昨天晚上在第二人生中参加了Amazon开发者小组讨论, 讨论主题有关Amazon的web服务: EC2, S3, ECS, SQS和FPS.

时间: 2007年9月7日
地点: http://slurl.com/secondlife/Amazon%20Developers%201/230/149/21

[10:01] Jeffronius Batra: Alright, good morning everyone!
[10:01] Alerque Thorne: goodmorning
[10:01] You: sorry, ciemaar.
[10:01] Ciemaar Flintoff: go morning
[10:01] Jeffronius Batra: Good to be back in-world; I was on vacation for most of August.
[10:02] Chase Marellan: Lucky man.
[10:02] cperciva Perfferle: Vacation? What’s that?
[10:02] Jeffronius Batra: I am going to do my best to stick to a regular schedule even when traveling (who needs sleep, anyway?)
[10:02] Jeffronius Batra: So we have some new folks here, and thanks for coming.
[10:02] Jeffronius Batra: In these chats we talk about the Amazon Web Services — EC2, S3, ECS, SQS, and FPS. I do my best to answer questions, and can also followup offline if need be.
[10:03] Jeffronius Batra: We have had several requests to do formal AWS presentations in here and I definitely want to do that sometime soon.
[10:03] Jeffronius Batra: Just need to get past my next big batch of travel — 2.5 week in Europe!
[10:04] Jeffronius Batra: So let’s start in with Q&A, and see where it goes!
[10:04] cperciva Perfferle: ok, I’ll start with a question: When will it be possible to pay for AWS services from an FPS account?
[10:05] Jeffronius Batra: Cperciva, very interesting question — how would that be better than simply charging AWS to your credit card? Would you want to do this so that you could pay from a bank account?

[10:06] cperciva Perfferle: It doesn’t matter to me yet, since you don’t have non-US accounts yet, but the last thing I want to end up doing is having people pay me in USD via FPS, get a bad exchange rate converting that to CAD, then pay for AWS on my credit card and pay an even worse exchange rate
[10:06] Jeffronius Batra: Ah, I see. Basically to avoid a double hit on currency exchanges. I will talk to the FPS product manager and will post the answer to the blog.
[10:07] Tobias Daguerre: hi
[10:07] Jeffronius Batra: Hi Tobias…
[10:07] Jeffronius Batra: I am still kind of new to FPS — it debuted during my time off and I haven’t spent a whole lot of time learning about it yet.
[10:07] cperciva Perfferle: yes, that’s exactly it. As I say, there’s no urgency, but assuming that you’re going to add support for international accounts at some point, it would be great if this can happen at or before that point.
[10:08] cperciva Perfferle: (and I’m sure Amazon will be happy to not pay credit card transaction costs on EC2/S3 billing, too)
[10:08] Jeffronius Batra: That’s great feedback. One thing you all should know is that the things I learn in this chat will be summarized and sent to the appropriate product managers within hours. So this is a great way for you to have some very direct input into our product planning process.
[10:09] Jeffronius Batra: I was at the SLCC (Second Life Community Convention) in Chicago last month and met a lot of AWS users there. In fact, between 1/3 and 1/2 of the attendees at my session knew what AWS was!
[10:10] Jeffronius Batra: A lot of the virtual worlds people are very excited by EC2 and by the prospect of on-demand world creation.
[10:10] Ciemaar Flintoff: so, what I’m wondering is how people are using SQS, seems like it would be handy in Second Life or metaverse development in general
[10:10] Ciemaar Flintoff: anybody get that hooked up?
[10:10] Jeffronius Batra: CIemaar, people are using SQS in a couple of different ways.
[10:11] Jeffronius Batra: One way is to use it as the bridge between internal and external processing. You could envision a system built from one or more EC2 instances, reading “work requests” from a queue. Outside applications would post requests to the queue.
[10:11] Jeffronius Batra: A work request could be something like “read document at URL http:// ??, scan it for keywords, and then store the results in S3.”
[10:12] Jeffronius Batra: Hi Jav and hi Cre80v.
[10:12] Jeffronius Batra: That’s one fairly interesting use case.
[10:12] Jeffronius Batra: Another is to use SQS as a variably sized buffer between different parts of an application, where each part might be itself running on one or more EC2 instances. That’s how Gigavox Media does it.
[10:13] Jeffronius Batra: Let me find a picture of their architecture….
[10:14] Ciemaar Flintoff: I saw their’s at your SLCC talk I believe, very impressive
[10:14] Jeffronius Batra: Using SQS lets them build very scalable apps.
[10:14] Jeffronius Batra: They actually monitor the processing time for each request and use that to regulate the number of EC2 instances.
[10:14] Jeffronius Batra: Ok, I found a good picture, hang on…
[10:16] Jeffronius Batra: Nothing like building the presentation on the fly…..
[10:16] Jeffronius Batra: Ok can you all see that?
[10:17] Jeffronius Batra: That’s the Gigavox Media architecture, build with EC2, S3, and SQS.
[10:17] Labsji Link: Hi Jeff is there a AWS library for javascript? as part of followup to last Chat I came across tiddlywiki.com a cool javascript based implementation of a wiki and more. It will be worthwhile to mash AWS with tiddlywiki. The mashup will be a good meeting ground for developers and nondeveloper users especially for mturk. TiddlyWiki +S3 be a website by iteslf!
[10:17] Jeffronius Batra: They upload raw podcasts, store them in S3, transcode them from MP2 to MP3, and then assemble them into finished podcasts with introductions, ads, and so forth.
[10:18] Jeffronius Batra: They do a fresh “build” of the podcasts every day, scaling up EC2 instances as needed.
[10:18] Lifing Krasner is Online
[10:18] Jeffronius Batra: Like I said, they monitor processing time (beginning to end) and use that to regulate the number of instances in operation.
[10:19] Jeffronius Batra: They told me that they spent less than $100 on services to build this entire architecture.
[10:19] Ciemaar Flintoff: wow, that include ther EC2 costs?
[10:19] Jeffronius Batra: I believe that it did. Doug Kaye was the architect (he’s been in these chats before).
[10:20] Jeffronius Batra: The very cool thing about these EC2-based models is that costs rise directly in line with usage.
[10:20] Jeffronius Batra: You don’t have to buy a big pile of servers just in case you need them.
[10:20] cperciva Perfferle: jeff, the costs rise in line with usage once your usage is high enough to need more than once instance.
[10:21] Ciemaar Flintoff: yeah, I ha thought that EC2 was course grained but maybe not at any reasonable scale
[10:21] cperciva Perfferle: I’m still waiting for smaller/cheaper instances to be available.
[10:21] Jeffronius Batra: Well, they don’t need any instances at all sometimes.
[10:21] Jeffronius Batra: How small of an instance would you like? Do you want less RAM, less CPU, less disk?
[10:22] cperciva Perfferle: yes to all of the above.
[10:22] Ciemaar Flintoff: well, I do some of my hosting with dedicated virtual servers, very fine grained, pay by the clock cycle and always up
[10:22] Jeffronius Batra: Ok, I will pass that along. I know we are working toward both larger and smaller instances.
[10:22] cperciva Perfferle: something like 1/16th or 1/40th of a physical box, instead of a quarter of a box
[10:23] Jeffronius Batra: Ciemaar, what would the effective stats be on something like that? Less than 1 GHz of processing?
[10:23] Ciemaar Flintoff: yeah, rather much less
[10:23] Jeffronius Batra: Wow, Cpercival, I think my digital camera has more processing and RAM than that!
[10:23] cperciva Perfferle: It looks like gigavox has all of their user-facing servers hosted traditionally
[10:23] cperciva Perfferle: I’d like to be able to have everything on EC2, including hosting a website and email and DNS
[10:24] Ciemaar Flintoff: I can e-mail you stats on my dev box, but basically it scales smoothly up to a 1.5GHz box if i use it heavily
[10:24] cperciva Perfferle: (which also requires a static IP address!)
[10:24] Jeffronius Batra: We are working on some refinements to how we deal with IP addresses which should address Cperciva’s request.
[10:24] Jeffronius Batra: Perhaps someone could create a second -tier virtual hosting business on top of EC2.
[10:25] Jeffronius Batra: We do see ourselves as creating the very low-level, vertical services, and leaving room for lots of other developers to innovate around and on top of what we have done.
[10:25] cperciva Perfferle: interesting idea, I hadn’t thought of that
[10:25] cperciva Perfferle: it would require a lot of flexibility with IP addresses, though
[10:26] Ciemaar Flintoff looks at the server guys section of his address book.
[10:27] cperciva Perfferle: I’d much rather be able to get a small (e.g., 100MHz, 128MB RAM, 10GB disk) EC2 instance for myself and have it running permanently
[10:27] cperciva Perfferle: It makes system administration much simpler if everything is an EC2 instance.
[10:27] Jeffronius Batra: Ok, that’s very helpful. Now how about large instances, more RAM, perhaps 64 bit? Anyone interested in those yet?
[10:27] You: yup, i need this option 2
[10:28] cperciva Perfferle: jeff, I’ve seen lots of people in the forums asking for that — but I don’t need anything larger myself
[10:28] Flip Matova: Yes, as usual we want larger instances. Not necessarily 64 bit though.
[10:28] Jeffronius Batra: Ciemaar, did we get to talk at SLCC? I met so many people there that I can’t keep track!
[10:29] Ciemaar Flintoff: Very briefly, my realname is Andy Fundinger, my cards probably in your stack
[10:29] Jeffronius Batra: Ah yes, we did. And we also connected on LinkedIn (all of these names are taxing my old brain).
[10:30] Jeffronius Batra: It is interesting that most of the questions we get here are about EC2!
[10:30] Ciemaar Flintoff: yeah, I went through all my networking services after SLCC–I feel so connected now

[10:31] Jeffronius Batra: Any questions about S3 or ECS?
[10:31] cperciva Perfferle: Jeff, when do we get an fsync / “have you propagated yet?” feature?
[10:31] cperciva Perfferle: (yes, I’m going to keep on asking until it arrives)
[10:31] Labsji Link: Is there a Javascript AWS library?
[10:32] Alerque Thorne: Jeffronious can I ask about the network situation behind EC2? I’ve just started playing around with the service and have some concerns about DDOS. I have been considering using it to scale some of my front ends, but I pretty receive targeted DDOS action.
[10:33] Jeffronius Batra: Labsji, JavaScript AWS, hmmmm. The Amazon Mashups book has a lot of good examples - http://www.amazon.com/Amazon-com-Mashups-Francis-Shanahan/dp/0470097779
[10:33] Jeffronius Batra: Alerque, are you concerned about our ability to handle the DDOS, or our response to it?
[10:34] Alerque Thorne: Actually my first concern was bandwidth costs. Is there any filtering done before the metering?
[10:34] Alerque Thorne: Or would all DDOS traffic be my responsibility to pay for and then filter?
[10:34] Jeffronius Batra: I don’t believe there is. We would meter it at the point where it reaches your EC2 instance.
[10:35] Jeffronius Batra: Is there a better way to do it?
[10:35] Alerque Thorne: I currently have my services on a 45mb OC3. It is not unusual for that to get filled up with traffic durring an attack. This doesn’t sound like a fun thing to have to cope with on EC2 if I use it on the forward side.
[10:36] Jeffronius Batra: That’s a lot of bits, even more than my teenagers use at home.
[10:36] Jeffronius Batra: I would have to talk to the EC2 team about this. If you want, you can send me an email (jbarr AT amazon.com) with more details.
[10:36] Alerque Thorne: Yes, it’s a pile. It doen’t happen all the time, but probably monthly.
[10:37] Jeffronius Batra: Are you making the wrong people get mad at you??
[10:37] Alerque Thorne: Simple answer: yes.
[10:37] Jeffronius Batra: Ok….
[10:38] Jeffronius Batra: I know that we support high-traffic users like Justin.tv and a bunch of web crawlers.
[10:38] Alerque Thorne: Most of it is pretty basic syn floods or other easily recognized stuff, my question would be you had any sort of filtering for this before traffic entered the ec2 network.
[10:39] Jeffronius Batra: Justin.tv did something like 3000 simultaneous video streams out of EC2.
[10:39] Jeffronius Batra: Ah, that kind of filtering (I was thinking IP address, Doh).
[10:39] Alerque Thorne: Wow, that’s a nice collection of bits too!
[10:39] Jeffronius Batra: I suspect that we do, but will confirm with the EC2 networking guru (he’s down the hall from me).
[10:39] Jeffronius Batra: Yeah, it is fun to see a bunch of folks do high-scale things like that from within the confines of an apartment.
[10:40] Jeffronius Batra: NIR — No Infrastructure Required.
[10:40] Jeffronius Batra: Also, the Second Life client downloads are served up from S3.
[10:40] Jeffronius Batra: They haven’t published new stats in a while, but the first time they did this they consumed 1 TB within 24 hours. That was a while ago and I have to assume that it has only grown since then.
[10:41] Alerque Thorne: MIR (minimal infustructure requred) — there are outstanding ip address/dns issues before we can ditch our hard boxes.
[10:41] Ciemaar Flintoff: and their images too I noticed
[10:41] cperciva Perfferle: So that’s why you like 2nd life so much — Amazon makes a profit every time someone new starts using 2nd life!
[10:42] Jeffronius Batra: Actually, the amazing thing about the downloads is that we didn’t know about it until they blogged about it. They wrote a nice post, “Amazon S3 for the Win” (http://blog.secondlife.com/2006/10/26/amazon-s3-for-the-win/).
[10:42] Jeffronius Batra: S3 usage is large enough that an extra 1 TB per day wasn’t something which made us panic.
[10:42] Jeffronius Batra: But I liked SL long before they started using it.
[10:43] Jeffronius Batra: There are some rumors that the SL server will run on EC2 at some point. This would be cool if/when they open source it.
[10:43] Jeffronius Batra: Once again, worlds on demand.
[10:43] Jeffronius Batra: Hi Marvel.
[10:43] Marvel Ousley waves
[10:44] Tobias Daguerre: i’ve got a question regarding accessing S3 buckets from an EC2 instance
[10:44] Jeffronius Batra: Ok, go for it.
[10:45] Tobias Daguerre: does amazon plan to provide anything like nfs for it?
[10:47] Ciemaar Flintoff: ok, SQS question, does anyone offer a Queue administrator too?
[10:47] Jeffronius Batra: We don’t have anything like that ourselves (NFS = Network File System, developed by Sun), but several developers have built some pretty interesting implementations themselves. There’s more information in the S3 forum.
[10:47] Ciemaar Flintoff: I’d like to poke at it some without having to question my code.
[10:48] Jeffronius Batra: CIemaar, I saw a very comprehensive tool from RightScale yesterday - it handles EC2, S3, and SQS.
[10:48] Jeffronius Batra: There is more info at http://info.rightscale.com/ .
[10:49] Ciemaar Flintoff: sounds good
[10:49] Jeffronius Batra: In the “shameless plug” department, a friend of mine just published his book on SL — http://www.amazon.com/Second-Life-Guide-Virtual-World/dp/0321501667 . I was a tech reviewer and I also wrote the chapter on LSL scripting.
[10:49] Jeffronius Batra: It is a good book, even my chapter.
[10:49] Jeffronius Batra: BTW, if you have typed into this chat and am ok with my publishing the transcript, please give me permission to do so. Just say “ok to publish”
[10:50] cperciva Perfferle: ok to publish
[10:50] Jeffronius Batra: I may post these into the forums to avoid clutting up the blog.
[10:50] Ciemaar Flintoff: ok to publish
[10:50] Alerque Thorne: ok to publish
[10:50] Jeffronius Batra: Awesome, thanks.
[10:50] Chase Marellan: ok to publish
[10:51] Jeffronius Batra: Is anyone here from London or Berlin? I still have some open meeting time in both of those cities and am always happy to meet with developers when I travel.
[10:51] Flip Matova: ok to publish
[10:52] Labsji Link: ok to publish
[10:53] Tobias Daguerre: ok to publish
[10:53] Jeffronius Batra: This goes for everyone on my team — Mike, Jinesh, and I always seek out 1:1 meetings as part of our travels. Our schedule is at http://evangelists.wetpaint.com/ and you can simpy edit the appropriate wiki page.
[10:53] Jeffronius Batra: We find that the direct contact with developers, be it through SL, face to face meetings, or forums, is really valuable.
[10:54] Chase Marellan: What do you do in those meetings? Just what we do here? Or is it more targeted?
[10:54] Chase Marellan: Or do you just have dinner.
[10:54] Jeffronius Batra: We generally discuss specific questions and issues, sometimes things that they would rather not share in public.
[10:55] Jeffronius Batra: I like to get a good understanding of how they are using our services — what works and even more importantly what doesn’t work.
[10:55] Chase Marellan: Gotcha, thanks.
[10:55] Jeffronius Batra: We write up what we learn each day and get it into the company ASAP.
[10:56] Chase Marellan: Cool.
[10:56] Second Life: You have left the group ‘Inside Millions of Us’.
[10:56] Second Life: You have left the group ‘Visual Studio’.
[10:56] Jeffronius Batra: Yes, it works wonders to be able to talk about real customers when making the case for new features. No ivory tower for us.
[10:56] Jeffronius Batra: And we do have dinner (or lunch or breakfast) too!
[10:56] Chase Marellan: Obviously it’s working.
[10:57] Jeffronius Batra: Oh yeah, definitely.
[10:58] Jeffronius Batra: Ok, I need to head downstairs for a RL meeting now, but feel free to chat with each other. Thanks so much for coming, everyone. As soon as I figure out a good place for them, the transcripts should be posted.
[10:58] Chase Marellan: Thanks, Jeff!
[10:59] Jeffronius Batra: Hope to see you next week (and tell your friends, blog about this, and so forth). The more the merrier.
[10:59] You: ty. Jeff..
[10:59] Alerque Thorne: Yes, thanks.
[10:59] Jav Meads: Cheers Jeff
[10:59] Labsji Link: Thanks jeff
[10:59] Ciemaar Flintoff: Thank you, see you next week?
[10:59] Flip Matova: Thanks Jeff!
[10:59] cperciva Perfferle: thanks jeff — good to see that these meetings are continuing now that you’re back from vacation!
[10:59] Jeffronius Batra: Yes, I will be in Europe but I know that the hotel works because I’ve been there before.
[11:00] Jeffronius Batra: See you next time!
[11:00] Marvel Ousley: Bye!