|
Discussion Forums
|
Thread: Speeds incredibly slow today?
 |
This thread is locked - replies are not allowed.
|
|
|
|
Replies:
34
-
Pages:
3
[
1
2
3
| Next
]
-
Last Post:
Jun 26, 2008 6:22 PM
by: Kathrin@AWS
|
|
|
Posts:
36
Registered:
6/2/08
|
|
|
|
Speeds incredibly slow today?
Posted:
Jun 2, 2008 12:53 PM PDT
|
|
|
I've been getting reports of very slow S3 download speeds (between 10 and 25k/s) from customers in various parts of the country ranging from Maine to Seattle (and EU customers) today. Is there a reason for this that hasn't been reported/commented on yet?
Speeds seem slower for me as well from one of our production servers on Level3 in Seattle. Typically I max our download speeds from that connection at up to 5MB/s, today things seem to be downloading at ~700k/s. I've tested other file sources and only S3 seems to be slow.
Are there any reported outages/slowdowns/problems that might explain this?
Rob
|
|
Posts:
3
Registered:
6/2/08
|
|
|
|
Re: Speeds incredibly slow today?
Posted:
Jun 2, 2008 1:54 PM PDT
in response to: entityat
|
|
|
We are basing our Javascripts in S3 and they are the one to produce traffic to our data center.
we are experiencing drop downs of the traffic to the data center and have reason to believe it is connected to the same issue.
we had one drop between 9-10am EST and another between 12:30 and 13:00 EST
Please report if you have issues there.
|
|
Posts:
36
Registered:
6/2/08
|
|
|
|
Re: Speeds incredibly slow today?
Posted:
Jun 6, 2008 10:42 AM PDT
in response to: entityat
|
|
|
Guys,
We're getting increased reports of this. The lack of response is really surprising even though I know we don't have a lot of technical reports to give, I count 3 threads on the front page of these forums talking about S3 being incredibly slow. What can I do to help my users diagnose this better and figure out why things are so ridiculously (we're talking 15-20k/s) slow? It's at the point that I've had to start looking into alternatives to S3 because while the performance now is really quite poor, the lack of response from anyone about what is going on is ridiculous.
Rob
|
|
Posts:
163
Registered:
2/8/06
|
|
|
|
Re: Speeds incredibly slow today?
Posted:
Jun 6, 2008 8:37 PM PDT
in response to: entityat
|
|
|
First, we take customer reports like this very seriously. We continuously work with customers via our forums to identify what may be causing behavior like this. We accumulate this data looking for general issues with Amazon S3, even those difficult to see in general statistics. For those of you who would like to be a part of our additional research, we encourage you to upload TCP dumps and any associated information, such as your error logs, to the following Amazon S3 bucket:
http://s3.amazonaws.com/0712upload/.
Secondly, we can tell you that we have not seen any general error rate or latency issues. Outside of those events we've posted on the AWS Service Health Dashboard, availability and latencies have been normal. Our availability has remained - and continues to remain - above our stated SLA (99.9%). We are committed to providing an excellent level of service. We will continue working on lowering latency and ever smaller rates of error. Some level of error is unavoidable given the nature of distributed computing.
If you’ve posted to this or other related threads regarding this issue, please send us as much information as you can by uploading it to the bucket mentioned above, and we’ll work with you to identify what may be causing the behavior you’re seeing.
Thanks,
Kathrin
|
|
Posts:
36
Registered:
6/2/08
|
|
|
|
Re: Speeds incredibly slow today?
Posted:
Jun 9, 2008 11:33 AM PDT
in response to: Kathrin@AWS
|
|
|
Kathrin,
I definitely appreciate the response. I'm a bit annoyed to find that despite there being posts all over your forums and having the number of customers complaining about speeds each day doubling, that you guys aren't seeing anything on your end.
Could you point me to the best location to provide some advanced troubleshooting logs? Unfortunately this means I'm going to need to get my customers to become tech ninjas to upload tcpdumps and the like, which is incredibly frustrating.
The reports from my customers seem to be widespread and not geographically based. I'm going to do what I can to track this down, but I'm really rather disappointed that there are so many posts in the forums right now about timeouts and slowdowns (which seem to be linked?) that have the response from Amazon that nothing is wrong.
Rob
|
|
Posts:
36
Registered:
6/2/08
|
|
|
|
Re: Speeds incredibly slow today?
Posted:
Jun 9, 2008 2:18 PM PDT
in response to: entityat
|
|
|
I just remotely connected to a customer's computer to verify. He was having downloads of about 20k/s.
Here are his speedtest results:
http://www.speedtest.net/result/281946945.png
I connected to his computer, verified that the connections were indeed slow. I opened a terminal window (he was using OSX) and tried to CURL to a public bucket rather than our authenticated bucket. Speeds semeed good (1.5MB/s). Then I attempted downloads from our authenticated bucket and they were back to normal.
He experienced no slowdowns from any sites other than S3 during this period.
Is there some possibility of misrouting to one or more of the endpoints?
So far, I have had reports of excessive slowness (10k/s when downloads are typically 600k/s or higher) from customers in the following areas:
Colorado
Drexel University in Philadelphia
Maine
Ireland (12k/s to the US bucket -- normally 100k/s)
Norway (14k/s to the US bucket -- normally 600-800k/s)
Switzerland (14k/s to the US bucket -- normally 300k/s)
I would really appreciate having some idea of where to look for help on this. I really love S3, but the inability to track what is definitely a problem is really frustrating. These are all reports from customers who are reporting that speeds are normally VERY fast, and are currently VERY slow. I can't tell all of them that they're imagining things, unfortunately.
Rob
|
|
Posts:
33
Registered:
8/16/07
|
|
|
|
Re: Speeds incredibly slow today?
Posted:
Jun 9, 2008 4:26 PM PDT
in response to: entityat
|
|
|
All of my sites that are hosted by s3 is not loading (timing out) or barely loading. I noticed this trend yesterday afternoon. It began to work again late evening through the morning however returned this afternoon (~3pm est). I appreciate an Explanation
or a solution or we may have to find other options.
thanks
|
|
Posts:
913
Registered:
12/13/06
|
|
|
|
Re: Speeds incredibly slow today?
Posted:
Jun 9, 2008 6:06 PM PDT
in response to: Courtney Campbell
|
|
|
Thanks, for the additional data, entityat. If you could please also post the name of the bucket/object experiencing the issue, we'll investigate further. It would also be helpful if you could provide the HTTP headers and the body of an Amazon S3 response. Any TCP dumps and error logs you could provide by uploading those to the bucket we've referenced above would also be very helpful. We'll do whatever we can to identify and help you resolve the issue you and your customers are seeing.
Courtney, any additional data you can provide will also help us to troubleshoot the issues you have reported.
|
|
Posts:
1
Registered:
6/9/08
|
|
|
|
Re: Speeds incredibly slow today?
Posted:
Jun 9, 2008 11:04 PM PDT
in response to: entityat
|
|
|
Our response rates are typically between .05 seconds to 3 seconds per request. We started to see increased latency yesterday, and are now running consistently in the 30-35 seconds request response time. This process is only grabbing a small 4 byte text file.
Prior to today, the longest response time we've seen (over the last week) has been 29 seconds on one occassion. 99.9% of our response times have been under 5 seconds. Right now, we're running 100% of our responses above 30 seconds for the last hour or two.
|
|
Posts:
36
Registered:
6/2/08
|
|
|
Posts:
6
Registered:
6/11/08
|
|
|
|
Re: Speeds incredibly slow today?
Posted:
Jun 11, 2008 12:53 PM PDT
in response to: entityat
|
|
|
I'm having an intermittent latency coming from Media Temple to S3 now. Sometimes it's fine. But other times it is so slow that the script times out when requesting a file off of S3. It has gotten really bad in the last 24 hours. If anyone has a suggestions on how to mitigate this problem (short of moving off of S3), I'd love to hear it.
|
|
Posts:
913
Registered:
12/13/06
|
|
|
|
Re: Speeds incredibly slow today?
Posted:
Jun 11, 2008 1:46 PM PDT
in response to: Benny O. Smith
|
|
|
Entityat, we appreciate the additional data, however, both of the IP's given are currently resolving to the East, so there doesn't appear to be a routing problem. Any luck getting a tcpdump from one of your users?
Tazzben, any additional data you can provide will also be very helpful. If possible, please post the response headers/body of a request or upload a tcpdump to the following Amazon S3 bucket:
http://s3.amazonaws.com/0712upload/
Thanks,
Justin
|
|
Posts:
7
Registered:
6/11/08
|
|
|
|
Re: Speeds incredibly slow today?
Posted:
Jun 11, 2008 2:45 PM PDT
in response to: entityat
|
|
|
I can second the issue, which is something we've been experiencing for a bit. For most of the time (i'd say 95%) things are like expected, however.. Every now and then everything goes really really slow.
Our bucketname is storage.filemobile.com.
If we'll keep on seeing this we'll probably end up doing a tcpdump of this and sending it over. Its a bit tough to catch though, as it doesn't always happen.
It does seem to be related to our bucket. Today we were experiencing excruciating speeds, which happened from different networks. (but all from the Toronto area..)
If there are, besides making TCP dumps and sending it over, some general best practices we can apply, I'd love to hear them.
|
|
Posts:
163
Registered:
2/8/06
|
|
|
|
Re: Speeds incredibly slow today?
Posted:
Jun 11, 2008 4:45 PM PDT
in response to: Justin@AWS
|
|
|
Just a quick update to let everyone know that we are actively working with the customers who have raised these issues, and are looking at the data they've provided. We’ll continue to do so and will post back to this thread with anything broadly applicable.
Thanks,
Kathrin
|
|
Posts:
36
Registered:
6/2/08
|
|
|
|
Re: Speeds incredibly slow today?
Posted:
Jun 11, 2008 6:51 PM PDT
in response to: Kathrin@AWS
|
|
|
Yeah I wanted to post in here to say that Justin and John from AWS have been in touch and very helpful in working with me to try to track down the problems. The slow speed reports still remain, but I've got a path that I can take now and provide to my customers to help us get data to check out the issues.
Rob
|
|
|
|
Sign in to the AWS Management Console
Create an AWS Account
