|
Discussion Forums
|
Thread: cloudfront is down in Europe
|
|
|
Replies:
19
-
Pages:
2
[
1
2
| Next
]
-
Last Post:
Jul 1, 2009 2:38 AM
by: fredericsidler
|
Threads:
[
Previous
|
Next
]
|
|
Posts:
22
Registered:
4/19/07
|
|
|
|
cloudfront is down in Europe
Posted:
Jun 30, 2009 12:45 AM PDT
|
|
|
Our static files are not there. It is working from US and Australia, but not from Europe. We are in Switzerland.
Not mention of that in status.aws.amazon.com
|
|
Posts:
17
Registered:
4/13/08
|
|
|
|
Re: cloudfront is down in Europe
Posted:
Jun 30, 2009 12:58 AM PDT
in response to: fredericsidler
|
|
|
Agreed, the nameservers for cloudfront are not working, as the following queries show:
% dig -t ns cloudfront.net
; <<>> DiG 9.4.3-P1 <<>> -t ns cloudfront.net
;; global options: printcmd
;; connection timed out; no servers could be reached
% dig d1k4qptiem4efl.cloudfront.net
; <<>> DiG 9.4.3-P1 <<>> d1k4qptiem4efl.cloudfront.net
;; global options: printcmd
;; connection timed out; no servers could be reached
I would very much like it if amazon could comment on this very, very soon. I'm going to write a guide on how to configure varnish to behave like cloudfront on an ec2 instance. I don't hope I need to finish it.
|
|
Posts:
252
Registered:
7/1/08
|
|
|
|
Re: cloudfront is down in Europe
Posted:
Jun 30, 2009 1:03 AM PDT
in response to: madssj
|
|
|
Starting at 11:50 PM PDT we began experiencing networking issues in our Frankfurt edge location. We have rerouted traffic to our other European edge locations.
We are actively working to resolve the issue in Frankfurt.
Regards,
The CloudFront Team
|
|
Posts:
17
Registered:
4/13/08
|
|
|
|
Re: cloudfront is down in Europe
Posted:
Jun 30, 2009 1:09 AM PDT
in response to: Tal@AWS
|
|
|
I'm still not seeing any dns replies, do you have any kind of eta for when the dns will be answering again?
|
|
Posts:
17
Registered:
4/13/08
|
|
|
|
Re: cloudfront is down in Europe
Posted:
Jun 30, 2009 1:37 AM PDT
in response to: madssj
|
|
|
I'm now getting dns replies, which sends a 307 Temporary Redirect to s3-external-3.amazonaws.com, so it works for some distributions and sends redirects for others. Thanks so far.
|
|
Posts:
2
Registered:
6/30/09
|
|
|
|
Re: cloudfront is down in Europe
Posted:
Jun 30, 2009 1:40 AM PDT
in response to: fredericsidler
|
|
|
This is a HUGE problem for us. Really huge.
I hope it'll be fixed asap.
I would also like to know what e why this happened.
Just to know how to fix it next time.
Cloudfront its a wonderful service, but you need to trust it.
Ciao.
Marco.
"Sh*t happens" - Forrest Gump
|
|
Posts:
17
Registered:
4/13/08
|
|
|
|
Re: cloudfront is down in Europe
Posted:
Jun 30, 2009 1:54 AM PDT
in response to: linkgroup
|
|
|
In case you're still having problems, could you check what the dns is resolving to and post the results together with your location.
|
|
Posts:
17
Registered:
4/13/08
|
|
|
|
Re: cloudfront is down in Europe
Posted:
Jun 30, 2009 2:03 AM PDT
in response to: Tal@AWS
|
|
|
Now that you seem to have solved most of this mess, please give me a very, very good explanation of why it took you from 11:50 PM PDT to around 1:05 AM PDT to solve the problem.
Also, why on earth did you not inform about the outage on your status page, but waited until the twispher cought it, before taking action?
Unless the answers to this are damn good, I (and I think I speak for a lot of people) will be thinking twice about my current cloudfront and aws usage.
I don't really mind the outage, but I'm pissed about the lack of information and time you spent on resolving the problem.
Message was edited by: madssj
|
|
Posts:
22
Registered:
4/19/07
|
|
|
|
Re: cloudfront is down in Europe
Posted:
Jun 30, 2009 2:12 AM PDT
in response to: madssj
|
|
|
Answer welcome too. It tooks us many minutes to find out what the problem was. And first reflex I got when I have a problem with AWS is status.aws.amazon.com
There was nothing related to Cloudfront at the moment I wrote the first message in this forum.
For me this is not something difficult to do. Put a simple file on each of your edge location and monitor them. A simple nagios setup can do that and alert you in case of problem.
I also would like to know why there was no mention of this problem on your status page.
|
|
Posts:
252
Registered:
7/1/08
|
|
|
|
Re: cloudfront is down in Europe
Posted:
Jun 30, 2009 2:32 AM PDT
in response to: madssj
|
|
|
Madssj,
We started investigating the issues in Frankfurt at 11:50 PDT when our alarming alerted us of the issue. We failed away most traffic by 12:12 after our investigation made it clear that we had a networking issue in Frankfurt. All remaining traffic was routed way from Frankfurt by 1:05 am. This did take longer than we would normally have expected. We are continuing to investigate the issue further.
We apologize that it took us longer than we would have liked to post to the AWS dashboard page. We spent that time actively focused on resolving the underlying root cause in order to effectively post to the AWS dashboard page.
We would also like to follow up with you separately via PM to understand some specifics around what you saw.
Regards,
The CloudFront Team
|
|
Posts:
23
Registered:
11/2/07
|
|
|
|
Re: cloudfront is down in Europe
Posted:
Jun 30, 2009 2:52 AM PDT
in response to: madssj
|
|
|
madssj,
With regards to getting 307 temporary redirects to s3-external-3.amazonaws.com, that is not something cloudfront is setup to do. Can you PM me with output from your tests so we can investigate what was going on?
Also, please include the output from
dig identity.cloudfront.net txt
dig resolver-identity.cloudfront.net txt
Thanks,
CloudFront team
|
|
Posts:
17
Registered:
4/13/08
|
|
|
|
Re: cloudfront is down in Europe
Posted:
Jun 30, 2009 2:59 AM PDT
in response to: Tal@AWS
|
|
|
> We started investigating the issues in Frankfurt at 11:50 PDT when our alarming alerted us of the issue.
Then that's when you should have posted a notice about it. Right now, you're sending the signal that "there is nothing wrong", when there is. Right now, the status icon is still green for cloudfront, when it's very clear that there has been some service downtime and outage.
You really should focus on your service announcements, as the current dashboard is not very friendly. A mailing list would be
very
welcome, yes, it is an old, old, technology, but it works. Well.
> We apologize that it took us longer than we would have liked to post to the AWS dashboard page. We spent that time actively focused on resolving the underlying root cause in order to effectively post to the AWS dashboard page.
Yes, that is what I'd have said in your place as well. It doesn't matter that you guys were spending your time resolving the issues when the world does not know of them. When you know about a problem, tell about it. I assume that it takes well under 2 minutes to update the status page with "Service disruptions in europe", so when people go there looking for answers, they'll know that something is wrong, and and you're looking into it.
For an extra bonus, do things like "Next update in 20 minutes" if you have no clue about the ETA of a given problem, and actually do the update at that time. It doesn't matter if that update in inconclusive, but it matters a lot that it is there. Especially with services like your where there is no real place to turn in the face of trouble.
> We would also like to follow up with you separately via PM to understand some specifics around what you saw.
Please do.
|
|
Posts:
17
Registered:
4/13/08
|
|
|
|
Re: cloudfront is down in Europe
Posted:
Jun 30, 2009 3:02 AM PDT
in response to: DavidR@AWS
|
|
|
> With regards to getting 307 temporary redirects to s3-external-3.amazonaws.com, that is not something cloudfront is setup to do. Can you PM me with output from your tests so we can investigate what was going on?
That was actually a brainfart by me, as I was setting up a varnish instance to gracefully handle the outage. I seem to have forgot about that face when I posed that information.
It can be seen here:
> % curl -I
http://cdn.imholsten.com/v18/swf/splash.swf
> HTTP/1.1 307 Temporary Redirect
> x-amz-request-id: E0B3606E874E10B1
> x-amz-id-2: Xqy1byPE/cdtkTqD+Cgpet4KPns/GGQlf2TG6+X3JEXAqn8vf+63bE0Sc0K92jVc
> Location:
http://media.imholsten.com.s3-external-3.amazonaws.com/v18/swf/splash.swf
> Content-Type: application/xml
> Server: AmazonS3
> Date: Tue, 30 Jun 2009 08:37:12 GMT
> X-Varnish: 1291517409
> Age: 0
> Via: 1.1 varnish
> Connection: keep-alive
Which was basiclly a half configured varnish proxy pointing on s3.amazonaws.com. So feel free to disregard that.
|
|
Posts:
2
Registered:
6/30/09
|
|
|
|
Re: cloudfront is down in Europe
Posted:
Jun 30, 2009 3:30 AM PDT
in response to: madssj
|
|
|
> Then that's when you should have posted a notice about it.Right now, you're sending the signal that "there is nothing wrong",when there is. Right now, the status icon is still green forcloudfront, when it's very clear that there has been some servicedowntime and outage.You really should focus on your service announcements, as the current dashboard is not very friendly. A mailing list would be
very
welcome, yes, it is an old, old, technology, but it works. Well.
> Itdoesn't matter that you guys were spending your time resolving theissues when the world does not know of them. When you know about aproblem, tell about it. I assume that it takes well under 2 minutes toupdate the status page with "Service disruptions in europe", so whenpeople go there looking for answers, they'll know that something iswrong, and and you're looking into it.
> For an extra bonus, do things like "Next update in 20 minutes"if you have no clue about the ETA of a given problem, and actually dothe update at that time. It doesn't matter if that update ininconclusive, but it matters a lot that it is there. Especially withservices like your where there is no real place to turn in the face oftrouble.
I deeply agree with madssj. Great proposals.
This is the minimum I expect from a class A service.
Ciao.
Marco.
|
|
Posts:
7
Registered:
9/29/08
|
|
|
|
Re: cloudfront is down in Europe
Posted:
Jun 30, 2009 4:07 AM PDT
in response to: fredericsidler
|
|
|
Do any of you guys have any idea, how to set up nagios(or do it some other way), to be checking if CF is working? I know how to do it for simple file in one location, but since this is a CDN, the file can be ok on one of the servers but unavailable on the rest of the servers. In that case I need to switch my application to serve static content from my primary server or S3 instead of CF. But I need someone to tell me, that part of CF is not working.
|
|
|
|
Sign in to the AWS Management Console
Create an AWS Account
