Discussion Forums



Thread: EC2 API outage

Welcome, Guest Help
Login Login


Permlink Replies: 40 - Pages: 3 [ 1 2 3 | Next ] - Last Post: Oct 4, 2008 10:59 AM by: Colin Rhodes
James@AWS

Posts: 623
Registered: 6/22/06
EC2 API outage
Posted: Sep 29, 2007 4:21 AM PDT
  Click to reply to this thread Reply

The EC2 control and monitoring API is currently experiencing an outage.

API calls are currently failing, but the majority of running instances appear unaffected.

We are investigating and will post more details as soon as we have them.

Regards
James


Thorsten von Eicken
RealName(TM)


Posts: 633
Registered: 3/24/06
Re: EC2 API outage
Posted: Sep 29, 2007 4:24 AM PDT   in response to: James@AWS
  Click to reply to this thread Reply

I was just trying to track this down. Thanks for being proactive and posting here. For everyone else, we're seeing this as the following error message:
"Unavailable: Unable to process request - no CM servers available"
Thorsten


stinger67

Posts: 30
Registered: 8/2/07
Re: EC2 API outage
Posted: Sep 29, 2007 4:28 AM PDT   in response to: James@AWS
  Click to reply to this thread Reply

Copy that James...


Thanks for the rapid response...

However, we have one instance which "went houdini"...
during a download.

Can't use the api, can't see the id number...


I'll keep checking back for a status on the tool belt!


renkooawis

Posts: 31
Registered: 1/9/07
Re: EC2 API outage
Posted: Sep 29, 2007 4:41 AM PDT   in response to: James@AWS
  Click to reply to this thread Reply

this is very bad.  we have three instances (maybe more) that have gone down just now and we can't fire up replacement instances without these tools...

i'm wondering if there are large problems if instances are going down and the API servers are also unavailable.

i'll post about the specific instance problems we know of but please, please get the API tools back up.


Thorsten von Eicken
RealName(TM)


Posts: 633
Registered: 3/24/06
Re: EC2 API outage
Posted: Sep 29, 2007 5:02 AM PDT   in response to: renkooawis
  Click to reply to this thread Reply

[message deleted]
Thorsten
Message was edited by: Thorsten von Eicken

zoopla

Posts: 18
Registered: 3/27/07
Re: EC2 API outage
Posted: Sep 29, 2007 5:52 AM PDT   in response to: Thorsten von Ei...
  Click to reply to this thread Reply

same problem here - at least one instance down and other instances slow to respond

Attila@AWS

Posts: 162
Registered: 11/2/06
Re: EC2 API outage
Posted: Sep 29, 2007 7:01 AM PDT   in response to: James@AWS
  Click to reply to this thread Reply

Hi everyone,
quick update: we are working as quickly as safely possible to restore functionality.
Please stay tuned for updates.
Thanks,
    Attila...

enomaly

Posts: 444
Registered: 9/3/06
Re: EC2 API outage
Posted: Sep 29, 2007 7:22 AM PDT   in response to: Attila@AWS
  Click to reply to this thread Reply

thanks for keeping us updated.

reuven
http://www.enomalylabs.com


Thorsten von Eicken
RealName(TM)


Posts: 633
Registered: 3/24/06
Re: EC2 API outage
Posted: Sep 29, 2007 7:39 AM PDT   in response to: Attila@AWS
  Click to reply to this thread Reply

We're seeing the API calls working for a few minutes now. There's been a brief up/down a few minutes earlier, let's hope for the best...
    Thorsten


Thorsten von Eicken
RealName(TM)


Posts: 633
Registered: 3/24/06
Outage feedback
Posted: Sep 29, 2007 7:47 AM PDT   in response to: Attila@AWS
  Click to reply to this thread Reply

The prompt initial notification by James was very appreciated!
I suggest you set-up a policy to follow-up with a post every 20 or 30 minutes, even if it's to say "no news". What I've found helpful in the past is to communicate an ETA, even if in vague terms, such as "any moment" or "probably not in the next half hour". This lets users decide what work-arounds they should start putting in place.

Regards,
Thorsten



Attila@AWS

Posts: 162
Registered: 11/2/06
Re: EC2 API outage
Posted: Sep 29, 2007 7:58 AM PDT   in response to: Thorsten von Ei...
  Click to reply to this thread Reply

Everyone,
we are continuing to work on this. We don't have an ETA yet. We will update this thread as soon as we know more.
Apologies for the inconvenience, and thanks for your patience.
Thanks,
    Attila...

Attila@AWS

Posts: 162
Registered: 11/2/06
Re: EC2 API outage
Posted: Sep 29, 2007 8:09 AM PDT   in response to: James@AWS
  Click to reply to this thread Reply

Hi everyone,
just a quick update: at this point API access is restored.
We will post more details as soon as we have them.
Please let us know if you see further issues.
Thanks,
    Attila...


Brian

Posts: 67
Registered: 7/25/07
Re: EC2 API outage
Posted: Sep 29, 2007 8:24 AM PDT   in response to: Attila@AWS
  Click to reply to this thread Reply

Hi,

I can get console output

brian@ubuntu-server:~$ ec2-get-console-output i-79e80510
i-79e80510
2007-09-29T15:21:52+0000


but am unable to access by ssh or ping the instance. Will I reboot the instance or wait?

Best Regards,
Brian

Attila@AWS

Posts: 162
Registered: 11/2/06
Re: EC2 API outage
Posted: Sep 29, 2007 8:29 AM PDT   in response to: Brian
  Click to reply to this thread Reply

Brian,
give us a moment, we'll take a look. Don't reboot the instance.
Take care,
    Attila...


Attila@AWS

Posts: 162
Registered: 11/2/06
Re: EC2 API outage
Posted: Sep 29, 2007 8:40 AM PDT   in response to: Attila@AWS
  Click to reply to this thread Reply

Hi Brian,
unfortunately, this instances was lost during our outage.
We will not be able to recover it.
The service is again taking requests, and you should be able to launch a replacement instance
Apologies for the inconvenience.
Thanks,
    Attila...



Point your RSS reader here for a feed of the latest messages in all forums