|
Discussion Forums
|
Thread: Internal Error when autoscaler tries to launch instance
 |
This question is not answered.
Helpful answers available: 2.
Correct answers available: 1.
|
|
|
|
Replies:
7
-
Pages:
1
-
Last Post:
Oct 19, 2009 3:53 PM
by: John@AWS
|
|
|
Posts:
2
Registered:
10/13/09
|
|
|
|
Internal Error when autoscaler tries to launch instance
Posted:
Oct 13, 2009 1:16 PM PDT
|
|
|
Hi,
When the autoscale functionality tries to launch a new instance, it gets an internal error message. What is causing this?
Thanks,
Bjorn
ACTIVITY 6da4719e-3aac-4e76-95b3-3609eb3cb452 2009-10-13T19:58:07Z Failed Internal Error At 2009-10-13 07:55:00Z an instance was started in response to a difference between desired and actual capacity, increasing the capacity from 2 to 3.
ACTIVITY fc38a2d4-2be6-46d6-9273-fff0c6cfb916 2009-10-13T19:51:05Z Failed Internal Error At 2009-10-13 07:48:00Z an instance was started in response to a difference between desired and actual capacity, increasing the capacity from 2 to 3.
ACTIVITY 039e9198-d86a-4b13-8413-4f464e4ce083 2009-10-13T19:45:54Z Failed Internal Error At 2009-10-13 07:42:59Z an instance was started in response to a difference between desired and actual capacity, increasing the capacity from 2 to 3.
ACTIVITY a1ef5e86-8673-4312-af3a-c63f1f535dcb 2009-10-13T19:42:13Z Failed Internal Error "At 2009-10-13 07:38:36Z a breaching trigger explicitly set group desired capacity changing the desired capacity from 2 to 3. At 2009-10-13 07:38:36Z trigger DynamicxxTrigger breached high threshold value for CPUUtilization, 60.0, adjusting the desired capacity from 2 to 3. At 2009-10-13 07:38:59Z an instance was started in response to a difference between desired and actual capacity, increasing the capacity from 2 to 3."
|
|
Posts:
2,112
Registered:
7/10/08
|
|
|
|
Re: Internal Error when autoscaler tries to launch instance
Posted:
Oct 13, 2009 1:56 PM PDT
in response to: bjornhij
|
|
|
What AMI are you trying to launch?
Is it a DevPay AMI?
Is the Auto Scaling Group part of an ELB?
Does the AMI's architecture match the architecture of the instance type specified in the Launch Configuration?
|
|
Posts:
2
Registered:
10/13/09
|
|
|
|
Re: Internal Error when autoscaler tries to launch instance
Posted:
Oct 13, 2009 11:18 PM PDT
in response to: Shlomo Swidler
|
|
|
The AMI was bundled by me, a few weeks ago. The internal error occured 5 times last night, but later it succeeded. Also, it succeeded several times in the past.
The auto scaling group is part of an ELB. Architecture matches and it is not a devpay AMI.
Any idea what could cause this and what I can do to prevent this? Now during peak time servers did not scale up, and our service became very slow.
|
|
Posts:
2,112
Registered:
7/10/08
|
|
|
|
Re: Internal Error when autoscaler tries to launch instance
Posted:
Oct 14, 2009 12:58 AM PDT
in response to: bjornhij
|
|
|
This problem has been reported before.
The underlying error is likely to be "InsufficientCapacity" in that availability zone.
Unfortunately Auto Scaling doesn't report the underlying error. And it doesn't yet have good tools to recover from such errors.
As a potential workaround:
You can write a program that follows the output of describe-auto-scaling-actions and launches a new instance in a different availability zone.
Maybe there's a better workaround. Anyone?
|
|
Posts:
16
Registered:
12/26/08
|
|
|
|
Re: Internal Error when autoscaler tries to launch instance
Posted:
Oct 14, 2009 2:12 PM PDT
in response to: bjornhij
|
|
|
bjornhij,
thanks for reporting this. I am looking into it to find out what exactly happened and will report my findings.
Regards,
Alex.
|
|
Posts:
12
Registered:
11/12/08
|
|
|
|
Re: Internal Error when autoscaler tries to launch instance
Posted:
Oct 18, 2009 12:33 PM PDT
in response to: Shlomo Swidler
|
|
|
AutoScaling currently reports InsufficientAddressCapacity, InsufficientInstanceCapacity, and InsufficientReservedInstanceCapacity as a generic Internal Error. This is unintentional, and will be remedied in our next release.
|
|
Posts:
2,112
Registered:
7/10/08
|
|
|
|
Re: Internal Error when autoscaler tries to launch instance
Posted:
Oct 18, 2009 1:31 PM PDT
in response to: John@AWS
|
|
|
Thanks for the update, John.
But I'm a little confused by your wording.
It seems to imply that the errors InsufficientReservedInstanceCapacity and InsufficientAddressCapacity can happen as a result of Auto Scaling activities. But these two errors (according to the docs here:
http://docs.amazonwebservices.com/AWSEC2/latest/APIReference/index.html?api-error-codes.html ) are caused by requesting reserved instances or Elastic IP addresses - which are not auto scaling activities.
Can you please elaborate?
|
|
Posts:
12
Registered:
11/12/08
|
|
|
|
Re: Internal Error when autoscaler tries to launch instance
Posted:
Oct 19, 2009 3:53 PM PDT
in response to: Shlomo Swidler
|
|
|
I'm to blame for your confusion. Auto Scaling activities will only show errors resulting from insufficient capacity. It wasn't appropriate to bring up the other errors in this context.
By the way, this announcement is very apropos to this thread:
http://developer.amazonwebservices.com/connect/ann.jspa?annID=515
We now support multiple availability zones for a single Auto Scaling group. If your Auto Scaling group is configured to span two zones, one of which is congested, and the other of which isn't, Auto Scaling will endeavor to match the group's desired capacity by launching in the uncongested zone.
The error reporting fix is still coming soon.
|
|
|
|
Sign in to the AWS Management Console
Create an AWS Account
