|
Discussion Forums
|
Thread: CPU performance 50% of expected
 |
This question is not answered.
Helpful answers available: 2.
Correct answers available: 1.
|
|
|
|
Replies:
7
-
Pages:
1
-
Last Post:
Sep 13, 2007 8:37 AM
by: jjc1138
|
|
|
Posts:
5
Registered:
9/6/07
|
|
|
|
CPU performance 50% of expected
Posted:
Sep 6, 2007 11:31 AM PDT
|
|
|
We have been experimenting with using EC2 instances as app servers for a Ruby on Rails application, and, confusingly, we have found performance to systematically be only 50% of what we expect. We are comparing performance to an Intel Core 2 Duo 1.8GHz machine, using only one core.
We wrote an extremely simple Ruby benchmark that generates a random list of 1,000,000 integers and then sorts it. It reports elapsed time like so (from our comparison machine):
user system total real
sort! 3.590000 0.010000 3.600000 (
3.631370
)
Note that this is a single-threaded benchmark; on our comparison machine, top and vmstat show "50%" CPU utilization since there is an idle core. Note that the last column shows elapsed "wall time", and it corresponds well to the "total" column, which is user+system.
On our EC2 instance, the benchmark reports:
user system total real
sort! 3.640000 0.000000 3.640000 (
8.078162
)
Note that the benchmark took over twice as many actual seconds to run, but that the reported time spent in user+system is nearly identical to our reference machine. This disparity makes it seem as though the virtualization layer has stolen every other CPU tick, and indeed, "top" and "vmstat" show 50% stolen cycles while the benchmark is running.
Our concern is not with the benchmark itself, but rather with our experience that our Rails application's observed performance is roughly 50% of its performance on what should be comparable hardware. Elapsed-time for rendering dynamic pages is systematically twice as long as our comparison machine. Our tests are with sequential page hits, exercising only one core, so the issue is not that we are comparing a fully-utilized dual-core machine to a virtual single-core machine.
In a nutshell, EC2's CPU performance seems to be only 50% of what it should be.
I have found several other threads where users observe that EC2 performance is roughly half of what they expect, and/or that reported CPU utilization never gets above 50% (which is dismissed as an artifact of usage reporting on top of the Xen virtualization layer). I haven't seen any resolution of the raw performance issue, though.
Could an Amazon employee please comment on the level of performance we can expect from an EC2 instance?
Thanks!
|
|
Posts:
5
Registered:
9/6/07
|
|
|
|
Re: CPU performance 50% of expected
Posted:
Sep 7, 2007 1:17 PM PDT
in response to: Alan Steele
|
|
|
In case anyone wants to observe this for themselves:
------------------------------ (ruby code)
require 'benchmark'
array = (1..1000000).map { rand }
Benchmark.bmbm do |x|
x.report("sort!") { array.dup.sort! }
end
------------------------------ (ruby code)
|
|
Posts:
57
Registered:
11/13/06
|
|
|
|
Re: CPU performance 50% of expected
Posted:
Sep 8, 2007 2:38 PM PDT
in response to: Alan Steele
|
|
|
One thing to note about your benchmark (and about you real world RoR performance) is that they are not only testing the CPU. They are also using the system RAM and the buses which lead there will be contended between the instances that are sharing the same machine.
EC2's performance claim of
"equivalent of a system with a 1.7Ghz x86 processor, 1.75GB of RAM[...]" is somewhat vague because it doesn't specify any of the rather important bits connecting those pieces together.
However, in the end, is the performance causing you actual problems, or is it merely a theoretical concern? If one instance is providing half the performance that you need then the obvious response would be: start another instance!
 You mentioned the rendering time for your Rails pages; is it causing a noticeably slow user experience, or is it a matter of server capacity?
|
|
Posts:
5
Registered:
9/6/07
|
|
|
|
Re: CPU performance 50% of expected
Posted:
Sep 8, 2007 9:20 PM PDT
in response to: jjc1138
|
|
|
Nonsense.
Look, it's like this.
1. Very simple ruby benchmarks, using various EC2 images including the default ones provided by Amazon, show 50% of normal perf vs. what's advertised for very simple operations.
2. We are not the first people to point this out. Go look at a half-dozen other threads like this, all unresolved, that you can find in the EC2 forums.
3. Yes, we are in fact noticing a human-visible impact in rendering time for more expensive operations.
4. Bottom line, I am getting 50% of the cpu power that I'm paying $70/month per instance for. I might as well go to ServerBeach and get the real thing if this is the perf I can expect from EC2.
I am a huge booster for the concept behind EC2 but I'm not willing to settle for 1/2 of what I'm paying for.
|
|
Posts:
3
Registered:
9/10/07
|
|
|
|
Re: CPU performance 50% of expected
Posted:
Sep 10, 2007 10:25 AM PDT
in response to: Alan Steele
|
|
|
I work with Alan and we have looked into this a little more. As we mention in the "50% CPU utilization thread" here,
http://developer.amazonwebservices.com/connect/thread.jspa?messageID=66988
it looks as though the problem is that our instance only has access to one core of a dual-core machine, and the virtualization layer enforces that we can only use half of that core's cycles. The result is that the virtual machine is almost exactly half as fast as it should be.
Does Amazon have any comment on this?
|
|
Posts:
89
Registered:
9/12/07
|
|
|
|
Re: CPU performance 50% of expected
Posted:
Sep 12, 2007 6:13 PM PDT
in response to: Mark A. Aiken
|
|
|
We can confirm that what you are seeing is expected behavior.
As mentioned in the other threads that you refer to, it is expected that instances will show ~50% CPU utilization.
We use a variety of actual processors with EC2, but our goal is to provide instances that perform consistently.
On different processors, this will be a different percentage of a real processor.
When we released EC2, we benchmarked against a common processor of that time and arrived at the 1.7 GHZ equivalency.
Since processors change quickly, we are working on expressing our CPU in less transient terms.
Finally, we are very interested in hearing about your application and your use case if you are bound by CPU performance.
We are evaluating ways to provide greater value to users with CPU intensive tasks and your input would be valuable.
If you would be interested in discussing your use case, please private message me.
Thanks for your interest in EC2,
Matt
|
|
Posts:
3
Registered:
9/10/07
|
|
|
|
Re: CPU performance 50% of expected
Posted:
Sep 12, 2007 8:09 PM PDT
in response to: MattG@AWS
|
|
|
Matt, thanks for looking into this. If you've investigated and this is really the level of CPU performance that EC2 is set up to provide, this seems like a frustrating disconnect with how EC2 has been billed. Making sure EC2 compares well to commodity hardware currently on offer from dedicated-hosting companies is a smart and attractive way to position the service, and is what I thought Amazon was aiming for.
In our case, though, our VM's kernel seems to think that the underlying processor is a 2.0GHz Opteron CPU. If that's true, I don't understand why the virtualization layer would be configured to provide only one half of one core to each hosted VM. Surely that cannot reasonably be expected to provide performance equivalent to "
a 1.7Ghz x86 processor"?
Our use case is simply that Ruby is a text-interpreted language and quite CPU-heavy. In-memory operations over a large number of objects are inevitably CPU-bound. Yes, user-visible operations should avoid heavy CPU lifting. Yes, we move CPU-intensive operations out of Ruby whenever they become a problem. Yes, a well-tuned application should be I/O bound. But our site is highly dynamic and CPU performance matters.
I ran the same simple Ruby benchmark on some old machines I had lying around and found:
Intel Pentium III Coppermine (introduced early 2000), 990MHz, 256Kb cache:
10.52s
EC2 instance:
8.08s
Intel Pentium 4 Northwood (introduced January 2002), 2.0GHz, 512K cache:
5.76s
Intel Core 2 Duo 4300 (introduced January 2007) 1.8GHz, 2MB cache:
3.63s
It would be silly to build out a farm of application servers using 7-year-old Pentium IIIs, don't you think?
|
|
Posts:
57
Registered:
11/13/06
|
|
|
|
Re: CPU performance 50% of expected
Posted:
Sep 13, 2007 8:37 AM PDT
in response to: MattG@AWS
|
|
|
This is a very interesting post. I had assumed (and have been telling my customers) that each instance has a dedicated CPU core. I take it that's actually incorrect?
|
|
|
|
Sign in to the AWS Management Console
Create an AWS Account
