http://blogs.vmware.com/vmtn/2007/03/bad_benchmark_r.html

There are some good reasons that VMware wants pre-publishing review of benchmarks using our software. There are also good reasons not to like this policy, and my personal view is that it probably should be phased out at some point. However, we see problems with virtual performance testing over and over again. Virtualization benchmarking is hard, and virtualization is still so new that people love to take ancedotes and generalize to the usefulness of virtualization technologies, for all time.

Case in point: this recent benchmark got picked up by Slashdot today: Load Testing a Virtual Web Application. It really isn’t a good test: they use VMware Server 1.0.1 (we don’t recommend using Server for high-throughput production uses!) instead of ESX Server, they don’t tune anything, etc.

This is a very fair point, the problem with these benchmarks is that they can be finger in the air type tests. Ultimately anyone wishing to compare virtualization products should be doing the due diligence part, downloading it/buying it and comparing the two products on their server, their network setting it up as per their documentation/standards and configuring it with the vendors recommendations.

Virtualization remains for many as the post suggests is a great tool for server consolidation, the ability to get rid of my legacy proliants and consolidate them on to one or two DL585s, freeing up power and datacenter space.  Would I use it in production? Sure, but as with any technology, I’d use it for what I thought was appropriate to the task at hand, if it’s a real time market data feed and it’s working on a physical platform, then maybe not, but if it’s an infrastructure server and it’s not doing much, then it’s the ideal candidate.