Performance Benchmark Results: JVM vs CLR

I was talking with someone the other day about Java vs .NET performance. I have never thought that much about it. He said that .NET was fast because it had a Just In Time compiler (“JIT”). I choked on my coffee. Java’s JVM has had a JIT for about 6 years now. Not only that, but I had never heard that .net was faster.
So, I decided to go and run some benchmarks for myself. Rather than dirty my machine by installing Windows, I fired up VMWare and installed it on a disposable file. I then started the arduous and frustrating experience of installing .net onto my VMWare pack of XP. Reading the EULA, I discovered that Microsoft had already foreseen the threat of benchmarks and banned disclosure of any benchmark information without written permission from them.
So, If I were to tell you that Java JVM based on JDK1.5 was 42% faster than .net’s CLR, based on a benchmark I had, that would be breaching the EULA. In reality I cannot even get started because of the mind boggling activation process required. Currently my 60 day trial activation key has been used too many times and my Visual Studio is disabled. I am enjoying the experience already!
There are some braver souls than me around. See http://www.shudo.net/jit/perf/. There benchmarks are current. They were last updated in December. They show that on SciMark, Java scored 324 vs .NET of 240. On Linpack, Java scored 413 vs .net’s 282. Adding the scores together gives 737 for Java and 522 for .net.
On their site there is a lot of detail and a lot of different versions. Check it out.
So, in complete agreement with the EULA, I can tell you that Java’s VM is 42% faster than .net’s CLR.
And the next time someone claims that .net is faster because it “has a JIT compiler” point them at the site.

Published
Categorized as Java

By Greg Luck

As Terracotta’s CTO, Greg (@gregrluck) is entrusted with understanding market and technology forces and the business drivers that impact Terracotta’s product innovation and customer success. He helps shape company and technology strategy and designs many of the features in Terracotta’s products. Greg came to Terracotta on the acquisition of the popular caching project Ehcache which he founded in 2003. Prior to joining Terracotta, Greg served as Chief Architect at Australian online travel giant Wotif.com. He also served as a lead consultant for ThoughtWorks on accounts in the United States and Australia, was CIO at Virgin Blue, Tempo Services, Stamford Hotels and Resorts and Australian Resorts and spent seven years as a Chartered Accountant in KPMG’s small business and insolvency divisions. He is a regular speaker at conferences and contributor of articles to the technical press.

8 comments

  1. You need to look again. The number your cite aren’t quite right, and your math is wrong — 240 + 282 != 695. It’s 522, making JDK 1.5 41% faster than .net on the combine mflops.
    Of course, you can’t really add mflops like you did, but the numbers do indicate that for math, the JDK is about 40% faster.

  2. Actually, the real reason why the CLR is “faster” than the JVM (for certain tasks, of course) is that Microsoft push a lot more tasks down to the native infrastructure that Windows provides. In other words, it does a lot less inside the CLR than Java does inside the JVM.
    While Microsoft focus on a single platform for the CLR, this is a logical choice for performance improvements.

  4. This is meaningless. Are they saying that the CLR is faster executing the same sequence of instructions on the same operating system as the JVM? Surely they don’t have the same instruction set so it’s impossible to compare “speed” directly. It’s just like comparing the “speed” of an Intel chipset with, say a G5. Clock speed alone means nothing – depends on what instructions are executing during each “tick” and efficiently they were compiled from the source code. If we could comare differently specified VMs for speed, I wonder if Cincom Smalltalk VM would beat both the CLR and the JVM. They’ve been doing this a lot longer than Sun or Microsoft.

  5. You used VMWare to run the benchmarks? There goes your credibility! Did you run both the Java and C# code on the same stack?

  7. Perhaps the .NET proponent was just ignorant of the state of the art on the Java platform. But it’s possible that he was having trouble communication about how on the CLR, the JIT is always used to produce native code. MSIL is never interpretted by the CLR whereas on the JVM, bytecode is often interpreted until hotspot identifies a method as requiring jitting.
    Also worth keeping in mind that the Microsoft JVM was the first to include a JIT, IIRC.

  8. Actually the .net results for the SciMark 2.0 Bench mark are out dated, and use a 1.0 version of .net.
    Out of curiosity, I ran the results on an XP box, using my 1.5 Java JRE, and .net 1.1, and .net 2.0 beta two, and here are the composite scores.
    Java 1.5: 230
    .Net 2.0 C#: 356
    .Net 2.0 C: 616
    .Net 1.1 C#: 340
    .Net 1.1 C: 575
    Note for the Java time, I just ran the bench mark off the web, so it may have incurred some
    browser over head.
    Personally, math oriented tests such as these can be misleading, as the two libraries can have such a wide variety of implementation, accuracy ect. Its not always a valid comparison. I’d love to see the see the SPEC JVM98 benchmark ported to C#.
    I’ll admit to being a .net fan, as well as Java, but I find .net to be generally 5-15% more performant then Java, which makes sense to me considering the CLR is designed specially for the windows platform, where as Java needs to provide cross platform support. I’m not sure what Sam’s point is. It seems valid to ask, you have a box, what runs similar tasks faster on it, the clr, or the jvm? I would be curious to see if java would have better performance ruining on the same physical box with Linux, instead of xp.

Comments are closed.