Benchmarking Interpreted, JIT compiled and precompiled Java

1
Benchmarking Interpreted, JIT compiled and
pre-compiled Java

• By Sitaram Iyer
• Algis Rudys
• Keith Layne
• Paul Thomas

2
Background

• Java prides itself in its
• Security
• Modularity
• Platform independence(what we will focus on)
• Creates architecture entirely in software
  including the instruction set and functional
  units. This is called the Java Virtual Machine.
• BUT - these factors can lead to degraded
  performance

3
Quick Points about JVM

• Stack-based
• Object Oriented Memory Addressing Model
• Supports Garbage collection
• Java APIs
• ISA uses bytecodes, a byte representing the
  instruction.
• Bytecodes are interpreted to functional unit
  calls and then to native machine code

4
BIG PROBLEM

• Interpretation is slow
• The fact that you must interpret each JVM
  instruction to native instructions can seriously
  degrade performance.
• Can we do better than interpretation inside the
  JVM?
• JIT(Just-in-Time compiler)
• Pre-compilation

5
WHAT IS A JIT?

• Under normal circumstances the JVM reads a
  collection of bytecodes in something called a
  classfile and interprets each bytecode, one by
  one. With JIT, the JVM reads in the entire
  classfile and compiles the set of bytecodes at
  once and then begins execution.

6
WHAT IS PRECOMPILATION?

• Precompilation means to take a collection of
  class files that compose a program, and compile
  them into a single binary executable native to
  the host machine. Although the executable is not
  portable like the class files, it should run
  faster.

7
OUR EXPERIMENT

• Benchmark a cross section of publicly available
  of JVMs that are representative of
• Interpreted JVM
• JIT Enable JVM
• PreCompiled version

8
JVMs tested

• JDK 1.1.4 no JIT
• JDK 1.1.6 no JIT and with JIT
• JDK 1.2B4 with JIT
• Kaffe, a non Sun based JVM(no JIT on SPARC)
• Toba, a Java to C translator

9
Benchmarks used

• Two categories of benchmarks
• Micro-benchmarks
• we used a modified version of the University of
  California, San Diego micro-benchmark package.
  That package tested
• array and loop operations
• method accesses
• exception handling
• garbage collection
• Macro-benchmarks to test real world performance
• SPECJVM
• ZIP, JavaCUP, etc.

10
JDK-1.1.6-nojit vs JDK-1.1.6-jit (advantages of a
JIT over interpretation)

• empty loop benchmark 20 times faster on JIT
• synchronized method benchmark - only 10-15
  faster on JIT - a possible bottleneck
• exception handling - 1.6 times better on non-JIT!
• algorithmic macro-benchmarks (Compress,
  MPEGAudio) - 8 to 9 times faster on JIT
• other macro-benchmarks - 1.5 to 3 times (library
  function calls)

11
JDK-1.1.6-jit vs JDK-1.2b4-jit (recent
improvements in JIT performance)

• method calls within same object - 5 times faster
• method call in different objects is 2 times
  faster, so jdk12b4-jit optimizes for spatial
  locality
• synchronized methods - improvement of 5 to 6
  times - fine grained locks?
• worse on exception handling - they're sacrificing
  this as they optimize the JIT
• similar performance of library functions like
  Zip/Unzip - libraries unchanged.

12
JDKs vs Toba (possibility of Java to C
translation)

• On SPARC, Toba has better performance than
  interpreted but worse than JIT with gcc
• Toba generates bad C code. They could try
  machine-dependent C code optimization
• -O (optimize flag) to toba slows it down!
  (generates even worse C code)
• exception handling is much better than JDK!
• - stack frames vs setjmp/longjmp
• garbage collection is much worse (20 times
  slower) than the JDKs.

13
JDK-1.1.6-nojit vs Kaffe (investigating non-Sun
java interpreters)

• Kaffe (w/o JIT) performs BADLY - being a Sun
  product is good for JDK.
• empty loop benchmark shows Kaffe five times
  slower.
• Exception handling 60 times slower!!
• On the x86 Kaffe has a JIT and the performance is
  comparable to JDK.

14
CONCLUSIONS

• JITs facilitate a large improvement in speed of
  execution over interpreted code.
• The fact that exception handling gets worse as
  the JIT is optimized is OK considering an
  exception is a rare case
• Kaffe's poor performance stemmed from the fact
  that it lacked a JIT(on x86 where is has a JIT it
  performs well). This goes to show that JIT is an
  extremely useful method for speed enhancement.

15
CONCLUSIONS

• Toba shows promise and is a good start in the
  direction towards a Java WAT(way ahead of time)
  compiler.
• It is obviously useful for standalone
  applications, where one is willing to compromise
  portability in some measure, for potentially
  improved performance and reliability.
• Work needs to be done on improving
• Java-gtC translation
• the garbage collection
• machine specific optimizations