Performance testing for Graal

[Petr Tůma]

Hi Jean-Philippe,

one thing we are doing is running benchmarks on every merge commit in the Graal GitHub repo. Obviously, our
results are platform specific (particular machine, OS and JDK), but they help get some idea of how Graal fares
compared to C2 and how the performance changes with time.

> - Where does Graal stand as of now as a JIT compiler in OpenJDK compared to C2?

I do not have the latest results for OpenJDK right now, but if comparison with C2 on Oracle JDK 11 can be any
indication, then the open source version of Graal is almost always on par with C2, often better. For example,
with the ScalaBench suite, we see Graal being about 38% faster on the factorie benchmark, about 9% faster on
the tmt benchmark, etc. (all this collected on 8 core 2.1 GHz Intel machines with Oracle JDK 11 JVM).

I can post a bit longer overview but I'm not sure if it's all that useful because the exact numbers depend on a
lot of configuration factors and it is better to run the benchmarks locally. Also, aggregating across multiple
benchmarks tends to give possibly misleading results (average speedup with arbitrary benchmark weights is
basically a useless number).

> - What are the major performance gaps of Graal compared to C2?

Unfortunately, I do not have enough insight into the compiler to answer this. I also understand the compiler is
being actively improved in many areas, and the answer to your question would therefore depend on which version
you have in mind.

> - What are the benchmarks traditionally used to measure the difference?

We have started with the traditional ones, like DaCapo and ScalaBench, plus SPECjbb and SPECjvm, but only in
selected configurations (accurate measurements consume quite a lot of CPU time). The "mx benchmark --list"
gives a bit longer list of benchmarks that can be executed directly by the mx harness ...

Regards, Petr


On 26/10/2018 02:39, Halimi, Jean-Philippe wrote:
> Hi,
> 
> I would like to scope out what efforts have been done to track the performance of Graal. I take a purposedly
> naïve approach to get the conversation started here:
> 
> - Where does Graal stand as of now as a JIT compiler in OpenJDK compared to C2? - What are the major 
> performance gaps of Graal compared to C2? - What are the benchmarks traditionally used to measure the 
> difference?
> 
> Jp
>