Performance testing for Graal

[Halimi, Jean-Philippe]

Hi Petr,

Thanks a lot for your valuable insights. Especially, I did not realize mx was benchmark-able.

Jp

-----Original Message-----
From: Petr Tůma [mailto:petr.tuma at d3s.mff.cuni.cz] 
Sent: Monday, October 29, 2018 9:29 AM
To: Halimi, Jean-Philippe <jean-philippe.halimi at intel.com>; graal-dev at openjdk.java.net
Subject: Re: Performance testing for Graal

Hi Jean-Philippe,

one thing we are doing is running benchmarks on every merge commit in the Graal GitHub repo. Obviously, our results are platform specific (particular machine, OS and JDK), but they help get some idea of how Graal fares compared to C2 and how the performance changes with time.

> - Where does Graal stand as of now as a JIT compiler in OpenJDK compared to C2?

I do not have the latest results for OpenJDK right now, but if comparison with C2 on Oracle JDK 11 can be any indication, then the open source version of Graal is almost always on par with C2, often better. For example, with the ScalaBench suite, we see Graal being about 38% faster on the factorie benchmark, about 9% faster on the tmt benchmark, etc. (all this collected on 8 core 2.1 GHz Intel machines with Oracle JDK 11 JVM).

I can post a bit longer overview but I'm not sure if it's all that useful because the exact numbers depend on a lot of configuration factors and it is better to run the benchmarks locally. Also, aggregating across multiple benchmarks tends to give possibly misleading results (average speedup with arbitrary benchmark weights is basically a useless number).

> - What are the major performance gaps of Graal compared to C2?

Unfortunately, I do not have enough insight into the compiler to answer this. I also understand the compiler is being actively improved in many areas, and the answer to your question would therefore depend on which version you have in mind.

> - What are the benchmarks traditionally used to measure the difference?

We have started with the traditional ones, like DaCapo and ScalaBench, plus SPECjbb and SPECjvm, but only in selected configurations (accurate measurements consume quite a lot of CPU time). The "mx benchmark --list"
gives a bit longer list of benchmarks that can be executed directly by the mx harness ...

Regards, Petr


On 26/10/2018 02:39, Halimi, Jean-Philippe wrote:
> Hi,
> 
> I would like to scope out what efforts have been done to track the 
> performance of Graal. I take a purposedly naïve approach to get the conversation started here:
> 
> - Where does Graal stand as of now as a JIT compiler in OpenJDK 
> compared to C2? - What are the major performance gaps of Graal 
> compared to C2? - What are the benchmarks traditionally used to measure the difference?
> 
> Jp
>