Benchmark scenario with high G1 performance degradation

[Stefan Johansson]

Hi Jens,

Thanks for reaching out and for providing such a good step-by-step guide 
on how to run the benchmark the same way you are.

I've tried with CMS, G1 and Parallel, both with 10g and 20g heap, but so 
far I can't reproduce your problems. It would be great if you could 
provide us with some more information. For example GC-logs and the 
result files. We might be able to dig something out of them.

Some comments below:

On 2017-04-20 04:49, Jens Wilke wrote:
> Hello,
>
> I am currently benchmarking different in-process caching libraries. In some
> benchmark scenarios I found very "odd" results when using
> the G1 garbage collector. In a particular benchmark scenario the performance
> (as measured in ops/s of the particular benchmark)
> drops to about 30% when compared to CMS, typically I expect (and observe) only
> a performance degradation to around 80% of  the performance with CMS.
 From my runs it looks like G1 is about 5-10% behind CMS and 10-15% 
behind Parallel for both JDK 8 and 9.

> In the blog post is a little bit more background of what I am doing:
> https://cruftex.net/2017/03/28/The-6-Memory-Metrics-You-Should-Track-in-Your-Java-Benchmarks.html
>
> For the particular scenario I have made the following observations:
>
> - Java 8 VM 25.131-b11: VmRSS, as reported by Linux, grows to 4.2 GB
> - Java 8 VM 25.131-b11: Average benchmark performance throughput is 2098524
> ops/s
> - Java 8 VM 25.131-b11: Throughput is quite steady
> - Java 9 WM 9-EA+165: VmRSS, as reported by Linux, grows to 6.3 GB
For me it is the other way around, running G1 with -Xmx10g I get VmRSS 6 
GB for JDK 8 and 4.6 GB for JDK 9. Throughput seems to be more or less 
on par between 8 and 9, but the results vary a bit so hard to say for 
sure without doing a deeper analysis.
> - Java 9 WM 9-EA+165: Average benchmark performance throughput is 566699 ops/s
> - Java 9 WM 9-EA+165: Throughput has big variations and the tendency to
> decrease
> - Java 9 WM 9-EA+165: Profiling shows that 44.19% of CPU cycles is spent in
> OtherRegionsTable::add_reference (for Java 8 G1 it is similar)
>
> And less quantified:
>
> - With Java 8 G1 it seems more worse
> - Scenarios with smaller heap/cache sizes don't show the high performance drop
> when comparing CMS and G1
> - Java 9 WM 9-EA+165 with the options -XX:+UseParallelGC -XX:
> +UseParallelOldGC, seems to have 50% performance of Java 8 and higher memory
> consumption
>    (What are the correct parameters to restore the old default behavior?)
You only have to set -XX:+UseParallelGC, that will enable the correct 
"old collector" as well (but setting UseParallelOldGC should be fine as 
it is the correct one). I see a bit higher memory consumption too, but 
the score is more or less the same between 8 and 9 for Parallel.
> - The overall GC activity and time spend for GC is quite low
>
> To reproduce the measurements:
>
> Hardware:
> Intel(R) Core(TM) i7-5600U CPU @ 2.60GHz, 1x4GB, 1x16GB @ DDR3 1600MHz, 2
> cores with hyperthreading enabled
> OS/Kernel:
> Linux version 4.4.0-72-generic (buildd at lcy01-17) (gcc version 5.4.0 20160609
> (Ubuntu 5.4.0-6ubuntu1~16.04.4) ) #93-Ubuntu SMP Fri Mar 31 14:07:41 UTC 2017
>
> git clone https://github.com/headissue/cache2k-benchmark.git
> cd cache2k-benchmark
> git checkout d68d7608f18ed6c5a10671f6dd3c48f76afdf0a8
> mvn -DskipTests package
> java -jar jmh-suite/target/benchmarks.jar \\.RandomSequenceBenchmark -jvmArgs
> -server\ -Xmx20G\ -XX:BiasedLockingStartupDelay=0\ -verbose:gc\ -XX:
> +PrintGCDetails -f 1 -wi 0 -i 10 -r 20s -t 4 -prof
> org.cache2k.benchmark.jmh.LinuxVmProfiler  -p
> cacheFactory=org.cache2k.benchmark.Cache2kFactory -rf json -rff result.json
I took a quick look at the blog as well, and there the system had 32GB 
ram and the runs were done with -Xmx10g. The system you describe here 
only have 20GB ram and you are using -Xmx20G, is that correct or has 
there been a typo? Otherwise you might be running into swapping issues, 
and that could explain your problems.

Thanks,
Stefan
> When not on Linux, strip "-prof org.cache2k.benchmark.jmh.LinuxVmProfiler".
>
> I have the feeling this could be worth a closer look.
> If there are any questions or things I can help with, let me know.
>
> I would be interested to know whether there is something that I can change in
> the code to avoid triggering this behavior.
>
> Best,
>
> Jens
>