Benchmark scenario with high G1 performance degradation

[Jens Wilke]

Hello,

I am currently benchmarking different in-process caching libraries. In some 
benchmark scenarios I found very "odd" results when using
the G1 garbage collector. In a particular benchmark scenario the performance 
(as measured in ops/s of the particular benchmark) 
drops to about 30% when compared to CMS, typically I expect (and observe) only 
a performance degradation to around 80% of  the performance with CMS.

In the blog post is a little bit more background of what I am doing:
https://cruftex.net/2017/03/28/The-6-Memory-Metrics-You-Should-Track-in-Your-Java-Benchmarks.html

For the particular scenario I have made the following observations:

- Java 8 VM 25.131-b11: VmRSS, as reported by Linux, grows to 4.2 GB
- Java 8 VM 25.131-b11: Average benchmark performance throughput is 2098524 
ops/s
- Java 8 VM 25.131-b11: Throughput is quite steady
- Java 9 WM 9-EA+165: VmRSS, as reported by Linux, grows to 6.3 GB
- Java 9 WM 9-EA+165: Average benchmark performance throughput is 566699 ops/s
- Java 9 WM 9-EA+165: Throughput has big variations and the tendency to 
decrease
- Java 9 WM 9-EA+165: Profiling shows that 44.19% of CPU cycles is spent in 
OtherRegionsTable::add_reference (for Java 8 G1 it is similar)

And less quantified:

- With Java 8 G1 it seems more worse
- Scenarios with smaller heap/cache sizes don't show the high performance drop 
when comparing CMS and G1
- Java 9 WM 9-EA+165 with the options -XX:+UseParallelGC -XX:
+UseParallelOldGC, seems to have 50% performance of Java 8 and higher memory 
consumption
  (What are the correct parameters to restore the old default behavior?)
- The overall GC activity and time spend for GC is quite low

To reproduce the measurements:

Hardware:
Intel(R) Core(TM) i7-5600U CPU @ 2.60GHz, 1x4GB, 1x16GB @ DDR3 1600MHz, 2 
cores with hyperthreading enabled
OS/Kernel:
Linux version 4.4.0-72-generic (buildd at lcy01-17) (gcc version 5.4.0 20160609 
(Ubuntu 5.4.0-6ubuntu1~16.04.4) ) #93-Ubuntu SMP Fri Mar 31 14:07:41 UTC 2017

git clone https://github.com/headissue/cache2k-benchmark.git
cd cache2k-benchmark
git checkout d68d7608f18ed6c5a10671f6dd3c48f76afdf0a8
mvn -DskipTests package
java -jar jmh-suite/target/benchmarks.jar \\.RandomSequenceBenchmark -jvmArgs 
-server\ -Xmx20G\ -XX:BiasedLockingStartupDelay=0\ -verbose:gc\ -XX:
+PrintGCDetails -f 1 -wi 0 -i 10 -r 20s -t 4 -prof 
org.cache2k.benchmark.jmh.LinuxVmProfiler  -p 
cacheFactory=org.cache2k.benchmark.Cache2kFactory -rf json -rff result.json

When not on Linux, strip "-prof org.cache2k.benchmark.jmh.LinuxVmProfiler".

I have the feeling this could be worth a closer look.
If there are any questions or things I can help with, let me know.

I would be interested to know whether there is something that I can change in 
the code to avoid triggering this behavior.

Best,

Jens

-- 
"Everything superfluous is wrong!"

   // Jens Wilke - headissue GmbH - Germany
 \//  https://headissue.com