ParallelGC vs G1GC performance

[Richard Warburton]

Hi Richard,

I have done some basic performance benchmarking using my own home baked JMH
> bench marks <https://github.com/AdoptOpenJDK/vmbenchmarks>
> (A proper article to come)
> Also I'm not a mirco benchmark expert, so some of these tests may be
> incorrect from a micro benchmark point of view.
>
> I used the following JVMs on Ubuntu Linux 64bit :
> 1.9.0-ea-b70 64bit
> 1.8.0_45 64bit
>
> All values are in operations per second, so higher is better.
> ParallelGC wins every time.
>
>
> *Java 8* *Java 8 G1GC* *Java 9* *Java 9 G1GC*  *DynamicLanguages.groovy*
> 402.142 352.266 397.495 370.121
> *ExceptionsBenchMark.throwRuntimeException*
> 885181.859 823326.931 910622.671 843228.203
> *GeneratePoiWorkBook.generatePoiXLSWorkBook* 34205.392 28162.636 35129.546
> 28693.55  *GeneratePoiWorkBook.generatePoiXLSXWorkBook* 1125.074 980.061
> 1172.468 1012.47  *GeneratePrimeNumbersWithForLoop.generatePrime*
> 231035.868
> 230904.102 231405.021 230599.11  *Jackson.jsonMashal* 30019.081 22232.937
> 31908.426 24242.968  *Jackson.jsonUnmashal* 26942.264 20443.724 26736.986
> 22404.13  *Jackson.xmlMashalJackson* 22868.064 19657.647 23112.766
> 19947.204
> *Jackson.xmlMashalJacksonStatic* 691051.644 441105.658 745753.342
> 487013.862
> *Jackson.xmlUnmashalJackson* 11620.793 9143.764 11681.19 9915.961
> *Jackson.xmlUnmashalJacksonStatic* 92897.944 70261.49 96174.969 73341.546
> *JaxbMashalUnMashal.mashal* 988.781 925.565 2587.721 2345.568
> *JaxbMashalUnMashal.mashalStatic* 164904.491 136569.201 207380.024
> 190202.406  *JaxbMashalUnMashal.unmashal* 873.89 783.885 1928.468 1721.395
> *JaxbMashalUnMashal.unmashalStatic* 11348.605 7578.279 12184.689 8682.734
> *Reflection.doReflection* 1676241.984 1572468.245 1738819.079 1738759.745
> *RegularExpressions.validateEmail* 514686.286 476725.225 535478.698
> 490515.516
>

Thanks for asking for review, that's always a good idea when it comes to
benchmarking. I'm sure other people have their expertise to offer, but
here's a few observations I would make. Any criticism offered is meant in a
constructive and educational spirit, please don't take anything personally
;)

1. Its pretty hard to read what you're pasting here given the lack of
formatting. Perhaps its just my email client (gmail) or setup that's making
it hard to read but IMO presentation of results is pretty important because
it enables better peer review.
2. You've only measured the throughput (operations/second) in your
benchmark. Its not that surprising that Parallel GC would "win" at this war
as its a throughput oriented collector. Collectors like G1 and CMS are
aimed at reducing the time of individual stop-the-world pauses. Your
benchmark doesn't measure that at all, even though its a really important
property for application developers - no matter whether its:
a. a website that doesn't want to have its customers sitting around waiting
for a page to load
b. a distributed computing cluster which uses network timeouts as a way to
detect failure and thus gets partitioned by a long GC pause
c. an application with extreme low pause requirements such as HFT or a
computer game that needs to consistently re-render its display every 16.6ms.
3. Consider whether you really want to microbenchmark a GC subsystem. I
wouldn't microbenchmark GC - there are too many factors like promotion rate
and generation sizing which you can tune to make your microbenchmark look
awesome but which don't apply in the real world.
4. I had a brief look at Jackson.java. You shouldn't be using static final
Strings as they would enable constant propagation which is unrealistic for
Strings being parsed off a network connection. See the JMH examples or
Alexey Shipilev's Devoxx Be talk for more details.

There's probably a lot more factors to think about in terms of your
specific benchmarks, but I'll give some other people a chance to chime in
now ;)

regards,

  Richard Warburton

  http://insightfullogic.com
  @RichardWarburto <http://twitter.com/richardwarburto>