G1 GC for 100GB+ heaps

[Bernd Eckenfels]

Am Thu, 23 Jul 2015 17:41:41 +0300
schrieb Kees Jan Koster <kjkoster at gmail.com>:

> Max pause times we aim for is 100ms, which looks to be entirely
> doable. Maybe we should set our goals a little more aggressively. ;-)

100ms sounds very aggressive for such large heaps, I would not expect
it. And its actually better to not aim for it then as well.

> We have a test cluster running with -XX:MaxGCPauseMillis=100 but I
> found that this actually results in an *average* of 100ms and not a
> max. Is that observation correct? What am I misinterpreting?

It is a hint and not very reliable at that. Neither as a max nor as a
average. But if you undershoot it will make things worse. Maybe try
200ms and see if your avarage does not change and the peaks become
flatter (not untypical).

Anyway, without seeing your logs and knowing your hardware specs its
hard to give more hints.

> > Consider that the G1 needs some extra space for operation. So at
> > 100G Java heap, and 128G RAM, the system might start to swap/thrash,
> > particular if other stuff is running there. I.e. monitor that using
> > e.g. vmstat. Should be avoided :)
> 
> Yes, these are dedicated machines and should never hit swap. We’ll
> keep an eye out to avoid the system hitting swap. Today the machines
> are running with 64GB heaps for that reason.

If you have 128GB and aim for a 100GB heap, keep 10GB for the rest of
the VM then you will have 10GB for filesystem cache. This should be
reflected in the swappiness setting for linux (5-10 for large machines
with application server load).

> > If you are running on Linux, completely disable Transparent Huge
> > Pages on Linux (use a search engine to get to know how it is done
> > on your particular distro). Always, we have found no exceptions.
> 
> Thank you for that advice. I got the same advice from Kirk Pepperdine
> this week. Our systems actually run with transparent huge pages
> enabled and I’ll ask the guys to switch that off.
> 
> $ cat /sys/kernel/mm/transparent_hugepage/enabled
> [always] madvise never

You might however reserve and turn on real large pages for the VM.
-XX:+UseLargePages. when using such a large heap it can safe a lot of
resources.

Gruss
Bernd