G1 GC for 100GB+ heaps

[Thomas Schatzl]

Hi,

On Wed, 2015-07-22 at 13:51 +0000, Kees Jan Koster wrote:
> Dear All,
> 
> Marcus Lagergren suggested I post these questions on this list. We
> are considering switching to using the G1 GC for a decently sized
> HBase cluster, and ran into some questions. Hope you can help me
> our, or point me to the place where I should ask.

This place is fine, although hotspot-gc-use might be more appropriate.

> First -> sizing: Our machines have 128GB of RAM. We run on the bare
> metal. Is there a practical limit to the heap size we should
> allocate to the JVM when using the G1 GC?

100G heaps are fine with G1. We have heard of quite a few successful
reports of using G1 with big data setups. (E.g.
https://databricks.com/blog/2015/05/28/tuning-java-garbage-collection-for-spark-applications.html 100G heap, significantly beating Parallel GC in throughput, https://software.intel.com/en-us/blogs/2014/06/18/part-1-tuning-java-garbage-collection-for-hbase 100G heap, 100ms latency) or people moving to G1 as default (https://issues.apache.org/jira/browse/CASSANDRA-7486 ) or just being happy with using G1 on smaller setups (https://wiki.apache.org/solr/ShawnHeisey) for this kind of workloads.

However you do not mention what your goals are (throughput or latency or
a mix of that), so it is hard to say whether G1 can meet your
expectations.

The larger your heap gets (like TB heaps) and the more stringent the
requirements are, depending on the application, it may start failing to
meet your expectations.

> What kind of region sizing
> should we use, or should we just let G1 do what it does?

Initially we recommend just setting heap size (Xms/Xmx) and pause time
goals (-XX:MaxGCPauseMillis). Depending on your results decreasing
G1NewSizePercent and increasing the number of marking threads (see the
first few links above).

> Second -> failure modes. Does G1 have any failure or fall-back modes
> that it will use for edge cases? How do we monitor for those?

The obvious last-effort failure mode in case of going out of memory is
issuing a full GC, on these heaps taking minutes which is typically not
acceptable.

The GC log will tell you about it (-XX:+PrintGCTimeStamps -XX:
+PrintGCDetails, search for "Full GC"). This typically occurs because
marking could not finish before going out of memory. In these cases,
decreasing the heap occupancy of when marking starts
(-XX:InitiatingHeapOccupancyPercent=n) and/or increasing the number of
marking threads (-XX:ConcGCThreads=n) helps.

Again, the first link gives some further explanation.

I think HBase in particular needs -XX:+ParallelRefProcEnabled to work
well (independent of collector used), see the links given.

> Finally: Are there any gotcha’s to keep in mind, or any tunables that
> we have to invest time into when we want to run smoothly with 100GB+
> heap sizes?

Consider that the G1 needs some extra space for operation. So at 100G
Java heap, and 128G RAM, the system might start to swap/thrash,
particular if other stuff is running there. I.e. monitor that using e.g.
vmstat. Should be avoided :)

If you are running on Linux, completely disable Transparent Huge Pages
on Linux (use a search engine to get to know how it is done on your
particular distro). Always, we have found no exceptions.

Other than that the above recommendations should be okay. If there are
particular issues you may want to come back with a log of a problematic
run with at least -XX:+PrintGCTimeStamps -XX:+PrintGCDetails set.

Thanks,
  Thomas