G1 trace humongous allocations

[Thomas Schatzl]

Hi Roy,

On Wed, 2018-08-08 at 15:14 +0800, Roy Zhang wrote:
> Hi All,
> 
> Currently we are evaluating G1 in our company, find uncertainty of
> humongous object is major show stopper issue to roll out G1 in our
> production. 
> Our heap size is 36G, region size is 16M determined ergonomically by
> JVM, we find there are about 800~1300 humongous regions during some
> period which is not acceptable, it was caused by our *ugly* code,
> after we fix it, we increase region size to 32M (max region size),
> but we still do see many (>600) short-lived humongous objects during
> some period. So we have to identify humongous object allocation
> pattern first.

Short-lived humongous objects that are not arrays of j.l.O. are not
that bad - G1 gets rid of them quickly. G1 tries to reclaim them every
GC.

> Based on archived hotspot gc use mail
> list http://mail.openjdk.java.net/pipermail/hotspot-gc-use/2013-
> December/date.html#1682 (5 years ago), charlie hunt and Krystal Mok
> mentioned AllocObjectOutsideTLAB event in JFR, does anyone have some
> experience in using it to identify humongous objects allocation
> pattern? BTW, our code base is very big, and we have many server
> types, goal for me is use some automated ways to find humongous
> object in our whole code base.

The tests in test/jdk/jdk/jfr/event/gc/detailed in the jdk/jdk11 tree
[1][2] show how to programmatically enable allocation related GC events
and process them online (from what I understand the code). Instead of
the used EventNames.AllocationRequiringGC it should be sufficient to
use EventNames.ObjectAllocationOutsideTLAB to get the recordings.

It has a property that stores the allocation size - you might want to
filter on that.

Requires JDK11 though to use that API - since you are evaluating you
might be using that release already. The JFR API in earlier releases
may be slightly different, and is restricted by Oracle JDK licensing
terms of that particular release.

Another (offline) option would be to take a JFR recording with the
event set and then programmatically parse that file. There should be
examples how to do that in the jfr tests too.

There is also the new Low-overhead heap profiling, which may or may not
help you [3].

Thanks,
  Thomas

[1] http://hg.openjdk.java.net/jdk/jdk11
[2] http://hg.openjdk.java.net/jdk/jdk11/file/2afc9aa349ed/test/jdk/jdk
/jfr/event/gc/detailed
[3] http://openjdk.java.net/jeps/331