G1GC is not able reclaim bytes

[Thomas Schatzl]

Hi,

   (cc'ed hotspot-gc-use as it is the appropriate mailing list for such 
quesstions).

On 13.08.19 17:01, shang xinli wrote:
> There are two connected full GC but they are not able to reclaim bytes. Here are gc logs.
> 
> 
>   [Times: user=0.35 sys=0.00, real=0.03 secs]
> 
> 2019-08-01T15:12:14.594+0000: 755123.074: [Full GC (Allocation Failure)  132G->132G(180G), 37.4525886 secs]
> 
>     [Eden: 0.0B(9216.0M)->0.0B(9216.0M) Survivors: 0.0B->0.0B Heap: 132.1G(180.0G)->132.1G(180.0G)], [Metaspace: 204550K->204459K(229376K)]
> 
>   [Times: user=68.14 sys=0.07, real=37.45 secs]
> 
> 2019-08-01T15:12:52.048+0000: 755160.527: [Full GC (Allocation Failure)  132G->132G(180G), 37.4584894 secs]
> 
>     [Eden: 0.0B(9216.0M)->0.0B(9216.0M) Survivors: 0.0B->0.0B Heap: 132.1G(180.0G)->132.1G(180.0G)], [Metaspace: 204459K->204451K(229376K)]
> 
>   [Times: user=68.38 sys=0.05, real=37.46 secs]
> 
> Here is the chart. We can see before and after GC, the heap is almost the same.
> [cid:65dca3e8-3bc4-4b87-93dc-997ba0e601f2]
> 
> 
> The service OOMs eventually, but we don't configure to generate heap dump. Is there a way to investigate why?
> 
>
The reason of the full gc might be humongous objects that use up lots of 
additional space given that you get full gcs at 132 GB out of 180.

So does your application happen to use many - and I mean many - large 
objects (~16M in size)?

You can add -XX:+UnlockExperimentalVMOptions -XX:G1LogLevel=finest to 
your command line? (I assume you use some JDK8 given the log output, but 
not sure)

Of particular interest are the following lines.

[Humongous Register: x.x ms]
   [Humongous Total: y]
   [Humongous Candidate: z]

Where y shows you the number of live humongous objects at the time of 
the GC(s).

If that value is rather high, we found the issue.

The reason is that objects that have a total size that is just above the 
humongous object threshold always use up full regions. This can waste a 
lot of space.

That threshold is >= half a region size, i.e. 16M in your case, since 
regions are 32M at that heap size by default. Such a humongous object 
wastes the other half of the region. Remember that an object adds a 
small header, so an object that has exactly 16M of usable data is 
actually 16M + a little bit in size.

If possible, try to lower your object size a bit. Then the waste will be 
much smaller in any case.

The only other option I can see is increasing the heap, or use a 
different collector.

Thanks,
   Thomas