java process memory usage is higher than Xmx

[Thomas Schatzl]

Hi Shengzhe,

On Mon, 2014-01-06 at 13:49 -0800, yao wrote:
> Hi Thomas,
>         Possibly remembered set size.
>         Can you enable -XX:+UnlockDiagnosticVMOptions and -XX:
>         +G1SummarizeRSetStats?
>         The line
>           Total heap region rem set sizes = 5256086K.  Max = 8640K.
>         gives you a good idea about remembered set size memory usage.
>  
> You are right, rem set occupies ~7GB 

Could you add -XX:+G1SummarizeConcMark? The GC then shows some details
about the work done during cleanup phases at VM exit.

At 7 GB remembered set size most likely the phase that tries to minimize
the remembered set is dominant.

It should show up as large "RS scrub total time" compared to the "Final
counting total time".
> 
>  Concurrent RS processed -1863202596 cards
>   Of 12780432 completed buffers:
>      12611979 ( 98.7%) by conc RS threads.
>        168453 (  1.3%) by mutator threads.
>   Conc RS threads times(s)
>           0.00     0.00     0.00     0.00     0.00     0.00     0.00
> 0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00
> 0.00     0.00     0.00    \
>  0.00
>   Total heap region rem set sizes = 7520648K.  Max = 13256K.
>   Static structures = 347K, free_lists = 28814K.
>     141012349 occupied cards represented.
>     Max size region =
> 93:(O)[0x00007fcb48000000,0x00007fcb4a000000,0x00007fcb4a000000], size
> = 13257K, occupied = 2639K.
>     Did 0 coarsenings.
> 
>         I copied above line from one of your responses to the "G1 GC
>         clean up
>         time is too long" thread, and it seems the remembered set
>         takes ~5GB
>         there.
> 
> I did set -XX:G1RSetRegionEntries=4096 to avoid coarsenings; however,
> the cleanup time seems not being reduced, it is still around 700
> milliseconds, althrough there is no coarsenings.
>
> Any hint for tuning ? Because I want process with G1 use the same heap
> as CMS to compare the performance. But I cannot do so if rem set is

You could still compare performance with the same total memory usage.

>  that large, the process will be likely killed by OOM killed if I gave
> more memory.

The default value of G1RSetRegionEntries at 32M region size should be
1536 (= (log(region size) - log(1MB) + 1) * G1RSetRegionEntriesBase by
default); the chosen value means that you allow G1 to keep a larger
remembered set (ie. less coarsening) per region.

The "RS scrub" part of cleanup is roughly dependent on remembered set
size, and this is the main knob to turn here imo.

So it seems that increasing the G1RSetRegionEntries is
counter-productive for decreasing gc cleanup time, because scrubbing
coarsened remembered sets looks fast. I do not have numbers though, just
a feeling.
Coarsening mostly increases gc pause time (RS Scan time to be exact).

Otoh you mentioned that gc cleanup time did not change when changing
G1RSetRegionEntries.

It's best to measure where the time is spent using the
G1SummarizeConcMark switch and then possibly change the
G1RSetRegionEntries value (and measuring the impact).

Thomas