Need help on G1 GC young gen Update RS and Scan RS pause reduction

[Thomas Schatzl]

Hi Amit,

On Fri, 2017-02-03 at 11:09 +0000, Amit Mishra wrote:
> Hi Thomas/team,
> 
> 
> I have put all parameters as per your suggestion but somehow the
> minor gc pauses are still haunting.
> 
> Attaching GC logs.
> 
> 
> bash-3.2$ grep -i young gcstats.log.10636|cut -d, -f2|awk -F" "
> '{print $1}'|awk '$1 > 1'
> 1.1273134
> 1.1683221
> 3.5504848
> 5.2693987

  looking at these log entries, there seems to be something going on
that seems outside of VM control:

- from one gc to another, just for these four gcs, sys time is
relatively high.

- for the last two occurrences, at least one thread is hanging in "Ext
Root Scanning" for almost all of the gc time for no obvious reason.

- there do not seem to be an unusually large amount of changes in the
amount of work done in the particular phases that would raise immediate
concerns to me.

Please try to find out the source of the high sys time and maybe even
what causes it. I can't help a lot in that area, but dtrace seems a
good starting point as suggested earlier.

I think we went through most obvious tunings now, but maybe somebody else has more ideas. I don't at this time.

The jdk (7u45) you are using is also very old, so even if we find that
there is something wrong with g1 in particular, I kind of doubt there
are many more useful knobs to turn with that version (or even
appropriate logging to find out about the actual issue). Since 7u45
release, there have been hundreds of changes that in particular improve
G1 performance, so please consider upgrading to something more recent
(at least latest 8u, preferably to me some test runs with 9ea).
Upgrading alone might already help.

Thanks,
  Thomas