[10] RFR (M/L): 8137099: G1 needs to "upgrade" GC within the safepoint if it can't allocate during that safepoint to avoid OoME

[Thomas Schatzl]

Hi all,

  can I have reviews for this change that fixes a race within the G1
collector that in conjunction with the GC locker causes it to OoME
prematurely?

The issue is that if a young gc of any kind (including gc locker caused
ones) fails to allocate enough memory, the scheduled full gc can be
cancelled by the gc locker, causing OoME in that thread. I.e. before
any full gc has been tried.

The solution is to, similar to other collectors (which corresponds to
the observation that they do not show this behavior), start a
compacting full gc in the same VM operation where the young GC has been
 started in to avoid this race to happen.

Axel Siebenborn (cc'ed) from SAP originally contributed a fix that
covered most situations; this change enhances it (and converts the
standalone test he wrote into a jtreg ones in addition to style
changes). I added us both as contributors.

Unfortunately the test has had to be put into the Problemlist, because
I did not manage to make it not to exhibit the failure in JDK-8192647
sometimes. I still see it as a useful addition as soon that one has
been fixed. For testing I manually checked the test output that the GC
type "upgrade" occurs as expected though.

Also, the test has been changed to be a stress test, and only execute
at most a certain amount of time.

CR:
https://bugs.openjdk.java.net/browse/JDK-8137099
Webrev:
http://cr.openjdk.java.net/~tschatzl/8137099/webrev/
Testing:
hs-tier1-3, manual execution of included test, checking output to be as
expected.

Thanks,
  Thomas