[10] RFR (M/L): 8137099: G1 needs to "upgrade" GC within the safepoint if it can't allocate during that safepoint to avoid OoME

[Hohensee, Paul]

In attempt_allocation_humongous in g1CollectedHeap.cpp, there’s a note that says there’s code duplication between it and attempt_allocation_slow. There may be a case of version skew between the code at line 553 and the lack of similar code at line 972. If you don’t want to retry the humongous allocation attempt at that point, then the comment at line 968 should be updated.

Thanks,
Paul

On 12/4/17, 7:58 AM, "hotspot-gc-dev on behalf of Thomas Schatzl" <hotspot-gc-dev-bounces at openjdk.java.net on behalf of thomas.schatzl at oracle.com> wrote:

    Hi all,
    
      can I have reviews for this change that fixes a race within the G1
    collector that in conjunction with the GC locker causes it to OoME
    prematurely?
    
    The issue is that if a young gc of any kind (including gc locker caused
    ones) fails to allocate enough memory, the scheduled full gc can be
    cancelled by the gc locker, causing OoME in that thread. I.e. before
    any full gc has been tried.
    
    The solution is to, similar to other collectors (which corresponds to
    the observation that they do not show this behavior), start a
    compacting full gc in the same VM operation where the young GC has been
     started in to avoid this race to happen.
    
    Axel Siebenborn (cc'ed) from SAP originally contributed a fix that
    covered most situations; this change enhances it (and converts the
    standalone test he wrote into a jtreg ones in addition to style
    changes). I added us both as contributors.
    
    Unfortunately the test has had to be put into the Problemlist, because
    I did not manage to make it not to exhibit the failure in JDK-8192647
    sometimes. I still see it as a useful addition as soon that one has
    been fixed. For testing I manually checked the test output that the GC
    type "upgrade" occurs as expected though.
    
    Also, the test has been changed to be a stress test, and only execute
    at most a certain amount of time.
    
    CR:
    https://bugs.openjdk.java.net/browse/JDK-8137099
    Webrev:
    http://cr.openjdk.java.net/~tschatzl/8137099/webrev/
    Testing:
    hs-tier1-3, manual execution of included test, checking output to be as
    expected.
    
    Thanks,
      Thomas