RFR (XS): 8140597: Initial mark caused by humongous allocations can prevent mixed collections

[Thomas Schatzl]

Hi all,

  can I have reviews for this bug in the GC state machinery that results
in mixed collections being completely skipped if an initial mark caused
by humongous allocation occurs between the end of marking activity and
before the last young-only gc that should initiate the mixed phase?

Basically the initial mark request overrides the transition from that
last young-only GC to the mixed gc.

There is a really detailed technical description about what happens in
the CR.

This behavior is really problematic in case you tuned (either manually
or automatically) the IHOP to the latest possible time to start marking.

This seems like a bug that has been in since at least 2010.

I also changed the code to actually assert if G1 tries to do an initial
mark in the last young gc before the mixed phase. That restriction may
be lifted in a later change again (i.e. if G1 notices that there is
nothing to reclaim at the start of that last young-only gc). But that is
another change :)

CR:
https://bugs.openjdk.java.net/browse/JDK-8140597
Webrev:
http://cr.openjdk.java.net/~tschatzl/8140597/webrev/
Testing:
jprt, vm.gc testlist, manual testing with the mentioned existing test in
the CR

Thanks,
  Thomas