RFR(S): 8004816: G1: Kitchensink failures after marking stack changes

[John Cuthbertson]

Hi Everyone,

Can I have a couple of volunteers review the fix for this CR? The webrev 
can be found at: http://cr.openjdk.java.net/~johnc/8004816/webrev.0/

Summary:
The reduced mark stack size that came in as part of the changes for 
8000244 exposed this issue. If the marking stack overflowed as a result 
of pushes from the serial reference processing closures, the marking 
stack overflow flag was not cleared. When marking eventually completed, 
after the subsequent restarting, the still set overflow flag was 
detected and resulted in a guarantee failure. I also discovered that the 
actual marking state was not correctly cleared for restarting for such 
an overflow and, as a result, some referent objects might have been 
skipped by marking (those that are reachable from the objects we failed 
to push as a result of the overflow).

Normally this is not a problem as the serial reference processing code 
is executed infrequently - except on systems with one or two cpus. The 
parallel reference processing closures reset the marking state correctly 
in the event of an overflow in the global marking stack.

Testing:
Kitchensink on 1 and 2 cpu systems with a very low mark stack size and 
marking verification.

Thanks,

JohnC