RFR(S): 8040803: G1: Concurrent mark hangs when mark stack overflows

[Per Liden]

Hi,

Could I please have a couple of reviews in this bug fix:

Summary: G1's concurrent marking can potentially hang forever if the 
global mark stack overflows and immediately after that a Full GC 
happens, which tries to abort the marking. The reason is that there's a 
race between detecting the overflow situation and detecting the abort 
signal. Threads detecting the overflow situation first will go into the 
overflow protocol and wait on a barrier for all threads to reach this 
state. However, threads detecting the abort signal first will terminate 
and never participate in the barrier.

This patch introduces an abort state and function on the 
WorkGangBarrierSync class, to unblock any threads waiting for the 
barrier to complete when the concurrent mark is aborted.

Bug: https://bugs.openjdk.java.net/browse/JDK-8040803
Webrev: http://cr.openjdk.java.net/~pliden/8040803/webrev.0/

/Per