RFR(S): 8010463: G1: Crashes with -UseTLAB and heap verification

[John Cuthbertson]

Hi Everyone,

I'm looking for a couple of reviews for the fix for these crashes. The 
webrev can be found at: http://cr.openjdk.java.net/~johnc/8010463/webrev.0/

Summary:
During JVM start up, with TLABs disabled, the JVM performs three 
separate verifications. The first is in universe2_init(), the second is 
in init_globals(), and the final one is in Threads::create_vm(). With 
TLABs enabled only one verification is performed during start up -  the 
one in Threads::create_vm(). These verifications are invoked by the main 
thread.

The problem here was that the G1 verification code was expecting to be 
invoked by the VMThread, at a safepoint. When TLABs are disabled the 
verification code was executed by main thread, triggering the assert. 
Relaxing the assert (to allow for execution during VM start up) is, 
unfortunately, not a good solution. There are parts of the root scanning 
code which assume the JVM is at a safepoint or has completed 
initialization. For example Threads::oops_do() assumes that the VMThread 
exists; CodeCache::oops_do() assumes that the CodeCache_lock is being 
held (or the JVM is at a safepoint); verifying G1's region sets assumes 
that the Heap_lock is being held (or the JVM is at a safepoint); etc.

When TLABs are enabled the verification from Threads::create_vm() skips 
verifying parts of the heap. The solution is to skip those parts of the 
verification even if TLABs are disabled. With just the changes in 
g1CollectedHeap.cpp, we would see the following:

With -UseTLABS:

> ----> universe2_init
> [Verifying threads (SKIPPING roots, heapRegionSets, heapRegions, 
> remset) syms strs zone dict cldg metaspace chunks hand C-heap code cache ]
> <---- universe2_init
> ---->init::verify
> [Verifying threads (SKIPPING roots, heapRegionSets, heapRegions, 
> remset) syms strs zone dict cldg metaspace chunks hand C-heap code cache ]
> <----init::verify
> --->create_vm:verify
> [Verifying threads (SKIPPING roots, heapRegionSets, heapRegions, 
> remset) syms strs zone dict cldg metaspace chunks hand C-heap code cache ]
> <---create_vm:verify

With +UseTLABS:

> ----> universe2_init
> <---- universe2_init
> ---->init::verify
> <----init::verify
> --->create_vm:verify
> [Verifying threads (SKIPPING roots, heapRegionSets, heapRegions, 
> remset) syms strs zone dict cldg metaspace chunks hand C-heap code cache ]
> <---create_vm:verify

Why do we perform  two additional verifications when TLABs are disabled? 
I've removed these in this fix. If someone can provide a reasonable 
justification, I'll add them back.

Additionally I've moved the verification code in Threads::create_vm() to 
after the VMThread is created. That way, as a future enhancement, the 
verification could be wrapped inside a VMOperation.

I've also included a regression test.

Testing:
The failing test case with G1 with and without TLABs enabled.
The regression test with all the collectors.
A jprt run (with -UseTLABS -XX:+VerifyBeforeGC) is in the queue.

Thanks,

JohnC