RFR(S): 7037756 Deadlock in compiler thread similiar to 6789220

[John Cuthbertson]

Hi Everyone,

Can I have a couple of volunteers to look over these changes? The webrev 
can be found at: http://cr.openjdk.java.net/~johnc/7037756/webrev.1

The issue here was very similar to the issue that caused 6789220 - the 
difference here was that the reference handler was blocked while waiting 
for the MethodCompileQueue_lock rather than waiting on a blocking 
compilation. To summarize:

Thread 6 (reference handler thread), while owning the pending list lock, 
requested a compilation and was blocked waiting on the 
MethodCompileQueue_lock.

Thread 11 (compiler thread 1), while owning the Compile_lock, attempted 
to allocate a Class mirror which triggered GC. In the GC it was blocked 
attempting to lock the pending list lock.

Thread 12 (compiler thread 2) was registering a compiled method and, 
while owning the MethodCompileQueue_lock, was blocked waiting on the 
Compile_lock.

The solution is to make the reference handler thread not block while 
holding the pending list lock. If the requesting thread is the reference 
handler thread, then an attempt is made to lock the 
MethodCompileQueue_lock in CompileBroker::compile_method_base and, if 
that is unsuccessful, we just return with enqueueing the compile task. 
Otherwise a regular blocking lock attempt is made. I also tweaked the 
fix made by Bengt for 6789220 to make all compilation requests by the 
reference handler thread non-blocking.

Testing: the failing test case has been running successfully on the 
VMSQE machine for 2 days (normally I see the deadlock after 20 minutes 
or so); the nsk tests; and a jprt job is the queue.

Thanks,

JohnC