review request for 6791815

[Xiaobin Lu]

6791815:  Fix for 6471657 can cause deadlock on non-Solaris platforms 
when initializing direct buffer support

Webrev: http://webrev.invokedynamic.info/xiaobin.lu/6791815/webrev/

Details:

When mutiple threads call NewDirectByteBuffer using JNI, deadlock could 
occur when thread A triggers the safepoint and thread B is busy waiting 
for thread A to get finished. However, the current implementation 
transfers thread B's state to be in VM state due to the fact that 
os::yield_all call could result into a os::sleep call on Solaris. The 
side effect of put thread B's state to be in VM state is that the 
VMThread will wait for thread B to finish before putting all Java 
threads to sleep. However, thread B is waiting for thread A to finish 
and thread A is waiting for the VMThread to return. Hence, a deadlock 
occurs.

The fix suggested by Tom is to put the ThreadInVMfromNative statement 
inside the loop so that there is a Window that thread B could change 
back its state to native and as a result, the VMThread could proceed to 
bring all the threads to safepoint. This is based on the fact that 
VMThread doesn't stop threads in native, the state check only occurs 
during the state transition.

I've tried hard to come up a local test to reproduce the problem, but 
without much luck. I have contacted the reporter so that I can give him 
the VM to verify that the problem has been fixed. I am pretty sure it 
has, but just to be safe.

Thanks,
-Xiaobin