RFR 8234613: JavaThread can escape back to Java from an ongoing handshake

[Patricio Chilano]

Hi,

This patch aims to address a current bug where, given the right 
combination of handshakes and external suspend/resume, a JavaThread can 
transition from a safe state back to Java without blocking for a 
still-in-progress handshake. In the description of the bug I added an 
example, tracing the state changes of the JavaThread as it goes through 
the different transitions until it escapes the handshake. Currently, the 
window of time for this issue to happen is so small that we do not see 
actual failures running tests. Running test SuspendAtExit.java and 
adding some small delay before restoring the JavaThread state in 
java_suspend_self_with_safepoint_check() can demonstrate the issue.
The proposed fix is to check again if we have a pending/in-progress 
handshake operation after executing ~ThreadInVMForHandshake().

Tested with mach5, tiers1-6 on all platforms (Linux, macOS, Windows and 
Solaris).

Bug: https://bugs.openjdk.java.net/browse/JDK-8234613
Webrev: http://cr.openjdk.java.net/~pchilanomate/8234613/v01/webrev/

Thanks,
Patricio