RFR(s): 8210303: VM_HandshakeAllThreads fails assert with "failed: blocked and not walkable"

[Robbin Ehn]

Hi all, please review.

VM thread checks if it can processes a handshake for a JavaThread. That
check will only return a stable value if the VM thread holds the handshake
semaphore (or at safepoint). To avoid an unnecessary grabbing of the semaphore
just to release it, the VM thread do an early check to see if there is any point
to do the stable check. But the method SafepointSynchronize::safepoint_safe() is
not suppose to handle unstable checks. This can causes a false positive from an
assert in safepoint_safe().

This change-set adds a local function for doing the unstable check without 
asserts. I do not want to expose a generic method for doing unstable safepoint 
safe test.

Since asserts are not in release builds, there is no indication of a bug in JDK
11. But since 11 is a LTS, this should also be considered for back-porting.

Note, in JDK 11 only ZGC uses handshakes, previously releases have no users of 
handshakes.

Webrev:
http://cr.openjdk.java.net/~rehn/8210303/webrev/index.html

Bug:
https://bugs.openjdk.java.net/browse/JDK-8210303

I could not reproduce it, sanity with t1-3 + handshake tests.

Thanks, Robbin