RFR: 8266963: Reentrance condition for safepoint/handshake

[Yude Lin]

Shenandoah hangs when running specjvm2008 derby. The reason is a Java Thread reenters safepoint/handshake and blocks on itself. Please checkout the bugid for more details. After discussion with @zhengyu123, we think this might not be Shenandoah-specific. I propose to add a check before processing the safepoint/handshake.

An alternative approach (also insight from @zhengyu123) is to move the check a little earlier to the specific place where the Java Thread do ThreadBlockInVM. To feel reassured that no more reentrance exists in other places, I still leave the check in safepoint/handshake as debug code. See https://github.com/openjdk/jdk/compare/master...linade:reentrancecond

I'd appreciate more of your thoughts on these as I understand it could be a rather critical part of the code.

-------------

Commit messages:
 - 8266963: Reentrance condition for safepoint/handshake

Changes: https://git.openjdk.java.net/jdk/pull/4028/files
 Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=4028&range=00
  Issue: https://bugs.openjdk.java.net/browse/JDK-8266963
  Stats: 29 lines in 2 files changed: 29 ins; 0 del; 0 mod
  Patch: https://git.openjdk.java.net/jdk/pull/4028.diff
  Fetch: git fetch https://git.openjdk.java.net/jdk pull/4028/head:pull/4028

PR: https://git.openjdk.java.net/jdk/pull/4028