RFR(S) 8238585: Use handshake for JvmtiEventControllerPrivate::enter_interp_only_mode() and don't make compiled methods on stack not_entrant

[Reingruber, Richard]

// Repost including hotspot runtime and gc lists.
// Dean Long suggested to do so, because the enhancement replaces a vm operation
// with a handshake.
// Original thread: http://mail.openjdk.java.net/pipermail/serviceability-dev/2020-February/030359.html

Hi,

could I please get reviews for this small enhancement in hotspot's jvmti implementation:

Webrev: http://cr.openjdk.java.net/~rrich/webrevs/8238585/webrev.0/
Bug:    https://bugs.openjdk.java.net/browse/JDK-8238585

The change avoids making all compiled methods on stack not_entrant when switching a java thread to
interpreter only execution for jvmti purposes. It is sufficient to deoptimize the compiled frames on stack.

Additionally a handshake is used instead of a vm operation to walk the stack and do the deoptimizations.

Testing: JCK and JTREG tests, also in Xcomp mode with fastdebug and release builds on all platforms.

Thanks, Richard.

See also my question if anyone knows a reason for making the compiled methods not_entrant:
http://mail.openjdk.java.net/pipermail/serviceability-dev/2020-January/030339.html