RFR: 8311557: [JVMCI] deadlock with JVMTI thread suspension

[Tom Rodriguez]

On Fri, 7 Jul 2023 06:13:21 GMT, Tom Rodriguez <never at openjdk.org> wrote:

> Java based JVMCI compiler threads are more like normal Java threads so they aren't `hidden_from_external_view` like the native compilers.  This can leak to deadlocks if you use JVMTI to suspend all threads since this will block the compiler queue and can block execution if background compilation is disabled.  It's reasonable to treat libgraal threads like native threads in this regard.  Making jargraal threads hidden too would interfere with using profiling and debugging tool on them so I've left that alone but it might be worth changing the JVMTI suspend and resume functions to explicitly skip compiler threads as well.

I did some local testing and unsurprisingly this changes make it impossible to attach a debugger to these threads.  In rare cases during development it might be necessary to this since there are still some helper Java calls with native JVMCI.  So I've added a new flag LibJVMCICompilerThreadHidden to control this behaviour.  I also refactored the code a bit because I figured I was missing some JVMCI_ONLY macros in the shared code.  Could I get a rereview?

-------------

PR Comment: https://git.openjdk.org/jdk/pull/14799#issuecomment-1672108594