RFR: 8273608: Deadlock when jcmd of OnError attaches to itself

[Thomas Stuefe]

On Wed, 22 Sep 2021 05:59:01 GMT, Xin Liu <xliu at openjdk.org> wrote:

>> src/hotspot/share/utilities/vmError.cpp line 1663:
>> 
>>> 1661:       out.print_raw_cr("\" ...");
>>> 1662: 
>>> 1663:       VMErrorThreadToNativeFromVM ttnfv(JavaThread::current_or_null());
>> 
>> Surely the current thread need not be a JavaThread here.
>
> Make sense. I haven't seen report_and_die() is called by NonJavaThread, but I agree we should cover that case.  it will be no-op for NonJavaThread because safepoint synchronization only checks JavaThreads.

I would move the RAII object out of this scope into the enclosing scope. Do it at the start of `if OnError[0] != NULL`. There is no need to repeatedly do this for every command.

(See my earlier comment, maybe we don't even need an RAII but a simple transition, done once, would be sufficient).

-------------

PR: https://git.openjdk.java.net/jdk/pull/5590