RFR: 8273608: Deadlock when jcmd of OnError attaches to itself [v8]

[David Holmes]

On Wed, 13 Oct 2021 04:39:33 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

>> Xin Liu has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Change to VM unconditionally as long as current thread is JavaThread.
>>   
>>   Hoist VMErrorForceNative out of While.
>
> src/hotspot/share/utilities/vmError.cpp line 1646:
> 
>> 1644:     // at safepoints.
>> 1645:     VMErrorForceInNative fn(Thread::current_or_null());
>> 1646: 
> 
> I have similar concerns as David about printing to acquire locks. Even if that is not a problem with tty specifically, its something to keep in mind (e.g. with the recent attempt to print logging via network sockets, such things can creep in).
> 
> I would in that case move the RAII object close around fork_and_exec. Only do this for fork, and undo it after returning, doing the printing stuff in the original state instead.

I suggested moving the RAII object so it is only used once - under the assumption that output stream usage was safe to do while in native. Otherwise we need this:

int ret = 0;
{
  VMErrorForceInNative fn(_thread);
  ret = os::fork_and_exec(cmd);
}
if (ret < 0) {
  out.print_cr("os::fork_and_exec failed: %s (%s=%d)",
                os::strerror(errno), os::errno_name(errno), errno);
}

which is a bit clunky but okay I can live with it. And note we already have `_thread` to use so don't need to call `Thread::current_or_null()`  (which should have been `Thread::current_or_null_safe()` in any case.)

-------------

PR: https://git.openjdk.java.net/jdk/pull/5590