RFR: 8273608: Deadlock when jcmd of OnError attaches to itself [v8]

[Xin Liu]

On Wed, 13 Oct 2021 05:02:50 GMT, David Holmes <dholmes at openjdk.org> wrote:

>> src/hotspot/share/utilities/vmError.cpp line 1646:
>> 
>>> 1644:     // at safepoints.
>>> 1645:     VMErrorForceInNative fn(Thread::current_or_null());
>>> 1646: 
>> 
>> I have similar concerns as David about printing to acquire locks. Even if that is not a problem with tty specifically, its something to keep in mind (e.g. with the recent attempt to print logging via network sockets, such things can creep in).
>> 
>> I would in that case move the RAII object close around fork_and_exec. Only do this for fork, and undo it after returning, doing the printing stuff in the original state instead.
>
> I suggested moving the RAII object so it is only used once - under the assumption that output stream usage was safe to do while in native. Otherwise we need this:
> 
> int ret = 0;
> {
>   VMErrorForceInNative fn(_thread);
>   ret = os::fork_and_exec(cmd);
> }
> if (ret < 0) {
>   out.print_cr("os::fork_and_exec failed: %s (%s=%d)",
>                 os::strerror(errno), os::errno_name(errno), errno);
> }
> 
> which is a bit clunky but okay I can live with it. And note we already have `_thread` to use so don't need to call `Thread::current_or_null()`  (which should have been `Thread::current_or_null_safe()` in any case.)

It's risky indeed.  if we can't eradicate the deadlock situation, I feel it's not worth it. 
Even though ttyLock works in Native now, it doesn't mean it will be safe in the future.

-------------

PR: https://git.openjdk.java.net/jdk/pull/5590