RFR: 8273608: Deadlock when jcmd of OnError attaches to itself [v7]

[David Holmes]

On 12/10/2021 4:15 pm, Xin Liu wrote:
> On Tue, 12 Oct 2021 05:27:15 GMT, David Holmes <dholmes at openjdk.org> wrote:
> 
>>> Xin Liu has updated the pull request incrementally with one additional commit since the last revision:
>>>
>>>    Revert JavaThreadInVMAndNative change.
>>>    
>>>    Only change the state to Native if current thread is Java Thread and it's in VM.
>>>    Restore thread state after fork_and_exec() if it's changed.
>>
>> src/hotspot/share/utilities/vmError.cpp line 1659:
>>
>>> 1657:       VMErrorForceInNative fn(Thread::current_or_null());
>>> 1658:       if (os::fork_and_exec(cmd) < 0) {
>>> 1659:         out.print_cr("os::fork_and_exec failed: %s (%s=%d)",
>>
>> I'm still concerned about using out.print_cr if you are _thread_in_native - is it safe? If it is, and if `print_raw` is also safe to execute when in native, then you could just apply the VMErrorForceInNative before the while loop.
> 
> I think it's safe. IIUC, a java thread in Native is safe if we keep track all oops in handles.
> In that sense, we prevent them from GC.  A fdStream should have nothing to do with oops and GC.

Can out.print_cr acquire a lock (e.g. ttyLock?) ? If so then it could 
block and that would use a ThreadBlockInVM which requires it to be 
_thread_in_vm. That is the kind of "safety" I'm concerned about.

David
-----

> -------------
> 
> PR: https://git.openjdk.java.net/jdk/pull/5590
>