RFR: 8273608: Deadlock when jcmd of OnError attaches to itself [v7]

[coleen.phillimore at oracle.com]

On 10/12/21 8:04 AM, David Holmes wrote:
> On 12/10/2021 4:15 pm, Xin Liu wrote:
>> On Tue, 12 Oct 2021 05:27:15 GMT, David Holmes <dholmes at openjdk.org> 
>> wrote:
>>
>>>> Xin Liu has updated the pull request incrementally with one 
>>>> additional commit since the last revision:
>>>>
>>>>    Revert JavaThreadInVMAndNative change.
>>>>       Only change the state to Native if current thread is Java 
>>>> Thread and it's in VM.
>>>>    Restore thread state after fork_and_exec() if it's changed.
>>>
>>> src/hotspot/share/utilities/vmError.cpp line 1659:
>>>
>>>> 1657:       VMErrorForceInNative fn(Thread::current_or_null());
>>>> 1658:       if (os::fork_and_exec(cmd) < 0) {
>>>> 1659:         out.print_cr("os::fork_and_exec failed: %s (%s=%d)",
>>>
>>> I'm still concerned about using out.print_cr if you are 
>>> _thread_in_native - is it safe? If it is, and if `print_raw` is also 
>>> safe to execute when in native, then you could just apply the 
>>> VMErrorForceInNative before the while loop.
>>
>> I think it's safe. IIUC, a java thread in Native is safe if we keep 
>> track all oops in handles.
>> In that sense, we prevent them from GC.  A fdStream should have 
>> nothing to do with oops and GC.
>
> Can out.print_cr acquire a lock (e.g. ttyLock?) ? If so then it could 
> block and that would use a ThreadBlockInVM which requires it to be 
> _thread_in_vm. That is the kind of "safety" I'm concerned about.

ttyLock is taken and released inside of print_cr, but it's a nosafepoint 
lock so would not block.
Coleen

>
> David
> -----
>
>> -------------
>>
>> PR: https://git.openjdk.java.net/jdk/pull/5590
>>