RFR: 8273608: Deadlock when jcmd of OnError attaches to itself [v2]

[Xin Liu]

On Wed, 29 Sep 2021 01:11:16 GMT, David Holmes <dholmes at openjdk.org> wrote:

>> Xin Liu has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains four additional commits since the last revision:
>> 
>>  - Add a new testcase for OutOfMemoryError thrown from NIO.
>>  - Make state changer one way in VMError.
>>    
>>    Add a test to show that jcmd %p won't get stuck.
>>  - Merge branch 'master' into JDK-8273608
>>  - 8273608: Deadlock when jcmd of OnError attaches to itself
>>    
>>    Allow custom command of OnError to attach to HotSpot itself. This patch sets
>>    the thread of report_and_die() to Native before os::fork_and_exec(cmd). This
>>    prevents cmds which require safepoint synchronization from deadlock.
>>    eg. OnError='jcmd %p Thread.print'.
>
> src/hotspot/share/runtime/mutexLocker.cpp line 390:
> 
>> 388:     owned_lock = next;
>> 389:   }
>> 390: #endif // ASSERT
> 
> Should we also clear `_owned_locks` after this so that it is still correct?

In debug build, Thread keeps tracks its owning mutexes and Mutex keeps tracks its owner. 
eg. Mutex::unlock() -> Mutex::set_owner(NULL) -> Mutex::set_owner_implementation(NULL). 

here it deletes owned_lock from its owner.  
https://github.com/openjdk/jdk/blob/master/src/hotspot/share/runtime/mutex.cpp#L483

-------------

PR: https://git.openjdk.java.net/jdk/pull/5590