RFR: 8273608: Deadlock when jcmd of OnError attaches to itself [v2]

[Xin Liu]

On Wed, 29 Sep 2021 01:10:00 GMT, David Holmes <dholmes at openjdk.org> wrote:

>> Xin Liu has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains four additional commits since the last revision:
>> 
>>  - Add a new testcase for OutOfMemoryError thrown from NIO.
>>  - Make state changer one way in VMError.
>>    
>>    Add a test to show that jcmd %p won't get stuck.
>>  - Merge branch 'master' into JDK-8273608
>>  - 8273608: Deadlock when jcmd of OnError attaches to itself
>>    
>>    Allow custom command of OnError to attach to HotSpot itself. This patch sets
>>    the thread of report_and_die() to Native before os::fork_and_exec(cmd). This
>>    prevents cmds which require safepoint synchronization from deadlock.
>>    eg. OnError='jcmd %p Thread.print'.
>
> src/hotspot/share/runtime/mutexLocker.cpp line 396:
> 
>> 394:        _mutex_array[i]->unlock();
>> 395:      }
>> 396:   }
> 
> Surely we don't need this in a debug build as we already unlocked all owned locks?

This logic is for release build. 

Yes, we don't need them in debug build. We should have released all owning mutexes above. 
It's no-op in debug build because owner() should be NULL.  `thread` isn't  NULL in this function. 

Here is current bookkeeping. From a thread, we can't find all owning mutexes. @pchilano said we can try _mutex_array in release build. I think the idea is that we are supposed to capture failure in debug build. 

| class       | debug                     | release      |   |
|-------------|---------------------------|--------------|---|
| Thread      | _owned_locks (listedlist) | N/A          |   |
| Mutex       | _owner                    | _owner       |   |
| MutexLocker | _mutex_array              | _mutex_array |   |

Shall we fix _mutex_array in release? I think we can change it to GrowableArray and keep track all mutexes in runtime.

-------------

PR: https://git.openjdk.java.net/jdk/pull/5590