RFR: 8273608: Deadlock when jcmd of OnError attaches to itself [v2]

[Xin Liu]

On Wed, 29 Sep 2021 01:22:05 GMT, David Holmes <dholmes at openjdk.org> wrote:

>> Xin Liu has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains four additional commits since the last revision:
>> 
>>  - Add a new testcase for OutOfMemoryError thrown from NIO.
>>  - Make state changer one way in VMError.
>>    
>>    Add a test to show that jcmd %p won't get stuck.
>>  - Merge branch 'master' into JDK-8273608
>>  - 8273608: Deadlock when jcmd of OnError attaches to itself
>>    
>>    Allow custom command of OnError to attach to HotSpot itself. This patch sets
>>    the thread of report_and_die() to Native before os::fork_and_exec(cmd). This
>>    prevents cmds which require safepoint synchronization from deadlock.
>>    eg. OnError='jcmd %p Thread.print'.
>
> test/hotspot/jtreg/runtime/ErrorHandling/TestOutOfMemoryErrorFromNIO.java line 89:
> 
>> 87:         output_single.stdoutShouldMatch("^" + msg); // match start of line only
>> 88: 
>> 89:         System.out.println("PASSED");
> 
> Should we also check for output related to execution of the OnError command?

This testcase will get stuck if I delete `transition_into_native()` in vmErrror.cpp
I will try to verify outputs from before command and after command.

-------------

PR: https://git.openjdk.java.net/jdk/pull/5590