RFR: 8273608: Deadlock when jcmd of OnError attaches to itself [v2]

[Xin Liu]

> This patch allows the custom commands of OnError to attach to HotSpot itself. 
> It sets the thread of report_and_die() to Native before os::fork_and_exec(cmd). 
> This prevents cmds which require safepoint synchronization from deadlock.
> eg. OnError='jcmd %p Thread.print'.
> 
> Without this patch, we will encounter a deadlock at safepoint synchronization. 
> `"main" #1`  is the very thread which executes `os::fork_and_exec(cmd)`.  
> 
> 
> Aborting due to java.lang.OutOfMemoryError: Java heap space
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  Internal Error (debug.cpp:364), pid=94632, tid=94633
> #  fatal error: OutOfMemory encountered: Java heap space
> #
> # JRE version: OpenJDK Runtime Environment (18.0) (build 18-internal+0-adhoc.xxinliu.jdk)
> # Java VM: OpenJDK 64-Bit Server VM (18-internal+0-adhoc.xxinliu.jdk, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
> # No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # /local/home/xxinliu/JDK-2085/hs_err_pid94632.log
> #
> # -XX:OnError="jcmd %p Thread.print"
> #   Executing /bin/sh -c "jcmd 94632 Thread.print" ...
> 94632:
> [10.616s][warning][safepoint]
> [10.616s][warning][safepoint] # SafepointSynchronize::begin: Timeout detected:
> [10.616s][warning][safepoint] # SafepointSynchronize::begin: Timed out while spinning to reach a safepoint.
> [10.616s][warning][safepoint] # SafepointSynchronize::begin: Threads which did not reach the safepoint:
> [10.616s][warning][safepoint] # "main" #1 prio=5 os_prio=0 cpu=236.97ms elapsed=10.61s tid=0x00007f01b00232f0 nid=94633 runnable  [0x00007f01b7a08000]
> [10.616s][warning][safepoint]    java.lang.Thread.State: RUNNABLE
> [10.616s][warning][safepoint]
> [10.616s][warning][safepoint] # SafepointSynchronize::begin: (End of list)

Xin Liu has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains four additional commits since the last revision:

 - Add a new testcase for OutOfMemoryError thrown from NIO.
 - Make state changer one way in VMError.
   
   Add a test to show that jcmd %p won't get stuck.
 - Merge branch 'master' into JDK-8273608
 - 8273608: Deadlock when jcmd of OnError attaches to itself
   
   Allow custom command of OnError to attach to HotSpot itself. This patch sets
   the thread of report_and_die() to Native before os::fork_and_exec(cmd). This
   prevents cmds which require safepoint synchronization from deadlock.
   eg. OnError='jcmd %p Thread.print'.

-------------

Changes:
  - all: https://git.openjdk.java.net/jdk/pull/5590/files
  - new: https://git.openjdk.java.net/jdk/pull/5590/files/afd1610d..bf684e5b

Webrevs:
 - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=5590&range=01
 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=5590&range=00-01

  Stats: 31232 lines in 1010 files changed: 21697 ins; 4654 del; 4881 mod
  Patch: https://git.openjdk.java.net/jdk/pull/5590.diff
  Fetch: git fetch https://git.openjdk.java.net/jdk pull/5590/head:pull/5590

PR: https://git.openjdk.java.net/jdk/pull/5590