RFR: 8273608: Deadlock when jcmd of OnError attaches to itself [v5]

[Xin Liu]

> This patch allows the custom commands of OnError to attach to HotSpot itself. 
> It sets the thread of report_and_die() to Native before os::fork_and_exec(cmd). 
> This prevents cmds which require safepoint synchronization from deadlock.
> eg. OnError='jcmd %p Thread.print'.
> 
> Without this patch, we will encounter a deadlock at safepoint synchronization. 
> `"main" #1`  is the very thread which executes `os::fork_and_exec(cmd)`.  
> 
> 
> Aborting due to java.lang.OutOfMemoryError: Java heap space
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  Internal Error (debug.cpp:364), pid=94632, tid=94633
> #  fatal error: OutOfMemory encountered: Java heap space
> #
> # JRE version: OpenJDK Runtime Environment (18.0) (build 18-internal+0-adhoc.xxinliu.jdk)
> # Java VM: OpenJDK 64-Bit Server VM (18-internal+0-adhoc.xxinliu.jdk, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
> # No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # /local/home/xxinliu/JDK-2085/hs_err_pid94632.log
> #
> # -XX:OnError="jcmd %p Thread.print"
> #   Executing /bin/sh -c "jcmd 94632 Thread.print" ...
> 94632:
> [10.616s][warning][safepoint]
> [10.616s][warning][safepoint] # SafepointSynchronize::begin: Timeout detected:
> [10.616s][warning][safepoint] # SafepointSynchronize::begin: Timed out while spinning to reach a safepoint.
> [10.616s][warning][safepoint] # SafepointSynchronize::begin: Threads which did not reach the safepoint:
> [10.616s][warning][safepoint] # "main" #1 prio=5 os_prio=0 cpu=236.97ms elapsed=10.61s tid=0x00007f01b00232f0 nid=94633 runnable  [0x00007f01b7a08000]
> [10.616s][warning][safepoint]    java.lang.Thread.State: RUNNABLE
> [10.616s][warning][safepoint]
> [10.616s][warning][safepoint] # SafepointSynchronize::begin: (End of list)

Xin Liu has updated the pull request incrementally with one additional commit since the last revision:

  Update per reviewers' feedbacks.
  
  1. rename the test to TestOnErrorWithSelfAttachingJCmd.java.
  2. rename java_current_transition_into_native() and add the static qualifier.
  3. make unlock_locks_on_error more general. ensure formal name is consistent
  with argument name.

-------------

Changes:
  - all: https://git.openjdk.java.net/jdk/pull/5590/files
  - new: https://git.openjdk.java.net/jdk/pull/5590/files/a9d439ad..2877637e

Webrevs:
 - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=5590&range=04
 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=5590&range=03-04

  Stats: 192 lines in 5 files changed: 91 ins; 95 del; 6 mod
  Patch: https://git.openjdk.java.net/jdk/pull/5590.diff
  Fetch: git fetch https://git.openjdk.java.net/jdk pull/5590/head:pull/5590

PR: https://git.openjdk.java.net/jdk/pull/5590