RFR: 8279124: VM does not handle SIGQUIT during initialization [v4]

[Xin Liu]

> In early stage of initialization, HotSpot doesn't handle SIGQUIT. The default signal preposition on Linux is to quit the process and generate coredump.
> 
> There are 2 applications for this signal.
> 1. There's a handshake protocol between sun.tools.attach and HotSpot.  VirtualMachineImpl sends SIGQUIT(3) if the AttachListener has not been initialized. It expects "Signal Dispatcher" to handle SIGQUIT and create AttachListener.
> 2. POSIX systems use SIGQUIT as SIGBREAK. After AttachListener is up, HotSpot will reinterpret the signal for thread dump.
> 
> It is possible that HotSpot is still initializing in Threads::create_vm() when SIGQUIT arrives. We should change JVM_HANDLE_XXX_SIGNAL to catch SIGQUIT and ignore it. It is installed os::init_2() and should cover the early stage of initialization. Later on, os::initialize_jdk_signal_support() still overwrites the signal handler of SIGQUIT if ReduceSignalUsage is false(default). 
> 
> Testing 
> 
> Before, this patch, once initialization takes long time, jcmd may quit the java process. 
> 
> $java -Xms64g -XX:+AlwaysPreTouch -Xlog:gc+heap=debug:stderr -XX:ParallelGCThreads=1 &
> [1] 9589
> [0.028s][debug][gc,heap] Minimum heap 68719476736  Initial heap 68719476736  Maximum heap 68719476736
> [0.034s][debug][gc,heap] Running G1 PreTouch with 1 workers for 16384 work units pre-touching 68719476736B.
> $jcmd 9589 VM.flags
> 9589:
> [1]  + 9589 quit       java -Xms64g -XX:+AlwaysPreTouch -Xlog:gc+heap=debug:stderr
> java.io.IOException: No such process
>         at jdk.attach/sun.tools.attach.VirtualMachineImpl.sendQuitTo(Native Method)
>         at jdk.attach/sun.tools.attach.VirtualMachineImpl.<init>(VirtualMachineImpl.java:100)
>         at jdk.attach/sun.tools.attach.AttachProviderImpl.attachVirtualMachine(AttachProviderImpl.java:58)
>         at jdk.attach/com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:207)
>         at jdk.jcmd/sun.tools.jcmd.JCmd.executeCommandForPid(JCmd.java:113)
>         at jdk.jcmd/sun.tools.jcmd.JCmd.main(JCmd.java:97)
> 
> 
> With this patch, neither jcmd nor kill -3 will disrupt java process 45850. 
> 
> $java -Xms64g -XX:+AlwaysPreTouch -XX:ParallelGCThreads=1 -Xlog:os+init=debug:stderr -version &
> [1] 45850
> $ kill -3 45850
> [6.902s][info][os,init] ignore BREAK_SIGNAL in the initialization phase.                                                                                                                                                                  
> $jcmd 45850 VM.flags
> 45850:
> [19.920s][info][os,init] ignore BREAK_SIGNAL in the initialization phase.
> [25.422s][info][os,init] ignore BREAK_SIGNAL in the initialization phase.
> [26.522s][info][os,init] ignore BREAK_SIGNAL in the initialization phase.
> [27.723s][info][os,init] ignore BREAK_SIGNAL in the initialization phase.
> [29.023s][info][os,init] ignore BREAK_SIGNAL in the initialization phase.
> [30.423s][info][os,init] ignore BREAK_SIGNAL in the initialization phase.
> com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file /proc/45850/root/tmp/.java_pid45850: target process 45850 doesn't respond within 10500ms or HotSpot VM not loaded
>         at jdk.attach/sun.tools.attach.VirtualMachineImpl.<init>(VirtualMachineImpl.java:105)
>         at jdk.attach/sun.tools.attach.AttachProviderImpl.attachVirtualMachine(AttachProviderImpl.java:58)
>         at jdk.attach/com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:207)
>         at jdk.jcmd/sun.tools.jcmd.JCmd.executeCommandForPid(JCmd.java:113)
>         at jdk.jcmd/sun.tools.jcmd.JCmd.main(JCmd.java:97)
> $openjdk version "19-internal" 2022-09-20
> OpenJDK Runtime Environment (fastdebug build 19-internal+0-adhoc.xxinliu.jdk)
> OpenJDK 64-Bit Server VM (fastdebug build 19-internal+0-adhoc.xxinliu.jdk, mixed mode, sharing)
> [1]  + 45850 done       java -Xms64g -XX:+AlwaysPreTouch -XX:ParallelGCThreads=1  -version

Xin Liu has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains five additional commits since the last revision:

 - Merge branch 'master' into JDK-8279124
 - Correct the typo.
 - Only install JVM_HANDLE_XXX_SIGNAL when ReduceSignalUsage is
   false(default value).
   
   This patch also adds a log message with 'os+init=info'.
 - Intercept SIGQUIT in the early stage of HotSpot.
 - 8279124: VirtualMachineImpl should check signal handler has installed before sending SIGQUIT

-------------

Changes:
  - all: https://git.openjdk.java.net/jdk/pull/7003/files
  - new: https://git.openjdk.java.net/jdk/pull/7003/files/0dddd00c..5f93d0ef

Webrevs:
 - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=7003&range=03
 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=7003&range=02-03

  Stats: 19148 lines in 564 files changed: 13063 ins; 3468 del; 2617 mod
  Patch: https://git.openjdk.java.net/jdk/pull/7003.diff
  Fetch: git fetch https://git.openjdk.java.net/jdk pull/7003/head:pull/7003

PR: https://git.openjdk.java.net/jdk/pull/7003