RFR: 8292695: SIGQUIT and jcmd attaching mechanism does not work with signal chaining library

[Thomas Stuefe]

On Thu, 8 Sep 2022 02:50:59 GMT, Man Cao <manc at openjdk.org> wrote:

> > > But there is nothing that prevents us from moving the SIGQUIT handler installation up in time to the hotspot signal handler initialization. Nobody says the SIGQUIT handler has to be installed after the signal dispatcher thread started.
> > 
> > 
> > Thank you for the great idea and experiment! Let me try this approach and rerun the tests.
> 
> Switched to the approach that installs the handler only once.
> 
> The SIGQUIT handler has to be installed after jdk_misc_signal_init() has initialized `pending_signals` and `sig_semaphore`, which are used by `os::signal_notify()`. This means SIGQUIT handler is still installed outside libjsig's JVM_begin_signal_setting/JVM_end_signal_setting window, so it not tracked by libjsig like other HotSpot signals.

You can just move `jdk_misc_signal_init` up a notch in `PosixSignals::init()`, to precede hotspot signal initialization. Semaphore initialization does not depend on anything much, its just a `sem_init()`. See my modified example: 

https://github.com/openjdk/jdk/commit/77826cf8253e5bda0ca7e694a2d100e4988d9e38

Then it works as planned, with SIGQUIT now being part of the normal hotspot VM signal installation.

-------------

PR: https://git.openjdk.org/jdk/pull/9955