RFR: 8292695: SIGQUIT and jcmd attaching mechanism does not work with signal chaining library

Tue Aug 30 01:24:34 UTC 2022

On Sat, 27 Aug 2022 21:03:21 GMT, David Holmes <dholmes at openjdk.org> wrote:

> My point is that this change seems to allow installation of a BREAK_SIGNAL handler even without -Xrs!

Got it. This is indeed the case. However, prior to [JDK-8279124](https://bugs.openjdk.org/browse/JDK-8279124), HotSpot also behaved in the same way, that it allows installation of a custom BREAK_SIGNAL handler even without -Xrs.

I verified with a modified TestSigquit.java, [here](https://github.com/caoman/jdk/commit/346132aa5af8d59406d0b98567d18d16868a3a32). It tests the "postpre" and "postpost" mode without -Xrs, which both uses libjsig.so:
- Without any change, the test times out waiting for the custom signal handler to be called.
- With JDK-8279124 [backed out](https://github.com/caoman/jdk/commit/0983baad17d78cf64e1f604abb232e581700a662), OR with this PR, the test passes.

Since HotSpot does not support chaining for SIGQUIT, I think it is OK to leave the behavior undefined if user installs a custom SIGQUIT handler without -Xrs. I.e. the custom handler may or may not work, and it could break HotSpot functionality. libjsig is not designed to prevent installing custom SIGQUIT handler.

-------------

PR: https://git.openjdk.org/jdk/pull/9955