RFR: 8292695: SIGQUIT and jcmd attaching mechanism does not work with signal chaining library [v2]

[Man Cao]

On Tue, 30 Aug 2022 06:39:11 GMT, David Holmes <dholmes at openjdk.org> wrote:

>>> My point is that this change seems to allow installation of a BREAK_SIGNAL handler even without -Xrs!
>> 
>> Got it. This is indeed the case. However, prior to [JDK-8279124](https://bugs.openjdk.org/browse/JDK-8279124), HotSpot also behaved in the same way, that it allows installation of a custom BREAK_SIGNAL handler even without -Xrs.
>> 
>> I verified with a modified TestSigquit.java, [here](https://github.com/caoman/jdk/commit/346132aa5af8d59406d0b98567d18d16868a3a32). It tests the "postpre" and "postpost" mode without -Xrs, which both uses libjsig.so:
>> - Without any change, the test times out waiting for the custom signal handler to be called.
>> - With JDK-8279124 [backed out](https://github.com/caoman/jdk/commit/0983baad17d78cf64e1f604abb232e581700a662), OR with this PR, the test passes.
>> 
>> Since HotSpot does not support chaining for SIGQUIT, I think it is OK to leave the behavior undefined if user installs a custom SIGQUIT handler without -Xrs. I.e. the custom handler may or may not work, and it could break HotSpot functionality. libjsig is not designed to prevent installing custom SIGQUIT handler.
>
> I am somewhat surprised by this. My reading of the jsig code is that once the VM has installed a SIGQUIT handler during VM init, then subsequent attempts to install a user handler will add the handler to the set to be chained, but because this is a VM signal the VM will never call that chained handler. With the proposed changes if the SIGQUIT handler has not been installed by the VM then the user handler will be installed directly - though I think the VM's handler would then replace that.
> 
> I find this all rather hard to follow and evaluate. As this seems to fix an issue I guess we can take it as-is and then keep an eye out for any unintended consequences. Though they will likely only come from external users so not something we will know till after the next release.

Updated with tstuefe@'s recommendations. It still keeps the behavior that a user can install a SIGQUIT handler without -Xrs.

-------------

PR: https://git.openjdk.org/jdk/pull/9955