Runtime.exec and SIGPIPE + SIG_IGN

Tue Apr 29 18:07:16 UTC 2025

Hi Thomas, Archie,

Generally, I would agree that a bit more cleanup/resetting of the signal 
state makes sense.
There is general unease about changing this kind of global state that 
has been stable for a long time.
An application that expects to change operating system state before 
calling exec and expecting it to be retained/passed to the child is 
expecting too much.

A change here should also be reviewed by Hotspot, signals are a 
sensitive area for them too.

Regards, Roger

On 4/29/25 10:55 AM, Thomas Stüfe wrote:
> Hi Archie!
>
> On Tue, Apr 29, 2025 at 4:17 PM Archie Cobbs <archie.cobbs at gmail.com> 
> wrote:
>
>     This reminds me of a bug I saw once decades ago: the JVM was not
>     setting the close-on-exec flag on new file descriptors, so child
>     processes were receiving copies of "leaked" file descriptors and
>     keeping them open indefinitely, causing the system to eventually
>     running out of file descriptors and crash.
>
>
> I remember. We fixed that in jspawnhelper.
>
>     The moral of that story was that control of the "process
>     environment" of a new process (i.e., which file descriptors are
>     open, signal handlers installed, etc.) created via the usual
>     fork/exec sequence clearly belongs to the bit of code that is
>     spawning that process for whatever reason: any inheritable part of
>     the process environment that the JVM might mess around with must
>     be reset before spawning child processes.
>
>
> Agreed.
>
>     So if some native code wants to configure SIGPIPE for SIG_IGN and
>     then fork/exec a child process that inherits that setting, then fine.
>
>     But if the JVM itself is doing its fork/exec for some unrelated
>     purpose (e.g., Runtime.exec()), it has the right and
>     responsibility to clean up the environment that the child process
>     is going to inherit to protect it from any changes due to the JVM
>     process, including native code.
>
>     So it seems like there are two possible bugs here:
>
>     1. Some JNI code links in a library that changes the (global)
>     setting for SIGPIPE. Is that allowed by the JNI specification?
>     Does the specification even say? It seems like this is not really
>     kosher, but anyway it happens to work by coincidence because
>     SIG_IGN and javaSignalHandler do the same thing (i.e., discard the
>     signal).
>
>
> To my knowledge the JNI spec says nothing about signals.
>
>     2. The JVM is not completely scrubbing the child's process
>     environment when it spawns a new process as it should: All signal
>     handlers should be reset to their default values (unless there is
>     some JVM-specific reason to set them differently).
>
>     It seems like #2 is a valid bug or at least a valid improvement,
>     and whether #1 is a bug depends on what you believe JNI libraries
>     are officially allowed to do.
>
>
> I agree. Note that we also have a slight inconsistency depending on 
> the order of third-party signal handler installation:
>
> 1) In a scenario like the one I described we start java, it installs 
> the SIGPIPE handler, which then gets preplaced by the third-party lib 
> with SIG_IGN, resulting in child processes running with SIG_IGN for 
> SIGPIPE.
> 2) If, however,r the libjvm.so gets embedded into a custom launcher, 
> which already had set SIG_IGN for SIGPIPE, the JVM will replace that 
> handler with its own, resulting in child processes running with SIG_DFL.
>
> This seems a bit arbitrary to me and seems to support the view that 
> this behavior is not intended.
>
>     -Archie
>
>     On Tue, Apr 29, 2025 at 4:05 AM Thomas Stüfe
>     <thomas.stuefe at gmail.com> wrote:
>
>         Hi,
>
>         I would like to gauge opinions on whether the following
>         scenario is a bug to fix or whether to accept it as standard
>         behavior.
>
>         ---
>
>         A customer has the following problem:
>
>         - The JVM invokes a third-party JNI library that sets the
>         signal disposition of SIGPIPE to SIG_IGN (Boo! in this case,
>         it is the FIPS nspr library, which does this unconditionally.)
>         - The JVM then spawns child processes
>         - All child processes now ignore SIGPIPE, which leads to
>         failures in automation scripts
>
>         I was surprised. I always assumed the signal disposition of
>         all signals would be reset to SIG_DFL when exec'ing. However,
>         seems I was wrong when it came to SIG_IGN. Posix documents for
>         execve() states [1]:
>
>         1) " Signals set to the default action (SIG_DFL) in the
>         calling process image shall be set to the default action in
>         the new process image."
>
>         and
>
>         2) "Except for SIGCHLD, signals set to be ignored (SIG_IGN) by
>         the calling process image shall be set to be ignored by the
>         new process image."
>
>         and
>
>         3) "Signals set to be caught by the calling process image
>         shall be set to the default action in the new process image
>         (see /<signal.h>/
>         <https://urldefense.com/v3/__https://pubs.opengroup.org/onlinepubs/009695399/basedefs/signal.h.html__;!!ACWV5N9M2RV99hQ!JGJYVEFhN3cSMzW41RLH_PK_mMddc3HnFEtCKYUXztE2-_9cjl_tIkMLO3y5_vav3Pt2ARimbCm44kuH8n_No6q2$>)."
>
>         (2) and (3) are the interesting parts. (2) means that if the
>         parent process ignores SIGPIPE, child processes will also
>         ignore SIGPIPE. (3) means that if the parent process has a
>         custom handler installed for SIGPIPE, child processes will be
>         started with SIG_DFL (default action) for SIGPIPE. The default
>         action for SIGPIPE is "terminate process".
>
>         The libjvm handles SIGPIPE. We install our
>         `javaSignalHandler`. We do this to eat up and ignore SIGPIPE.
>         That we "manually" ignore the signal beside the point here -
>         what counts is that we set a custom signal handler for
>         SIGPIPE. Therefore, on execve, we behave according to rule (3)
>         and start the child with SIG_DFL as SIGPIPE disposition. As it
>         should be.
>
>         If third-party code sets the handler to SIG_IGN as in this
>         example, exeve will behave according to rule (2) and the child
>         will start with SIG_IGN as SIGPIPE disposition.
>
>         The libjig can be used to workaround this scenario, but I
>         wonder if that is more of an accident. The libjsig.so will
>         preserve the JVM's SIGPIPE handler even if third-party code
>         attempts to set it to SIG_IGN. That means that exeve still
>         behaves according to rule (2): sets child's SIGPIPE
>         disposition to SIG_DFL.
>
>         ----
>
>         But I wonder whether this should not be considered a bug to
>         fix regardless of the jsig.so workaround? In jspawnhelper, we
>         clean the environment from various effects when exec'ing;
>         among other things, we reset the signal block mask for the
>         process. The "ignore" state of processes could be considered
>         along the same line. We could reset all signal handlers to
>         SIG_DFL before execing the child.
>
>         I know that this area is super-tricky and problems are
>         notoriously difficult to analyze; we should therefore be
>         extremely careful not to break downward compatibility. Still,
>         what do people think? Should be fix this in jspawnhelper?
>
>         Thanks, Thomas
>
>         (cc'ing Roger)
>
>         [1]
>         https://pubs.opengroup.org/onlinepubs/009695399/functions/exec.html
>         <https://urldefense.com/v3/__https://pubs.opengroup.org/onlinepubs/009695399/functions/exec.html__;!!ACWV5N9M2RV99hQ!JGJYVEFhN3cSMzW41RLH_PK_mMddc3HnFEtCKYUXztE2-_9cjl_tIkMLO3y5_vav3Pt2ARimbCm44kuH8vLbF0yj$>
>
>
>
>     -- 
>     Archie L. Cobbs
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/core-libs-dev/attachments/20250429/b06a0146/attachment-0001.htm>