Runtime.exec and SIGPIPE + SIG_IGN

[Thomas Stüfe]

Hi David,

On Fri, May 9, 2025 at 8:44 AM David Holmes <david.holmes at oracle.com> wrote:

> Hi Thomas,
>
> On 29/04/2025 7:04 pm, Thomas Stüfe wrote:
> > Hi,
> >
> > I would like to gauge opinions on whether the following scenario is a
> > bug to fix or whether to accept it as standard behavior.
> >
> > ---
> >
> > A customer has the following problem:
> >
> > - The JVM invokes a third-party JNI library that sets the signal
> > disposition of SIGPIPE to SIG_IGN (Boo! in this case, it is the FIPS
> > nspr library, which does this unconditionally.)
> > - The JVM then spawns child processes
> > - All child processes now ignore SIGPIPE, which leads to failures in
> > automation scripts
> >
> > I was surprised. I always assumed the signal disposition of all signals
> > would be reset to SIG_DFL when exec'ing. However, seems I was wrong when
> > it came to SIG_IGN. Posix documents for execve() states [1]:
> >
> > 1) " Signals set to the default action (SIG_DFL) in the calling process
> > image shall be set to the default action in the new process image."
> >
> > and
> >
> > 2) "Except for SIGCHLD, signals set to be ignored (SIG_IGN) by the
> > calling process image shall be set to be ignored by the new process
> image."
> >
> > and
> >
> > 3) "Signals set to be caught by the calling process image shall be set
> > to the default action in the new process image (see /<signal.h>/
> > <https://pubs.opengroup.org/onlinepubs/009695399/basedefs/signal.h.html
> >)."
> >
> > (2) and (3) are the interesting parts. (2) means that if the parent
> > process ignores SIGPIPE, child processes will also ignore SIGPIPE. (3)
> > means that if the parent process has a custom handler installed for
> > SIGPIPE, child processes will be started with SIG_DFL (default action)
> > for SIGPIPE. The default action for SIGPIPE is "terminate process".
> >
> > The libjvm handles SIGPIPE. We install our `javaSignalHandler`. We do
> > this to eat up and ignore SIGPIPE. That we "manually" ignore the signal
> > beside the point here - what counts is that we set a custom signal
> > handler for SIGPIPE. Therefore, on execve, we behave according to rule
> > (3) and start the child with SIG_DFL as SIGPIPE disposition. As it
> > should be.
> >
> > If third-party code sets the handler to SIG_IGN as in this example,
> > exeve will behave according to rule (2) and the child will start with
> > SIG_IGN as SIGPIPE disposition.
> >
> > The libjig can be used to workaround this scenario, but I wonder if that
> > is more of an accident. The libjsig.so will preserve the JVM's SIGPIPE
> > handler even if third-party code attempts to set it to SIG_IGN. That
> > means that exeve still behaves according to rule (2): sets child's
> > SIGPIPE disposition to SIG_DFL.
> >
> > ----
> >
> > But I wonder whether this should not be considered a bug to fix
> > regardless of the jsig.so workaround? In jspawnhelper, we clean the
> > environment from various effects when exec'ing; among other things, we
> > reset the signal block mask for the process. The "ignore" state of
> > processes could be considered along the same line. We could reset all
> > signal handlers to SIG_DFL before execing the child.
> >
> > I know that this area is super-tricky and problems are notoriously
> > difficult to analyze; we should therefore be extremely careful not to
> > break downward compatibility. Still, what do people think? Should be fix
> > this in jspawnhelper?
>
> I think at most, the exec'ing code could restore the signals the VM has
> modified to their default disposition - which in practice would mean
> only those set to SIG_IGN would need to explicitly be changed, which in
> turn may mean only SIGPIPE is affected?
>

Agreed, if I fix this in jspawnhelper, I fix it very conservatively for
only this signal.


>
> Even so, a library/application that ignores SIGPIPE and exec's other
> processes which it expects also ignore SIGPIPE, would not be happy with
> such a change.
>
>
The problem is that libjsig, albeit working, is clunky to use in practice
and requires control of the environment the customer may not have.

And there are no good solutions, since whatever decision one does it won't
work for a portion of customers. The way it works now is broken for at
least all accidental users of this FIPS library. They may not even know
that they use this library. And it is inconsistent, since the behavior is
different depending on whether the SIG_IGN had been set before or after VM
creation. So, it differs for custom launchers e.g. that set up signal
handling upfront before loading the libjvm.

My guess would be that the behavior now bothers more customers than a reset
to SIG_DFL for SIGPIPE would.

Cheers, Thomas

P.S. before doing any behavioral change, I plan to add good extensive
tracing to jspawnhelper, in order to make it simple to analyze situations
like this.

In the customer scenario described maybe the simplest fix is to have
> them use libjsig.
>
> Cheers,
> David
>
> >
> > Thanks, Thomas
> >
> > (cc'ing Roger)
> >
> > [1] https://pubs.opengroup.org/onlinepubs/009695399/functions/exec.html
> > <https://pubs.opengroup.org/onlinepubs/009695399/functions/exec.html>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/core-libs-dev/attachments/20250509/dba7af8a/attachment-0001.htm>