RFR: 8360791: [ubsan] Adjust signal handling when using clang

[Kim Barrett]

On Fri, 27 Jun 2025 09:30:19 GMT, Matthias Baesken <mbaesken at openjdk.org> wrote:

> A couple of tests e.g. VendorInfoPluginsTest but also some Hotspot like runtime/ErrorHandling/CreateCoredumpOnCrash.java put (write) to special addresses like 0 to provoke crashs.
> However this does not work well with ubsan-enabled binaries on the clang based platforms (macOS, AIX).
> The mentioned tests generate a SIGSEGV.
> 
> Some other tests generate a SIGBUS, e.g.
> 
> runtime/memory/ReserveMemory.java
> runtime/memory/ReadFromNoaccessArea.java
> 
> and this leads to similar issues with ubsan-enabled binaries.
> 
> We should adjust the signal handling with the sanitizer options, how to do this is documented here :
> https://github.com/google/sanitizers/wiki/SanitizerCommonFlags

make/data/ubsan/ubsan_default_options.c line 69:

> 67:           "handle_segv=0,"
> 68:           "handle_sigbus=0,"
> 69: #endif

I wonder if this should be unconditional, rather than conditional on `__clang__`.

And maybe we should similarly disable all of the other UBSAN signal handlers,
to avoid interfering with those installed by the JVM. UBSAN has potential
handlers for SIGSEGV, SIGBUS, SIGILL, and SIGFPE.  The JVM installs handlers
for all of those:

https://github.com/openjdk/jdk/blob/12196baf6700d00c244747cfa22767e532a4a963/src/hotspot/os/posix/signals_posix.cpp#L1316-L1320

https://github.com/openjdk/jdk/blob/12196baf6700d00c244747cfa22767e532a4a963/src/hotspot/os/posix/jvm_posix.cpp#L46-L56

Also, the SanitizerCommonFlags documentation has a bug in the description of
these handle_XXX options. They are described as having values of 0, 1, or 2,
but the given defaults are true or false.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/26015#discussion_r2172316093