RFR: 8360791: [ubsan] Adjust signal handling when using clang

[Magnus Ihse Bursie]

On Fri, 27 Jun 2025 15:45:22 GMT, Kim Barrett <kbarrett at openjdk.org> wrote:

>> A couple of tests e.g. VendorInfoPluginsTest but also some Hotspot like runtime/ErrorHandling/CreateCoredumpOnCrash.java put (write) to special addresses like 0 to provoke crashs.
>> However this does not work well with ubsan-enabled binaries on the clang based platforms (macOS, AIX).
>> The mentioned tests generate a SIGSEGV.
>> 
>> Some other tests generate a SIGBUS, e.g.
>> 
>> runtime/memory/ReserveMemory.java
>> runtime/memory/ReadFromNoaccessArea.java
>> 
>> and this leads to similar issues with ubsan-enabled binaries.
>> 
>> We should adjust the signal handling with the sanitizer options, how to do this is documented here :
>> https://github.com/google/sanitizers/wiki/SanitizerCommonFlags
>
> make/data/ubsan/ubsan_default_options.c line 69:
> 
>> 67:           "handle_segv=0,"
>> 68:           "handle_sigbus=0,"
>> 69: #endif
> 
> I wonder if this should be unconditional, rather than conditional on `__clang__`.
> 
> And maybe we should similarly disable all of the other UBSAN signal handlers,
> to avoid interfering with those installed by the JVM. UBSAN has potential
> handlers for SIGSEGV, SIGBUS, SIGILL, and SIGFPE.  The JVM installs handlers
> for all of those:
> 
> https://github.com/openjdk/jdk/blob/12196baf6700d00c244747cfa22767e532a4a963/src/hotspot/os/posix/signals_posix.cpp#L1316-L1320
> 
> https://github.com/openjdk/jdk/blob/12196baf6700d00c244747cfa22767e532a4a963/src/hotspot/os/posix/jvm_posix.cpp#L46-L56
> 
> Also, the SanitizerCommonFlags documentation has a bug in the description of
> these handle_XXX options. They are described as having values of 0, 1, or 2,
> but the given defaults are true or false.

I agree, this does not seem to be limited to clang, unless the signal handling stuff differs in ubsan in different compilers.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/26015#discussion_r2172334545