RFR: 8368365: ASAN errors should produce hs-err files and core dumps [v4]

[Thomas Stuefe]

On Wed, 24 Sep 2025 14:23:45 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

>> When we run with ASAN enabled and ASAN catches an error, it reports, then stops the JVM. hs-err files and crash dumps at that point would be incredibly useful, though. The ASAN error report itself is seldom enlightening since it only contains native stacks.
>> 
>> This patch makes it possible to get hs-err files and subsequent crash dumps by using the error callback functionality of ASAN. It registers a callback which gets called when ASAN catches an error. We then - carefully - print out the ASAN report to stderr (as ASAN itself would have done) and proceeed to end the JVM with a `fatal()`, which gives us an hs-err file, a callstack at that point including Java frames, and - if enabled - core dumps.
>> 
>> Tested on Fedora 42 and Debian 12, both manually and by running the new companion jtreg test.
>
> Thomas Stuefe has updated the pull request incrementally with one additional commit since the last revision:
> 
>   fix

Hi all,

After feedback by @MBaesken and the question of @dean-long, I reworked the patch to make the JVM exhibit exactly the same behaviors with regard to core files that a standard ASAN-instrumented binary would show. There is a longer story behind that; see the lengthy comment in address.cpp.

The gist of it: ASAN, by default, inhibits core file generation. To get cores, one needs to set `abort_on_error=1` and `disable_coredump=0`. The first version of my patch was more permissive and allowed core files despite these settings; that was bad (thanks @MBaesken for reminding me) since in mass integration tests done with ASAN, you don't want cores enabled. Note that cores can also get enormous with ASAN.

The new patch exhibits the same behavior as a standard ASAN-instrumented binary: core files are only generated if `abort_on_error=1` and `disable_coredump=0`. By default, this prevents core files. This overrules +CreateCoredumpOnCrash in the JVM (by default enabled).

I think this is a good compromise. Anyone wanting to get cores with ASAN would use the standard ASAN options for this.

@dean-long 
> I haven't tried it yet, but what happens if we cause ASAN to abort by setting the flag in the environment:
export ASAN_OPTIONS=abort_on_error=1. Do we get a useful stack trace in the hs_err file?

Yes, hs-err files are now generated in all cases. That is independent of core generation. 

We now 1) print the ASAN report to stderr, then 2) dump an hs-err file, which also contains the ASAN report, then optionally 3) create a core dump.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/27446#issuecomment-3328850811