RFR: 8368365: ASAN errors should produce hs-err files and core dumps [v4]

[Ashutosh Mehra]

On Wed, 24 Sep 2025 14:23:45 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

>> When we run with ASAN enabled and ASAN catches an error, it reports, then stops the JVM. hs-err files and crash dumps at that point would be incredibly useful, though. The ASAN error report itself is seldom enlightening since it only contains native stacks.
>> 
>> After this patch, the JVM will always produce hs-err files when an ASAN-report happens. It will *only* produce core files if ASAN_OPTIONS `disable_coredump=0` and `abort_on_error=1` and the JVM option `CreateCoredumpOnCrash` had not been disabled (and the limit for core file size is high enough etc, all the usual restrictions on OS level still apply).
>> 
>> This means that ASAN builds, by default, will continue to *not* generate cores, since ASAN default options inhibit that. See detail in the comments below.
>> 
>> ---
>> 
>> Tested on Fedora 42 and Debian 12, both manually and by running the new companion jtreg test.
>
> Thomas Stuefe has updated the pull request incrementally with one additional commit since the last revision:
> 
>   fix

This looks good to me. I have a couple of questions/points:
1. Is `__asan_set_error_report_callback` the documented way for applications to install callback? I couldn't find information about this. It would be good to add a link to the doc, if there is one, as a comment in the code.
2. Should there be a test for `abort_on_error=1:disable_coredump=0` case where the JVM is expected to generate a core file.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/27446#issuecomment-3339046947