RFR: 8368365: ASAN errors should produce hs-err files and core dumps

[Kerem Kat]

On Tue, 23 Sep 2025 07:37:12 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

> When we run with ASAN enabled and ASAN catches an error, it reports, then stops the JVM. hs-err files and crash dumps at that point would be incredibly useful, though. The ASAN error report itself is seldom enlightening since it only contains native stacks.
> 
> This patch makes it possible to get hs-err files and subsequent crash dumps by using the error callback functionality of ASAN. It registers a callback which gets called when ASAN catches an error. We then - carefully - print out the ASAN report to stderr (as ASAN itself would have done) and proceeed to end the JVM with a `fatal()`, which gives us an hs-err file, a callstack at that point including Java frames, and - if enabled - core dumps.
> 
> Tested on Fedora 42 and Debian 12, both manually and by running the new companion jtreg test.

Changes requested by krk (no project role).

src/hotspot/share/sanitizers/address.cpp line 65:

> 63:     log_info(asan)("*** Failed to install JVM callback for ASAN. ASAN errors will not generate hs-err files. ***");
> 64:   }
> 65:   //__asan_set_error_report_callback(asan_error_callback);

Commented code from debugging?

-------------

PR Review: https://git.openjdk.org/jdk/pull/27446#pullrequestreview-3257176050
PR Review Comment: https://git.openjdk.org/jdk/pull/27446#discussion_r2371790409