RFR: 8368176: ASAN should not inhibit hs-err file generation

[SendaoYan]

On Sat, 20 Sep 2025 07:33:34 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

> ASAN, when catching an error, will abort the process.
> 
> Two things control this:
> 1) the compiler option `-fsanitize-recover=address` (resp. `-fno-sanitize-recover=address`. This controls whether, once ASAN returns from its error report, the compiler-generated ASAN stubs will abort the process. This is by default set to `-fno-sanitize-recover=address`, so we won't recover.
> 2) The runtime option `halt_on_error` controls whether ASAN itself returns from its error handler or whether it aborts the process. This, by default, is set to `1`, so by default ASAN aborts.
> 
> We "double abort" in the sense that two options are overlaid and both prevent the process from continuing.
> 
> I propose that we set, during build time for ASAN builds, the option `-fsanitize-recover=address`. Now, we can control whether to abort or not using the runtime setting `halt_on_error=0`. By default, we still will abort, since `halt_on_error=1`. So, the default behavior won't change. However, we can now at least decide to do it differently.
> 
> What would that give us?
> 
> By aborting right away, ASAN denies the JVM the option to catch the error and write an hs-err file. Of course, not every error that ASAN catches will result in a segfault or in an assertion. The JVM could lurch on for a bit before it stumbles. However, the chance for the JVM to stop on its own very soon after a memory corruption happens is pretty good. Then we get a hs-err file and a crash dump in close correlation to the error ASAN caught.
> 
> And even if there is no close relationship between the original ASAN error and the eventual segfault/assertion (think ASAN sees a double free, JVM continues, and after a while asserts somewhere else as a remote consequence of the error - the stacks in the hs-err file won't be related to the original error) - the hs-err file is shock-full of helpful information about running threads (see also [JDK-8368124](https://bugs.openjdk.org/browse/JDK-8368124)), memory mappings, JVM flags, etc. All of that would make it easier to understand the ASAN report.
> 
> And even if the JVM survives, one can still attach to the still living process and grab thread dumps, VM.info reports, heap dumps etc.

Hi, jdk support address sanitizer and leak sanitizer for now. It seems that this PR only concern address sanitizer, does the leak sanitizer also need this enhancement

-------------

PR Comment: https://git.openjdk.org/jdk/pull/27404#issuecomment-3315002815