RFR: 8368176: ASAN should not inhibit hs-err file generation

[SendaoYan]

On Sat, 20 Sep 2025 07:33:34 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

> ASAN, when catching an error, will abort the process.
> 
> Two things control this:
> 1) the compiler option `-fsanitize-recover=address` (resp. `-fno-sanitize-recover=address`. This controls whether, once ASAN returns from its error report, the compiler-generated ASAN stubs will abort the process. This is by default set to `-fno-sanitize-recover=address`, so we won't recover.
> 2) The runtime option `halt_on_error` controls whether ASAN itself returns from its error handler or whether it aborts the process. This, by default, is set to `1`, so by default ASAN aborts.
> 
> We "double abort" in the sense that two options are overlaid and both prevent the process from continuing.
> 
> I propose that we set, during build time for ASAN builds, the option `-fsanitize-recover=address`. Now, we can control whether to abort or not using the runtime setting `halt_on_error=0`. By default, we still will abort, since `halt_on_error=1`. So, the default behavior won't change. However, we can now at least decide to do it differently.
> 
> What would that give us?
> 
> By aborting right away, ASAN denies the JVM the option to catch the error and write an hs-err file. Of course, not every error that ASAN catches will result in a segfault or in an assertion. The JVM could lurch on for a bit before it stumbles. However, the chance for the JVM to stop on its own very soon after a memory corruption happens is pretty good. Then we get a hs-err file and a crash dump in close correlation to the error ASAN caught.
> 
> And even if there is no close relationship between the original ASAN error and the eventual segfault/assertion (think ASAN sees a double free, JVM continues, and after a while asserts somewhere else as a remote consequence of the error - the stacks in the hs-err file won't be related to the original error) - the hs-err file is shock-full of helpful information about running threads (see also [JDK-8368124](https://bugs.openjdk.org/browse/JDK-8368124)), memory mappings, JVM flags, etc. All of that would make it easier to understand the ASAN report.
> 
> And even if the JVM survives, one can still attach to the still living process and grab thread dumps, VM.info reports, heap dumps etc.

make/autoconf/jdk-options.m4 line 464:

> 462:         elif test "x$TOOLCHAIN_TYPE" = "xmicrosoft"; then
> 463:           # -Oy- is equivalent to -fno-omit-frame-pointer in GCC/Clang.
> 464:           ASAN_CFLAGS="-fsanitize=address -Oy- -DADDRESS_SANITIZER"

Does the `-fsanitize-recover=address` compiler options supported when TOOLCHAIN_TYPE == microsoft

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/27404#discussion_r2365646161