RFR: 8368365: ASAN errors should produce hs-err files and core dumps [v4]

[Thomas Stuefe]

On Fri, 26 Sep 2025 14:51:44 GMT, Ashutosh Mehra <asmehra at openjdk.org> wrote:

>> Thomas Stuefe has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   fix
>
> This looks good to me. I have a couple of questions/points:
> 1. Is `__asan_set_error_report_callback` the documented way for applications to install callback? I couldn't find information about this. It would be good to add a link to the doc, if there is one, as a comment in the code.
> 2. Should there be a test for `abort_on_error=1:disable_coredump=0` case where the JVM is expected to generate a core file.

Thank you, @ashu-mehra .

> This looks good to me. I have a couple of questions/points:
> 
> 1. Is `__asan_set_error_report_callback` the documented way for applications to install callback? I couldn't find information about this. It would be good to add a link to the doc, if there is one, as a comment in the code.

Documentation for Asan is sparse in general, but it is documented in the header file. I don't know if it was ever not supported - the dlsym'ing I did out of an abundance of caution in case there are older Asan versions around without that functionality.

> 
> 2. Should there be a test for `abort_on_error=1:disable_coredump=0` case where the JVM is expected to generate a core file.

I thought about that too, but testing for core files is tricky. For one, there are many ways core file dumping could fail. Prediction of the core file  path is difficult (with systemd, you'd need to interpret the sysctl kernel.core-pattern value - we don't even manage to do that correctly in hotspot when displaying the core file dumping message). And with Asan, core file size is a bit unpredictable.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/27446#issuecomment-3340090888