<!DOCTYPE html>
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
</head>
<body>
Hi,<br>
<br>
We've been hitting a problem in production that I think might be a
bug in hotspot's signal handling. Let me know if this should go
somewhere else.<br>
<br>
The issue is that when a native library crashes due to memory
corruption (like an invalid free() call), the JVM exits immediately
without generating any core dump or error report, even though we
have -XX:+CreateCoredumpOnCrash enabled.<br>
<br>
Here's what we're seeing when it crashes:
<pre> munmap_chunk(): invalid pointer</pre>
<br>
Or when using tcmalloc:
<pre> src/tcmalloc.cc:333] Attempt to free invalid pointer 0xffff38000b60</pre>
<br>
We're running with:
<pre> JAVA_TOOL_OPTIONS=-XX:+CreateCoredumpOnCrash -XX:ErrorFile=/core-dumps/hs_err_pid%p.log</pre>
<br>
But when these crashes happen, we get nothing - just the error
message above and the process dies. This makes debugging really
difficult, especially since the crashes happen randomly in
production.<br>
<br>
After digging through the hotspot source, I noticed that signal
handlers are installed for SIGSEGV, SIGBUS, SIGFPE, etc., but not
for SIGABRT:<br>
<br>
<a class="moz-txt-link-freetext"
href="https://github.com/openjdk/jdk/blob/37dc1be67d4c15a040dc99dbc105c3269c65063d/src/hotspot/os/posix/signals_posix.cpp#L1352-L1358">https://github.com/openjdk/jdk/blob/37dc1be67d4c15a040dc99dbc105c3269c65063d/src/hotspot/os/posix/signals_posix.cpp#L1352-L1358</a><br>
<br>
When glibc detects the memory corruption, it calls abort() which
raises SIGABRT. Since there's no handler for it, the JVM can't catch
it and generate the diagnostics.<br>
<br>
To demonstrate the issue, I put together a small reproduction case:<br>
<br>
<a class="moz-txt-link-freetext"
href="https://github.com/atorrescogollo/poc-jdk-sigabrt-coredump-handling">https://github.com/atorrescogollo/poc-jdk-sigabrt-coredump-handling</a><br>
<br>
The repo has a Spring Boot app with three endpoints that show the
problem:<br>
<br>
1. /crash/unsafe - Uses Java Unsafe to write to address 0<br>
Result: SIGSEGV -> Works correctly, generates hs_err file<br>
<br>
2. /crash/null - JNI code that dereferences a null pointer<br>
Result: SIGSEGV -> Works correctly, generates hs_err file<br>
<br>
3. /crash/free - JNI code that calls free() on a stack variable<br>
Result: SIGABRT -> BROKEN, just prints "munmap_chunk():
invalid pointer" and dies<br>
<br>
You can reproduce it with:
<pre> docker-compose up -d
curl localhost:8080/crash/free
docker-compose logs</pre>
<br>
And you'll see it just prints the error and exits, no hs_err file
gets created.<br>
<br>
I also tested a potential fix by adding SIGABRT handling to hotspot.
With that change, scenario 3 correctly generates an hs_err file and
core dump. The patch basically:<br>
<p>
<a class="moz-txt-link-freetext"
href="https://github.com/atorrescogollo/poc-jdk-sigabrt-coredump-bug/blob/main/jdk17.patch">https://github.com/atorrescogollo/poc-jdk-sigabrt-coredump-bug/blob/main/jdk17.patch</a></p>
- Adds set_signal_handler(SIGABRT) in signals_posix.cpp<br>
- Resets SIGABRT to SIG_DFL before calling abort() in os_posix.cpp
to avoid recursive handling<br>
<br>
After applying it, the /crash/free endpoint generates proper
diagnostics:
<pre> # SIGABRT (0x6) at pc=0x0000ffffbd177608 (sent by kill), pid=1, tid=41
# Problematic frame:
# C [libc.so.6+0x87608]
# Core dump will be written. Default location: //core
# An error report file with more information is saved as:
# /core-dumps/java_error1.log</pre>
<br>
I'm not sure if there's a specific reason why SIGABRT isn't handled
currently. If there is, are there any alternative approaches to
capture diagnostics when native libraries trigger abort()? For us
and probably others dealing with native library bugs in production,
having some way to get these diagnostics would be really valuable.<br>
<br>
Thanks,<br>
<br>
Álvaro<br>
<br>
</body>
</html>