<div dir="ltr">Hm. It unfortunately does not show where it was poisoned, as the Arena uses large chunks with multiple separate allocations per chunk. ASan only keeps track of malloc/free. But if I change the Arena implementation when building under ASan to just use a right-sized chunk for each request, it should be able to show us. I'll try that and bump this once I get something more definitive.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Feb 9, 2023 at 1:41 PM <<a href="mailto:dean.long@oracle.com">dean.long@oracle.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Can ASan show where the memory was freed? We've had crashes in
the past (like JDK-8270028) that could be related to memory
corruption or how ResourceArea recycles memory. The allocation
below seems to be using an Arena without a ResourceArea, but if
some other code used the same arena wrapped in a ResourceArea,
then it seems like that could lead to potential problems.<br>
</p>
<p>dl<br>
</p>
<div>On 2/9/23 8:59 AM, Justin King wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi,
<div><br>
</div>
<div>I was looking at instrumenting Arena again for ASan.
The WIP patch is <a href="https://github.com/jcking/jdk/commit/047d4aa9a091cf5a84b9308454862e39666ca253" target="_blank">047d4aa9a091cf5a84b9308454862e39666ca253</a>.
I ran back into the <a href="https://bugs.openjdk.org/browse/JDK-8298984" target="_blank">suspicious logic in C2</a> where
nodes are used after calling Arena::Afree. The first issue is
present in Node::destruct, which I fixed by moving the call to
Arena::Afree to the bottom of the function (addressed in
patch). The second issue came up after in Compile::Compile,
the stack trace is below. It looks like there are residual
freed nodes being operated on? Maybe we are failing to
unregister a temporary node from the node list? Maybe related
to clone_map which returns SafePointNode?</div>
<div><br>
</div>
<div><font face="monospace">==3146540==ERROR: AddressSanitizer:
use-after-poison on address 0x62d00996b370 at pc
0x7f9f93048335 bp 0x7f9ed29fae20 sp 0x7f9ed29fae18<br>
READ of size 4 at 0x62d00996b370 thread T13<br>
#0 0x7f9f93048334 in
Unique_Node_List::remove_useless_nodes(VectorSet&)
src/hotspot/share/opto/node.cpp:2967<br>
#1 0x7f9f932124b3 in
PhaseRemoveUseless::PhaseRemoveUseless(PhaseGVN*,
Unique_Node_List*, Phase::PhaseNumber)
src/hotspot/share/opto/phaseX.cpp:423<br>
#2 0x7f9f91621beb in Compile::Compile(ciEnv*, ciMethod*,
int, Options, DirectiveSet*)
src/hotspot/share/opto/compile.cpp:797<br>
#3 0x7f9f912e37fe in C2Compiler::compile_method(ciEnv*,
ciMethod*, int, bool, DirectiveSet*)
src/hotspot/share/opto/c2compiler.cpp:113<br>
#4 0x7f9f91638e07 in
CompileBroker::invoke_compiler_on_method(CompileTask*)
src/hotspot/share/compiler/compileBroker.cpp:2237<br>
#5 0x7f9f9163bfd7 in
CompileBroker::compiler_thread_loop()
src/hotspot/share/compiler/compileBroker.cpp:1916<br>
#6 0x7f9f921e3eec in JavaThread::thread_main_inner()
src/hotspot/share/runtime/javaThread.cpp:710<br>
#7 0x7f9f921e434f in JavaThread::thread_main_inner()
src/hotspot/share/runtime/javaThread.cpp:689<br>
#8 0x7f9f921e434f in JavaThread::run()
src/hotspot/share/runtime/javaThread.cpp:695<br>
#9 0x7f9f93aa3f55 in Thread::call_run()
src/hotspot/share/runtime/thread.cpp:224<br>
#10 0x7f9f9310144f in thread_native_entry
src/hotspot/os/linux/os_linux.cpp:737<br>
#11 0x7f9f960a7fd3 in start_thread
nptl/pthread_create.c:442<br>
#12 0x7f9f9612866b in clone3
../sysdeps/unix/sysv/linux/x86_64/clone3.S:81<br>
<br>
0x62d00996b370 is located 20336 bytes inside of 32744-byte
region [0x62d009966400,0x62d00996e3e8)<br>
allocated by thread T13 here:<br>
#0 0x7f9f962b89cf in __interceptor_malloc
../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69<br>
#1 0x7f9f930e915a in os::malloc(unsigned long, MEMFLAGS,
NativeCallStack const&)
src/hotspot/share/runtime/os.cpp:672<br>
#2 0x7f9f90c8a08a in Chunk::operator new(unsigned long,
AllocFailStrategy::AllocFailEnum, unsigned long)
src/hotspot/share/memory/arena.cpp:190<br>
#3 0x7f9f90c8a08a in Arena::grow(unsigned long,
AllocFailStrategy::AllocFailEnum)
src/hotspot/share/memory/arena.cpp:325<br>
#4 0x7f9f932109f5 in Arena::internal_amalloc(unsigned
long, AllocFailStrategy::AllocFailEnum)
src/hotspot/share/memory/arena.hpp:113<br>
#5 0x7f9f932109f5 in Arena::Amalloc(unsigned long,
AllocFailStrategy::AllocFailEnum)
src/hotspot/share/memory/arena.hpp:133<br>
#6 0x7f9f932109f5 in NodeHash::NodeHash(Arena*, unsigned
int) src/hotspot/share/opto/phaseX.cpp:68<br>
#7 0x7f9f932293c7 in PhaseValues::PhaseValues(Arena*,
unsigned int) src/hotspot/share/opto/phaseX.cpp:697<br>
#8 0x7f9f9161f678 in PhaseGVN::PhaseGVN(Arena*, unsigned
int) src/hotspot/share/opto/phaseX.hpp:415<br>
#9 0x7f9f9161f678 in Compile::Compile(ciEnv*, ciMethod*,
int, Options, DirectiveSet*)
src/hotspot/share/opto/compile.cpp:714<br>
#10 0x7f9f912e37fe in C2Compiler::compile_method(ciEnv*,
ciMethod*, int, bool, DirectiveSet*)
src/hotspot/share/opto/c2compiler.cpp:113<br>
#11 0x7f9f91638e07 in
CompileBroker::invoke_compiler_on_method(CompileTask*)
src/hotspot/share/compiler/compileBroker.cpp:2237<br>
#12 0x7f9f9163bfd7 in
CompileBroker::compiler_thread_loop()
src/hotspot/share/compiler/compileBroker.cpp:1916<br>
#13 0x7f9f921e3eec in JavaThread::thread_main_inner()
src/hotspot/share/runtime/javaThread.cpp:710<br>
#14 0x7f9f921e434f in JavaThread::thread_main_inner()
src/hotspot/share/runtime/javaThread.cpp:689<br>
#15 0x7f9f921e434f in JavaThread::run()
src/hotspot/share/runtime/javaThread.cpp:695<br>
#16 0x7f9f93aa3f55 in Thread::call_run()
src/hotspot/share/runtime/thread.cpp:224<br>
#17 0x7f9f9310144f in thread_native_entry
src/hotspot/os/linux/os_linux.cpp:737<br>
#18 0x7f9f960a7fd3 in start_thread
nptl/pthread_create.c:442<br>
<br>
Thread T13 created by T1 here:<br>
#0 0x7f9f96249726 in __interceptor_pthread_create
../../../../src/libsanitizer/asan/asan_interceptors.cpp:207<br>
#1 0x7f9f93102d88 in os::create_thread(Thread*,
os::ThreadType, unsigned long)
src/hotspot/os/linux/os_linux.cpp:888<br>
#2 0x7f9f91693d93 in
CompilerThread::CompilerThread(CompileQueue*,
CompilerCounters*)
src/hotspot/share/compiler/compilerThread.cpp:34<br>
#3 0x7f9f91625c7c in
CompileBroker::make_thread(CompileBroker::ThreadType,
_jobject*, CompileQueue*, AbstractCompiler*, JavaThread*)
src/hotspot/share/compiler/compileBroker.cpp:842<br>
#4 0x7f9f91628f71 in
CompileBroker::init_compiler_threads()
src/hotspot/share/compiler/compileBroker.cpp:943<br>
#5 0x7f9f9162a464 in
CompileBroker::compilation_init_phase1(JavaThread*)
src/hotspot/share/compiler/compileBroker.cpp:654<br>
#6 0x7f9f93adc3a4 in Threads::create_vm(JavaVMInitArgs*,
bool*) src/hotspot/share/runtime/threads.cpp:701<br>
#7 0x7f9f92465b51 in JNI_CreateJavaVM_inner
src/hotspot/share/prims/jni.cpp:3588<br>
#8 0x7f9f92465b51 in JNI_CreateJavaVM
src/hotspot/share/prims/jni.cpp:3674<br>
#9 0x7f9f968d2e25 in InitializeJVM
src/java.base/share/native/libjli/java.c:1459<br>
#10 0x7f9f968d2e25 in JavaMain
src/java.base/share/native/libjli/java.c:413<br>
#11 0x7f9f968db708 in ThreadJavaMain
src/java.base/unix/native/libjli/java_md.c:650<br>
#12 0x7f9f960a7fd3 in start_thread
nptl/pthread_create.c:442<br>
<br>
Thread T1 created by T0 here:<br>
#0 0x7f9f96249726 in __interceptor_pthread_create
../../../../src/libsanitizer/asan/asan_interceptors.cpp:207<br>
#1 0x7f9f968dd3a1 in CallJavaMainInNewThread
src/java.base/unix/native/libjli/java_md.c:691<br>
#2 0x7f9f968d822d in ContinueInNewThread
src/java.base/share/native/libjli/java.c:2280<br>
#3 0x7f9f968d96ae in JLI_Launch
src/java.base/share/native/libjli/java.c:340<br>
#4 0x5594a81c337c in main
src/java.base/share/native/launcher/main.c:166<br>
#5 0x7f9f96046189 in __libc_start_call_main
../sysdeps/nptl/libc_start_call_main.h:58</font><br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr"><span><br>
<table style="margin:0px;padding:20px 0px 0px;font-family:"Times New Roman"" cellspacing="0" cellpadding="0" border="0">
<tbody style="margin:0px;padding:0px">
<tr style="margin:0px;padding:0px">
<td style="padding:0px 20px 0px 0px;vertical-align:top;border-right:1px solid rgb(213,213,213)" valign="top"><img src="https://www.gstatic.com/images/branding/product/1x/googleg_64dp.png" alt="Google Logo" style="margin: 0px; padding: 0px; display: block; height: auto;" width="72"></td>
<td style="padding:0px 0px 0px 20px">
<table style="margin:0px;padding:0px" cellspacing="0" cellpadding="0" border="0">
<tbody style="margin:0px;padding:0px">
<tr style="margin:0px;padding:0px">
<td colspan="2" style="padding:1px 0px 5px;font-family:Arial,Helvetica,Verdana,sans-serif;font-size:13px;line-height:13px;color:rgb(56,58,53);font-weight:700">Justin
King</td>
</tr>
<tr style="margin:0px;padding:0px">
<td colspan="2" style="padding:0px 0px 5px;font-family:Arial,Helvetica,Verdana,sans-serif;font-size:11px;line-height:13px;color:rgb(56,58,53)">Software
Engineer</td>
</tr>
<tr style="margin:0px;padding:0px">
<td colspan="2" style="padding:0px 0px 5px;font-family:Arial,Helvetica,Verdana,sans-serif;font-size:11px;line-height:13px;color:rgb(56,58,53)"><a href="mailto:jcking@google.com" target="_blank">jcking@google.com</a></td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<table style="border:none;border-collapse:collapse">
<colgroup><col width="85"><col width="539"></colgroup><tbody>
<tr style="height:0pt">
<td style="border-width:1pt;border-style:solid;border-color:rgb(255,255,255);vertical-align:middle;padding:4.32pt"><br>
</td>
<td style="border-width:1pt;border-style:solid;border-color:rgb(255,255,255);vertical-align:middle;padding:4.32pt"><br>
</td>
</tr>
</tbody>
</table>
</span></div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><span><br><table border="0" cellspacing="0" cellpadding="0" style="margin:0px;padding:20px 0px 0px;font-family:"Times New Roman""><tbody style="margin:0px;padding:0px"><tr style="margin:0px;padding:0px"><td valign="top" style="padding:0px 20px 0px 0px;vertical-align:top;border-right:1px solid rgb(213,213,213)"><img src="https://www.gstatic.com/images/branding/product/1x/googleg_64dp.png" alt="Google Logo" width="72" style="margin: 0px; padding: 0px; display: block; height: auto;"></td><td style="padding:0px 0px 0px 20px"><table border="0" cellspacing="0" cellpadding="0" style="margin:0px;padding:0px"><tbody style="margin:0px;padding:0px"><tr style="margin:0px;padding:0px"><td colspan="2" style="padding:1px 0px 5px;font-family:Arial,Helvetica,Verdana,sans-serif;font-size:13px;line-height:13px;color:rgb(56,58,53);font-weight:700">Justin King</td></tr><tr style="margin:0px;padding:0px"><td colspan="2" style="padding:0px 0px 5px;font-family:Arial,Helvetica,Verdana,sans-serif;font-size:11px;line-height:13px;color:rgb(56,58,53)">Software Engineer</td></tr><tr style="margin:0px;padding:0px"><td colspan="2" style="padding:0px 0px 5px;font-family:Arial,Helvetica,Verdana,sans-serif;font-size:11px;line-height:13px;color:rgb(56,58,53)"><a href="mailto:jcking@google.com" target="_blank">jcking@google.com</a></td></tr></tbody></table></td></tr></tbody></table><table style="border:none;border-collapse:collapse"><colgroup><col width="85"><col width="539"></colgroup><tbody><tr style="height:0pt"><td style="border-width:1pt;border-style:solid;border-color:rgb(255,255,255);vertical-align:middle;padding:4.32pt"></td><td style="border-width:1pt;border-style:solid;border-color:rgb(255,255,255);vertical-align:middle;padding:4.32pt"><br></td></tr></tbody></table></span></div></div>