<div dir="ltr">Looks like there is definitely a lingering use-after-free like bug, maybe multiple. One is related to the copying around of Unique_Node_List and the copy then modifying the list, causing a Arealloc, with the original (which is then used again) seeing the free'd block. I'll see about pinpointing the issues at a later date.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Feb 14, 2023 at 9:27 AM Vladimir Kozlov <<a href="mailto:vladimir.kozlov@oracle.com">vladimir.kozlov@oracle.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Justin,<br>
<br>
Please file a bug for this issue. We need to keep records.<br>
<br>
Thanks,<br>
Vladimir K<br>
<br>
On 2/10/23 12:42 PM, Justin King wrote:<br>
> Hm. That size is still too big, I don't think the node is 224 bytes. Let me double check again.<br>
> <br>
> On Fri, Feb 10, 2023 at 12:06 PM Justin King <<a href="mailto:jcking@google.com" target="_blank">jcking@google.com</a> <mailto:<a href="mailto:jcking@google.com" target="_blank">jcking@google.com</a>>> wrote:<br>
> <br>
> Looks to be the temporary node created by clone_map is leaking. I think there is a missing call to undo some of the<br>
> work done by clone_map.<br>
> <br>
> ==3591618==ERROR: AddressSanitizer: use-after-poison on address 0x6110000e20b8 at pc 0x7f7d8cac1875 bp<br>
> 0x7f7d1028be20 sp 0x7f7d1028be18<br>
> READ of size 4 at 0x6110000e20b8 thread T13<br>
> #0 0x7f7d8cac1874 in Unique_Node_List::remove_useless_nodes(VectorSet&) src/hotspot/share/opto/node.cpp:2967<br>
> #1 0x7f7d8cc90b03 in PhaseRemoveUseless::PhaseRemoveUseless(PhaseGVN*, Unique_Node_List*, Phase::PhaseNumber)<br>
> src/hotspot/share/opto/phaseX.cpp:423<br>
> #2 0x7f7d8b03853b in Compile::Compile(ciEnv*, ciMethod*, int, Options, DirectiveSet*)<br>
> src/hotspot/share/opto/compile.cpp:797<br>
> #3 0x7f7d8ace8ece in C2Compiler::compile_method(ciEnv*, ciMethod*, int, bool, DirectiveSet*)<br>
> src/hotspot/share/opto/c2compiler.cpp:113<br>
> #4 0x7f7d8b0507f8 in CompileBroker::invoke_compiler_on_method(CompileTask*)<br>
> src/hotspot/share/compiler/compileBroker.cpp:2237<br>
> #5 0x7f7d8b053e57 in CompileBroker::compiler_thread_loop() src/hotspot/share/compiler/compileBroker.cpp:1916<br>
> #6 0x7f7d8bc1f3a8 in JavaThread::thread_main_inner() src/hotspot/share/runtime/javaThread.cpp:710<br>
> #7 0x7f7d8bc1f99f in JavaThread::thread_main_inner() src/hotspot/share/runtime/javaThread.cpp:689<br>
> #8 0x7f7d8bc1f99f in JavaThread::run() src/hotspot/share/runtime/javaThread.cpp:695<br>
> #9 0x7f7d8d535b35 in Thread::call_run() src/hotspot/share/runtime/thread.cpp:224<br>
> #10 0x7f7d8cb7de8f in thread_native_entry src/hotspot/os/linux/os_linux.cpp:737<br>
> #11 0x7f7d8faa7fd3 in start_thread nptl/pthread_create.c:442<br>
> #12 0x7f7d8fb2866b in clone3 ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81<br>
> <br>
> 0x6110000e20b8 is located 56 bytes inside of 224-byte region [0x6110000e2080,0x6110000e2160)<br>
> allocated by thread T13 here:<br>
> #0 0x7f7d8fcb89cf in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69<br>
> #1 0x7f7d8cb65b9a in os::malloc(unsigned long, MEMFLAGS, NativeCallStack const&)<br>
> src/hotspot/share/runtime/os.cpp:672<br>
> #2 0x7f7d8a68c47e in Chunk::operator new(unsigned long, AllocFailStrategy::AllocFailEnum, unsigned long)<br>
> src/hotspot/share/memory/arena.cpp:190<br>
> #3 0x7f7d8a68c47e in Arena::grow(unsigned long, AllocFailStrategy::AllocFailEnum)<br>
> src/hotspot/share/memory/arena.cpp:345<br>
> #4 0x7f7d8cac1c44 in Arena::internal_amalloc(unsigned long, AllocFailStrategy::AllocFailEnum)<br>
> src/hotspot/share/memory/arena.hpp:113<br>
> #5 0x7f7d8cac1c44 in Arena::AmallocWords(unsigned long, AllocFailStrategy::AllocFailEnum)<br>
> src/hotspot/share/memory/arena.hpp:140<br>
> #6 0x7f7d8cac1c44 in Node::clone() const src/hotspot/share/opto/node.cpp:495<br>
> #7 0x7f7d8b980c38 in GraphKit::clone_map() src/hotspot/share/opto/graphKit.cpp:727<br>
> #8 0x7f7d8c538b44 in LibraryCallKit::inline_unsafe_load_store(BasicType, LibraryCallKit::LoadStoreKind,<br>
> LibraryCallKit::AccessKind) src/hotspot/share/opto/library_call.cpp:2599<br>
> #9 0x7f7d8c5aad3e in LibraryCallKit::try_to_inline(int) src/hotspot/share/opto/library_call.cpp:416<br>
> #10 0x7f7d8c5ae34b in LibraryIntrinsic::generate(JVMState*) src/hotspot/share/opto/library_call.cpp:116<br>
> #11 0x7f7d8b4430e9 in Parse::do_call() src/hotspot/share/opto/doCall.cpp:662<br>
> #12 0x7f7d8cc4c1ef in Parse::do_one_bytecode() src/hotspot/share/opto/parse2.cpp:2704<br>
> #13 0x7f7d8cc1f3f6 in Parse::do_one_block() src/hotspot/share/opto/parse1.cpp:1554<br>
> #14 0x7f7d8cc207ce in Parse::do_all_blocks() src/hotspot/share/opto/parse1.cpp:706<br>
> #15 0x7f7d8cc2a214 in Parse::Parse(JVMState*, ciMethod*, float) src/hotspot/share/opto/parse1.cpp:613<br>
> #16 0x7f7d8acec235 in ParseGenerator::generate(JVMState*) src/hotspot/share/opto/callGenerator.cpp:99<br>
> #17 0x7f7d8b03738c in Compile::Compile(ciEnv*, ciMethod*, int, Options, DirectiveSet*)<br>
> src/hotspot/share/opto/compile.cpp:763<br>
> #18 0x7f7d8ace8ece in C2Compiler::compile_method(ciEnv*, ciMethod*, int, bool, DirectiveSet*)<br>
> src/hotspot/share/opto/c2compiler.cpp:113<br>
> #19 0x7f7d8b0507f8 in CompileBroker::invoke_compiler_on_method(CompileTask*)<br>
> src/hotspot/share/compiler/compileBroker.cpp:2237<br>
> #20 0x7f7d8b053e57 in CompileBroker::compiler_thread_loop() src/hotspot/share/compiler/compileBroker.cpp:1916<br>
> #21 0x7f7d8bc1f3a8 in JavaThread::thread_main_inner() src/hotspot/share/runtime/javaThread.cpp:710<br>
> #22 0x7f7d8bc1f99f in JavaThread::thread_main_inner() src/hotspot/share/runtime/javaThread.cpp:689<br>
> #23 0x7f7d8bc1f99f in JavaThread::run() src/hotspot/share/runtime/javaThread.cpp:695<br>
> #24 0x7f7d8d535b35 in Thread::call_run() src/hotspot/share/runtime/thread.cpp:224<br>
> #25 0x7f7d8cb7de8f in thread_native_entry src/hotspot/os/linux/os_linux.cpp:737<br>
> #26 0x7f7d8faa7fd3 in start_thread nptl/pthread_create.c:442<br>
> <br>
> <br>
> <br>
> On Thu, Feb 9, 2023 at 2:31 PM Justin King <<a href="mailto:jcking@google.com" target="_blank">jcking@google.com</a> <mailto:<a href="mailto:jcking@google.com" target="_blank">jcking@google.com</a>>> wrote:<br>
> <br>
> Hm. It unfortunately does not show where it was poisoned, as the Arena uses large chunks with multiple separate<br>
> allocations per chunk. ASan only keeps track of malloc/free. But if I change the Arena implementation when<br>
> building under ASan to just use a right-sized chunk for each request, it should be able to show us. I'll try<br>
> that and bump this once I get something more definitive.<br>
> <br>
> On Thu, Feb 9, 2023 at 1:41 PM <<a href="mailto:dean.long@oracle.com" target="_blank">dean.long@oracle.com</a> <mailto:<a href="mailto:dean.long@oracle.com" target="_blank">dean.long@oracle.com</a>>> wrote:<br>
> <br>
> Can ASan show where the memory was freed? We've had crashes in the past (like JDK-8270028) that could be<br>
> related to memory corruption or how ResourceArea recycles memory. The allocation below seems to be using an<br>
> Arena without a ResourceArea, but if some other code used the same arena wrapped in a ResourceArea, then it<br>
> seems like that could lead to potential problems.<br>
> <br>
> dl<br>
> <br>
> On 2/9/23 8:59 AM, Justin King wrote:<br>
>> Hi,<br>
>><br>
>> I was looking at instrumenting Arena again for ASan. The WIP patch is<br>
>> 047d4aa9a091cf5a84b9308454862e39666ca253<br>
>> <<a href="https://github.com/jcking/jdk/commit/047d4aa9a091cf5a84b9308454862e39666ca253" rel="noreferrer" target="_blank">https://github.com/jcking/jdk/commit/047d4aa9a091cf5a84b9308454862e39666ca253</a>>. I ran back into the<br>
>> suspicious logic in C2 <<a href="https://bugs.openjdk.org/browse/JDK-8298984" rel="noreferrer" target="_blank">https://bugs.openjdk.org/browse/JDK-8298984</a>> where nodes are used after calling<br>
>> Arena::Afree. The first issue is present in Node::destruct, which I fixed by moving the call to<br>
>> Arena::Afree to the bottom of the function (addressed in patch). The second issue came up after in<br>
>> Compile::Compile, the stack trace is below. It looks like there are residual freed nodes being operated<br>
>> on? Maybe we are failing to unregister a temporary node from the node list? Maybe related to clone_map<br>
>> which returns SafePointNode?<br>
>><br>
>> ==3146540==ERROR: AddressSanitizer: use-after-poison on address 0x62d00996b370 at pc 0x7f9f93048335 bp<br>
>> 0x7f9ed29fae20 sp 0x7f9ed29fae18<br>
>> READ of size 4 at 0x62d00996b370 thread T13<br>
>> #0 0x7f9f93048334 in Unique_Node_List::remove_useless_nodes(VectorSet&)<br>
>> src/hotspot/share/opto/node.cpp:2967<br>
>> #1 0x7f9f932124b3 in PhaseRemoveUseless::PhaseRemoveUseless(PhaseGVN*, Unique_Node_List*,<br>
>> Phase::PhaseNumber) src/hotspot/share/opto/phaseX.cpp:423<br>
>> #2 0x7f9f91621beb in Compile::Compile(ciEnv*, ciMethod*, int, Options, DirectiveSet*)<br>
>> src/hotspot/share/opto/compile.cpp:797<br>
>> #3 0x7f9f912e37fe in C2Compiler::compile_method(ciEnv*, ciMethod*, int, bool, DirectiveSet*)<br>
>> src/hotspot/share/opto/c2compiler.cpp:113<br>
>> #4 0x7f9f91638e07 in CompileBroker::invoke_compiler_on_method(CompileTask*)<br>
>> src/hotspot/share/compiler/compileBroker.cpp:2237<br>
>> #5 0x7f9f9163bfd7 in CompileBroker::compiler_thread_loop()<br>
>> src/hotspot/share/compiler/compileBroker.cpp:1916<br>
>> #6 0x7f9f921e3eec in JavaThread::thread_main_inner() src/hotspot/share/runtime/javaThread.cpp:710<br>
>> #7 0x7f9f921e434f in JavaThread::thread_main_inner() src/hotspot/share/runtime/javaThread.cpp:689<br>
>> #8 0x7f9f921e434f in JavaThread::run() src/hotspot/share/runtime/javaThread.cpp:695<br>
>> #9 0x7f9f93aa3f55 in Thread::call_run() src/hotspot/share/runtime/thread.cpp:224<br>
>> #10 0x7f9f9310144f in thread_native_entry src/hotspot/os/linux/os_linux.cpp:737<br>
>> #11 0x7f9f960a7fd3 in start_thread nptl/pthread_create.c:442<br>
>> #12 0x7f9f9612866b in clone3 ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81<br>
>><br>
>> 0x62d00996b370 is located 20336 bytes inside of 32744-byte region [0x62d009966400,0x62d00996e3e8)<br>
>> allocated by thread T13 here:<br>
>> #0 0x7f9f962b89cf in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69<br>
>> #1 0x7f9f930e915a in os::malloc(unsigned long, MEMFLAGS, NativeCallStack const&)<br>
>> src/hotspot/share/runtime/os.cpp:672<br>
>> #2 0x7f9f90c8a08a in Chunk::operator new(unsigned long, AllocFailStrategy::AllocFailEnum, unsigned<br>
>> long) src/hotspot/share/memory/arena.cpp:190<br>
>> #3 0x7f9f90c8a08a in Arena::grow(unsigned long, AllocFailStrategy::AllocFailEnum)<br>
>> src/hotspot/share/memory/arena.cpp:325<br>
>> #4 0x7f9f932109f5 in Arena::internal_amalloc(unsigned long, AllocFailStrategy::AllocFailEnum)<br>
>> src/hotspot/share/memory/arena.hpp:113<br>
>> #5 0x7f9f932109f5 in Arena::Amalloc(unsigned long, AllocFailStrategy::AllocFailEnum)<br>
>> src/hotspot/share/memory/arena.hpp:133<br>
>> #6 0x7f9f932109f5 in NodeHash::NodeHash(Arena*, unsigned int) src/hotspot/share/opto/phaseX.cpp:68<br>
>> #7 0x7f9f932293c7 in PhaseValues::PhaseValues(Arena*, unsigned int) src/hotspot/share/opto/phaseX.cpp:697<br>
>> #8 0x7f9f9161f678 in PhaseGVN::PhaseGVN(Arena*, unsigned int) src/hotspot/share/opto/phaseX.hpp:415<br>
>> #9 0x7f9f9161f678 in Compile::Compile(ciEnv*, ciMethod*, int, Options, DirectiveSet*)<br>
>> src/hotspot/share/opto/compile.cpp:714<br>
>> #10 0x7f9f912e37fe in C2Compiler::compile_method(ciEnv*, ciMethod*, int, bool, DirectiveSet*)<br>
>> src/hotspot/share/opto/c2compiler.cpp:113<br>
>> #11 0x7f9f91638e07 in CompileBroker::invoke_compiler_on_method(CompileTask*)<br>
>> src/hotspot/share/compiler/compileBroker.cpp:2237<br>
>> #12 0x7f9f9163bfd7 in CompileBroker::compiler_thread_loop()<br>
>> src/hotspot/share/compiler/compileBroker.cpp:1916<br>
>> #13 0x7f9f921e3eec in JavaThread::thread_main_inner() src/hotspot/share/runtime/javaThread.cpp:710<br>
>> #14 0x7f9f921e434f in JavaThread::thread_main_inner() src/hotspot/share/runtime/javaThread.cpp:689<br>
>> #15 0x7f9f921e434f in JavaThread::run() src/hotspot/share/runtime/javaThread.cpp:695<br>
>> #16 0x7f9f93aa3f55 in Thread::call_run() src/hotspot/share/runtime/thread.cpp:224<br>
>> #17 0x7f9f9310144f in thread_native_entry src/hotspot/os/linux/os_linux.cpp:737<br>
>> #18 0x7f9f960a7fd3 in start_thread nptl/pthread_create.c:442<br>
>><br>
>> Thread T13 created by T1 here:<br>
>> #0 0x7f9f96249726 in __interceptor_pthread_create<br>
>> ../../../../src/libsanitizer/asan/asan_interceptors.cpp:207<br>
>> #1 0x7f9f93102d88 in os::create_thread(Thread*, os::ThreadType, unsigned long)<br>
>> src/hotspot/os/linux/os_linux.cpp:888<br>
>> #2 0x7f9f91693d93 in CompilerThread::CompilerThread(CompileQueue*, CompilerCounters*)<br>
>> src/hotspot/share/compiler/compilerThread.cpp:34<br>
>> #3 0x7f9f91625c7c in CompileBroker::make_thread(CompileBroker::ThreadType, _jobject*, CompileQueue*,<br>
>> AbstractCompiler*, JavaThread*) src/hotspot/share/compiler/compileBroker.cpp:842<br>
>> #4 0x7f9f91628f71 in CompileBroker::init_compiler_threads()<br>
>> src/hotspot/share/compiler/compileBroker.cpp:943<br>
>> #5 0x7f9f9162a464 in CompileBroker::compilation_init_phase1(JavaThread*)<br>
>> src/hotspot/share/compiler/compileBroker.cpp:654<br>
>> #6 0x7f9f93adc3a4 in Threads::create_vm(JavaVMInitArgs*, bool*) src/hotspot/share/runtime/threads.cpp:701<br>
>> #7 0x7f9f92465b51 in JNI_CreateJavaVM_inner src/hotspot/share/prims/jni.cpp:3588<br>
>> #8 0x7f9f92465b51 in JNI_CreateJavaVM src/hotspot/share/prims/jni.cpp:3674<br>
>> #9 0x7f9f968d2e25 in InitializeJVM src/java.base/share/native/libjli/java.c:1459<br>
>> #10 0x7f9f968d2e25 in JavaMain src/java.base/share/native/libjli/java.c:413<br>
>> #11 0x7f9f968db708 in ThreadJavaMain src/java.base/unix/native/libjli/java_md.c:650<br>
>> #12 0x7f9f960a7fd3 in start_thread nptl/pthread_create.c:442<br>
>><br>
>> Thread T1 created by T0 here:<br>
>> #0 0x7f9f96249726 in __interceptor_pthread_create<br>
>> ../../../../src/libsanitizer/asan/asan_interceptors.cpp:207<br>
>> #1 0x7f9f968dd3a1 in CallJavaMainInNewThread src/java.base/unix/native/libjli/java_md.c:691<br>
>> #2 0x7f9f968d822d in ContinueInNewThread src/java.base/share/native/libjli/java.c:2280<br>
>> #3 0x7f9f968d96ae in JLI_Launch src/java.base/share/native/libjli/java.c:340<br>
>> #4 0x5594a81c337c in main src/java.base/share/native/launcher/main.c:166<br>
>> #5 0x7f9f96046189 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58<br>
>><br>
>> -- <br>
>><br>
>> Google Logo <br>
>> Justin King<br>
>> Software Engineer<br>
>> <a href="mailto:jcking@google.com" target="_blank">jcking@google.com</a> <mailto:<a href="mailto:jcking@google.com" target="_blank">jcking@google.com</a>><br>
>><br>
>><br>
>> <br>
>><br>
> <br>
> <br>
> -- <br>
> <br>
> Google Logo <br>
> Justin King<br>
> Software Engineer<br>
> <a href="mailto:jcking@google.com" target="_blank">jcking@google.com</a> <mailto:<a href="mailto:jcking@google.com" target="_blank">jcking@google.com</a>><br>
> <br>
> <br>
> <br>
> <br>
> <br>
> -- <br>
> <br>
> Google Logo <br>
> Justin King<br>
> Software Engineer<br>
> <a href="mailto:jcking@google.com" target="_blank">jcking@google.com</a> <mailto:<a href="mailto:jcking@google.com" target="_blank">jcking@google.com</a>><br>
> <br>
> <br>
> <br>
> <br>
> <br>
> -- <br>
> <br>
> Google Logo <br>
> Justin King<br>
> Software Engineer<br>
> <a href="mailto:jcking@google.com" target="_blank">jcking@google.com</a> <mailto:<a href="mailto:jcking@google.com" target="_blank">jcking@google.com</a>><br>
> <br>
> <br>
> <br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><span><br><table border="0" cellspacing="0" cellpadding="0" style="margin:0px;padding:20px 0px 0px;font-family:"Times New Roman""><tbody style="margin:0px;padding:0px"><tr style="margin:0px;padding:0px"><td valign="top" style="padding:0px 20px 0px 0px;vertical-align:top;border-right:1px solid rgb(213,213,213)"><img src="https://www.gstatic.com/images/branding/product/1x/googleg_64dp.png" alt="Google Logo" width="72" style="margin: 0px; padding: 0px; display: block; height: auto;"></td><td style="padding:0px 0px 0px 20px"><table border="0" cellspacing="0" cellpadding="0" style="margin:0px;padding:0px"><tbody style="margin:0px;padding:0px"><tr style="margin:0px;padding:0px"><td colspan="2" style="padding:1px 0px 5px;font-family:Arial,Helvetica,Verdana,sans-serif;font-size:13px;line-height:13px;color:rgb(56,58,53);font-weight:700">Justin King</td></tr><tr style="margin:0px;padding:0px"><td colspan="2" style="padding:0px 0px 5px;font-family:Arial,Helvetica,Verdana,sans-serif;font-size:11px;line-height:13px;color:rgb(56,58,53)">Software Engineer</td></tr><tr style="margin:0px;padding:0px"><td colspan="2" style="padding:0px 0px 5px;font-family:Arial,Helvetica,Verdana,sans-serif;font-size:11px;line-height:13px;color:rgb(56,58,53)"><a href="mailto:jcking@google.com" target="_blank">jcking@google.com</a></td></tr></tbody></table></td></tr></tbody></table><table style="border:none;border-collapse:collapse"><colgroup><col width="85"><col width="539"></colgroup><tbody><tr style="height:0pt"><td style="border-width:1pt;border-style:solid;border-color:rgb(255,255,255);vertical-align:middle;padding:4.32pt"></td><td style="border-width:1pt;border-style:solid;border-color:rgb(255,255,255);vertical-align:middle;padding:4.32pt"><br></td></tr></tbody></table></span></div></div>