RFR: 8178287: AsyncGetCallTrace fails to traverse valid Java stacks [v3]

[Ludovic Henry]

On Sun, 11 Jul 2021 22:21:31 GMT, Andrei Pangin <apangin at openjdk.org> wrote:

>> Ludovic Henry has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Fix comments
>
> Hi Ludovic,
> 
> Thank you for working on this long-standing bug.
> I like the idea of the proposed solution, but unfortunately it cannot be applied as is. Since the stack walking code runs inside a signal handler, it is very limited in things it can do. In particular, it must not allocate, acquire locks, etc. In your implementation, FrameParser does allocate though.
> 
> The issue is not just theoretical: when I ran JDK with this patch with async-profiler, I immediately got the following deadlock:
> 
> 
> (gdb) bt
> #0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
> #1  0x00007fa2363ca025 in __GI___pthread_mutex_lock (mutex=0x7fa235da5440 <tc_mutex>)
>     at ../nptl/pthread_mutex_lock.c:80
> #2  0x00007fa235696cb6 in ThreadCritical::ThreadCritical() () from /usr/java/jdk-18/lib/server/libjvm.so
> #3  0x00007fa234b6fe53 in Chunk::next_chop() () from /usr/java/jdk-18/lib/server/libjvm.so
> #4  0x00007fa234e88523 in frame::safe_for_sender(JavaThread*) () from /usr/java/jdk-18/lib/server/libjvm.so
> #5  0x00007fa234e838f2 in vframeStreamForte::forte_next() () from /usr/java/jdk-18/lib/server/libjvm.so
> #6  0x00007fa2349fbb9b in forte_fill_call_trace_given_top(JavaThread*, ASGCT_CallTrace*, int, frame) [clone .isra.20]
>     () from /usr/java/jdk-18/lib/server/libjvm.so
> #7  0x00007fa234e8426e in AsyncGetCallTrace () from /usr/java/jdk-18/lib/server/libjvm.so
> #8  0x00007fa228519312 in Profiler::getJavaTraceAsync(void*, ASGCT_CallFrame*, int) ()
>    from /mnt/c/Users/Andrei/java/async-profiler/build/libasyncProfiler.so
> #9  0x00007fa228519c72 in Profiler::recordSample(void*, unsigned long long, int, Event*) ()
>    from /mnt/c/Users/Andrei/java/async-profiler/build/libasyncProfiler.so
> #10 0x00007fa2285164f8 in WallClock::signalHandler(int, siginfo_t*, void*) ()
>    from /mnt/c/Users/Andrei/java/async-profiler/build/libasyncProfiler.so
> #11 <signal handler called>
> #12 __pthread_mutex_unlock_usercnt (decr=1, mutex=0x7fa235da5440 <tc_mutex>) at pthread_mutex_unlock.c:41
> #13 __GI___pthread_mutex_unlock (mutex=0x7fa235da5440 <tc_mutex>) at pthread_mutex_unlock.c:356
> #14 0x00007fa235696d3b in ThreadCritical::~ThreadCritical() () from /usr/java/jdk-18/lib/server/libjvm.so
> #15 0x00007fa234b6fe71 in Chunk::next_chop() () from /usr/java/jdk-18/lib/server/libjvm.so
> #16 0x00007fa234d1ca62 in ClassFileParser::parse_method(ClassFileStream const*, bool, ConstantPool const*, AccessFlags*, JavaThread*) () from /usr/java/jdk-18/lib/server/libjvm.so
> #17 0x00007fa234d1e338 in ClassFileParser::parse_methods(ClassFileStream const*, bool, AccessFlags*, bool*, bool*, JavaThread*) () from /usr/java/jdk-18/lib/server/libjvm.so
> #18 0x00007fa234d22459 in ClassFileParser::parse_stream(ClassFileStream const*, JavaThread*) ()
>    from /usr/java/jdk-18/lib/server/libjvm.so
> #19 0x00007fa234d2291c in ClassFileParser::ClassFileParser(ClassFileStream*, Symbol*, ClassLoaderData*, ClassLoadInfo const*, ClassFileParser::Publicity, JavaThread*) () from /usr/java/jdk-18/lib/server/libjvm.so
> #20 0x00007fa2351febb6 in KlassFactory::create_from_stream(ClassFileStream*, Symbol*, ClassLoaderData*, ClassLoadInfo const&, JavaThread*) ()
>    from /usr/java/jdk-18/lib/server/libjvm.so
> #21 0x00007fa235645b40 in SystemDictionary::resolve_class_from_stream(ClassFileStream*, Symbol*, Handle, ClassLoadInfo const&, JavaThread*) ()
>    from /usr/java/jdk-18/lib/server/libjvm.so
> #22 0x00007fa2350bad0a in jvm_define_class_common(char const*, _jobject*, signed char const*, int, _jobject*, char const*, JavaThread*) [clone .constprop.299] ()
>    from /usr/java/jdk-18/lib/server/libjvm.so
> #23 0x00007fa2350bae6d in JVM_DefineClassWithSource () from /usr/java/jdk-18/lib/server/libjvm.so
> #24 0x00007fa236a0ee12 in Java_java_lang_ClassLoader_defineClass1 () from /usr/java/jdk-18/lib/libjava.so

@apangin Thanks for pointing that out! I'm updating it right now and should be pushing an update very soon. I'll also add examples on how it impacts JFR.

-------------

PR: https://git.openjdk.java.net/jdk/pull/4436