Java stack unwinder for Linux + gdb7.10+

Tue May 31 13:20:00 UTC 2016

Adding this to the hotspot repo sounds like the way to go.

I would suggest, though, that the scripts come with a comprehensive set of tests so that we find out when changes break the scripts. This would avoid the sad state that the serviceability agent is currently in.

/Staffan

> On 31 maj 2016, at 15:13, Mikael Gerdin <mikael.gerdin at oracle.com> wrote:
> 
> Hi Andrew,
> 
> On 2016-05-31 15:09, Andrew Dinn wrote:
>> I probably should have sent this here initially rather than to jdk9-dev.
>> Anyway, I hope subscribers to this list will find this interesting.
> 
> I saw it, tried it and it worked after replacing a bunch of
> -        except Exception, arg:
> +        except Exception as arg:
> 
> and adjusting the sources to account for jdk 9 changes which I don't yet have in my project workspace.
> 
> I think it might make more sense to put this in the hotspot repo, similar to the serviceability agent, since it's functionality is tied to a particular revision of the sources.
> 
> /Mikael
> 
>> 
>> regards,
>> 
>> 
>> Andrew Dinn
>> -----------
>> 
>> -------- Forwarded Message --------
>> Subject: Java stack unwinder for Linux + gdb7.10+
>> Date: Tue, 31 May 2016 12:02:50 +0100
>> From: Andrew Dinn <adinn at redhat.com>
>> To: jdk9-dev at openjdk.java.net <jdk9-dev at openjdk.java.net>
>> CC: code-tools-dev at openjdk.java.net
>> 
>> What's this all about?
>> ----------------------
>> 
>> Recently, I have been working on implementing support for unwinding Java
>> stacks in gdb using the new python stack unwinder API provided from gdb
>> 7.10.1 onwards. I have managed to provide a pure python solution i.e.
>> one that that needs no modifications to the VM itself. So, if you use
>> gdb to debug operation of the JVM then this implementation will allow
>> you to use the bt command as $DEITY (and, indeed, the gdb team) intended.
>> 
>> Attached at the bottom of this note is a before and after showing how
>> backtraces look without/with this package installed.
>> 
>> Note that the unwinder works when using gdb to debug either a live
>> process or a core file.
>> 
>> Also, note that at present gdb only uses python unwinders to display
>> frame info underneath the bt command. I am working with the gdb devs to
>> see if they can extend the use of the unwinder to other situations e.g.
>> to display the current frame when single-stepping.
>> 
>> Can I try it out?
>> -----------------
>> 
>> Yes, please do!
>> 
>> Indeed, I am posting this to jdk9-dev because I want as many people as
>> possible to try out the unwinder and provide feedback on how it works.
>> So, please use it and respond -- positive or negative I would love to
>> hear what you think. I'm happy to hear offline or on this list. You
>> choose whether or not you think others on this list might want to hear
>> your comments.
>> 
>> How do I obtain/use it?
>> -----------------------
>> 
>> The unwinder code is available from the following hg repo
>> 
>>   http://icedtea.classpath.org/people/adinn/unwinder
>> 
>> See the README in that repo for details of how to install and use it
>> (gdb will auto-load it once you have configured things correctly)
>> 
>> To summarize some important details:
>> 
>> The extension requires gdb 7.10+ and you there are two unwinder
>> implementations, one for jdk8 and one for jdk9.
>> 
>>   The jdk8 unwinder should work for any jdk8 built from the jdk8u repo.
>> In particular, it will work for /product releases/ available with your
>> Linux distro so long as the distro supports installation of minimal
>> debuginfo for those releases. For example, if you are using RHEL or
>> fedora then you can simply debuginfo-install the debug info
>> corresponding to your jdk8 java package and then install the uinwinder
>> under /etc/alternatives/java_sdk_1.8.0. If you want to use it with your
>> own jdk8 build then you need to build a debug release which includes a
>> libjvm.debuginfo product (well, I guess you could always hack the
>> product make process so it generates libjvm.debuginfo ).
>> 
>>   The jdk9 unwinder will only work for your own debug builds and even
>> then only for builds which include the  b119 tag. It doesn't work for
>> product builds because  ... well, there are no product builds available
>> from any distros just yet which, a fortiori, means there are no product
>> builds where you can install minimal debug info. Unfortunately, that
>> includes the currently available early access program builds which don't
>> ship libjvm.debuginfo.
>> 
>> Ok, so why also the b119 restriction? That's because between b118 and
>> b119 the stack and code heap layouts changed and this required the
>> unwinder code to change accordingly. An older version of the unwinder
>> did work with b118 and earlier builds but tempus fugit, hey ho ....
>> 
>> 
>> How does it do that?
>> --------------------
>> 
>> For those who are curious the python code is your friend (apologies for
>> my python noobness -- I mean, eeeuurrrggghhh, python :-). Basically, the
>> unwinder works without change to the JVM because it understands a few
>> details of how:
>> 
>>   the code cache is laid out
>>   code blobs/methods are organised
>>   bci-to-line number maps are organized
>>   code address-to-bci maps are organized
>> 
>> So, essentially, it uses the gdb inferior access functions to poke
>> around in JVM memory, allowing it to work out what the state of the
>> stack is and what method, file, bci and line number a specific
>> instruction address corresponds to. That includes using gdb to resolve a
>> few symbols in order to identify memory layouts and a few important
>> global addresses.
>> 
>> This dependency has been kept to a bare minimum so that the unwinder
>> will be as resilient as possible to changes in the JVM.
>> 
>> Can we have this in OpenJDK?
>> ----------------------------
>> 
>> The current code is LGPL licensed. I'd be very happy to contribute it to
>> OpenJDK as part of the code tools set for JDK developers to use and Red
>> Hat would be very happy to maintain it once included. Ideally, Red Hat
>> would like to ship the tool as part of OpenJDK product releases for use
>> in any distro where minimal debug info is available with the product
>> release. We see great benefit in the opportunity it provides for support
>> staff to print stack traces from core files (clearly, shipping with
>> OpenJDK would help to ensure that changes in the JVM and required
>> changes in the python code were able to be kept in sync).
>> 
>> I have copied this note to the code-tools-dev list on the assumption
>> that it is the correct place to open up discussion of whether it
>> could/should be provided as part of OpenJDK. If not can someone please
>> let me know where else I should start that discussion.
>> 
>> regards,
>> 
>> 
>> Andrew Dinn
>> -----------
>> Senior Principal Software Engineer
>> Red Hat UK Ltd
>> Registered in England and Wales under Company Registration No. 03798903
>> Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander
>> 
>> ----- 8< -------- 8< -------- 8< -------- 8< -------- 8< -------- 8< ---
>> 
>> Normal gdb backtrace for java thread fails ot unwind through Java frames
>> 
>> (gdb) thread 16
>> [Switching to thread 16 (Thread 0x7fffa1fff700 (LWP 4930))]
>> #0 0x00007fffe1113484 in ?? ()
>> (gdb) bt 3
>> #0  0x00007fffe1113484 in ?? ()
>> #1  0x00007fffd3c00c18 in ?? ()
>> #2  0x0000000000000000 in ?? ()
>> 
>> Backtrace with the unwinder installed
>> 
>> (gdb) thread 2
>> [Switching to thread 2 (Thread 0x7ffff7fcc700 (LWP 4906))]
>> #0  0x00007ffff79b6540 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>    from /lib64/libpthread.so.0
>> (gdb) bt 10
>> #0  0x00007ffff79b6540 in pthread_cond_wait@@GLIBC_2.3.2 ()
>>     at /lib64/libpthread.so.0
>> #1  0x00007ffff64ae82d in os::PlatformEvent::park() (this=0x7ffff0019700)
>>     at
>> /home/adinn/redhat/openjdk/jdk9/hs/hotspot/src/os/linux/vm/os_linux.cpp:5582
>> #2  0x00007ffff6486f59 in ObjectMonitor::wait(long, bool, Thread*)
>> (this=0x7fff90008b80, millis=0, interruptible=true,
>> __the_thread__=0x7ffff0018800)
>>     at
>> /home/adinn/redhat/openjdk/jdk9/hs/hotspot/src/share/vm/runtime/objectMonitor.cpp:1490
>> #3  0x00007ffff66518cb in ObjectSynchronizer::wait(Handle, long,
>> Thread*) (obj=..., millis=0, __the_thread__=0x7ffff0018800)
>>     at
>> /home/adinn/redhat/openjdk/jdk9/hs/hotspot/src/share/vm/runtime/synchronizer.cpp:496
>> #4  0x00007ffff618fa54 in JVM_MonitorWait(JNIEnv*, jobject, jlong)
>> (env=0x7ffff0018a28, handle=0x7ffff7fcb738, ms=0)
>>     at
>> /home/adinn/redhat/openjdk/jdk9/hs/hotspot/src/share/vm/prims/jvm.cpp:614
>> #5  0x00007fffd8924f7d in [interpreted: bc = 0]
>> java.lang.Object.wait(long) ()
>>     at java/lang/Object.java
>> #6  0x00007fffd89034a3 in [interpreted: bc = 38]
>> java.lang.Thread.join(long) () at java/lang/Thread.java:1352
>> #7  0x00007fffd89034a3 in [interpreted: bc = 2] java.lang.Thread.join() ()
>>     at java/lang/Thread.java:1426
>> #8  0x00007fffd89034a3 in [interpreted: bc = 35]
>> HelloV.main(java.lang.String) () at HelloV.java:80
>> #9  0x00007fffd88f89f1 in StubRoutines (1) ()
>> #10 0x00007ffff6109bb3 in JavaCalls::call_helper(JavaValue*,
>> methodHandle const&, JavaCallArguments*, Thread*)
>> (result=0x7ffff7fcbd30, method=..., args=0x7ffff7fcbc10,
>> __the_thread__=0x7ffff0018800)
>>     at
>> /home/adinn/redhat/openjdk/jdk9/hs/hotspot/src/share/vm/runtime/javaCalls.cpp:409
>> (gdb) thread 26
>> [Switching to thread 26 (Thread 0x7fffa1fff700 (LWP 4930))]
>> #0  0x00007fffe03cb664 in ?? ()
>> (gdb) bt 5
>> #0  0x00007fffe03cb664 in [compiled offset = 0x3a4] HelloV$1.run() ()
>>     at HelloV.java:63
>> #1  0x00007fffd88f89f1 in StubRoutines (1) ()
>> #2  0x00007ffff6109bb3 in JavaCalls::call_helper(JavaValue*,
>> methodHandle const&, JavaCallArguments*, Thread*)
>> (result=0x7fffa1ffecc0, method=..., args=0x7fffa1ffec00,
>> __the_thread__=0x7ffff0471800)
>>     at
>> /home/adinn/redhat/openjdk/jdk9/hs/hotspot/src/share/vm/runtime/javaCalls.cpp:409
>> #3  0x00007ffff64ad409 in os::os_exception_wrapper(void (*)(JavaValue*,
>> methodHandle const&, JavaCallArguments*, Thread*), JavaValue*,
>> methodHandle const&, JavaCallArguments*, Thread*) (f=0x7ffff610951a
>> <JavaCalls::call_helper(JavaValue*, methodHandle const&,
>> JavaCallArguments*, Thread*)>, value=0x7fffa1ffecc0, method=...,
>> args=0x7fffa1ffec00, thread=0x7ffff0471800)
>>     at
>> /home/adinn/redhat/openjdk/jdk9/hs/hotspot/src/os/linux/vm/os_linux.cpp:5128
>> #4  0x00007ffff6109502 in JavaCalls::call(JavaValue*, methodHandle
>> const&, JavaCallArguments*, Thread*) (result=0x7fffa1ffecc0, method=...,
>> args=0x7fffa1ffec00, __the_thread__=0x7ffff0471800)
>>     at
>> /home/adinn/redhat/openjdk/jdk9/hs/hotspot/src/share/vm/runtime/javaCalls.cpp:298
>> #5  0x00007ffff610877b in JavaCalls::call_virtual(JavaValue*,
>> KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)
>> (result=0x7fffa1ffecc0, spec_klass=..., name=0x7ffff4031338,
>> signature=0x7ffff4033220, args=0x7fffa1ffec00,
>> __the_thread__=0x7ffff0471800)
>>     at
>> /home/adinn/redhat/openjdk/jdk9/hs/hotspot/src/share/vm/runtime/javaCalls.cpp:193
>> (gdb)
>> 
>> 
>> 
>> 