Fwd: Java stack unwinder for Linux + gdb7.10+

Tue May 31 13:13:55 UTC 2016

Hi Andrew,

On 2016-05-31 15:09, Andrew Dinn wrote:
> I probably should have sent this here initially rather than to jdk9-dev.
> Anyway, I hope subscribers to this list will find this interesting.

I saw it, tried it and it worked after replacing a bunch of
-        except Exception, arg:
+        except Exception as arg:

and adjusting the sources to account for jdk 9 changes which I don't yet 
have in my project workspace.

I think it might make more sense to put this in the hotspot repo, 
similar to the serviceability agent, since it's functionality is tied to 
a particular revision of the sources.

/Mikael

>
> regards,
>
>
> Andrew Dinn
> -----------
>
> -------- Forwarded Message --------
> Subject: Java stack unwinder for Linux + gdb7.10+
> Date: Tue, 31 May 2016 12:02:50 +0100
> From: Andrew Dinn <adinn at redhat.com>
> To: jdk9-dev at openjdk.java.net <jdk9-dev at openjdk.java.net>
> CC: code-tools-dev at openjdk.java.net
>
> What's this all about?
> ----------------------
>
> Recently, I have been working on implementing support for unwinding Java
> stacks in gdb using the new python stack unwinder API provided from gdb
> 7.10.1 onwards. I have managed to provide a pure python solution i.e.
> one that that needs no modifications to the VM itself. So, if you use
> gdb to debug operation of the JVM then this implementation will allow
> you to use the bt command as $DEITY (and, indeed, the gdb team) intended.
>
> Attached at the bottom of this note is a before and after showing how
> backtraces look without/with this package installed.
>
> Note that the unwinder works when using gdb to debug either a live
> process or a core file.
>
> Also, note that at present gdb only uses python unwinders to display
> frame info underneath the bt command. I am working with the gdb devs to
> see if they can extend the use of the unwinder to other situations e.g.
> to display the current frame when single-stepping.
>
> Can I try it out?
> -----------------
>
> Yes, please do!
>
> Indeed, I am posting this to jdk9-dev because I want as many people as
> possible to try out the unwinder and provide feedback on how it works.
> So, please use it and respond -- positive or negative I would love to
> hear what you think. I'm happy to hear offline or on this list. You
> choose whether or not you think others on this list might want to hear
> your comments.
>
> How do I obtain/use it?
> -----------------------
>
> The unwinder code is available from the following hg repo
>
>    http://icedtea.classpath.org/people/adinn/unwinder
>
> See the README in that repo for details of how to install and use it
> (gdb will auto-load it once you have configured things correctly)
>
> To summarize some important details:
>
> The extension requires gdb 7.10+ and you there are two unwinder
> implementations, one for jdk8 and one for jdk9.
>
>    The jdk8 unwinder should work for any jdk8 built from the jdk8u repo.
> In particular, it will work for /product releases/ available with your
> Linux distro so long as the distro supports installation of minimal
> debuginfo for those releases. For example, if you are using RHEL or
> fedora then you can simply debuginfo-install the debug info
> corresponding to your jdk8 java package and then install the uinwinder
> under /etc/alternatives/java_sdk_1.8.0. If you want to use it with your
> own jdk8 build then you need to build a debug release which includes a
> libjvm.debuginfo product (well, I guess you could always hack the
> product make process so it generates libjvm.debuginfo ).
>
>    The jdk9 unwinder will only work for your own debug builds and even
> then only for builds which include the  b119 tag. It doesn't work for
> product builds because  ... well, there are no product builds available
> from any distros just yet which, a fortiori, means there are no product
> builds where you can install minimal debug info. Unfortunately, that
> includes the currently available early access program builds which don't
> ship libjvm.debuginfo.
>
> Ok, so why also the b119 restriction? That's because between b118 and
> b119 the stack and code heap layouts changed and this required the
> unwinder code to change accordingly. An older version of the unwinder
> did work with b118 and earlier builds but tempus fugit, hey ho ....
>
>
> How does it do that?
> --------------------
>
> For those who are curious the python code is your friend (apologies for
> my python noobness -- I mean, eeeuurrrggghhh, python :-). Basically, the
> unwinder works without change to the JVM because it understands a few
> details of how:
>
>    the code cache is laid out
>    code blobs/methods are organised
>    bci-to-line number maps are organized
>    code address-to-bci maps are organized
>
> So, essentially, it uses the gdb inferior access functions to poke
> around in JVM memory, allowing it to work out what the state of the
> stack is and what method, file, bci and line number a specific
> instruction address corresponds to. That includes using gdb to resolve a
> few symbols in order to identify memory layouts and a few important
> global addresses.
>
> This dependency has been kept to a bare minimum so that the unwinder
> will be as resilient as possible to changes in the JVM.
>
> Can we have this in OpenJDK?
> ----------------------------
>
> The current code is LGPL licensed. I'd be very happy to contribute it to
> OpenJDK as part of the code tools set for JDK developers to use and Red
> Hat would be very happy to maintain it once included. Ideally, Red Hat
> would like to ship the tool as part of OpenJDK product releases for use
> in any distro where minimal debug info is available with the product
> release. We see great benefit in the opportunity it provides for support
> staff to print stack traces from core files (clearly, shipping with
> OpenJDK would help to ensure that changes in the JVM and required
> changes in the python code were able to be kept in sync).
>
> I have copied this note to the code-tools-dev list on the assumption
> that it is the correct place to open up discussion of whether it
> could/should be provided as part of OpenJDK. If not can someone please
> let me know where else I should start that discussion.
>
> regards,
>
>
> Andrew Dinn
> -----------
> Senior Principal Software Engineer
> Red Hat UK Ltd
> Registered in England and Wales under Company Registration No. 03798903
> Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander
>
> ----- 8< -------- 8< -------- 8< -------- 8< -------- 8< -------- 8< ---
>
> Normal gdb backtrace for java thread fails ot unwind through Java frames
>
> (gdb) thread 16
> [Switching to thread 16 (Thread 0x7fffa1fff700 (LWP 4930))]
> #0 0x00007fffe1113484 in ?? ()
> (gdb) bt 3
> #0  0x00007fffe1113484 in ?? ()
> #1  0x00007fffd3c00c18 in ?? ()
> #2  0x0000000000000000 in ?? ()
>
> Backtrace with the unwinder installed
>
> (gdb) thread 2
> [Switching to thread 2 (Thread 0x7ffff7fcc700 (LWP 4906))]
> #0  0x00007ffff79b6540 in pthread_cond_wait@@GLIBC_2.3.2 ()
>     from /lib64/libpthread.so.0
> (gdb) bt 10
> #0  0x00007ffff79b6540 in pthread_cond_wait@@GLIBC_2.3.2 ()
>      at /lib64/libpthread.so.0
> #1  0x00007ffff64ae82d in os::PlatformEvent::park() (this=0x7ffff0019700)
>      at
> /home/adinn/redhat/openjdk/jdk9/hs/hotspot/src/os/linux/vm/os_linux.cpp:5582
> #2  0x00007ffff6486f59 in ObjectMonitor::wait(long, bool, Thread*)
> (this=0x7fff90008b80, millis=0, interruptible=true,
> __the_thread__=0x7ffff0018800)
>      at
> /home/adinn/redhat/openjdk/jdk9/hs/hotspot/src/share/vm/runtime/objectMonitor.cpp:1490
> #3  0x00007ffff66518cb in ObjectSynchronizer::wait(Handle, long,
> Thread*) (obj=..., millis=0, __the_thread__=0x7ffff0018800)
>      at
> /home/adinn/redhat/openjdk/jdk9/hs/hotspot/src/share/vm/runtime/synchronizer.cpp:496
> #4  0x00007ffff618fa54 in JVM_MonitorWait(JNIEnv*, jobject, jlong)
> (env=0x7ffff0018a28, handle=0x7ffff7fcb738, ms=0)
>      at
> /home/adinn/redhat/openjdk/jdk9/hs/hotspot/src/share/vm/prims/jvm.cpp:614
> #5  0x00007fffd8924f7d in [interpreted: bc = 0]
> java.lang.Object.wait(long) ()
>      at java/lang/Object.java
> #6  0x00007fffd89034a3 in [interpreted: bc = 38]
> java.lang.Thread.join(long) () at java/lang/Thread.java:1352
> #7  0x00007fffd89034a3 in [interpreted: bc = 2] java.lang.Thread.join() ()
>      at java/lang/Thread.java:1426
> #8  0x00007fffd89034a3 in [interpreted: bc = 35]
> HelloV.main(java.lang.String) () at HelloV.java:80
> #9  0x00007fffd88f89f1 in StubRoutines (1) ()
> #10 0x00007ffff6109bb3 in JavaCalls::call_helper(JavaValue*,
> methodHandle const&, JavaCallArguments*, Thread*)
> (result=0x7ffff7fcbd30, method=..., args=0x7ffff7fcbc10,
> __the_thread__=0x7ffff0018800)
>      at
> /home/adinn/redhat/openjdk/jdk9/hs/hotspot/src/share/vm/runtime/javaCalls.cpp:409
> (gdb) thread 26
> [Switching to thread 26 (Thread 0x7fffa1fff700 (LWP 4930))]
> #0  0x00007fffe03cb664 in ?? ()
> (gdb) bt 5
> #0  0x00007fffe03cb664 in [compiled offset = 0x3a4] HelloV$1.run() ()
>      at HelloV.java:63
> #1  0x00007fffd88f89f1 in StubRoutines (1) ()
> #2  0x00007ffff6109bb3 in JavaCalls::call_helper(JavaValue*,
> methodHandle const&, JavaCallArguments*, Thread*)
> (result=0x7fffa1ffecc0, method=..., args=0x7fffa1ffec00,
> __the_thread__=0x7ffff0471800)
>      at
> /home/adinn/redhat/openjdk/jdk9/hs/hotspot/src/share/vm/runtime/javaCalls.cpp:409
> #3  0x00007ffff64ad409 in os::os_exception_wrapper(void (*)(JavaValue*,
> methodHandle const&, JavaCallArguments*, Thread*), JavaValue*,
> methodHandle const&, JavaCallArguments*, Thread*) (f=0x7ffff610951a
> <JavaCalls::call_helper(JavaValue*, methodHandle const&,
> JavaCallArguments*, Thread*)>, value=0x7fffa1ffecc0, method=...,
> args=0x7fffa1ffec00, thread=0x7ffff0471800)
>      at
> /home/adinn/redhat/openjdk/jdk9/hs/hotspot/src/os/linux/vm/os_linux.cpp:5128
> #4  0x00007ffff6109502 in JavaCalls::call(JavaValue*, methodHandle
> const&, JavaCallArguments*, Thread*) (result=0x7fffa1ffecc0, method=...,
> args=0x7fffa1ffec00, __the_thread__=0x7ffff0471800)
>      at
> /home/adinn/redhat/openjdk/jdk9/hs/hotspot/src/share/vm/runtime/javaCalls.cpp:298
> #5  0x00007ffff610877b in JavaCalls::call_virtual(JavaValue*,
> KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)
> (result=0x7fffa1ffecc0, spec_klass=..., name=0x7ffff4031338,
> signature=0x7ffff4033220, args=0x7fffa1ffec00,
> __the_thread__=0x7ffff0471800)
>      at
> /home/adinn/redhat/openjdk/jdk9/hs/hotspot/src/share/vm/runtime/javaCalls.cpp:193
> (gdb)
>
>
>
>