Fwd: Java stack unwinder for Linux + gdb7.10+

Tue May 31 13:09:47 UTC 2016

I probably should have sent this here initially rather than to jdk9-dev.
Anyway, I hope subscribers to this list will find this interesting.

regards,

Andrew Dinn
-----------

-------- Forwarded Message --------
Subject: Java stack unwinder for Linux + gdb7.10+
Date: Tue, 31 May 2016 12:02:50 +0100
From: Andrew Dinn <adinn at redhat.com>
To: jdk9-dev at openjdk.java.net <jdk9-dev at openjdk.java.net>
CC: code-tools-dev at openjdk.java.net

What's this all about?
----------------------

Recently, I have been working on implementing support for unwinding Java
stacks in gdb using the new python stack unwinder API provided from gdb
7.10.1 onwards. I have managed to provide a pure python solution i.e.
one that that needs no modifications to the VM itself. So, if you use
gdb to debug operation of the JVM then this implementation will allow
you to use the bt command as $DEITY (and, indeed, the gdb team) intended.

Attached at the bottom of this note is a before and after showing how
backtraces look without/with this package installed.

Note that the unwinder works when using gdb to debug either a live
process or a core file.

Also, note that at present gdb only uses python unwinders to display
frame info underneath the bt command. I am working with the gdb devs to
see if they can extend the use of the unwinder to other situations e.g.
to display the current frame when single-stepping.

Can I try it out?
-----------------

Yes, please do!

Indeed, I am posting this to jdk9-dev because I want as many people as
possible to try out the unwinder and provide feedback on how it works.
So, please use it and respond -- positive or negative I would love to
hear what you think. I'm happy to hear offline or on this list. You
choose whether or not you think others on this list might want to hear
your comments.

How do I obtain/use it?
-----------------------

The unwinder code is available from the following hg repo

  http://icedtea.classpath.org/people/adinn/unwinder

See the README in that repo for details of how to install and use it
(gdb will auto-load it once you have configured things correctly)

To summarize some important details:

The extension requires gdb 7.10+ and you there are two unwinder
implementations, one for jdk8 and one for jdk9.

  The jdk8 unwinder should work for any jdk8 built from the jdk8u repo.
In particular, it will work for /product releases/ available with your
Linux distro so long as the distro supports installation of minimal
debuginfo for those releases. For example, if you are using RHEL or
fedora then you can simply debuginfo-install the debug info
corresponding to your jdk8 java package and then install the uinwinder
under /etc/alternatives/java_sdk_1.8.0. If you want to use it with your
own jdk8 build then you need to build a debug release which includes a
libjvm.debuginfo product (well, I guess you could always hack the
product make process so it generates libjvm.debuginfo ).

  The jdk9 unwinder will only work for your own debug builds and even
then only for builds which include the  b119 tag. It doesn't work for
product builds because  ... well, there are no product builds available
from any distros just yet which, a fortiori, means there are no product
builds where you can install minimal debug info. Unfortunately, that
includes the currently available early access program builds which don't
ship libjvm.debuginfo.

Ok, so why also the b119 restriction? That's because between b118 and
b119 the stack and code heap layouts changed and this required the
unwinder code to change accordingly. An older version of the unwinder
did work with b118 and earlier builds but tempus fugit, hey ho ....

How does it do that?
--------------------

For those who are curious the python code is your friend (apologies for
my python noobness -- I mean, eeeuurrrggghhh, python :-). Basically, the
unwinder works without change to the JVM because it understands a few
details of how:

  the code cache is laid out
  code blobs/methods are organised
  bci-to-line number maps are organized
  code address-to-bci maps are organized

So, essentially, it uses the gdb inferior access functions to poke
around in JVM memory, allowing it to work out what the state of the
stack is and what method, file, bci and line number a specific
instruction address corresponds to. That includes using gdb to resolve a
few symbols in order to identify memory layouts and a few important
global addresses.

This dependency has been kept to a bare minimum so that the unwinder
will be as resilient as possible to changes in the JVM.

Can we have this in OpenJDK?
----------------------------

The current code is LGPL licensed. I'd be very happy to contribute it to
OpenJDK as part of the code tools set for JDK developers to use and Red
Hat would be very happy to maintain it once included. Ideally, Red Hat
would like to ship the tool as part of OpenJDK product releases for use
in any distro where minimal debug info is available with the product
release. We see great benefit in the opportunity it provides for support
staff to print stack traces from core files (clearly, shipping with
OpenJDK would help to ensure that changes in the JVM and required
changes in the python code were able to be kept in sync).

I have copied this note to the code-tools-dev list on the assumption
that it is the correct place to open up discussion of whether it
could/should be provided as part of OpenJDK. If not can someone please
let me know where else I should start that discussion.

regards,

Andrew Dinn
-----------
Senior Principal Software Engineer
Red Hat UK Ltd
Registered in England and Wales under Company Registration No. 03798903
Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander

----- 8< -------- 8< -------- 8< -------- 8< -------- 8< -------- 8< ---

Normal gdb backtrace for java thread fails ot unwind through Java frames

(gdb) thread 16
[Switching to thread 16 (Thread 0x7fffa1fff700 (LWP 4930))]
#0 0x00007fffe1113484 in ?? ()
(gdb) bt 3
#0  0x00007fffe1113484 in ?? ()
#1  0x00007fffd3c00c18 in ?? ()
#2  0x0000000000000000 in ?? ()

Backtrace with the unwinder installed

(gdb) thread 2
[Switching to thread 2 (Thread 0x7ffff7fcc700 (LWP 4906))]
#0  0x00007ffff79b6540 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
(gdb) bt 10
#0  0x00007ffff79b6540 in pthread_cond_wait@@GLIBC_2.3.2 ()
    at /lib64/libpthread.so.0
#1  0x00007ffff64ae82d in os::PlatformEvent::park() (this=0x7ffff0019700)
    at
/home/adinn/redhat/openjdk/jdk9/hs/hotspot/src/os/linux/vm/os_linux.cpp:5582
#2  0x00007ffff6486f59 in ObjectMonitor::wait(long, bool, Thread*)
(this=0x7fff90008b80, millis=0, interruptible=true,
__the_thread__=0x7ffff0018800)
    at
/home/adinn/redhat/openjdk/jdk9/hs/hotspot/src/share/vm/runtime/objectMonitor.cpp:1490
#3  0x00007ffff66518cb in ObjectSynchronizer::wait(Handle, long,
Thread*) (obj=..., millis=0, __the_thread__=0x7ffff0018800)
    at
/home/adinn/redhat/openjdk/jdk9/hs/hotspot/src/share/vm/runtime/synchronizer.cpp:496
#4  0x00007ffff618fa54 in JVM_MonitorWait(JNIEnv*, jobject, jlong)
(env=0x7ffff0018a28, handle=0x7ffff7fcb738, ms=0)
    at
/home/adinn/redhat/openjdk/jdk9/hs/hotspot/src/share/vm/prims/jvm.cpp:614
#5  0x00007fffd8924f7d in [interpreted: bc = 0]
java.lang.Object.wait(long) ()
    at java/lang/Object.java
#6  0x00007fffd89034a3 in [interpreted: bc = 38]
java.lang.Thread.join(long) () at java/lang/Thread.java:1352
#7  0x00007fffd89034a3 in [interpreted: bc = 2] java.lang.Thread.join() ()
    at java/lang/Thread.java:1426
#8  0x00007fffd89034a3 in [interpreted: bc = 35]
HelloV.main(java.lang.String) () at HelloV.java:80
#9  0x00007fffd88f89f1 in StubRoutines (1) ()
#10 0x00007ffff6109bb3 in JavaCalls::call_helper(JavaValue*,
methodHandle const&, JavaCallArguments*, Thread*)
(result=0x7ffff7fcbd30, method=..., args=0x7ffff7fcbc10,
__the_thread__=0x7ffff0018800)
    at
/home/adinn/redhat/openjdk/jdk9/hs/hotspot/src/share/vm/runtime/javaCalls.cpp:409
(gdb) thread 26
[Switching to thread 26 (Thread 0x7fffa1fff700 (LWP 4930))]
#0  0x00007fffe03cb664 in ?? ()
(gdb) bt 5
#0  0x00007fffe03cb664 in [compiled offset = 0x3a4] HelloV$1.run() ()
    at HelloV.java:63
#1  0x00007fffd88f89f1 in StubRoutines (1) ()
#2  0x00007ffff6109bb3 in JavaCalls::call_helper(JavaValue*,
methodHandle const&, JavaCallArguments*, Thread*)
(result=0x7fffa1ffecc0, method=..., args=0x7fffa1ffec00,
__the_thread__=0x7ffff0471800)
    at
/home/adinn/redhat/openjdk/jdk9/hs/hotspot/src/share/vm/runtime/javaCalls.cpp:409
#3  0x00007ffff64ad409 in os::os_exception_wrapper(void (*)(JavaValue*,
methodHandle const&, JavaCallArguments*, Thread*), JavaValue*,
methodHandle const&, JavaCallArguments*, Thread*) (f=0x7ffff610951a
<JavaCalls::call_helper(JavaValue*, methodHandle const&,
JavaCallArguments*, Thread*)>, value=0x7fffa1ffecc0, method=...,
args=0x7fffa1ffec00, thread=0x7ffff0471800)
    at
/home/adinn/redhat/openjdk/jdk9/hs/hotspot/src/os/linux/vm/os_linux.cpp:5128
#4  0x00007ffff6109502 in JavaCalls::call(JavaValue*, methodHandle
const&, JavaCallArguments*, Thread*) (result=0x7fffa1ffecc0, method=...,
args=0x7fffa1ffec00, __the_thread__=0x7ffff0471800)
    at
/home/adinn/redhat/openjdk/jdk9/hs/hotspot/src/share/vm/runtime/javaCalls.cpp:298
#5  0x00007ffff610877b in JavaCalls::call_virtual(JavaValue*,
KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)
(result=0x7fffa1ffecc0, spec_klass=..., name=0x7ffff4031338,
signature=0x7ffff4033220, args=0x7fffa1ffec00,
__the_thread__=0x7ffff0471800)
    at
/home/adinn/redhat/openjdk/jdk9/hs/hotspot/src/share/vm/runtime/javaCalls.cpp:193
(gdb)