gdb and OpenJDK

Sun Feb 15 19:55:58 UTC 2015

Alexandre,

I think what Erik suggested was if there was some way the JVM could expose data in a format that is easy to interpret by other tools (such as the python gdb plugin, but also plugins for other debuggers, or SA). Of course this would have to be data, not code, so that it would be available in core files as well. I haven’t seen the python module you have written so I don’t know how complex is it, but we should think of ways to make such code even simpler if possible. If we had data exposed in an easy-to-read format it would perhaps make maintenance of these tools simpler. We have a problem with SA today that it is way to dependent on the code in the JVM - a small change in data structures in the JVM will break SA, something we hare looking for solutions to.

Thanks,
/Staffan

> On 14 feb 2015, at 00:13, Alexander Smundak <asmundak at google.com> wrote:
> 
> You have to solve two problems to be able to use GDB to debug mixed language
> applications: how to walk the stack, and how to symbolize the frames.
> And, it has
> to work on core dumps, so it cannot rely on executing the code in debuggee.
> You can use JIT reader interface to walk the stack (that was my first attempt),
> but there is no binary interface to symbolize the frames, it can be
> achieved only
> via Python plugin. Having two pieces of code, a binary (shared object)
> containing JIT reader written in C or C++, and a Python module for symbolizing
> frames, is really inconvenient.
> 
> GDB is very much set on using Python as extension language, it exposes quite
> sizable piece of its internals to it: you can programmatically issue commands,
> implement new CLI commands, inspect debuggee, programmatically respond
> to debuggee signals (will prove to be quite handy for JVM, Python code can
> handle SIGSEGV intelligently by passing it to JVM to handle null
> pointer exception
> if it happens in Java code).
> IMHO it's a fairly good trade-off, despite the understandable reluctance to use
> yet another language.
> 
> On Fri, Feb 13, 2015 at 2:02 AM, Erik Helin <erik.helin at oracle.com> wrote:
>> On 2015-02-13, Erik Helin wrote:
>>> On 2015-02-12, Alexander Smundak wrote:
>>>> It was the previous implementation that used the serviceability agent,
>>>> sorry for the confusion. The new one replicates the necessary
>>>> functionality of the serviceability agent instead. All the operations
>>>> on the debuggee (catching events, reading writing to the debuggee
>>>> memory, etc.) are performed by the GDB.
>>>> The Python code consists of two "plugins". The first does "unwinding",
>>>> that is, finding out the caller's frame for a given frame. JIT frames
>>>> need unwinding because JIT code uses frame pointer as a general
>>>> purpose register (I am speaking of x86_64), so GDB cannot unwind it.
>>>> So the unwinding plugin locates nmethod instance containing given PC,
>>>> fetches the frame size from it, and is thus able to locate the
>>>> previous frame.
>>>> The second plugin symbolizes Java frames (GDB already has the
>>>> interface for that called FrameDecorators), and it requires sizable
>>>> amount of code to traverse various JVM structures to do that.
>>>> The Python module is currently about 2500 lines of code. About 10% of
>>>> this code is platform-dependent (I have implementations for x86_64 and
>>>> PowerPC64 and working on Aarch64).
>>> 
>>> First of all, thanks for putting work into this, having Java stacktraces in
>>> GDB would be really helpful.
>>> 
>>> Just a quick question, instead of rewriting the stack walking code in
>>> Python, would it be possible to use gdbjit [0]? Apparently, this is used
>>> by LLVM MC JIT and I also think that V8 has some basic support for it.
>>> The benefit of using something like gdbjit would be to not have the
>>> stack walking code in multiple places:
>>> - the vm
>>> - SA
>>> - Python for gdb
>> 
>> I just had a discussion with Staffan about this, and I just want to
>> clarify my comment: instead of rewriting the stack walking code in
>> multiple places, can we add some code in the VM that aids the
>> development of tools like these? We will of course require platform/tool
>> specific code, and probably some duplication, but how little duplication
>> can we get away with? gdbjit seems like it can consume a custom debug
>> info format, maybe we can design a very lightweight debug info format
>> that then can be picked up by a variety of tools (lldb, windbg, gdb,
>> SA, etc...)?
>> 
>> Thanks,
>> Erik
>> 
>>> Thanks,
>>> Erik
>>> 
>>> [0]: https://sourceware.org/gdb/onlinedocs/gdb/JIT-Interface.html
>>> 
>>>> Sasha
>>>> 
>>>> On Thu, Feb 12, 2015 at 2:14 AM, Volker Simonis
>>>> <volker.simonis at gmail.com> wrote:
>>>>> Hi Jeremy,
>>>>> 
>>>>> nice to see that somebody is working on this topic.
>>>>> 
>>>>> Can you please detail a little bit on how this would work?
>>>>> As far as I understand you extend gdb to support "frame undwinder"
>>>>> plugins written in Python.
>>>>> 
>>>>> One of these plugins, which you want to contribute to the OpenJDK, can
>>>>> be used to unwind native Java/HotSpot stack frames by calling out to
>>>>> the serviceability agent. Can you please explain how this could work?
>>>>> If you debug a Java process with gdb, gdb attaches to that process
>>>>> using "ptrace". The SA agent uses "ptrace" as well but as far as I
>>>>> know, only once process can attach to another process at the same
>>>>> time. So how is this callback from gdb to the SA agent supposed to
>>>>> work?
>>>>> 
>>>>> Regards,
>>>>> Volker
>>>>> 
>>>>> 
>>>>> On Wed, Feb 11, 2015 at 11:27 PM, Jeremy Manson <jeremymanson at google.com> wrote:
>>>>>> Hey folks,
>>>>>> 
>>>>>> I think I've mentioned to some of the people on this list that we (and by
>>>>>> we, I mean Sasha Smundak) have been working on interoperability between gdb
>>>>>> and Java.  Basically, what we have now allows us to get a backtrace in gdb
>>>>>> that includes Java frames with parameter values (no local variables yet).
>>>>>> 
>>>>>> Needless to say, this has been terribly, terribly helpful in our JVM and JNI
>>>>>> debugging.
>>>>>> 
>>>>>> Support for this feature is basically two parts:
>>>>>> 
>>>>>> First, gdb needed to be extended to support the ability to plug in a frame
>>>>>> unwinder.  The process of submitting this to gdb (which has been ongoing for
>>>>>> quite a while) is finally starting to make reasonable progress.
>>>>>> 
>>>>>> Next, we need a plugin for OpenJDK.  We have a plugin written that calls out
>>>>>> to the serviceability agent.  (It's written in Python, per gdb's
>>>>>> requirements).
>>>>>> 
>>>>>> Pending the gdb support being finalized, we'd love to contribute the plugin
>>>>>> to OpenJDK.    What should we do make that happen?  Write a JEP and submit
>>>>>> it to this list?  Technically, neither of us are committers (which you need
>>>>>> to be to submit a JEP), but I imagine we can work around that.
>>>>>> 
>>>>>> Here's the gdb patch, for anyone interested:
>>>>>> 
>>>>>> https://sourceware.org/ml/gdb-patches/2014-12/msg00408.html
>>>>>> 
>>>>>> Thanks!
>>>>>> 
>>>>>> Jeremy