PowerPC: core file option not available with serviceability tools
Maynard Johnson
maynardj at us.ibm.com
Fri Nov 14 17:00:27 UTC 2014
On 10/07/2014 08:35 AM, Maynard Johnson wrote:
> On 10/07/2014 03:58 AM, Volker Simonis wrote:
>> Hi Maynard,
>>
>> I'm now back from JavaOne and can look at this issue. Could you please
>> share your current implementation so I can reproduce your problem more
>> easily.
> See attachment. The two patches in the attached tar file apply to a jdk9-dev snapshot from July. I haven't even tried forward-porting to current upstream code, so I don't know how well they would apply.
>>
>> By the way, you can find the ppc frame layout description of all the
>> different frame types (native, interpreted, compiled) in
>> /hotspot/src/cpu/ppc/vm/frame_ppc.hpp. The different frame::sender()
>> implementations (in frame_x86.cpp, frame_ppc.cpp, ..) contain all the
> Yes, I've been studying those files, but I freely admit I don't have a good grasp of the code yet.
>> gory details about how to walk a frame. That's what you have to
>> implement in Java in order to get a full stack trace from the
>> serviceability tools. On x86 the frame pointer (i.e. the ebp register
>> points to the last frame pointer while (frame pointer - 1) points to
>> the return pc.
> Ummm . . . Stack addresses grow downward, and I was under the impression 'return pc' was the word on the stack directly before the ebp, which would mean return pc is at 'frame pointer + 1'. Or am I off base here? Nevertheless, my question below concerns the "pc", not the "return pc". Perhaps I'm misunderstanding "pc" in this context; but even so, the x86 code still seems wrong:
>
> this.pc = raw_sp.getAddressAt(-1 * VM.getVM().getAddressSize());
>
> If, in fact, 'this.pc' is supposed to represent 'return pc' in this context, then I would think the code should be:
>
> this.pc = raw_fp.getAddressAt(VM.getVM().getAddressSize());
>
> I hope you can help set me on the right track. As you can see, I'm lost in the weeds right now. :-)
Hi, Volker,
I presume you've been too busy to help out with this, so I had another go at it, and was able to make
quite a bit of progress. The main issue was that I was unwittingly building my openJDK source
with the "--with-jvm-interpreter=cpp" configure option! Once I discovered that and removed that option,
the stack started making sense, since my new ppc64 agent's reflection of the stack assumes the use
of the template interpreter (as does the x86 agent code which the ppc64 stuff is based on).
So now I have jmap, jsadebugd, and jstack working for ppc64 on both big endian and little endian -- almost.
There are a couple issues yet with jstack that I'm stuck on. I will post the patches as an RFC
and document those issues. If you could find time to review the patches and perhaps help out with
the open issues, I would appreciate it. Thanks!
-Maynard
>
> -Maynard
>>
>> Regards,
>> Volker
>>
>>
>> On Tue, Sep 30, 2014 at 12:42 AM, Maynard Johnson <maynardj at us.ibm.com> wrote:
>>> On 07/09/2014 12:38 PM, Volker Simonis wrote:
>>>> On Wed, Jul 9, 2014 at 3:45 PM, Maynard Johnson <maynardj at us.ibm.com> wrote:
>>>>> On 07/04/2014 10:59 AM, Volker Simonis wrote:
>>>>>> Hi Maynard,
>>>>>>
>>>>>> we (i.e. SAP) do not currently support the SA agent on Linux/PPC64 and
>>>>>> AIX (we have other proprietary servicibility tools). Because of that
>>>>>> (and because SA isn't specified by the SE specification) porting the
>>>>>> SA agent was no top priority until now. But there are no technical
>>>>>> reasons why it should not work (it's just a lack of resources). Of
>>>>>> course contributions are always highly welcome:)
>>>>>>
>>>>>> That said, the SA agent library and jar file actually gets build. If
>>>>>> you do a complete build you'll find them under:
>>>>>>
>>>>>> hotspot/linux_ppc64_compiler2/generated/sa-jdi.jar
>>>>>> hotspot/linux_ppc64_compiler2/{product,fastdebug,debug}/libsaproc.so
>>>>>>
>>>>>> in the build directory. They are just not copied into the jdk
>>>>>> workspace and the created images because they don't work out of the
>>>>>> box.
>>>>>>
>>>>>> The following two patches for the jdk9 top-level and hotspot
>>>>>> repositories respectively should fix the build such that the agent
>>>>>> files will be correctly copied into the images.
>>>>>>
>>>>>> http://cr.openjdk.java.net/~simonis/webrevs/sa_toplevel
>>>>>> http://cr.openjdk.java.net/~simonis/webrevs/sa_hotspot/
>>>>>>
>>>>>> They will get you to the point where for example 'jstack' will run up
>>>>>> to the following point:
>>>>> Ok, great. This should be enough to get me started. I should have time to begin on this later this week or early next week.
>>>>
>>>> Hi Maynard,
>>>>
>>>> great to welcome you in the ppc64 porting team:)
>>>>
>>>>> I may come knocking at your "door" for some occasional help, but I'll try to keep that to a minimum.
>>> Hi, Volker. Knock, knock. :-)
>>> I was preoccupied for a while this summer rolling out the latest release of oprofile (for which I'm the maintainer), but am now coming back to this task. I've implemented what I believe are all of the necessary ppc64-specific Java files to enable the jstack and jmap tools to work on core files. I've also updated hotspot/agent/src/os/linux/LinuxDebuggerLocal.c to implement the accumulation of register data on ppc64 vi ptrace. But now I've run into a problem I need help with.
>>>
>>> When I run jstack on my POWER7 system, it gets stuck in a loop in sun.jvm.hotspot.tools.StackTrace::run. There's an inner for-loop there where cur.getLastJavaVFrameDbg() is called ('cur' is a JavaThread). For the first JavaThread, we do return from getLastJavaVFrameDbg(), just as we do when running jstack on my Intel laptop. But for the second JavaThread, we never return from getLastJavaVFrameDbg() on ppc64. I believe the root of the problem is in my new sun.jvm.hotspot.runtime.ppc64.PPC64Frame class. The getLastJavaVFrameDbg method calls getCurrentFrameGuess, which is implemented in the new sun.jvm.hotspot.runtime.linux_ppc64.LinuxPPC64JavaThreadPDAccess class. In both ppc64 and x86, this first level xxxCurrentFrameGuess object is instantiated with a 'pc' value of null, so getCurrentFrameGuess then new's up a xxxFrame object, passing in the SP and FP, but no PC. The implementation of the PPC64Frame(Address,Address) constructor is currently identical to the X86Frame !
c!
> ons!
>>> tructor, b
>>> ut is almost certainly incorrect. In this constructor, the 'pc' is set as follows:
>>> this.pc = raw_sp.getAddressAt(-1 * VM.getVM().getAddressSize());
>>>
>>> This works fine on X86, but not on ppc64. But I'm not understanding how this even works on X86. From what I understand, the data below the stack pointer on X86 is the "red zone". How is that being used as a pc? But more importantly, do you know how I can ascertain what the 'pc' value should be for ppc64?
>>>
>>> Thanks in advance for any assistance you can give.
>>>
>>> -Maynard
>>>
>>>>
>>>> Please feel free to ask any questions. The OpenJDK project and
>>>> especially the HotSpot part are known to take some getting used to.
>>>>
>>>>> I was wondering if a bug report should be opened in JBS, just to record that the issue is being worked. Thoughts?
>>>>
>>>> I have opened "8049715: PPC64: First steps to enable SA on
>>>> Linux/PPC64" (https://bugs.openjdk.java.net/browse/JDK-8049715) for
>>>> the patch which I sent you with the last mail. I've already sent out
>>>> webrevs for that change and hopefully it will be fixed within the next
>>>> few days.
>>>>
>>>> For the actual port of the ppc64-specific stuff I opened bug "8049716
>>>> PPC64: Implement SA on Linux/PPC64"
>>>> (https://bugs.openjdk.java.net/browse/JDK-8049716). I can also help
>>>> with hosting the webrevs, once you have a running version.
>>>>
>>>> Regards,
>>>> Volker
>>>>
>>>>>
>>>>> -Maynard
>>>>>>
>>>>>>> images/j2sdk-image/bin/jstack ./jdk/bin/java core.13547
>>>>>> Attaching to core core.13547 from executable ./jdk/bin/java, please wait...
>>>>>> WARNING: Hotspot VM version
>>>>>> 1.9.0-internal-debug-d046063_2014_07_04_11_46-b00 does not match with
>>>>>> SA version 1.9.0-internal-debug-d046063_2014_07_04_11_46-b00. You may
>>>>>> see unexpected results.
>>>>>> Debugger attached successfully.
>>>>>> Server compiler detected.
>>>>>> JVM version is 1.9.0-internal-debug-d046063_2014_07_04_11_46-b00
>>>>>> Deadlock Detection:
>>>>>>
>>>>>> Exception in thread "main" java.lang.reflect.InvocationTargetException
>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:484)
>>>>>> at sun.tools.jstack.JStack.runJStackTool(JStack.java:140)
>>>>>> at sun.tools.jstack.JStack.main(JStack.java:106)
>>>>>> Caused by: java.lang.ExceptionInInitializerError
>>>>>> at sun.jvm.hotspot.runtime.VM.getThreads(VM.java:610)
>>>>>> at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:54)
>>>>>> at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:39)
>>>>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:62)
>>>>>> at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>>>>>> at sun.jvm.hotspot.tools.JStack.run(JStack.java:66)
>>>>>> at sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:260)
>>>>>> at sun.jvm.hotspot.tools.Tool.start(Tool.java:223)
>>>>>> at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
>>>>>> at sun.jvm.hotspot.tools.JStack.main(JStack.java:92)
>>>>>> ... 6 more
>>>>>> Caused by: java.lang.RuntimeException: OS/CPU combination linux/ppc64
>>>>>> not yet supported
>>>>>> at sun.jvm.hotspot.runtime.Threads.initialize(Threads.java:97)
>>>>>> at sun.jvm.hotspot.runtime.Threads.access$000(Threads.java:42)
>>>>>> at sun.jvm.hotspot.runtime.Threads$1.update(Threads.java:52)
>>>>>> at sun.jvm.hotspot.runtime.VM.registerVMInitializedObserver(VM.java:394)
>>>>>> at sun.jvm.hotspot.runtime.Threads.<clinit>(Threads.java:50)
>>>>>> ... 16 more
>>>>>>
>>>>>> And that's the point where I was saying that "contributions are always
>>>>>> highly welcome:)"
>>>>>>
>>>>>> Now all the Linux/PPC64 specific class under
>>>>>> hotspot/agent/src/share/classes/ would have to be implemented (e.g.
>>>>>> sun/jvm/hotspot/runtime/amd64/AMD64CurrentFrameGuess). Are you
>>>>>> interested in contributing to this project?
>>>>>>
>>>>>> Regards,
>>>>>> Volker
>>>>>>
>>>>>> PS: I cc-ed serviceability-dev because I remember that they started a
>>>>>> poll a while ago about who is using the SA tools. I'm therefore not
>>>>>> quite sure what's the current status and what are the future plan for
>>>>>> these libraries.
>>>>>>
>>>>>>
>>>>>> On Thu, Jul 3, 2014 at 9:04 PM, Maynard Johnson <maynardj at us.ibm.com> wrote:
>>>>>>> Hi, all,
>>>>>>> On my Intel laptop, I note that certain jdk9 serviceability tools -- jstack, jmap, jsadebugd -- have an option to pass a core file instead of a process ID; for example:
>>>>>>>
>>>>>>> $ jstack -h
>>>>>>> Usage:
>>>>>>> jstack [-l] <pid>
>>>>>>> (to connect to running process)
>>>>>>> jstack -F [-m] [-l] <pid>
>>>>>>> (to connect to a hung process)
>>>>>>> jstack [-m] [-l] <executable> <core>
>>>>>>> (to connect to a core file)
>>>>>>> jstack [-m] [-l] [server_id@]<remote server IP or hostname>
>>>>>>> (to connect to a remote debug server)
>>>>>>>
>>>>>>> But on my PowerLinux box, the core file option is missing from the usage output. I see that jdk9-dev/jdk/src/share/classes/sun/tools/jstack/JStack.java requires the existence of sun.jvm.hotspot.tools.JStack for the core file option to be made available. On my Intel system, the sun.jvm.hotspot.tools.JStack class is packaged in sa-jdi.jar in <jdk9Dev-install>/jvm/openjdk-1.9.0-internal/lib/. But the sa-jdi.jar is missing on PowerPC. Is there a technical reason for this or is it an oversight?
>>>>>>>
>>>>>>> The jsadebugd tool does not run at all on PowerLinux; it gets the following error:
>>>>>>>
>>>>>>> Error: Could not find or load main class sun.jvm.hotspot.jdi.SADebugServer
>>>>>>>
>>>>>>> On my Intel system, the SADebugServer class is packaged in the sa-jdi.jar mentioned above.
>>>>>>>
>>>>>>> I've spent the past day or so looking at makefiles until I'm cross-eyed, but haven't yet found where the issue might be. Any tips would be appreciated.
>>>>>>>
>>>>>>> Thanks.
>>>>>>> -Maynard
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
More information about the ppc-aix-port-dev
mailing list