[PATCH RFC 0/2] Add linux/ppc64 support for Hotspot serviceability agent to read core files

[Volker Simonis]

On Wed, Dec 3, 2014 at 7:33 PM, Maynard Johnson <maynardj at us.ibm.com> wrote:
> On 12/03/2014 02:40 AM, Volker Simonis wrote:
>> On Tue, Dec 2, 2014 at 9:50 PM, Maynard Johnson <maynardj at us.ibm.com> wrote:
>>> On 12/02/2014 11:45 AM, Volker Simonis wrote:
>>>> Hi Maynard,
>>>>
>>>> did you intentionally answered only to me or can we take the discussion back to the mailing list?
>>>
>>> Yes, it was intentional. I figured that if we did indeed have some functional problems to fix,
>>> we may want to discuss off-list until we agree we're ready for full patch review. I apologize if
>>> that's not following normal process. Let me know how and when you'd like me to bring this discussion
>>> back to the list.
> Hi, Volker,
> You've answered all of my concerns -- thanks for your patience.  Can you please let me know the
> process for getting the web review moving?
>
>>>
>>>> Please find further comments inline:
>>>>
>>>> On Tue, Dec 2, 2014 at 12:16 AM, Maynard Johnson <maynardj at us.ibm.com <mailto:maynardj at us.ibm.com>> wrote:
>>>>> On 11/25/2014 12:40 PM, Volker Simonis wrote:
>>>>>> Hi Maynard,
>>>>>>
>>>>>> so here finally comes my version of your change. I've tested 'jstack'
>>>>>> with no, '-F' and '-m' flags on PIDs and core files on BigEndian
>>>>>> Linux/PPC64 and it seems to work quite stable now. I've also done some
>>>>>> basic tests with 'jmap' with no, '-histo' and '-F' flags and 'jinfo'.
>>>>> Hi, Volker,
>>>>> Thank you very much for your help with the patch I posted.
>>>>> I tried out your new patch and my results aren't quite as expected.  I wonder
>>>>> if I need to update my source tree as it's a few weeks old by now.  Anyway,
>>>>> running 'jstack `which java` core', I usually get something like the following
>>>>> for the thread that's running the actual java app code:
>>>>
>>>> I've cloned today a fresh copy of jdk9-dev, applied the patch and build the images. Everything worked as expected. Could you please also try with a fresh version of jdk9-dev.
>>>
>>> OK, I did the same today.
>>>
>>>>
>>>>>
>>>>> ---------------------------------
>>>>> Thread 22585: (state = IN_JAVA, current Java SP = null
>>>>> )
>>>>>  - test.main(java.lang.String[]) @bci=100, line=54, pc=0x00003fff8800953c, Method*=0x00003fff7dc00998 (Interpreted frame)
>>>>> ---------------------------------
>>>>>
>>>>> As you can see, there's no compiled frame output where I expect I should see something.
>>>>> I get this pretty consistently on both BE and LE. I also see the same thing running 'jstack -F <pid>'
>>>>> on BE.
>>>>
>>>> That's not unusual. The problem with your example is that it above test.main all the other methods are compiled AND inlined. That means that there's just one physical frame above test.main. If the stack tracing routine can't make sense of the first frame, in not only skips that frame, but also all the virtual java frames which are contained inside this physical frame. You can try to run your example with "-XX:-Inline" to avoid inlining and see deeper stack traces.
>>>
>>> Thanks for the tip. Using "-XX:-Inline" does show more stack.  Sometimes the stack looks like following:
>>>
>>> ---------------------------------
>>> Thread 18959: (state = IN_JAVA)
>>>  - test.get_my_chars(int, int, char[], java.lang.String, int) @bci=34, line=14 (Compiled frame; information may be imprecise)
>>>  - test.run_test() @bci=63, line=31 (Compiled frame)
>>>  - test.main(java.lang.String[]) @bci=100, line=54 (Interpreted frame)
>>> ---------------------------------
>>>
>>> But usually the stack just has "main" and "run_test", always with the same line numbers, whether or not get_my_chars is included in the stack.  But as I mentioned in my earlier note, line #31 in run_test is the first line of a for-loop, not the (expected) call to get_my_chars.
>>>
>>
>> Again, PC to line/BCImapping is only precise at safepoints. There's no
>> chance to get this 100% right. But that's the same on all
>> architectures. If you run with -XX:+PrintInlining you will what has
>> been inlined which may help to interprete the missing parts in a stack
>> trace which was not taken at a safepoint.
>>
>>> In the approximate 10 times that I re-ran my test with the "-XX:-Inline" (sometimes killing it with SIGSEGV to get a core file; sometimes using 'jstack -F'), I twice got an NPE:
>>>
>>> java.lang.NullPointerException
>>>         at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:88)
>>>         at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
>>>             . . .
>>>
>>> Line 88 in StackTrace.java is
>>>
>>>     tty.print(" - " + method.externalNameAndSignature() +
>>>     " @bci=" + vf.getBCI());
>>>
>>> I'll work on debugging this.
>>>
>>
>> Yes, that should be fixed. Maybe we need some more checks if 'method'
>> and 'vf' are valid.
> In fact, I'm consistently seeing the same NPE on x86 when I run the java app with "-XX:-Inline"
> and then do 'jstack -F <pid>'. Do you know if there's a bug report open already for this?
> The jdk9 source on my x86 system was a couple months (or so) old, so I wanted to see if
> this still fails with current upstream code. I refreshed my source tree (via get_source.sh)
> and tried to rebuild it, but I got a compile failure.  I imagine someone else is already

Just as a short note (I'll answer all the other questions tomorrow).
You should call 'bash common/bin/hgforest.sh pull -u' in the top level
repositoy for updating the whole forest. "get_source.sh" will just
fetch a new clone and copy it over the old version. This may be the
source of your your build problem because it may leave old files in
your workspace.

> working on that.
>
> -Maynard
>>
> [snip]
>