system profilers and incomplete stacks

Tue Jun 17 07:06:45 UTC 2014

G'Day David,

Looks like it's JDK-6276264; I'll answer these questions anyone in case
anyone was interested...

On Mon, Jun 16, 2014 at 10:48 PM, David Holmes <david.holmes at oracle.com>
wrote:

> Hi Brendan,
>
> I'm not quite seeing the complete context here so a few queries:
>
> 1. Which platform are you running Dtrace on? (hotspot has to provide
> helper d-scripts to allow Dtrace to walk Java stacks)
>

OmniOS.

I'd also tried Oracle Linux, but it doesn't look like they have done
jstack() yet.

> 2. What are perf-events? How do they try to walk stacks?
>

The Linux perf command, which is sometimes known as perf_events. Like
DTrace, it's hard to summarize as it does many things, including static and
dynamic probe instrumentation, sampling, and profiling of CPU performance
counters. It's probably the most important of all the Linux tools in this
area because it is in the mainline kernel.

It has a couple of stack walking modes: frame pointers, and libdw DWARF
unwind. More modes could be added.

>
> 3. Does "jstack -m" show full stacks for the apps where you see problem?
>

I'll have to compare it more closely, but on first look it appears to match
the broken stacks. Eg:

[...]
----------------- t at 21 -----------------
0x00007ffffee69f3a      ioctl + 0xa
0x00007fffe7cbc8b2      <Unknown compiled code>
0x00007fffe7dea54c      <Unknown compiled code>
----------------- t at 23 -----------------
0x00007ffffee6a96a      __writev + 0xa
0x00007ffff86e8abe      Java_sun_nio_ch_FileDispatcherImpl_writev0 + 0x2e
0x00007fffe7ca258b      <Unknown compiled code>
0x00007fffe7d5c500      *
sun.nio.ch.SocketDispatcher.writev(java.io.FileDescriptor, long, int) bci:4
line:51 (Compiled frame)
* sun.nio.ch.IOUtil.write(java.io.FileDescriptor, java.nio.ByteBuffer[],
int, int, sun.nio.ch.NativeDispatcher) bci:277 line:148 (Compiled frame)
----------------- t at 24 -----------------
0x00007ffffee6a96a      __writev + 0xa
0x00007ffff86e8abe      Java_sun_nio_ch_FileDispatcherImpl_writev0 + 0x2e
0x00007fffe7ca258b      <Unknown compiled code>
0x00007fffe7d5c500      *
sun.nio.ch.SocketDispatcher.writev(java.io.FileDescriptor, long, int) bci:4
line:51 (Compiled frame)
* sun.nio.ch.IOUtil.write(java.io.FileDescriptor, java.nio.ByteBuffer[],
int, int, sun.nio.ch.NativeDispatcher) bci:277 line:148 (Compiled frame)
----------------- t at 25 -----------------
[...]

Thanks,

Brendan

> Thanks,
> David
>
>
> On 17/06/2014 10:14 AM, Brendan Gregg wrote:
>
>> Thanks but no, I'm aware of that bug and workarounds (I'm using the
>> LD_AUDIT_64=/usr/lib/dtrace/64/libdtrace_forceload.so workaround, which
>> isn't mentioned in the bug comments, but probably should be). That bug
>> is about missing symbols, but the stacks shown in that bug still go all
>> the way to thread_start. My stacks often don't.
>>
>> For simple programs, the stacks are complete. But something complex (eg,
>> vert.x with event loops), and the stacks are often incomplete, one frame
>> only. Very much like what I see with -fomit-frame-pointer, although this
>> is hotspot, not gcc. Such incomplete stacks are seen using either DTrace
>> or perf_events.
>>
>> It was suggested to me to email the hotspot developers, because this may
>> well be a hotspot optimization they are familiar with. It may also be
>> something really obvious, like that the JVM breaks native stacks due to
>> optimized frames / green threads / etc, and there is absolutely no way
>> around it (no way to disable it). If that's true, it may also mean that
>> the DTrace jstack() action has always had this issue. I'm still reading
>> the source...
>>
>> Brendan
>>
>>
>>
>> On Mon, Jun 16, 2014 at 4:04 AM, Staffan Larsen
>> <staffan.larsen at oracle.com <mailto:staffan.larsen at oracle.com>> wrote:
>>
>>     I think this is the bug you are looking at:
>>     https://bugs.openjdk.java.net/browse/JDK-7187999, but I’ll defer to
>>     someone else to confirm.
>>
>>     /Staffan
>>
>>
>>     On 16 jun 2014, at 12:47, Roland Westrelin
>>     <roland.westrelin at oracle.com <mailto:roland.westrelin at oracle.com>>
>>     wrote:
>>
>>      Forwarding to serviceability alias where this question belongs I
>>>     think.
>>>
>>>     Begin forwarded message:
>>>
>>>      *From: *Brendan Gregg <brendan.d.gregg at gmail.com
>>>>     <mailto:brendan.d.gregg at gmail.com>>
>>>>     *Subject: **system profilers and incomplete stacks*
>>>>     *Date: *June 12, 2014 at 7:15:54 PM GMT+2
>>>>     *To: *hotspot-compiler-dev at openjdk.java.net
>>>>     <mailto:hotspot-compiler-dev at openjdk.java.net>
>>>>
>>>>
>>>>     G'Day,
>>>>
>>>>     Is there a way to run hotspot so that a system profiler (eg,
>>>>     DTrace, or Linux perf_events) can measure complete stacks? I
>>>>     often get incomplete, partial stacks, with one or a few frames
>>>>     only. I'm not worried about symbols right now, what I'd like is
>>>>     to walk stacks all the way down to thread start.
>>>>
>>>>     I've been browsing the hotspot code, but haven't found out how
>>>>     yet. I suspect it's related to Java optimized frames, and has
>>>>     ditched the frame pointer. I was looking for an equivalent
>>>>     -fno-omit-frame-pointer option.
>>>>
>>>>     Here's an example:
>>>>
>>>>     # dtrace -n 'profile-99 /execname == "java"/ { @[jstack(100,
>>>>     8000)] = count(); }'
>>>>     [...]
>>>>                   org/mozilla/javascript/
>>>>     ScriptableObject.createSlot(Ljava/lang/String;II)Lorg/
>>>> mozilla/javascript/ScriptableObject$Slot;*
>>>>                   0x884acce8200002da
>>>>                     1
>>>>
>>>>
>>>>     sun/nio/ch/SocketChannelImpl.read(Ljava/nio/ByteBuffer;)I*
>>>>                   0xffffffff20007f4b
>>>>                     1
>>>>
>>>>
>>>>     org/mozilla/javascript/ScriptRuntime.newObjectLiteral([Ljava/lang/
>>>> Object;[Ljava/lang/Object;[ILorg/mozilla/javascript/
>>>> Context;Lorg/mozilla/javascript/Scriptable;)Lorg/
>>>> mozilla/javascript/Scriptable;*
>>>>                   0xa20000041
>>>>                     1
>>>>     [...]
>>>>
>>>>     I see similar incomplete stacks with Linux perf_events. Oracle
>>>>     JDKs from 6 to 8, and OpenJDK.
>>>>
>>>>     thanks,
>>>>
>>>>     Brendan
>>>>     --
>>>>     http://www.brendangregg.com <http://www.brendangregg.com/>
>>>>
>>>
>>>
>>
>>
>>
>> --
>> http://www.brendangregg.com
>>
>

-- 
http://www.brendangregg.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/serviceability-dev/attachments/20140617/68651754/attachment-0001.html>