system profilers and incomplete stacks

[David Holmes]

Hi Brendan,

I'm not quite seeing the complete context here so a few queries:

1. Which platform are you running Dtrace on? (hotspot has to provide 
helper d-scripts to allow Dtrace to walk Java stacks)

2. What are perf-events? How do they try to walk stacks?

3. Does "jstack -m" show full stacks for the apps where you see problem?

Thanks,
David

On 17/06/2014 10:14 AM, Brendan Gregg wrote:
> Thanks but no, I'm aware of that bug and workarounds (I'm using the
> LD_AUDIT_64=/usr/lib/dtrace/64/libdtrace_forceload.so workaround, which
> isn't mentioned in the bug comments, but probably should be). That bug
> is about missing symbols, but the stacks shown in that bug still go all
> the way to thread_start. My stacks often don't.
>
> For simple programs, the stacks are complete. But something complex (eg,
> vert.x with event loops), and the stacks are often incomplete, one frame
> only. Very much like what I see with -fomit-frame-pointer, although this
> is hotspot, not gcc. Such incomplete stacks are seen using either DTrace
> or perf_events.
>
> It was suggested to me to email the hotspot developers, because this may
> well be a hotspot optimization they are familiar with. It may also be
> something really obvious, like that the JVM breaks native stacks due to
> optimized frames / green threads / etc, and there is absolutely no way
> around it (no way to disable it). If that's true, it may also mean that
> the DTrace jstack() action has always had this issue. I'm still reading
> the source...
>
> Brendan
>
>
>
> On Mon, Jun 16, 2014 at 4:04 AM, Staffan Larsen
> <staffan.larsen at oracle.com <mailto:staffan.larsen at oracle.com>> wrote:
>
>     I think this is the bug you are looking at:
>     https://bugs.openjdk.java.net/browse/JDK-7187999, but I’ll defer to
>     someone else to confirm.
>
>     /Staffan
>
>
>     On 16 jun 2014, at 12:47, Roland Westrelin
>     <roland.westrelin at oracle.com <mailto:roland.westrelin at oracle.com>>
>     wrote:
>
>>     Forwarding to serviceability alias where this question belongs I
>>     think.
>>
>>     Begin forwarded message:
>>
>>>     *From: *Brendan Gregg <brendan.d.gregg at gmail.com
>>>     <mailto:brendan.d.gregg at gmail.com>>
>>>     *Subject: **system profilers and incomplete stacks*
>>>     *Date: *June 12, 2014 at 7:15:54 PM GMT+2
>>>     *To: *hotspot-compiler-dev at openjdk.java.net
>>>     <mailto:hotspot-compiler-dev at openjdk.java.net>
>>>
>>>     G'Day,
>>>
>>>     Is there a way to run hotspot so that a system profiler (eg,
>>>     DTrace, or Linux perf_events) can measure complete stacks? I
>>>     often get incomplete, partial stacks, with one or a few frames
>>>     only. I'm not worried about symbols right now, what I'd like is
>>>     to walk stacks all the way down to thread start.
>>>
>>>     I've been browsing the hotspot code, but haven't found out how
>>>     yet. I suspect it's related to Java optimized frames, and has
>>>     ditched the frame pointer. I was looking for an equivalent
>>>     -fno-omit-frame-pointer option.
>>>
>>>     Here's an example:
>>>
>>>     # dtrace -n 'profile-99 /execname == "java"/ { @[jstack(100,
>>>     8000)] = count(); }'
>>>     [...]
>>>                   org/mozilla/javascript/
>>>     ScriptableObject.createSlot(Ljava/lang/String;II)Lorg/mozilla/javascript/ScriptableObject$Slot;*
>>>                   0x884acce8200002da
>>>                     1
>>>
>>>
>>>     sun/nio/ch/SocketChannelImpl.read(Ljava/nio/ByteBuffer;)I*
>>>                   0xffffffff20007f4b
>>>                     1
>>>
>>>
>>>     org/mozilla/javascript/ScriptRuntime.newObjectLiteral([Ljava/lang/Object;[Ljava/lang/Object;[ILorg/mozilla/javascript/Context;Lorg/mozilla/javascript/Scriptable;)Lorg/mozilla/javascript/Scriptable;*
>>>                   0xa20000041
>>>                     1
>>>     [...]
>>>
>>>     I see similar incomplete stacks with Linux perf_events. Oracle
>>>     JDKs from 6 to 8, and OpenJDK.
>>>
>>>     thanks,
>>>
>>>     Brendan
>>>     --
>>>     http://www.brendangregg.com <http://www.brendangregg.com/>
>>
>
>
>
>
> --
> http://www.brendangregg.com