system profilers and incomplete stacks

[serguei.spitsyn at oracle.com]

Hi Brendan,

We are aware of these issues and work with the Solaris team to fix them 
in JDK 9.
One is the frame pointer is used by the server compiler as a general 
purpose register on intel.
Another is about the virtual (or inlined) frames.

There are a couple of related bugs:
    https://bugs.openjdk.java.net/browse/JDK-6617153
    https://bugs.openjdk.java.net/browse/JDK-6276264

There can be more issues filed on this.
Please, note, that the jstack action is not implemented on Linux yet.

Thanks,
Serguei


On 6/16/14 5:14 PM, Brendan Gregg wrote:
> Thanks but no, I'm aware of that bug and workarounds (I'm using the 
> LD_AUDIT_64=/usr/lib/dtrace/64/libdtrace_forceload.so workaround, 
> which isn't mentioned in the bug comments, but probably should be). 
> That bug is about missing symbols, but the stacks shown in that bug 
> still go all the way to thread_start. My stacks often don't.
>
> For simple programs, the stacks are complete. But something complex 
> (eg, vert.x with event loops), and the stacks are often incomplete, 
> one frame only. Very much like what I see with -fomit-frame-pointer, 
> although this is hotspot, not gcc. Such incomplete stacks are seen 
> using either DTrace or perf_events.
>
> It was suggested to me to email the hotspot developers, because this 
> may well be a hotspot optimization they are familiar with. It may also 
> be something really obvious, like that the JVM breaks native stacks 
> due to optimized frames / green threads / etc, and there is absolutely 
> no way around it (no way to disable it). If that's true, it may also 
> mean that the DTrace jstack() action has always had this issue. I'm 
> still reading the source...
>
> Brendan
>
>
>
> On Mon, Jun 16, 2014 at 4:04 AM, Staffan Larsen 
> <staffan.larsen at oracle.com <mailto:staffan.larsen at oracle.com>> wrote:
>
>     I think this is the bug you are looking at:
>     https://bugs.openjdk.java.net/browse/JDK-7187999, but I’ll defer
>     to someone else to confirm.
>
>     /Staffan
>
>
>     On 16 jun 2014, at 12:47, Roland Westrelin
>     <roland.westrelin at oracle.com <mailto:roland.westrelin at oracle.com>>
>     wrote:
>
>>     Forwarding to serviceability alias where this question belongs I
>>     think.
>>
>>     Begin forwarded message:
>>
>>>     *From: *Brendan Gregg <brendan.d.gregg at gmail.com
>>>     <mailto:brendan.d.gregg at gmail.com>>
>>>     *Subject: **system profilers and incomplete stacks*
>>>     *Date: *June 12, 2014 at 7:15:54 PM GMT+2
>>>     *To: *hotspot-compiler-dev at openjdk.java.net
>>>     <mailto:hotspot-compiler-dev at openjdk.java.net>
>>>
>>>     G'Day,
>>>
>>>     Is there a way to run hotspot so that a system profiler (eg,
>>>     DTrace, or Linux perf_events) can measure complete stacks? I
>>>     often get incomplete, partial stacks, with one or a few frames
>>>     only. I'm not worried about symbols right now, what I'd like is
>>>     to walk stacks all the way down to thread start.
>>>
>>>     I've been browsing the hotspot code, but haven't found out how
>>>     yet. I suspect it's related to Java optimized frames, and has
>>>     ditched the frame pointer. I was looking for an equivalent
>>>     -fno-omit-frame-pointer option.
>>>
>>>     Here's an example:
>>>
>>>     # dtrace -n 'profile-99 /execname == "java"/ { @[jstack(100,
>>>     8000)] = count(); }'
>>>     [...]
>>>     org/mozilla/javascript/
>>>     ScriptableObject.createSlot(Ljava/lang/String;II)Lorg/mozilla/javascript/ScriptableObject$Slot;*
>>>     0x884acce8200002da
>>>                     1
>>>
>>>     sun/nio/ch/SocketChannelImpl.read(Ljava/nio/ByteBuffer;)I*
>>>     0xffffffff20007f4b
>>>                     1
>>>
>>>     org/mozilla/javascript/ScriptRuntime.newObjectLiteral([Ljava/lang/Object;[Ljava/lang/Object;[ILorg/mozilla/javascript/Context;Lorg/mozilla/javascript/Scriptable;)Lorg/mozilla/javascript/Scriptable;*
>>>                   0xa20000041
>>>                     1
>>>     [...]
>>>
>>>     I see similar incomplete stacks with Linux perf_events. Oracle
>>>     JDKs from 6 to 8, and OpenJDK.
>>>
>>>     thanks,
>>>
>>>     Brendan
>>>     -- 
>>>     http://www.brendangregg.com <http://www.brendangregg.com/>
>>
>
>
>
>
> -- 
> http://www.brendangregg.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/hotspot-compiler-dev/attachments/20140616/65ab5ca0/attachment-0001.html>