system profilers and incomplete stacks

Wed Oct 22 06:43:43 UTC 2014

On 22 okt 2014, at 02:10, Brendan Gregg <brendan.d.gregg at gmail.com> wrote:

> G'Day,
> 
> I checked the JDK 9 early access releases, but didn't see anything for
> JDK-6276264.

That bug has been closed as a duplicate of https://bugs.openjdk.java.net/browse/JDK-6617153, which is still open.

> I've also since learned that Twitter has an OpenJDK fork
> with frame pointers disabled, for the same purpose: stack profiling
> (using Linux perf_events). Might this be worked on for JDK 9?

I would welcome that change - perhaps Twitter can contribute it?

Thanks,
/Staffan

> I can
> help test. thanks,
> 
> Brendan
> 
> On Mon, Jun 16, 2014 at 11:52 PM, Brendan Gregg
> <brendan.d.gregg at gmail.com> wrote:
>> G'Day Serguei,
>> 
>> On Mon, Jun 16, 2014 at 10:45 PM, serguei.spitsyn at oracle.com
>> <serguei.spitsyn at oracle.com> wrote:
>>> 
>>> Hi Brendan,
>>> 
>>> We are aware of these issues and work with the Solaris team to fix them in
>>> JDK 9.
>>> One is the frame pointer is used by the server compiler as a general
>>> purpose register on intel.
>>> Another is about the virtual (or inlined) frames.
>>> 
>>> There are a couple of related bugs:
>>>   https://bugs.openjdk.java.net/browse/JDK-6617153
>>>   https://bugs.openjdk.java.net/browse/JDK-6276264
>>> 
>>> There can be more issues filed on this.
>> 
>> 
>> Ah, thanks, it's JDK-6276264.
>> 
>> As Tom Rodriguez said at the time (2005): "The server VM uses the frame
>> pointer as an allocatable register and there's no way to turn that off." I
>> was really hoping there was a way to turn that off, like
>> -fno-omit-frame-pointer.
>> 
>> This also means DTrace jstack() has never worked fully. For the applications
>> I tried it on, 50% of stacks were incomplete. Perhaps it wasn't that bad in
>> 2005. I've been getting more mileage today from Java profilers.
>> 
>>> Please, note, that the jstack action is not implemented on Linux yet.
>> 
>> 
>> Linux doesn't have DTrace jstack(), no, but its perf_events does has support
>> for loading an auxiliary file of symbols, which can created via a Java agent
>> for that purpose (eg, https://github.com/jrudolph/perf-map-agent). But that
>> hasn't been working fully for the same reason - incomplete stacks.
>> 
>> Brendan
>> 
>>> 
>>> Thanks,
>>> Serguei
>>> 
>>> 
>>> 
>>> On 6/16/14 5:14 PM, Brendan Gregg wrote:
>>> 
>>> Thanks but no, I'm aware of that bug and workarounds (I'm using the
>>> LD_AUDIT_64=/usr/lib/dtrace/64/libdtrace_forceload.so workaround, which
>>> isn't mentioned in the bug comments, but probably should be). That bug is
>>> about missing symbols, but the stacks shown in that bug still go all the way
>>> to thread_start. My stacks often don't.
>>> 
>>> For simple programs, the stacks are complete. But something complex (eg,
>>> vert.x with event loops), and the stacks are often incomplete, one frame
>>> only. Very much like what I see with -fomit-frame-pointer, although this is
>>> hotspot, not gcc. Such incomplete stacks are seen using either DTrace or
>>> perf_events.
>>> 
>>> It was suggested to me to email the hotspot developers, because this may
>>> well be a hotspot optimization they are familiar with. It may also be
>>> something really obvious, like that the JVM breaks native stacks due to
>>> optimized frames / green threads / etc, and there is absolutely no way
>>> around it (no way to disable it). If that's true, it may also mean that the
>>> DTrace jstack() action has always had this issue. I'm still reading the
>>> source...
>>> 
>>> Brendan
>>> 
>>> 
>>> 
>>> On Mon, Jun 16, 2014 at 4:04 AM, Staffan Larsen
>>> <staffan.larsen at oracle.com> wrote:
>>>> 
>>>> I think this is the bug you are looking at:
>>>> https://bugs.openjdk.java.net/browse/JDK-7187999, but I’ll defer to someone
>>>> else to confirm.
>>>> 
>>>> /Staffan
>>>> 
>>>> 
>>>> On 16 jun 2014, at 12:47, Roland Westrelin <roland.westrelin at oracle.com>
>>>> wrote:
>>>> 
>>>> Forwarding to serviceability alias where this question belongs I think.
>>>> 
>>>> Begin forwarded message:
>>>> 
>>>> From: Brendan Gregg <brendan.d.gregg at gmail.com>
>>>> Subject: system profilers and incomplete stacks
>>>> Date: June 12, 2014 at 7:15:54 PM GMT+2
>>>> To: hotspot-compiler-dev at openjdk.java.net
>>>> 
>>>> G'Day,
>>>> 
>>>> Is there a way to run hotspot so that a system profiler (eg, DTrace, or
>>>> Linux perf_events) can measure complete stacks? I often get incomplete,
>>>> partial stacks, with one or a few frames only. I'm not worried about symbols
>>>> right now, what I'd like is to walk stacks all the way down to thread start.
>>>> 
>>>> I've been browsing the hotspot code, but haven't found out how yet. I
>>>> suspect it's related to Java optimized frames, and has ditched the frame
>>>> pointer. I was looking for an equivalent -fno-omit-frame-pointer option.
>>>> 
>>>> Here's an example:
>>>> 
>>>> # dtrace -n 'profile-99 /execname == "java"/ { @[jstack(100, 8000)] =
>>>> count(); }'
>>>> [...]
>>>>              org/mozilla/javascript/
>>>> 
>>>> ScriptableObject.createSlot(Ljava/lang/String;II)Lorg/mozilla/javascript/ScriptableObject$Slot;*
>>>>              0x884acce8200002da
>>>>                1
>>>> 
>>>>              sun/nio/ch/SocketChannelImpl.read(Ljava/nio/ByteBuffer;)I*
>>>>              0xffffffff20007f4b
>>>>                1
>>>> 
>>>> 
>>>> org/mozilla/javascript/ScriptRuntime.newObjectLiteral([Ljava/lang/Object;[Ljava/lang/Object;[ILorg/mozilla/javascript/Context;Lorg/mozilla/javascript/Scriptable;)Lorg/mozilla/javascript/Scriptable;*
>>>>              0xa20000041
>>>>                1
>>>> [...]
>>>> 
>>>> I see similar incomplete stacks with Linux perf_events. Oracle JDKs from
>>>> 6 to 8, and OpenJDK.
>>>> 
>>>> thanks,
>>>> 
>>>> Brendan
>>>> --
>>>> http://www.brendangregg.com
>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> http://www.brendangregg.com
>>> 
>>> 
>> 
>> 
>> 
>> --
>> http://www.brendangregg.com