system profilers and incomplete stacks

Wed Oct 22 00:10:31 UTC 2014

G'Day,

I checked the JDK 9 early access releases, but didn't see anything for
JDK-6276264. I've also since learned that Twitter has an OpenJDK fork
with frame pointers disabled, for the same purpose: stack profiling
(using Linux perf_events). Might this be worked on for JDK 9? I can
help test. thanks,

Brendan

On Mon, Jun 16, 2014 at 11:52 PM, Brendan Gregg
<brendan.d.gregg at gmail.com> wrote:
> G'Day Serguei,
>
> On Mon, Jun 16, 2014 at 10:45 PM, serguei.spitsyn at oracle.com
> <serguei.spitsyn at oracle.com> wrote:
>>
>> Hi Brendan,
>>
>> We are aware of these issues and work with the Solaris team to fix them in
>> JDK 9.
>> One is the frame pointer is used by the server compiler as a general
>> purpose register on intel.
>> Another is about the virtual (or inlined) frames.
>>
>> There are a couple of related bugs:
>>    https://bugs.openjdk.java.net/browse/JDK-6617153
>>    https://bugs.openjdk.java.net/browse/JDK-6276264
>>
>> There can be more issues filed on this.
>
>
> Ah, thanks, it's JDK-6276264.
>
> As Tom Rodriguez said at the time (2005): "The server VM uses the frame
> pointer as an allocatable register and there's no way to turn that off." I
> was really hoping there was a way to turn that off, like
> -fno-omit-frame-pointer.
>
> This also means DTrace jstack() has never worked fully. For the applications
> I tried it on, 50% of stacks were incomplete. Perhaps it wasn't that bad in
> 2005. I've been getting more mileage today from Java profilers.
>
>> Please, note, that the jstack action is not implemented on Linux yet.
>
>
> Linux doesn't have DTrace jstack(), no, but its perf_events does has support
> for loading an auxiliary file of symbols, which can created via a Java agent
> for that purpose (eg, https://github.com/jrudolph/perf-map-agent). But that
> hasn't been working fully for the same reason - incomplete stacks.
>
> Brendan
>
>>
>> Thanks,
>> Serguei
>>
>>
>>
>> On 6/16/14 5:14 PM, Brendan Gregg wrote:
>>
>> Thanks but no, I'm aware of that bug and workarounds (I'm using the
>> LD_AUDIT_64=/usr/lib/dtrace/64/libdtrace_forceload.so workaround, which
>> isn't mentioned in the bug comments, but probably should be). That bug is
>> about missing symbols, but the stacks shown in that bug still go all the way
>> to thread_start. My stacks often don't.
>>
>> For simple programs, the stacks are complete. But something complex (eg,
>> vert.x with event loops), and the stacks are often incomplete, one frame
>> only. Very much like what I see with -fomit-frame-pointer, although this is
>> hotspot, not gcc. Such incomplete stacks are seen using either DTrace or
>> perf_events.
>>
>> It was suggested to me to email the hotspot developers, because this may
>> well be a hotspot optimization they are familiar with. It may also be
>> something really obvious, like that the JVM breaks native stacks due to
>> optimized frames / green threads / etc, and there is absolutely no way
>> around it (no way to disable it). If that's true, it may also mean that the
>> DTrace jstack() action has always had this issue. I'm still reading the
>> source...
>>
>> Brendan
>>
>>
>>
>> On Mon, Jun 16, 2014 at 4:04 AM, Staffan Larsen
>> <staffan.larsen at oracle.com> wrote:
>>>
>>> I think this is the bug you are looking at:
>>> https://bugs.openjdk.java.net/browse/JDK-7187999, but I’ll defer to someone
>>> else to confirm.
>>>
>>> /Staffan
>>>
>>>
>>> On 16 jun 2014, at 12:47, Roland Westrelin <roland.westrelin at oracle.com>
>>> wrote:
>>>
>>> Forwarding to serviceability alias where this question belongs I think.
>>>
>>> Begin forwarded message:
>>>
>>> From: Brendan Gregg <brendan.d.gregg at gmail.com>
>>> Subject: system profilers and incomplete stacks
>>> Date: June 12, 2014 at 7:15:54 PM GMT+2
>>> To: hotspot-compiler-dev at openjdk.java.net
>>>
>>> G'Day,
>>>
>>> Is there a way to run hotspot so that a system profiler (eg, DTrace, or
>>> Linux perf_events) can measure complete stacks? I often get incomplete,
>>> partial stacks, with one or a few frames only. I'm not worried about symbols
>>> right now, what I'd like is to walk stacks all the way down to thread start.
>>>
>>> I've been browsing the hotspot code, but haven't found out how yet. I
>>> suspect it's related to Java optimized frames, and has ditched the frame
>>> pointer. I was looking for an equivalent -fno-omit-frame-pointer option.
>>>
>>> Here's an example:
>>>
>>> # dtrace -n 'profile-99 /execname == "java"/ { @[jstack(100, 8000)] =
>>> count(); }'
>>> [...]
>>>               org/mozilla/javascript/
>>>
>>> ScriptableObject.createSlot(Ljava/lang/String;II)Lorg/mozilla/javascript/ScriptableObject$Slot;*
>>>               0x884acce8200002da
>>>                 1
>>>
>>>               sun/nio/ch/SocketChannelImpl.read(Ljava/nio/ByteBuffer;)I*
>>>               0xffffffff20007f4b
>>>                 1
>>>
>>>
>>> org/mozilla/javascript/ScriptRuntime.newObjectLiteral([Ljava/lang/Object;[Ljava/lang/Object;[ILorg/mozilla/javascript/Context;Lorg/mozilla/javascript/Scriptable;)Lorg/mozilla/javascript/Scriptable;*
>>>               0xa20000041
>>>                 1
>>> [...]
>>>
>>> I see similar incomplete stacks with Linux perf_events. Oracle JDKs from
>>> 6 to 8, and OpenJDK.
>>>
>>> thanks,
>>>
>>> Brendan
>>> --
>>> http://www.brendangregg.com
>>>
>>>
>>>
>>
>>
>>
>> --
>> http://www.brendangregg.com
>>
>>
>
>
>
> --
> http://www.brendangregg.com