Object allocation tracing / profilers

[Jeremy Manson]

We have a patch for tracking method calls, too.  It doesn't do callbacks on
method invocation, but it does provide a list of method invocations.  You
can already do callbacks on method entry and exit with JVMTI, but it is
expensive, which is why we built that.  If there's interest, we can provide
it.

As far as the allocation tracking patch - you will have to feel comfortable
patching and building your own JDK, and writing a blob of native code
loaded by the JVM that will dlsym into libjvm.so to fish out the method to
register your callback.  Does that sound like something you feel
comfortable doing?

(I'm about to send the forward-port patch out to get code-reviewed for JDK8
before leaving for the first half of vacation, so if that doesn't scare
you, then I can send it the week after next, or so.)

Jeremy


On Fri, Feb 7, 2014 at 1:35 PM, Robert Stupp <snazy at snazy.de> wrote:

> Jeremy, if you have a patch for JDK8 or 9 and some short instructions, i'd
> like to try it (OSX or Linux) :)
> I think it's ok if there's no way to do JDK callbacks (as long as the
> class, array length, thread and maybe (piece of) the its stack trace is
> available).
> But for now I only need the class and array length.
>
> Regarding the fact, that profiler output regarding allocated objects is
> wrong (compared to non-instrumented code) there should be a solution. I'm
> afraid that many others optimize their code by considering the output of a
> profiler as "the truth". Since all numbers (either object allocations or
> method runtimes) are based on instrumented code - they must be considered
> an at least airy estimate with a probability to be completely different
> compared to non-profiled/instrumented code.
>
> Maybe it is possible to extend the official JVMTI API to provide a new
> profiling API. IMO it would be faster and less intrusive and therefore
> better comparable against a real production system.
> That API would perform callbacks with information about the object, its
> class, array length, thread and stack trace (up to a configurable length,
> possibly empty). It should be considered that only a specific set of
> threads, set of classes is of interest. Such an API would need to provide a
> part to track method calls with similar filters - it would not need to do
> any callbacks for inlined methods. Stack trace information could also
> include information which method is compiled. The events could be placed by
> application threads in some queue and a separate thread could poll for
> events and perform the callbacks - means that the real profiling code is
> independent from the application code.
>
> Instrumentation itself could still be used for more complicated stuff like
> hunting for handle leaks (file, jdbc, etc).
>
> These are only some thoughts.
>
> Robert
>
> Am 07.02.2014 um 02:13 schrieb Jeremy Manson <jeremymanson at google.com>:
>
> You aren't doing anything wrong.  More detail:
>
> - Anything bytecode based (like my instrumenter, which you tried) isn't
> going to let you know what has been escape analyzed away.  You can only do
> bytecode rewriting before escape analysis runs, and before the escape
> analysis runs, there is no way to find out what will be escape analyzed
> away.
>
> - A bytecode-based instrumenter will also not pick up allocations that
> occur via JNI, which is the problem that VMObjectAlloc solves.
>
> - If you happen to be on a platform that supports it, and you've compiled
> the JDK correctly, you can use dtrace to capture Java allocations.
>
> - Because of its limitations and performance issues, we've actually
> stopped doing bytecode-based instrumentation at Google in favor of
> instrumenting the VM directly.  We now use JVMTI-style callbacks whenever
> an allocation happens.  If people are interested, I can provide the patch
> (in fact, I forward ported it to JDK8 yesterday!), but it is rather user
> unfriendly, so you have to be rather ambitious to use it.  On the plus
> side, it can take into account escape analyzed objects (right now, it
> doesn't instrument them, but this could conceivably be changed).
>
> - On the down side, if you take this approach, you can't call back into
> Java from the JVM callbacks.  This is because calling back into Java may
> require the JVM to come to a safepoint, which you can't do in the middle of
> allocation (which is where the callback would occur).
>
> Jeremy
>
>
>
> On Thu, Feb 6, 2014 at 2:36 PM, Robert Stupp <snazy at snazy.de> wrote:
>
>> Hi,
>>
>> I'm trying to trace/count object allocations - especially regarding the
>> ObjectIn/OutputStream stuff.
>>
>> I know two API points: JVMTI and instrumentation. JVMTI fires an event
>> for nearly everything except allocations from bytecode - instrumentation
>> allows to transform bytecode (e.g. to issue a callback on object
>> allocation).
>>
>> But I am not very satisfied - maybe I oversee something...
>> What I need is the number of "real" allocations. I do not want to count
>> allocations that Hotspot would normally eliminate - for example Hotspot
>> seems to eliminate instances of ArrayList$Itr under some circumstances.
>>
>> I played around with this instrumentation code:
>> https://code.google.com/p/java-allocation-instrumenter/
>> But this one and any other profiler (JProfiler) I tried always gave me
>> the theoretical number of instances of ArrayList$Itr. This reason seems to
>> be as follows:
>> With instrumentation/transformation constructors get a piece of code
>> "injected" which calls a "profile object allocation method". This method
>> gets the new object instance, which is used to get the Class object and to
>> inquire the shallow object size on the heap - and it gets the current
>> thread's stack trace.
>>
>> My suggestion is, that Hotspot cannot eliminate object allocations (for
>> example of ArrayList$Itr) because the profiler bytecode needs that object
>> instance and does other things, that are "strange" to Hotspot.
>>
>> I hoped that I can get callbacks using JVMTI but due to the restrictions
>> mentioned here (
>> http://docs.oracle.com/javase/7/docs/platform/jvmti/jvmti.html#VMObjectAlloc)
>> there are no callbacks for allocations caused by bytecode.
>> VMObjectAlloc events occur during the start and end of the application -
>> but not a single event is fired while the application is running (I've
>> attached my JVMTI code).
>>
>> Am I doing anything wrong? Is there a possibility to count only the
>> "real" instances? If not, wouldn't it be nice to have such a functionality
>> in JVMTI or a callback functionality for
>> http://download.java.net/jdk8/docs/api/java/lang/instrument/Instrumentation.html
>> ?
>> My fear is that all profiler results that all developers see are
>> inaccurate because the code injected by profilers influences Hotspot too
>> much.
>>
>> -
>> Robert
>>
>>
>
>