Trace object allocation flag proposal

[Tom Rodriguez]

On Dec 15, 2010, at 11:21 AM, Y. S. Ramakrishna wrote:

> 
> 
> On 12/15/10 10:24, Xiaobin Lu wrote:
>> One of the problems JVMTI has is that once it is enabled, it will fire all kinds of events which will cause even more overheads. Correct me if I am wrong here.
> 
> I didn't think JVMTI is "all or nothing" (if i understood yr statement).
> But i will let the serviceability/JVMTI cognoscenti comment.

We recently fixed a bug that created unneeded overhead in compiled code when running with JVMTI agents.  Previously just the existence of a JVMTI agent turned on some options in the compilers that kept extra locals alive but the fix for 6943485 fixed that.  I think the only compiled overhead you should see these days are when you turn on things like can_post_on_exceptions and can_access_locals.

tom

> 
>> DTrace provides obj-alloc probes which is exactly what many people like to have on Linux. And that is exactly what I am proposing here.
> 
> Ah, i see. (Sorry for not reading through yr proposal before talking.)
> But orthogonally to yr proposal, which is narrowly targeted to addressing
> this specific issue related to object allocation,
> why not use Linux system tap or other to mimic as much as possible the
> Dtrace probes provided by the JVM, rather than making this point fix (if
> i understand yr proposal) for just one aspect of what's offered by
> Dtrace?
> 
> Over to the serviceability folk; and out.
> -- ramki
> 
>> Thanks,
>> -Xiaobin
>> On Wed, Dec 15, 2010 at 10:06 AM, Y. S. Ramakrishna <y.s.ramakrishna at oracle.com <mailto:y.s.ramakrishna at oracle.com>> wrote:
>>    Doesn't JVMTI already provide hooks (BCI?) for tracing allocation?
>>    Perhaps that's found to be too obtrusive? Don't Java IDE's
>>    use JVMTI to already deliver this kind of info to developers?
>>    Pardon my naivette re JVMTI capabilities in this regard.
>>    -- ramki
>>    (java-serviceability-tools-illiterate)
>>    On 12/15/10 10:00, Xiaobin Lu wrote:
>>        Thanks for your feedback, David.
>>        Let me try to clarify with some more background information.
>>        One of the problems many application has is the object
>>        allocation problem (Duh ...). Over releases, GC becomes more and
>>        more frequent. It is pretty hard for one single team to find out
>>        where the problem goes wrong. The existing tool such as jhat or
>>        jmap is very hard to work with giant heap dump file. So the flag
>>        I propose is mostly for diagnostic purpose, it won't be enabled
>>        by default in production. Having said so, if we make it as a
>>        manageable flag, we can turn it on when GC goes wild.
>>        Another value this flag could provide is to find out why
>>        sometime OOM occurs. For example, someone who wrote a search
>>        application on top of Lucene framework suddenly found a crash
>>        due to OOM. The stack trace points to Lucene code which is hard
>>        to instrument, so this flag could provide insight to why the
>>        allocation fails. Maybe it is due to a corrupted length value
>>        passed to "new byte[length];" etc.
>>        I like your idea on per-thread basis, however, for a lot of web
>>        application, thread comes and go. It is pretty hard to pin point
>>        on which thread I want to trace the object allocation.
>>        To answer your last question, what I am really interested in
>>        finding out is what type of object allocation makes GC happens
>>        more frequent. Randomly taking snapshot of heap using jmap is
>>        absolutely not an idea way to do so since not only it generates
>>        a giant file which is difficult to work with, also it will pause
>>        the application and cause negative impact to the application
>>        throughput. After I get the type of hot allocation, if it
>>        happens to be an application level object type such as
>>        com.Xyz.search.IndexReader, I can instrument the constructor to
>>        dump the caller stack. People here also suggests it would be
>>        nice if we could dump the allocation stack trace for some
>>        particular hot types.
>>        I could propose the diff for your folks to review.
>>        Thanks,
>>        -Xiaobin
>>        On Tue, Dec 14, 2010 at 11:58 PM, David Holmes
>>        <David.Holmes at oracle.com <mailto:David.Holmes at oracle.com>
>>        <mailto:David.Holmes at oracle.com
>>        <mailto:David.Holmes at oracle.com>>> wrote:
>>           Hi Xiaobin,
>>           The problem with tracing like this is that to be useful the
>>        tracing
>>           must be unobtrusive and be able to handle getting called
>>        millions of
>>           times (which is what will happen in those GC scenarios you
>>           describe). The sheer volume of data generated as you suggest
>>        below
>>           would overwhelm the app and be pretty hard to work with.
>>           Per-thread statistics of particular types (or of objects
>>        larger than
>>           a certain size) might be more useful, with a dump able to be
>>           requested on-demand.
>>           But I think you'd need to be able to combine this with heap dump
>>           info to be useful.
>>           It really depends on exactly what info you want to be able to
>>           deduce: simple number of objects of given types, hot allocation
>>           sites, hot threads, ...
>>           Cheers,
>>           David
>>           Xiaobin Lu said the following on 12/15/10 17:07:
>>               Hi folks,
>>               I would like to propose a way to trace object allocation on
>>               Linux. On Solaris, we have DTrace which is pretty nice.
>>        But on
>>               Linux, it is almost impossible to do so. Correct me if I am
>>               wrong here.
>>               So I am thinking to add a manageable VM flag and let's
>>        call it
>>               TraceObjectAllocation. When enabled, we can output
>>        something like:
>>               thread id: 10     class name: java/lang/reflect/Method                           size: 80 bytes
>>               thread id: 10     class name: [Ljava/lang/Class;                                  size: 16 bytes
>>               thread id: 10     class name: [C                                                                         size: 56 bytes
>>               thread id: 10     class name: java/lang/reflect/Method                          size: 80 bytes
>>               thread id: 10     class name: [Ljava/lang/Class;                                 size: 16 bytes
>>               As you could imagine, this could be very useful to keep
>>        track of
>>               object allocation behavior in the app. Some smart tool
>>        can take
>>               advantage of this to print a histogram (like top 10 hot
>>               allocations) of object allocation. I would like to know your
>>               thoughts and suggestions on this.
>>               I have a detailed proposal on this attached in PDF file.
>>               Thanks,
>>               -Xiaobin