Trace object allocation flag proposal

Dmitry Samersoff Dmitry.Samersoff at oracle.com
Wed Dec 15 11:03:08 PST 2010


Xiaobin,

We have three independent diagnostic systems in hotspot:

1. JVMTI
2. DTrace
3. Conventional logging

It's "not good" from long term perspective and I think we should unify it.

My personal opinion - we should have only JVMTI and make it as 
lightweight as possible.

-Dmitry



On 2010-12-15 21:24, Xiaobin Lu wrote:
> One of the problems JVMTI has is that once it is enabled, it will fire
> all kinds of events which will cause even more overheads. Correct me if
> I am wrong here.
>
> DTrace provides obj-alloc probes which is exactly what many people like
> to have on Linux. And that is exactly what I am proposing here.
>
> Thanks,
>
> -Xiaobin
>
> On Wed, Dec 15, 2010 at 10:06 AM, Y. S. Ramakrishna
> <y.s.ramakrishna at oracle.com <mailto:y.s.ramakrishna at oracle.com>> wrote:
>
>     Doesn't JVMTI already provide hooks (BCI?) for tracing allocation?
>     Perhaps that's found to be too obtrusive? Don't Java IDE's
>     use JVMTI to already deliver this kind of info to developers?
>
>     Pardon my naivette re JVMTI capabilities in this regard.
>     -- ramki
>     (java-serviceability-tools-illiterate)
>
>
>     On 12/15/10 10:00, Xiaobin Lu wrote:
>
>         Thanks for your feedback, David.
>
>         Let me try to clarify with some more background information.
>
>         One of the problems many application has is the object
>         allocation problem (Duh ...). Over releases, GC becomes more and
>         more frequent. It is pretty hard for one single team to find out
>         where the problem goes wrong. The existing tool such as jhat or
>         jmap is very hard to work with giant heap dump file. So the flag
>         I propose is mostly for diagnostic purpose, it won't be enabled
>         by default in production. Having said so, if we make it as a
>         manageable flag, we can turn it on when GC goes wild.
>
>         Another value this flag could provide is to find out why
>         sometime OOM occurs. For example, someone who wrote a search
>         application on top of Lucene framework suddenly found a crash
>         due to OOM. The stack trace points to Lucene code which is hard
>         to instrument, so this flag could provide insight to why the
>         allocation fails. Maybe it is due to a corrupted length value
>         passed to "new byte[length];" etc.
>
>         I like your idea on per-thread basis, however, for a lot of web
>         application, thread comes and go. It is pretty hard to pin point
>         on which thread I want to trace the object allocation.
>
>         To answer your last question, what I am really interested in
>         finding out is what type of object allocation makes GC happens
>         more frequent. Randomly taking snapshot of heap using jmap is
>         absolutely not an idea way to do so since not only it generates
>         a giant file which is difficult to work with, also it will pause
>         the application and cause negative impact to the application
>         throughput. After I get the type of hot allocation, if it
>         happens to be an application level object type such as
>         com.Xyz.search.IndexReader, I can instrument the constructor to
>         dump the caller stack. People here also suggests it would be
>         nice if we could dump the allocation stack trace for some
>         particular hot types.
>
>         I could propose the diff for your folks to review.
>
>         Thanks,
>
>         -Xiaobin
>
>         On Tue, Dec 14, 2010 at 11:58 PM, David Holmes
>         <David.Holmes at oracle.com <mailto:David.Holmes at oracle.com>
>         <mailto:David.Holmes at oracle.com
>         <mailto:David.Holmes at oracle.com>>> wrote:
>
>             Hi Xiaobin,
>
>             The problem with tracing like this is that to be useful the
>         tracing
>             must be unobtrusive and be able to handle getting called
>         millions of
>             times (which is what will happen in those GC scenarios you
>             describe). The sheer volume of data generated as you suggest
>         below
>             would overwhelm the app and be pretty hard to work with.
>
>             Per-thread statistics of particular types (or of objects
>         larger than
>             a certain size) might be more useful, with a dump able to be
>             requested on-demand.
>
>             But I think you'd need to be able to combine this with heap dump
>             info to be useful.
>
>             It really depends on exactly what info you want to be able to
>             deduce: simple number of objects of given types, hot allocation
>             sites, hot threads, ...
>
>             Cheers,
>             David
>
>             Xiaobin Lu said the following on 12/15/10 17:07:
>
>                 Hi folks,
>
>                 I would like to propose a way to trace object allocation on
>                 Linux. On Solaris, we have DTrace which is pretty nice.
>         But on
>                 Linux, it is almost impossible to do so. Correct me if I am
>                 wrong here.
>
>                 So I am thinking to add a manageable VM flag and let's
>         call it
>                 TraceObjectAllocation. When enabled, we can output
>         something like:
>
>                 thread id: 10     class name: java/lang/reflect/Method
>                           size: 80 bytes
>                 thread id: 10     class name: [Ljava/lang/Class;
>                            size: 16 bytes
>                 thread id: 10     class name: [C
>                                                   size: 56 bytes
>                 thread id: 10     class name: java/lang/reflect/Method
>                          size: 80 bytes
>                 thread id: 10     class name: [Ljava/lang/Class;
>                           size: 16 bytes
>
>                 As you could imagine, this could be very useful to keep
>         track of
>                 object allocation behavior in the app. Some smart tool
>         can take
>                 advantage of this to print a histogram (like top 10 hot
>                 allocations) of object allocation. I would like to know your
>                 thoughts and suggestions on this.
>
>                 I have a detailed proposal on this attached in PDF file.
>
>                 Thanks,
>
>                 -Xiaobin
>
>
>
>


-- 
Dmitry Samersoff
J2SE Sustaining team, SPB04
* Give Rabbit time and he'll always get the answer ...


More information about the hotspot-runtime-dev mailing list