Trace object allocation flag proposal
Tony Guan
guanxiaohua at gmail.com
Wed Dec 15 11:28:50 PST 2010
Hi Xiaobin,
A reminder for the allocation site profiling, you can check JVMTI
sample heapTracker or hprof, this might be what you need for the time
being.
As to the question what type of the objects will cause the most
garbage collections,I think it's equivalent to the question "At what
point the heap is filled up, and what the next allocation request
looks like." I think the stack trace at the GC time is more relevant
to this question than the next allocation type.
If you are concerned about the overhead using JVMTI, I am sure that if
you don't enable a certain event type, the callbacks will not be
called for that event. The overhead is just the similar to DTrace.
Please refer to the following src code in OpenJDK:
inline void post_allocation_notify(KlassHandle klass, oop obj) {
// support low memory notifications (no-op if not enabled)
LowMemoryDetector::detect_low_memory_for_collected_pools();
// support for JVMTI VMObjectAlloc event (no-op if not enabled)
JvmtiExport::vm_object_alloc_event_collector(obj);
if (DTraceAllocProbes) {
// support for Dtrace object alloc event (no-op most of the time)
if (klass() != NULL && klass()->klass_part()->name() != NULL) {
SharedRuntime::dtrace_object_alloc(obj);
}
}
}
Thanks.
Tony (Xiaohua Guan)
>
> 1. Re: Trace object allocation flag proposal (David Holmes)
> 2. hg: jdk7/hotspot-rt/hotspot: 2 new changesets
> (staffan.larsen at oracle.com)
> 3. Re: Trace object allocation flag proposal (Xiaobin Lu)
> 4. Re: Trace object allocation flag proposal (Y. S. Ramakrishna)
> 5. Re: Trace object allocation flag proposal (Xiaobin Lu)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Wed, 15 Dec 2010 17:58:09 +1000
> From: David Holmes <David.Holmes at oracle.com>
> Subject: Re: Trace object allocation flag proposal
> To: Xiaobin Lu <jacklusf at gmail.com>
> Cc: hotspot-runtime-dev at openjdk.java.net
> Message-ID: <4D087511.30902 at oracle.com>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> Hi Xiaobin,
>
> The problem with tracing like this is that to be useful the tracing must
> be unobtrusive and be able to handle getting called millions of times
> (which is what will happen in those GC scenarios you describe). The
> sheer volume of data generated as you suggest below would overwhelm the
> app and be pretty hard to work with.
>
> Per-thread statistics of particular types (or of objects larger than a
> certain size) might be more useful, with a dump able to be requested
> on-demand.
>
> But I think you'd need to be able to combine this with heap dump info to
> be useful.
>
> It really depends on exactly what info you want to be able to deduce:
> simple number of objects of given types, hot allocation sites, hot
> threads, ...
>
> Cheers,
> David
>
> Xiaobin Lu said the following on 12/15/10 17:07:
>> Hi folks,
>>
>> I would like to propose a way to trace object allocation on Linux. On
>> Solaris, we have DTrace which is pretty nice. But on Linux, it is almost
>> impossible to do so. Correct me if I am wrong here.
>>
>> So I am thinking to add a manageable VM flag and let's call it
>> TraceObjectAllocation. When enabled, we can output something like:
>>
>> thread id: 10 class name: java/lang/reflect/Method size:
>> 80 bytes
>> thread id: 10 class name: [Ljava/lang/Class;
>> size: 16 bytes
>> thread id: 10 class name: [C
>> size: 56 bytes
>> thread id: 10 class name: java/lang/reflect/Method size:
>> 80 bytes
>> thread id: 10 class name: [Ljava/lang/Class; size:
>> 16 bytes
>>
>> As you could imagine, this could be very useful to keep track of object
>> allocation behavior in the app. Some smart tool can take advantage of
>> this to print a histogram (like top 10 hot allocations) of object
>> allocation. I would like to know your thoughts and suggestions on this.
>>
>> I have a detailed proposal on this attached in PDF file.
>>
>> Thanks,
>>
>> -Xiaobin
> Message: 3
> Date: Wed, 15 Dec 2010 10:00:50 -0800
> From: Xiaobin Lu <jacklusf at gmail.com>
> Subject: Re: Trace object allocation flag proposal
> To: David Holmes <David.Holmes at oracle.com>
> Cc: hotspot-runtime-dev at openjdk.java.net
> Message-ID:
> <AANLkTik4uxHXy-p1eLWDA46yXuenEcdRJV90z4dni6MR at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> Thanks for your feedback, David.
>
> Let me try to clarify with some more background information.
>
> One of the problems many application has is the object allocation problem
> (Duh ...). Over releases, GC becomes more and more frequent. It is pretty
> hard for one single team to find out where the problem goes wrong. The
> existing tool such as jhat or jmap is very hard to work with giant heap dump
> file. So the flag I propose is mostly for diagnostic purpose, it won't be
> enabled by default in production. Having said so, if we make it as a
> manageable flag, we can turn it on when GC goes wild.
>
> Another value this flag could provide is to find out why sometime OOM
> occurs. For example, someone who wrote a search application on top of Lucene
> framework suddenly found a crash due to OOM. The stack trace points to
> Lucene code which is hard to instrument, so this flag could provide insight
> to why the allocation fails. Maybe it is due to a corrupted length value
> passed to "new byte[length];" etc.
>
> I like your idea on per-thread basis, however, for a lot of web application,
> thread comes and go. It is pretty hard to pin point on which thread I want
> to trace the object allocation.
>
> To answer your last question, what I am really interested in finding out is
> what type of object allocation makes GC happens more frequent. Randomly
> taking snapshot of heap using jmap is absolutely not an idea way to do so
> since not only it generates a giant file which is difficult to work with,
> also it will pause the application and cause negative impact to the
> application throughput. After I get the type of hot allocation, if it
> happens to be an application level object type such as
> com.Xyz.search.IndexReader, I can instrument the constructor to dump the
> caller stack. People here also suggests it would be nice if we could dump
> the allocation stack trace for some particular hot types.
>
> I could propose the diff for your folks to review.
>
> Thanks,
>
> -Xiaobin
>
> On Tue, Dec 14, 2010 at 11:58 PM, David Holmes <David.Holmes at oracle.com>wrote:
>
>> Hi Xiaobin,
>>
>> The problem with tracing like this is that to be useful the tracing must be
>> unobtrusive and be able to handle getting called millions of times (which is
>> what will happen in those GC scenarios you describe). The sheer volume of
>> data generated as you suggest below would overwhelm the app and be pretty
>> hard to work with.
>>
>> Per-thread statistics of particular types (or of objects larger than a
>> certain size) might be more useful, with a dump able to be requested
>> on-demand.
>>
>> But I think you'd need to be able to combine this with heap dump info to be
>> useful.
>>
>> It really depends on exactly what info you want to be able to deduce:
>> simple number of objects of given types, hot allocation sites, hot threads,
>> ...
>>
>> Cheers,
>> David
>>
>> Xiaobin Lu said the following on 12/15/10 17:07:
>>
>> Hi folks,
>>>
>>> I would like to propose a way to trace object allocation on Linux. On
>>> Solaris, we have DTrace which is pretty nice. But on Linux, it is almost
>>> impossible to do so. Correct me if I am wrong here.
>>>
>>> So I am thinking to add a manageable VM flag and let's call it
>>> TraceObjectAllocation. When enabled, we can output something like:
>>>
>>> thread id: 10 class name: java/lang/reflect/Method size:
>>> 80 bytes
>>> thread id: 10 class name: [Ljava/lang/Class; size:
>>> 16 bytes
>>> thread id: 10 class name: [C
>>> size: 56 bytes
>>> thread id: 10 class name: java/lang/reflect/Method size: 80
>>> bytes
>>> thread id: 10 class name: [Ljava/lang/Class; size:
>>> 16 bytes
>>>
>>> As you could imagine, this could be very useful to keep track of object
>>> allocation behavior in the app. Some smart tool can take advantage of this
>>> to print a histogram (like top 10 hot allocations) of object allocation. I
>>> would like to know your thoughts and suggestions on this.
>>>
>>> I have a detailed proposal on this attached in PDF file.
>>>
>>> Thanks,
>>>
>>> -Xiaobin
>>>
>>>
>>>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/attachments/20101215/8e050e58/attachment-0001.html
>
> ------------------------------
>
> Message: 4
> Date: Wed, 15 Dec 2010 10:06:00 -0800
> From: "Y. S. Ramakrishna" <y.s.ramakrishna at oracle.com>
> Subject: Re: Trace object allocation flag proposal
> To: Xiaobin Lu <jacklusf at gmail.com>
> Cc: hotspot-runtime-dev at openjdk.java.net
> Message-ID: <4D090388.1040704 at oracle.com>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> Doesn't JVMTI already provide hooks (BCI?) for tracing allocation?
> Perhaps that's found to be too obtrusive? Don't Java IDE's
> use JVMTI to already deliver this kind of info to developers?
>
> Pardon my naivette re JVMTI capabilities in this regard.
> -- ramki
> (java-serviceability-tools-illiterate)
>
> On 12/15/10 10:00, Xiaobin Lu wrote:
>> Thanks for your feedback, David.
>>
>> Let me try to clarify with some more background information.
>>
>> One of the problems many application has is the object allocation
>> problem (Duh ...). Over releases, GC becomes more and more frequent. It
>> is pretty hard for one single team to find out where the problem goes
>> wrong. The existing tool such as jhat or jmap is very hard to work with
>> giant heap dump file. So the flag I propose is mostly for diagnostic
>> purpose, it won't be enabled by default in production. Having said so,
>> if we make it as a manageable flag, we can turn it on when GC goes wild.
>>
>> Another value this flag could provide is to find out why sometime OOM
>> occurs. For example, someone who wrote a search application on top of
>> Lucene framework suddenly found a crash due to OOM. The stack trace
>> points to Lucene code which is hard to instrument, so this flag could
>> provide insight to why the allocation fails. Maybe it is due to a
>> corrupted length value passed to "new byte[length];" etc.
>>
>> I like your idea on per-thread basis, however, for a lot of web
>> application, thread comes and go. It is pretty hard to pin point on
>> which thread I want to trace the object allocation.
>>
>> To answer your last question, what I am really interested in finding out
>> is what type of object allocation makes GC happens more frequent.
>> Randomly taking snapshot of heap using jmap is absolutely not an idea
>> way to do so since not only it generates a giant file which is difficult
>> to work with, also it will pause the application and cause negative
>> impact to the application throughput. After I get the type of hot
>> allocation, if it happens to be an application level object type such as
>> com.Xyz.search.IndexReader, I can instrument the constructor to dump the
>> caller stack. People here also suggests it would be nice if we could
>> dump the allocation stack trace for some particular hot types.
>>
>> I could propose the diff for your folks to review.
>>
>> Thanks,
>>
>> -Xiaobin
>>
>> On Tue, Dec 14, 2010 at 11:58 PM, David Holmes <David.Holmes at oracle.com
>> <mailto:David.Holmes at oracle.com>> wrote:
>>
>> Hi Xiaobin,
>>
>> The problem with tracing like this is that to be useful the tracing
>> must be unobtrusive and be able to handle getting called millions of
>> times (which is what will happen in those GC scenarios you
>> describe). The sheer volume of data generated as you suggest below
>> would overwhelm the app and be pretty hard to work with.
>>
>> Per-thread statistics of particular types (or of objects larger than
>> a certain size) might be more useful, with a dump able to be
>> requested on-demand.
>>
>> But I think you'd need to be able to combine this with heap dump
>> info to be useful.
>>
>> It really depends on exactly what info you want to be able to
>> deduce: simple number of objects of given types, hot allocation
>> sites, hot threads, ...
>>
>> Cheers,
>> David
>>
>> Xiaobin Lu said the following on 12/15/10 17:07:
>>
>> Hi folks,
>>
>> I would like to propose a way to trace object allocation on
>> Linux. On Solaris, we have DTrace which is pretty nice. But on
>> Linux, it is almost impossible to do so. Correct me if I am
>> wrong here.
>>
>> So I am thinking to add a manageable VM flag and let's call it
>> TraceObjectAllocation. When enabled, we can output something like:
>>
>> thread id: 10 class name: java/lang/reflect/Method
>> size: 80 bytes
>> thread id: 10 class name: [Ljava/lang/Class;
>> size: 16 bytes
>> thread id: 10 class name: [C
>> size: 56 bytes
>> thread id: 10 class name: java/lang/reflect/Method
>> size: 80 bytes
>> thread id: 10 class name: [Ljava/lang/Class;
>> size: 16 bytes
>>
>> As you could imagine, this could be very useful to keep track of
>> object allocation behavior in the app. Some smart tool can take
>> advantage of this to print a histogram (like top 10 hot
>> allocations) of object allocation. I would like to know your
>> thoughts and suggestions on this.
>>
>> I have a detailed proposal on this attached in PDF file.
>>
>> Thanks,
>>
>> -Xiaobin
>>
>>
>>
>
>
> ------------------------------
>
> Message: 5
> Date: Wed, 15 Dec 2010 10:24:40 -0800
> From: Xiaobin Lu <jacklusf at gmail.com>
> Subject: Re: Trace object allocation flag proposal
> To: Y.S.Ramakrishna at oracle.com
> Cc: hotspot-runtime-dev at openjdk.java.net
> Message-ID:
> <AANLkTimzUiMNkUFfS2jaC0a+zFezqZA6UZjrPCK2-Lq9 at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> One of the problems JVMTI has is that once it is enabled, it will fire all
> kinds of events which will cause even more overheads. Correct me if I am
> wrong here.
>
> DTrace provides obj-alloc probes which is exactly what many people like to
> have on Linux. And that is exactly what I am proposing here.
>
> Thanks,
>
> -Xiaobin
>
> On Wed, Dec 15, 2010 at 10:06 AM, Y. S. Ramakrishna <
> y.s.ramakrishna at oracle.com> wrote:
>
>> Doesn't JVMTI already provide hooks (BCI?) for tracing allocation?
>> Perhaps that's found to be too obtrusive? Don't Java IDE's
>> use JVMTI to already deliver this kind of info to developers?
>>
>> Pardon my naivette re JVMTI capabilities in this regard.
>> -- ramki
>> (java-serviceability-tools-illiterate)
>>
>>
>> On 12/15/10 10:00, Xiaobin Lu wrote:
>>
>>> Thanks for your feedback, David.
>>>
>>> Let me try to clarify with some more background information.
>>>
>>> One of the problems many application has is the object allocation problem
>>> (Duh ...). Over releases, GC becomes more and more frequent. It is pretty
>>> hard for one single team to find out where the problem goes wrong. The
>>> existing tool such as jhat or jmap is very hard to work with giant heap dump
>>> file. So the flag I propose is mostly for diagnostic purpose, it won't be
>>> enabled by default in production. Having said so, if we make it as a
>>> manageable flag, we can turn it on when GC goes wild.
>>>
>>> Another value this flag could provide is to find out why sometime OOM
>>> occurs. For example, someone who wrote a search application on top of Lucene
>>> framework suddenly found a crash due to OOM. The stack trace points to
>>> Lucene code which is hard to instrument, so this flag could provide insight
>>> to why the allocation fails. Maybe it is due to a corrupted length value
>>> passed to "new byte[length];" etc.
>>>
>>> I like your idea on per-thread basis, however, for a lot of web
>>> application, thread comes and go. It is pretty hard to pin point on which
>>> thread I want to trace the object allocation.
>>>
>>> To answer your last question, what I am really interested in finding out
>>> is what type of object allocation makes GC happens more frequent. Randomly
>>> taking snapshot of heap using jmap is absolutely not an idea way to do so
>>> since not only it generates a giant file which is difficult to work with,
>>> also it will pause the application and cause negative impact to the
>>> application throughput. After I get the type of hot allocation, if it
>>> happens to be an application level object type such as
>>> com.Xyz.search.IndexReader, I can instrument the constructor to dump the
>>> caller stack. People here also suggests it would be nice if we could dump
>>> the allocation stack trace for some particular hot types.
>>>
>>> I could propose the diff for your folks to review.
>>>
>>> Thanks,
>>>
>>> -Xiaobin
>>>
>>> On Tue, Dec 14, 2010 at 11:58 PM, David Holmes <David.Holmes at oracle.com<mailto:
>>> David.Holmes at oracle.com>> wrote:
>>>
>>> Hi Xiaobin,
>>>
>>> The problem with tracing like this is that to be useful the tracing
>>> must be unobtrusive and be able to handle getting called millions of
>>> times (which is what will happen in those GC scenarios you
>>> describe). The sheer volume of data generated as you suggest below
>>> would overwhelm the app and be pretty hard to work with.
>>>
>>> Per-thread statistics of particular types (or of objects larger than
>>> a certain size) might be more useful, with a dump able to be
>>> requested on-demand.
>>>
>>> But I think you'd need to be able to combine this with heap dump
>>> info to be useful.
>>>
>>> It really depends on exactly what info you want to be able to
>>> deduce: simple number of objects of given types, hot allocation
>>> sites, hot threads, ...
>>>
>>> Cheers,
>>> David
>>>
>>> Xiaobin Lu said the following on 12/15/10 17:07:
>>>
>>> Hi folks,
>>>
>>> I would like to propose a way to trace object allocation on
>>> Linux. On Solaris, we have DTrace which is pretty nice. But on
>>> Linux, it is almost impossible to do so. Correct me if I am
>>> wrong here.
>>>
>>> So I am thinking to add a manageable VM flag and let's call it
>>> TraceObjectAllocation. When enabled, we can output something like:
>>>
>>> thread id: 10 class name: java/lang/reflect/Method
>>> size: 80 bytes
>>> thread id: 10 class name: [Ljava/lang/Class;
>>> size: 16 bytes
>>> thread id: 10 class name: [C
>>> size: 56 bytes
>>> thread id: 10 class name: java/lang/reflect/Method
>>> size: 80 bytes
>>> thread id: 10 class name: [Ljava/lang/Class;
>>> size: 16 bytes
>>>
>>> As you could imagine, this could be very useful to keep track of
>>> object allocation behavior in the app. Some smart tool can take
>>> advantage of this to print a histogram (like top 10 hot
>>> allocations) of object allocation. I would like to know your
>>> thoughts and suggestions on this.
>>>
>>> I have a detailed proposal on this attached in PDF file.
>>>
>>> Thanks,
>>>
>>> -Xiaobin
>>>
>>>
>>>
>>>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/attachments/20101215/d9df125a/attachment.html
>
> End of hotspot-runtime-dev Digest, Vol 42, Issue 18
> ***************************************************
>
More information about the hotspot-runtime-dev
mailing list