From bccheng at google.com  Fri Oct 26 01:04:15 2007
From: bccheng at google.com (Ben Cheng)
Date: Thu, 25 Oct 2007 18:04:15 -0700
Subject: gc log analysis tools
Message-ID: <bcf6f6d40710251804i5c7aed9duab74e246a23a7088@mail.gmail.com>

Hi,

I was trying to get a sense of total GC time spent in a Java app using
1.6JDK. The gc logs (w/ and wo/ parallel gc) seem to break the ancient
tool
gc_analyze.pl found on the web.

Is there an update to date tool to produce the desired information for new
JDKs?

Thanks in advance,
-Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/hotspot-gc-dev/attachments/20071025/bcec99e2/attachment.htm>

From neojia at gmail.com  Fri Oct 26 01:07:47 2007
From: neojia at gmail.com (Neo Jia)
Date: Thu, 25 Oct 2007 18:07:47 -0700
Subject: gc log analysis tools
In-Reply-To: <bcf6f6d40710251804i5c7aed9duab74e246a23a7088@mail.gmail.com>
References: <bcf6f6d40710251804i5c7aed9duab74e246a23a7088@mail.gmail.com>
Message-ID: <5d649bdb0710251807y2d446ab2ye7dba0e5b0025a25@mail.gmail.com>

Ben,

What is your opion? I had one but not sure if it would work for you or not (

I have not worked on JDK for a while ...

Thanks,
Neo

On 10/25/07, Ben Cheng <bccheng at google.com> wrote:
> Hi,
>
> I was trying to get a sense of total GC time spent in a Java app using 1.6
> JDK. The gc logs (w/ and wo/ parallel gc) seem to break the ancient tool
> gc_analyze.pl found on the web.
>
> Is there an update to date tool to produce the desired information for new
> JDKs?
>
> Thanks in advance,
> -Ben
>


-- 
I would remember that if researchers were not ambitious
probably today we haven't the technology we are using!


From bccheng at google.com  Fri Oct 26 05:00:01 2007
From: bccheng at google.com (Ben Cheng)
Date: Thu, 25 Oct 2007 22:00:01 -0700
Subject: gc log analysis tools
In-Reply-To: <5d649bdb0710251807y2d446ab2ye7dba0e5b0025a25@mail.gmail.com>
References: <bcf6f6d40710251804i5c7aed9duab74e246a23a7088@mail.gmail.com>
	<5d649bdb0710251807y2d446ab2ye7dba0e5b0025a25@mail.gmail.com>
Message-ID: <bcf6f6d40710252200m53b38eeay6bbd2969d5e4ee75@mail.gmail.com>

The reason I am interested in this information is because I have a benchmark
whose score is X using 32-bit VM and 2G of heap. When I switched to 64-bit
build of the same VM, the score dropped to 0.77X with the same amount of
heap. If I increase the heap size to 3.5G (on a machine with 4GB of RAM),
the score went to to 1.40X.

What I need is pretty basic - it sums up all the gc time from the verbose
log during a benchmark run, as I want to verify if the change in gc time
correlates to the change in performance. I can write a similar script to do
that, but I just want to check to see if there is a working tool out there
already. Apparently the verbose gc log format changed from 1.5 and beyond
and the old script is not compatible.

Thanks,
-Ben

On 10/25/07, Neo Jia <neojia at gmail.com> wrote:
>
> Ben,
>
> What is your opion? I had one but not sure if it would work for you or not
> (
>
> I have not worked on JDK for a while ...
>
> Thanks,
> Neo
>
> On 10/25/07, Ben Cheng <bccheng at google.com> wrote:
> > Hi,
> >
> > I was trying to get a sense of total GC time spent in a Java app using
> 1.6
> > JDK. The gc logs (w/ and wo/ parallel gc) seem to break the ancient tool
> > gc_analyze.pl found on the web.
> >
> > Is there an update to date tool to produce the desired information for
> new
> > JDKs?
> >
> > Thanks in advance,
> > -Ben
> >
>
>
> --
> I would remember that if researchers were not ambitious
> probably today we haven't the technology we are using!
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/hotspot-gc-dev/attachments/20071025/16194f56/attachment.htm>

From neojia at gmail.com  Fri Oct 26 06:13:40 2007
From: neojia at gmail.com (Neo Jia)
Date: Thu, 25 Oct 2007 23:13:40 -0700
Subject: gc log analysis tools
In-Reply-To: <bcf6f6d40710252200m53b38eeay6bbd2969d5e4ee75@mail.gmail.com>
References: <bcf6f6d40710251804i5c7aed9duab74e246a23a7088@mail.gmail.com>
	<5d649bdb0710251807y2d446ab2ye7dba0e5b0025a25@mail.gmail.com>
	<bcf6f6d40710252200m53b38eeay6bbd2969d5e4ee75@mail.gmail.com>
Message-ID: <5d649bdb0710252313r7ee83969t207e41a6c499905@mail.gmail.com>

Ben,

Which one should be better? lower score or higher score? I assume it
should be higher based on what you said here.

And, what is the relation between score and time? Will the execution
time longer with a low score?

And how many minor GC and full GC in each of your execution? Just grep
and wc them.

For 32-bit and 64-bit machine, at least the default heap layout is
different. Are you trying to use same amount of nursery space and
mature space.

And for 64-bit machine, the Jitter is much more better than 32-bit
machine, a lot of optimization is added.

So ...

Thanks,
Neo

On 10/25/07, Ben Cheng <bccheng at google.com> wrote:
> The reason I am interested in this information is because I have a benchmark
> whose score is X using 32-bit VM and 2G of heap. When I switched to 64-bit
> build of the same VM, the score dropped to 0.77X with the same amount of
> heap. If I increase the heap size to 3.5G (on a machine with 4GB of RAM),
> the score went to to 1.40X.
>
> What I need is pretty basic - it sums up all the gc time from the verbose
> log during a benchmark run, as I want to verify if the change in gc time
> correlates to the change in performance. I can write a similar script to do
> that, but I just want to check to see if there is a working tool out there
> already. Apparently the verbose gc log format changed from 1.5 and beyond
> and the old script is not compatible.
>
> Thanks,
> -Ben
>
>
> On 10/25/07, Neo Jia <neojia at gmail.com> wrote:
> > Ben,
> >
> > What is your opion? I had one but not sure if it would work for you or not
> (
> >
> > I have not worked on JDK for a while ...
> >
> > Thanks,
> > Neo
> >
> > On 10/25/07, Ben Cheng < bccheng at google.com> wrote:
> > > Hi,
> > >
> > > I was trying to get a sense of total GC time spent in a Java app using
> 1.6
> > > JDK. The gc logs (w/ and wo/ parallel gc) seem to break the ancient tool
> > > gc_analyze.pl found on the web.
> > >
> > > Is there an update to date tool to produce the desired information for
> new
> > > JDKs?
> > >
> > > Thanks in advance,
> > > -Ben
> > >
> >
> >
> > --
> > I would remember that if researchers were not ambitious
> > probably today we haven't the technology we are using!
> >
>
>


-- 
I would remember that if researchers were not ambitious
probably today we haven't the technology we are using!


From bccheng at google.com  Fri Oct 26 06:22:54 2007
From: bccheng at google.com (Ben Cheng)
Date: Thu, 25 Oct 2007 23:22:54 -0700
Subject: gc log analysis tools
In-Reply-To: <5d649bdb0710252313r7ee83969t207e41a6c499905@mail.gmail.com>
References: <bcf6f6d40710251804i5c7aed9duab74e246a23a7088@mail.gmail.com>
	<5d649bdb0710251807y2d446ab2ye7dba0e5b0025a25@mail.gmail.com>
	<bcf6f6d40710252200m53b38eeay6bbd2969d5e4ee75@mail.gmail.com>
	<5d649bdb0710252313r7ee83969t207e41a6c499905@mail.gmail.com>
Message-ID: <bcf6f6d40710252322r7e4a3fdcq9c67a2a37dcb6f38@mail.gmail.com>

On 10/25/07, Neo Jia <neojia at gmail.com> wrote:
>
> Ben,
>
> Which one should be better? lower score or higher score? I assume it
> should be higher based on what you said here.


Oops, I forgot to clarify that. Yes, the higher the better.

And, what is the relation between score and time? Will the execution
> time longer with a low score?



The benchmark is doing fixed workload, so the score is simply calculated as
work/s.

And how many minor GC and full GC in each of your execution? Just grep
> and wc them.



Almost all of them are minor GCs.

For 32-bit and 64-bit machine, at least the default heap layout is
> different. Are you trying to use same amount of nursery space and
> mature space.



I think I am just using the default ration. I only specify -Xmx and -Xms to
the same amount without using -Xmn.


> And for 64-bit machine, the Jitter is much more better than 32-bit
> machine, a lot of optimization is added.


I believe so, and that's why the 64-bit VM eventually provides 40% speedup
with larger heap is given. I think under the same amount of heap space there
are more GC activities in 64-bit VM due to larger object sizes. I will
grep/wc the total gc time then.

Thanks,
-Ben

So ...
>
> Thanks,
> Neo
>
> On 10/25/07, Ben Cheng <bccheng at google.com> wrote:
> > The reason I am interested in this information is because I have a
> benchmark
> > whose score is X using 32-bit VM and 2G of heap. When I switched to
> 64-bit
> > build of the same VM, the score dropped to 0.77X with the same amount of
> > heap. If I increase the heap size to 3.5G (on a machine with 4GB of
> RAM),
> > the score went to to 1.40X.
> >
> > What I need is pretty basic - it sums up all the gc time from the
> verbose
> > log during a benchmark run, as I want to verify if the change in gc time
> > correlates to the change in performance. I can write a similar script to
> do
> > that, but I just want to check to see if there is a working tool out
> there
> > already. Apparently the verbose gc log format changed from 1.5 and
> beyond
> > and the old script is not compatible.
> >
> > Thanks,
> > -Ben
> >
> >
> > On 10/25/07, Neo Jia <neojia at gmail.com> wrote:
> > > Ben,
> > >
> > > What is your opion? I had one but not sure if it would work for you or
> not
> > (
> > >
> > > I have not worked on JDK for a while ...
> > >
> > > Thanks,
> > > Neo
> > >
> > > On 10/25/07, Ben Cheng < bccheng at google.com> wrote:
> > > > Hi,
> > > >
> > > > I was trying to get a sense of total GC time spent in a Java app
> using
> > 1.6
> > > > JDK. The gc logs (w/ and wo/ parallel gc) seem to break the ancient
> tool
> > > > gc_analyze.pl found on the web.
> > > >
> > > > Is there an update to date tool to produce the desired information
> for
> > new
> > > > JDKs?
> > > >
> > > > Thanks in advance,
> > > > -Ben
> > > >
> > >
> > >
> > > --
> > > I would remember that if researchers were not ambitious
> > > probably today we haven't the technology we are using!
> > >
> >
> >
>
>
> --
> I would remember that if researchers were not ambitious
> probably today we haven't the technology we are using!
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/hotspot-gc-dev/attachments/20071025/baea995a/attachment.htm>

From neojia at gmail.com  Fri Oct 26 07:19:55 2007
From: neojia at gmail.com (Neo Jia)
Date: Fri, 26 Oct 2007 00:19:55 -0700
Subject: gc log analysis tools
In-Reply-To: <bcf6f6d40710252322r7e4a3fdcq9c67a2a37dcb6f38@mail.gmail.com>
References: <bcf6f6d40710251804i5c7aed9duab74e246a23a7088@mail.gmail.com>
	<5d649bdb0710251807y2d446ab2ye7dba0e5b0025a25@mail.gmail.com>
	<bcf6f6d40710252200m53b38eeay6bbd2969d5e4ee75@mail.gmail.com>
	<5d649bdb0710252313r7ee83969t207e41a6c499905@mail.gmail.com>
	<bcf6f6d40710252322r7e4a3fdcq9c67a2a37dcb6f38@mail.gmail.com>
Message-ID: <5d649bdb0710260019m28f6a9b7xb39a29e9899bb67a@mail.gmail.com>

On 10/25/07, Ben Cheng <bccheng at google.com> wrote:
>
>
>
> On 10/25/07, Neo Jia <neojia at gmail.com> wrote:
> > Ben,
> >
> > Which one should be better? lower score or higher score? I assume it
> > should be higher based on what you said here.
>
>
> Oops, I forgot to clarify that. Yes, the higher the better.
>
> > And, what is the relation between score and time? Will the execution
> > time longer with a low score?
>
>
>
> The benchmark is doing fixed workload, so the score is simply calculated as
> work/s.
>
> > And how many minor GC and full GC in each of your execution? Just grep
> > and wc them.
>
>
>
> Almost all of them are minor GCs.

You may try the same ratio for these two configurations and check if
there is any difference.

Neo

>
> > For 32-bit and 64-bit machine, at least the default heap layout is
> > different. Are you trying to use same amount of nursery space and
> > mature space.
>
>
>
> I think I am just using the default ration. I only specify -Xmx and -Xms to
> the same amount without using -Xmn.
>
> > And for 64-bit machine, the Jitter is much more better than 32-bit
> > machine, a lot of optimization is added.
>
>
> I believe so, and that's why the 64-bit VM eventually provides 40% speedup
> with larger heap is given. I think under the same amount of heap space there
> are more GC activities in 64-bit VM due to larger object sizes. I will
> grep/wc the total gc time then.
>
> Thanks,
> -Ben
>
>
> > So ...
> >
> > Thanks,
> > Neo
> >
> > On 10/25/07, Ben Cheng <bccheng at google.com > wrote:
> > > The reason I am interested in this information is because I have a
> benchmark
> > > whose score is X using 32-bit VM and 2G of heap. When I switched to
> 64-bit
> > > build of the same VM, the score dropped to 0.77X with the same amount of
> > > heap. If I increase the heap size to 3.5G (on a machine with 4GB of
> RAM),
> > > the score went to to 1.40X.
> > >
> > > What I need is pretty basic - it sums up all the gc time from the
> verbose
> > > log during a benchmark run, as I want to verify if the change in gc time
> > > correlates to the change in performance. I can write a similar script to
> do
> > > that, but I just want to check to see if there is a working tool out
> there
> > > already. Apparently the verbose gc log format changed from 1.5 and
> beyond
> > > and the old script is not compatible.
> > >
> > > Thanks,
> > > -Ben
> > >
> > >
> > > On 10/25/07, Neo Jia < neojia at gmail.com> wrote:
> > > > Ben,
> > > >
> > > > What is your opion? I had one but not sure if it would work for you or
> not
> > > (
> > > >
> > > > I have not worked on JDK for a while ...
> > > >
> > > > Thanks,
> > > > Neo
> > > >
> > > > On 10/25/07, Ben Cheng < bccheng at google.com> wrote:
> > > > > Hi,
> > > > >
> > > > > I was trying to get a sense of total GC time spent in a Java app
> using
> > > 1.6
> > > > > JDK. The gc logs (w/ and wo/ parallel gc) seem to break the ancient
> tool
> > > > > gc_analyze.pl found on the web.
> > > > >
> > > > > Is there an update to date tool to produce the desired information
> for
> > > new
> > > > > JDKs?
> > > > >
> > > > > Thanks in advance,
> > > > > -Ben
> > > > >
> > > >
> > > >
> > > > --
> > > > I would remember that if researchers were not ambitious
> > > > probably today we haven't the technology we are using!
> > > >
> > >
> > >
> >
> >
> > --
> > I would remember that if researchers were not ambitious
> > probably today we haven't the technology we are using!
> >
>
>


-- 
I would remember that if researchers were not ambitious
probably today we haven't the technology we are using!


From Jon.Masamitsu at Sun.COM  Fri Oct 26 07:54:37 2007
From: Jon.Masamitsu at Sun.COM (Jon Masamitsu)
Date: Fri, 26 Oct 2007 00:54:37 -0700
Subject: gc log analysis tools
In-Reply-To: <bcf6f6d40710252200m53b38eeay6bbd2969d5e4ee75@mail.gmail.com>
References: <bcf6f6d40710251804i5c7aed9duab74e246a23a7088@mail.gmail.com>
	<5d649bdb0710251807y2d446ab2ye7dba0e5b0025a25@mail.gmail.com>
	<bcf6f6d40710252200m53b38eeay6bbd2969d5e4ee75@mail.gmail.com>
Message-ID: <47219D3D.7040109@sun.com>

Ben,

It's typical to see lower performance with the 64bit VM.  A drop of 23%
seems a little high
but is in the ball park.  The drop in performance comes from the
increased memory traffic
due to the larger objects.  

Hope that helps.

Jon

PS.  I don't think the output from -verbosegc changed at all between jdk
1.4.2 and jdk 5.
If it did, I think the change was very minor.  One difference you might
see with jdk 5 is that
you could be getting a different default garbage collector.   That might
be the source of
differences in the logs you're trying to analyze.

Ben Cheng wrote:

> The reason I am interested in this information is because I have a
> benchmark whose score is X using 32-bit VM and 2G of heap. When I
> switched to 64-bit build of the same VM, the score dropped to 0.77X
> with the same amount of heap. If I increase the heap size to 3.5G (on
> a machine with 4GB of RAM), the score went to to 1.40X.
>
> What I need is pretty basic - it sums up all the gc time from the
> verbose log during a benchmark run, as I want to verify if the change
> in gc time correlates to the change in performance. I can write a
> similar script to do that, but I just want to check to see if there is
> a working tool out there already. Apparently the verbose gc log format
> changed from 1.5 and beyond and the old script is not compatible.
>
> Thanks,
> -Ben
>
> On 10/25/07, * Neo Jia* <neojia at gmail.com <mailto:neojia at gmail.com>>
> wrote:
>
>     Ben,
>
>     What is your opion? I had one but not sure if it would work for
>     you or not (
>
>     I have not worked on JDK for a while ...
>
>     Thanks,
>     Neo
>
>     On 10/25/07, Ben Cheng < bccheng at google.com
>     <mailto:bccheng at google.com>> wrote:
>     > Hi,
>     >
>     > I was trying to get a sense of total GC time spent in a Java app
>     using 1.6
>     > JDK. The gc logs (w/ and wo/ parallel gc) seem to break the
>     ancient tool
>     > gc_analyze.pl found on the web.
>     >
>     > Is there an update to date tool to produce the desired
>     information for new
>     > JDKs?
>     >
>     > Thanks in advance,
>     > -Ben
>     >
>
>
>     --
>     I would remember that if researchers were not ambitious
>     probably today we haven't the technology we are using!
>
>




From Peter.Kessler at Sun.COM  Fri Oct 26 16:36:49 2007
From: Peter.Kessler at Sun.COM (Peter B. Kessler)
Date: Fri, 26 Oct 2007 09:36:49 -0700
Subject: gc log analysis tools
In-Reply-To: <bcf6f6d40710251804i5c7aed9duab74e246a23a7088@mail.gmail.com>
References: <bcf6f6d40710251804i5c7aed9duab74e246a23a7088@mail.gmail.com>
Message-ID: <472217A1.6060005@Sun.COM>

Ben Cheng wrote:

> Hi,
> 
> I was trying to get a sense of total GC time spent in a Java app using 
> 1.6 JDK. The gc logs (w/ and wo/ parallel gc) seem to break the ancient 
> tool gc_analyze.pl found on the web.
> 
> Is there an update to date tool to produce the desired information for 
> new JDKs?
> 
> Thanks in advance,
> -Ben


I'll see if we can get an updated version of PrintGCStats out to
replace the ancient one available at

     http://java.sun.com/developer/technicalArticles/Programming/turbo/

That should give you the kinds of statistics you are asking for.

			... peter


From tony.printezis at sun.com  Fri Oct 26 17:18:10 2007
From: tony.printezis at sun.com (Tony Printezis)
Date: Fri, 26 Oct 2007 13:18:10 -0400
Subject: gc log analysis tools
In-Reply-To: <472217A1.6060005@Sun.COM>
References: <bcf6f6d40710251804i5c7aed9duab74e246a23a7088@mail.gmail.com>
	<472217A1.6060005@Sun.COM>
Message-ID: <47222152.1080208@sun.com>

Hi Ben,

Additionally, we are trying to open source a graphical tool we have 
developed, called GChisto, that can show you a lot of stats about your 
log, as well as other interesting graphs. We have been using it to 
analyze customer logs and it has been very helpful. Once we have the OK 
to open source it, we'll announce it on this list.

Regards,

Tony

Peter B. Kessler wrote:
> Ben Cheng wrote:
>
>> Hi,
>>
>> I was trying to get a sense of total GC time spent in a Java app 
>> using 1.6 JDK. The gc logs (w/ and wo/ parallel gc) seem to break the 
>> ancient tool gc_analyze.pl found on the web.
>>
>> Is there an update to date tool to produce the desired information 
>> for new JDKs?
>>
>> Thanks in advance,
>> -Ben
>
>
> I'll see if we can get an updated version of PrintGCStats out to
> replace the ancient one available at
>
>     http://java.sun.com/developer/technicalArticles/Programming/turbo/
>
> That should give you the kinds of statistics you are asking for.
>
>             ... peter

-- 
----------------------------------------------------------------------
| Tony Printezis, Staff Engineer    | Sun Microsystems Inc.          |
|                                   | MS BUR02-311                   |
| e-mail: tony.printezis at sun.com    | 35 Network Drive               |
| office: +1 781 442 0998 (x20998)  | Burlington, MA01803-0902, USA  |
----------------------------------------------------------------------
e-mail client: Thunderbird (Solaris)




From bccheng at google.com  Fri Oct 26 17:19:55 2007
From: bccheng at google.com (Ben Cheng)
Date: Fri, 26 Oct 2007 10:19:55 -0700
Subject: gc log analysis tools
In-Reply-To: <472217A1.6060005@Sun.COM>
References: <bcf6f6d40710251804i5c7aed9duab74e246a23a7088@mail.gmail.com>
	<472217A1.6060005@Sun.COM>
Message-ID: <bcf6f6d40710261019tbf97447r9480523d4983d94d@mail.gmail.com>

Thank you Peter. I just visisted the turbo... page and the PrintGCStats
script there is working perfectly with the gc log I collected the other day.

-Ben

On 10/26/07, Peter B. Kessler <Peter.Kessler at sun.com> wrote:
>
> Ben Cheng wrote:
>
> > Hi,
> >
> > I was trying to get a sense of total GC time spent in a Java app using
> > 1.6 JDK. The gc logs (w/ and wo/ parallel gc) seem to break the ancient
> > tool gc_analyze.pl found on the web.
> >
> > Is there an update to date tool to produce the desired information for
> > new JDKs?
> >
> > Thanks in advance,
> > -Ben
>
>
> I'll see if we can get an updated version of PrintGCStats out to
> replace the ancient one available at
>
>      http://java.sun.com/developer/technicalArticles/Programming/turbo/
>
> That should give you the kinds of statistics you are asking for.
>
>                         ... peter
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/hotspot-gc-dev/attachments/20071026/3f2324d6/attachment.htm>