From Andreas.Mueller at mgm-tp.com  Sat Nov  2 15:15:15 2013
From: Andreas.Mueller at mgm-tp.com (=?iso-8859-1?Q?Andreas_M=FCller?=)
Date: Sat, 2 Nov 2013 22:15:15 +0000
Subject: G1 log traces: rounding errors!
Message-ID: <46FF8393B58AD84D95E444264805D98FBDDF1504@edata01.mgm-edv.de>

Hi all,

has anybody ever noticed that the G1 collector (in Java 7 with larger heaps) writes gc.log traces like that:
433.966: [GC pause (young) 13G->12G(16G), 0.0431884 secs]
I am a bit concerned about the rounding errors when I use those traces to do a quantitative analysis:(
What about that GC trace line:
417.747: [GC pause (young) 12G->12G(16G), 0.0443063 secs]
I am sure that the collector did some work here, but any meaningful parsing of that line must come to the conclusion that 0 bytes of heap have been freed.
On the other hand, I find traces like this one:
48.809: [GC pause (young) 10G->9981M(16G), 0.0607958 secs]

Is there any information about the rounding policy in these traces?

Best regards
Andreas

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20131102/a2a17628/attachment.html 

From thomas.schatzl at oracle.com  Sun Nov  3 03:16:00 2013
From: thomas.schatzl at oracle.com (Thomas Schatzl)
Date: Sun, 03 Nov 2013 12:16:00 +0100
Subject: G1 log traces: rounding errors!
In-Reply-To: <46FF8393B58AD84D95E444264805D98FBDDF1504@edata01.mgm-edv.de>
References: <46FF8393B58AD84D95E444264805D98FBDDF1504@edata01.mgm-edv.de>
Message-ID: <1383477360.2693.4.camel@cirrus>

On Sat, 2013-11-02 at 22:15 +0000, Andreas M?ller wrote:
> Hi all,
>  
> has anybody ever noticed that the G1 collector (in Java 7 with larger
> heaps) writes gc.log traces like that:
>
> 433.966: [GC pause (young) 13G->12G(16G), 0.0431884 secs]
>
> I am a bit concerned about the rounding errors when I use those traces
> to do a quantitative analysisL

These problems are known.

> What about that GC trace line:
>
> 417.747: [GC pause (young) 12G->12G(16G), 0.0443063 secs]
>
> I am sure that the collector did some work here, but any meaningful
> parsing of that line must come to the conclusion that 0 bytes of heap
> have been freed.
>
> On the other hand, I find traces like this one:
>
> 48.809: [GC pause (young) 10G->9981M(16G), 0.0607958 secs]
> 
> Is there any information about the rounding policy in these traces?

globalDefinitions.hpp, line 210:

template <class T>
inline T byte_size_in_proper_unit(T s) {
#ifdef _LP64
  if (s >= 10*G) {
    return (T)(s/G);
  }
#endif
  if (s >= 10*M) {
    return (T)(s/M);
  } else if (s >= 10*K) {
    return (T)(s/K);
  } else {
    return s;
  }
}

where G/M/K are 2^30/2^20/2^10 respectively.

Probably it would be better to make sure to preserve more digits in
these messages and do better rounding.

Thomas



From michal at frajt.eu  Tue Nov  5 02:26:53 2013
From: michal at frajt.eu (Michal Frajt)
Date: Tue,  5 Nov 2013 11:26:53 +0100
Subject: Survivor space class historgram print (know your bad garbage)
In-Reply-To: <CABzyjykarchvP9Cr2hzxSiktKC=4bJY=dDV78zCEG=x4Yqu-ag@mail.gmail.com>
References: =?iso-8859-1?q?=3C01e201cca4b9=24de88e400=249b9aac00=24=40frajt=2Eeu?=
	=?iso-8859-1?q?=3E_=3CCABzyjynyN8=3DTHDUOFRFosda0p0cgyRfEdBZFuTtFZCnw?=
	=?iso-8859-1?q?O=2DU56Q=40mail=2Egmail=2Ecom=3E_=3C4EC507E0=2E1080905?=
	=?iso-8859-1?q?=40oracle=2Ecom=3E_=3CCABzyjykarchvP9Cr2hzxSiktKC=3D4b?=
	=?iso-8859-1?q?JY=3DdDV78zCEG=3Dx4Yqu=2Dag=40mail=2Egmail=2Ecom=3E?=
Message-ID: <MVSD0T$F923495C346FD7F121B654B08538D920@frajt.eu>

Hi Ramki,We have again a situation where the survivor space histogram would help us to clarify an application behavior. Would you be able to find the non-integrated code reporting the survivor space class histogram on every
                  minor collection run? We are maintaining our own openjdk version where we would simply try to integrate it again. I assume that the  code was extended around the ageTable class.Thanks,Michal
Od: hotspot-gc-dev-bounces at openjdk.java.net
Komu: "Tony Printezis" tony.printezis at oracle.com
Kopie: hotspot-gc-use at openjdk.java.net
Datum: Thu, 17 Nov 2011 08:53:32 -0800
P?edmet: Re: Survivor space class historgram print (know your bad garbage)


> 
Yes, that seems like a good place to expose such functionality. The question of refactoring to allow> the stats gathering to happen at low cost when disabled of course still remains, I guess.> thanks!> -- ramki
> On Thu, Nov 17, 2011 at 5:10 AM, Tony Printezis <tony.printezis at oracle.com> wrote:> 

  
    
  
  > 
    No current plans to integrate this. But this is something we could
    consider as part of our ongoing effort to support Mission Control.> 
    > 
    Tony> 
    > 
    On 11/16/2011 7:01 PM, Srinivas Ramakrishna wrote:
    AFAR, it was not integrated at that time because of
      the performance impact even when the feature was turned off.> 
      I believe it would be possible to refactor the code, at some cost,
      to get this to work without that performance> 
      impact, but that didn't get done. It might be time to revisit that
      code and do the requisite refactoring. Tony et al?> 
      > 
      -- ramki> 
      > 
      > 
      > On Wed, Nov 16, 2011 at 3:45 PM, Michal
        Frajt <michal at frajt.eu>
        wrote:> 
        
          > 
            > 
              > Hi,
              > 
                ?
              > Is there a way to
                  get printed a survivor space class histogram on every
                  minor collection run? Tony once provided us a special
                  jdk build containing this feature but it got never
                  integrated as a print flag into the main hotspot
                  version. It was very useful for understanding and
                  identifying promoted objects to the old gen.
                  Additionally we were looking to get a class histogram
                  print for the eden space but there was no easy way to
                  implement it. The eden space class histogram would
                  help to identify garbage invoking minor collection
                  runs. It could be as well used to check the impact of
                  the scalar replacements.
              > ?
              > Would it be
                  possible to reimplement both histogram prints?
              > ?
              > Thanks,
              > Michal
            
          
          > 
          _______________________________________________> 
          hotspot-gc-use mailing list> 
          hotspot-gc-use at openjdk.java.net> 
          http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use> 
          > 
        
      
      > 
      > 
      
      > 
      > _______________________________________________
hotspot-gc-use mailing list
hotspot-gc-use at openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use

    
  




-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20131105/4e8a9021/attachment.html 

From ysr1729 at gmail.com  Thu Nov 14 13:36:09 2013
From: ysr1729 at gmail.com (Srinivas Ramakrishna)
Date: Thu, 14 Nov 2013 13:36:09 -0800
Subject: Survivor space class historgram print (know your bad garbage)
In-Reply-To: <MVSD0T$F923495C346FD7F121B654B08538D920@frajt.eu>
References: <01e201cca4b9$de88e400$9b9aac00$@frajt.eu>
	<CABzyjynyN8=THDUOFRFosda0p0cgyRfEdBZFuTtFZCnwO-U56Q@mail.gmail.com>
	<4EC507E0.1080905@oracle.com>
	<CABzyjykarchvP9Cr2hzxSiktKC=4bJY=dDV78zCEG=x4Yqu-ag@mail.gmail.com>
	<MVSD0T$F923495C346FD7F121B654B08538D920@frajt.eu>
Message-ID: <CABzyjyk9+_LjbBoGFep=OiFn4pwJxULQqTALz2BpC9CJbDzX5w@mail.gmail.com>

Hi Michal --

That code was at that time internal to Sun/Oracle and was not integrated
into the product for the reasons I described. Since Tony and I are now both
outside of Oracle, I don't know if the code was kept or lost. If it is
there, perhaps someone inside Oracle could help with it, especially if it's
in Mission Control in some form. Don't know because AFAIK that's a for fee
product and code may not necessarily be open source (most likely not).

You should be able to look at the age table interface and build the
functionality into your JVM if you wanted. If you are a source licensee,
and the support models are what they used to be, you'd be able to talk with
your Oracle liaison to get further help. You can look at the code that does
age table updates while copying survivor objects to see how it's done. I
can't recall how the original was done (and even if I did, I couldn't
describe code that is not open source and which i was privy to while a
Sun/Oracle employee, nor can I provide actual help to you unless the code
was open sourced), but you can think of how you might do that census over
the population of interest to you at a time of interest to you and do it
accordingly.

all the best!
-- ramki



On Tue, Nov 5, 2013 at 2:26 AM, Michal Frajt <michal at frajt.eu> wrote:

>  Hi Ramki,
>
> We have again a situation where the survivor space histogram would help
> us to clarify an application behavior. Would you be able to find the
> non-integrated code reporting the survivor space class histogram on every
> minor collection run? We are maintaining our own openjdk version where we
> would simply try to integrate it again. I assume that the code was extended
> around the ageTable class.
>
> Thanks,
> Michal
>
> Od: hotspot-gc-dev-bounces at openjdk.java.net
> Komu: "Tony Printezis" tony.printezis at oracle.com
> Kopie: hotspot-gc-use at openjdk.java.net
> Datum: Thu, 17 Nov 2011 08:53:32 -0800
> P?edmet: Re: Survivor space class historgram print (know your bad garbage)
>
> > Yes, that seems like a good place to expose such functionality. The
> question of refactoring to allow
> > the stats gathering to happen at low cost when disabled of course still
> remains, I guess.
>
> > thanks!
> > -- ramki
>
> > On Thu, Nov 17, 2011 at 5:10 AM, Tony Printezis <
> tony.printezis at oracle.com> wrote:
>
>> >
>> > No current plans to integrate this. But this is something we could
>> consider as part of our ongoing effort to support Mission Control.
>> >
>> > Tony
>>
>> >
>> > On 11/16/2011 7:01 PM, Srinivas Ramakrishna wrote:
>>
>> AFAR, it was not integrated at that time because of the performance
>> impact even when the feature was turned off.
>> > I believe it would be possible to refactor the code, at some cost, to
>> get this to work without that performance
>> > impact, but that didn't get done. It might be time to revisit that code
>> and do the requisite refactoring. Tony et al?
>> >
>> > -- ramki
>> >
>> >
>> >
>> > On Wed, Nov 16, 2011 at 3:45 PM, Michal Frajt <michal at frajt.eu> wrote:
>> >
>>>
>>> >
>>> >
>>>
>>> > Hi,
>>>
>>> >
>>>
>>> > Is there a way to get printed a survivor space class histogram on
>>> every minor collection run? Tony once provided us a special jdk build
>>> containing this feature but it got never integrated as a print flag into
>>> the main hotspot version. It was very useful for understanding and
>>> identifying promoted objects to the old gen. Additionally we were looking
>>> to get a class histogram print for the eden space but there was no easy way
>>> to implement it. The eden space class histogram would help to identify
>>> garbage invoking minor collection runs. It could be as well used to check
>>> the impact of the scalar replacements.
>>>
>>> >
>>>
>>> > Would it be possible to reimplement both histogram prints?
>>>
>>> >
>>>
>>> > Thanks,
>>>
>>> > Michal
>>>
>>> > _______________________________________________
>>> > hotspot-gc-use mailing list
>>> > hotspot-gc-use at openjdk.java.net
>>> > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>> >
>>> >
>>
>>
>> >
>> >
>> >
>>
>> > _______________________________________________
>> hotspot-gc-use mailing listhotspot-gc-use at openjdk.java.nethttp://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20131114/d0f862da/attachment.html 

From ysr1729 at gmail.com  Thu Nov 14 13:46:34 2013
From: ysr1729 at gmail.com (Srinivas Ramakrishna)
Date: Thu, 14 Nov 2013 13:46:34 -0800
Subject: GC performance regression between 7u11->7u45?
Message-ID: <CABzyjym2mJjyf0AqDueyqj5pep+JA-myagDuj2W7pcX85qrQDA@mail.gmail.com>

Is anyone, especially the perf and gc teams inside Oracle, aware of any
parallel young (PS) gc performance regressions
from 7u11 to 7u45?

I am seeing some 20-30% regression in minor gc times (and a lesser increase
in promotion volume per scavenge),
on some random testing of a largish application. Haven't dug into the
details yet, but wondered if this was noted
on any of the Oracle perf/gc or community perf/gc radars and if there's any
guidance available.

As I delve deeper the nature of this apparent GC regression, which I just
became aware of, I'll post more.

thanks for any help!
-- ramki
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20131114/6b07fefc/attachment.html 

From jon.masamitsu at oracle.com  Thu Nov 14 14:57:21 2013
From: jon.masamitsu at oracle.com (Jon Masamitsu)
Date: Thu, 14 Nov 2013 14:57:21 -0800
Subject: GC performance regression between 7u11->7u45?
In-Reply-To: <CABzyjym2mJjyf0AqDueyqj5pep+JA-myagDuj2W7pcX85qrQDA@mail.gmail.com>
References: <CABzyjym2mJjyf0AqDueyqj5pep+JA-myagDuj2W7pcX85qrQDA@mail.gmail.com>
Message-ID: <52855551.3090801@oracle.com>

Ramki,

I don't think we've seen such a regression.  Fix the heap size and
turn off UseAdaptivesizePolicy.

What platform?

Jon

On 11/14/13 1:46 PM, Srinivas Ramakrishna wrote:
>
> Is anyone, especially the perf and gc teams inside Oracle, aware of 
> any parallel young (PS) gc performance regressions
> from 7u11 to 7u45?
>
> I am seeing some 20-30% regression in minor gc times (and a lesser 
> increase in promotion volume per scavenge),
> on some random testing of a largish application. Haven't dug into the 
> details yet, but wondered if this was noted
> on any of the Oracle perf/gc or community perf/gc radars and if 
> there's any guidance available.
>
> As I delve deeper the nature of this apparent GC regression, which I 
> just became aware of, I'll post more.
>
> thanks for any help!
> -- ramki
>
>
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20131114/8c01ecb9/attachment.html 

From ysr1729 at gmail.com  Thu Nov 14 15:32:45 2013
From: ysr1729 at gmail.com (Srinivas Ramakrishna)
Date: Thu, 14 Nov 2013 15:32:45 -0800
Subject: GC performance regression between 7u11->7u45?
In-Reply-To: <52855551.3090801@oracle.com>
References: <CABzyjym2mJjyf0AqDueyqj5pep+JA-myagDuj2W7pcX85qrQDA@mail.gmail.com>
	<52855551.3090801@oracle.com>
Message-ID: <CABzyjyk6kPUzabgJue_OB3acbOy7uCnkUdZOT65Cg=-Q3qYSdw@mail.gmail.com>

Thanks Jon.

intel/linux, c2, ps gc.

Will turn off adaptive size policy. Survivor space sizing seems to be
different causing more promotion, at a high level.

Haven't tried to narrow down to VM or JDK yet, but survivor size diffs may
be key, so yr suggestion to turn off adaptive size may be the thing.

Also need to get more data to see how reproducible it is with workload etc.

thanks!
-- ramki



On Thu, Nov 14, 2013 at 2:57 PM, Jon Masamitsu <jon.masamitsu at oracle.com>wrote:

>  Ramki,
>
> I don't think we've seen such a regression.  Fix the heap size and
> turn off UseAdaptivesizePolicy.
>
> What platform?
>
> Jon
>
>
> On 11/14/13 1:46 PM, Srinivas Ramakrishna wrote:
>
>
>  Is anyone, especially the perf and gc teams inside Oracle, aware of any
> parallel young (PS) gc performance regressions
>  from 7u11 to 7u45?
>
>  I am seeing some 20-30% regression in minor gc times (and a lesser
> increase in promotion volume per scavenge),
>  on some random testing of a largish application. Haven't dug into the
> details yet, but wondered if this was noted
> on any of the Oracle perf/gc or community perf/gc radars and if there's
> any guidance available.
>
>  As I delve deeper the nature of this apparent GC regression, which I just
> became aware of, I'll post more.
>
> thanks for any help!
>  -- ramki
>
>
> _______________________________________________
> hotspot-gc-use mailing listhotspot-gc-use at openjdk.java.nethttp://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>
>
>
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20131114/be705a35/attachment.html 

From alexey.ragozin at gmail.com  Fri Nov 15 19:29:16 2013
From: alexey.ragozin at gmail.com (Alexey Ragozin)
Date: Sat, 16 Nov 2013 07:29:16 +0400
Subject: Survivor space class historgram print (know your bad garbage)
Message-ID: <CAMgTVmL1hOVSbQCCwo7OH3b_Dp1y-Dn6GcWPYv+gRkRvRFqL=Q@mail.gmail.com>

Hi Michal,

This is not exactly what you are asking for, but you may find it useable.

Take a look at sjk hh command (https://github.com/aragozin/jvm-tools#hh).

With -dead option it will print you histogram of garbage in your heap
(underhood it take normal jmap-style histogram then forces GC then
takes another histogram and calculates diff).

I have found you very useful for analyzing "garbage" problem.

Another feature sjk helping with memory troubleshooting - reporting
per thread allocation rate in real time (JMX is used underhood).

Regards,
Alexey

On Tue, Nov 5, 2013 at 2:26 AM, Michal Frajt <michal at frajt.eu> wrote:

>  Hi Ramki,
>
> We have again a situation where the survivor space histogram would help
> us to clarify an application behavior. Would you be able to find the
> non-integrated code reporting the survivor space class histogram on every
> minor collection run? We are maintaining our own openjdk version where we
> would simply try to integrate it again. I assume that the code was extended
> around the ageTable class.
>
> Thanks,
> Michal
>
> Od: hotspot-gc-dev-bounces at openjdk.java.net
> Komu: "Tony Printezis" tony.printezis at oracle.com
> Kopie: hotspot-gc-use at openjdk.java.net
> Datum: Thu, 17 Nov 2011 08:53:32 -0800
> P?edmet: Re: Survivor space class historgram print (know your bad garbage)
>
> > Yes, that seems like a good place to expose such functionality. The
> question of refactoring to allow
> > the stats gathering to happen at low cost when disabled of course still
> remains, I guess.
>
> > thanks!
> > -- ramki
>
> > On Thu, Nov 17, 2011 at 5:10 AM, Tony Printezis <
> tony.printezis at oracle.com> wrote:
>
>> >
>> > No current plans to integrate this. But this is something we could
>> consider as part of our ongoing effort to support Mission Control.
>> >
>> > Tony
>>
>> >
>> > On 11/16/2011 7:01 PM, Srinivas Ramakrishna wrote:
>>
>> AFAR, it was not integrated at that time because of the performance
>> impact even when the feature was turned off.
>> > I believe it would be possible to refactor the code, at some cost, to
>> get this to work without that performance
>> > impact, but that didn't get done. It might be time to revisit that code
>> and do the requisite refactoring. Tony et al?
>> >
>> > -- ramki
>> >
>> >
>> >
>> > On Wed, Nov 16, 2011 at 3:45 PM, Michal Frajt <michal at frajt.eu> wrote:
>> >
>>>
>>> >
>>> >
>>>
>>> > Hi,
>>>
>>> >
>>>
>>> > Is there a way to get printed a survivor space class histogram on
>>> every minor collection run? Tony once provided us a special jdk build
>>> containing this feature but it got never integrated as a print flag into
>>> the main hotspot version. It was very useful for understanding and
>>> identifying promoted objects to the old gen. Additionally we were looking
>>> to get a class histogram print for the eden space but there was no easy way
>>> to implement it. The eden space class histogram would help to identify
>>> garbage invoking minor collection runs. It could be as well used to check
>>> the impact of the scalar replacements.
>>>
>>> >
>>>
>>> > Would it be possible to reimplement both histogram prints?
>>>
>>> >
>>>
>>> > Thanks,
>>>
>>> > Michal
>>>
>>> > ______________________________
_________________
>>> > hotspot-gc-use mailing list
>>> > hotspot-gc-use at openjdk.java.net
>>> > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>> >
>>> >
>>
>>
>> >
>> >
>> >
>>
>> > _______________________________________________
>> hotspot-gc-use mailing listhotspot-gc-use at openjdk.java.nethttp://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>
>>
>