From kaj.nygren at mag.se  Wed May  6 05:06:13 2009
From: kaj.nygren at mag.se (Kaj Nygren)
Date: Wed, 06 May 2009 14:06:13 +0200
Subject: Strange long ParNew collections
Message-ID: <4A017D35.7000208@mag.se>

Hi,

We are having problems with occational long ParNew pauses in our 
application running on a Jboss server. The server runs well for most of 
the time but sometimes we get ParNew collections that can take up to 120 
seconds. Most of the time we have several consecutive slow collections, 
which are more or less killing the server under high load.

We are using JDK 1.5.0_16 on Windows Server 2003, and are using ParNew + 
Concurrent CMS.

Our GC configuration is:

wrapper.java.additional.1=-Dprogram.name=run.bat
wrapper.java.additional.2=-server
wrapper.java.additional.3=-Xms19000m
wrapper.java.additional.4=-Xmx19000m
wrapper.java.additional.5="-XX:NewSize=1000m"
wrapper.java.additional.6="-XX:MaxNewSize=1000m"
wrapper.java.additional.7="-XX:MaxPermSize=256m"
wrapper.java.additional.8="-XX:PermSize=256m"
wrapper.java.additional.11="-Xloggc:c:\mygclog.gc"
wrapper.java.additional.12="-XX:+UseConcMarkSweepGC"
wrapper.java.additional.13="-XX:+PrintGCDetails"
wrapper.java.additional.14="-XX:-PrintTenuringDistribution"
wrapper.java.additional.15="-XX:+PrintHeapAtGC"
wrapper.java.additional.16="-XX:+DisableExplicitGC"
wrapper.java.additional.17="-XX:+PrintGCTimeStamps"
wrapper.java.additional.18="-XX:CMSInitiatingOccupancyFraction=50"
wrapper.java.additional.19="-XX:+CMSIncrementalMode"
wrapper.java.additional.20="-XX:+CMSIncrementalPacing"
wrapper.java.additional.21="-verbose:gc"

Usually ParNew is quick as can be seen here:

614276,653: [ParNew: 935058K->0K(1023040K), 0,0624699 secs] 
12383872K->11459948K(19455040K) icms_dc=22 Heap after gc invocations=57822:
 par new generation   total 1023040K, used 0K [0x000000007fff0000, 
0x00000000be7f0000, 0x00000000be7f0000)
  eden space 1022080K,   0% used [0x000000007fff0000, 
0x000000007fff0000, 0x00000000be610000)
  from space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
0x00000000be7f0000)
  to   space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
0x00000000be700000)
 concurrent mark-sweep generation total 18432000K, used 11459948K 
[0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
 concurrent-mark-sweep perm gen total 262144K, used 112069K 
[0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
}
, 0,0628860 secs]

But suddenly it becomes really slow:

620659,717: [ParNew: 1022080K->0K(1023040K), 1,3313817 secs] 
12688913K->11692925K(19455040K) icms_dc=10 Heap after gc invocations=58587:
 par new generation   total 1023040K, used 0K [0x000000007fff0000, 
0x00000000be7f0000, 0x00000000be7f0000)
  eden space 1022080K,   0% used [0x000000007fff0000, 
0x000000007fff0000, 0x00000000be610000)
  from space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
0x00000000be700000)
  to   space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
0x00000000be7f0000)
 concurrent mark-sweep generation total 18432000K, used 11692925K 
[0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
 concurrent-mark-sweep perm gen total 262144K, used 112098K 
[0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
}
, 1,3319554 secs]
620663,221: [GC {Heap before gc invocations=58587:
 par new generation   total 1023040K, used 1022080K [0x000000007fff0000, 
0x00000000be7f0000, 0x00000000be7f0000)
  eden space 1022080K, 100% used [0x000000007fff0000, 
0x00000000be610000, 0x00000000be610000)
  from space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
0x00000000be700000)
  to   space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
0x00000000be7f0000)
 concurrent mark-sweep generation total 18432000K, used 11692925K 
[0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
 concurrent-mark-sweep perm gen total 262144K, used 112106K 
[0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
620663,222: [ParNew: 1022080K->0K(1023040K), 10,2909893 secs] 
12715005K->11936864K(19455040K) icms_dc=10 Heap after gc invocations=58588:
 par new generation   total 1023040K, used 0K [0x000000007fff0000, 
0x00000000be7f0000, 0x00000000be7f0000)
  eden space 1022080K,   0% used [0x000000007fff0000, 
0x000000007fff0000, 0x00000000be610000)
  from space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
0x00000000be7f0000)
  to   space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
0x00000000be700000)
 concurrent mark-sweep generation total 18432000K, used 11936864K 
[0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
 concurrent-mark-sweep perm gen total 262144K, used 112106K 
[0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
}
, 10,2915548 secs]
620676,392: [GC {Heap before gc invocations=58588:
 par new generation   total 1023040K, used 1022080K [0x000000007fff0000, 
0x00000000be7f0000, 0x00000000be7f0000)
  eden space 1022080K, 100% used [0x000000007fff0000, 
0x00000000be610000, 0x00000000be610000)
  from space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
0x00000000be7f0000)
  to   space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
0x00000000be700000)
 concurrent mark-sweep generation total 18432000K, used 11936864K 
[0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
 concurrent-mark-sweep perm gen total 262144K, used 112106K 
[0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
620676,392: [ParNew: 1022080K->0K(1023040K), 3,8905950 secs] 
12958944K->12020225K(19455040K) icms_dc=10 Heap after gc invocations=58589:
 par new generation   total 1023040K, used 0K [0x000000007fff0000, 
0x00000000be7f0000, 0x00000000be7f0000)
  eden space 1022080K,   0% used [0x000000007fff0000, 
0x000000007fff0000, 0x00000000be610000)
  from space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
0x00000000be700000)
  to   space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
0x00000000be7f0000)
 concurrent mark-sweep generation total 18432000K, used 12020225K 
[0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
 concurrent-mark-sweep perm gen total 262144K, used 112106K 
[0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
}
, 3,8911113 secs]
620682,667: [GC {Heap before gc invocations=58589:
 par new generation   total 1023040K, used 1022080K [0x000000007fff0000, 
0x00000000be7f0000, 0x00000000be7f0000)
  eden space 1022080K, 100% used [0x000000007fff0000, 
0x00000000be610000, 0x00000000be610000)
  from space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
0x00000000be700000)
  to   space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
0x00000000be7f0000)
 concurrent mark-sweep generation total 18432000K, used 12020225K 
[0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
 concurrent-mark-sweep perm gen total 262144K, used 112112K 
[0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
620682,668: [ParNew: 1022080K->0K(1023040K), 9,4186088 secs] 
13042305K->12214571K(19455040K) icms_dc=10 Heap after gc invocations=58590:
 par new generation   total 1023040K, used 0K [0x000000007fff0000, 
0x00000000be7f0000, 0x00000000be7f0000)
  eden space 1022080K,   0% used [0x000000007fff0000, 
0x000000007fff0000, 0x00000000be610000)
  from space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
0x00000000be7f0000)
  to   space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
0x00000000be700000)
 concurrent mark-sweep generation total 18432000K, used 12214571K 
[0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
 concurrent-mark-sweep perm gen total 262144K, used 112112K 
[0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
}
, 9,4191551 secs]

We also have problems with occational promotion failed, which I assume 
is due to fragmentation.
I have attatched a longer log, which shows this behaviour and ends with 
a promotion failed.

Does anybody have a glue as to what is going wrong, and how we can 
change our gc configuration to reduce the problems?
Any help is greatly appreciated!

Regards

Kaj Nygren
kaj at mag.se

-------------- next part --------------
A non-text attachment was scrubbed...
Name: strangegclog.zip
Type: application/x-zip-compressed
Size: 268061 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20090506/2c27e6e1/attachment.bin 

From Y.S.Ramakrishna at Sun.COM  Wed May  6 14:56:28 2009
From: Y.S.Ramakrishna at Sun.COM (Y.S.Ramakrishna at Sun.COM)
Date: Wed, 06 May 2009 14:56:28 -0700
Subject: Strange long ParNew collections
In-Reply-To: <4A017D35.7000208@mag.se>
References: <4A017D35.7000208@mag.se>
Message-ID: <4A02078C.4080904@Sun.COM>

This may or may not be a cause of the sudden spikes in
ParNew collection, but I'd suggest using the survivor
spaces to age objects for at least a few collections
before promoting to the old gen. This reduces pressure
on the old gen allocation (during scavenge) improving
scavenges and on CMS collection as well.

I'd suggest trying -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=4
to start off with and iterate based on data from -XX:+PrintTenuringDistribution.

It is possible that the promotion of very short-lived objects fragments
the old gen and causes old gen allocation (for promotion) to slow down.
It can also cause lots of mutation in the old gen and adversely affect
card-scanning times further slowing down scavenges.

-- ramki

On 05/06/09 05:06, Kaj Nygren wrote:
> Hi,
> 
> We are having problems with occational long ParNew pauses in our 
> application running on a Jboss server. The server runs well for most of 
> the time but sometimes we get ParNew collections that can take up to 120 
> seconds. Most of the time we have several consecutive slow collections, 
> which are more or less killing the server under high load.
> 
> We are using JDK 1.5.0_16 on Windows Server 2003, and are using ParNew + 
> Concurrent CMS.
> 
> Our GC configuration is:
> 
> wrapper.java.additional.1=-Dprogram.name=run.bat
> wrapper.java.additional.2=-server
> wrapper.java.additional.3=-Xms19000m
> wrapper.java.additional.4=-Xmx19000m
> wrapper.java.additional.5="-XX:NewSize=1000m"
> wrapper.java.additional.6="-XX:MaxNewSize=1000m"
> wrapper.java.additional.7="-XX:MaxPermSize=256m"
> wrapper.java.additional.8="-XX:PermSize=256m"
> wrapper.java.additional.11="-Xloggc:c:\mygclog.gc"
> wrapper.java.additional.12="-XX:+UseConcMarkSweepGC"
> wrapper.java.additional.13="-XX:+PrintGCDetails"
> wrapper.java.additional.14="-XX:-PrintTenuringDistribution"
> wrapper.java.additional.15="-XX:+PrintHeapAtGC"
> wrapper.java.additional.16="-XX:+DisableExplicitGC"
> wrapper.java.additional.17="-XX:+PrintGCTimeStamps"
> wrapper.java.additional.18="-XX:CMSInitiatingOccupancyFraction=50"
> wrapper.java.additional.19="-XX:+CMSIncrementalMode"
> wrapper.java.additional.20="-XX:+CMSIncrementalPacing"
> wrapper.java.additional.21="-verbose:gc"
> 
> Usually ParNew is quick as can be seen here:
> 
> 614276,653: [ParNew: 935058K->0K(1023040K), 0,0624699 secs] 
> 12383872K->11459948K(19455040K) icms_dc=22 Heap after gc invocations=57822:
> par new generation   total 1023040K, used 0K [0x000000007fff0000, 
> 0x00000000be7f0000, 0x00000000be7f0000)
>  eden space 1022080K,   0% used [0x000000007fff0000, 0x000000007fff0000, 
> 0x00000000be610000)
>  from space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
> 0x00000000be7f0000)
>  to   space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
> 0x00000000be700000)
> concurrent mark-sweep generation total 18432000K, used 11459948K 
> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
> concurrent-mark-sweep perm gen total 262144K, used 112069K 
> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
> }
> , 0,0628860 secs]
> 
> But suddenly it becomes really slow:
> 
> 620659,717: [ParNew: 1022080K->0K(1023040K), 1,3313817 secs] 
> 12688913K->11692925K(19455040K) icms_dc=10 Heap after gc invocations=58587:
> par new generation   total 1023040K, used 0K [0x000000007fff0000, 
> 0x00000000be7f0000, 0x00000000be7f0000)
>  eden space 1022080K,   0% used [0x000000007fff0000, 0x000000007fff0000, 
> 0x00000000be610000)
>  from space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
> 0x00000000be700000)
>  to   space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
> 0x00000000be7f0000)
> concurrent mark-sweep generation total 18432000K, used 11692925K 
> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
> concurrent-mark-sweep perm gen total 262144K, used 112098K 
> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
> }
> , 1,3319554 secs]
> 620663,221: [GC {Heap before gc invocations=58587:
> par new generation   total 1023040K, used 1022080K [0x000000007fff0000, 
> 0x00000000be7f0000, 0x00000000be7f0000)
>  eden space 1022080K, 100% used [0x000000007fff0000, 0x00000000be610000, 
> 0x00000000be610000)
>  from space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
> 0x00000000be700000)
>  to   space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
> 0x00000000be7f0000)
> concurrent mark-sweep generation total 18432000K, used 11692925K 
> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
> concurrent-mark-sweep perm gen total 262144K, used 112106K 
> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
> 620663,222: [ParNew: 1022080K->0K(1023040K), 10,2909893 secs] 
> 12715005K->11936864K(19455040K) icms_dc=10 Heap after gc invocations=58588:
> par new generation   total 1023040K, used 0K [0x000000007fff0000, 
> 0x00000000be7f0000, 0x00000000be7f0000)
>  eden space 1022080K,   0% used [0x000000007fff0000, 0x000000007fff0000, 
> 0x00000000be610000)
>  from space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
> 0x00000000be7f0000)
>  to   space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
> 0x00000000be700000)
> concurrent mark-sweep generation total 18432000K, used 11936864K 
> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
> concurrent-mark-sweep perm gen total 262144K, used 112106K 
> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
> }
> , 10,2915548 secs]
> 620676,392: [GC {Heap before gc invocations=58588:
> par new generation   total 1023040K, used 1022080K [0x000000007fff0000, 
> 0x00000000be7f0000, 0x00000000be7f0000)
>  eden space 1022080K, 100% used [0x000000007fff0000, 0x00000000be610000, 
> 0x00000000be610000)
>  from space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
> 0x00000000be7f0000)
>  to   space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
> 0x00000000be700000)
> concurrent mark-sweep generation total 18432000K, used 11936864K 
> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
> concurrent-mark-sweep perm gen total 262144K, used 112106K 
> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
> 620676,392: [ParNew: 1022080K->0K(1023040K), 3,8905950 secs] 
> 12958944K->12020225K(19455040K) icms_dc=10 Heap after gc invocations=58589:
> par new generation   total 1023040K, used 0K [0x000000007fff0000, 
> 0x00000000be7f0000, 0x00000000be7f0000)
>  eden space 1022080K,   0% used [0x000000007fff0000, 0x000000007fff0000, 
> 0x00000000be610000)
>  from space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
> 0x00000000be700000)
>  to   space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
> 0x00000000be7f0000)
> concurrent mark-sweep generation total 18432000K, used 12020225K 
> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
> concurrent-mark-sweep perm gen total 262144K, used 112106K 
> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
> }
> , 3,8911113 secs]
> 620682,667: [GC {Heap before gc invocations=58589:
> par new generation   total 1023040K, used 1022080K [0x000000007fff0000, 
> 0x00000000be7f0000, 0x00000000be7f0000)
>  eden space 1022080K, 100% used [0x000000007fff0000, 0x00000000be610000, 
> 0x00000000be610000)
>  from space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
> 0x00000000be700000)
>  to   space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
> 0x00000000be7f0000)
> concurrent mark-sweep generation total 18432000K, used 12020225K 
> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
> concurrent-mark-sweep perm gen total 262144K, used 112112K 
> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
> 620682,668: [ParNew: 1022080K->0K(1023040K), 9,4186088 secs] 
> 13042305K->12214571K(19455040K) icms_dc=10 Heap after gc invocations=58590:
> par new generation   total 1023040K, used 0K [0x000000007fff0000, 
> 0x00000000be7f0000, 0x00000000be7f0000)
>  eden space 1022080K,   0% used [0x000000007fff0000, 0x000000007fff0000, 
> 0x00000000be610000)
>  from space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
> 0x00000000be7f0000)
>  to   space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
> 0x00000000be700000)
> concurrent mark-sweep generation total 18432000K, used 12214571K 
> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
> concurrent-mark-sweep perm gen total 262144K, used 112112K 
> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
> }
> , 9,4191551 secs]
> 
> We also have problems with occational promotion failed, which I assume 
> is due to fragmentation.
> I have attatched a longer log, which shows this behaviour and ends with 
> a promotion failed.
> 
> Does anybody have a glue as to what is going wrong, and how we can 
> change our gc configuration to reduce the problems?
> Any help is greatly appreciated!
> 
> Regards
> 
> Kaj Nygren
> kaj at mag.se
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use



From Jon.Masamitsu at Sun.COM  Wed May  6 15:06:52 2009
From: Jon.Masamitsu at Sun.COM (Jon Masamitsu)
Date: Wed, 06 May 2009 15:06:52 -0700
Subject: Strange long ParNew collections
In-Reply-To: <4A017D35.7000208@mag.se>
References: <4A017D35.7000208@mag.se>
Message-ID: <4A0209FC.3030106@Sun.COM>

Kaj,

I don't see anything in the logs that points to
a specific problem.

Why are you using CMSIncrementalMode?

By default the CMS collector promotes all
objects that survive 1 minor collection
(into the tenured generation).  You might
find it useful to use larger survivor
spaces and a larger tenuring threshold.
See

http://blogs.sun.com/jonthecollector/entry/the_fault_with_defaults

for some suggestions of values to try.  You might also benefit
from a larger young.  Have you tried larger values? 2g? Or maybe
even 4g

Jon

On 05/06/09 05:06, Kaj Nygren wrote:
> Hi,
> 
> We are having problems with occational long ParNew pauses in our 
> application running on a Jboss server. The server runs well for most of 
> the time but sometimes we get ParNew collections that can take up to 120 
> seconds. Most of the time we have several consecutive slow collections, 
> which are more or less killing the server under high load.
> 
> We are using JDK 1.5.0_16 on Windows Server 2003, and are using ParNew + 
> Concurrent CMS.
> 
> Our GC configuration is:
> 
> wrapper.java.additional.1=-Dprogram.name=run.bat
> wrapper.java.additional.2=-server
> wrapper.java.additional.3=-Xms19000m
> wrapper.java.additional.4=-Xmx19000m
> wrapper.java.additional.5="-XX:NewSize=1000m"
> wrapper.java.additional.6="-XX:MaxNewSize=1000m"
> wrapper.java.additional.7="-XX:MaxPermSize=256m"
> wrapper.java.additional.8="-XX:PermSize=256m"
> wrapper.java.additional.11="-Xloggc:c:\mygclog.gc"
> wrapper.java.additional.12="-XX:+UseConcMarkSweepGC"
> wrapper.java.additional.13="-XX:+PrintGCDetails"
> wrapper.java.additional.14="-XX:-PrintTenuringDistribution"
> wrapper.java.additional.15="-XX:+PrintHeapAtGC"
> wrapper.java.additional.16="-XX:+DisableExplicitGC"
> wrapper.java.additional.17="-XX:+PrintGCTimeStamps"
> wrapper.java.additional.18="-XX:CMSInitiatingOccupancyFraction=50"
> wrapper.java.additional.19="-XX:+CMSIncrementalMode"
> wrapper.java.additional.20="-XX:+CMSIncrementalPacing"
> wrapper.java.additional.21="-verbose:gc"
> 
> Usually ParNew is quick as can be seen here:
> 
> 614276,653: [ParNew: 935058K->0K(1023040K), 0,0624699 secs] 
> 12383872K->11459948K(19455040K) icms_dc=22 Heap after gc invocations=57822:
> par new generation   total 1023040K, used 0K [0x000000007fff0000, 
> 0x00000000be7f0000, 0x00000000be7f0000)
>  eden space 1022080K,   0% used [0x000000007fff0000, 0x000000007fff0000, 
> 0x00000000be610000)
>  from space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
> 0x00000000be7f0000)
>  to   space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
> 0x00000000be700000)
> concurrent mark-sweep generation total 18432000K, used 11459948K 
> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
> concurrent-mark-sweep perm gen total 262144K, used 112069K 
> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
> }
> , 0,0628860 secs]
> 
> But suddenly it becomes really slow:
> 
> 620659,717: [ParNew: 1022080K->0K(1023040K), 1,3313817 secs] 
> 12688913K->11692925K(19455040K) icms_dc=10 Heap after gc invocations=58587:
> par new generation   total 1023040K, used 0K [0x000000007fff0000, 
> 0x00000000be7f0000, 0x00000000be7f0000)
>  eden space 1022080K,   0% used [0x000000007fff0000, 0x000000007fff0000, 
> 0x00000000be610000)
>  from space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
> 0x00000000be700000)
>  to   space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
> 0x00000000be7f0000)
> concurrent mark-sweep generation total 18432000K, used 11692925K 
> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
> concurrent-mark-sweep perm gen total 262144K, used 112098K 
> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
> }
> , 1,3319554 secs]
> 620663,221: [GC {Heap before gc invocations=58587:
> par new generation   total 1023040K, used 1022080K [0x000000007fff0000, 
> 0x00000000be7f0000, 0x00000000be7f0000)
>  eden space 1022080K, 100% used [0x000000007fff0000, 0x00000000be610000, 
> 0x00000000be610000)
>  from space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
> 0x00000000be700000)
>  to   space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
> 0x00000000be7f0000)
> concurrent mark-sweep generation total 18432000K, used 11692925K 
> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
> concurrent-mark-sweep perm gen total 262144K, used 112106K 
> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
> 620663,222: [ParNew: 1022080K->0K(1023040K), 10,2909893 secs] 
> 12715005K->11936864K(19455040K) icms_dc=10 Heap after gc invocations=58588:
> par new generation   total 1023040K, used 0K [0x000000007fff0000, 
> 0x00000000be7f0000, 0x00000000be7f0000)
>  eden space 1022080K,   0% used [0x000000007fff0000, 0x000000007fff0000, 
> 0x00000000be610000)
>  from space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
> 0x00000000be7f0000)
>  to   space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
> 0x00000000be700000)
> concurrent mark-sweep generation total 18432000K, used 11936864K 
> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
> concurrent-mark-sweep perm gen total 262144K, used 112106K 
> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
> }
> , 10,2915548 secs]
> 620676,392: [GC {Heap before gc invocations=58588:
> par new generation   total 1023040K, used 1022080K [0x000000007fff0000, 
> 0x00000000be7f0000, 0x00000000be7f0000)
>  eden space 1022080K, 100% used [0x000000007fff0000, 0x00000000be610000, 
> 0x00000000be610000)
>  from space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
> 0x00000000be7f0000)
>  to   space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
> 0x00000000be700000)
> concurrent mark-sweep generation total 18432000K, used 11936864K 
> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
> concurrent-mark-sweep perm gen total 262144K, used 112106K 
> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
> 620676,392: [ParNew: 1022080K->0K(1023040K), 3,8905950 secs] 
> 12958944K->12020225K(19455040K) icms_dc=10 Heap after gc invocations=58589:
> par new generation   total 1023040K, used 0K [0x000000007fff0000, 
> 0x00000000be7f0000, 0x00000000be7f0000)
>  eden space 1022080K,   0% used [0x000000007fff0000, 0x000000007fff0000, 
> 0x00000000be610000)
>  from space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
> 0x00000000be700000)
>  to   space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
> 0x00000000be7f0000)
> concurrent mark-sweep generation total 18432000K, used 12020225K 
> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
> concurrent-mark-sweep perm gen total 262144K, used 112106K 
> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
> }
> , 3,8911113 secs]
> 620682,667: [GC {Heap before gc invocations=58589:
> par new generation   total 1023040K, used 1022080K [0x000000007fff0000, 
> 0x00000000be7f0000, 0x00000000be7f0000)
>  eden space 1022080K, 100% used [0x000000007fff0000, 0x00000000be610000, 
> 0x00000000be610000)
>  from space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
> 0x00000000be700000)
>  to   space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
> 0x00000000be7f0000)
> concurrent mark-sweep generation total 18432000K, used 12020225K 
> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
> concurrent-mark-sweep perm gen total 262144K, used 112112K 
> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
> 620682,668: [ParNew: 1022080K->0K(1023040K), 9,4186088 secs] 
> 13042305K->12214571K(19455040K) icms_dc=10 Heap after gc invocations=58590:
> par new generation   total 1023040K, used 0K [0x000000007fff0000, 
> 0x00000000be7f0000, 0x00000000be7f0000)
>  eden space 1022080K,   0% used [0x000000007fff0000, 0x000000007fff0000, 
> 0x00000000be610000)
>  from space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
> 0x00000000be7f0000)
>  to   space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
> 0x00000000be700000)
> concurrent mark-sweep generation total 18432000K, used 12214571K 
> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
> concurrent-mark-sweep perm gen total 262144K, used 112112K 
> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
> }
> , 9,4191551 secs]
> 
> We also have problems with occational promotion failed, which I assume 
> is due to fragmentation.
> I have attatched a longer log, which shows this behaviour and ends with 
> a promotion failed.
> 
> Does anybody have a glue as to what is going wrong, and how we can 
> change our gc configuration to reduce the problems?
> Any help is greatly appreciated!
> 
> Regards
> 
> Kaj Nygren
> kaj at mag.se
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use


From kaj at mag.se  Thu May  7 01:06:37 2009
From: kaj at mag.se (Kaj Nygren)
Date: Thu, 07 May 2009 10:06:37 +0200
Subject: Strange long ParNew collections
In-Reply-To: <4A02078C.4080904@Sun.COM>
References: <4A017D35.7000208@mag.se> <4A02078C.4080904@Sun.COM>
Message-ID: <4A02968D.3050601@mag.se>

Hi,

Thank you for your information!

I will try the settings and get back within a few days to let you now if 
that solved the problem.

/Kaj

Y.S.Ramakrishna at Sun.COM skrev:
> This may or may not be a cause of the sudden spikes in
> ParNew collection, but I'd suggest using the survivor
> spaces to age objects for at least a few collections
> before promoting to the old gen. This reduces pressure
> on the old gen allocation (during scavenge) improving
> scavenges and on CMS collection as well.
>
> I'd suggest trying -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=4
> to start off with and iterate based on data from -XX:+PrintTenuringDistribution.
>
> It is possible that the promotion of very short-lived objects fragments
> the old gen and causes old gen allocation (for promotion) to slow down.
> It can also cause lots of mutation in the old gen and adversely affect
> card-scanning times further slowing down scavenges.
>
> -- ramki
>
> On 05/06/09 05:06, Kaj Nygren wrote:
>   
>> Hi,
>>
>> We are having problems with occational long ParNew pauses in our 
>> application running on a Jboss server. The server runs well for most of 
>> the time but sometimes we get ParNew collections that can take up to 120 
>> seconds. Most of the time we have several consecutive slow collections, 
>> which are more or less killing the server under high load.
>>
>> We are using JDK 1.5.0_16 on Windows Server 2003, and are using ParNew + 
>> Concurrent CMS.
>>
>> Our GC configuration is:
>>
>> wrapper.java.additional.1=-Dprogram.name=run.bat
>> wrapper.java.additional.2=-server
>> wrapper.java.additional.3=-Xms19000m
>> wrapper.java.additional.4=-Xmx19000m
>> wrapper.java.additional.5="-XX:NewSize=1000m"
>> wrapper.java.additional.6="-XX:MaxNewSize=1000m"
>> wrapper.java.additional.7="-XX:MaxPermSize=256m"
>> wrapper.java.additional.8="-XX:PermSize=256m"
>> wrapper.java.additional.11="-Xloggc:c:\mygclog.gc"
>> wrapper.java.additional.12="-XX:+UseConcMarkSweepGC"
>> wrapper.java.additional.13="-XX:+PrintGCDetails"
>> wrapper.java.additional.14="-XX:-PrintTenuringDistribution"
>> wrapper.java.additional.15="-XX:+PrintHeapAtGC"
>> wrapper.java.additional.16="-XX:+DisableExplicitGC"
>> wrapper.java.additional.17="-XX:+PrintGCTimeStamps"
>> wrapper.java.additional.18="-XX:CMSInitiatingOccupancyFraction=50"
>> wrapper.java.additional.19="-XX:+CMSIncrementalMode"
>> wrapper.java.additional.20="-XX:+CMSIncrementalPacing"
>> wrapper.java.additional.21="-verbose:gc"
>>
>> Usually ParNew is quick as can be seen here:
>>
>> 614276,653: [ParNew: 935058K->0K(1023040K), 0,0624699 secs] 
>> 12383872K->11459948K(19455040K) icms_dc=22 Heap after gc invocations=57822:
>> par new generation   total 1023040K, used 0K [0x000000007fff0000, 
>> 0x00000000be7f0000, 0x00000000be7f0000)
>>  eden space 1022080K,   0% used [0x000000007fff0000, 0x000000007fff0000, 
>> 0x00000000be610000)
>>  from space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>> 0x00000000be7f0000)
>>  to   space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>> 0x00000000be700000)
>> concurrent mark-sweep generation total 18432000K, used 11459948K 
>> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>> concurrent-mark-sweep perm gen total 262144K, used 112069K 
>> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>> }
>> , 0,0628860 secs]
>>
>> But suddenly it becomes really slow:
>>
>> 620659,717: [ParNew: 1022080K->0K(1023040K), 1,3313817 secs] 
>> 12688913K->11692925K(19455040K) icms_dc=10 Heap after gc invocations=58587:
>> par new generation   total 1023040K, used 0K [0x000000007fff0000, 
>> 0x00000000be7f0000, 0x00000000be7f0000)
>>  eden space 1022080K,   0% used [0x000000007fff0000, 0x000000007fff0000, 
>> 0x00000000be610000)
>>  from space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>> 0x00000000be700000)
>>  to   space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>> 0x00000000be7f0000)
>> concurrent mark-sweep generation total 18432000K, used 11692925K 
>> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>> concurrent-mark-sweep perm gen total 262144K, used 112098K 
>> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>> }
>> , 1,3319554 secs]
>> 620663,221: [GC {Heap before gc invocations=58587:
>> par new generation   total 1023040K, used 1022080K [0x000000007fff0000, 
>> 0x00000000be7f0000, 0x00000000be7f0000)
>>  eden space 1022080K, 100% used [0x000000007fff0000, 0x00000000be610000, 
>> 0x00000000be610000)
>>  from space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>> 0x00000000be700000)
>>  to   space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>> 0x00000000be7f0000)
>> concurrent mark-sweep generation total 18432000K, used 11692925K 
>> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>> concurrent-mark-sweep perm gen total 262144K, used 112106K 
>> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>> 620663,222: [ParNew: 1022080K->0K(1023040K), 10,2909893 secs] 
>> 12715005K->11936864K(19455040K) icms_dc=10 Heap after gc invocations=58588:
>> par new generation   total 1023040K, used 0K [0x000000007fff0000, 
>> 0x00000000be7f0000, 0x00000000be7f0000)
>>  eden space 1022080K,   0% used [0x000000007fff0000, 0x000000007fff0000, 
>> 0x00000000be610000)
>>  from space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>> 0x00000000be7f0000)
>>  to   space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>> 0x00000000be700000)
>> concurrent mark-sweep generation total 18432000K, used 11936864K 
>> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>> concurrent-mark-sweep perm gen total 262144K, used 112106K 
>> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>> }
>> , 10,2915548 secs]
>> 620676,392: [GC {Heap before gc invocations=58588:
>> par new generation   total 1023040K, used 1022080K [0x000000007fff0000, 
>> 0x00000000be7f0000, 0x00000000be7f0000)
>>  eden space 1022080K, 100% used [0x000000007fff0000, 0x00000000be610000, 
>> 0x00000000be610000)
>>  from space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>> 0x00000000be7f0000)
>>  to   space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>> 0x00000000be700000)
>> concurrent mark-sweep generation total 18432000K, used 11936864K 
>> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>> concurrent-mark-sweep perm gen total 262144K, used 112106K 
>> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>> 620676,392: [ParNew: 1022080K->0K(1023040K), 3,8905950 secs] 
>> 12958944K->12020225K(19455040K) icms_dc=10 Heap after gc invocations=58589:
>> par new generation   total 1023040K, used 0K [0x000000007fff0000, 
>> 0x00000000be7f0000, 0x00000000be7f0000)
>>  eden space 1022080K,   0% used [0x000000007fff0000, 0x000000007fff0000, 
>> 0x00000000be610000)
>>  from space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>> 0x00000000be700000)
>>  to   space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>> 0x00000000be7f0000)
>> concurrent mark-sweep generation total 18432000K, used 12020225K 
>> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>> concurrent-mark-sweep perm gen total 262144K, used 112106K 
>> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>> }
>> , 3,8911113 secs]
>> 620682,667: [GC {Heap before gc invocations=58589:
>> par new generation   total 1023040K, used 1022080K [0x000000007fff0000, 
>> 0x00000000be7f0000, 0x00000000be7f0000)
>>  eden space 1022080K, 100% used [0x000000007fff0000, 0x00000000be610000, 
>> 0x00000000be610000)
>>  from space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>> 0x00000000be700000)
>>  to   space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>> 0x00000000be7f0000)
>> concurrent mark-sweep generation total 18432000K, used 12020225K 
>> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>> concurrent-mark-sweep perm gen total 262144K, used 112112K 
>> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>> 620682,668: [ParNew: 1022080K->0K(1023040K), 9,4186088 secs] 
>> 13042305K->12214571K(19455040K) icms_dc=10 Heap after gc invocations=58590:
>> par new generation   total 1023040K, used 0K [0x000000007fff0000, 
>> 0x00000000be7f0000, 0x00000000be7f0000)
>>  eden space 1022080K,   0% used [0x000000007fff0000, 0x000000007fff0000, 
>> 0x00000000be610000)
>>  from space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>> 0x00000000be7f0000)
>>  to   space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>> 0x00000000be700000)
>> concurrent mark-sweep generation total 18432000K, used 12214571K 
>> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>> concurrent-mark-sweep perm gen total 262144K, used 112112K 
>> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>> }
>> , 9,4191551 secs]
>>
>> We also have problems with occational promotion failed, which I assume 
>> is due to fragmentation.
>> I have attatched a longer log, which shows this behaviour and ends with 
>> a promotion failed.
>>
>> Does anybody have a glue as to what is going wrong, and how we can 
>> change our gc configuration to reduce the problems?
>> Any help is greatly appreciated!
>>
>> Regards
>>
>> Kaj Nygren
>> kaj at mag.se
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>     
>
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>   



From kaj at mag.se  Thu May  7 01:45:56 2009
From: kaj at mag.se (Kaj Nygren)
Date: Thu, 07 May 2009 10:45:56 +0200
Subject: Strange long ParNew collections
In-Reply-To: <4A0209FC.3030106@Sun.COM>
References: <4A017D35.7000208@mag.se> <4A0209FC.3030106@Sun.COM>
Message-ID: <4A029FC4.1090303@mag.se>

Hi Jon,

Thanks for your input!

Well it was around a year ago that we turned on incremental mode, but if 
memory serves me right we had problems with getting "Concurrent Mode 
Failure" rather often (probably CMS kicking in to late and not keeping up).

It is an 8 processor machine but the application is keeping it rather 
busy. We searched around for possible solutions and found several 
sources that recommended running iCMS, and tuning the threshold to start 
CMS gc earlier. We turned on iCMS and that helped. The application has 
changed a bit since then so maybe that's not needed anymore though.  
(This might have been a bogus change by us though, gc internals is still 
a bit voodoo for me, although I've tried to read up as much as I can 
about it :)

We will start by specifying a SurvivorRatio and a TenuringThreshold, and 
see if that helps. After that if that's needed I'll try increasing the 
spaces as well.

/Kaj


Jon Masamitsu skrev:
> Kaj,
>
> I don't see anything in the logs that points to
> a specific problem.
>
> Why are you using CMSIncrementalMode?
>
> By default the CMS collector promotes all
> objects that survive 1 minor collection
> (into the tenured generation).  You might
> find it useful to use larger survivor
> spaces and a larger tenuring threshold.
> See
>
> http://blogs.sun.com/jonthecollector/entry/the_fault_with_defaults
>
> for some suggestions of values to try.  You might also benefit
> from a larger young.  Have you tried larger values? 2g? Or maybe
> even 4g
>
> Jon
>
> On 05/06/09 05:06, Kaj Nygren wrote:
>   
>> Hi,
>>
>> We are having problems with occational long ParNew pauses in our 
>> application running on a Jboss server. The server runs well for most of 
>> the time but sometimes we get ParNew collections that can take up to 120 
>> seconds. Most of the time we have several consecutive slow collections, 
>> which are more or less killing the server under high load.
>>
>> We are using JDK 1.5.0_16 on Windows Server 2003, and are using ParNew + 
>> Concurrent CMS.
>>
>> Our GC configuration is:
>>
>> wrapper.java.additional.1=-Dprogram.name=run.bat
>> wrapper.java.additional.2=-server
>> wrapper.java.additional.3=-Xms19000m
>> wrapper.java.additional.4=-Xmx19000m
>> wrapper.java.additional.5="-XX:NewSize=1000m"
>> wrapper.java.additional.6="-XX:MaxNewSize=1000m"
>> wrapper.java.additional.7="-XX:MaxPermSize=256m"
>> wrapper.java.additional.8="-XX:PermSize=256m"
>> wrapper.java.additional.11="-Xloggc:c:\mygclog.gc"
>> wrapper.java.additional.12="-XX:+UseConcMarkSweepGC"
>> wrapper.java.additional.13="-XX:+PrintGCDetails"
>> wrapper.java.additional.14="-XX:-PrintTenuringDistribution"
>> wrapper.java.additional.15="-XX:+PrintHeapAtGC"
>> wrapper.java.additional.16="-XX:+DisableExplicitGC"
>> wrapper.java.additional.17="-XX:+PrintGCTimeStamps"
>> wrapper.java.additional.18="-XX:CMSInitiatingOccupancyFraction=50"
>> wrapper.java.additional.19="-XX:+CMSIncrementalMode"
>> wrapper.java.additional.20="-XX:+CMSIncrementalPacing"
>> wrapper.java.additional.21="-verbose:gc"
>>
>> Usually ParNew is quick as can be seen here:
>>
>> 614276,653: [ParNew: 935058K->0K(1023040K), 0,0624699 secs] 
>> 12383872K->11459948K(19455040K) icms_dc=22 Heap after gc invocations=57822:
>> par new generation   total 1023040K, used 0K [0x000000007fff0000, 
>> 0x00000000be7f0000, 0x00000000be7f0000)
>>  eden space 1022080K,   0% used [0x000000007fff0000, 0x000000007fff0000, 
>> 0x00000000be610000)
>>  from space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>> 0x00000000be7f0000)
>>  to   space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>> 0x00000000be700000)
>> concurrent mark-sweep generation total 18432000K, used 11459948K 
>> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>> concurrent-mark-sweep perm gen total 262144K, used 112069K 
>> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>> }
>> , 0,0628860 secs]
>>
>> But suddenly it becomes really slow:
>>
>> 620659,717: [ParNew: 1022080K->0K(1023040K), 1,3313817 secs] 
>> 12688913K->11692925K(19455040K) icms_dc=10 Heap after gc invocations=58587:
>> par new generation   total 1023040K, used 0K [0x000000007fff0000, 
>> 0x00000000be7f0000, 0x00000000be7f0000)
>>  eden space 1022080K,   0% used [0x000000007fff0000, 0x000000007fff0000, 
>> 0x00000000be610000)
>>  from space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>> 0x00000000be700000)
>>  to   space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>> 0x00000000be7f0000)
>> concurrent mark-sweep generation total 18432000K, used 11692925K 
>> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>> concurrent-mark-sweep perm gen total 262144K, used 112098K 
>> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>> }
>> , 1,3319554 secs]
>> 620663,221: [GC {Heap before gc invocations=58587:
>> par new generation   total 1023040K, used 1022080K [0x000000007fff0000, 
>> 0x00000000be7f0000, 0x00000000be7f0000)
>>  eden space 1022080K, 100% used [0x000000007fff0000, 0x00000000be610000, 
>> 0x00000000be610000)
>>  from space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>> 0x00000000be700000)
>>  to   space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>> 0x00000000be7f0000)
>> concurrent mark-sweep generation total 18432000K, used 11692925K 
>> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>> concurrent-mark-sweep perm gen total 262144K, used 112106K 
>> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>> 620663,222: [ParNew: 1022080K->0K(1023040K), 10,2909893 secs] 
>> 12715005K->11936864K(19455040K) icms_dc=10 Heap after gc invocations=58588:
>> par new generation   total 1023040K, used 0K [0x000000007fff0000, 
>> 0x00000000be7f0000, 0x00000000be7f0000)
>>  eden space 1022080K,   0% used [0x000000007fff0000, 0x000000007fff0000, 
>> 0x00000000be610000)
>>  from space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>> 0x00000000be7f0000)
>>  to   space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>> 0x00000000be700000)
>> concurrent mark-sweep generation total 18432000K, used 11936864K 
>> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>> concurrent-mark-sweep perm gen total 262144K, used 112106K 
>> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>> }
>> , 10,2915548 secs]
>> 620676,392: [GC {Heap before gc invocations=58588:
>> par new generation   total 1023040K, used 1022080K [0x000000007fff0000, 
>> 0x00000000be7f0000, 0x00000000be7f0000)
>>  eden space 1022080K, 100% used [0x000000007fff0000, 0x00000000be610000, 
>> 0x00000000be610000)
>>  from space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>> 0x00000000be7f0000)
>>  to   space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>> 0x00000000be700000)
>> concurrent mark-sweep generation total 18432000K, used 11936864K 
>> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>> concurrent-mark-sweep perm gen total 262144K, used 112106K 
>> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>> 620676,392: [ParNew: 1022080K->0K(1023040K), 3,8905950 secs] 
>> 12958944K->12020225K(19455040K) icms_dc=10 Heap after gc invocations=58589:
>> par new generation   total 1023040K, used 0K [0x000000007fff0000, 
>> 0x00000000be7f0000, 0x00000000be7f0000)
>>  eden space 1022080K,   0% used [0x000000007fff0000, 0x000000007fff0000, 
>> 0x00000000be610000)
>>  from space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>> 0x00000000be700000)
>>  to   space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>> 0x00000000be7f0000)
>> concurrent mark-sweep generation total 18432000K, used 12020225K 
>> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>> concurrent-mark-sweep perm gen total 262144K, used 112106K 
>> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>> }
>> , 3,8911113 secs]
>> 620682,667: [GC {Heap before gc invocations=58589:
>> par new generation   total 1023040K, used 1022080K [0x000000007fff0000, 
>> 0x00000000be7f0000, 0x00000000be7f0000)
>>  eden space 1022080K, 100% used [0x000000007fff0000, 0x00000000be610000, 
>> 0x00000000be610000)
>>  from space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>> 0x00000000be700000)
>>  to   space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>> 0x00000000be7f0000)
>> concurrent mark-sweep generation total 18432000K, used 12020225K 
>> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>> concurrent-mark-sweep perm gen total 262144K, used 112112K 
>> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>> 620682,668: [ParNew: 1022080K->0K(1023040K), 9,4186088 secs] 
>> 13042305K->12214571K(19455040K) icms_dc=10 Heap after gc invocations=58590:
>> par new generation   total 1023040K, used 0K [0x000000007fff0000, 
>> 0x00000000be7f0000, 0x00000000be7f0000)
>>  eden space 1022080K,   0% used [0x000000007fff0000, 0x000000007fff0000, 
>> 0x00000000be610000)
>>  from space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>> 0x00000000be7f0000)
>>  to   space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>> 0x00000000be700000)
>> concurrent mark-sweep generation total 18432000K, used 12214571K 
>> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>> concurrent-mark-sweep perm gen total 262144K, used 112112K 
>> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>> }
>> , 9,4191551 secs]
>>
>> We also have problems with occational promotion failed, which I assume 
>> is due to fragmentation.
>> I have attatched a longer log, which shows this behaviour and ends with 
>> a promotion failed.
>>
>> Does anybody have a glue as to what is going wrong, and how we can 
>> change our gc configuration to reduce the problems?
>> Any help is greatly appreciated!
>>
>> Regards
>>
>> Kaj Nygren
>> kaj at mag.se
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>     
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>   



From Jon.Masamitsu at Sun.COM  Thu May  7 09:10:22 2009
From: Jon.Masamitsu at Sun.COM (Jon Masamitsu)
Date: Thu, 07 May 2009 09:10:22 -0700
Subject: Strange long ParNew collections
In-Reply-To: <4A029FC4.1090303@mag.se>
References: <4A017D35.7000208@mag.se> <4A0209FC.3030106@Sun.COM>
	<4A029FC4.1090303@mag.se>
Message-ID: <4A0307EE.1050204@Sun.COM>

Kaj,

iCMS may have worked better for you because it might
kick off a collection earlier.  But iCMS was intended for
use on 1 or 2 processor machines where doing work
concurrently would hog one (maybe the only one)
processor so would not feel very concurrent to the
application.  I think your use of the lower initiating
occupancy would have gotten you the result you needed.
iCMS generally runs longer and leaves more floating
garbage (dead objects that are not recognized as
garbage until the next collection).  I'd suggest that you
do some testing without the incremental mode and
with the larger survivor spaces and tenuring threshold
and see how that does.  The concurrent mode failures
can often be reduced when the survivor spaces are
used.

Jon

Kaj Nygren wrote:

>Hi Jon,
>
>Thanks for your input!
>
>Well it was around a year ago that we turned on incremental mode, but if 
>memory serves me right we had problems with getting "Concurrent Mode 
>Failure" rather often (probably CMS kicking in to late and not keeping up).
>
>It is an 8 processor machine but the application is keeping it rather 
>busy. We searched around for possible solutions and found several 
>sources that recommended running iCMS, and tuning the threshold to start 
>CMS gc earlier. We turned on iCMS and that helped. The application has 
>changed a bit since then so maybe that's not needed anymore though.  
>(This might have been a bogus change by us though, gc internals is still 
>a bit voodoo for me, although I've tried to read up as much as I can 
>about it :)
>
>We will start by specifying a SurvivorRatio and a TenuringThreshold, and 
>see if that helps. After that if that's needed I'll try increasing the 
>spaces as well.
>
>/Kaj
>
>
>Jon Masamitsu skrev:
>  
>
>>Kaj,
>>
>>I don't see anything in the logs that points to
>>a specific problem.
>>
>>Why are you using CMSIncrementalMode?
>>
>>By default the CMS collector promotes all
>>objects that survive 1 minor collection
>>(into the tenured generation).  You might
>>find it useful to use larger survivor
>>spaces and a larger tenuring threshold.
>>See
>>
>>http://blogs.sun.com/jonthecollector/entry/the_fault_with_defaults
>>
>>for some suggestions of values to try.  You might also benefit
>>from a larger young.  Have you tried larger values? 2g? Or maybe
>>even 4g
>>
>>Jon
>>
>>On 05/06/09 05:06, Kaj Nygren wrote:
>>  
>>    
>>
>>>Hi,
>>>
>>>We are having problems with occational long ParNew pauses in our 
>>>application running on a Jboss server. The server runs well for most of 
>>>the time but sometimes we get ParNew collections that can take up to 120 
>>>seconds. Most of the time we have several consecutive slow collections, 
>>>which are more or less killing the server under high load.
>>>
>>>We are using JDK 1.5.0_16 on Windows Server 2003, and are using ParNew + 
>>>Concurrent CMS.
>>>
>>>Our GC configuration is:
>>>
>>>wrapper.java.additional.1=-Dprogram.name=run.bat
>>>wrapper.java.additional.2=-server
>>>wrapper.java.additional.3=-Xms19000m
>>>wrapper.java.additional.4=-Xmx19000m
>>>wrapper.java.additional.5="-XX:NewSize=1000m"
>>>wrapper.java.additional.6="-XX:MaxNewSize=1000m"
>>>wrapper.java.additional.7="-XX:MaxPermSize=256m"
>>>wrapper.java.additional.8="-XX:PermSize=256m"
>>>wrapper.java.additional.11="-Xloggc:c:\mygclog.gc"
>>>wrapper.java.additional.12="-XX:+UseConcMarkSweepGC"
>>>wrapper.java.additional.13="-XX:+PrintGCDetails"
>>>wrapper.java.additional.14="-XX:-PrintTenuringDistribution"
>>>wrapper.java.additional.15="-XX:+PrintHeapAtGC"
>>>wrapper.java.additional.16="-XX:+DisableExplicitGC"
>>>wrapper.java.additional.17="-XX:+PrintGCTimeStamps"
>>>wrapper.java.additional.18="-XX:CMSInitiatingOccupancyFraction=50"
>>>wrapper.java.additional.19="-XX:+CMSIncrementalMode"
>>>wrapper.java.additional.20="-XX:+CMSIncrementalPacing"
>>>wrapper.java.additional.21="-verbose:gc"
>>>
>>>Usually ParNew is quick as can be seen here:
>>>
>>>614276,653: [ParNew: 935058K->0K(1023040K), 0,0624699 secs] 
>>>12383872K->11459948K(19455040K) icms_dc=22 Heap after gc invocations=57822:
>>>par new generation   total 1023040K, used 0K [0x000000007fff0000, 
>>>0x00000000be7f0000, 0x00000000be7f0000)
>>> eden space 1022080K,   0% used [0x000000007fff0000, 0x000000007fff0000, 
>>>0x00000000be610000)
>>> from space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>>>0x00000000be7f0000)
>>> to   space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>>>0x00000000be700000)
>>>concurrent mark-sweep generation total 18432000K, used 11459948K 
>>>[0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>>>concurrent-mark-sweep perm gen total 262144K, used 112069K 
>>>[0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>>>}
>>>, 0,0628860 secs]
>>>
>>>But suddenly it becomes really slow:
>>>
>>>620659,717: [ParNew: 1022080K->0K(1023040K), 1,3313817 secs] 
>>>12688913K->11692925K(19455040K) icms_dc=10 Heap after gc invocations=58587:
>>>par new generation   total 1023040K, used 0K [0x000000007fff0000, 
>>>0x00000000be7f0000, 0x00000000be7f0000)
>>> eden space 1022080K,   0% used [0x000000007fff0000, 0x000000007fff0000, 
>>>0x00000000be610000)
>>> from space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>>>0x00000000be700000)
>>> to   space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>>>0x00000000be7f0000)
>>>concurrent mark-sweep generation total 18432000K, used 11692925K 
>>>[0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>>>concurrent-mark-sweep perm gen total 262144K, used 112098K 
>>>[0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>>>}
>>>, 1,3319554 secs]
>>>620663,221: [GC {Heap before gc invocations=58587:
>>>par new generation   total 1023040K, used 1022080K [0x000000007fff0000, 
>>>0x00000000be7f0000, 0x00000000be7f0000)
>>> eden space 1022080K, 100% used [0x000000007fff0000, 0x00000000be610000, 
>>>0x00000000be610000)
>>> from space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>>>0x00000000be700000)
>>> to   space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>>>0x00000000be7f0000)
>>>concurrent mark-sweep generation total 18432000K, used 11692925K 
>>>[0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>>>concurrent-mark-sweep perm gen total 262144K, used 112106K 
>>>[0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>>>620663,222: [ParNew: 1022080K->0K(1023040K), 10,2909893 secs] 
>>>12715005K->11936864K(19455040K) icms_dc=10 Heap after gc invocations=58588:
>>>par new generation   total 1023040K, used 0K [0x000000007fff0000, 
>>>0x00000000be7f0000, 0x00000000be7f0000)
>>> eden space 1022080K,   0% used [0x000000007fff0000, 0x000000007fff0000, 
>>>0x00000000be610000)
>>> from space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>>>0x00000000be7f0000)
>>> to   space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>>>0x00000000be700000)
>>>concurrent mark-sweep generation total 18432000K, used 11936864K 
>>>[0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>>>concurrent-mark-sweep perm gen total 262144K, used 112106K 
>>>[0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>>>}
>>>, 10,2915548 secs]
>>>620676,392: [GC {Heap before gc invocations=58588:
>>>par new generation   total 1023040K, used 1022080K [0x000000007fff0000, 
>>>0x00000000be7f0000, 0x00000000be7f0000)
>>> eden space 1022080K, 100% used [0x000000007fff0000, 0x00000000be610000, 
>>>0x00000000be610000)
>>> from space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>>>0x00000000be7f0000)
>>> to   space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>>>0x00000000be700000)
>>>concurrent mark-sweep generation total 18432000K, used 11936864K 
>>>[0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>>>concurrent-mark-sweep perm gen total 262144K, used 112106K 
>>>[0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>>>620676,392: [ParNew: 1022080K->0K(1023040K), 3,8905950 secs] 
>>>12958944K->12020225K(19455040K) icms_dc=10 Heap after gc invocations=58589:
>>>par new generation   total 1023040K, used 0K [0x000000007fff0000, 
>>>0x00000000be7f0000, 0x00000000be7f0000)
>>> eden space 1022080K,   0% used [0x000000007fff0000, 0x000000007fff0000, 
>>>0x00000000be610000)
>>> from space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>>>0x00000000be700000)
>>> to   space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>>>0x00000000be7f0000)
>>>concurrent mark-sweep generation total 18432000K, used 12020225K 
>>>[0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>>>concurrent-mark-sweep perm gen total 262144K, used 112106K 
>>>[0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>>>}
>>>, 3,8911113 secs]
>>>620682,667: [GC {Heap before gc invocations=58589:
>>>par new generation   total 1023040K, used 1022080K [0x000000007fff0000, 
>>>0x00000000be7f0000, 0x00000000be7f0000)
>>> eden space 1022080K, 100% used [0x000000007fff0000, 0x00000000be610000, 
>>>0x00000000be610000)
>>> from space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>>>0x00000000be700000)
>>> to   space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>>>0x00000000be7f0000)
>>>concurrent mark-sweep generation total 18432000K, used 12020225K 
>>>[0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>>>concurrent-mark-sweep perm gen total 262144K, used 112112K 
>>>[0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>>>620682,668: [ParNew: 1022080K->0K(1023040K), 9,4186088 secs] 
>>>13042305K->12214571K(19455040K) icms_dc=10 Heap after gc invocations=58590:
>>>par new generation   total 1023040K, used 0K [0x000000007fff0000, 
>>>0x00000000be7f0000, 0x00000000be7f0000)
>>> eden space 1022080K,   0% used [0x000000007fff0000, 0x000000007fff0000, 
>>>0x00000000be610000)
>>> from space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>>>0x00000000be7f0000)
>>> to   space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>>>0x00000000be700000)
>>>concurrent mark-sweep generation total 18432000K, used 12214571K 
>>>[0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>>>concurrent-mark-sweep perm gen total 262144K, used 112112K 
>>>[0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>>>}
>>>, 9,4191551 secs]
>>>
>>>We also have problems with occational promotion failed, which I assume 
>>>is due to fragmentation.
>>>I have attatched a longer log, which shows this behaviour and ends with 
>>>a promotion failed.
>>>
>>>Does anybody have a glue as to what is going wrong, and how we can 
>>>change our gc configuration to reduce the problems?
>>>Any help is greatly appreciated!
>>>
>>>Regards
>>>
>>>Kaj Nygren
>>>kaj at mag.se
>>>
>>>
>>>------------------------------------------------------------------------
>>>
>>>_______________________________________________
>>>hotspot-gc-use mailing list
>>>hotspot-gc-use at openjdk.java.net
>>>http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>>    
>>>      
>>>
>>_______________________________________________
>>hotspot-gc-use mailing list
>>hotspot-gc-use at openjdk.java.net
>>http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>>  
>>    
>>
>
>_______________________________________________
>hotspot-gc-use mailing list
>hotspot-gc-use at openjdk.java.net
>http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
>  
>


From kaj at mag.se  Tue May 19 04:27:40 2009
From: kaj at mag.se (Kaj Nygren)
Date: Tue, 19 May 2009 13:27:40 +0200
Subject: Strange long ParNew collections
In-Reply-To: <4A02078C.4080904@Sun.COM>
References: <4A017D35.7000208@mag.se> <4A02078C.4080904@Sun.COM>
Message-ID: <4A1297AC.5060205@mag.se>

Hi,

Thanks for your help. I wanted to give an update and have a couple of 
more questions :)

We did change to -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=4  and now 
the ParNew collections are working a lot better. We have also turned off 
incremental mode as Jon suggested.

Now we do have one problem left though, it seems that the CMS-GC gets 
stuck sometimes during parallell rescan. The output looks like this:

24004,062: [CMS-concurrent-sweep: 1,755/1,932 secs]
24004,062: [CMS-concurrent-reset-start]
24004,163: [CMS-concurrent-reset: 0,101/0,101 secs]
24006,163: [GC [1 CMS-initial-mark: 10348978K(18432000K)] 
10521837K(19353600K), 0,1055176 secs]
24006,269: [CMS-concurrent-mark-start]
24010,642: [CMS-concurrent-mark: 4,373/4,373 secs]
24010,642: [CMS-concurrent-preclean-start]
24010,730: [CMS-concurrent-preclean: 0,085/0,088 secs]
24010,730: [CMS-concurrent-abortable-preclean-start]
 CMS: abort preclean due to time 24011,824: 
[CMS-concurrent-abortable-preclean: 0,792/1,094 secs]
24011,828: [GC[YG occupancy: 332774 K (921600 K)]
24011,829: [Rescan (parallel)

Then the JVM hangs for more than 5 minutes at which time our watchdog 
determines that the server is hung and restarts it. It did happen with 
iCMS and apparently it still happens using CMS.

I searched around and found one similar report:

http://mail.openjdk.java.net/pipermail/hotspot-gc-use/2009-January/000327.html

I know this is just a guess, but could we be experiencing the same bug? 
If so, am I correct in interpreting the bug-reports that it should be 
fixed in 1.5.0_18? (we are running 1.5.0_16 at the moment).

/Kaj

Y.S.Ramakrishna at Sun.COM skrev:
> This may or may not be a cause of the sudden spikes in
> ParNew collection, but I'd suggest using the survivor
> spaces to age objects for at least a few collections
> before promoting to the old gen. This reduces pressure
> on the old gen allocation (during scavenge) improving
> scavenges and on CMS collection as well.
>
> I'd suggest trying -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=4
> to start off with and iterate based on data from 
> -XX:+PrintTenuringDistribution.
>
> It is possible that the promotion of very short-lived objects fragments
> the old gen and causes old gen allocation (for promotion) to slow down.
> It can also cause lots of mutation in the old gen and adversely affect
> card-scanning times further slowing down scavenges.
>
> -- ramki
>
> On 05/06/09 05:06, Kaj Nygren wrote:
>> Hi,
>>
>> We are having problems with occational long ParNew pauses in our 
>> application running on a Jboss server. The server runs well for most 
>> of the time but sometimes we get ParNew collections that can take up 
>> to 120 seconds. Most of the time we have several consecutive slow 
>> collections, which are more or less killing the server under high load.
>>
>> We are using JDK 1.5.0_16 on Windows Server 2003, and are using 
>> ParNew + Concurrent CMS.
>>
>> Our GC configuration is:
>>
>> wrapper.java.additional.1=-Dprogram.name=run.bat
>> wrapper.java.additional.2=-server
>> wrapper.java.additional.3=-Xms19000m
>> wrapper.java.additional.4=-Xmx19000m
>> wrapper.java.additional.5="-XX:NewSize=1000m"
>> wrapper.java.additional.6="-XX:MaxNewSize=1000m"
>> wrapper.java.additional.7="-XX:MaxPermSize=256m"
>> wrapper.java.additional.8="-XX:PermSize=256m"
>> wrapper.java.additional.11="-Xloggc:c:\mygclog.gc"
>> wrapper.java.additional.12="-XX:+UseConcMarkSweepGC"
>> wrapper.java.additional.13="-XX:+PrintGCDetails"
>> wrapper.java.additional.14="-XX:-PrintTenuringDistribution"
>> wrapper.java.additional.15="-XX:+PrintHeapAtGC"
>> wrapper.java.additional.16="-XX:+DisableExplicitGC"
>> wrapper.java.additional.17="-XX:+PrintGCTimeStamps"
>> wrapper.java.additional.18="-XX:CMSInitiatingOccupancyFraction=50"
>> wrapper.java.additional.19="-XX:+CMSIncrementalMode"
>> wrapper.java.additional.20="-XX:+CMSIncrementalPacing"
>> wrapper.java.additional.21="-verbose:gc"
>>
>> Usually ParNew is quick as can be seen here:
>>
>> 614276,653: [ParNew: 935058K->0K(1023040K), 0,0624699 secs] 
>> 12383872K->11459948K(19455040K) icms_dc=22 Heap after gc 
>> invocations=57822:
>> par new generation   total 1023040K, used 0K [0x000000007fff0000, 
>> 0x00000000be7f0000, 0x00000000be7f0000)
>>  eden space 1022080K,   0% used [0x000000007fff0000, 
>> 0x000000007fff0000, 0x00000000be610000)
>>  from space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>> 0x00000000be7f0000)
>>  to   space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>> 0x00000000be700000)
>> concurrent mark-sweep generation total 18432000K, used 11459948K 
>> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>> concurrent-mark-sweep perm gen total 262144K, used 112069K 
>> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>> }
>> , 0,0628860 secs]
>>
>> But suddenly it becomes really slow:
>>
>> 620659,717: [ParNew: 1022080K->0K(1023040K), 1,3313817 secs] 
>> 12688913K->11692925K(19455040K) icms_dc=10 Heap after gc 
>> invocations=58587:
>> par new generation   total 1023040K, used 0K [0x000000007fff0000, 
>> 0x00000000be7f0000, 0x00000000be7f0000)
>>  eden space 1022080K,   0% used [0x000000007fff0000, 
>> 0x000000007fff0000, 0x00000000be610000)
>>  from space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>> 0x00000000be700000)
>>  to   space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>> 0x00000000be7f0000)
>> concurrent mark-sweep generation total 18432000K, used 11692925K 
>> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>> concurrent-mark-sweep perm gen total 262144K, used 112098K 
>> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>> }
>> , 1,3319554 secs]
>> 620663,221: [GC {Heap before gc invocations=58587:
>> par new generation   total 1023040K, used 1022080K 
>> [0x000000007fff0000, 0x00000000be7f0000, 0x00000000be7f0000)
>>  eden space 1022080K, 100% used [0x000000007fff0000, 
>> 0x00000000be610000, 0x00000000be610000)
>>  from space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>> 0x00000000be700000)
>>  to   space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>> 0x00000000be7f0000)
>> concurrent mark-sweep generation total 18432000K, used 11692925K 
>> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>> concurrent-mark-sweep perm gen total 262144K, used 112106K 
>> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>> 620663,222: [ParNew: 1022080K->0K(1023040K), 10,2909893 secs] 
>> 12715005K->11936864K(19455040K) icms_dc=10 Heap after gc 
>> invocations=58588:
>> par new generation   total 1023040K, used 0K [0x000000007fff0000, 
>> 0x00000000be7f0000, 0x00000000be7f0000)
>>  eden space 1022080K,   0% used [0x000000007fff0000, 
>> 0x000000007fff0000, 0x00000000be610000)
>>  from space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>> 0x00000000be7f0000)
>>  to   space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>> 0x00000000be700000)
>> concurrent mark-sweep generation total 18432000K, used 11936864K 
>> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>> concurrent-mark-sweep perm gen total 262144K, used 112106K 
>> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>> }
>> , 10,2915548 secs]
>> 620676,392: [GC {Heap before gc invocations=58588:
>> par new generation   total 1023040K, used 1022080K 
>> [0x000000007fff0000, 0x00000000be7f0000, 0x00000000be7f0000)
>>  eden space 1022080K, 100% used [0x000000007fff0000, 
>> 0x00000000be610000, 0x00000000be610000)
>>  from space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>> 0x00000000be7f0000)
>>  to   space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>> 0x00000000be700000)
>> concurrent mark-sweep generation total 18432000K, used 11936864K 
>> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>> concurrent-mark-sweep perm gen total 262144K, used 112106K 
>> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>> 620676,392: [ParNew: 1022080K->0K(1023040K), 3,8905950 secs] 
>> 12958944K->12020225K(19455040K) icms_dc=10 Heap after gc 
>> invocations=58589:
>> par new generation   total 1023040K, used 0K [0x000000007fff0000, 
>> 0x00000000be7f0000, 0x00000000be7f0000)
>>  eden space 1022080K,   0% used [0x000000007fff0000, 
>> 0x000000007fff0000, 0x00000000be610000)
>>  from space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>> 0x00000000be700000)
>>  to   space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>> 0x00000000be7f0000)
>> concurrent mark-sweep generation total 18432000K, used 12020225K 
>> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>> concurrent-mark-sweep perm gen total 262144K, used 112106K 
>> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>> }
>> , 3,8911113 secs]
>> 620682,667: [GC {Heap before gc invocations=58589:
>> par new generation   total 1023040K, used 1022080K 
>> [0x000000007fff0000, 0x00000000be7f0000, 0x00000000be7f0000)
>>  eden space 1022080K, 100% used [0x000000007fff0000, 
>> 0x00000000be610000, 0x00000000be610000)
>>  from space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>> 0x00000000be700000)
>>  to   space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>> 0x00000000be7f0000)
>> concurrent mark-sweep generation total 18432000K, used 12020225K 
>> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>> concurrent-mark-sweep perm gen total 262144K, used 112112K 
>> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>> 620682,668: [ParNew: 1022080K->0K(1023040K), 9,4186088 secs] 
>> 13042305K->12214571K(19455040K) icms_dc=10 Heap after gc 
>> invocations=58590:
>> par new generation   total 1023040K, used 0K [0x000000007fff0000, 
>> 0x00000000be7f0000, 0x00000000be7f0000)
>>  eden space 1022080K,   0% used [0x000000007fff0000, 
>> 0x000000007fff0000, 0x00000000be610000)
>>  from space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>> 0x00000000be7f0000)
>>  to   space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>> 0x00000000be700000)
>> concurrent mark-sweep generation total 18432000K, used 12214571K 
>> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>> concurrent-mark-sweep perm gen total 262144K, used 112112K 
>> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>> }
>> , 9,4191551 secs]
>>
>> We also have problems with occational promotion failed, which I 
>> assume is due to fragmentation.
>> I have attatched a longer log, which shows this behaviour and ends 
>> with a promotion failed.
>>
>> Does anybody have a glue as to what is going wrong, and how we can 
>> change our gc configuration to reduce the problems?
>> Any help is greatly appreciated!
>>
>> Regards
>>
>> Kaj Nygren
>> kaj at mag.se
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> hotspot-gc-use mailing list
>> hotspot-gc-use at openjdk.java.net
>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use



From Y.S.Ramakrishna at Sun.COM  Tue May 19 11:15:10 2009
From: Y.S.Ramakrishna at Sun.COM (Y.S.Ramakrishna at Sun.COM)
Date: Tue, 19 May 2009 11:15:10 -0700
Subject: Strange long ParNew collections
In-Reply-To: <4A1297AC.5060205@mag.se>
References: <4A017D35.7000208@mag.se> <4A02078C.4080904@Sun.COM>
	<4A1297AC.5060205@mag.se>
Message-ID: <4A12F72E.2000906@Sun.COM>

Yes, from your description, it's very likely you are
running into the bug that was fixed in CR 6786503. The
fix is available in 1.5u19 (and 6u14 and open jdk 7).

Your Sun support can confirm that this is indeed the bug if you send
them a series of pstacks taken during the "hang" (i suppose you
would need to have your watchdog get the gcore/pstack before
restarting the prcoess since the onset of the bug is somewhat
unpredictable and you can't be waiting for the event before
your watchdog restarts the process; or you can turn off your
watchdog if in a test environment and collect the pstacks
manually, since the "hang" usually lasts for a while, even if
you have gone out for a cup of coffee :-).

-- ramki

On 05/19/09 04:27, Kaj Nygren wrote:
> Hi,
> 
> Thanks for your help. I wanted to give an update and have a couple of 
> more questions :)
> 
> We did change to -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=4  and now 
> the ParNew collections are working a lot better. We have also turned off 
> incremental mode as Jon suggested.
> 
> Now we do have one problem left though, it seems that the CMS-GC gets 
> stuck sometimes during parallell rescan. The output looks like this:
> 
> 24004,062: [CMS-concurrent-sweep: 1,755/1,932 secs]
> 24004,062: [CMS-concurrent-reset-start]
> 24004,163: [CMS-concurrent-reset: 0,101/0,101 secs]
> 24006,163: [GC [1 CMS-initial-mark: 10348978K(18432000K)] 
> 10521837K(19353600K), 0,1055176 secs]
> 24006,269: [CMS-concurrent-mark-start]
> 24010,642: [CMS-concurrent-mark: 4,373/4,373 secs]
> 24010,642: [CMS-concurrent-preclean-start]
> 24010,730: [CMS-concurrent-preclean: 0,085/0,088 secs]
> 24010,730: [CMS-concurrent-abortable-preclean-start]
>  CMS: abort preclean due to time 24011,824: 
> [CMS-concurrent-abortable-preclean: 0,792/1,094 secs]
> 24011,828: [GC[YG occupancy: 332774 K (921600 K)]
> 24011,829: [Rescan (parallel)
> 
> Then the JVM hangs for more than 5 minutes at which time our watchdog 
> determines that the server is hung and restarts it. It did happen with 
> iCMS and apparently it still happens using CMS.
> 
> I searched around and found one similar report:
> 
> http://mail.openjdk.java.net/pipermail/hotspot-gc-use/2009-January/000327.html
> 
> I know this is just a guess, but could we be experiencing the same bug? 
> If so, am I correct in interpreting the bug-reports that it should be 
> fixed in 1.5.0_18? (we are running 1.5.0_16 at the moment).
> 
> /Kaj
> 
> Y.S.Ramakrishna at Sun.COM skrev:
>> This may or may not be a cause of the sudden spikes in
>> ParNew collection, but I'd suggest using the survivor
>> spaces to age objects for at least a few collections
>> before promoting to the old gen. This reduces pressure
>> on the old gen allocation (during scavenge) improving
>> scavenges and on CMS collection as well.
>>
>> I'd suggest trying -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=4
>> to start off with and iterate based on data from 
>> -XX:+PrintTenuringDistribution.
>>
>> It is possible that the promotion of very short-lived objects fragments
>> the old gen and causes old gen allocation (for promotion) to slow down.
>> It can also cause lots of mutation in the old gen and adversely affect
>> card-scanning times further slowing down scavenges.
>>
>> -- ramki
>>
>> On 05/06/09 05:06, Kaj Nygren wrote:
>>> Hi,
>>>
>>> We are having problems with occational long ParNew pauses in our 
>>> application running on a Jboss server. The server runs well for most 
>>> of the time but sometimes we get ParNew collections that can take up 
>>> to 120 seconds. Most of the time we have several consecutive slow 
>>> collections, which are more or less killing the server under high load.
>>>
>>> We are using JDK 1.5.0_16 on Windows Server 2003, and are using 
>>> ParNew + Concurrent CMS.
>>>
>>> Our GC configuration is:
>>>
>>> wrapper.java.additional.1=-Dprogram.name=run.bat
>>> wrapper.java.additional.2=-server
>>> wrapper.java.additional.3=-Xms19000m
>>> wrapper.java.additional.4=-Xmx19000m
>>> wrapper.java.additional.5="-XX:NewSize=1000m"
>>> wrapper.java.additional.6="-XX:MaxNewSize=1000m"
>>> wrapper.java.additional.7="-XX:MaxPermSize=256m"
>>> wrapper.java.additional.8="-XX:PermSize=256m"
>>> wrapper.java.additional.11="-Xloggc:c:\mygclog.gc"
>>> wrapper.java.additional.12="-XX:+UseConcMarkSweepGC"
>>> wrapper.java.additional.13="-XX:+PrintGCDetails"
>>> wrapper.java.additional.14="-XX:-PrintTenuringDistribution"
>>> wrapper.java.additional.15="-XX:+PrintHeapAtGC"
>>> wrapper.java.additional.16="-XX:+DisableExplicitGC"
>>> wrapper.java.additional.17="-XX:+PrintGCTimeStamps"
>>> wrapper.java.additional.18="-XX:CMSInitiatingOccupancyFraction=50"
>>> wrapper.java.additional.19="-XX:+CMSIncrementalMode"
>>> wrapper.java.additional.20="-XX:+CMSIncrementalPacing"
>>> wrapper.java.additional.21="-verbose:gc"
>>>
>>> Usually ParNew is quick as can be seen here:
>>>
>>> 614276,653: [ParNew: 935058K->0K(1023040K), 0,0624699 secs] 
>>> 12383872K->11459948K(19455040K) icms_dc=22 Heap after gc 
>>> invocations=57822:
>>> par new generation   total 1023040K, used 0K [0x000000007fff0000, 
>>> 0x00000000be7f0000, 0x00000000be7f0000)
>>>  eden space 1022080K,   0% used [0x000000007fff0000, 
>>> 0x000000007fff0000, 0x00000000be610000)
>>>  from space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>>> 0x00000000be7f0000)
>>>  to   space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>>> 0x00000000be700000)
>>> concurrent mark-sweep generation total 18432000K, used 11459948K 
>>> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>>> concurrent-mark-sweep perm gen total 262144K, used 112069K 
>>> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>>> }
>>> , 0,0628860 secs]
>>>
>>> But suddenly it becomes really slow:
>>>
>>> 620659,717: [ParNew: 1022080K->0K(1023040K), 1,3313817 secs] 
>>> 12688913K->11692925K(19455040K) icms_dc=10 Heap after gc 
>>> invocations=58587:
>>> par new generation   total 1023040K, used 0K [0x000000007fff0000, 
>>> 0x00000000be7f0000, 0x00000000be7f0000)
>>>  eden space 1022080K,   0% used [0x000000007fff0000, 
>>> 0x000000007fff0000, 0x00000000be610000)
>>>  from space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>>> 0x00000000be700000)
>>>  to   space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>>> 0x00000000be7f0000)
>>> concurrent mark-sweep generation total 18432000K, used 11692925K 
>>> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>>> concurrent-mark-sweep perm gen total 262144K, used 112098K 
>>> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>>> }
>>> , 1,3319554 secs]
>>> 620663,221: [GC {Heap before gc invocations=58587:
>>> par new generation   total 1023040K, used 1022080K 
>>> [0x000000007fff0000, 0x00000000be7f0000, 0x00000000be7f0000)
>>>  eden space 1022080K, 100% used [0x000000007fff0000, 
>>> 0x00000000be610000, 0x00000000be610000)
>>>  from space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>>> 0x00000000be700000)
>>>  to   space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>>> 0x00000000be7f0000)
>>> concurrent mark-sweep generation total 18432000K, used 11692925K 
>>> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>>> concurrent-mark-sweep perm gen total 262144K, used 112106K 
>>> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>>> 620663,222: [ParNew: 1022080K->0K(1023040K), 10,2909893 secs] 
>>> 12715005K->11936864K(19455040K) icms_dc=10 Heap after gc 
>>> invocations=58588:
>>> par new generation   total 1023040K, used 0K [0x000000007fff0000, 
>>> 0x00000000be7f0000, 0x00000000be7f0000)
>>>  eden space 1022080K,   0% used [0x000000007fff0000, 
>>> 0x000000007fff0000, 0x00000000be610000)
>>>  from space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>>> 0x00000000be7f0000)
>>>  to   space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>>> 0x00000000be700000)
>>> concurrent mark-sweep generation total 18432000K, used 11936864K 
>>> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>>> concurrent-mark-sweep perm gen total 262144K, used 112106K 
>>> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>>> }
>>> , 10,2915548 secs]
>>> 620676,392: [GC {Heap before gc invocations=58588:
>>> par new generation   total 1023040K, used 1022080K 
>>> [0x000000007fff0000, 0x00000000be7f0000, 0x00000000be7f0000)
>>>  eden space 1022080K, 100% used [0x000000007fff0000, 
>>> 0x00000000be610000, 0x00000000be610000)
>>>  from space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>>> 0x00000000be7f0000)
>>>  to   space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>>> 0x00000000be700000)
>>> concurrent mark-sweep generation total 18432000K, used 11936864K 
>>> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>>> concurrent-mark-sweep perm gen total 262144K, used 112106K 
>>> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>>> 620676,392: [ParNew: 1022080K->0K(1023040K), 3,8905950 secs] 
>>> 12958944K->12020225K(19455040K) icms_dc=10 Heap after gc 
>>> invocations=58589:
>>> par new generation   total 1023040K, used 0K [0x000000007fff0000, 
>>> 0x00000000be7f0000, 0x00000000be7f0000)
>>>  eden space 1022080K,   0% used [0x000000007fff0000, 
>>> 0x000000007fff0000, 0x00000000be610000)
>>>  from space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>>> 0x00000000be700000)
>>>  to   space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>>> 0x00000000be7f0000)
>>> concurrent mark-sweep generation total 18432000K, used 12020225K 
>>> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>>> concurrent-mark-sweep perm gen total 262144K, used 112106K 
>>> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>>> }
>>> , 3,8911113 secs]
>>> 620682,667: [GC {Heap before gc invocations=58589:
>>> par new generation   total 1023040K, used 1022080K 
>>> [0x000000007fff0000, 0x00000000be7f0000, 0x00000000be7f0000)
>>>  eden space 1022080K, 100% used [0x000000007fff0000, 
>>> 0x00000000be610000, 0x00000000be610000)
>>>  from space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>>> 0x00000000be700000)
>>>  to   space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>>> 0x00000000be7f0000)
>>> concurrent mark-sweep generation total 18432000K, used 12020225K 
>>> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>>> concurrent-mark-sweep perm gen total 262144K, used 112112K 
>>> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>>> 620682,668: [ParNew: 1022080K->0K(1023040K), 9,4186088 secs] 
>>> 13042305K->12214571K(19455040K) icms_dc=10 Heap after gc 
>>> invocations=58590:
>>> par new generation   total 1023040K, used 0K [0x000000007fff0000, 
>>> 0x00000000be7f0000, 0x00000000be7f0000)
>>>  eden space 1022080K,   0% used [0x000000007fff0000, 
>>> 0x000000007fff0000, 0x00000000be610000)
>>>  from space 960K,   0% used [0x00000000be700000, 0x00000000be700000, 
>>> 0x00000000be7f0000)
>>>  to   space 960K,   0% used [0x00000000be610000, 0x00000000be610000, 
>>> 0x00000000be700000)
>>> concurrent mark-sweep generation total 18432000K, used 12214571K 
>>> [0x00000000be7f0000, 0x00000005237f0000, 0x00000005237f0000)
>>> concurrent-mark-sweep perm gen total 262144K, used 112112K 
>>> [0x00000005237f0000, 0x00000005337f0000, 0x00000005337f0000)
>>> }
>>> , 9,4191551 secs]
>>>
>>> We also have problems with occational promotion failed, which I 
>>> assume is due to fragmentation.
>>> I have attatched a longer log, which shows this behaviour and ends 
>>> with a promotion failed.
>>>
>>> Does anybody have a glue as to what is going wrong, and how we can 
>>> change our gc configuration to reduce the problems?
>>> Any help is greatly appreciated!
>>>
>>> Regards
>>>
>>> Kaj Nygren
>>> kaj at mag.se
>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>> _______________________________________________
>>> hotspot-gc-use mailing list
>>> hotspot-gc-use at openjdk.java.net
>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
> 
> _______________________________________________
> hotspot-gc-use mailing list
> hotspot-gc-use at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use



From George.Porter at Sun.COM  Wed May 20 16:33:21 2009
From: George.Porter at Sun.COM (George Porter)
Date: Wed, 20 May 2009 16:33:21 -0700
Subject: Question about GC and network throughput problems
Message-ID: <5AC02859-1C62-4FE0-8C50-B9678548EF94@Sun.COM>

Hello everyone,

I'm writing with a question on behalf of the Apache HBase project,  
which is an open-source implementation of Google's BigTable system,  
written in 100% Java.  One of the developers ran into the following  
problem, and I wasn't sure where to look, and so I was hoping that  
someone on this list might have some pointers or suggestions that I  
could try.

Thank you in advance for your time,
George

"I was wondering if you could help us out w/ some issues we're having  
performance-wise.

The areas we need to look at next are rpc and the garbage collector.   
The rpc we can handle ourselves.  I was wondering if you could point  
us at someone who could help with the latter.  Maybe you know some  
java GC expert who is up to the challenge?

GC pauses hamper upload rates -- halving or quartering over time --  
and pauses mid-fetch bring our overall fetching throughput down as  
well as latency.  Some time has been spent messing w/ GC options,  
using different engines and different tunings to some effect (so far,  
we're seeing one engine to upload but need another to serve), but  
we're not expert and we're supposed to be a 'general-use' application  
fielding uploads and serving concurrently."

They are using Java 1.6.0_14 running on Linux x86_64.