From tanman12345 at yahoo.com Fri Jun 3 15:17:14 2011 From: tanman12345 at yahoo.com (Erwin) Date: Fri, 3 Jun 2011 15:17:14 -0700 (PDT) Subject: help with long par new times Message-ID: <593141.16056.qm@web111104.mail.gq1.yahoo.com> > Hello, > > We performed load testing in our environment today and > there were 3 high par new times. There were no failure in > promotions and no full gc, the par new just took long on 7 > occasions. The long times were: 4.1402018, 19.4847804, > 18.2093573, 11.7687385, 3.8394909, 3.1041563 and 2.1444040. > The load test was ran for about 3 hours. > Any help would be appreciated. I'm attaching the > native_stdout.log. I'm also showing the jvm args below. > M5000 with 16GB physical memory > Min heap - 4096 > Mam heap - 6016 > -server > -Xmn1000m > -XX:PermSize=512m > -XX:+UseConcMarkSweepGC > -XX:+HeapDumpOnOutOfMemoryError > -DUseSunHttpHandler=true > -Djavax.xml.soap.MessageFactory=weblogic.xml.saaj.MessageFactoryImpl > -Doracle.jdbc.V8Compatible=true > -XX:CMSInitiatingOccupancyFraction=70 > -XX:+CMSClassUnloadingEnabled > -verbose:gc > -XX:+PrintGCTimeStamps > -XX:+PrintGCDetails > -XX:-TraceClassUnloading > -XX:+PrintHeapAtGC > -XX:MaxPermSize=694m > -XX:+DisableExplicitGC > -XX:+CMSParallelRemarkEnabled > > Any help would be appreciated. > Thanks, > Erwin -------------- next part -------------- A non-text attachment was scrubbed... Name: App1_700_psu_06032011_native_stdout.zip Type: application/x-zip-compressed Size: 20447 bytes Desc: not available Url : http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20110603/c3428317/attachment-0001.bin From jon.masamitsu at oracle.com Fri Jun 3 16:46:08 2011 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Fri, 03 Jun 2011 16:46:08 -0700 Subject: help with long par new times In-Reply-To: <593141.16056.qm@web111104.mail.gq1.yahoo.com> References: <593141.16056.qm@web111104.mail.gq1.yahoo.com> Message-ID: <4DE97240.1000701@oracle.com> Your system time is rather high on this GC. Check the other long GC's to see if there is similar high system times. 8706.691: [GC 8706.691: [ParNew: 921599K->95151K(921600K), 18.2093573 secs] 2598 628K->1818996K(4091904K), 18.2099612 secs] [Times: user=5.11 sys=33.09, real=18. 21 secs] Something other than GC is likely contributing to these long pauses. On 6/3/2011 3:17 PM, Erwin wrote: >> Hello, >> >> We performed load testing in our environment today and >> there were 3 high par new times. There were no failure in >> promotions and no full gc, the par new just took long on 7 >> occasions. The long times were: 4.1402018, 19.4847804, >> 18.2093573, 11.7687385, 3.8394909, 3.1041563 and 2.1444040. >> The load test was ran for about 3 hours. >> Any help would be appreciated. I'm attaching the >> native_stdout.log. I'm also showing the jvm args below. >> M5000 with 16GB physical memory >> Min heap - 4096 >> Mam heap - 6016 >> -server >> -Xmn1000m >> -XX:PermSize=512m >> -XX:+UseConcMarkSweepGC >> -XX:+HeapDumpOnOutOfMemoryError >> -DUseSunHttpHandler=true >> -Djavax.xml.soap.MessageFactory=weblogic.xml.saaj.MessageFactoryImpl >> -Doracle.jdbc.V8Compatible=true >> -XX:CMSInitiatingOccupancyFraction=70 >> -XX:+CMSClassUnloadingEnabled >> -verbose:gc >> -XX:+PrintGCTimeStamps >> -XX:+PrintGCDetails >> -XX:-TraceClassUnloading >> -XX:+PrintHeapAtGC >> -XX:MaxPermSize=694m >> -XX:+DisableExplicitGC >> -XX:+CMSParallelRemarkEnabled >> >> Any help would be appreciated. >> Thanks, >> Erwin >> >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20110603/47eb889c/attachment.html From Stefan.Wachter at gmx.de Tue Jun 7 10:16:20 2011 From: Stefan.Wachter at gmx.de (Stefan Wachter) Date: Tue, 07 Jun 2011 19:16:20 +0200 Subject: Various GC (G1GC, CMS): Meaning of CollectionTime attribute of MBeans Message-ID: <4DEE5CE4.4080308@gmx.de> Hi all, I have a cluster with Tomcat instances running on the nodes. (The cluster nodes are 64-Bit Linux machines with 37 GB memory and 8 processors. Two Tomcat instances run on each machine. Every Tomcat instance has 10 GB of heap available. The nodes have Java 1.6.0_24 installed.) I would like to compare differenct GC settings and reduce the stop-the-world times. Is this possible using the information available by JMX? At the moment I have three different settings with the following MBeans being registered: 1. No garbage collection configuration -> "PS MarkSweep" and "PS Scavenge". 2. Option "-XX:+UseG1GC" -> "G1 Old Generation" and "G1 Young Generation". 3. Option "-XX:+UseConcMarkSweepGC" -> "ConcurrentMarkSweep" and "ParNew". Each of the MBeans has a "CollectionTime" attribute. What is the meaning of the attribute for each of the MBeans? Does the attribute reflect "stop-the-world" times? Should the attribute values of the two MBeans used in each configuration be summed? If using the information available via JMX is not enough in order to determine stop-the-world times, how could this be done? Thanks you for your attention & best regards Stefan From Stefan.Wachter at gmx.de Thu Jun 16 00:23:05 2011 From: Stefan.Wachter at gmx.de (Stefan Wachter) Date: Thu, 16 Jun 2011 09:23:05 +0200 Subject: G1GC: many consecutive full gcs Message-ID: <4DF9AF59.90708@gmx.de> Hi all, the G1GC runs nicely for some time but then it starts to have bursts of consecutive full gcs (see log below). Can anyone explain that behaviour? The JVM is stared with these options: -Xmx10000M -Xms1024M -XX:MaxPermSize=1024m -XX:+UseG1GC -XX:MaxGCPauseMillis=2000 -XX:GCPauseIntervalMillis=10000 --Stefan 31314.569: [GC pause (young) 3286M->3234M(8937M), 0.0485530 secs] 31374.932: [GC pause (young) 3279M->3236M(8937M), 0.0486560 secs] 31837.688: [GC pause (young) 4614M->3252M(8937M), 0.0739370 secs] 33528.784: [GC pause (young) 5997M->3330M(8937M), 0.1543940 secs] 35372.029: [GC pause (young) 6038M->3405M(8937M), 0.2364930 secs] 36508.343: [Full GC 8525M->2903M(9678M), 8.9121160 secs] 38204.291: [Full GC 8647M->2295M(10000M), 7.4105940 secs] 39038.017: [Full GC 8790M->2092M(6975M), 6.7242110 secs] 39047.736: [Full GC 8996M->2089M(9998M), 6.0741630 secs] 39095.551: [Full GC 8936M->2090M(6969M), 5.5573390 secs] 39104.326: [Full GC 9175M->2092M(9997M), 5.3413860 secs] 39112.760: [Full GC 9392M->2094M(9978M), 5.4884150 secs] 41965.304: [Full GC 9549M->2112M(7043M), 6.4774550 secs] 43830.656: [GC pause (young) 4615M->2309M(7043M), 0.1727720 secs] 46024.398: [GC pause (young) 4638M->2497M(7043M), 0.3039900 secs] 46068.009: [Full GC 8832M->2237M(10000M), 6.6717340 secs] 46077.691: [Full GC 9322M->2187M(9987M), 5.8603570 secs] 46086.731: [Full GC 9487M->2189M(9996M), 5.8521360 secs] 46124.713: [Full GC 8792M->2185M(9895M), 5.7782260 secs] 46133.366: [Full GC 9297M->2188M(9999M), 5.7263360 secs] 46142.238: [Full GC 9509M->2189M(9985M), 5.5704850 secs] 47454.530: [Full GC 9676M->2295M(7653M), 7.3169840 secs] 48163.422: [GC pause (young) 4937M->2431M(7653M), 0.2379570 secs] 48887.388: [GC pause (young) 5007M->2497M(7653M), 0.1999790 secs] From david.ely at unboundid.com Thu Jun 16 13:09:38 2011 From: david.ely at unboundid.com (David Ely) Date: Thu, 16 Jun 2011 15:09:38 -0500 Subject: Feedback requested: HotSpot GC logging improvements In-Reply-To: <4BE96ACD.2070805@oracle.com> References: <4BE3194E.902@oracle.com> <4BE43C97.6000805@oracle.com> <9694A6C3D68A4249BD9E1A875B6BA81E105CCD27@bos0ex01.corp.attivio.com> <4BE96ACD.2070805@oracle.com> Message-ID: I'm curious if there was any update on this issue. Is there someplace I can track whether this is in JDK 7 or not? Thanks. David On Tue, May 11, 2010 at 9:33 AM, Tony Printezis wrote: > Martin, > > Hi, thanks for the feedback. > > Martin Serrano wrote: > > I would prefer to have a consistent suffix (like .log), in the filename. > Perhaps you could support just the %d format for the counter in the > generated log name. > > > Well, if you allow parameters in the log file name, like 'foo.%d.%n.log' > then folks can give their one suffix. I don't think we want to start > adding one... > > Tony > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20110616/952212a1/attachment.html From tony.printezis at oracle.com Thu Jun 16 13:10:52 2011 From: tony.printezis at oracle.com (Tony Printezis) Date: Thu, 16 Jun 2011 16:10:52 -0400 Subject: Feedback requested: HotSpot GC logging improvements In-Reply-To: References: <4BE3194E.902@oracle.com> <4BE43C97.6000805@oracle.com> <9694A6C3D68A4249BD9E1A875B6BA81E105CCD27@bos0ex01.corp.attivio.com> <4BE96ACD.2070805@oracle.com> Message-ID: <4DFA634C.7020805@oracle.com> David, We haven't had a chance to work on the %d format.... Tony On 06/16/2011 04:09 PM, David Ely wrote: > I'm curious if there was any update on this issue. Is there someplace > I can track whether this is in JDK 7 or not? Thanks. > > David > > On Tue, May 11, 2010 at 9:33 AM, Tony Printezis > > wrote: > > Martin, > > Hi, thanks for the feedback. > > Martin Serrano wrote: > > I would prefer to have a consistent suffix (like .log), in the > filename. Perhaps you could support just the %d format for the > counter in the generated log name. > > > Well, if you allow parameters in the log file name, like > 'foo.%d.%n.log' > then folks can give their one suffix. I don't think we want to start > adding one... > > Tony > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20110616/52b7cc0a/attachment.html From aaisinzon at guidewire.com Thu Jun 16 14:02:04 2011 From: aaisinzon at guidewire.com (Alex Aisinzon) Date: Thu, 16 Jun 2011 14:02:04 -0700 Subject: hotspot-gc-use Digest, Vol 40, Issue 4 In-Reply-To: References: Message-ID: Stefan I have had similar experience with G1 and reported it here. I tried the option to try to limit GC pauses (-XX:MaxGCPauseMillis with 1s and then 2s). It did not improve things. I am happy to try anything else to help improve G1. Alex A -----Original Message----- From: hotspot-gc-use-bounces at openjdk.java.net [mailto:hotspot-gc-use-bounces at openjdk.java.net] On Behalf Of hotspot-gc-use-request at openjdk.java.net Sent: Thursday, June 16, 2011 12:00 PM To: hotspot-gc-use at openjdk.java.net Subject: hotspot-gc-use Digest, Vol 40, Issue 4 Send hotspot-gc-use mailing list submissions to hotspot-gc-use at openjdk.java.net To subscribe or unsubscribe via the World Wide Web, visit http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use or, via email, send a message with subject or body 'help' to hotspot-gc-use-request at openjdk.java.net You can reach the person managing the list at hotspot-gc-use-owner at openjdk.java.net When replying, please edit your Subject line so it is more specific than "Re: Contents of hotspot-gc-use digest..." Today's Topics: 1. G1GC: many consecutive full gcs (Stefan Wachter) ---------------------------------------------------------------------- Message: 1 Date: Thu, 16 Jun 2011 09:23:05 +0200 From: Stefan Wachter Subject: G1GC: many consecutive full gcs To: hotspot-gc-use at openjdk.java.net Message-ID: <4DF9AF59.90708 at gmx.de> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Hi all, the G1GC runs nicely for some time but then it starts to have bursts of consecutive full gcs (see log below). Can anyone explain that behaviour? The JVM is stared with these options: -Xmx10000M -Xms1024M -XX:MaxPermSize=1024m -XX:+UseG1GC -XX:MaxGCPauseMillis=2000 -XX:GCPauseIntervalMillis=10000 --Stefan 31314.569: [GC pause (young) 3286M->3234M(8937M), 0.0485530 secs] 31374.932: [GC pause (young) 3279M->3236M(8937M), 0.0486560 secs] 31837.688: [GC pause (young) 4614M->3252M(8937M), 0.0739370 secs] 33528.784: [GC pause (young) 5997M->3330M(8937M), 0.1543940 secs] 35372.029: [GC pause (young) 6038M->3405M(8937M), 0.2364930 secs] 36508.343: [Full GC 8525M->2903M(9678M), 8.9121160 secs] 38204.291: [Full GC 8647M->2295M(10000M), 7.4105940 secs] 39038.017: [Full GC 8790M->2092M(6975M), 6.7242110 secs] 39047.736: [Full GC 8996M->2089M(9998M), 6.0741630 secs] 39095.551: [Full GC 8936M->2090M(6969M), 5.5573390 secs] 39104.326: [Full GC 9175M->2092M(9997M), 5.3413860 secs] 39112.760: [Full GC 9392M->2094M(9978M), 5.4884150 secs] 41965.304: [Full GC 9549M->2112M(7043M), 6.4774550 secs] 43830.656: [GC pause (young) 4615M->2309M(7043M), 0.1727720 secs] 46024.398: [GC pause (young) 4638M->2497M(7043M), 0.3039900 secs] 46068.009: [Full GC 8832M->2237M(10000M), 6.6717340 secs] 46077.691: [Full GC 9322M->2187M(9987M), 5.8603570 secs] 46086.731: [Full GC 9487M->2189M(9996M), 5.8521360 secs] 46124.713: [Full GC 8792M->2185M(9895M), 5.7782260 secs] 46133.366: [Full GC 9297M->2188M(9999M), 5.7263360 secs] 46142.238: [Full GC 9509M->2189M(9985M), 5.5704850 secs] 47454.530: [Full GC 9676M->2295M(7653M), 7.3169840 secs] 48163.422: [GC pause (young) 4937M->2431M(7653M), 0.2379570 secs] 48887.388: [GC pause (young) 5007M->2497M(7653M), 0.1999790 secs] ------------------------------ _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use End of hotspot-gc-use Digest, Vol 40, Issue 4 ********************************************* From y.s.ramakrishna at oracle.com Thu Jun 16 15:20:19 2011 From: y.s.ramakrishna at oracle.com (Y. S. Ramakrishna) Date: Thu, 16 Jun 2011 15:20:19 -0700 Subject: G1GC: many consecutive full gcs (was Re: hotspot-gc-use Digest, Vol 40, Issue 4) In-Reply-To: References: Message-ID: <4DFA81A3.1050802@oracle.com> Stefan and Alex -- (1) what's the version of the JVM used? (2) could you try PermSize == MaxPermSize and see if it made any difference? (3) do you have lots of WeakReference or Finalizer objects in your system? (4) is what you have below your complete GC log (i.e. no gaps between first and last line?) (5) what do you see when using CMS (in case you tried it)? (6) do you have a simple test case that you can share with us that exhibits this behaviour? thanks! -- ramki On 06/16/11 14:02, Alex Aisinzon wrote: > Stefan > > I have had similar experience with G1 and reported it here. > I tried the option to try to limit GC pauses (-XX:MaxGCPauseMillis with > 1s and then 2s). It did not improve things. > I am happy to try anything else to help improve G1. > > Alex A > > -----Original Message----- > From: hotspot-gc-use-bounces at openjdk.java.net > [mailto:hotspot-gc-use-bounces at openjdk.java.net] On Behalf Of > hotspot-gc-use-request at openjdk.java.net > Sent: Thursday, June 16, 2011 12:00 PM > To: hotspot-gc-use at openjdk.java.net > Subject: hotspot-gc-use Digest, Vol 40, Issue 4 > > Send hotspot-gc-use mailing list submissions to > hotspot-gc-use at openjdk.java.net > > To subscribe or unsubscribe via the World Wide Web, visit > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > or, via email, send a message with subject or body 'help' to > hotspot-gc-use-request at openjdk.java.net > > You can reach the person managing the list at > hotspot-gc-use-owner at openjdk.java.net > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of hotspot-gc-use digest..." > > > Today's Topics: > > 1. G1GC: many consecutive full gcs (Stefan Wachter) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 16 Jun 2011 09:23:05 +0200 > From: Stefan Wachter > Subject: G1GC: many consecutive full gcs > To: hotspot-gc-use at openjdk.java.net > Message-ID: <4DF9AF59.90708 at gmx.de> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > Hi all, > > the G1GC runs nicely for some time but then it starts to have bursts of > consecutive full gcs (see log below). Can anyone explain that behaviour? > > The JVM is stared with these options: > > -Xmx10000M > -Xms1024M > -XX:MaxPermSize=1024m > -XX:+UseG1GC > -XX:MaxGCPauseMillis=2000 > -XX:GCPauseIntervalMillis=10000 > > --Stefan > > > 31314.569: [GC pause (young) 3286M->3234M(8937M), 0.0485530 secs] > 31374.932: [GC pause (young) 3279M->3236M(8937M), 0.0486560 secs] > 31837.688: [GC pause (young) 4614M->3252M(8937M), 0.0739370 secs] > 33528.784: [GC pause (young) 5997M->3330M(8937M), 0.1543940 secs] > 35372.029: [GC pause (young) 6038M->3405M(8937M), 0.2364930 secs] > 36508.343: [Full GC 8525M->2903M(9678M), 8.9121160 secs] > 38204.291: [Full GC 8647M->2295M(10000M), 7.4105940 secs] > 39038.017: [Full GC 8790M->2092M(6975M), 6.7242110 secs] > 39047.736: [Full GC 8996M->2089M(9998M), 6.0741630 secs] > 39095.551: [Full GC 8936M->2090M(6969M), 5.5573390 secs] > 39104.326: [Full GC 9175M->2092M(9997M), 5.3413860 secs] > 39112.760: [Full GC 9392M->2094M(9978M), 5.4884150 secs] > 41965.304: [Full GC 9549M->2112M(7043M), 6.4774550 secs] > 43830.656: [GC pause (young) 4615M->2309M(7043M), 0.1727720 secs] > 46024.398: [GC pause (young) 4638M->2497M(7043M), 0.3039900 secs] > 46068.009: [Full GC 8832M->2237M(10000M), 6.6717340 secs] > 46077.691: [Full GC 9322M->2187M(9987M), 5.8603570 secs] > 46086.731: [Full GC 9487M->2189M(9996M), 5.8521360 secs] > 46124.713: [Full GC 8792M->2185M(9895M), 5.7782260 secs] > 46133.366: [Full GC 9297M->2188M(9999M), 5.7263360 secs] > 46142.238: [Full GC 9509M->2189M(9985M), 5.5704850 secs] > 47454.530: [Full GC 9676M->2295M(7653M), 7.3169840 secs] > 48163.422: [GC pause (young) 4937M->2431M(7653M), 0.2379570 secs] > 48887.388: [GC pause (young) 5007M->2497M(7653M), 0.1999790 secs] > > > > ------------------------------ > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > > End of hotspot-gc-use Digest, Vol 40, Issue 4 > ********************************************* > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From aaisinzon at guidewire.com Thu Jun 16 16:27:47 2011 From: aaisinzon at guidewire.com (Alex Aisinzon) Date: Thu, 16 Jun 2011 16:27:47 -0700 Subject: G1GC: many consecutive full gcs (was Re: hotspot-gc-use Digest, Vol 40, Issue 4) In-Reply-To: <4DFA81A3.1050802@oracle.com> References: <4DFA81A3.1050802@oracle.com> Message-ID: Ramki Thanks for the attention. Comments are inlined. Regards Alex A -----Original Message----- From: Y. S. Ramakrishna [mailto:y.s.ramakrishna at oracle.com] Sent: Thursday, June 16, 2011 3:20 PM To: Alex Aisinzon Cc: hotspot-gc-use at openjdk.java.net; stefan.wachter at gmx.de Subject: Re: G1GC: many consecutive full gcs (was Re: hotspot-gc-use Digest, Vol 40, Issue 4) Stefan and Alex -- (1) what's the version of the JVM used? Sun JVM 1.6 64 bit update 25. (2) could you try PermSize == MaxPermSize and see if it made any difference? I always used PermSize == MaxPermSize. (3) do you have lots of WeakReference or Finalizer objects in your system? I am not sure about Weak references. I know that some finalizers exist and come from open source package we leverage. Do you want me to ask further about weak references? (4) is what you have below your complete GC log (i.e. no gaps between first and last line?) I am happy to share my own GC logs. They are monstrous. (5) what do you see when using CMS (in case you tried it)? CMS worked surprisingly well: it scales to a higher load and experienced some very rare longer pauses. At this stage, CMS seemed a better bet than G1. (6) do you have a simple test case that you can share with us that exhibits this behaviour? Our performance test involves running our full app and operating many workflows. This is done only in in house performance tests and cannot be easily replicated outside. thanks! -- ramki On 06/16/11 14:02, Alex Aisinzon wrote: > Stefan > > I have had similar experience with G1 and reported it here. > I tried the option to try to limit GC pauses (-XX:MaxGCPauseMillis with > 1s and then 2s). It did not improve things. > I am happy to try anything else to help improve G1. > > Alex A > > -----Original Message----- > From: hotspot-gc-use-bounces at openjdk.java.net > [mailto:hotspot-gc-use-bounces at openjdk.java.net] On Behalf Of > hotspot-gc-use-request at openjdk.java.net > Sent: Thursday, June 16, 2011 12:00 PM > To: hotspot-gc-use at openjdk.java.net > Subject: hotspot-gc-use Digest, Vol 40, Issue 4 > > Send hotspot-gc-use mailing list submissions to > hotspot-gc-use at openjdk.java.net > > To subscribe or unsubscribe via the World Wide Web, visit > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > or, via email, send a message with subject or body 'help' to > hotspot-gc-use-request at openjdk.java.net > > You can reach the person managing the list at > hotspot-gc-use-owner at openjdk.java.net > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of hotspot-gc-use digest..." > > > Today's Topics: > > 1. G1GC: many consecutive full gcs (Stefan Wachter) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 16 Jun 2011 09:23:05 +0200 > From: Stefan Wachter > Subject: G1GC: many consecutive full gcs > To: hotspot-gc-use at openjdk.java.net > Message-ID: <4DF9AF59.90708 at gmx.de> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > Hi all, > > the G1GC runs nicely for some time but then it starts to have bursts of > consecutive full gcs (see log below). Can anyone explain that behaviour? > > The JVM is stared with these options: > > -Xmx10000M > -Xms1024M > -XX:MaxPermSize=1024m > -XX:+UseG1GC > -XX:MaxGCPauseMillis=2000 > -XX:GCPauseIntervalMillis=10000 > > --Stefan > > > 31314.569: [GC pause (young) 3286M->3234M(8937M), 0.0485530 secs] > 31374.932: [GC pause (young) 3279M->3236M(8937M), 0.0486560 secs] > 31837.688: [GC pause (young) 4614M->3252M(8937M), 0.0739370 secs] > 33528.784: [GC pause (young) 5997M->3330M(8937M), 0.1543940 secs] > 35372.029: [GC pause (young) 6038M->3405M(8937M), 0.2364930 secs] > 36508.343: [Full GC 8525M->2903M(9678M), 8.9121160 secs] > 38204.291: [Full GC 8647M->2295M(10000M), 7.4105940 secs] > 39038.017: [Full GC 8790M->2092M(6975M), 6.7242110 secs] > 39047.736: [Full GC 8996M->2089M(9998M), 6.0741630 secs] > 39095.551: [Full GC 8936M->2090M(6969M), 5.5573390 secs] > 39104.326: [Full GC 9175M->2092M(9997M), 5.3413860 secs] > 39112.760: [Full GC 9392M->2094M(9978M), 5.4884150 secs] > 41965.304: [Full GC 9549M->2112M(7043M), 6.4774550 secs] > 43830.656: [GC pause (young) 4615M->2309M(7043M), 0.1727720 secs] > 46024.398: [GC pause (young) 4638M->2497M(7043M), 0.3039900 secs] > 46068.009: [Full GC 8832M->2237M(10000M), 6.6717340 secs] > 46077.691: [Full GC 9322M->2187M(9987M), 5.8603570 secs] > 46086.731: [Full GC 9487M->2189M(9996M), 5.8521360 secs] > 46124.713: [Full GC 8792M->2185M(9895M), 5.7782260 secs] > 46133.366: [Full GC 9297M->2188M(9999M), 5.7263360 secs] > 46142.238: [Full GC 9509M->2189M(9985M), 5.5704850 secs] > 47454.530: [Full GC 9676M->2295M(7653M), 7.3169840 secs] > 48163.422: [GC pause (young) 4937M->2431M(7653M), 0.2379570 secs] > 48887.388: [GC pause (young) 5007M->2497M(7653M), 0.1999790 secs] > > > > ------------------------------ > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > > End of hotspot-gc-use Digest, Vol 40, Issue 4 > ********************************************* > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From y.s.ramakrishna at oracle.com Fri Jun 17 08:10:51 2011 From: y.s.ramakrishna at oracle.com (Y. Srinivas Ramakrishna) Date: Fri, 17 Jun 2011 08:10:51 -0700 Subject: G1GC: many consecutive full gcs (was Re: hotspot-gc-use Digest, Vol 40, Issue 4) In-Reply-To: <4DFAFB95.7070405@gmx.de> References: <4DFA81A3.1050802@oracle.com> <4DFAFB95.7070405@gmx.de> Message-ID: <4DFB6E7B.8000200@oracle.com> Thanks for the responses and the extra information, Stefan and Alex. Stefan, please try PermSize == MaxPermSize at the next opportunity, although it's likely not to have any effect based on a preliminary reading of the info you provided below. As regards class histogram, please note that this would induce a pause as long as the full gc's you see (or may be a bit shorter), so use it with care when you obtain that information. Meanwhile, we'll look over the information you have provided and get back to you once we have made some headway. Thanks for bringing this to our attention and for the data so far. -- ramki On 6/17/2011 12:00 AM, Stefan Wachter wrote: > Hi Ramki, > > thanks for your interest. See my answers below. > > Cheers, > > Stefan > > > On 06/17/2011 12:20 AM, Y. S. Ramakrishna wrote: >> Stefan and Alex -- >> >> (1) what's the version of the JVM used? > > Java(TM) SE Runtime Environment (build 1.6.0_24-b07) > Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode) > >> >> (2) could you try PermSize == MaxPermSize and see if it made any >> difference? > > PermSize was not set until now. I try this setting and can report results the next time our cluster > nodes are switched. > >> >> (3) do you have lots of WeakReference or Finalizer objects in your system? > > I have no information about this. I guess a heap dump would provide that information. I will ask our > admins if it is possible to take a heap dump. > >> >> (4) is what you have below your complete GC log (i.e. no gaps between first and last line?) > > Yes, that is the original output of the GC. No lines were removed. >> >> (5) what do you see when using CMS (in case you tried it)? > > CMS seems to behave better (see attached gc.log). The settings used with CMS are: > > -Xmx10000M > -Xms1024M > -XX:MaxPermSize=1024m > -XX:+PrintGCDateStamps > -XX:+UseConcMarkSweepGC > >> >> (6) do you have a simple test case that you can share with >> us that exhibits this behaviour? > > Unfortunately I have no simple test case. The output is from our life system running a tomcat with > various wars deployed. > >> >> thanks! >> -- ramki >> >> On 06/16/11 14:02, Alex Aisinzon wrote: >>> Stefan >>> >>> I have had similar experience with G1 and reported it here. >>> I tried the option to try to limit GC pauses (-XX:MaxGCPauseMillis with >>> 1s and then 2s). It did not improve things. >>> I am happy to try anything else to help improve G1. >>> >>> Alex A >>> >>> -----Original Message----- >>> From: hotspot-gc-use-bounces at openjdk.java.net >>> [mailto:hotspot-gc-use-bounces at openjdk.java.net] On Behalf Of >>> hotspot-gc-use-request at openjdk.java.net >>> Sent: Thursday, June 16, 2011 12:00 PM >>> To: hotspot-gc-use at openjdk.java.net >>> Subject: hotspot-gc-use Digest, Vol 40, Issue 4 >>> >>> Send hotspot-gc-use mailing list submissions to >>> hotspot-gc-use at openjdk.java.net >>> >>> To subscribe or unsubscribe via the World Wide Web, visit >>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>> or, via email, send a message with subject or body 'help' to >>> hotspot-gc-use-request at openjdk.java.net >>> >>> You can reach the person managing the list at >>> hotspot-gc-use-owner at openjdk.java.net >>> >>> When replying, please edit your Subject line so it is more specific >>> than "Re: Contents of hotspot-gc-use digest..." >>> >>> >>> Today's Topics: >>> >>> 1. G1GC: many consecutive full gcs (Stefan Wachter) >>> >>> >>> ---------------------------------------------------------------------- >>> >>> Message: 1 >>> Date: Thu, 16 Jun 2011 09:23:05 +0200 >>> From: Stefan Wachter >>> Subject: G1GC: many consecutive full gcs >>> To: hotspot-gc-use at openjdk.java.net >>> Message-ID: <4DF9AF59.90708 at gmx.de> >>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed >>> >>> Hi all, >>> >>> the G1GC runs nicely for some time but then it starts to have bursts of consecutive full gcs (see >>> log below). Can anyone explain that behaviour? >>> >>> The JVM is stared with these options: >>> >>> -Xmx10000M >>> -Xms1024M >>> -XX:MaxPermSize=1024m >>> -XX:+UseG1GC >>> -XX:MaxGCPauseMillis=2000 >>> -XX:GCPauseIntervalMillis=10000 >>> >>> --Stefan >>> >>> >>> 31314.569: [GC pause (young) 3286M->3234M(8937M), 0.0485530 secs] >>> 31374.932: [GC pause (young) 3279M->3236M(8937M), 0.0486560 secs] >>> 31837.688: [GC pause (young) 4614M->3252M(8937M), 0.0739370 secs] >>> 33528.784: [GC pause (young) 5997M->3330M(8937M), 0.1543940 secs] >>> 35372.029: [GC pause (young) 6038M->3405M(8937M), 0.2364930 secs] >>> 36508.343: [Full GC 8525M->2903M(9678M), 8.9121160 secs] >>> 38204.291: [Full GC 8647M->2295M(10000M), 7.4105940 secs] >>> 39038.017: [Full GC 8790M->2092M(6975M), 6.7242110 secs] >>> 39047.736: [Full GC 8996M->2089M(9998M), 6.0741630 secs] >>> 39095.551: [Full GC 8936M->2090M(6969M), 5.5573390 secs] >>> 39104.326: [Full GC 9175M->2092M(9997M), 5.3413860 secs] >>> 39112.760: [Full GC 9392M->2094M(9978M), 5.4884150 secs] >>> 41965.304: [Full GC 9549M->2112M(7043M), 6.4774550 secs] >>> 43830.656: [GC pause (young) 4615M->2309M(7043M), 0.1727720 secs] >>> 46024.398: [GC pause (young) 4638M->2497M(7043M), 0.3039900 secs] >>> 46068.009: [Full GC 8832M->2237M(10000M), 6.6717340 secs] >>> 46077.691: [Full GC 9322M->2187M(9987M), 5.8603570 secs] >>> 46086.731: [Full GC 9487M->2189M(9996M), 5.8521360 secs] >>> 46124.713: [Full GC 8792M->2185M(9895M), 5.7782260 secs] >>> 46133.366: [Full GC 9297M->2188M(9999M), 5.7263360 secs] >>> 46142.238: [Full GC 9509M->2189M(9985M), 5.5704850 secs] >>> 47454.530: [Full GC 9676M->2295M(7653M), 7.3169840 secs] >>> 48163.422: [GC pause (young) 4937M->2431M(7653M), 0.2379570 secs] >>> 48887.388: [GC pause (young) 5007M->2497M(7653M), 0.1999790 secs] >>> >>> >>> >>> ------------------------------ >>> >>> _______________________________________________ >>> hotspot-gc-use mailing list >>> hotspot-gc-use at openjdk.java.net >>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>> >>> >>> End of hotspot-gc-use Digest, Vol 40, Issue 4 >>> ********************************************* >>> _______________________________________________ >>> hotspot-gc-use mailing list >>> hotspot-gc-use at openjdk.java.net >>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> > From Stefan.Wachter at gmx.de Fri Jun 17 00:00:37 2011 From: Stefan.Wachter at gmx.de (Stefan Wachter) Date: Fri, 17 Jun 2011 09:00:37 +0200 Subject: G1GC: many consecutive full gcs (was Re: hotspot-gc-use Digest, Vol 40, Issue 4) In-Reply-To: <4DFA81A3.1050802@oracle.com> References: <4DFA81A3.1050802@oracle.com> Message-ID: <4DFAFB95.7070405@gmx.de> Hi Ramki, thanks for your interest. See my answers below. Cheers, Stefan On 06/17/2011 12:20 AM, Y. S. Ramakrishna wrote: > Stefan and Alex -- > > (1) what's the version of the JVM used? Java(TM) SE Runtime Environment (build 1.6.0_24-b07) Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode) > > (2) could you try PermSize == MaxPermSize and see if it made any > difference? PermSize was not set until now. I try this setting and can report results the next time our cluster nodes are switched. > > (3) do you have lots of WeakReference or Finalizer objects in your > system? I have no information about this. I guess a heap dump would provide that information. I will ask our admins if it is possible to take a heap dump. > > (4) is what you have below your complete GC log (i.e. no gaps between > first and last line?) Yes, that is the original output of the GC. No lines were removed. > > (5) what do you see when using CMS (in case you tried it)? CMS seems to behave better (see attached gc.log). The settings used with CMS are: -Xmx10000M -Xms1024M -XX:MaxPermSize=1024m -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC > > (6) do you have a simple test case that you can share with > us that exhibits this behaviour? Unfortunately I have no simple test case. The output is from our life system running a tomcat with various wars deployed. > > thanks! > -- ramki > > On 06/16/11 14:02, Alex Aisinzon wrote: >> Stefan >> >> I have had similar experience with G1 and reported it here. >> I tried the option to try to limit GC pauses (-XX:MaxGCPauseMillis with >> 1s and then 2s). It did not improve things. >> I am happy to try anything else to help improve G1. >> >> Alex A >> >> -----Original Message----- >> From: hotspot-gc-use-bounces at openjdk.java.net >> [mailto:hotspot-gc-use-bounces at openjdk.java.net] On Behalf Of >> hotspot-gc-use-request at openjdk.java.net >> Sent: Thursday, June 16, 2011 12:00 PM >> To: hotspot-gc-use at openjdk.java.net >> Subject: hotspot-gc-use Digest, Vol 40, Issue 4 >> >> Send hotspot-gc-use mailing list submissions to >> hotspot-gc-use at openjdk.java.net >> >> To subscribe or unsubscribe via the World Wide Web, visit >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> or, via email, send a message with subject or body 'help' to >> hotspot-gc-use-request at openjdk.java.net >> >> You can reach the person managing the list at >> hotspot-gc-use-owner at openjdk.java.net >> >> When replying, please edit your Subject line so it is more specific >> than "Re: Contents of hotspot-gc-use digest..." >> >> >> Today's Topics: >> >> 1. G1GC: many consecutive full gcs (Stefan Wachter) >> >> >> ---------------------------------------------------------------------- >> >> Message: 1 >> Date: Thu, 16 Jun 2011 09:23:05 +0200 >> From: Stefan Wachter >> Subject: G1GC: many consecutive full gcs >> To: hotspot-gc-use at openjdk.java.net >> Message-ID: <4DF9AF59.90708 at gmx.de> >> Content-Type: text/plain; charset=ISO-8859-1; format=flowed >> >> Hi all, >> >> the G1GC runs nicely for some time but then it starts to have bursts >> of consecutive full gcs (see log below). Can anyone explain that >> behaviour? >> >> The JVM is stared with these options: >> >> -Xmx10000M >> -Xms1024M >> -XX:MaxPermSize=1024m >> -XX:+UseG1GC >> -XX:MaxGCPauseMillis=2000 >> -XX:GCPauseIntervalMillis=10000 >> >> --Stefan >> >> >> 31314.569: [GC pause (young) 3286M->3234M(8937M), 0.0485530 secs] >> 31374.932: [GC pause (young) 3279M->3236M(8937M), 0.0486560 secs] >> 31837.688: [GC pause (young) 4614M->3252M(8937M), 0.0739370 secs] >> 33528.784: [GC pause (young) 5997M->3330M(8937M), 0.1543940 secs] >> 35372.029: [GC pause (young) 6038M->3405M(8937M), 0.2364930 secs] >> 36508.343: [Full GC 8525M->2903M(9678M), 8.9121160 secs] >> 38204.291: [Full GC 8647M->2295M(10000M), 7.4105940 secs] >> 39038.017: [Full GC 8790M->2092M(6975M), 6.7242110 secs] >> 39047.736: [Full GC 8996M->2089M(9998M), 6.0741630 secs] >> 39095.551: [Full GC 8936M->2090M(6969M), 5.5573390 secs] >> 39104.326: [Full GC 9175M->2092M(9997M), 5.3413860 secs] >> 39112.760: [Full GC 9392M->2094M(9978M), 5.4884150 secs] >> 41965.304: [Full GC 9549M->2112M(7043M), 6.4774550 secs] >> 43830.656: [GC pause (young) 4615M->2309M(7043M), 0.1727720 secs] >> 46024.398: [GC pause (young) 4638M->2497M(7043M), 0.3039900 secs] >> 46068.009: [Full GC 8832M->2237M(10000M), 6.6717340 secs] >> 46077.691: [Full GC 9322M->2187M(9987M), 5.8603570 secs] >> 46086.731: [Full GC 9487M->2189M(9996M), 5.8521360 secs] >> 46124.713: [Full GC 8792M->2185M(9895M), 5.7782260 secs] >> 46133.366: [Full GC 9297M->2188M(9999M), 5.7263360 secs] >> 46142.238: [Full GC 9509M->2189M(9985M), 5.5704850 secs] >> 47454.530: [Full GC 9676M->2295M(7653M), 7.3169840 secs] >> 48163.422: [GC pause (young) 4937M->2431M(7653M), 0.2379570 secs] >> 48887.388: [GC pause (young) 5007M->2497M(7653M), 0.1999790 secs] >> >> >> >> ------------------------------ >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> >> End of hotspot-gc-use Digest, Vol 40, Issue 4 >> ********************************************* >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > -------------- next part -------------- A non-text attachment was scrubbed... Name: gc.log Type: text/x-log Size: 691269 bytes Desc: not available Url : http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20110617/0a01cec9/attachment-0001.bin From Stefan.Wachter at gmx.de Mon Jun 27 06:43:38 2011 From: Stefan.Wachter at gmx.de (Stefan Wachter) Date: Mon, 27 Jun 2011 15:43:38 +0200 Subject: G1GC: many consecutive full gcs (was Re: hotspot-gc-use Digest, Vol 40, Issue 4) In-Reply-To: <4DFB6E7B.8000200@oracle.com> References: <4DFA81A3.1050802@oracle.com> <4DFAFB95.7070405@gmx.de> <4DFB6E7B.8000200@oracle.com> Message-ID: <4E08890A.9030601@gmx.de> Hi Ramki, I took a heap dump and checked for WeakReferences and Finalizer objects. The heap dump contained 36,071,040 instances with 12,405 instances being WeakReference instances. I am not sure if this is a lot or not. The "Basic info" section that is displayed in NetBeans reports "Number of objects pending for finalization: 0". Corresponds this to the number of Finalizer objects you asked for? The complete heap summary is: Total bytes: 2,472,537,236 Total classes: 15,953 Total instances: 36,071,040 Classloaders: 648 GC roots: 3,138 Number of objects pending for finalization: 0 Cheers, Stefan On 06/17/2011 05:10 PM, Y. Srinivas Ramakrishna wrote: > Thanks for the responses and the extra information, Stefan and Alex. > > Stefan, please try PermSize == MaxPermSize at the next opportunity, > although it's likely not to have any effect based on a preliminary > reading of the info you provided below. > > As regards class histogram, please note that this would induce a > pause as long as the full gc's you see (or may be a bit shorter), > so use it with care when you obtain that information. > > Meanwhile, we'll look over the information you have provided and > get back to you once we have made some headway. Thanks for bringing > this to our attention and for the data so far. > > -- ramki > > On 6/17/2011 12:00 AM, Stefan Wachter wrote: >> Hi Ramki, >> >> thanks for your interest. See my answers below. >> >> Cheers, >> >> Stefan >> >> >> On 06/17/2011 12:20 AM, Y. S. Ramakrishna wrote: >>> Stefan and Alex -- >>> >>> (1) what's the version of the JVM used? >> >> Java(TM) SE Runtime Environment (build 1.6.0_24-b07) >> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode) >> >>> >>> (2) could you try PermSize == MaxPermSize and see if it made any >>> difference? >> >> PermSize was not set until now. I try this setting and can report >> results the next time our cluster >> nodes are switched. >> >>> >>> (3) do you have lots of WeakReference or Finalizer objects in your >>> system? >> >> I have no information about this. I guess a heap dump would provide >> that information. I will ask our >> admins if it is possible to take a heap dump. >> >>> >>> (4) is what you have below your complete GC log (i.e. no gaps >>> between first and last line?) >> >> Yes, that is the original output of the GC. No lines were removed. >>> >>> (5) what do you see when using CMS (in case you tried it)? >> >> CMS seems to behave better (see attached gc.log). The settings used >> with CMS are: >> >> -Xmx10000M >> -Xms1024M >> -XX:MaxPermSize=1024m >> -XX:+PrintGCDateStamps >> -XX:+UseConcMarkSweepGC >> >>> >>> (6) do you have a simple test case that you can share with >>> us that exhibits this behaviour? >> >> Unfortunately I have no simple test case. The output is from our life >> system running a tomcat with >> various wars deployed. >> >>> >>> thanks! >>> -- ramki >>> >>> On 06/16/11 14:02, Alex Aisinzon wrote: >>>> Stefan >>>> >>>> I have had similar experience with G1 and reported it here. >>>> I tried the option to try to limit GC pauses (-XX:MaxGCPauseMillis >>>> with >>>> 1s and then 2s). It did not improve things. >>>> I am happy to try anything else to help improve G1. >>>> >>>> Alex A >>>> >>>> -----Original Message----- >>>> From: hotspot-gc-use-bounces at openjdk.java.net >>>> [mailto:hotspot-gc-use-bounces at openjdk.java.net] On Behalf Of >>>> hotspot-gc-use-request at openjdk.java.net >>>> Sent: Thursday, June 16, 2011 12:00 PM >>>> To: hotspot-gc-use at openjdk.java.net >>>> Subject: hotspot-gc-use Digest, Vol 40, Issue 4 >>>> >>>> Send hotspot-gc-use mailing list submissions to >>>> hotspot-gc-use at openjdk.java.net >>>> >>>> To subscribe or unsubscribe via the World Wide Web, visit >>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>>> or, via email, send a message with subject or body 'help' to >>>> hotspot-gc-use-request at openjdk.java.net >>>> >>>> You can reach the person managing the list at >>>> hotspot-gc-use-owner at openjdk.java.net >>>> >>>> When replying, please edit your Subject line so it is more specific >>>> than "Re: Contents of hotspot-gc-use digest..." >>>> >>>> >>>> Today's Topics: >>>> >>>> 1. G1GC: many consecutive full gcs (Stefan Wachter) >>>> >>>> >>>> ---------------------------------------------------------------------- >>>> >>>> Message: 1 >>>> Date: Thu, 16 Jun 2011 09:23:05 +0200 >>>> From: Stefan Wachter >>>> Subject: G1GC: many consecutive full gcs >>>> To: hotspot-gc-use at openjdk.java.net >>>> Message-ID: <4DF9AF59.90708 at gmx.de> >>>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed >>>> >>>> Hi all, >>>> >>>> the G1GC runs nicely for some time but then it starts to have >>>> bursts of consecutive full gcs (see >>>> log below). Can anyone explain that behaviour? >>>> >>>> The JVM is stared with these options: >>>> >>>> -Xmx10000M >>>> -Xms1024M >>>> -XX:MaxPermSize=1024m >>>> -XX:+UseG1GC >>>> -XX:MaxGCPauseMillis=2000 >>>> -XX:GCPauseIntervalMillis=10000 >>>> >>>> --Stefan >>>> >>>> >>>> 31314.569: [GC pause (young) 3286M->3234M(8937M), 0.0485530 secs] >>>> 31374.932: [GC pause (young) 3279M->3236M(8937M), 0.0486560 secs] >>>> 31837.688: [GC pause (young) 4614M->3252M(8937M), 0.0739370 secs] >>>> 33528.784: [GC pause (young) 5997M->3330M(8937M), 0.1543940 secs] >>>> 35372.029: [GC pause (young) 6038M->3405M(8937M), 0.2364930 secs] >>>> 36508.343: [Full GC 8525M->2903M(9678M), 8.9121160 secs] >>>> 38204.291: [Full GC 8647M->2295M(10000M), 7.4105940 secs] >>>> 39038.017: [Full GC 8790M->2092M(6975M), 6.7242110 secs] >>>> 39047.736: [Full GC 8996M->2089M(9998M), 6.0741630 secs] >>>> 39095.551: [Full GC 8936M->2090M(6969M), 5.5573390 secs] >>>> 39104.326: [Full GC 9175M->2092M(9997M), 5.3413860 secs] >>>> 39112.760: [Full GC 9392M->2094M(9978M), 5.4884150 secs] >>>> 41965.304: [Full GC 9549M->2112M(7043M), 6.4774550 secs] >>>> 43830.656: [GC pause (young) 4615M->2309M(7043M), 0.1727720 secs] >>>> 46024.398: [GC pause (young) 4638M->2497M(7043M), 0.3039900 secs] >>>> 46068.009: [Full GC 8832M->2237M(10000M), 6.6717340 secs] >>>> 46077.691: [Full GC 9322M->2187M(9987M), 5.8603570 secs] >>>> 46086.731: [Full GC 9487M->2189M(9996M), 5.8521360 secs] >>>> 46124.713: [Full GC 8792M->2185M(9895M), 5.7782260 secs] >>>> 46133.366: [Full GC 9297M->2188M(9999M), 5.7263360 secs] >>>> 46142.238: [Full GC 9509M->2189M(9985M), 5.5704850 secs] >>>> 47454.530: [Full GC 9676M->2295M(7653M), 7.3169840 secs] >>>> 48163.422: [GC pause (young) 4937M->2431M(7653M), 0.2379570 secs] >>>> 48887.388: [GC pause (young) 5007M->2497M(7653M), 0.1999790 secs] >>>> >>>> >>>> >>>> ------------------------------ >>>> >>>> _______________________________________________ >>>> hotspot-gc-use mailing list >>>> hotspot-gc-use at openjdk.java.net >>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>>> >>>> >>>> End of hotspot-gc-use Digest, Vol 40, Issue 4 >>>> ********************************************* >>>> _______________________________________________ >>>> hotspot-gc-use mailing list >>>> hotspot-gc-use at openjdk.java.net >>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>> >> > > From y.s.ramakrishna at oracle.com Mon Jun 27 10:46:16 2011 From: y.s.ramakrishna at oracle.com (Y. Srinivas Ramakrishna) Date: Mon, 27 Jun 2011 10:46:16 -0700 Subject: G1GC: many consecutive full gcs (was Re: hotspot-gc-use Digest, Vol 40, Issue 4) In-Reply-To: <4E08890A.9030601@gmx.de> References: <4DFA81A3.1050802@oracle.com> <4DFAFB95.7070405@gmx.de> <4DFB6E7B.8000200@oracle.com> <4E08890A.9030601@gmx.de> Message-ID: <4E08C1E8.9000609@oracle.com> No smoking guns yet. I should have asked for this earlier, sorry i forgot... Could you share data from +PrintHeapAtGC +PrintHeapAtGCExtended ? -- ramki On 6/27/2011 6:43 AM, Stefan Wachter wrote: > Hi Ramki, > > I took a heap dump and checked for WeakReferences and Finalizer objects. > > The heap dump contained 36,071,040 instances with 12,405 instances being WeakReference instances. I > am not sure if this is a lot or not. The "Basic info" section that is displayed in NetBeans reports > "Number of objects pending for finalization: 0". Corresponds this to the number of Finalizer objects > you asked for? > > The complete heap summary is: > > Total bytes: 2,472,537,236 > Total classes: 15,953 > Total instances: 36,071,040 > Classloaders: 648 > GC roots: 3,138 > Number of objects pending for finalization: 0 > > > Cheers, > > Stefan > > On 06/17/2011 05:10 PM, Y. Srinivas Ramakrishna wrote: >> Thanks for the responses and the extra information, Stefan and Alex. >> >> Stefan, please try PermSize == MaxPermSize at the next opportunity, >> although it's likely not to have any effect based on a preliminary >> reading of the info you provided below. >> >> As regards class histogram, please note that this would induce a >> pause as long as the full gc's you see (or may be a bit shorter), >> so use it with care when you obtain that information. >> >> Meanwhile, we'll look over the information you have provided and >> get back to you once we have made some headway. Thanks for bringing >> this to our attention and for the data so far. >> >> -- ramki >> >> On 6/17/2011 12:00 AM, Stefan Wachter wrote: >>> Hi Ramki, >>> >>> thanks for your interest. See my answers below. >>> >>> Cheers, >>> >>> Stefan >>> >>> >>> On 06/17/2011 12:20 AM, Y. S. Ramakrishna wrote: >>>> Stefan and Alex -- >>>> >>>> (1) what's the version of the JVM used? >>> >>> Java(TM) SE Runtime Environment (build 1.6.0_24-b07) >>> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode) >>> >>>> >>>> (2) could you try PermSize == MaxPermSize and see if it made any >>>> difference? >>> >>> PermSize was not set until now. I try this setting and can report results the next time our cluster >>> nodes are switched. >>> >>>> >>>> (3) do you have lots of WeakReference or Finalizer objects in your system? >>> >>> I have no information about this. I guess a heap dump would provide that information. I will ask our >>> admins if it is possible to take a heap dump. >>> >>>> >>>> (4) is what you have below your complete GC log (i.e. no gaps between first and last line?) >>> >>> Yes, that is the original output of the GC. No lines were removed. >>>> >>>> (5) what do you see when using CMS (in case you tried it)? >>> >>> CMS seems to behave better (see attached gc.log). The settings used with CMS are: >>> >>> -Xmx10000M >>> -Xms1024M >>> -XX:MaxPermSize=1024m >>> -XX:+PrintGCDateStamps >>> -XX:+UseConcMarkSweepGC >>> >>>> >>>> (6) do you have a simple test case that you can share with >>>> us that exhibits this behaviour? >>> >>> Unfortunately I have no simple test case. The output is from our life system running a tomcat with >>> various wars deployed. >>> >>>> >>>> thanks! >>>> -- ramki >>>> >>>> On 06/16/11 14:02, Alex Aisinzon wrote: >>>>> Stefan >>>>> >>>>> I have had similar experience with G1 and reported it here. >>>>> I tried the option to try to limit GC pauses (-XX:MaxGCPauseMillis with >>>>> 1s and then 2s). It did not improve things. >>>>> I am happy to try anything else to help improve G1. >>>>> >>>>> Alex A >>>>> >>>>> -----Original Message----- >>>>> From: hotspot-gc-use-bounces at openjdk.java.net >>>>> [mailto:hotspot-gc-use-bounces at openjdk.java.net] On Behalf Of >>>>> hotspot-gc-use-request at openjdk.java.net >>>>> Sent: Thursday, June 16, 2011 12:00 PM >>>>> To: hotspot-gc-use at openjdk.java.net >>>>> Subject: hotspot-gc-use Digest, Vol 40, Issue 4 >>>>> >>>>> Send hotspot-gc-use mailing list submissions to >>>>> hotspot-gc-use at openjdk.java.net >>>>> >>>>> To subscribe or unsubscribe via the World Wide Web, visit >>>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>>>> or, via email, send a message with subject or body 'help' to >>>>> hotspot-gc-use-request at openjdk.java.net >>>>> >>>>> You can reach the person managing the list at >>>>> hotspot-gc-use-owner at openjdk.java.net >>>>> >>>>> When replying, please edit your Subject line so it is more specific >>>>> than "Re: Contents of hotspot-gc-use digest..." >>>>> >>>>> >>>>> Today's Topics: >>>>> >>>>> 1. G1GC: many consecutive full gcs (Stefan Wachter) >>>>> >>>>> >>>>> ---------------------------------------------------------------------- >>>>> >>>>> Message: 1 >>>>> Date: Thu, 16 Jun 2011 09:23:05 +0200 >>>>> From: Stefan Wachter >>>>> Subject: G1GC: many consecutive full gcs >>>>> To: hotspot-gc-use at openjdk.java.net >>>>> Message-ID: <4DF9AF59.90708 at gmx.de> >>>>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed >>>>> >>>>> Hi all, >>>>> >>>>> the G1GC runs nicely for some time but then it starts to have bursts of consecutive full gcs (see >>>>> log below). Can anyone explain that behaviour? >>>>> >>>>> The JVM is stared with these options: >>>>> >>>>> -Xmx10000M >>>>> -Xms1024M >>>>> -XX:MaxPermSize=1024m >>>>> -XX:+UseG1GC >>>>> -XX:MaxGCPauseMillis=2000 >>>>> -XX:GCPauseIntervalMillis=10000 >>>>> >>>>> --Stefan >>>>> >>>>> >>>>> 31314.569: [GC pause (young) 3286M->3234M(8937M), 0.0485530 secs] >>>>> 31374.932: [GC pause (young) 3279M->3236M(8937M), 0.0486560 secs] >>>>> 31837.688: [GC pause (young) 4614M->3252M(8937M), 0.0739370 secs] >>>>> 33528.784: [GC pause (young) 5997M->3330M(8937M), 0.1543940 secs] >>>>> 35372.029: [GC pause (young) 6038M->3405M(8937M), 0.2364930 secs] >>>>> 36508.343: [Full GC 8525M->2903M(9678M), 8.9121160 secs] >>>>> 38204.291: [Full GC 8647M->2295M(10000M), 7.4105940 secs] >>>>> 39038.017: [Full GC 8790M->2092M(6975M), 6.7242110 secs] >>>>> 39047.736: [Full GC 8996M->2089M(9998M), 6.0741630 secs] >>>>> 39095.551: [Full GC 8936M->2090M(6969M), 5.5573390 secs] >>>>> 39104.326: [Full GC 9175M->2092M(9997M), 5.3413860 secs] >>>>> 39112.760: [Full GC 9392M->2094M(9978M), 5.4884150 secs] >>>>> 41965.304: [Full GC 9549M->2112M(7043M), 6.4774550 secs] >>>>> 43830.656: [GC pause (young) 4615M->2309M(7043M), 0.1727720 secs] >>>>> 46024.398: [GC pause (young) 4638M->2497M(7043M), 0.3039900 secs] >>>>> 46068.009: [Full GC 8832M->2237M(10000M), 6.6717340 secs] >>>>> 46077.691: [Full GC 9322M->2187M(9987M), 5.8603570 secs] >>>>> 46086.731: [Full GC 9487M->2189M(9996M), 5.8521360 secs] >>>>> 46124.713: [Full GC 8792M->2185M(9895M), 5.7782260 secs] >>>>> 46133.366: [Full GC 9297M->2188M(9999M), 5.7263360 secs] >>>>> 46142.238: [Full GC 9509M->2189M(9985M), 5.5704850 secs] >>>>> 47454.530: [Full GC 9676M->2295M(7653M), 7.3169840 secs] >>>>> 48163.422: [GC pause (young) 4937M->2431M(7653M), 0.2379570 secs] >>>>> 48887.388: [GC pause (young) 5007M->2497M(7653M), 0.1999790 secs] >>>>> >>>>> >>>>> >>>>> ------------------------------ >>>>> >>>>> _______________________________________________ >>>>> hotspot-gc-use mailing list >>>>> hotspot-gc-use at openjdk.java.net >>>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>>>> >>>>> >>>>> End of hotspot-gc-use Digest, Vol 40, Issue 4 >>>>> ********************************************* >>>>> _______________________________________________ >>>>> hotspot-gc-use mailing list >>>>> hotspot-gc-use at openjdk.java.net >>>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>>> >>> >> >> >