From jamesnichols3 at gmail.com Mon Jul 14 06:12:03 2008 From: jamesnichols3 at gmail.com (James Nichols) Date: Mon, 14 Jul 2008 09:12:03 -0400 Subject: Failing promption guarantee? Message-ID: <83a51e120807140612n42600e3cma1be6588d146d65b@mail.gmail.com> Hi, I have two questions. I'm pretty sure I'm on the right track with this, but I'm getting close to pushing out a new release and would really love an extra set of eyes on this problem. The first is that I think I might be hitting the promotion guarantee, as shown by this GC output below. As you can see, both eden and from are 100% full, but I'm pretty sure that it's all garbage, as evidenced by the precipitous drop in the young and tenured generation. Does the output for the actual GC saying ParNew indicate that it was trying to do a young collection, but ended up doing a full GC instead? It also looks like CMS would've helped out in this case. 157148.797: [GC {Heap before gc invocations=545: par new generation total 1100288K, used 1100288K [0x00002aaab21b0000, 0x00002aaafedb0000, 0x00002aaafedb0000) eden space 943104K, 100% used [0x00002aaab21b0000, 0x00002aaaebab0000, 0x00002aaaebab0000) from space 157184K, 100% used [0x00002aaaf5430000, 0x00002aaafedb0000, 0x00002aaafedb0000) to space 157184K, 0% used [0x00002aaaebab0000, 0x00002aaaebab0000, 0x00002aaaf5430000) tenured generation total 2936832K, used 2014622K [0x00002aaafedb0000, 0x00002aabb21b0000, 0x00002aabb21b0000) the space 2936832K, 68% used [0x00002aaafedb0000, 0x00002aab79d17ba0, 0x00002aab79d17c00, 0x00002aabb21b0000) compacting perm gen total 262144K, used 136065K [0x00002aabb21b0000, 0x00002aabc21b0000, 0x00002aabc21b0000) the space 262144K, 51% used [0x00002aabb21b0000, 0x00002aabba6906a0, 0x00002aabba690800, 0x00002aabc21b0000) No shared spaces configured. 157148.797: [ParNew: 1100288K->1100288K(1100288K), 0.0000400 secs]157148.797: [Tenured: 2014622K->1135456K(2936832K), 17.1537650 secs] 3114910K->1135456K(4037120K)Heap after gc invocations=546: par new generation total 1100288K, used 0K [0x00002aaab21b0000, 0x00002aaafedb0000, 0x00002aaafedb0000) eden space 943104K, 0% used [0x00002aaab21b0000, 0x00002aaab21b0000, 0x00002aaaebab0000) from space 157184K, 0% used [0x00002aaaf5430000, 0x00002aaaf5430000, 0x00002aaafedb0000) to space 157184K, 0% used [0x00002aaaebab0000, 0x00002aaaebab0000, 0x00002aaaf5430000) tenured generation total 2936832K, used 1135456K [0x00002aaafedb0000, 0x00002aabb21b0000, 0x00002aabb21b0000) the space 2936832K, 38% used [0x00002aaafedb0000, 0x00002aab44288078, 0x00002aab44288200, 0x00002aabb21b0000) compacting perm gen total 262144K, used 136065K [0x00002aabb21b0000, 0x00002aabc21b0000, 0x00002aabc21b0000) the space 262144K, 51% used [0x00002aabb21b0000, 0x00002aabba6906a0, 0x00002aabba690800, 0x00002aabc21b0000) No shared spaces configured. } , 17.1543400 secs] Total time for which application threads were stopped: 17.2148700 seconds Here are my JVM arguments: JAVA_OPTS="-Xms4096m -Xmx4096m -XX:NewSize=1228M -XX:MaxNewSize=1228M -XX:MaxTenuringThreshold=4 -XX:SurvivorRatio=6 -XX:+ScavengeBeforeFullGC -XX:PermSize=256M -XX:MaxPermSize=256M -XX:+UseParNewGC -XX:ParallelGCThreads=3 -verbosegc -XX:+DisableExplicitGC -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -XX:+PrintClassHistogram -Xloggc:/var/log/jboss/gc.dat -Dsun.net.client.defaultConnectTimeout=10000" The second question is that I'm on JMV rev jdk1.5.0_12. I see that rev 16 is out. Is there a way to get a list of any GC bug fixes in this rev? Also, has is there a way to gauge the stability of this rev know that it has been in the wild for a while? I'm just a little nervous to make the change since I got burned pretty badly with like rev 08. Jim -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20080714/70283eb3/attachment.html From Jon.Masamitsu at Sun.COM Mon Jul 14 07:25:33 2008 From: Jon.Masamitsu at Sun.COM (Jon Masamitsu) Date: Mon, 14 Jul 2008 07:25:33 -0700 Subject: Failing promption guarantee? In-Reply-To: <83a51e120807140612n42600e3cma1be6588d146d65b@mail.gmail.com> References: <83a51e120807140612n42600e3cma1be6588d146d65b@mail.gmail.com> Message-ID: <487B61DD.8020404@Sun.COM> Jim, Yes, the log entry is for a young collection that was started but which could not be done because there was not enough room in the tenured (old) generation to safely do the young collection. You can see the young collection did not collect any garbage > 157148.797: [ParNew: 1100288K->1100288K(1100288K), The before occupancy 1100288K is the same as the after occupancy 1100288K. That's followed by the indication that a tenured collection was done. 157148.797: Tenured: 2014622K->1135456K(2936832K) That only happens when there is not enough room in the tenured gen to support the young gen collection (yes, the promotion guarantee). CMS is the low pause collector so would be helpful if you were trying to avoid the long 17 sec pause. Throughput with the CMS collector is typically less so depending on whether you want the low pause or the throughput, you can make your choice. You're using UseParNewGC to do the parallel collection in the young gen. It's is perhaps more common to use UseParallelGC to do the parallel collection. Both collectors do a parallel collection of the young gen and pretty much perform equally well but some applications (because of their allocation pattern) do better with one than the other. It's something you can experiment with if you're interested. The UseParallelGC collector has a relaxed promotion guarantee so in situations where the heap is nearly full, it generally performs better than UseParNewGC (at least in jdk5). In the past I've seen lists of all bugs fixed in a release in the release notes. Sorry, I don't know if that is still the practice. I haven't heard anything about the relative stability of jdk5_16 so I'm no help there. Jon James Nichols wrote: > Hi, > > I have two questions. > > I'm pretty sure I'm on the right track with this, but I'm getting close > to pushing out a new release and would really love an extra set of eyes > on this problem. The first is that I think I might be hitting the > promotion guarantee, as shown by this GC output below. As you can see, > both eden and from are 100% full, but I'm pretty sure that it's all > garbage, as evidenced by the precipitous drop in the young and tenured > generation. Does the output for the actual GC saying ParNew indicate > that it was trying to do a young collection, but ended up doing a full > GC instead? It also looks like CMS would've helped out in this case. > > 157148.797: [GC {Heap before gc invocations=545: > par new generation total 1100288K, used 1100288K [0x00002aaab21b0000, > 0x00002aaafedb0000, 0x00002aaafedb0000) > eden space 943104K, 100% used [0x00002aaab21b0000, 0x00002aaaebab0000, > 0x00002aaaebab0000) > from space 157184K, 100% used [0x00002aaaf5430000, 0x00002aaafedb0000, > 0x00002aaafedb0000) > to space 157184K, 0% used [0x00002aaaebab0000, 0x00002aaaebab0000, > 0x00002aaaf5430000) > tenured generation total 2936832K, used 2014622K [0x00002aaafedb0000, > 0x00002aabb21b0000, 0x00002aabb21b0000) > the space 2936832K, 68% used [0x00002aaafedb0000, > 0x00002aab79d17ba0, 0x00002aab79d17c00, 0x00002aabb21b0000) > compacting perm gen total 262144K, used 136065K [0x00002aabb21b0000, > 0x00002aabc21b0000, 0x00002aabc21b0000) > the space 262144K, 51% used [0x00002aabb21b0000, 0x00002aabba6906a0, > 0x00002aabba690800, 0x00002aabc21b0000) > No shared spaces configured. > 157148.797: [ParNew: 1100288K->1100288K(1100288K), 0.0000400 > secs]157148.797: [Tenured: 2014622K->1135456K(2936832K), 17.1537650 > secs] 3114910K->1135456K(4037120K)Heap after gc invocations=546: > par new generation total 1100288K, used 0K [0x00002aaab21b0000, > 0x00002aaafedb0000, 0x00002aaafedb0000) > eden space 943104K, 0% used [0x00002aaab21b0000, 0x00002aaab21b0000, > 0x00002aaaebab0000) > from space 157184K, 0% used [0x00002aaaf5430000, 0x00002aaaf5430000, > 0x00002aaafedb0000) > to space 157184K, 0% used [0x00002aaaebab0000, 0x00002aaaebab0000, > 0x00002aaaf5430000) > tenured generation total 2936832K, used 1135456K [0x00002aaafedb0000, > 0x00002aabb21b0000, 0x00002aabb21b0000) > the space 2936832K, 38% used [0x00002aaafedb0000, > 0x00002aab44288078, 0x00002aab44288200, 0x00002aabb21b0000) > compacting perm gen total 262144K, used 136065K [0x00002aabb21b0000, > 0x00002aabc21b0000, 0x00002aabc21b0000) > the space 262144K, 51% used [0x00002aabb21b0000, 0x00002aabba6906a0, > 0x00002aabba690800, 0x00002aabc21b0000) > No shared spaces configured. > } > , 17.1543400 secs] > Total time for which application threads were stopped: 17.2148700 seconds > > > Here are my JVM arguments: > > JAVA_OPTS="-Xms4096m -Xmx4096m -XX:NewSize=1228M -XX:MaxNewSize=1228M > -XX:MaxTenuringThreshold=4 -XX:SurvivorRatio=6 -XX:+ScavengeBeforeFullGC > -XX:PermSize=256M -XX:MaxPermSize=256M -XX:+UseParNewGC > -XX:ParallelGCThreads=3 -verbosegc -XX:+DisableExplicitGC > -XX:+PrintTenuringDistribution -XX:+PrintGCDetails > -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC > -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime > -XX:+PrintClassHistogram -Xloggc:/var/log/jboss/gc.dat > -Dsun.net.client.defaultConnectTimeout=10000" > > > > The second question is that I'm on JMV rev jdk1.5.0_12. I see that rev > 16 is out. Is there a way to get a list of any GC bug fixes in this > rev? Also, has is there a way to gauge the stability of this rev know > that it has been in the wild for a while? I'm just a little nervous to > make the change since I got burned pretty badly with like rev 08. > > Jim > > > ------------------------------------------------------------------------ > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From doug.jones at eds.com Mon Jul 14 14:21:22 2008 From: doug.jones at eds.com (Jones, Doug H) Date: Tue, 15 Jul 2008 09:21:22 +1200 Subject: Failing promption guarantee? In-Reply-To: <83a51e120807140612n42600e3cma1be6588d146d65b@mail.gmail.com> References: <83a51e120807140612n42600e3cma1be6588d146d65b@mail.gmail.com> Message-ID: <027FCB5D4C65CC4CA714042A4EE8CC6B048AA7CB@nzprm231.apac.corp.eds.com> Hi Jim, Your log extract doesn't show how long normal collections of the tenured area are taking. I suspect in the situation below it will revert to a single-thread Stop-the-World collection, hence the 17 seconds. The precursor of your problem is the line 'tenured generation total 2936832K, used 2014622K' in the first heap dump - there is less than NewSize free space remaining in tenured, so forcing the collection of tenured before the scavenge. CMS GC will give you much shorter pauses during collection of the tenured area, though as Jon M rightly says at the expense of a greater use of resources. One other advantage of CMS GC is that it allows you to control the percent full when a GC kicks in (so avoiding such 'promotion failures'). Normally the JVM should dynamically adjust this, using a JIT plus a bit to spare policy. But if your app suddenly creates a lots more objects over a short period of time, it won't be able to anticipate this. Doug. ________________________________ From: hotspot-gc-use-bounces at openjdk.java.net [mailto:hotspot-gc-use-bounces at openjdk.java.net] On Behalf Of James Nichols Sent: Tuesday, 15 July 2008 1:12 a.m. To: hotspot-gc-use at openjdk.java.net Subject: Failing promption guarantee? Hi, I have two questions. I'm pretty sure I'm on the right track with this, but I'm getting close to pushing out a new release and would really love an extra set of eyes on this problem. The first is that I think I might be hitting the promotion guarantee, as shown by this GC output below. As you can see, both eden and from are 100% full, but I'm pretty sure that it's all garbage, as evidenced by the precipitous drop in the young and tenured generation. Does the output for the actual GC saying ParNew indicate that it was trying to do a young collection, but ended up doing a full GC instead? It also looks like CMS would've helped out in this case. 157148.797: [GC {Heap before gc invocations=545: par new generation total 1100288K, used 1100288K [0x00002aaab21b0000, 0x00002aaafedb0000, 0x00002aaafedb0000) eden space 943104K, 100% used [0x00002aaab21b0000, 0x00002aaaebab0000, 0x00002aaaebab0000) from space 157184K, 100% used [0x00002aaaf5430000, 0x00002aaafedb0000, 0x00002aaafedb0000) to space 157184K, 0% used [0x00002aaaebab0000, 0x00002aaaebab0000, 0x00002aaaf5430000) tenured generation total 2936832K, used 2014622K [0x00002aaafedb0000, 0x00002aabb21b0000, 0x00002aabb21b0000) the space 2936832K, 68% used [0x00002aaafedb0000, 0x00002aab79d17ba0, 0x00002aab79d17c00, 0x00002aabb21b0000) compacting perm gen total 262144K, used 136065K [0x00002aabb21b0000, 0x00002aabc21b0000, 0x00002aabc21b0000) the space 262144K, 51% used [0x00002aabb21b0000, 0x00002aabba6906a0, 0x00002aabba690800, 0x00002aabc21b0000) No shared spaces configured. 157148.797: [ParNew: 1100288K->1100288K(1100288K), 0.0000400 secs]157148.797: [Tenured: 2014622K->1135456K(2936832K), 17.1537650 secs] 3114910K->1135456K(4037120K)Heap after gc invocations=546: par new generation total 1100288K, used 0K [0x00002aaab21b0000, 0x00002aaafedb0000, 0x00002aaafedb0000) eden space 943104K, 0% used [0x00002aaab21b0000, 0x00002aaab21b0000, 0x00002aaaebab0000) from space 157184K, 0% used [0x00002aaaf5430000, 0x00002aaaf5430000, 0x00002aaafedb0000) to space 157184K, 0% used [0x00002aaaebab0000, 0x00002aaaebab0000, 0x00002aaaf5430000) tenured generation total 2936832K, used 1135456K [0x00002aaafedb0000, 0x00002aabb21b0000, 0x00002aabb21b0000) the space 2936832K, 38% used [0x00002aaafedb0000, 0x00002aab44288078, 0x00002aab44288200, 0x00002aabb21b0000) compacting perm gen total 262144K, used 136065K [0x00002aabb21b0000, 0x00002aabc21b0000, 0x00002aabc21b0000) the space 262144K, 51% used [0x00002aabb21b0000, 0x00002aabba6906a0, 0x00002aabba690800, 0x00002aabc21b0000) No shared spaces configured. } , 17.1543400 secs] Total time for which application threads were stopped: 17.2148700 seconds Here are my JVM arguments: JAVA_OPTS="-Xms4096m -Xmx4096m -XX:NewSize=1228M -XX:MaxNewSize=1228M -XX:MaxTenuringThreshold=4 -XX:SurvivorRatio=6 -XX:+ScavengeBeforeFullGC -XX:PermSize=256M -XX:MaxPermSize=256M -XX:+UseParNewGC -XX:ParallelGCThreads=3 -verbosegc -XX:+DisableExplicitGC -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -XX:+PrintClassHistogram -Xloggc:/var/log/jboss/gc.dat -Dsun.net.client.defaultConnectTimeout=10000" The second question is that I'm on JMV rev jdk1.5.0_12. I see that rev 16 is out. Is there a way to get a list of any GC bug fixes in this rev? Also, has is there a way to gauge the stability of this rev know that it has been in the wild for a while? I'm just a little nervous to make the change since I got burned pretty badly with like rev 08. Jim -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20080715/942dad13/attachment.html From jamesnichols3 at gmail.com Wed Jul 16 14:49:56 2008 From: jamesnichols3 at gmail.com (James Nichols) Date: Wed, 16 Jul 2008 17:49:56 -0400 Subject: Very short application run time and stops Message-ID: <83a51e120807161449h743239ffgf676a188bb6e85f0@mail.gmail.com> I was just running some workload through an application on a test environment. For some reason, I had a run of very short application run times and pauses, as shown in this GC snippet. The young generation and old generation both had low occupancy rates right before this occurred. After this log output, the next GC showed Eden at 99% full, which was then collected in under a second. I'm kinda perplexed as to what is going on when this occurred as there isn't any GC related information output during this time. When the workload began against the application a bunch of threads became active doing work, but I don't believe they were creating excessive amounts of garbage. Total time for which application threads were stopped: 0.2914130 seconds Application time: 2.0577050 seconds Total time for which application threads were stopped: 0.0234770 seconds Application time: 0.0021460 seconds Total time for which application threads were stopped: 0.0316840 seconds Application time: 0.0016000 seconds Total time for which application threads were stopped: 0.0218700 seconds Application time: 0.1401370 seconds Total time for which application threads were stopped: 0.0401360 seconds Application time: 0.0007750 seconds Total time for which application threads were stopped: 0.0235050 seconds Application time: 0.0018830 seconds Total time for which application threads were stopped: 0.0327830 seconds Application time: 0.0010240 seconds Total time for which application threads were stopped: 0.0218930 seconds Application time: 0.0008780 seconds Total time for which application threads were stopped: 0.0375390 seconds Application time: 0.0006810 seconds Total time for which application threads were stopped: 0.0212840 seconds Application time: 0.0010380 seconds Total time for which application threads were stopped: 0.0314600 seconds Application time: 0.0007640 seconds Attached is the zipped up GC log. Here are my GC arguments: /usr/java/jdk1.5.0_12/bin/java -Dprogram.name=run.sh -server -Xms4096m -Xmx4096m -XX:NewSize=1228M -XX:MaxNewSize=1228M -XX:MaxTenuringThreshold=4 -XX:SurvivorRatio=6 -XX:+ScavengeBeforeFullGC -XX:PermSize=256M -XX:MaxPermSize=256M -XX:+HandlePromotionFailure -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ParallelGCThreads=3 -XX:+CMSParallelRemarkEnabled -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -verbosegc -XX:+DisableExplicitGC -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -XX:+PrintClassHistogram -Xloggc:/var/log/jboss/gc.dat -Dsun.net.client.defaultConnectTimeout=10000 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20080716/ff6960a4/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: gc.zip Type: application/zip Size: 172444 bytes Desc: not available Url : http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20080716/ff6960a4/attachment.zip From justin at techadvise.com Tue Jul 22 08:07:06 2008 From: justin at techadvise.com (Justin Ellison) Date: Tue, 22 Jul 2008 10:07:06 -0500 Subject: Work queue overflow during re-mark? Message-ID: Hello all, I'm running 1.4.2_17 on Solaris 9 Sparc. When I was running 1.4.2_12 on the same system, I was experiencing bug 6558100. I was able to get a fix backported to 1.4.2, and the fix was released in 1.4.2_17. The bug where the VM crashed is certainly resolved, but occasionally I still get *a lot* of these lines printed to stdout: Java HotSpot(TM) Server VM warning: Work queue overflow during re-mark By a lot, I mean 144,347 times since the VM was started 6 days ago. The usage scenario is a website running jsps that make heavy use of caching in heap. The log entries seem to follow cases where the caches are being cleared. This makes sense, as a cache full of objects that all of a sudden become available for GC might fill a fixed queue. My question is twofold - what is the impact of this, and is there any workaround for it? I'm willing to sacrifice a little heap to increase that array size if need be. Also, do those objects get GC'd next time around, or do they stick around? Justin From Jon.Masamitsu at Sun.COM Tue Jul 22 12:22:20 2008 From: Jon.Masamitsu at Sun.COM (Jon Masamitsu) Date: Tue, 22 Jul 2008 12:22:20 -0700 Subject: Work queue overflow during re-mark? In-Reply-To: References: Message-ID: <4886336C.1060403@Sun.COM> Justin, The queue mentioned is a per thread queue that is used to remember objects that need to be examined. When the queue overflows, the objects are put in a global queue which expands as needed and which is shared by other GC threads. The objects do get examined in the current GC (it's just remembered via the global queue). The cost of an overflow is that the global queue is used and access to the global queue must be synchronized. I don't recall if there is a workaround for this. I don't think there is. Maybe someone else on the alias knows. Jon Justin Ellison wrote: > Hello all, > > I'm running 1.4.2_17 on Solaris 9 Sparc. > > When I was running 1.4.2_12 on the same system, I was experiencing bug > 6558100. I was able to get a fix backported to 1.4.2, and the fix was > released in 1.4.2_17. > > The bug where the VM crashed is certainly resolved, but occasionally I > still get *a lot* of these lines printed to stdout: > Java HotSpot(TM) Server VM warning: Work queue overflow during re-mark > > By a lot, I mean 144,347 times since the VM was started 6 days ago. > > The usage scenario is a website running jsps that make heavy use of > caching in heap. The log entries seem to follow cases where the > caches are being cleared. This makes sense, as a cache full of > objects that all of a sudden become available for GC might fill a > fixed queue. > > My question is twofold - what is the impact of this, and is there any > workaround for it? I'm willing to sacrifice a little heap to increase > that array size if need be. Also, do those objects get GC'd next time > around, or do they stick around? > > Justin > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From michael.finocchiaro at gmail.com Thu Jul 31 08:01:18 2008 From: michael.finocchiaro at gmail.com (Michael Finocchiaro) Date: Thu, 31 Jul 2008 17:01:18 +0200 Subject: Blog posting about HotSpot Message-ID: <8b61e5430807310801i298cd13fsabf3257cd3f2887e@mail.gmail.com> For those who are interested in an overview of what I have learned about HotSpot and Garbage Collection, please see the pdf on my blog here Feel welcome to browse and comment on the rest of the blog (just starting to be honest) but I thought folks might get something out of my garbage collection info. Cheers, Fino -- Michael Finocchiaro michael.finocchiaro at gmail.com Mobile Telephone: +33 6 67 90 64 39 MSN: le_fino at hotmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20080731/9f527d0e/attachment.html