June 2008 - hotspot-gc-dev

Re: GCOverheadLimit does not work ?
by Jon Masamitsu 09 Jun '08

09 Jun '08

David, There is a bug somewhere here that relates to the GCOverheadLimit. I think the problem is that the survivor spaces are not being used and the survivor space is large enough so that the calculation of the amount of free space in the heap is > 2%. The survivor spaces are not being used because only full collections are being done later in the run and the survivor spaces are used for minor (young generation) collections. Though the description of the GCOverheadLimit refers to the amount of the heap collected, it really should just say that amount of free space in the heap. That's a bug in the specification for the parameter. I'm not sure that an out-of-memory should be thrown if there is 2% of the heap free - that's in keeping with trying to minimize the false positives resulting from this feature. Jon David Tavoularis wrote: > Hi, > > It seems that the mechanism GCOverheadLimit did not trigger, even if more than 98% of the total time is spent in garbage collection and less than 2% of the heap is recovered. The process spent 3 hours in this state, performing Full GC after Full GC. > > I am using Parallel Collector on Java6u4/Solaris9. Is it a bug or maybe I do not understand this feature ? > > "TimeStamp","GC duration","Duration between 2 FullGC" > 57356.444 27.46 27.471 > 57383.915 27.14 27.137 > 57411.052 27.63 27.738 > 57438.79 33.72 33.724 > 57472.514 30.07 30.087 > 57502.601 27.84 27.842 > 57530.443 28.46 28.519 > 57558.962 29.03 29.032 > 57587.994 29.31 29.321 > > Full GC logs are available at http://people.via.ecp.fr/~tony/tmp/gc_200802282216.zip > > Thanks in advance > _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use(a)openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use

1 0

Proposal: MaxTenuringThreshold and age field size changes
by Ciciora, Paul 06 Jun '08

06 Jun '08

I just came across a neglected process here that was last tuned so long ago it was still using the never tenure approach. So yesterday I was down stairs retuning. My goal was to reduce object promotions, minor collection frequency and reduce collection times. So here is what I did - This is before JVM_ARGS="${JVM_ARGS} -XX:NewSize=100m" JVM_ARGS="${JVM_ARGS} -XX:MaxNewSize=100m" JVM_ARGS="${JVM_ARGS} -XX:SurvivorRatio=128" JVM_ARGS="${JVM_ARGS} -XX:MaxTenuringThreshold=0" JVM_ARGS="${JVM_ARGS} -XX:TargetSurvivorRatio=100" JVM_ARGS="${JVM_ARGS} -XX:CMSInitiatingOccupancyFraction=40" This is first guess based on what I thought would be close JVM_ARGS="${JVM_ARGS} -XX:NewSize=200m" JVM_ARGS="${JVM_ARGS} -XX:MaxNewSize=200m" JVM_ARGS="${JVM_ARGS} -XX:SurvivorRatio=10" JVM_ARGS="${JVM_ARGS} -XX:MaxTenuringThreshold=15" JVM_ARGS="${JVM_ARGS} -XX:-CMSPermGenPrecleaningEnabled" JVM_ARGS="${JVM_ARGS} -XX:TargetSurvivorRatio=50" JVM_ARGS="${JVM_ARGS} -XX:CMSInitiatingOccupancyFraction=60" The test ran really well. I achieved all three goals right off the bat. So the only thing to do was adjust the tenuring to minimize the copies and still minimize promotions as well. The data indicated 4 slots would be optimal. After re-running my numbers did not improve. In fact the collection time increased slightly. So I upped the slots to 6 to maximize object death at the expense of an extra 2 copies. The results were significantly better. The lesson here is promotions are much more expensive than copies. I thought I'd share that. -----Original Message----- From: hotspot-gc-dev-bounces(a)openjdk.java.net [mailto:hotspot-gc-dev-bounces@openjdk.java.net] On Behalf Of hotspot-gc-dev-request(a)openjdk.java.net Sent: Thursday, June 05, 2008 2:00 PM To: hotspot-gc-dev(a)openjdk.java.net Subject: hotspot-gc-dev Digest, Vol 12, Issue 4 Send hotspot-gc-dev mailing list submissions to hotspot-gc-dev(a)openjdk.java.net To subscribe or unsubscribe via the World Wide Web, visit http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-dev or, via email, send a message with subject or body 'help' to hotspot-gc-dev-request(a)openjdk.java.net You can reach the person managing the list at hotspot-gc-dev-owner(a)openjdk.java.net When replying, please edit your Subject line so it is more specific than "Re: Contents of hotspot-gc-dev digest..." Today's Topics: 1. Re: Proposal: MaxTenuringThreshold and age field size changes (Y Srinivas Ramakrishna) 2. Re: Proposal: MaxTenuringThreshold and age field size changes (kirk) 3. Re: Proposal: MaxTenuringThreshold and age field size changes (Jon Masamitsu) 4. hg: jdk7/hotspot-gc/hotspot: 6629727: assertion in set_trap_state() in methodDataOop.hpp is too strong. (jon.masamitsu(a)sun.com) ---------------------------------------------------------------------- Message: 1 Date: Thu, 05 Jun 2008 01:03:11 -0700 From: Y Srinivas Ramakrishna <Y.S.Ramakrishna(a)Sun.COM> Subject: Re: Proposal: MaxTenuringThreshold and age field size changes To: Nicolas Michael <email(a)nmichael.de> Cc: Tony Printezis <tony.printezis(a)Sun.COM>, hotspot-gc-dev(a)openjdk.java.net Message-ID: <f81dbe8d5a24.48473b4f(a)sun.com> Content-Type: text/plain; charset=us-ascii Hi Nick -- You make a good point. I think there are scenarios in which the use of PLAB's in the survivor spaces can cause survivor space overflow, even when the total space in the survivor space should have otherwise been enough. That then sets in motion a series of "nepotistical" cycles, which are caused when young objects Z that are prematurely promoted die in the old generation while holding references to now dead objects Z' in the young generation. Call these objects "zombies" because they are dead but not recognized as such. Z keep Z' alive, because a scavenge does not know that Z is dead, because it considers all references from the old gen as roots. Worse, if Z' has references to Z and Z' stays in the young generation forever (which they can under the circumstances you describe) then Z will not be recognized as dead by a CMS collection (which currently treats all objects in the young generation as roots). This is a well-understood problem when spaces are collected independently in this manner. (The workaround is to have the CMS collector not treat the young generation as a source of roots but rather to mark through the young gen objects starting from roots.) Of course when MTT=15, then eventually every such Z' will be forced to promote to the old gen and the garbage cycle will all (hopefully) move into the old gen and thence will be reclaimed by the old gen CMS collection. This would explain the behaviour difference you saw. Now to come to the first point I made above: the use of multiple scavenger threads and their use of PLABs can sometimes cause this kind of overflow to happen (especially if there is the occasional large object). You might want to switch off the use of survivor space PLAB's (or fix them at a vey small modest value) or just use a single-threaded scavenger and see that this kind of behaviour might reduce because overflow becomes much less likely. I think there is an open bug to tune the adaptive PLAB sizing code to eliminate this kind of pathological behaviour, but we have not had the opportunity to get to that bug. If you have PrintGCDetails logs, they would probably show the premature promotion happening. PrintTenuringDistribution would not show any objects of age greater than 2 initially, yet some objects would be seen to be promoted, and then by virtue of the cross-generational references from a promoted zombie, we would artificially expand the lifetime of objects in the young generation (creating zombies Z') and so on. I believe it was partially this kind of behaviour on the part of generational scavenger implementations that caused some people in the past to start advocating the clearing of all references in objects that they knew they would be dropping references to (which of course we all know is difficult to do correctly and fraught with all kinds of problems and errors). -- ramki ------------------------------ Message: 2 Date: Thu, 05 Jun 2008 09:51:42 +0200 From: kirk <kirk.pepperdine(a)gmail.com> Subject: Re: Proposal: MaxTenuringThreshold and age field size changes To: Jon Masamitsu <Jon.Masamitsu(a)Sun.COM> Cc: hotspot-gc-dev(a)openjdk.java.net, Tony Printezis <tony.printezis(a)Sun.COM>, Y Srinivas Ramakrishna <Y.S.Ramakrishna(a)Sun.COM> Message-ID: <48479B0E.5060403(a)javaperformancetuning.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Hi really enjoyed reading Nick's description of how he ran into trouble and what he did to fix it. My preference to delay premature promotions has always been to increase the size of survivor spaces. Using larger tenuring thresholds hasn't sounded appealing because of the cost of copying. Having GC run less often always seemed like a more desirable goal. And I think this is especially true in young where it is mostly about live object harvesting and having GC run less often gives object a better chance to expire. If these bits are so important I think that limiting MMT 31 seems reasonable at least today. If you have a bit to spare, than 63 seems even safer. Regards, Kirk Pepperdine Jon Masamitsu wrote: > Since higher MTT's are important for some applications > and we really don't know how high is high enough, should > we take John P's suggestion a step further and use the > upper 15 bits of any MTT as the tenuring age (i.e., > setting MTT to 60, 61, 62 or 63 would tenure at > age 60)? We just have to increment the age every > n-th scavenge as appropriate. > > Y Srinivas Ramakrishna wrote On 06/04/08 14:54,: > > >> Hi Nick -- >> >> thanks for sharing that experience and for the nice description! >> Looking at the PrintTenuringDistribiution for your application >> with the old, 5-age bit JVM run with MTT=24 would probably be >> illuminating. By the way, would you expect the objects surviving >> 24 scavenges (in that original configuration) to live for a pretty long time? >> Do you know if your object age distribution has a long thin tail, >> and, if so, where it falls to 0? If it is the case that most applications >> have a long thin tail and that the population in that tail (age > MTT) >> is large, then NeverTenure is probably a bad idea. >> >> As you found, the basic rule is always that if a small transient overload >> causes survivor space overflow, that in turn can cause a much longer-term "ringing effect" >> because of "nepotism" (which begets more nepotism, ....,) the effects of >> which can last much longer than the initial transient that set it off. >> And, yes, NeverTenure will lead to overflow in long-tailed distributions >> unless your survivor spaces are large enough to accomodate the tail; >> which is just another way of saying, if the tail does not fit, you will >> have overflow which will cause nepotism and its ill-effects. >> >> Thanks again for sharing that experience and for the nice >> explanation of your experiments. >> >> -- ramki >> >> >> >> >>> So, as a conclusion: Yes, we are missing the 5th age bit. NeverTenure >>> >>> works very bad for us. MTT=15 with our original eden size is too >>> small, >>> but increasing the eden size allows us to get similar behavior with >>> MTT=15 as with the original configuration (MTT=24 and 5 age bits). >>> >>> I think, Tony's suggestion to limit the configurable MTT to 2^n-1 >>> (with >>> n being the age bits) is a good solution. At least according to my >>> tests, this is much better then activating the "never tenure" policy >>> when the user is not aware of this. >>> >>> I hope some of this may have been helpful for you. I will send some more >>> detailed results and logfiles directly to Tony. >>> >>> Nick. >>> >>> >>> > > > ------------------------------ Message: 3 Date: Thu, 05 Jun 2008 08:39:01 -0700 From: Jon Masamitsu <Jon.Masamitsu(a)Sun.COM> Subject: Re: Proposal: MaxTenuringThreshold and age field size changes To: hotspot-gc-dev(a)openjdk.java.net Message-ID: <48480895.7010502(a)Sun.COM> Content-Type: text/plain; format=flowed; charset=ISO-8859-1 Jon Masamitsu wrote: > Since higher MTT's are important for some applications > and we really don't know how high is high enough, should > we take John P's suggestion a step further and use the > upper 15 bits of any MTT as the tenuring age (i.e. Sorry, guys. That should have been 4 bits. You can stop wondering where that heck I got 15 bits now. > setting MTT to 60, 61, 62 or 63 would tenure at > age 60)? We just have to increment the age every > n-th scavenge as appropriate. > > Y Srinivas Ramakrishna wrote On 06/04/08 14:54,: > >> Hi Nick -- >> >> thanks for sharing that experience and for the nice description! >> Looking at the PrintTenuringDistribiution for your application >> with the old, 5-age bit JVM run with MTT=24 would probably be >> illuminating. By the way, would you expect the objects surviving >> 24 scavenges (in that original configuration) to live for a pretty long time? >> Do you know if your object age distribution has a long thin tail, >> and, if so, where it falls to 0? If it is the case that most applications >> have a long thin tail and that the population in that tail (age > MTT) >> is large, then NeverTenure is probably a bad idea. >> >> As you found, the basic rule is always that if a small transient overload >> causes survivor space overflow, that in turn can cause a much longer-term "ringing effect" >> because of "nepotism" (which begets more nepotism, ....,) the effects of >> which can last much longer than the initial transient that set it off. >> And, yes, NeverTenure will lead to overflow in long-tailed distributions >> unless your survivor spaces are large enough to accomodate the tail; >> which is just another way of saying, if the tail does not fit, you will >> have overflow which will cause nepotism and its ill-effects. >> >> Thanks again for sharing that experience and for the nice >> explanation of your experiments. >> >> -- ramki >> >> >> >>> So, as a conclusion: Yes, we are missing the 5th age bit. NeverTenure >>> >>> works very bad for us. MTT=15 with our original eden size is too >>> small, >>> but increasing the eden size allows us to get similar behavior with >>> MTT=15 as with the original configuration (MTT=24 and 5 age bits). >>> >>> I think, Tony's suggestion to limit the configurable MTT to 2^n-1 >>> (with >>> n being the age bits) is a good solution. At least according to my >>> tests, this is much better then activating the "never tenure" policy >>> when the user is not aware of this. >>> >>> I hope some of this may have been helpful for you. I will send some more >>> detailed results and logfiles directly to Tony. >>> >>> Nick. >>> >>> > ------------------------------ Message: 4 Date: Thu, 05 Jun 2008 17:56:18 +0000 From: jon.masamitsu(a)sun.com Subject: hg: jdk7/hotspot-gc/hotspot: 6629727: assertion in set_trap_state() in methodDataOop.hpp is too strong. To: jdk7-changes(a)openjdk.java.net, hotspot-gc-dev(a)openjdk.java.net Message-ID: <20080605175622.8667328041(a)hg.openjdk.java.net> Changeset: 0b27f3512f9e Author: jmasa Date: 2008-06-04 13:51 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/0b27f3512f9e 6629727: assertion in set_trap_state() in methodDataOop.hpp is too strong. Summary: The assertion can failure due to race conditions. Reviewed-by: never ! src/share/vm/oops/methodDataOop.hpp End of hotspot-gc-dev Digest, Vol 12, Issue 4 *********************************************

2 1

hg: jdk7/hotspot-gc/hotspot: 6629727: assertion in set_trap_state() in methodDataOop.hpp is too strong.
by jon.masamitsu＠sun.com 05 Jun '08

05 Jun '08

Changeset: 0b27f3512f9e Author: jmasa Date: 2008-06-04 13:51 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/0b27f3512f9e 6629727: assertion in set_trap_state() in methodDataOop.hpp is too strong. Summary: The assertion can failure due to race conditions. Reviewed-by: never ! src/share/vm/oops/methodDataOop.hpp

1 0

Proposal: MaxTenuringThreshold and age field size changes
by Tony Printezis 05 Jun '08

05 Jun '08

Hi all, We are considering making some changes to the way the MaxTenuringThreshold parameter is handled and to the size of the age field. We'd appreciate your feedback. *Background.* The -XX:MaxTenuringThreshold=n (or MTT for short) parameter dictates how many times objects will be copied within the young generation (i.e., between the survivor spaces) before they are promoted (tenured) to the old generation. Each object has an "age" field in its header which is incremented every time an object is copied within the young generation. When the age field reaches the value of MTT, the object is promoted to the old generation (I've left out some detail here...). The parameter -XX:+NeverTenure tells the GC never to tenure objects willingly (they will be promoted only when the target survivor space is full). (out of curiosity: does anyone actually use -XX:+NeverTenure?) Originally, in the HotSpot JVM, we had 5 bits per object for the age field (for a max value of 31, so values of MTT would make sense if they were <= 31). A couple of years ago (since 5u6 IIRC), the age field "lost" one bit and it now only has 4 (for a max value of 15). *The Problem.* We have got reports from customers that, when they use a MTT > 15 in the latest JVMs (say, when they use their older settings), the tenuring policy behaves strangely and piles up objects on age 15 (i.e., objects reach age 15 and are not then promoted, but are kept at age 15). This happens because if MTT is set to a value higher than the max age (which is currently 15, as I said earlier), the policy essentially behaves the same as -XX:+NeverTenure. We have got at least two reports that this has a detrimental effect on performance since (a) objects are either retained too long in the young gen, imposing copying overhead, or (b) objects are promoted too early, since objects much older than them, and which they should have got promoted some time ago, filled up the survivor space, forcing younger objects to be promoted instead. We also got some reports from customers that the decrease in age field size has actually created performance issues for them (they get better GC behavior if they keep objects in the young gen longer, so that they can reclaim more of them in the young gen and promote less). *Change Proposal.* We are thinking of applying the following changes: - If MTT is set by the user to a value > max age, we will actually set it to max age. So, a high MTT value will not be equivalent to setting -XX:+NeverTenure (if users really want they never tenure policy, they'd get it with the -XX:+NeverTenure parameter). We believe that this is the more natural interpretation of the MTT parameter (and it will be less confusing to users with "old" settings), even though it will behave differently to the way it currently behaves. - Even though we cannot add an extra bit to the age field in the 32-bit JVM (the space in an object's mark word is at a premium in the 32-bit JVM), we think we can increase the age field size, and as a result the max value of MTT, in the 64-bit JVM (to even higher than 5 bits too). We feel this is a worthwhile change, given that 64-bit JVM users will be able to use MTT values > 15, despite the confusion that it might cause (the max MTT value will be different in the 32-bit and 64-bit JVMs; incidentally, the "compressed oops" JVM will also have the larger age fields too). Let us know what you think, Tony, HS GC Group -- ---------------------------------------------------------------------- | Tony Printezis, Staff Engineer | Sun Microsystems Inc. | | | MS BUR02-311 | | e-mail: tony.printezis(a)sun.com | 35 Network Drive | | office: +1 781 442 0998 (x20998) | Burlington, MA01803-0902, USA | ---------------------------------------------------------------------- e-mail client: Thunderbird (Solaris)

6 14

question from customer
by Randy Crihfield 03 Jun '08

03 Jun '08

Hello Sun experts, I work for ESPN.com and one of our teams here is looking for serious garbage collection expertise. In a "past life" I worked at Sun for 5+ years and one of my former co-workers (Randy Crihfield who's now in the JVM group) suggested I might try hitting up this alias for some help. In a nutshell: * the fantasy games group here at ESPN.com has some pretty serious issues with memory consumption and (specifically) garbage collection. * They are looking for someone intimately familiar with the ins and o= uts of the Java 1.5 / 1.6 garbage collector... a certified GC bad-a** :) * They are more than willing to PAY for this expertise since it's alm= ost certainly not something that can be solved in simple back & forth ema= il thread or 30 minute phone call - except for the Sun JDK, all of our stuff is home grown Java code. * They are somewhat leary of who might show up if they were to just g= o to professional services :) So I thought I might see if your team can or knows of someone who can provide assistance. Please let me know if this is something any of yo= u might be of assistance with. Thanks for your time and feel free to call me at the work # below if that works better for you. -------- Sean Comerford, Software Engineer ESPN.com Site Architecture Group Office: 860.766.6454 Cell: 860.951.6973

4 3