From randyc at east.sun.com Tue Jun 3 14:42:58 2008 From: randyc at east.sun.com (Randy Crihfield) Date: Tue, 3 Jun 2008 10:42:58 -0400 (EDT) Subject: question from customer Message-ID: <200806031442.m53Egw4P001067@east.sun.com> Hello Sun experts, I work for ESPN.com and one of our teams here is looking for serious garbage collection expertise. In a "past life" I worked at Sun for 5+ years and one of my former co-workers (Randy Crihfield who's now in the JVM group) suggested I might try hitting up this alias for some help. In a nutshell: * the fantasy games group here at ESPN.com has some pretty serious issues with memory consumption and (specifically) garbage collection. * They are looking for someone intimately familiar with the ins and o= uts of the Java 1.5 / 1.6 garbage collector... a certified GC bad-a** :) * They are more than willing to PAY for this expertise since it's alm= ost certainly not something that can be solved in simple back & forth ema= il thread or 30 minute phone call - except for the Sun JDK, all of our stuff is home grown Java code. * They are somewhat leary of who might show up if they were to just g= o to professional services :) So I thought I might see if your team can or knows of someone who can provide assistance. Please let me know if this is something any of yo= u might be of assistance with. Thanks for your time and feel free to call me at the work # below if that works better for you. -------- Sean Comerford, Software Engineer ESPN.com Site Architecture Group Office: 860.766.6454 Cell: 860.951.6973 From Jon.Masamitsu at Sun.COM Tue Jun 3 15:16:38 2008 From: Jon.Masamitsu at Sun.COM (Jon Masamitsu) Date: Tue, 03 Jun 2008 08:16:38 -0700 Subject: question from customer In-Reply-To: <200806031442.m53Egw4P001067@east.sun.com> References: <200806031442.m53Egw4P001067@east.sun.com> Message-ID: <48456056.8020605@Sun.COM> Sean, Knowing a little more about the problem might help us direct your inquiry. Is there a gc log (-XX:+PrintGCDetails) that you can share? Jon PS. The GC engineering team is on this alias so you're asking the right people. Depending on the problem, you might get directed to professional services but let's look at the problem first. Randy Crihfield wrote: > Hello Sun experts, > > I work for ESPN.com and one of our teams here is looking for serious > garbage collection expertise. > > In a "past life" I worked at Sun for 5+ years and one of my former > co-workers (Randy Crihfield who's now in the JVM group) suggested I > might try hitting up this alias for some help. > > In a nutshell: > > * the fantasy games group here at ESPN.com has some pretty serious > issues with memory consumption and (specifically) garbage collection. > > * They are looking for someone intimately familiar with the ins and o= > uts > of the Java 1.5 / 1.6 garbage collector... a certified GC bad-a** :) > > * They are more than willing to PAY for this expertise since it's alm= > ost > certainly not something that can be solved in simple back & forth ema= > il > thread or 30 minute phone call - except for the Sun JDK, all of our > stuff is home grown Java code. > > * They are somewhat leary of who might show up if they were to just g= > o > to professional services :) > > So I thought I might see if your team can or knows of someone who can > provide assistance. Please let me know if this is something any of yo= > u > might be of assistance with. > > Thanks for your time and feel free to call me at the work # below if > that works better for you. > > -------- > Sean Comerford, Software Engineer > ESPN.com Site Architecture Group > Office: 860.766.6454 Cell: 860.951.6973 From Sean.Comerford at espn3.com Tue Jun 3 15:26:23 2008 From: Sean.Comerford at espn3.com (Comerford, Sean) Date: Tue, 3 Jun 2008 15:26:23 +0000 Subject: question from customer In-Reply-To: <48456056.8020605@Sun.COM> References: <200806031442.m53Egw4P001067@east.sun.com> <48456056.8020605@Sun.COM> Message-ID: <70B8C67AEAD69D48AA873E99C275E9500991A473@SM-NYNY-VXMB01A.nena.wdpr.disney.com> Hi Jon - thanks for the quick response. My understanding is the issues are more of extreme scaling / # of objects being GC'd than any specific problem that can be pinpointed in a log. But Mike Polikoff (whom I've copied on this email) is in charge of the fantasy engineering group and can explain far better that I the issues they are looking to get help with. So I'll defer to him at this point. -------- Sean Comerford, Software Engineer ESPN.com Site Architecture Group Office: 860.766.6454 Cell: 860.951.6973 > -----Original Message----- > From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM] > Sent: Tuesday, June 03, 2008 11:17 AM > To: Randy Crihfield > Cc: hotspot-gc-dev at openjdk.java.net; Comerford, Sean > Subject: Re: question from customer > > Sean, > > Knowing a little more about the problem might help us direct > your inquiry. Is there a gc log (-XX:+PrintGCDetails) that > you can share? > > Jon > > PS. The GC engineering team is on this alias so you're > asking the right people. Depending on the problem, you might > get directed to professional services but let's look at the > problem first. > > Randy Crihfield wrote: > > Hello Sun experts, > > > > I work for ESPN.com and one of our teams here is looking > for serious > > garbage collection expertise. > > > > In a "past life" I worked at Sun for 5+ years and one of my former > > co-workers (Randy Crihfield who's now in the JVM group) suggested I > > might try hitting up this alias for some help. > > > > In a nutshell: > > > > * the fantasy games group here at ESPN.com has some pretty serious > > issues with memory consumption and (specifically) garbage > collection. > > > > * They are looking for someone intimately familiar with the > ins and o= > > uts of the Java 1.5 / 1.6 garbage collector... a certified > GC bad-a** > > :) > > > > * They are more than willing to PAY for this expertise > since it's alm= > > ost certainly not something that can be solved in simple > back & forth > > ema= il thread or 30 minute phone call - except for the Sun > JDK, all > > of our stuff is home grown Java code. > > > > * They are somewhat leary of who might show up if they were > to just g= > > o to professional services :) > > > > So I thought I might see if your team can or knows of > someone who can > > provide assistance. Please let me know if this is something > any of yo= > > u might be of assistance with. > > > > Thanks for your time and feel free to call me at the work # > below if > > that works better for you. > > > > -------- > > Sean Comerford, Software Engineer > > ESPN.com Site Architecture Group > > Office: 860.766.6454 Cell: 860.951.6973 > From fw at deneb.enyo.de Tue Jun 3 19:44:01 2008 From: fw at deneb.enyo.de (Florian Weimer) Date: Tue, 03 Jun 2008 21:44:01 +0200 Subject: question from customer In-Reply-To: <70B8C67AEAD69D48AA873E99C275E9500991A473@SM-NYNY-VXMB01A.nena.wdpr.disney.com> (Sean Comerford's message of "Tue, 3 Jun 2008 15:26:23 +0000") References: <200806031442.m53Egw4P001067@east.sun.com> <48456056.8020605@Sun.COM> <70B8C67AEAD69D48AA873E99C275E9500991A473@SM-NYNY-VXMB01A.nena.wdpr.disney.com> Message-ID: <87prqyjoce.fsf@mid.deneb.enyo.de> * Sean Comerford: > But Mike Polikoff (whom I've copied on this email) is in charge of the > fantasy engineering group and can explain far better that I the issues > they are looking to get help with. So I'll defer to him at this point. Uhm, folks, are you aware that this is a public mailing list? From email at nmichael.de Wed Jun 4 20:17:03 2008 From: email at nmichael.de (Nicolas Michael) Date: Wed, 04 Jun 2008 22:17:03 +0200 Subject: Proposal: MaxTenuringThreshold and age field size changes In-Reply-To: <483F08D6.9060803@sun.com> References: <483DD7EC.7050500@sun.com> <483F08D6.9060803@sun.com> Message-ID: <4846F83F.9050808@nmichael.de> Hi, we are one of the customers Tony was referring to that are "missing" the lost age bit. During the last days, I ran some tests in different configurations (that's why I reply only now). Tony asked me to post the results of my analysis, so here it is! Probably some of it is interesting for you... We use ParNew/CMS as garbage collectors. Our goal is to collect all medium-lived objects for most scenarios before they tenure into the old gen. Therefore, we've been using a MaxTenuringThreshold (MTT) of 24, which proved to be a good trade-off between collecting all medium-lived objects in the young heap for most scenarios, while at the other hand not copy objects of some other scenarios (that live a little longer, i.e. more than 24 cycles) too often. Unfortunately, with the lost age bit, configuring MTT=24 currently activates the NeverTenure policy (which we didn't notice when updating to the latest Java 5 releases). I now ran test for different scenarios (using the latest Java 6 build) to figure out how to best configure MTT for our application. It turned out, that MTT=24 ("never tenure") is a bad idea for us. For scenarios with medium-lived objects that survive more than 24 gc cycles, this increases the copying overhead significantly, leading to much longer young gc times (it's better to tenure them into old after a while instead of copying them too often). Also (and this was most surprising to me!), even for scenarios where all medium-lived objects die after 3-4 gc cycles, these objects are sometimes *not* collected, piling up at age 15. Both lead to the situation that the entire survivor space fills up to 100%. Sometimes, 90% of this is garbage, that is still being copied at each gc cycle. Also, since the survivor spaces are full, other objects that even may be younger than those in the survivor spaces, need to tenure into old because there's no space for them in the survivor spaces. While MTT=24 used to be "good" for JVMs with 5 age bits, now with only 4 bits, this setting turned out to be rather bad. It not only increased young gc times significantly, but also caused objects to prematurely tenure into old (causing unecessary CMS cycles) and also increased the CPU usage (due to more gc work and probably also because of L1/L2 cache effects when copying more objects than necessary). With MTT=15, we had much shorter young gc times, and we never filled up the survivor spaces. However, MTT=15 is too short for many scenarios: Now we have for many scenarios objects tenuring into old which didn't tenure before with MTT=24 and 5 age bits. Doubling the eded size solved this problem. Since the gc frequency this way decreased by factor 2, 15 gc cycles now last as many seconds as (with half the eden) 30 cycles would have lasted before. Since the gc time (almost) only depends on the number of surviving objects and not the eden size, the effect from doubling the eden size on the gc times was negligible: For scenarios where no objects were tenuring into old with the small eden, there was no change in gc times at all (after all, the same amount of objects were surviving a gc). For scenarios where with the small eden, some objects were tenuring which now, with the large eden, don't tenure any more, there were more objects surviving a gc in the young heap, so the young gc times increased a little. However, since we avoid CMS cycles with the larger eden by keeping those objects in the young gen, this pays off again. Generally, young gc times with MTT=24 ("never tenure") are around factor 2-4 of those with MTT=15, depending on the scenario (130ms vs 25ms, 180ms vs 80ms). This is mostly because much more objects survive a gc cycle (including many dead objects) with MTT=24. So, as a conclusion: Yes, we are missing the 5th age bit. NeverTenure works very bad for us. MTT=15 with our original eden size is too small, but increasing the eden size allows us to get similar behavior with MTT=15 as with the original configuration (MTT=24 and 5 age bits). I think, Tony's suggestion to limit the configurable MTT to 2^n-1 (with n being the age bits) is a good solution. At least according to my tests, this is much better then activating the "never tenure" policy when the user is not aware of this. I hope some of this may have been helpful for you. I will send some more detailed results and logfiles directly to Tony. Nick. Tony Printezis schrieb: > Ramki, > > Y Srinivas Ramakrishna wrote: >> In the 64 bit JVM, I would rather vote for reserving the available >> bits for >> possible future use rather than use them for age. > I think, in the 64-bit JVM, we have 26 bits available on the mark word. > I think we should be able to use at least one of them to expand the age > field. In fact, I was going to vote that we use two and expand the age > field to 6 bits for a max value of 63. >> It would seem that 15 >> would almost always be sufficient, and if not there is always >> NeverTenure :-) >> [But I guess your customers are evidence that 15 is not enough in some >> cases; > I know of at least two customers who see better GC behavior with a MTT > > 15. >> where do we draw the line? I guess that we do not really know, and that >> you are suggesting we give the users as many age bits as we >> "reasonably can" for >> a specific platform/implementation.] >> >> On the other hand, if people feel more age bits are needed for the 64-bit >> case, I would like us to agree that the actual number available in a >> given release would be bounded below by 4, but would otherwise be >> free to change from one release to another, along with the semantics >> for ceiling the MTT as suggested in your note below. [For example: "Hey, >> where did my age bit go? I was specifying MTT=256 in JDK 19 >> and was happy with the performance, and now the JVM's ceiling the MTT >> at 128 since JDK 20."] >> > I think this is definitely reasonable. And, in fact, it makes using a > 2n-1 ceiling for MTT, as we discussed, even more important. :-) >> Just out of curiosity, and to do a very rough survey, should one do a >> quick >> poll of hotspot-gc-use to see how many people on the >> list missed the age-bit that vanished across the 5u5->5u6 transition >> and wished the bit were back? >> > I can do that. >> Anyway, ... my $0.02 :-) >> > Tony >> ----- Original Message ----- >> From: Tony Printezis >> Date: Wednesday, May 28, 2008 3:15 pm >> Subject: Proposal: MaxTenuringThreshold and age field size changes >> To: "hotspot-gc-dev at openjdk.java.net" >> >> >> >>> Hi all, >>> >>> We are considering making some changes to the way the >>> MaxTenuringThreshold parameter is handled and to the size of the age >>> field. We'd appreciate your feedback. >>> >>> *Background.* The -XX:MaxTenuringThreshold=n (or MTT for short) >>> parameter dictates how many times objects will be copied within the >>> young generation (i.e., between the survivor spaces) before they are >>> promoted (tenured) to the old generation. Each object has an "age" >>> field in its header which is incremented every time an object is >>> copied within the young generation. When the age field reaches the >>> value of MTT, the >>> object is promoted to the old generation (I've left out some detail >>> here...). The parameter -XX:+NeverTenure tells the GC never to tenure >>> objects willingly (they will be promoted only when the target survivor >>> space is full). >>> >>> (out of curiosity: does anyone actually use -XX:+NeverTenure?) >>> >>> Originally, in the HotSpot JVM, we had 5 bits per object for the age >>> field (for a max value of 31, so values of MTT would make sense if >>> they were <= 31). A couple of years ago (since 5u6 IIRC), the age >>> field "lost" one bit and it now only has 4 (for a max value of 15). >>> >>> *The Problem.* We have got reports from customers that, when they use >>> a MTT > 15 in the latest JVMs (say, when they use their older settings), >>> the tenuring policy behaves strangely and piles up objects on age 15 >>> (i.e., objects reach age 15 and are not then promoted, but are kept at >>> age 15). This happens because if MTT is set to a value higher than the >>> max age (which is currently 15, as I said earlier), the policy >>> essentially behaves the same as -XX:+NeverTenure. We have got at least >>> two reports that this has a detrimental effect on performance since >>> (a) objects are either retained too long in the young gen, imposing >>> copying overhead, or (b) objects are promoted too early, since >>> objects much older than them, and which they should have got promoted >>> some time ago, filled up the survivor space, forcing younger objects >>> to be promoted instead. We also got some reports from customers that >>> the decrease in >>> age field size has actually created performance issues for them (they >>> get better GC behavior if they keep objects in the young gen longer, >>> so that they can reclaim more of them in the young gen and promote >>> less). >>> >>> *Change Proposal.* We are thinking of applying the following changes: >>> >>> - If MTT is set by the user to a value > max age, we will actually set >>> it to max age. So, a high MTT value will not be equivalent to setting >>> -XX:+NeverTenure (if users really want they never tenure policy, >>> they'd get it with the -XX:+NeverTenure parameter). We believe that >>> this is the more natural interpretation of the MTT parameter (and it >>> will be less >>> confusing to users with "old" settings), even though it will behave >>> differently to the way it currently behaves. >>> >>> - Even though we cannot add an extra bit to the age field in the >>> 32-bit JVM (the space in an object's mark word is at a premium in the >>> 32-bit >>> JVM), we think we can increase the age field size, and as a result the >>> max value of MTT, in the 64-bit JVM (to even higher than 5 bits too). >>> We feel this is a worthwhile change, given that 64-bit JVM users will be >>> able to use MTT values > 15, despite the confusion that it might cause >>> (the max MTT value will be different in the 32-bit and 64-bit JVMs; >>> incidentally, the "compressed oops" JVM will also have the larger age >>> fields too). >>> >>> Let us know what you think, >>> >>> Tony, HS GC Group >>> >>> -- >>> ---------------------------------------------------------------------- >>> | Tony Printezis, Staff Engineer | Sun Microsystems Inc. | >>> | | MS BUR02-311 | >>> | e-mail: tony.printezis at sun.com | 35 Network Drive | >>> | office: +1 781 442 0998 (x20998) | Burlington, MA01803-0902, USA | >>> ---------------------------------------------------------------------- >>> e-mail client: Thunderbird (Solaris) >>> >>> >>> > From Y.S.Ramakrishna at Sun.COM Wed Jun 4 21:54:34 2008 From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna) Date: Wed, 04 Jun 2008 14:54:34 -0700 Subject: Proposal: MaxTenuringThreshold and age field size changes In-Reply-To: <4846F83F.9050808@nmichael.de> References: <483DD7EC.7050500@sun.com> <483F08D6.9060803@sun.com> <4846F83F.9050808@nmichael.de> Message-ID: Hi Nick -- thanks for sharing that experience and for the nice description! Looking at the PrintTenuringDistribiution for your application with the old, 5-age bit JVM run with MTT=24 would probably be illuminating. By the way, would you expect the objects surviving 24 scavenges (in that original configuration) to live for a pretty long time? Do you know if your object age distribution has a long thin tail, and, if so, where it falls to 0? If it is the case that most applications have a long thin tail and that the population in that tail (age > MTT) is large, then NeverTenure is probably a bad idea. As you found, the basic rule is always that if a small transient overload causes survivor space overflow, that in turn can cause a much longer-term "ringing effect" because of "nepotism" (which begets more nepotism, ....,) the effects of which can last much longer than the initial transient that set it off. And, yes, NeverTenure will lead to overflow in long-tailed distributions unless your survivor spaces are large enough to accomodate the tail; which is just another way of saying, if the tail does not fit, you will have overflow which will cause nepotism and its ill-effects. Thanks again for sharing that experience and for the nice explanation of your experiments. -- ramki > So, as a conclusion: Yes, we are missing the 5th age bit. NeverTenure > > works very bad for us. MTT=15 with our original eden size is too > small, > but increasing the eden size allows us to get similar behavior with > MTT=15 as with the original configuration (MTT=24 and 5 age bits). > > I think, Tony's suggestion to limit the configurable MTT to 2^n-1 > (with > n being the age bits) is a good solution. At least according to my > tests, this is much better then activating the "never tenure" policy > when the user is not aware of this. > > I hope some of this may have been helpful for you. I will send some more > detailed results and logfiles directly to Tony. > > Nick. From Jon.Masamitsu at Sun.COM Thu Jun 5 00:59:31 2008 From: Jon.Masamitsu at Sun.COM (Jon Masamitsu) Date: Wed, 04 Jun 2008 17:59:31 -0700 Subject: Proposal: MaxTenuringThreshold and age field size changes In-Reply-To: References: <483DD7EC.7050500@sun.com> <483F08D6.9060803@sun.com> <4846F83F.9050808@nmichael.de> Message-ID: <48473A73.5050705@sun.com> Since higher MTT's are important for some applications and we really don't know how high is high enough, should we take John P's suggestion a step further and use the upper 15 bits of any MTT as the tenuring age (i.e., setting MTT to 60, 61, 62 or 63 would tenure at age 60)? We just have to increment the age every n-th scavenge as appropriate. Y Srinivas Ramakrishna wrote On 06/04/08 14:54,: >Hi Nick -- > >thanks for sharing that experience and for the nice description! >Looking at the PrintTenuringDistribiution for your application >with the old, 5-age bit JVM run with MTT=24 would probably be >illuminating. By the way, would you expect the objects surviving >24 scavenges (in that original configuration) to live for a pretty long time? >Do you know if your object age distribution has a long thin tail, >and, if so, where it falls to 0? If it is the case that most applications >have a long thin tail and that the population in that tail (age > MTT) >is large, then NeverTenure is probably a bad idea. > >As you found, the basic rule is always that if a small transient overload >causes survivor space overflow, that in turn can cause a much longer-term "ringing effect" >because of "nepotism" (which begets more nepotism, ....,) the effects of >which can last much longer than the initial transient that set it off. >And, yes, NeverTenure will lead to overflow in long-tailed distributions >unless your survivor spaces are large enough to accomodate the tail; >which is just another way of saying, if the tail does not fit, you will >have overflow which will cause nepotism and its ill-effects. > >Thanks again for sharing that experience and for the nice >explanation of your experiments. > >-- ramki > > > >>So, as a conclusion: Yes, we are missing the 5th age bit. NeverTenure >> >>works very bad for us. MTT=15 with our original eden size is too >>small, >>but increasing the eden size allows us to get similar behavior with >>MTT=15 as with the original configuration (MTT=24 and 5 age bits). >> >>I think, Tony's suggestion to limit the configurable MTT to 2^n-1 >>(with >>n being the age bits) is a good solution. At least according to my >>tests, this is much better then activating the "never tenure" policy >>when the user is not aware of this. >> >>I hope some of this may have been helpful for you. I will send some more >>detailed results and logfiles directly to Tony. >> >>Nick. >> >> From email at nmichael.de Thu Jun 5 06:26:57 2008 From: email at nmichael.de (Nicolas Michael) Date: Thu, 05 Jun 2008 08:26:57 +0200 Subject: Proposal: MaxTenuringThreshold and age field size changes In-Reply-To: References: <483DD7EC.7050500@sun.com> <483F08D6.9060803@sun.com> <4846F83F.9050808@nmichael.de> Message-ID: <48478731.4020503@nmichael.de> Hi Ramki, please see inline. Y Srinivas Ramakrishna schrieb: > Hi Nick -- > > thanks for sharing that experience and for the nice description! > Looking at the PrintTenuringDistribiution for your application > with the old, 5-age bit JVM run with MTT=24 would probably be > illuminating. By the way, would you expect the objects surviving > 24 scavenges (in that original configuration) to live for a pretty long time? > Do you know if your object age distribution has a long thin tail, > and, if so, where it falls to 0? If it is the case that most applications > have a long thin tail and that the population in that tail (age > MTT) > is large, then NeverTenure is probably a bad idea. Basically, all our scenarios work like this: Upon an incoming request, we create some hundred kilobytes of objects. Most of those objects die after this request has been processed, which usually takes something between 10ms and 100ms. A few kilobytes of objects remain alive in the Java heap until the session is terminated, which is signaled through a further request. The session length may vary between few seconds, some minutes or even hours, depending on the scenario. There are no session-related objects dying "in between". So, assume a fix session duration of 60 seconds for all requests, this means that all objects that are not just temporary and survive the first gc cycle will only die after 60 seconds. With a gc interval of, let's say, 6 seconds, this would be 10 cycles. MTT >= 10 would be sufficient to collect all those objects in the young gen. As you can see, in this example there is no "thin tail": We have the same amount of objects at each age 1-10. Let's assume a session duration of 120 seconds. As you can easily see, with the same gc pattern, all those objects would survive 20 gc cycles. With a max MTT of 15, they would all tenure into old after 15 cycles. It's quite obvious that for this scenario, it would be better to set MTT=1, since we would avoid copying those objects 14 times before all of them tenure anyway. Another way would be to use "never tenure" for 120 second sessions. This would allow to keep them in the young gen for 20 cycles, provided the survivor spaces are large enough. But imagine a third scenario with a session duration of 10 minutes. Such a scenario would definately overflow the survivor spaces with the "never tenure" policy. We need a set of JVM settings that fits all scenarios. Assuming that scenarios with mixed session durations run simultaneously, our aim is to collect the objects for most scenarios in the young gen (before any of them tenure), and accept that the objects of some scenarios (after a reasonable amount of copying) tenure into old. The question is where we draw that line... With our original gc intervals, 24 cycles seemed to be a good trade-off. Now, that we can only set MTT<=15, MTT=15 with a streched gc interval (by enlarging the eden) achieves the same. > As you found, the basic rule is always that if a small transient overload > causes survivor space overflow, that in turn can cause a much longer-term "ringing effect" > because of "nepotism" (which begets more nepotism, ....,) the effects of > which can last much longer than the initial transient that set it off. > And, yes, NeverTenure will lead to overflow in long-tailed distributions > unless your survivor spaces are large enough to accomodate the tail; > which is just another way of saying, if the tail does not fit, you will > have overflow which will cause nepotism and its ill-effects. Unfortunately, not only long-tailed distributions lead to survivor space overflow! This is what I meant with my sentence "most surprisingly..." in my previous mail: I've run a scenario where all object died after 2 gc cycles. With MTT=15, they were nicely collected after 2 cycles and the survivor spaces were only something like 10% full. The *same* scenario with MTT=24 ("never tenure") filled up the survivor space to 100%, even causing tenuring of (live or dead, I don't know) objects into the old gen. It's obvious that 90% of the objects filling up the survivor spaces must have been dead already, just gc didn't collect them! This doesn't happen always, but often (so it is reproducable). To make it more clear: The same scenario with the same configuration (MTT=24) doesn't necessarily fill up the survivor spaces to 100%. I've also had runs where it only filled 20% of the survivor spaces. That's still factor 2 of MTT=15 (which still means, there's 50% garbage being copied around), but not as worse as filling them up to 100%. And it get's even better: During a 30 minute run, I've even seen a change in gc behavior (without any change in the load): For the first 20 minutes, the target survivor space was always 100% full after a collection. Then, all of a sudden, from the next collection on, it was only 20% full for the remaining 10 minutes. Tony has an explanation for this (Tony? You can probably explain this better than me?). This is one of the main reasons why "never tenure" works very poor for us. Nick. > > Thanks again for sharing that experience and for the nice > explanation of your experiments. > > -- ramki > >> So, as a conclusion: Yes, we are missing the 5th age bit. NeverTenure >> >> works very bad for us. MTT=15 with our original eden size is too >> small, >> but increasing the eden size allows us to get similar behavior with >> MTT=15 as with the original configuration (MTT=24 and 5 age bits). >> >> I think, Tony's suggestion to limit the configurable MTT to 2^n-1 >> (with >> n being the age bits) is a good solution. At least according to my >> tests, this is much better then activating the "never tenure" policy >> when the user is not aware of this. >> >> I hope some of this may have been helpful for you. I will send some more >> detailed results and logfiles directly to Tony. >> >> Nick. > From Y.S.Ramakrishna at Sun.COM Thu Jun 5 08:03:11 2008 From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna) Date: Thu, 05 Jun 2008 01:03:11 -0700 Subject: Proposal: MaxTenuringThreshold and age field size changes In-Reply-To: <48478731.4020503@nmichael.de> References: <483DD7EC.7050500@sun.com> <483F08D6.9060803@sun.com> <4846F83F.9050808@nmichael.de> <48478731.4020503@nmichael.de> Message-ID: Hi Nick -- You make a good point. I think there are scenarios in which the use of PLAB's in the survivor spaces can cause survivor space overflow, even when the total space in the survivor space should have otherwise been enough. That then sets in motion a series of "nepotistical" cycles, which are caused when young objects Z that are prematurely promoted die in the old generation while holding references to now dead objects Z' in the young generation. Call these objects "zombies" because they are dead but not recognized as such. Z keep Z' alive, because a scavenge does not know that Z is dead, because it considers all references from the old gen as roots. Worse, if Z' has references to Z and Z' stays in the young generation forever (which they can under the circumstances you describe) then Z will not be recognized as dead by a CMS collection (which currently treats all objects in the young generation as roots). This is a well-understood problem when spaces are collected independently in this manner. (The workaround is to have the CMS collector not treat the young generation as a source of roots but rather to mark through the young gen objects starting from roots.) Of course when MTT=15, then eventually every such Z' will be forced to promote to the old gen and the garbage cycle will all (hopefully) move into the old gen and thence will be reclaimed by the old gen CMS collection. This would explain the behaviour difference you saw. Now to come to the first point I made above: the use of multiple scavenger threads and their use of PLABs can sometimes cause this kind of overflow to happen (especially if there is the occasional large object). You might want to switch off the use of survivor space PLAB's (or fix them at a vey small modest value) or just use a single-threaded scavenger and see that this kind of behaviour might reduce because overflow becomes much less likely. I think there is an open bug to tune the adaptive PLAB sizing code to eliminate this kind of pathological behaviour, but we have not had the opportunity to get to that bug. If you have PrintGCDetails logs, they would probably show the premature promotion happening. PrintTenuringDistribution would not show any objects of age greater than 2 initially, yet some objects would be seen to be promoted, and then by virtue of the cross-generational references from a promoted zombie, we would artificially expand the lifetime of objects in the young generation (creating zombies Z') and so on. I believe it was partially this kind of behaviour on the part of generational scavenger implementations that caused some people in the past to start advocating the clearing of all references in objects that they knew they would be dropping references to (which of course we all know is difficult to do correctly and fraught with all kinds of problems and errors). -- ramki From kirk.pepperdine at gmail.com Thu Jun 5 07:51:42 2008 From: kirk.pepperdine at gmail.com (kirk) Date: Thu, 05 Jun 2008 09:51:42 +0200 Subject: Proposal: MaxTenuringThreshold and age field size changes In-Reply-To: <48473A73.5050705@sun.com> References: <483DD7EC.7050500@sun.com> <483F08D6.9060803@sun.com> <4846F83F.9050808@nmichael.de> <48473A73.5050705@sun.com> Message-ID: <48479B0E.5060403@javaperformancetuning.com> Hi really enjoyed reading Nick's description of how he ran into trouble and what he did to fix it. My preference to delay premature promotions has always been to increase the size of survivor spaces. Using larger tenuring thresholds hasn't sounded appealing because of the cost of copying. Having GC run less often always seemed like a more desirable goal. And I think this is especially true in young where it is mostly about live object harvesting and having GC run less often gives object a better chance to expire. If these bits are so important I think that limiting MMT 31 seems reasonable at least today. If you have a bit to spare, than 63 seems even safer. Regards, Kirk Pepperdine Jon Masamitsu wrote: > Since higher MTT's are important for some applications > and we really don't know how high is high enough, should > we take John P's suggestion a step further and use the > upper 15 bits of any MTT as the tenuring age (i.e., > setting MTT to 60, 61, 62 or 63 would tenure at > age 60)? We just have to increment the age every > n-th scavenge as appropriate. > > Y Srinivas Ramakrishna wrote On 06/04/08 14:54,: > > >> Hi Nick -- >> >> thanks for sharing that experience and for the nice description! >> Looking at the PrintTenuringDistribiution for your application >> with the old, 5-age bit JVM run with MTT=24 would probably be >> illuminating. By the way, would you expect the objects surviving >> 24 scavenges (in that original configuration) to live for a pretty long time? >> Do you know if your object age distribution has a long thin tail, >> and, if so, where it falls to 0? If it is the case that most applications >> have a long thin tail and that the population in that tail (age > MTT) >> is large, then NeverTenure is probably a bad idea. >> >> As you found, the basic rule is always that if a small transient overload >> causes survivor space overflow, that in turn can cause a much longer-term "ringing effect" >> because of "nepotism" (which begets more nepotism, ....,) the effects of >> which can last much longer than the initial transient that set it off. >> And, yes, NeverTenure will lead to overflow in long-tailed distributions >> unless your survivor spaces are large enough to accomodate the tail; >> which is just another way of saying, if the tail does not fit, you will >> have overflow which will cause nepotism and its ill-effects. >> >> Thanks again for sharing that experience and for the nice >> explanation of your experiments. >> >> -- ramki >> >> >> >> >>> So, as a conclusion: Yes, we are missing the 5th age bit. NeverTenure >>> >>> works very bad for us. MTT=15 with our original eden size is too >>> small, >>> but increasing the eden size allows us to get similar behavior with >>> MTT=15 as with the original configuration (MTT=24 and 5 age bits). >>> >>> I think, Tony's suggestion to limit the configurable MTT to 2^n-1 >>> (with >>> n being the age bits) is a good solution. At least according to my >>> tests, this is much better then activating the "never tenure" policy >>> when the user is not aware of this. >>> >>> I hope some of this may have been helpful for you. I will send some more >>> detailed results and logfiles directly to Tony. >>> >>> Nick. >>> >>> >>> > > > From Jon.Masamitsu at Sun.COM Thu Jun 5 15:39:01 2008 From: Jon.Masamitsu at Sun.COM (Jon Masamitsu) Date: Thu, 05 Jun 2008 08:39:01 -0700 Subject: Proposal: MaxTenuringThreshold and age field size changes In-Reply-To: <48473A73.5050705@sun.com> References: <483DD7EC.7050500@sun.com> <483F08D6.9060803@sun.com> <4846F83F.9050808@nmichael.de> <48473A73.5050705@sun.com> Message-ID: <48480895.7010502@Sun.COM> Jon Masamitsu wrote: > Since higher MTT's are important for some applications > and we really don't know how high is high enough, should > we take John P's suggestion a step further and use the > upper 15 bits of any MTT as the tenuring age (i.e. Sorry, guys. That should have been 4 bits. You can stop wondering where that heck I got 15 bits now. > setting MTT to 60, 61, 62 or 63 would tenure at > age 60)? We just have to increment the age every > n-th scavenge as appropriate. > > Y Srinivas Ramakrishna wrote On 06/04/08 14:54,: > >> Hi Nick -- >> >> thanks for sharing that experience and for the nice description! >> Looking at the PrintTenuringDistribiution for your application >> with the old, 5-age bit JVM run with MTT=24 would probably be >> illuminating. By the way, would you expect the objects surviving >> 24 scavenges (in that original configuration) to live for a pretty long time? >> Do you know if your object age distribution has a long thin tail, >> and, if so, where it falls to 0? If it is the case that most applications >> have a long thin tail and that the population in that tail (age > MTT) >> is large, then NeverTenure is probably a bad idea. >> >> As you found, the basic rule is always that if a small transient overload >> causes survivor space overflow, that in turn can cause a much longer-term "ringing effect" >> because of "nepotism" (which begets more nepotism, ....,) the effects of >> which can last much longer than the initial transient that set it off. >> And, yes, NeverTenure will lead to overflow in long-tailed distributions >> unless your survivor spaces are large enough to accomodate the tail; >> which is just another way of saying, if the tail does not fit, you will >> have overflow which will cause nepotism and its ill-effects. >> >> Thanks again for sharing that experience and for the nice >> explanation of your experiments. >> >> -- ramki >> >> >> >>> So, as a conclusion: Yes, we are missing the 5th age bit. NeverTenure >>> >>> works very bad for us. MTT=15 with our original eden size is too >>> small, >>> but increasing the eden size allows us to get similar behavior with >>> MTT=15 as with the original configuration (MTT=24 and 5 age bits). >>> >>> I think, Tony's suggestion to limit the configurable MTT to 2^n-1 >>> (with >>> n being the age bits) is a good solution. At least according to my >>> tests, this is much better then activating the "never tenure" policy >>> when the user is not aware of this. >>> >>> I hope some of this may have been helpful for you. I will send some more >>> detailed results and logfiles directly to Tony. >>> >>> Nick. >>> >>> > From jon.masamitsu at sun.com Thu Jun 5 17:56:18 2008 From: jon.masamitsu at sun.com (jon.masamitsu at sun.com) Date: Thu, 05 Jun 2008 17:56:18 +0000 Subject: hg: jdk7/hotspot-gc/hotspot: 6629727: assertion in set_trap_state() in methodDataOop.hpp is too strong. Message-ID: <20080605175622.8667328041@hg.openjdk.java.net> Changeset: 0b27f3512f9e Author: jmasa Date: 2008-06-04 13:51 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/0b27f3512f9e 6629727: assertion in set_trap_state() in methodDataOop.hpp is too strong. Summary: The assertion can failure due to race conditions. Reviewed-by: never ! src/share/vm/oops/methodDataOop.hpp From Ciciora at cboe.com Fri Jun 6 19:46:17 2008 From: Ciciora at cboe.com (Ciciora, Paul) Date: Fri, 6 Jun 2008 14:46:17 -0500 Subject: Proposal: MaxTenuringThreshold and age field size changes References: Message-ID: <8CDFE9CE1B9A344E9AA47C27B7C58DD8314300@MSMAIL.cboent.cboe.com> I just came across a neglected process here that was last tuned so long ago it was still using the never tenure approach. So yesterday I was down stairs retuning. My goal was to reduce object promotions, minor collection frequency and reduce collection times. So here is what I did - This is before JVM_ARGS="${JVM_ARGS} -XX:NewSize=100m" JVM_ARGS="${JVM_ARGS} -XX:MaxNewSize=100m" JVM_ARGS="${JVM_ARGS} -XX:SurvivorRatio=128" JVM_ARGS="${JVM_ARGS} -XX:MaxTenuringThreshold=0" JVM_ARGS="${JVM_ARGS} -XX:TargetSurvivorRatio=100" JVM_ARGS="${JVM_ARGS} -XX:CMSInitiatingOccupancyFraction=40" This is first guess based on what I thought would be close JVM_ARGS="${JVM_ARGS} -XX:NewSize=200m" JVM_ARGS="${JVM_ARGS} -XX:MaxNewSize=200m" JVM_ARGS="${JVM_ARGS} -XX:SurvivorRatio=10" JVM_ARGS="${JVM_ARGS} -XX:MaxTenuringThreshold=15" JVM_ARGS="${JVM_ARGS} -XX:-CMSPermGenPrecleaningEnabled" JVM_ARGS="${JVM_ARGS} -XX:TargetSurvivorRatio=50" JVM_ARGS="${JVM_ARGS} -XX:CMSInitiatingOccupancyFraction=60" The test ran really well. I achieved all three goals right off the bat. So the only thing to do was adjust the tenuring to minimize the copies and still minimize promotions as well. The data indicated 4 slots would be optimal. After re-running my numbers did not improve. In fact the collection time increased slightly. So I upped the slots to 6 to maximize object death at the expense of an extra 2 copies. The results were significantly better. The lesson here is promotions are much more expensive than copies. I thought I'd share that. -----Original Message----- From: hotspot-gc-dev-bounces at openjdk.java.net [mailto:hotspot-gc-dev-bounces at openjdk.java.net] On Behalf Of hotspot-gc-dev-request at openjdk.java.net Sent: Thursday, June 05, 2008 2:00 PM To: hotspot-gc-dev at openjdk.java.net Subject: hotspot-gc-dev Digest, Vol 12, Issue 4 Send hotspot-gc-dev mailing list submissions to hotspot-gc-dev at openjdk.java.net To subscribe or unsubscribe via the World Wide Web, visit http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-dev or, via email, send a message with subject or body 'help' to hotspot-gc-dev-request at openjdk.java.net You can reach the person managing the list at hotspot-gc-dev-owner at openjdk.java.net When replying, please edit your Subject line so it is more specific than "Re: Contents of hotspot-gc-dev digest..." Today's Topics: 1. Re: Proposal: MaxTenuringThreshold and age field size changes (Y Srinivas Ramakrishna) 2. Re: Proposal: MaxTenuringThreshold and age field size changes (kirk) 3. Re: Proposal: MaxTenuringThreshold and age field size changes (Jon Masamitsu) 4. hg: jdk7/hotspot-gc/hotspot: 6629727: assertion in set_trap_state() in methodDataOop.hpp is too strong. (jon.masamitsu at sun.com) ---------------------------------------------------------------------- Message: 1 Date: Thu, 05 Jun 2008 01:03:11 -0700 From: Y Srinivas Ramakrishna Subject: Re: Proposal: MaxTenuringThreshold and age field size changes To: Nicolas Michael Cc: Tony Printezis , hotspot-gc-dev at openjdk.java.net Message-ID: Content-Type: text/plain; charset=us-ascii Hi Nick -- You make a good point. I think there are scenarios in which the use of PLAB's in the survivor spaces can cause survivor space overflow, even when the total space in the survivor space should have otherwise been enough. That then sets in motion a series of "nepotistical" cycles, which are caused when young objects Z that are prematurely promoted die in the old generation while holding references to now dead objects Z' in the young generation. Call these objects "zombies" because they are dead but not recognized as such. Z keep Z' alive, because a scavenge does not know that Z is dead, because it considers all references from the old gen as roots. Worse, if Z' has references to Z and Z' stays in the young generation forever (which they can under the circumstances you describe) then Z will not be recognized as dead by a CMS collection (which currently treats all objects in the young generation as roots). This is a well-understood problem when spaces are collected independently in this manner. (The workaround is to have the CMS collector not treat the young generation as a source of roots but rather to mark through the young gen objects starting from roots.) Of course when MTT=15, then eventually every such Z' will be forced to promote to the old gen and the garbage cycle will all (hopefully) move into the old gen and thence will be reclaimed by the old gen CMS collection. This would explain the behaviour difference you saw. Now to come to the first point I made above: the use of multiple scavenger threads and their use of PLABs can sometimes cause this kind of overflow to happen (especially if there is the occasional large object). You might want to switch off the use of survivor space PLAB's (or fix them at a vey small modest value) or just use a single-threaded scavenger and see that this kind of behaviour might reduce because overflow becomes much less likely. I think there is an open bug to tune the adaptive PLAB sizing code to eliminate this kind of pathological behaviour, but we have not had the opportunity to get to that bug. If you have PrintGCDetails logs, they would probably show the premature promotion happening. PrintTenuringDistribution would not show any objects of age greater than 2 initially, yet some objects would be seen to be promoted, and then by virtue of the cross-generational references from a promoted zombie, we would artificially expand the lifetime of objects in the young generation (creating zombies Z') and so on. I believe it was partially this kind of behaviour on the part of generational scavenger implementations that caused some people in the past to start advocating the clearing of all references in objects that they knew they would be dropping references to (which of course we all know is difficult to do correctly and fraught with all kinds of problems and errors). -- ramki ------------------------------ Message: 2 Date: Thu, 05 Jun 2008 09:51:42 +0200 From: kirk Subject: Re: Proposal: MaxTenuringThreshold and age field size changes To: Jon Masamitsu Cc: hotspot-gc-dev at openjdk.java.net, Tony Printezis , Y Srinivas Ramakrishna Message-ID: <48479B0E.5060403 at javaperformancetuning.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Hi really enjoyed reading Nick's description of how he ran into trouble and what he did to fix it. My preference to delay premature promotions has always been to increase the size of survivor spaces. Using larger tenuring thresholds hasn't sounded appealing because of the cost of copying. Having GC run less often always seemed like a more desirable goal. And I think this is especially true in young where it is mostly about live object harvesting and having GC run less often gives object a better chance to expire. If these bits are so important I think that limiting MMT 31 seems reasonable at least today. If you have a bit to spare, than 63 seems even safer. Regards, Kirk Pepperdine Jon Masamitsu wrote: > Since higher MTT's are important for some applications > and we really don't know how high is high enough, should > we take John P's suggestion a step further and use the > upper 15 bits of any MTT as the tenuring age (i.e., > setting MTT to 60, 61, 62 or 63 would tenure at > age 60)? We just have to increment the age every > n-th scavenge as appropriate. > > Y Srinivas Ramakrishna wrote On 06/04/08 14:54,: > > >> Hi Nick -- >> >> thanks for sharing that experience and for the nice description! >> Looking at the PrintTenuringDistribiution for your application >> with the old, 5-age bit JVM run with MTT=24 would probably be >> illuminating. By the way, would you expect the objects surviving >> 24 scavenges (in that original configuration) to live for a pretty long time? >> Do you know if your object age distribution has a long thin tail, >> and, if so, where it falls to 0? If it is the case that most applications >> have a long thin tail and that the population in that tail (age > MTT) >> is large, then NeverTenure is probably a bad idea. >> >> As you found, the basic rule is always that if a small transient overload >> causes survivor space overflow, that in turn can cause a much longer-term "ringing effect" >> because of "nepotism" (which begets more nepotism, ....,) the effects of >> which can last much longer than the initial transient that set it off. >> And, yes, NeverTenure will lead to overflow in long-tailed distributions >> unless your survivor spaces are large enough to accomodate the tail; >> which is just another way of saying, if the tail does not fit, you will >> have overflow which will cause nepotism and its ill-effects. >> >> Thanks again for sharing that experience and for the nice >> explanation of your experiments. >> >> -- ramki >> >> >> >> >>> So, as a conclusion: Yes, we are missing the 5th age bit. NeverTenure >>> >>> works very bad for us. MTT=15 with our original eden size is too >>> small, >>> but increasing the eden size allows us to get similar behavior with >>> MTT=15 as with the original configuration (MTT=24 and 5 age bits). >>> >>> I think, Tony's suggestion to limit the configurable MTT to 2^n-1 >>> (with >>> n being the age bits) is a good solution. At least according to my >>> tests, this is much better then activating the "never tenure" policy >>> when the user is not aware of this. >>> >>> I hope some of this may have been helpful for you. I will send some more >>> detailed results and logfiles directly to Tony. >>> >>> Nick. >>> >>> >>> > > > ------------------------------ Message: 3 Date: Thu, 05 Jun 2008 08:39:01 -0700 From: Jon Masamitsu Subject: Re: Proposal: MaxTenuringThreshold and age field size changes To: hotspot-gc-dev at openjdk.java.net Message-ID: <48480895.7010502 at Sun.COM> Content-Type: text/plain; format=flowed; charset=ISO-8859-1 Jon Masamitsu wrote: > Since higher MTT's are important for some applications > and we really don't know how high is high enough, should > we take John P's suggestion a step further and use the > upper 15 bits of any MTT as the tenuring age (i.e. Sorry, guys. That should have been 4 bits. You can stop wondering where that heck I got 15 bits now. > setting MTT to 60, 61, 62 or 63 would tenure at > age 60)? We just have to increment the age every > n-th scavenge as appropriate. > > Y Srinivas Ramakrishna wrote On 06/04/08 14:54,: > >> Hi Nick -- >> >> thanks for sharing that experience and for the nice description! >> Looking at the PrintTenuringDistribiution for your application >> with the old, 5-age bit JVM run with MTT=24 would probably be >> illuminating. By the way, would you expect the objects surviving >> 24 scavenges (in that original configuration) to live for a pretty long time? >> Do you know if your object age distribution has a long thin tail, >> and, if so, where it falls to 0? If it is the case that most applications >> have a long thin tail and that the population in that tail (age > MTT) >> is large, then NeverTenure is probably a bad idea. >> >> As you found, the basic rule is always that if a small transient overload >> causes survivor space overflow, that in turn can cause a much longer-term "ringing effect" >> because of "nepotism" (which begets more nepotism, ....,) the effects of >> which can last much longer than the initial transient that set it off. >> And, yes, NeverTenure will lead to overflow in long-tailed distributions >> unless your survivor spaces are large enough to accomodate the tail; >> which is just another way of saying, if the tail does not fit, you will >> have overflow which will cause nepotism and its ill-effects. >> >> Thanks again for sharing that experience and for the nice >> explanation of your experiments. >> >> -- ramki >> >> >> >>> So, as a conclusion: Yes, we are missing the 5th age bit. NeverTenure >>> >>> works very bad for us. MTT=15 with our original eden size is too >>> small, >>> but increasing the eden size allows us to get similar behavior with >>> MTT=15 as with the original configuration (MTT=24 and 5 age bits). >>> >>> I think, Tony's suggestion to limit the configurable MTT to 2^n-1 >>> (with >>> n being the age bits) is a good solution. At least according to my >>> tests, this is much better then activating the "never tenure" policy >>> when the user is not aware of this. >>> >>> I hope some of this may have been helpful for you. I will send some more >>> detailed results and logfiles directly to Tony. >>> >>> Nick. >>> >>> > ------------------------------ Message: 4 Date: Thu, 05 Jun 2008 17:56:18 +0000 From: jon.masamitsu at sun.com Subject: hg: jdk7/hotspot-gc/hotspot: 6629727: assertion in set_trap_state() in methodDataOop.hpp is too strong. To: jdk7-changes at openjdk.java.net, hotspot-gc-dev at openjdk.java.net Message-ID: <20080605175622.8667328041 at hg.openjdk.java.net> Changeset: 0b27f3512f9e Author: jmasa Date: 2008-06-04 13:51 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/0b27f3512f9e 6629727: assertion in set_trap_state() in methodDataOop.hpp is too strong. Summary: The assertion can failure due to race conditions. Reviewed-by: never ! src/share/vm/oops/methodDataOop.hpp End of hotspot-gc-dev Digest, Vol 12, Issue 4 ********************************************* From Y.S.Ramakrishna at Sun.COM Fri Jun 6 19:59:02 2008 From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna) Date: Fri, 06 Jun 2008 12:59:02 -0700 Subject: Proposal: MaxTenuringThreshold and age field size changes In-Reply-To: <8CDFE9CE1B9A344E9AA47C27B7C58DD8314300@MSMAIL.cboent.cboe.com> References: <8CDFE9CE1B9A344E9AA47C27B7C58DD8314300@MSMAIL.cboent.cboe.com> Message-ID: > > The lesson here is promotions are much more expensive than copies. I > thought I'd share that. And, just to state the somewhat obvious warning, for the benefit of the unwary ... the relative costs of the two depend on what the old generation allocator/collector are vs the young generation collector. So you might need to retune were you to use a different memory manager for the old generation, for example. -- ramki From martinrb at google.com Sun Jun 8 22:12:52 2008 From: martinrb at google.com (Martin Buchholz) Date: Sun, 8 Jun 2008 15:12:52 -0700 Subject: Breadth-first vs. Depth-first Message-ID: <1ccfd1c10806081512q69ce07a1qabdbfaad36d2a872@mail.gmail.com> JavaOne 2008 technical session PDFs are now available http://developers.sun.com/learning/javaoneonline/j1online.jsp?track=javase&yr=2008 I enjoyed the discussion of breadh-first vs. depth-first GC copying in the JavaOne talk http://developers.sun.com/learning/javaoneonline/2008/pdf/TS-6434.pdf I'd like to suggest that this is a fruitful avenue for investigation, if hotspot engineers go further than simply switching whole-hog to depth-first. Profiling can find reference fields that are rarely dereferenced, and also ones that are likely to be dereferenced soon after the object itself is accessed. The former can be sequestered by the next copying GC into an unloved-object ghetto, while the latter can be stored contiguously with its parent object. Tricky to implement, and will certainly make GC slower, but might make the runtime of the mutator win big. And should be easier to implement than true object inlining. This ought to be very similar to the moving of dead or cold code fragments inside a method away from the hot code, that hotspot already does. Martin From tony.printezis at sun.com Mon Jun 9 15:51:04 2008 From: tony.printezis at sun.com (Tony Printezis) Date: Mon, 09 Jun 2008 11:51:04 -0400 Subject: Breadth-first vs. Depth-first In-Reply-To: <1ccfd1c10806081512q69ce07a1qabdbfaad36d2a872@mail.gmail.com> References: <1ccfd1c10806081512q69ce07a1qabdbfaad36d2a872@mail.gmail.com> Message-ID: <484D5168.4020806@sun.com> Martin, I'm personally skeptical on whether we should pursue this further. There are a couple of reasons why. Such profiling could be expensive and I'm not convinced that we will be able to reclaim back all the extra cost (the benefit we got from depth-first was very dependent on the architecture; some architectures got a lot, some got peanuts; the latter might be penalized, instead of benefit, due to the extra profiling). Additionally, the benefit from depth-first really depends on the GC used. For the GCs in HotSpot, the only opportunity we have to re-arrange objects is when new objects are copied from the eden to the survivor spaces and from the survivor spaces to the old generation. After that, you're stuck with the order in which objects moved to the old generation, as the old generation typically doesn't re-order objects (UseParallelGC and UseParallelOldGC do sliding compaction and they maintain object order, CMS doesn't typically do compaction; in fact, due to its in-place de-allocation policy, CMS cannot even guarantee that two objects that are allocated one after the other, will actually be located in the same order in the old gen). As an aside, Garbage-First: http://developers.sun.com/learning/javaoneonline/2008/pdf/TS-5419.pdf might have the edge over the other GCs in that respect, as it does copying (not sliding) GC in the old generation too and it might have more opportunities to re-order objects. Now, think about the following (very realistic!) scenario. An application has a non-trivial initialization phase and a lot of objects created during it will survive the lifetime of that application. After the initialization phase, the application goes into a steady-state phase (this is the "request serving" phase). Now, you would think that the initialization and steady-state phases will have quite different access patterns (you see where this is going, aren't you?). So, the objects created during the initialization phase will be copied according to the initialization access pattern, which might or might not be what you want during the steady-state phase. I think this is an example that shows that, in some (many?) cases, when the decision would be made on which objects to cluster together, it would be made prematurely. Now, do I think depth-first was a bad idea? Nope. I think it was a good idea, not because depth-first is really good, but instead because breadth-first was pathological in many many cases. Regarding, the GC times vs. mutator speed trade-off, I was actually expecting depth-first to have exactly this effect: slower GCs, but faster mutator threads. Interestingly, in most tests I ran, GCs didn't slow down much and, in some cases, they actually speeded up. I think this could have been due to the new order also benefiting the GC too, as well as the mutators threads. My two cents, Tony, HS GC Group Martin Buchholz wrote: > JavaOne 2008 technical session PDFs are now available > > http://developers.sun.com/learning/javaoneonline/j1online.jsp?track=javase&yr=2008 > > I enjoyed the discussion of breadh-first vs. depth-first GC copying > in the JavaOne talk > http://developers.sun.com/learning/javaoneonline/2008/pdf/TS-6434.pdf > > I'd like to suggest that this is a fruitful avenue for investigation, > if hotspot engineers go further than simply switching whole-hog > to depth-first. > > Profiling can find reference fields that are rarely dereferenced, > and also ones that are likely to be dereferenced soon after the object > itself is accessed. > > The former can be sequestered by the next copying GC into > an unloved-object ghetto, while the latter can be stored contiguously > with its parent object. Tricky to implement, and will certainly > make GC slower, but might make the runtime of the mutator win big. > And should be easier to implement than true object inlining. > > This ought to be very similar to the moving of dead or cold code > fragments inside a method away from the hot code, that hotspot > already does. > > Martin > -- ---------------------------------------------------------------------- | Tony Printezis, Staff Engineer | Sun Microsystems Inc. | | | MS BUR02-311 | | e-mail: tony.printezis at sun.com | 35 Network Drive | | office: +1 781 442 0998 (x20998) | Burlington, MA01803-0902, USA | ---------------------------------------------------------------------- e-mail client: Thunderbird (Solaris) From Jon.Masamitsu at Sun.COM Mon Jun 9 16:08:27 2008 From: Jon.Masamitsu at Sun.COM (Jon Masamitsu) Date: Mon, 09 Jun 2008 09:08:27 -0700 Subject: GCOverheadLimit does not work ? In-Reply-To: References: Message-ID: <484D557B.1060201@Sun.COM> David, There is a bug somewhere here that relates to the GCOverheadLimit. I think the problem is that the survivor spaces are not being used and the survivor space is large enough so that the calculation of the amount of free space in the heap is > 2%. The survivor spaces are not being used because only full collections are being done later in the run and the survivor spaces are used for minor (young generation) collections. Though the description of the GCOverheadLimit refers to the amount of the heap collected, it really should just say that amount of free space in the heap. That's a bug in the specification for the parameter. I'm not sure that an out-of-memory should be thrown if there is 2% of the heap free - that's in keeping with trying to minimize the false positives resulting from this feature. Jon David Tavoularis wrote: > Hi, > > It seems that the mechanism GCOverheadLimit did not trigger, even if more than 98% of the total time is spent in garbage collection and less than 2% of the heap is recovered. The process spent 3 hours in this state, performing Full GC after Full GC. > > I am using Parallel Collector on Java6u4/Solaris9. Is it a bug or maybe I do not understand this feature ? > > "TimeStamp","GC duration","Duration between 2 FullGC" > 57356.444 27.46 27.471 > 57383.915 27.14 27.137 > 57411.052 27.63 27.738 > 57438.79 33.72 33.724 > 57472.514 30.07 30.087 > 57502.601 27.84 27.842 > 57530.443 28.46 28.519 > 57558.962 29.03 29.032 > 57587.994 29.31 29.321 > > Full GC logs are available at http://people.via.ecp.fr/~tony/tmp/gc_200802282216.zip > > Thanks in advance > _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From igor.veresov at sun.com Mon Jun 9 16:20:52 2008 From: igor.veresov at sun.com (igor.veresov at sun.com) Date: Mon, 09 Jun 2008 16:20:52 +0000 Subject: hg: jdk7/hotspot-gc/hotspot: 6711930: NUMA allocator: ParOld can create a hole less than minimal object size in the lgrp chunk Message-ID: <20080609162056.CD68A283F3@hg.openjdk.java.net> Changeset: d1635bf93939 Author: iveresov Date: 2008-06-09 07:18 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/d1635bf93939 6711930: NUMA allocator: ParOld can create a hole less than minimal object size in the lgrp chunk Summary: The fix takes care of three issues that can create a hole less a minimal object in the lgrp chunk Reviewed-by: ysr, apetrusenko ! src/share/vm/gc_implementation/shared/immutableSpace.cpp ! src/share/vm/gc_implementation/shared/immutableSpace.hpp ! src/share/vm/gc_implementation/shared/mutableNUMASpace.cpp ! src/share/vm/gc_implementation/shared/mutableNUMASpace.hpp ! src/share/vm/gc_implementation/shared/mutableSpace.cpp ! src/share/vm/gc_implementation/shared/mutableSpace.hpp From Igor.Veresov at Sun.COM Mon Jun 9 16:25:29 2008 From: Igor.Veresov at Sun.COM (Igor Veresov) Date: Mon, 09 Jun 2008 20:25:29 +0400 Subject: Breadth-first vs. Depth-first In-Reply-To: <484D5168.4020806@sun.com> References: <1ccfd1c10806081512q69ce07a1qabdbfaad36d2a872@mail.gmail.com> <484D5168.4020806@sun.com> Message-ID: <200806092025.30027.igor.veresov@sun.com> A paper comes to mind: http://www.ssw.uni-linz.ac.at/Research/Papers/Wimmer06/ igor On Monday 09 June 2008 19:51:04 Tony Printezis wrote: > Martin, > > I'm personally skeptical on whether we should pursue this further. There > are a couple of reasons why. > > Such profiling could be expensive and I'm not convinced that we will be > able to reclaim back all the extra cost (the benefit we got from > depth-first was very dependent on the architecture; some architectures > got a lot, some got peanuts; the latter might be penalized, instead of > benefit, due to the extra profiling). > > Additionally, the benefit from depth-first really depends on the GC > used. For the GCs in HotSpot, the only opportunity we have to re-arrange > objects is when new objects are copied from the eden to the survivor > spaces and from the survivor spaces to the old generation. After that, > you're stuck with the order in which objects moved to the old > generation, as the old generation typically doesn't re-order objects > (UseParallelGC and UseParallelOldGC do sliding compaction and they > maintain object order, CMS doesn't typically do compaction; in fact, due > to its in-place de-allocation policy, CMS cannot even guarantee that two > objects that are allocated one after the other, will actually be located > in the same order in the old gen). > > As an aside, Garbage-First: > > http://developers.sun.com/learning/javaoneonline/2008/pdf/TS-5419.pdf > > might have the edge over the other GCs in that respect, as it does > copying (not sliding) GC in the old generation too and it might have > more opportunities to re-order objects. > > Now, think about the following (very realistic!) scenario. An > application has a non-trivial initialization phase and a lot of objects > created during it will survive the lifetime of that application. After > the initialization phase, the application goes into a steady-state phase > (this is the "request serving" phase). Now, you would think that the > initialization and steady-state phases will have quite different access > patterns (you see where this is going, aren't you?). So, the objects > created during the initialization phase will be copied according to the > initialization access pattern, which might or might not be what you want > during the steady-state phase. I think this is an example that shows > that, in some (many?) cases, when the decision would be made on which > objects to cluster together, it would be made prematurely. > > Now, do I think depth-first was a bad idea? Nope. I think it was a good > idea, not because depth-first is really good, but instead because > breadth-first was pathological in many many cases. > > Regarding, the GC times vs. mutator speed trade-off, I was actually > expecting depth-first to have exactly this effect: slower GCs, but > faster mutator threads. Interestingly, in most tests I ran, GCs didn't > slow down much and, in some cases, they actually speeded up. I think > this could have been due to the new order also benefiting the GC too, as > well as the mutators threads. > > My two cents, > > Tony, HS GC Group > > Martin Buchholz wrote: > > JavaOne 2008 technical session PDFs are now available > > > > http://developers.sun.com/learning/javaoneonline/j1online.jsp?track=javas > >e&yr=2008 > > > > I enjoyed the discussion of breadh-first vs. depth-first GC copying > > in the JavaOne talk > > http://developers.sun.com/learning/javaoneonline/2008/pdf/TS-6434.pdf > > > > I'd like to suggest that this is a fruitful avenue for investigation, > > if hotspot engineers go further than simply switching whole-hog > > to depth-first. > > > > Profiling can find reference fields that are rarely dereferenced, > > and also ones that are likely to be dereferenced soon after the object > > itself is accessed. > > > > The former can be sequestered by the next copying GC into > > an unloved-object ghetto, while the latter can be stored contiguously > > with its parent object. Tricky to implement, and will certainly > > make GC slower, but might make the runtime of the mutator win big. > > And should be easier to implement than true object inlining. > > > > This ought to be very similar to the moving of dead or cold code > > fragments inside a method away from the hot code, that hotspot > > already does. > > > > Martin From martinrb at google.com Mon Jun 9 17:16:25 2008 From: martinrb at google.com (Martin Buchholz) Date: Mon, 9 Jun 2008 10:16:25 -0700 Subject: Breadth-first vs. Depth-first In-Reply-To: <484D5168.4020806@sun.com> References: <1ccfd1c10806081512q69ce07a1qabdbfaad36d2a872@mail.gmail.com> <484D5168.4020806@sun.com> Message-ID: <1ccfd1c10806091016o7391c91amfe72edf6a4d3e0c3@mail.gmail.com> Tony, Thanks for the reply. I continue to see parallels between code hotspots and data hotspots. In both cases there are few opportunities to rectify previous bad decisions. There is no way to declare that the initialization phase of the application is over, and the "money phase" is starting. Identifying code hotspots pays off better, because code inlining may give rise to further optimizations, unlike data colocation. I imagine that data colocation will increase in value over time, with memory access becoming a more important part of application performance. I wonder why they y'all never seem to use a technique from the AOT compiler world - provide a profiling mode to allow users to run the app in for a "typical" run, then use that profiling information in future runs. Martin (definitely *not* a hotspot engineer) On Mon, Jun 9, 2008 at 8:51 AM, Tony Printezis wrote: > Martin, > > I'm personally skeptical on whether we should pursue this further. There are > a couple of reasons why. > > Such profiling could be expensive and I'm not convinced that we will be able > to reclaim back all the extra cost (the benefit we got from depth-first was > very dependent on the architecture; some architectures got a lot, some got > peanuts; the latter might be penalized, instead of benefit, due to the extra > profiling). > > Additionally, the benefit from depth-first really depends on the GC used. > For the GCs in HotSpot, the only opportunity we have to re-arrange objects > is when new objects are copied from the eden to the survivor spaces and from > the survivor spaces to the old generation. After that, you're stuck with the > order in which objects moved to the old generation, as the old generation > typically doesn't re-order objects (UseParallelGC and UseParallelOldGC do > sliding compaction and they maintain object order, CMS doesn't typically do > compaction; in fact, due to its in-place de-allocation policy, CMS cannot > even guarantee that two objects that are allocated one after the other, will > actually be located in the same order in the old gen). > > As an aside, Garbage-First: > > http://developers.sun.com/learning/javaoneonline/2008/pdf/TS-5419.pdf > > might have the edge over the other GCs in that respect, as it does copying > (not sliding) GC in the old generation too and it might have more > opportunities to re-order objects. > > Now, think about the following (very realistic!) scenario. An application > has a non-trivial initialization phase and a lot of objects created during > it will survive the lifetime of that application. After the initialization > phase, the application goes into a steady-state phase (this is the "request > serving" phase). Now, you would think that the initialization and > steady-state phases will have quite different access patterns (you see where > this is going, aren't you?). So, the objects created during the > initialization phase will be copied according to the initialization access > pattern, which might or might not be what you want during the steady-state > phase. I think this is an example that shows that, in some (many?) cases, > when the decision would be made on which objects to cluster together, it > would be made prematurely. > > Now, do I think depth-first was a bad idea? Nope. I think it was a good > idea, not because depth-first is really good, but instead because > breadth-first was pathological in many many cases. > > Regarding, the GC times vs. mutator speed trade-off, I was actually > expecting depth-first to have exactly this effect: slower GCs, but faster > mutator threads. Interestingly, in most tests I ran, GCs didn't slow down > much and, in some cases, they actually speeded up. I think this could have > been due to the new order also benefiting the GC too, as well as the > mutators threads. > > My two cents, > > Tony, HS GC Group > > Martin Buchholz wrote: >> >> JavaOne 2008 technical session PDFs are now available >> >> >> http://developers.sun.com/learning/javaoneonline/j1online.jsp?track=javase&yr=2008 >> >> I enjoyed the discussion of breadh-first vs. depth-first GC copying >> in the JavaOne talk >> http://developers.sun.com/learning/javaoneonline/2008/pdf/TS-6434.pdf >> >> I'd like to suggest that this is a fruitful avenue for investigation, >> if hotspot engineers go further than simply switching whole-hog >> to depth-first. >> >> Profiling can find reference fields that are rarely dereferenced, >> and also ones that are likely to be dereferenced soon after the object >> itself is accessed. >> >> The former can be sequestered by the next copying GC into >> an unloved-object ghetto, while the latter can be stored contiguously >> with its parent object. Tricky to implement, and will certainly >> make GC slower, but might make the runtime of the mutator win big. >> And should be easier to implement than true object inlining. >> >> This ought to be very similar to the moving of dead or cold code >> fragments inside a method away from the hot code, that hotspot >> already does. >> >> Martin >> > > -- > ---------------------------------------------------------------------- > | Tony Printezis, Staff Engineer | Sun Microsystems Inc. | > | | MS BUR02-311 | > | e-mail: tony.printezis at sun.com | 35 Network Drive | > | office: +1 781 442 0998 (x20998) | Burlington, MA01803-0902, USA | > ---------------------------------------------------------------------- > e-mail client: Thunderbird (Solaris) > > > From Peter.Kessler at Sun.COM Mon Jun 9 18:28:45 2008 From: Peter.Kessler at Sun.COM (Peter B. Kessler) Date: Mon, 09 Jun 2008 11:28:45 -0700 Subject: GCOverheadLimit does not work ? In-Reply-To: References: <484D557B.1060201@Sun.COM> Message-ID: <484D765D.2000500@Sun.COM> There's product(uintx, GCHeapFreeLimit, 2, \ "Minimum percentage of free space after a full GC before an " \ "OutOfMemoryError is thrown (used with GCTimeLimit)") \ But I have to say that I've never gotten it to behave the way I thought it worked. ... peter David Tavoularis wrote: > Hi Jon, > > I would like to understand the computation of the percentage for the following Full GC example : > > 57587.994: [Full GC [PSYoungGen: 262144K->260978K(317184K)] [PSOldGen: 1310719K->1310719K(1310720K)] 1572863K->1571698K(1627904K) [PSPermGen: 17242K->17242K(20480K)] > Heap after GC : > PSYoungGen total 317184K, used 260978K (eden=262144K, from=OK (out of 55040K)) > PSOldGen total 1310720K, used 1310719K > PSPermGen total 20480K, used 17242K > > So the percentage is computed like this : ((317184K-260978K)+(1310720K-1310719K)+(20480K-17242K))/(317184K+1310720K+20480K)=59445K/1648384K=3.6% > 2% > > The fix would be to compute it like this : ((262144K-260978K)+(1310720K-1310719K)+(20480K-17242K))/(317184K+1310720K+20480K)=4405K/1648384K=0.3% < 2% > > > > Is there a WA to this issue ? > > 1. Is is possible to change by property the percentage from 2% to 5% ? > > 2. Maybe I could reduce the young size from 256MB to 128MB, and it will automatically reduce (?) the survivor from 55040K to 27520K ? And the percentage would be in this case 1.9% and the OOM would be triggered. > > > Thanks in advance for your explanations > _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From Jon.Masamitsu at Sun.COM Mon Jun 9 18:55:07 2008 From: Jon.Masamitsu at Sun.COM (Jon Masamitsu) Date: Mon, 09 Jun 2008 11:55:07 -0700 Subject: GCOverheadLimit does not work ? In-Reply-To: References: <484D557B.1060201@Sun.COM> Message-ID: <484D7C8B.6070301@Sun.COM> David Tavoularis wrote: > Hi Jon, > > I would like to understand the computation of the percentage for the following Full GC example : > > 57587.994: [Full GC [PSYoungGen: 262144K->260978K(317184K)] [PSOldGen: 1310719K->1310719K(1310720K)] 1572863K->1571698K(1627904K) [PSPermGen: 17242K->17242K(20480K)] > Heap after GC : > PSYoungGen total 317184K, used 260978K (eden=262144K, from=OK (out of 55040K)) > PSOldGen total 1310720K, used 1310719K > PSPermGen total 20480K, used 17242K > > So the percentage is computed like this : ((317184K-260978K)+(1310720K-1310719K)+(20480K-17242K))/(317184K+1310720K+20480K)=59445K/1648384K=3.6% > 2% > If you want a definitive answer, please take a look at the sources. As I recall it, the perm gen is not counted so it's more like ((317184K-260978K)+(1310720K-1310719K)) / (317184K+1310720K) = 3.5% > The fix would be to compute it like this : ((262144K-260978K)+(1310720K-1310719K)+(20480K-17242K))/(317184K+1310720K+20480K)=4405K/1648384K=0.3% < 2% > I don't think that is always the right answer. If the survivor space is being used (i.e., objects are copied to it and collected from it), then it should be counted. In a young gen collection I would expect it to be counted all the time but there might be exceptions (e.g., an always tenure policy). When it's a full collection if it was being used it should be counted. I would expect it to be used when there are young gen collections occurring between full collections. If the collections are back-to-back full collections and the survivor space is not being used it might make sense to not count it. Sorry, it's just not clear to me. Counting it sometimes and not counting it others might surprise some users. > > > Is there a WA to this issue ? WA? > > 1. Is is possible to change by property the percentage from 2% to 5% ? -XX:GCHeapFreeLimit=5 > > 2. Maybe I could reduce the young size from 256MB to 128MB, and it will automatically reduce (?) the survivor from 55040K to 27520K ? And the percentage would be in this case 1.9% and the OOM would be triggered. > That might work but what is your goal? This feature is to just alert unsuspecting users to a very bad GC situation. You already know you have a problem here, right? _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From tony.printezis at sun.com Tue Jun 10 18:07:07 2008 From: tony.printezis at sun.com (Tony Printezis) Date: Tue, 10 Jun 2008 14:07:07 -0400 Subject: Breadth-first vs. Depth-first In-Reply-To: <1ccfd1c10806091016o7391c91amfe72edf6a4d3e0c3@mail.gmail.com> References: <1ccfd1c10806081512q69ce07a1qabdbfaad36d2a872@mail.gmail.com> <484D5168.4020806@sun.com> <1ccfd1c10806091016o7391c91amfe72edf6a4d3e0c3@mail.gmail.com> Message-ID: <484EC2CB.80701@sun.com> Martin, Martin Buchholz wrote: > Tony, > > Thanks for the reply. > > I continue to see parallels between code hotspots and data hotspots. > In both cases there are few opportunities to rectify previous bad > decisions. Yes, but in the case of code generation, recompiling a method can be (generally) done in a quite low-overhead manner. However, re-arranging the location of objects is much much more intrusive. > There is no way to declare that the initialization phase of > the application is over, and the "money phase" is starting. > Maybe, there should be? > Identifying code hotspots pays off better, because code inlining > may give rise to further optimizations, unlike data colocation. > > I imagine that data colocation will increase in value over time, > with memory access becoming a more important part of > application performance. > I totally agree with this. > I wonder why they y'all never seem to use a technique from the > AOT compiler world - provide a profiling mode to allow users > to run the app in for a "typical" run, then use that profiling > information in future runs. > I've never been enamored with the idea, I have to say. I just don't know how willing customers will be to do training runs. And remember to redo the training runs when their code changes. And do the training run with a realistic input ("Hey, your GC sucks. I did a training run with a 1MB DB input and when I loaded 50GB, the application went slower." Well, yeah....). Given that some of our customers are lurking here, I'd love to hear from them on this, actually... Tony > Martin (definitely *not* a hotspot engineer) > > On Mon, Jun 9, 2008 at 8:51 AM, Tony Printezis wrote: > >> Martin, >> >> I'm personally skeptical on whether we should pursue this further. There are >> a couple of reasons why. >> >> Such profiling could be expensive and I'm not convinced that we will be able >> to reclaim back all the extra cost (the benefit we got from depth-first was >> very dependent on the architecture; some architectures got a lot, some got >> peanuts; the latter might be penalized, instead of benefit, due to the extra >> profiling). >> >> Additionally, the benefit from depth-first really depends on the GC used. >> For the GCs in HotSpot, the only opportunity we have to re-arrange objects >> is when new objects are copied from the eden to the survivor spaces and from >> the survivor spaces to the old generation. After that, you're stuck with the >> order in which objects moved to the old generation, as the old generation >> typically doesn't re-order objects (UseParallelGC and UseParallelOldGC do >> sliding compaction and they maintain object order, CMS doesn't typically do >> compaction; in fact, due to its in-place de-allocation policy, CMS cannot >> even guarantee that two objects that are allocated one after the other, will >> actually be located in the same order in the old gen). >> >> As an aside, Garbage-First: >> >> http://developers.sun.com/learning/javaoneonline/2008/pdf/TS-5419.pdf >> >> might have the edge over the other GCs in that respect, as it does copying >> (not sliding) GC in the old generation too and it might have more >> opportunities to re-order objects. >> >> Now, think about the following (very realistic!) scenario. An application >> has a non-trivial initialization phase and a lot of objects created during >> it will survive the lifetime of that application. After the initialization >> phase, the application goes into a steady-state phase (this is the "request >> serving" phase). Now, you would think that the initialization and >> steady-state phases will have quite different access patterns (you see where >> this is going, aren't you?). So, the objects created during the >> initialization phase will be copied according to the initialization access >> pattern, which might or might not be what you want during the steady-state >> phase. I think this is an example that shows that, in some (many?) cases, >> when the decision would be made on which objects to cluster together, it >> would be made prematurely. >> >> Now, do I think depth-first was a bad idea? Nope. I think it was a good >> idea, not because depth-first is really good, but instead because >> breadth-first was pathological in many many cases. >> >> Regarding, the GC times vs. mutator speed trade-off, I was actually >> expecting depth-first to have exactly this effect: slower GCs, but faster >> mutator threads. Interestingly, in most tests I ran, GCs didn't slow down >> much and, in some cases, they actually speeded up. I think this could have >> been due to the new order also benefiting the GC too, as well as the >> mutators threads. >> >> My two cents, >> >> Tony, HS GC Group >> >> Martin Buchholz wrote: >> >>> JavaOne 2008 technical session PDFs are now available >>> >>> >>> http://developers.sun.com/learning/javaoneonline/j1online.jsp?track=javase&yr=2008 >>> >>> I enjoyed the discussion of breadh-first vs. depth-first GC copying >>> in the JavaOne talk >>> http://developers.sun.com/learning/javaoneonline/2008/pdf/TS-6434.pdf >>> >>> I'd like to suggest that this is a fruitful avenue for investigation, >>> if hotspot engineers go further than simply switching whole-hog >>> to depth-first. >>> >>> Profiling can find reference fields that are rarely dereferenced, >>> and also ones that are likely to be dereferenced soon after the object >>> itself is accessed. >>> >>> The former can be sequestered by the next copying GC into >>> an unloved-object ghetto, while the latter can be stored contiguously >>> with its parent object. Tricky to implement, and will certainly >>> make GC slower, but might make the runtime of the mutator win big. >>> And should be easier to implement than true object inlining. >>> >>> This ought to be very similar to the moving of dead or cold code >>> fragments inside a method away from the hot code, that hotspot >>> already does. >>> >>> Martin >>> >>> >> -- >> ---------------------------------------------------------------------- >> | Tony Printezis, Staff Engineer | Sun Microsystems Inc. | >> | | MS BUR02-311 | >> | e-mail: tony.printezis at sun.com | 35 Network Drive | >> | office: +1 781 442 0998 (x20998) | Burlington, MA01803-0902, USA | >> ---------------------------------------------------------------------- >> e-mail client: Thunderbird (Solaris) >> >> >> >> -- ---------------------------------------------------------------------- | Tony Printezis, Staff Engineer | Sun Microsystems Inc. | | | MS BUR02-311 | | e-mail: tony.printezis at sun.com | 35 Network Drive | | office: +1 781 442 0998 (x20998) | Burlington, MA01803-0902, USA | ---------------------------------------------------------------------- e-mail client: Thunderbird (Solaris) From rocketraman at fastmail.fm Tue Jun 10 18:23:45 2008 From: rocketraman at fastmail.fm (Raman Gupta) Date: Tue, 10 Jun 2008 14:23:45 -0400 Subject: Breadth-first vs. Depth-first In-Reply-To: <484EC2CB.80701@sun.com> References: <1ccfd1c10806081512q69ce07a1qabdbfaad36d2a872@mail.gmail.com> <484D5168.4020806@sun.com> <1ccfd1c10806091016o7391c91amfe72edf6a4d3e0c3@mail.gmail.com> <484EC2CB.80701@sun.com> Message-ID: <484EC6B1.4040007@fastmail.fm> Tony Printezis wrote: > I've never been enamored with the idea, I have to say. I just don't know > how willing customers will be to do training runs. And remember to redo > the training runs when their code changes. And do the training run with > a realistic input ("Hey, your GC sucks. I did a training run with a 1MB > DB input and when I loaded 50GB, the application went slower." Well, > yeah....). > > Given that some of our customers are lurking here, I'd love to hear from > them on this, actually... As a customer, I can say... please no training runs! In my opinion, that makes an already complicated system even more so, for what I would think is marginal gain. Something you may want to consider is allowing applications an interface into the GC subsystem (I've brought this up on this list before as well). In addition to the interface methods I suggested previously, you could add: public void startingApplicationPreloadingPhase(); public void startingApplicationSteadyStatePhase(); Applications that have the preloading/steady state behavior can then signal the GC subsystem with the state of the application, which allows you to start "training" the GC only at steady state, without any guesswork on the part of the GC subsystem. Any applications not using those functions will simply run as they do today. Cheers, Raman Gupta From Paul.Hohensee at Sun.COM Tue Jun 10 18:22:24 2008 From: Paul.Hohensee at Sun.COM (Paul Hohensee) Date: Tue, 10 Jun 2008 14:22:24 -0400 Subject: Breadth-first vs. Depth-first In-Reply-To: <484EC2CB.80701@sun.com> References: <1ccfd1c10806081512q69ce07a1qabdbfaad36d2a872@mail.gmail.com> <484D5168.4020806@sun.com> <1ccfd1c10806091016o7391c91amfe72edf6a4d3e0c3@mail.gmail.com> <484EC2CB.80701@sun.com> Message-ID: <484EC660.7050100@sun.com> 'Ergo writ large' proposes a repository that would contain history of actual runs, so no explicit training runs would be necessary. No resources though. Paul Tony Printezis wrote: > Martin, > > Martin Buchholz wrote: >> Tony, >> >> Thanks for the reply. >> >> I continue to see parallels between code hotspots and data hotspots. >> In both cases there are few opportunities to rectify previous bad >> decisions. > Yes, but in the case of code generation, recompiling a method can be > (generally) done in a quite low-overhead manner. However, re-arranging > the location of objects is much much more intrusive. >> There is no way to declare that the initialization phase of >> the application is over, and the "money phase" is starting. >> > Maybe, there should be? >> Identifying code hotspots pays off better, because code inlining >> may give rise to further optimizations, unlike data colocation. >> >> I imagine that data colocation will increase in value over time, >> with memory access becoming a more important part of >> application performance. >> > I totally agree with this. >> I wonder why they y'all never seem to use a technique from the >> AOT compiler world - provide a profiling mode to allow users >> to run the app in for a "typical" run, then use that profiling >> information in future runs. >> > I've never been enamored with the idea, I have to say. I just don't > know how willing customers will be to do training runs. And remember > to redo the training runs when their code changes. And do the training > run with a realistic input ("Hey, your GC sucks. I did a training run > with a 1MB DB input and when I loaded 50GB, the application went > slower." Well, yeah....). > > Given that some of our customers are lurking here, I'd love to hear > from them on this, actually... > > Tony >> Martin (definitely *not* a hotspot engineer) >> >> On Mon, Jun 9, 2008 at 8:51 AM, Tony Printezis >> wrote: >> >>> Martin, >>> >>> I'm personally skeptical on whether we should pursue this further. >>> There are >>> a couple of reasons why. >>> >>> Such profiling could be expensive and I'm not convinced that we will >>> be able >>> to reclaim back all the extra cost (the benefit we got from >>> depth-first was >>> very dependent on the architecture; some architectures got a lot, >>> some got >>> peanuts; the latter might be penalized, instead of benefit, due to >>> the extra >>> profiling). >>> >>> Additionally, the benefit from depth-first really depends on the GC >>> used. >>> For the GCs in HotSpot, the only opportunity we have to re-arrange >>> objects >>> is when new objects are copied from the eden to the survivor spaces >>> and from >>> the survivor spaces to the old generation. After that, you're stuck >>> with the >>> order in which objects moved to the old generation, as the old >>> generation >>> typically doesn't re-order objects (UseParallelGC and >>> UseParallelOldGC do >>> sliding compaction and they maintain object order, CMS doesn't >>> typically do >>> compaction; in fact, due to its in-place de-allocation policy, CMS >>> cannot >>> even guarantee that two objects that are allocated one after the >>> other, will >>> actually be located in the same order in the old gen). >>> >>> As an aside, Garbage-First: >>> >>> http://developers.sun.com/learning/javaoneonline/2008/pdf/TS-5419.pdf >>> >>> might have the edge over the other GCs in that respect, as it does >>> copying >>> (not sliding) GC in the old generation too and it might have more >>> opportunities to re-order objects. >>> >>> Now, think about the following (very realistic!) scenario. An >>> application >>> has a non-trivial initialization phase and a lot of objects created >>> during >>> it will survive the lifetime of that application. After the >>> initialization >>> phase, the application goes into a steady-state phase (this is the >>> "request >>> serving" phase). Now, you would think that the initialization and >>> steady-state phases will have quite different access patterns (you >>> see where >>> this is going, aren't you?). So, the objects created during the >>> initialization phase will be copied according to the initialization >>> access >>> pattern, which might or might not be what you want during the >>> steady-state >>> phase. I think this is an example that shows that, in some (many?) >>> cases, >>> when the decision would be made on which objects to cluster >>> together, it >>> would be made prematurely. >>> >>> Now, do I think depth-first was a bad idea? Nope. I think it was a good >>> idea, not because depth-first is really good, but instead because >>> breadth-first was pathological in many many cases. >>> >>> Regarding, the GC times vs. mutator speed trade-off, I was actually >>> expecting depth-first to have exactly this effect: slower GCs, but >>> faster >>> mutator threads. Interestingly, in most tests I ran, GCs didn't slow >>> down >>> much and, in some cases, they actually speeded up. I think this >>> could have >>> been due to the new order also benefiting the GC too, as well as the >>> mutators threads. >>> >>> My two cents, >>> >>> Tony, HS GC Group >>> >>> Martin Buchholz wrote: >>> >>>> JavaOne 2008 technical session PDFs are now available >>>> >>>> >>>> http://developers.sun.com/learning/javaoneonline/j1online.jsp?track=javase&yr=2008 >>>> >>>> >>>> I enjoyed the discussion of breadh-first vs. depth-first GC copying >>>> in the JavaOne talk >>>> http://developers.sun.com/learning/javaoneonline/2008/pdf/TS-6434.pdf >>>> >>>> I'd like to suggest that this is a fruitful avenue for investigation, >>>> if hotspot engineers go further than simply switching whole-hog >>>> to depth-first. >>>> >>>> Profiling can find reference fields that are rarely dereferenced, >>>> and also ones that are likely to be dereferenced soon after the object >>>> itself is accessed. >>>> >>>> The former can be sequestered by the next copying GC into >>>> an unloved-object ghetto, while the latter can be stored contiguously >>>> with its parent object. Tricky to implement, and will certainly >>>> make GC slower, but might make the runtime of the mutator win big. >>>> And should be easier to implement than true object inlining. >>>> >>>> This ought to be very similar to the moving of dead or cold code >>>> fragments inside a method away from the hot code, that hotspot >>>> already does. >>>> >>>> Martin >>>> >>>> >>> -- >>> ---------------------------------------------------------------------- >>> | Tony Printezis, Staff Engineer | Sun Microsystems Inc. | >>> | | MS BUR02-311 | >>> | e-mail: tony.printezis at sun.com | 35 Network Drive | >>> | office: +1 781 442 0998 (x20998) | Burlington, MA01803-0902, USA | >>> ---------------------------------------------------------------------- >>> e-mail client: Thunderbird (Solaris) >>> >>> >>> >>> > From john.coomes at sun.com Tue Jun 10 22:47:40 2008 From: john.coomes at sun.com (john.coomes at sun.com) Date: Tue, 10 Jun 2008 22:47:40 +0000 Subject: hg: jdk7/hotspot-gc/hotspot: 40 new changesets Message-ID: <20080610224857.BCD9C28840@hg.openjdk.java.net> Changeset: 24706b95d959 Author: xdono Date: 2008-04-24 12:12 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/24706b95d959 Added tag jdk7-b26 for changeset ad0b851458ff ! .hgtags Changeset: c70a245cad3a Author: dcubed Date: 2008-05-09 08:55 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/c70a245cad3a 6670684: 4/5 SA command universe did not print out CMS space information Summary: Forward port of Yumin's fix for 6670684 from HSX-11; Yumin verified the port was correct. Reviewed-by: dcubed ! agent/make/Makefile ! agent/src/share/classes/sun/jvm/hotspot/HSDB.java ! agent/src/share/classes/sun/jvm/hotspot/SALauncherLoader.java ! agent/src/share/classes/sun/jvm/hotspot/bugspot/Main.java ! agent/src/share/classes/sun/jvm/hotspot/jdi/SAJDIClassLoader.java + agent/src/share/classes/sun/jvm/hotspot/memory/BinaryTreeDictionary.java ! agent/src/share/classes/sun/jvm/hotspot/memory/CompactibleFreeListSpace.java ! agent/src/share/classes/sun/jvm/hotspot/memory/DefNewGeneration.java + agent/src/share/classes/sun/jvm/hotspot/memory/FreeList.java + agent/src/share/classes/sun/jvm/hotspot/memory/LinearAllocBlock.java ! agent/src/share/classes/sun/jvm/hotspot/ui/AnnotatedMemoryPanel.java ! agent/src/share/classes/sun/jvm/hotspot/ui/CommandProcessorPanel.java ! agent/src/share/classes/sun/jvm/hotspot/ui/DebuggerConsolePanel.java ! agent/src/share/classes/sun/jvm/hotspot/ui/HighPrecisionJScrollBar.java ! agent/src/share/classes/sun/jvm/hotspot/ui/JFrameWrapper.java ! agent/src/share/classes/sun/jvm/hotspot/ui/treetable/JTreeTable.java ! src/share/vm/gc_implementation/concurrentMarkSweep/binaryTreeDictionary.hpp ! src/share/vm/gc_implementation/concurrentMarkSweep/freeList.hpp ! src/share/vm/gc_implementation/concurrentMarkSweep/vmStructs_cms.hpp ! src/share/vm/runtime/vmStructs.cpp Changeset: 6ab92ec09f70 Author: dcubed Date: 2008-05-09 09:11 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/6ab92ec09f70 Merge Changeset: 09c2ba680204 Author: kvn Date: 2008-05-15 22:40 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/09c2ba680204 6700102: c2 assertion "counter_changed,"failed dependencies, but counter didn't change")" with AggressiveOpts Summary: Bytecode Escape Analyzer does not have the check for the case described in 6389127. Reviewed-by: never ! src/share/vm/ci/bcEscapeAnalyzer.cpp Changeset: 723be81c1212 Author: kvn Date: 2008-05-15 22:43 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/723be81c1212 6701887: JDK7 server VM in endless loop in Node::dominates Summary: The method Node::dominates loops in the dead code which does not have a Region node. Reviewed-by: jrose, never ! src/share/vm/opto/c2_globals.hpp ! src/share/vm/opto/memnode.cpp ! src/share/vm/opto/node.cpp Changeset: 5bba3366a9a2 Author: dcubed Date: 2008-05-16 13:42 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/5bba3366a9a2 Merge ! agent/src/share/classes/sun/jvm/hotspot/HSDB.java ! src/share/vm/runtime/vmStructs.cpp Changeset: a3e5744fafda Author: dcubed Date: 2008-05-20 09:47 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/a3e5744fafda Merge Changeset: a49545cab84a Author: ohair Date: 2008-05-27 09:47 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/a49545cab84a 6563752: Build and test JDK7 with Sun Studio 12 Express compilers (prep makefiles) Summary: Allows for building with SS12, no longer requires SS11, warns if not SS11 for now. Once SS12 is validated and performance measurements look ok, SS12 will be the validated compiler. Reviewed-by: sspitsyn, ikrylov ! make/jprt.config ! make/solaris/makefiles/debug.make ! make/solaris/makefiles/dtrace.make ! make/solaris/makefiles/fastdebug.make ! make/solaris/makefiles/jvmg.make ! make/solaris/makefiles/optimized.make ! make/solaris/makefiles/product.make ! make/solaris/makefiles/sparc.make ! make/solaris/makefiles/sparcWorks.make ! make/solaris/makefiles/sparcv9.make Changeset: af059c49e677 Author: ohair Date: 2008-05-28 10:16 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/af059c49e677 6703308: Fix jprt.properties to allow for jdk6 and jdk7 builds Summary: Allows for jprt submit -release option to select jdk version and proper build targets. Reviewed-by: jcoomes ! make/jprt.properties Changeset: 27f13876aef3 Author: iveresov Date: 2008-05-30 03:53 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/27f13876aef3 Merge Changeset: 8aa010f60e0f Author: rasbold Date: 2008-05-20 06:32 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/8aa010f60e0f Merge Changeset: 885ed790ecf0 Author: kvn Date: 2008-05-21 10:45 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/885ed790ecf0 6695810: null oop passed to encode_heap_oop_not_null Summary: fix several problems in C2 related to Escape Analysis and Compressed Oops. Reviewed-by: never, jrose ! src/cpu/sparc/vm/sparc.ad ! src/cpu/x86/vm/x86_64.ad ! src/share/vm/opto/callnode.cpp ! src/share/vm/opto/cfgnode.cpp ! src/share/vm/opto/compile.cpp ! src/share/vm/opto/connode.cpp ! src/share/vm/opto/connode.hpp ! src/share/vm/opto/escape.cpp ! src/share/vm/opto/library_call.cpp ! src/share/vm/opto/macro.cpp ! src/share/vm/opto/matcher.cpp ! src/share/vm/opto/memnode.cpp ! src/share/vm/opto/memnode.hpp ! src/share/vm/opto/node.cpp ! src/share/vm/opto/output.cpp ! src/share/vm/opto/type.cpp ! src/share/vm/opto/type.hpp + test/compiler/6689060/Test.java + test/compiler/6695810/Test.java Changeset: c436414a719e Author: kvn Date: 2008-05-21 13:46 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/c436414a719e 6703890: Compressed Oops: add LoadNKlass node to generate narrow oops (32-bits) compare instructions Summary: Add LoadNKlass and CMoveN nodes, use CmpN and ConN nodes to generate narrow oops compare instructions. Reviewed-by: never, rasbold ! src/cpu/sparc/vm/assembler_sparc.cpp ! src/cpu/sparc/vm/assembler_sparc.hpp ! src/cpu/sparc/vm/relocInfo_sparc.cpp ! src/cpu/sparc/vm/relocInfo_sparc.hpp ! src/cpu/sparc/vm/sparc.ad ! src/cpu/x86/vm/assembler_x86_64.cpp ! src/cpu/x86/vm/assembler_x86_64.hpp ! src/cpu/x86/vm/relocInfo_x86.cpp ! src/cpu/x86/vm/relocInfo_x86.hpp ! src/cpu/x86/vm/x86_64.ad ! src/share/vm/adlc/forms.cpp ! src/share/vm/adlc/formssel.cpp ! src/share/vm/includeDB_core ! src/share/vm/opto/classes.hpp ! src/share/vm/opto/compile.cpp ! src/share/vm/opto/connode.cpp ! src/share/vm/opto/connode.hpp ! src/share/vm/opto/doCall.cpp ! src/share/vm/opto/escape.cpp ! src/share/vm/opto/graphKit.cpp ! src/share/vm/opto/lcm.cpp ! src/share/vm/opto/library_call.cpp ! src/share/vm/opto/loopopts.cpp ! src/share/vm/opto/matcher.cpp ! src/share/vm/opto/memnode.cpp ! src/share/vm/opto/memnode.hpp ! src/share/vm/opto/parse1.cpp ! src/share/vm/opto/parseHelper.cpp ! src/share/vm/runtime/globals.hpp Changeset: 437d03ea40b1 Author: kvn Date: 2008-05-21 16:31 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/437d03ea40b1 6703888: Compressed Oops: use the 32-bits gap after klass in a object Summary: Use the gap also for a narrow oop field and a boxing object value. Reviewed-by: coleenp, never ! src/cpu/sparc/vm/sharedRuntime_sparc.cpp ! src/cpu/sparc/vm/sparc.ad ! src/cpu/x86/vm/sharedRuntime_x86_32.cpp ! src/cpu/x86/vm/sharedRuntime_x86_64.cpp ! src/share/vm/ci/ciInstanceKlass.cpp ! src/share/vm/classfile/classFileParser.cpp ! src/share/vm/classfile/javaClasses.cpp ! src/share/vm/classfile/javaClasses.hpp ! src/share/vm/gc_implementation/includeDB_gc_shared ! src/share/vm/oops/arrayOop.hpp ! src/share/vm/oops/instanceKlass.hpp ! src/share/vm/oops/instanceKlassKlass.cpp ! src/share/vm/oops/instanceOop.hpp Changeset: aaa1137c5ef4 Author: sgoldman Date: 2008-05-28 12:42 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/aaa1137c5ef4 6707485: bytecodeInterpreterWithChecks.xsl is malformed Summary: xsl output tag not at top level Reviewed-by: never, kvn, rasbold Contributed-by: gnu_andrew at member.fsf.org ! src/share/vm/interpreter/bytecodeInterpreterWithChecks.xml ! src/share/vm/interpreter/bytecodeInterpreterWithChecks.xsl Changeset: feeb96a45707 Author: coleenp Date: 2008-05-28 21:06 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/feeb96a45707 6696264: assert("narrow oop can never be zero") for GCBasher & ParNewGC Summary: decouple set_klass() with zeroing the gap when compressed. Reviewed-by: kvn, ysr, jrose ! src/cpu/sparc/vm/assembler_sparc.cpp ! src/cpu/sparc/vm/assembler_sparc.hpp ! src/cpu/sparc/vm/templateTable_sparc.cpp ! src/cpu/x86/vm/assembler_x86_64.cpp ! src/cpu/x86/vm/assembler_x86_64.hpp ! src/cpu/x86/vm/templateTable_x86_64.cpp ! src/share/vm/gc_implementation/parNew/parNewGeneration.cpp ! src/share/vm/gc_interface/collectedHeap.inline.hpp ! src/share/vm/interpreter/bytecodeInterpreter.cpp ! src/share/vm/memory/space.cpp ! src/share/vm/oops/oop.hpp ! src/share/vm/oops/oop.inline.hpp Changeset: 7793bd37a336 Author: kvn Date: 2008-05-29 12:04 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/7793bd37a336 6705887: Compressed Oops: generate x64 addressing and implicit null checks with narrow oops Summary: Generate addresses and implicit null checks with narrow oops to avoid decoding. Reviewed-by: jrose, never ! src/cpu/x86/vm/assembler_x86_32.hpp ! src/cpu/x86/vm/assembler_x86_64.hpp ! src/cpu/x86/vm/x86_64.ad ! src/os_cpu/linux_x86/vm/assembler_linux_x86_32.cpp ! src/os_cpu/linux_x86/vm/assembler_linux_x86_64.cpp ! src/os_cpu/solaris_x86/vm/assembler_solaris_x86_32.cpp ! src/os_cpu/solaris_x86/vm/assembler_solaris_x86_64.cpp ! src/os_cpu/windows_x86/vm/assembler_windows_x86_32.cpp ! src/os_cpu/windows_x86/vm/assembler_windows_x86_64.cpp ! src/share/vm/opto/callnode.hpp ! src/share/vm/opto/chaitin.cpp ! src/share/vm/opto/compile.cpp ! src/share/vm/opto/connode.cpp ! src/share/vm/opto/connode.hpp ! src/share/vm/opto/escape.cpp ! src/share/vm/opto/macro.cpp ! src/share/vm/opto/matcher.cpp ! src/share/vm/opto/matcher.hpp ! src/share/vm/opto/memnode.cpp ! src/share/vm/opto/node.hpp Changeset: 9148c65abefc Author: rasbold Date: 2008-05-29 16:22 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/9148c65abefc 6695049: (coll) Create an x86 intrinsic for Arrays.equals Summary: Intrinsify java/util/Arrays.equals(char[], char[]) Reviewed-by: kvn, never ! src/cpu/x86/vm/x86_32.ad ! src/cpu/x86/vm/x86_64.ad ! src/share/vm/classfile/vmSymbols.hpp ! src/share/vm/opto/classes.hpp ! src/share/vm/opto/lcm.cpp ! src/share/vm/opto/library_call.cpp ! src/share/vm/opto/loopnode.cpp ! src/share/vm/opto/matcher.cpp ! src/share/vm/opto/memnode.cpp ! src/share/vm/opto/memnode.hpp ! src/share/vm/runtime/arguments.cpp ! src/share/vm/runtime/globals.hpp Changeset: 02cc988a9fdc Author: rasbold Date: 2008-05-30 07:22 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/02cc988a9fdc Merge Changeset: e3d2692f8442 Author: trims Date: 2008-05-20 19:50 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/e3d2692f8442 Merge Changeset: 0e13255adcb0 Author: trims Date: 2008-05-30 14:30 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/0e13255adcb0 Merge Changeset: c14dab40ed9b Author: xdono Date: 2008-05-22 09:37 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/c14dab40ed9b Added tag jdk7-b27 for changeset e3d2692f8442 ! .hgtags Changeset: 3e4b7b5b2b4b Author: trims Date: 2008-05-30 14:31 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/3e4b7b5b2b4b Merge Changeset: 9077d695a1b0 Author: trims Date: 2008-05-30 14:50 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/9077d695a1b0 6709213: Update Build number for HS13 b02 Summary: Bump up build number to 02 Reviewed-by: jcoomes ! make/hotspot_version Changeset: 510f98a80563 Author: rasbold Date: 2008-06-03 13:14 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/510f98a80563 6709972: runThese failed with assert(false,"bad AD file") Summary: guard AryEqNode construction with has_match_rule() test, set SpecialArraysEquals default off Reviewed-by: kvn, never ! src/share/vm/opto/library_call.cpp ! src/share/vm/runtime/globals.hpp Changeset: f2759c126e9d Author: rasbold Date: 2008-06-03 15:38 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/f2759c126e9d Merge Changeset: 6b648fefb395 Author: kamg Date: 2008-05-22 13:03 -0400 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/6b648fefb395 6705523: Fix for 6695506 will violate spec when used in JDK6 Summary: Make max classfile version number dependent on JDK version Reviewed-by: acorn, never ! src/cpu/sparc/vm/sharedRuntime_sparc.cpp ! src/share/vm/classfile/classFileParser.cpp ! src/share/vm/runtime/java.hpp Changeset: 2a8ec427fbe1 Author: kamg Date: 2008-05-29 14:06 -0400 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/2a8ec427fbe1 6706604: Copyright headers need to be changed to GPL. Summary: Update the copyrights Reviewed-by: ohair ! src/share/vm/interpreter/bytecodeInterpreterWithChecks.xml ! src/share/vm/interpreter/bytecodeInterpreterWithChecks.xsl ! test/compiler/6659207/Test.java ! test/compiler/6661247/Test.java ! test/compiler/6663621/IVTest.java Changeset: 6d172e3548cb Author: coleenp Date: 2008-06-05 17:02 -0400 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/6d172e3548cb 6695819: verify_oopx rax: broken oop in decode_heap_oop Summary: Code in gen_subtype_check was encoding rax as an oop on a path where rax was not an oop. Reviewed-by: never, kvn ! src/cpu/sparc/vm/assembler_sparc.cpp ! src/cpu/x86/vm/assembler_x86_64.cpp ! src/cpu/x86/vm/interp_masm_x86_64.cpp Changeset: 1f809e010142 Author: kamg Date: 2008-06-06 13:43 -0400 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/1f809e010142 Merge ! src/cpu/sparc/vm/assembler_sparc.cpp ! src/cpu/sparc/vm/sharedRuntime_sparc.cpp ! src/cpu/x86/vm/assembler_x86_64.cpp ! src/share/vm/classfile/classFileParser.cpp ! src/share/vm/interpreter/bytecodeInterpreterWithChecks.xml ! src/share/vm/interpreter/bytecodeInterpreterWithChecks.xsl Changeset: b9ebd46331d2 Author: kvn Date: 2008-06-04 14:03 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/b9ebd46331d2 6710654: SAJDI failures with Compressed Oops Summary: Use correct offset for the java.lang.Class _klass field in SA. Reviewed-by: jrose, never ! agent/src/share/classes/sun/jvm/hotspot/oops/OopUtilities.java Changeset: 823298b11afc Author: never Date: 2008-06-04 21:56 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/823298b11afc 6709165: Tests hang or misbahve with HS 13.0-b01 on solaris-sparcv9 Reviewed-by: kvn, jrose ! src/cpu/sparc/vm/sparc.ad Changeset: 44abbb0d4c18 Author: kvn Date: 2008-06-05 13:02 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/44abbb0d4c18 6709093: Compressed Oops: reduce size of compiled methods Summary: exclude UEP size from nmethod code size and use narrow klass oop to load prototype header. Reviewed-by: jrose, never ! src/cpu/sparc/vm/sparc.ad ! src/cpu/x86/vm/assembler_x86_64.cpp ! src/cpu/x86/vm/assembler_x86_64.hpp ! src/cpu/x86/vm/x86_64.ad ! src/share/vm/ci/ciMethod.cpp ! src/share/vm/opto/compile.cpp Changeset: d4dbd9f91680 Author: never Date: 2008-06-05 15:43 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/d4dbd9f91680 6711083: 64bit JVM crashes with Internal Error (type.cpp:763) - ShouldNotReachHere() with enabled COOPs Summary: Add NarrowOop to various xmeet routines Reviewed-by: kvn, sgoldman, jrose, rasbold ! src/share/vm/opto/type.cpp Changeset: 65fe2bd88839 Author: never Date: 2008-06-05 21:44 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/65fe2bd88839 6614100: EXCEPTION_ACCESS_VIOLATION while running Eclipse with 1.6.0_05-ea Reviewed-by: kvn, jrose, rasbold ! src/share/vm/opto/cfgnode.cpp Changeset: 8759d37f2524 Author: rasbold Date: 2008-06-06 11:47 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/8759d37f2524 6711701: disable compressed oops by default Summary: comment out code that turns on compressed oops Reviewed-by: never, phh ! src/share/vm/runtime/arguments.cpp Changeset: cf1821c649d9 Author: never Date: 2008-06-06 14:34 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/cf1821c649d9 Merge ! src/cpu/x86/vm/assembler_x86_64.cpp Changeset: 790e66e5fbac Author: coleenp Date: 2008-06-09 11:51 -0400 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/790e66e5fbac 6687581: Make CMS work with compressed oops Summary: Make FreeChunk read markword instead of LSB in _klass pointer to indicate that it's a FreeChunk for compressed oops. Reviewed-by: ysr, jmasa ! agent/src/share/classes/sun/jvm/hotspot/memory/CompactibleFreeListSpace.java ! agent/src/share/classes/sun/jvm/hotspot/memory/FreeChunk.java ! agent/src/share/classes/sun/jvm/hotspot/oops/Mark.java ! src/share/vm/gc_implementation/concurrentMarkSweep/compactibleFreeListSpace.cpp ! src/share/vm/gc_implementation/concurrentMarkSweep/concurrentMarkSweepGeneration.cpp ! src/share/vm/gc_implementation/concurrentMarkSweep/freeBlockDictionary.hpp ! src/share/vm/gc_implementation/concurrentMarkSweep/freeChunk.cpp + src/share/vm/gc_implementation/concurrentMarkSweep/freeChunk.hpp ! src/share/vm/gc_implementation/concurrentMarkSweep/vmStructs_cms.hpp ! src/share/vm/gc_implementation/includeDB_gc_concurrentMarkSweep ! src/share/vm/oops/markOop.hpp ! src/share/vm/runtime/arguments.cpp ! src/share/vm/runtime/vmStructs.cpp Changeset: c0ecab83e6f3 Author: never Date: 2008-06-10 09:57 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/c0ecab83e6f3 Merge ! agent/src/share/classes/sun/jvm/hotspot/memory/CompactibleFreeListSpace.java ! src/share/vm/gc_implementation/concurrentMarkSweep/vmStructs_cms.hpp ! src/share/vm/runtime/vmStructs.cpp Changeset: 3ad4bacbcdbe Author: jcoomes Date: 2008-06-10 11:14 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/3ad4bacbcdbe Merge From jon.masamitsu at sun.com Thu Jun 12 23:38:44 2008 From: jon.masamitsu at sun.com (jon.masamitsu at sun.com) Date: Thu, 12 Jun 2008 23:38:44 +0000 Subject: hg: jdk7/hotspot-gc/hotspot: 2 new changesets Message-ID: <20080612233849.BCE6F28A0B@hg.openjdk.java.net> Changeset: 35ca13d63fe8 Author: jmasa Date: 2008-06-10 07:26 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/35ca13d63fe8 6688799: Second fix for Guarantee failure "Unexpected dirty card found" Summary: Expand cardtable without committing over existing regions. Reviewed-by: apetrusenko ! src/share/vm/memory/cardTableModRefBS.cpp Changeset: 4fab66d72627 Author: jmasa Date: 2008-06-12 14:23 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/4fab66d72627 Merge From aaisinzon at guidewire.com Thu Jun 19 18:30:33 2008 From: aaisinzon at guidewire.com (Alex Aisinzon) Date: Thu, 19 Jun 2008 11:30:33 -0700 Subject: JVM options to get full visibility on generation behavior and GC log analysis tools. Message-ID: <545E8962B7529546962672A564039F990DB06F18@exchange.guidewire.com> All I spend much time analyzing the behavior of the various generations. Their optimal sizing provides good performance improvements. I have several questions: * I add the following parameters to get good GC logs: "-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime". I wonder if some additional options are useful. By example, the timestamps only track time since the JVM was started. To correlate an incident with application time stamps, it would be much more useful to have those time stamps use regular dates (2008-06-17 11:58:11,23 by example). * I have not found a good tool to analyze these logs and have to do that analysis by hand. HPJTune (HP) and GCViewer (TagTraum) are sometimes good enough but does not work when using more refined GC options like Parallel New Garbage Collection.. PrintGCStats is a last one that I have not found very useful either. Are there other tools available that someone could recommend? In comparison, the IBM JDK provides a single GC output format by using the verbose:gc option. IBM provides some very good tools to analyze those logs. Thanks in advance for helping tuning the Sun JDK as well as possible. This is for Sun JDK 1.5. Alex Aisinzon -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From john.coomes at sun.com Thu Jun 19 21:52:43 2008 From: john.coomes at sun.com (john.coomes at sun.com) Date: Thu, 19 Jun 2008 21:52:43 +0000 Subject: hg: jdk7/hotspot-gc/hotspot: 6676016: ParallelOldGC leaks memory Message-ID: <20080619215246.928A828EF8@hg.openjdk.java.net> Changeset: 05712c37c828 Author: jcoomes Date: 2008-06-18 18:36 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/05712c37c828 6676016: ParallelOldGC leaks memory Summary: ensure that GCTask threads release resource and handle memory Reviewed-by: jmasa, chrisphi ! src/share/vm/gc_implementation/parallelScavenge/psParallelCompact.cpp From john.coomes at sun.com Sat Jun 21 05:39:56 2008 From: john.coomes at sun.com (john.coomes at sun.com) Date: Sat, 21 Jun 2008 05:39:56 +0000 Subject: hg: jdk7/hotspot-gc/hotspot: 15 new changesets Message-ID: <20080621054024.01B5F280AC@hg.openjdk.java.net> Changeset: 6d13fcb3663f Author: kvn Date: 2008-06-13 14:49 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/6d13fcb3663f 6714404: Add UseStringCache switch to enable String caching under AggressiveOpts Summary: Poke String.stringCacheEnabled during vm initialization Reviewed-by: never ! src/share/vm/classfile/vmSymbols.hpp ! src/share/vm/runtime/globals.hpp ! src/share/vm/runtime/thread.cpp Changeset: 44a553b2809d Author: kvn Date: 2008-06-13 15:08 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/44a553b2809d 6714406: Node::dominates() does not always check for TOP Summary: Add missed checks for TOP and missed checks for non-dominating cases Reviewed-by: rasbold, jrose, never ! src/share/vm/opto/memnode.cpp ! src/share/vm/opto/node.cpp Changeset: abe7181cbe8a Author: xdono Date: 2008-06-10 10:22 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/abe7181cbe8a Added tag jdk7-b28 for changeset c14dab40ed9b ! .hgtags Changeset: 4f91c08b3e44 Author: trims Date: 2008-06-17 15:27 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/4f91c08b3e44 Merge Changeset: 6470a2a42f92 Author: xlu Date: 2008-06-10 16:39 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/6470a2a42f92 6647068: libjvm.so is not built PIC Summary: Remove the preferred base address built with libjvm.so and unified the PIC approach on Solaris & Linux. Reviewed-by: never, kamg Contributed-by: xiaobin.lu at sun.com ! make/defs.make ! make/linux/makefiles/gcc.make ! make/linux/makefiles/rules.make ! make/solaris/makefiles/rules.make Changeset: f139919897d2 Author: xlu Date: 2008-06-17 09:59 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/f139919897d2 6681796: hotspot build failure on gcc 4.2.x (ubuntu 8.04) w/ openjdk 6 Summary: Fixed the cast from char* to const char* and tried to use const char* as much as possible Reviewed-by: never, kamg Contributed-by: xiaobin.lu at sun.com ! src/os/linux/vm/jvm_linux.cpp ! src/os/linux/vm/os_linux.cpp ! src/os/linux/vm/os_linux.hpp Changeset: 66cdb30329a6 Author: blacklion Date: 2008-06-19 06:02 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/66cdb30329a6 Merge Changeset: 86658812ca35 Author: ohair Date: 2008-06-13 14:52 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/86658812ca35 6714043: Minor fix to SS12 makefile changes Reviewed-by: sspitsyn ! make/solaris/makefiles/jvmg.make Changeset: 81675a0d3c75 Author: ohair Date: 2008-06-13 14:53 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/81675a0d3c75 Merge Changeset: 05d7b04bda37 Author: dcubed Date: 2008-06-14 21:53 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/05d7b04bda37 Merge Changeset: d6c28083ace6 Author: dcubed Date: 2008-06-16 13:38 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/d6c28083ace6 Merge Changeset: dfedd0e7fa9c Author: dcubed Date: 2008-06-17 09:33 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/dfedd0e7fa9c Merge Changeset: f20b3fcbb16f Author: dcubed Date: 2008-06-19 09:59 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/f20b3fcbb16f Merge Changeset: e619218327a7 Author: trims Date: 2008-06-20 11:47 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/e619218327a7 6717462: Update build number for HS14 B01 Summary: Change Hotspot version and build number for opening of HS14 Reviewed-by: jcoomes ! make/hotspot_version Changeset: 337e0e51cd6b Author: jcoomes Date: 2008-06-20 13:59 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/337e0e51cd6b Merge From rainer.jung at kippdata.de Thu Jun 19 20:19:00 2008 From: rainer.jung at kippdata.de (Rainer Jung) Date: Thu, 19 Jun 2008 22:19:00 +0200 Subject: JVM options to get full visibility on generation behavior and GC loganalysis tools. In-Reply-To: <545E8962B7529546962672A564039F990DB06F18@exchange.guidewire.com> References: <545E8962B7529546962672A564039F990DB06F18@exchange.guidewire.com> Message-ID: <485ABF34.5000303@kippdata.de> Alex Aisinzon schrieb: > I have several questions: > > * I add the following parameters to get good GC logs: > ?-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC > -XX:+PrintGCApplicationConcurrentTime > -XX:+PrintGCApplicationStoppedTime". I wonder if some additional > options are useful. By example, the timestamps only track time > since the JVM was started. To correlate an incident with > application time stamps, it would be much more useful to have > those time stamps use regular dates (2008-06-17 11:58:11,23 by > example). Also useful: -XX:+PrintTenuringDistribution Timestamps: yes unfortunately until lately there were no absolute timestamps available, only relative to JVM start. In 1.6.0_04 the new flag -XX:+PrintGCDateStamps has been implemented. See http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6517301 and http://java.sun.com/javase/6/webnotes/ReleaseNotes.html (look for the bug id). > * I have not found a good tool to analyze these logs and have to do > that analysis by hand. HPJTune (HP) and GCViewer (TagTraum) are > sometimes good enough but does not work when using more refined GC > options like Parallel New Garbage Collection.. PrintGCStats is a > last one that I have not found very useful either. Are there other > tools available that someone could recommend? The same question form here: if someone did the work of writing a parser for those human readable but not really machine usable logs which would work with various flag combinations of verbosity, GC algorithms and JVM versions that would be really great. By the way: some things are unfortunately logically missing in the log files, like HeapAtGC at the end of a CMS. You can only get the status for the memory regions when the next minor GC runs, but not at the end of a CMS :( > In comparison, the IBM JDK provides a single GC output format by using > the verbose:gc option. IBM provides some very good tools to analyze > those logs. Regards, Rainer _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From Jon.Masamitsu at Sun.COM Mon Jun 23 14:00:39 2008 From: Jon.Masamitsu at Sun.COM (Jon Masamitsu) Date: Mon, 23 Jun 2008 07:00:39 -0700 Subject: JVM options to get full visibility on generation behavior and GC loganalysis tools. In-Reply-To: <485ABF34.5000303@kippdata.de> References: <545E8962B7529546962672A564039F990DB06F18@exchange.guidewire.com> <485ABF34.5000303@kippdata.de> Message-ID: <485FAC87.1070500@Sun.COM> You can find source for a tool called GChisto at https://gchisto.dev.java.net/ It's open source. It parses PrintGCDetails output but if you turn on flags which cause printing to be interspersed with the PrintGCDetails output, it may or may not work. Regarding PrintHeapAtGC output at the end of a CMS collection, it's not there probably as a matter of the format of the output. The PrintHeapAtGC includes the "top" of a generation where "top" is the next space in a generation to be allocated. CMS is a non moving collector (i.e., it maintains its free space in free lists) so there is in general no "top". Rainer Jung wrote: > Alex Aisinzon schrieb: >> I have several questions: >> >> * I add the following parameters to get good GC logs: >> ?-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC >> -XX:+PrintGCApplicationConcurrentTime >> -XX:+PrintGCApplicationStoppedTime". I wonder if some additional >> options are useful. By example, the timestamps only track time >> since the JVM was started. To correlate an incident with >> application time stamps, it would be much more useful to have >> those time stamps use regular dates (2008-06-17 11:58:11,23 by >> example). > > Also useful: -XX:+PrintTenuringDistribution > Timestamps: yes unfortunately until lately there were no absolute > timestamps available, only relative to JVM start. In 1.6.0_04 the new > flag -XX:+PrintGCDateStamps has been implemented. See > > http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6517301 > and > http://java.sun.com/javase/6/webnotes/ReleaseNotes.html (look for the > bug id). > >> * I have not found a good tool to analyze these logs and have to do >> that analysis by hand. HPJTune (HP) and GCViewer (TagTraum) are >> sometimes good enough but does not work when using more refined GC >> options like Parallel New Garbage Collection.. PrintGCStats is a >> last one that I have not found very useful either. Are there other >> tools available that someone could recommend? > > The same question form here: if someone did the work of writing a parser > for those human readable but not really machine usable logs which would > work with various flag combinations of verbosity, GC algorithms and JVM > versions that would be really great. > > By the way: some things are unfortunately logically missing in the log > files, like HeapAtGC at the end of a CMS. You can only get the status > for the memory regions when the next minor GC runs, but not at the end > of a CMS :( > >> In comparison, the IBM JDK provides a single GC output format by using >> the verbose:gc option. IBM provides some very good tools to analyze >> those logs. > > Regards, > > Rainer > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From dragonken at gmail.com Tue Jun 24 07:59:30 2008 From: dragonken at gmail.com (Ken--@newsgroupstats.hk) Date: Tue, 24 Jun 2008 00:59:30 -0700 (PDT) Subject: Big GC with unknow reason Message-ID: <18085880.post@talk.nabble.com> First of all, I dunno is it appropriate to ask gc tuning question here. Please forgive me if I post in wrong place. Well, I was chat with Sun GC Tuning Team long time ago and seek for gc tuning advise. The team was dismissed. Today, I google Jon Masamitsu and finally get here. :> So, my question is: Any idea why there was a big gc at 51789? We are using jdk1.6.0_04 on windows32 with this set of vm option: c:\progra~1\Java\jdk1.6.0_04\bin\java -server -verbose:gc -XX:+PrintGCDetails -Xloggc:itrade_gc_gc.log -Xms1024M -Xmx1612M -XX:NewSize=128M -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:SurvivorRatio=6 -XX:CMSInitiatingOccupancyFraction=80 -XX:TargetSurvivorRatio=90 -Dcom.sun.management.jmxremote.port=18000 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false 51789.930: [Full GC 51789.930: [CMS: 403366K->126465K(917504K), 19.1967956 secs] 430862K->126465K(1032192K), [CMS Perm : 23152K->23072K(23296K)], 19.1972027 secs] [Times: user=0.94 sys=0.17, real=19.19 secs] The heap is around 4x% full at that time and the machine has 8G physical memory. From the logs, it show that it spend 19.19 secs (real), what is user/sys/real? Any info about it? I guess the vm is spending many time on memory swapping but task mgr show that the peak is around 4G. Thanks and Regards, Ken http://www.nabble.com/file/p18085880/2008-06-24_155647.gif -- View this message in context: http://www.nabble.com/Big-GC-with-unknow-reason-tp18085880p18085880.html Sent from the OpenJDK Hotspot Garbage Collection mailing list archive at Nabble.com. From Jon.Masamitsu at Sun.COM Tue Jun 24 15:38:27 2008 From: Jon.Masamitsu at Sun.COM (Jon Masamitsu) Date: Tue, 24 Jun 2008 08:38:27 -0700 Subject: Big GC with unknow reason In-Reply-To: <18085880.post@talk.nabble.com> References: <18085880.post@talk.nabble.com> Message-ID: <486114F3.6070702@Sun.COM> Your perm gen is nearly full CMS Perm : 23152K->23072K(23296K) so that might be the cause for the collection. Try increasing the initial size of the perm generation with -XX:PermSize=. Or maybe the maximum size of the perm gen with -XX:MaxPermSize=. The cms collector does not collect the perm gen by default. Add the flag -XX:+CMSClassUnloadingEnabled to turn on the collection of the perm gen. Ken-- at newsgroupstats.hk wrote: > First of all, I dunno is it appropriate to ask gc tuning question here. > Please forgive me if I post in wrong place. > > Well, I was chat with Sun GC Tuning Team long time ago and seek for gc > tuning advise. The team was dismissed. Today, I google Jon Masamitsu and > finally get here. :> > > So, my question is: Any idea why there was a big gc at 51789? > > We are using jdk1.6.0_04 on windows32 with this set of vm option: > > c:\progra~1\Java\jdk1.6.0_04\bin\java -server -verbose:gc > -XX:+PrintGCDetails -Xloggc:itrade_gc_gc.log -Xms1024M -Xmx1612M > -XX:NewSize=128M -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC > -XX:+UseParNewGC -XX:SurvivorRatio=6 -XX:CMSInitiatingOccupancyFraction=80 > -XX:TargetSurvivorRatio=90 -Dcom.sun.management.jmxremote.port=18000 > -Dcom.sun.management.jmxremote.authenticate=false > -Dcom.sun.management.jmxremote.ssl=false > > 51789.930: [Full GC 51789.930: [CMS: 403366K->126465K(917504K), 19.1967956 > secs] 430862K->126465K(1032192K), [CMS Perm : 23152K->23072K(23296K)], > 19.1972027 secs] [Times: user=0.94 sys=0.17, real=19.19 secs] > > The heap is around 4x% full at that time and the machine has 8G physical > memory. From the logs, it show that it spend 19.19 secs (real), what is > user/sys/real? Any info about it? I guess the vm is spending many time on > memory swapping but task mgr show that the peak is around 4G. See the unix command time(1) for the definitions of user, sys, and real. Not sure what it is on windows. Basically they are user = user application cpu time sys = system cpu time real = wall clock time Yes, it looks like the VM is waiting on something. > > Thanks and Regards, > Ken > > http://www.nabble.com/file/p18085880/2008-06-24_155647.gif > >