From Y.S.Ramakrishna at Sun.COM Thu Feb 7 18:24:00 2008 From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna) Date: Thu, 07 Feb 2008 10:24:00 -0800 Subject: request for review (XXS): 6659981 Message-ID: Diffs below: a 1 line change. Fixed 6659981: VMark Stability Test crash x64 Solaris 64 bit 6.0 VM When +ParallelRefProcEnabled is specified either on a single core platform or prset, or ParallelGCThreads=0 is explicitly specified, the reference processor was assuming that it would have the requisite number of parallel workers (of which there are none) to get the job done. Fix verified: yes Verification testing: specjvm98 with all collectors with +ParallelRefProcEnabled and running either or MP or 1-core platform with ParallelGCThreads set to {0,1, 2}. ------- referenceProcessor.cpp ------- *** /tmp/sccs.MraOvL Thu Feb 7 09:56:18 2008 --- referenceProcessor.cpp Thu Feb 7 09:52:34 2008 *************** *** 85,91 **** ReferenceProcessor* rp = new ReferenceProcessor(span, atomic_discovery, mt_discovery, mt_degree, ! mt_processing); if (rp == NULL) { vm_exit_during_initialization("Could not allocate ReferenceProcessor object"); } --- 85,91 ---- ReferenceProcessor* rp = new ReferenceProcessor(span, atomic_discovery, mt_discovery, mt_degree, ! mt_processing && parallel_gc_threads > 0); if (rp == NULL) { vm_exit_during_initialization("Could not allocate ReferenceProcessor object"); } From aaisinzon at guidewire.com Tue Feb 12 20:56:57 2008 From: aaisinzon at guidewire.com (Alex Aisinzon) Date: Tue, 12 Feb 2008 12:56:57 -0800 Subject: Back to back Full Garbage collections when conditions for Full GC do not seem met Message-ID: <545E8962B7529546962672A564039F990B90349A@exchange.guidewire.com> Hi all A customer of ours is experiencing instances whereby the Sun JDK 1.4.2_16 on Windows goes through many Full GCs for no good reason. The Sun JDK parameters used are: -Xms1280m -Xmx1280m -XX:NewSize=500m -XX:MaxNewSize=500m -XX:PermSize=150m -XX:MaxPermSize=150m -server -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC I have enclosed a snippet of the garbage collection logs with minor GC before and after (ProblematicFullGC.zip). >From timestamp 21928.044 to 21958.226 + 5 seconds aka 35 seconds, the JVM goes through 8 Full GCs with hardly any time in between. To the users, the JVM is frozen. The JVM needs to get restarted to get back to a good state. The most surprising is that the new generation is almost empty (1 or 2 MB max used out of 460MB) in all but the first Full GC and that there is a lot of free space in the tenured generation (more than 400MB). Our thinking is that the JVM may be triggering a Full GC because of the "Young Generation Guarantee". If such is the case, the behaviour is not consistent across platforms: we have run a similar load test in house with Sun JDK 1.4.2_16 on Linux with the identical parameters ("-server -Xms1280m -Xmx1280m -XX:NewSize=500m -XX:MaxNewSize=500m -XX:PermSize=150m -XX:MaxPermSize=150m -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCTimeStamps -verbose:gc". In that case, the Full GCs occur very normally though the tenured space is much fuller (140-180MB free only). A snippet of the GC logs is enclosed at OKGC.zip. It shows several Full GCs with a healthy dose of minor GC in between. So far, we have recommended to the customer that they use a smaller New Generation, as this may make it less likely to get closer to the "Young Generation Guarantee" threshold. Nevertheless, as the math is not there for that "Young Generation Guarantee", we want to dig further into this. Thanks in advance for any help. Alex Aisinzon -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ProblematicFullGC.zip Type: application/x-zip-compressed Size: 1187 bytes Desc: ProblematicFullGC.zip URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OKGC.zip Type: application/x-zip-compressed Size: 14900 bytes Desc: OKGC.zip URL: -------------- next part -------------- _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From Jon.Masamitsu at Sun.COM Tue Feb 12 21:22:27 2008 From: Jon.Masamitsu at Sun.COM (Jon Masamitsu) Date: Tue, 12 Feb 2008 13:22:27 -0800 Subject: Back to back Full Garbage collections when conditions for Full GC do not seem met In-Reply-To: <545E8962B7529546962672A564039F990B90349A@exchange.guidewire.com> References: <545E8962B7529546962672A564039F990B90349A@exchange.guidewire.com> Message-ID: <47B20E13.7020604@Sun.COM> Alex, The log in ProblematicFullGC.zip is from the serial GC collector which is subject to the "young generation guarantee". The log from OKGC.zip is from the parallel (UseParallelGC) collector and that collector is not subject to the "young generation guarantee". I don't see yet why the collections are starting in ProblematicFullGC.zip. Jon Alex Aisinzon wrote: > > Hi all > > A customer of ours is experiencing instances whereby the Sun JDK > 1.4.2_16 on Windows goes through many Full GCs for no good reason. > > The Sun JDK parameters used are: -Xms1280m -Xmx1280m -XX:NewSize=500m > -XX:MaxNewSize=500m -XX:PermSize=150m -XX:MaxPermSize=150m -server > -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC > > I have enclosed a snippet of the garbage collection logs with minor GC > before and after (ProblematicFullGC.zip). > > From timestamp 21928.044 to 21958.226 + 5 seconds aka 35 seconds, the > JVM goes through 8 Full GCs with hardly any time in between. > > To the users, the JVM is frozen. The JVM needs to get restarted to get > back to a good state. > > The most surprising is that the new generation is almost empty (1 or 2 > MB max used out of 460MB) in all but the first Full GC and that there > is a lot of free space in the tenured generation (more than 400MB). > > Our thinking is that the JVM may be triggering a Full GC because of > the ?Young Generation Guarantee?. > > If such is the case, the behaviour is not consistent across platforms: > we have run a similar load test in house with Sun JDK 1.4.2_16 on > Linux with the identical parameters (?-server -Xms1280m -Xmx1280m > -XX:NewSize=500m -XX:MaxNewSize=500m -XX:PermSize=150m > -XX:MaxPermSize=150m -XX:+PrintGCDetails -XX:+PrintHeapAtGC > -XX:+PrintGCTimeStamps -verbose:gc?. In that case, the Full GCs occur > very normally though the tenured space is much fuller (140-180MB free > only). A snippet of the GC logs is enclosed at OKGC.zip. It shows > several Full GCs with a healthy dose of minor GC in between. > > So far, we have recommended to the customer that they use a smaller > New Generation, as this may make it less likely to get closer to the > ?Young Generation Guarantee? threshold. Nevertheless, as the math is > not there for that ?Young Generation Guarantee?, we want to dig > further into this. > > Thanks in advance for any help. > > Alex Aisinzon > > ------------------------------------------------------------------------ > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From aaisinzon at guidewire.com Tue Feb 12 21:26:08 2008 From: aaisinzon at guidewire.com (Alex Aisinzon) Date: Tue, 12 Feb 2008 13:26:08 -0800 Subject: Back to back Full Garbage collections when conditions for Full GC do not seem met In-Reply-To: <47B20E13.7020604@Sun.COM> References: <545E8962B7529546962672A564039F990B90349A@exchange.guidewire.com> <47B20E13.7020604@Sun.COM> Message-ID: <545E8962B7529546962672A564039F990B903514@exchange.guidewire.com> Jon Thanks for the amazingly quick answer. Would you want me to gather additional information? If so, please provide me details on how to gather that additional information. One theory that I am testing is that, in the customer settings, the "-server" option was not in front. I was told that, in that case, the JVM would not pick that option. To check this further, I just restarted a perf run with the -server not in front, like the customer. Regards Alex Aisinzon -----Original Message----- From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM] Sent: Tuesday, February 12, 2008 1:22 PM To: Alex Aisinzon Cc: hotspot-gc-use at openjdk.java.net Subject: Re: Back to back Full Garbage collections when conditions for Full GC do not seem met Alex, The log in ProblematicFullGC.zip is from the serial GC collector which is subject to the "young generation guarantee". The log from OKGC.zip is from the parallel (UseParallelGC) collector and that collector is not subject to the "young generation guarantee". I don't see yet why the collections are starting in ProblematicFullGC.zip. Jon Alex Aisinzon wrote: > > Hi all > > A customer of ours is experiencing instances whereby the Sun JDK > 1.4.2_16 on Windows goes through many Full GCs for no good reason. > > The Sun JDK parameters used are: -Xms1280m -Xmx1280m -XX:NewSize=500m > -XX:MaxNewSize=500m -XX:PermSize=150m -XX:MaxPermSize=150m -server > -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC > > I have enclosed a snippet of the garbage collection logs with minor GC > before and after (ProblematicFullGC.zip). > > From timestamp 21928.044 to 21958.226 + 5 seconds aka 35 seconds, the > JVM goes through 8 Full GCs with hardly any time in between. > > To the users, the JVM is frozen. The JVM needs to get restarted to get > back to a good state. > > The most surprising is that the new generation is almost empty (1 or 2 > MB max used out of 460MB) in all but the first Full GC and that there > is a lot of free space in the tenured generation (more than 400MB). > > Our thinking is that the JVM may be triggering a Full GC because of > the "Young Generation Guarantee". > > If such is the case, the behaviour is not consistent across platforms: > we have run a similar load test in house with Sun JDK 1.4.2_16 on > Linux with the identical parameters ("-server -Xms1280m -Xmx1280m > -XX:NewSize=500m -XX:MaxNewSize=500m -XX:PermSize=150m > -XX:MaxPermSize=150m -XX:+PrintGCDetails -XX:+PrintHeapAtGC > -XX:+PrintGCTimeStamps -verbose:gc". In that case, the Full GCs occur > very normally though the tenured space is much fuller (140-180MB free > only). A snippet of the GC logs is enclosed at OKGC.zip. It shows > several Full GCs with a healthy dose of minor GC in between. > > So far, we have recommended to the customer that they use a smaller > New Generation, as this may make it less likely to get closer to the > "Young Generation Guarantee" threshold. Nevertheless, as the math is > not there for that "Young Generation Guarantee", we want to dig > further into this. > > Thanks in advance for any help. > > Alex Aisinzon > > ------------------------------------------------------------------------ > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From aaisinzon at guidewire.com Tue Feb 12 21:50:09 2008 From: aaisinzon at guidewire.com (Alex Aisinzon) Date: Tue, 12 Feb 2008 13:50:09 -0800 Subject: Back to back Full Garbage collections when conditions for Full GC do not seem met References: <545E8962B7529546962672A564039F990B90349A@exchange.guidewire.com> <47B20E13.7020604@Sun.COM> Message-ID: <545E8962B7529546962672A564039F990B903577@exchange.guidewire.com> Jon What you said about the different collectors used may actually be the most enlightening piece. The JVM options corresponding to the ProblematicFullGC.zip logs do not have "-server" as the first parameter. Therefore, we have reasons to believe that, in that case, the JVM is running in client mode. The JVM options corresponding to the OKGC.zip used the "-server" option as the first parameter. Is Serial GC the default for the client VM and Parallel GC the default for the server VM? In that light, we would ask our customer to use the -server option properly aka as a first parameter. Our application is actually certified on Server mode only. Thanks for letting us know. Regards Alex A -----Original Message----- From: Alex Aisinzon Sent: Tuesday, February 12, 2008 1:26 PM To: 'Jon.Masamitsu at Sun.COM' Cc: hotspot-gc-use at openjdk.java.net Subject: RE: Back to back Full Garbage collections when conditions for Full GC do not seem met Jon Thanks for the amazingly quick answer. Would you want me to gather additional information? If so, please provide me details on how to gather that additional information. One theory that I am testing is that, in the customer settings, the "-server" option was not in front. I was told that, in that case, the JVM would not pick that option. To check this further, I just restarted a perf run with the -server not in front, like the customer. Regards Alex Aisinzon -----Original Message----- From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM] Sent: Tuesday, February 12, 2008 1:22 PM To: Alex Aisinzon Cc: hotspot-gc-use at openjdk.java.net Subject: Re: Back to back Full Garbage collections when conditions for Full GC do not seem met Alex, The log in ProblematicFullGC.zip is from the serial GC collector which is subject to the "young generation guarantee". The log from OKGC.zip is from the parallel (UseParallelGC) collector and that collector is not subject to the "young generation guarantee". I don't see yet why the collections are starting in ProblematicFullGC.zip. Jon Alex Aisinzon wrote: > > Hi all > > A customer of ours is experiencing instances whereby the Sun JDK > 1.4.2_16 on Windows goes through many Full GCs for no good reason. > > The Sun JDK parameters used are: -Xms1280m -Xmx1280m -XX:NewSize=500m > -XX:MaxNewSize=500m -XX:PermSize=150m -XX:MaxPermSize=150m -server > -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC > > I have enclosed a snippet of the garbage collection logs with minor GC > before and after (ProblematicFullGC.zip). > > From timestamp 21928.044 to 21958.226 + 5 seconds aka 35 seconds, the > JVM goes through 8 Full GCs with hardly any time in between. > > To the users, the JVM is frozen. The JVM needs to get restarted to get > back to a good state. > > The most surprising is that the new generation is almost empty (1 or 2 > MB max used out of 460MB) in all but the first Full GC and that there > is a lot of free space in the tenured generation (more than 400MB). > > Our thinking is that the JVM may be triggering a Full GC because of > the "Young Generation Guarantee". > > If such is the case, the behaviour is not consistent across platforms: > we have run a similar load test in house with Sun JDK 1.4.2_16 on > Linux with the identical parameters ("-server -Xms1280m -Xmx1280m > -XX:NewSize=500m -XX:MaxNewSize=500m -XX:PermSize=150m > -XX:MaxPermSize=150m -XX:+PrintGCDetails -XX:+PrintHeapAtGC > -XX:+PrintGCTimeStamps -verbose:gc". In that case, the Full GCs occur > very normally though the tenured space is much fuller (140-180MB free > only). A snippet of the GC logs is enclosed at OKGC.zip. It shows > several Full GCs with a healthy dose of minor GC in between. > > So far, we have recommended to the customer that they use a smaller > New Generation, as this may make it less likely to get closer to the > "Young Generation Guarantee" threshold. Nevertheless, as the math is > not there for that "Young Generation Guarantee", we want to dig > further into this. > > Thanks in advance for any help. > > Alex Aisinzon > > ------------------------------------------------------------------------ > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From Jon.Masamitsu at Sun.COM Tue Feb 12 22:04:22 2008 From: Jon.Masamitsu at Sun.COM (Jon Masamitsu) Date: Tue, 12 Feb 2008 14:04:22 -0800 Subject: Back to back Full Garbage collections when conditions for Full GC do not seem met In-Reply-To: <545E8962B7529546962672A564039F990B903577@exchange.guidewire.com> References: <545E8962B7529546962672A564039F990B90349A@exchange.guidewire.com> <47B20E13.7020604@Sun.COM> <545E8962B7529546962672A564039F990B903577@exchange.guidewire.com> Message-ID: <47B217E6.5000605@Sun.COM> Alex Aisinzon wrote: > Jon > > What you said about the different collectors used may actually be the > most enlightening piece. > The JVM options corresponding to the ProblematicFullGC.zip logs do not > have "-server" as the first parameter. Therefore, we have reasons to > believe that, in that case, the JVM is running in client mode. > The JVM options corresponding to the OKGC.zip used the "-server" option > as the first parameter. > Is Serial GC the default for the client VM and Parallel GC the default > for the server VM? > The serial GC is definitely the default collector for the client VM. I'm a little puzzle how the UseParallelGC collector is being in the OKGC log. I don't think it is used by default. It's an older release so something might have changed since I worked with it but I thought that UseParallelGC would explicitly have to be turned on for it to run. Does the customer have AggressiveHeap turned on? That would turn on UseParallelGC. > In that light, we would ask our customer to use the -server option > properly aka as a first parameter. > Our application is actually certified on Server mode only. > > Thanks for letting us know. > > Regards > > Alex A > > > > -----Original Message----- > From: Alex Aisinzon > Sent: Tuesday, February 12, 2008 1:26 PM > To: 'Jon.Masamitsu at Sun.COM' > Cc: hotspot-gc-use at openjdk.java.net > Subject: RE: Back to back Full Garbage collections when conditions for > Full GC do not seem met > > Jon > > Thanks for the amazingly quick answer. > Would you want me to gather additional information? If so, please > provide me details on how to gather that additional information. > One theory that I am testing is that, in the customer settings, the > "-server" option was not in front. > I was told that, in that case, the JVM would not pick that option. > To check this further, I just restarted a perf run with the -server not > in front, like the customer. > > Regards > > Alex Aisinzon > > -----Original Message----- > From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM] > Sent: Tuesday, February 12, 2008 1:22 PM > To: Alex Aisinzon > Cc: hotspot-gc-use at openjdk.java.net > Subject: Re: Back to back Full Garbage collections when conditions for > Full GC do not seem met > > Alex, > > The log in ProblematicFullGC.zip is from the serial GC collector which > is subject to the "young generation guarantee". The log from OKGC.zip > is from the parallel (UseParallelGC) collector and that collector is > not subject to the "young generation guarantee". > > I don't see yet why the collections are starting in > ProblematicFullGC.zip. > > Jon > > Alex Aisinzon wrote: > >> Hi all >> >> A customer of ours is experiencing instances whereby the Sun JDK >> 1.4.2_16 on Windows goes through many Full GCs for no good reason. >> >> The Sun JDK parameters used are: -Xms1280m -Xmx1280m -XX:NewSize=500m >> -XX:MaxNewSize=500m -XX:PermSize=150m -XX:MaxPermSize=150m -server >> -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC >> >> I have enclosed a snippet of the garbage collection logs with minor GC >> > > >> before and after (ProblematicFullGC.zip). >> >> From timestamp 21928.044 to 21958.226 + 5 seconds aka 35 seconds, the >> JVM goes through 8 Full GCs with hardly any time in between. >> >> To the users, the JVM is frozen. The JVM needs to get restarted to get >> > > >> back to a good state. >> >> The most surprising is that the new generation is almost empty (1 or 2 >> > > >> MB max used out of 460MB) in all but the first Full GC and that there >> is a lot of free space in the tenured generation (more than 400MB). >> >> Our thinking is that the JVM may be triggering a Full GC because of >> the "Young Generation Guarantee". >> >> If such is the case, the behaviour is not consistent across platforms: >> > > >> we have run a similar load test in house with Sun JDK 1.4.2_16 on >> Linux with the identical parameters ("-server -Xms1280m -Xmx1280m >> -XX:NewSize=500m -XX:MaxNewSize=500m -XX:PermSize=150m >> -XX:MaxPermSize=150m -XX:+PrintGCDetails -XX:+PrintHeapAtGC >> -XX:+PrintGCTimeStamps -verbose:gc". In that case, the Full GCs occur >> very normally though the tenured space is much fuller (140-180MB free >> only). A snippet of the GC logs is enclosed at OKGC.zip. It shows >> several Full GCs with a healthy dose of minor GC in between. >> >> So far, we have recommended to the customer that they use a smaller >> New Generation, as this may make it less likely to get closer to the >> "Young Generation Guarantee" threshold. Nevertheless, as the math is >> not there for that "Young Generation Guarantee", we want to dig >> further into this. >> >> Thanks in advance for any help. >> >> Alex Aisinzon >> >> >> > ------------------------------------------------------------------------ > >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From aaisinzon at guidewire.com Wed Feb 13 00:15:41 2008 From: aaisinzon at guidewire.com (Alex Aisinzon) Date: Tue, 12 Feb 2008 16:15:41 -0800 Subject: Back to back Full Garbage collections when conditions for Full GC do not seem met In-Reply-To: <47B217E6.5000605@Sun.COM> References: <545E8962B7529546962672A564039F990B90349A@exchange.guidewire.com> <47B20E13.7020604@Sun.COM> <545E8962B7529546962672A564039F990B903577@exchange.guidewire.com> <47B217E6.5000605@Sun.COM> Message-ID: <545E8962B7529546962672A564039F990B903874@exchange.guidewire.com> Jon The theory that the customer was using the client mode seems not to be true. We realize that some nodes were using client and some other were using server but that the excessive Full GC seems to happen also on the JVMs using the server mode. The point about the usage of different GC policies is interesting though: Could you share with me the GC log pattern to GC policy mapping? In other words, as you indicate in your blog (http://blogs.sun.com/jonthecollector/), there are 3 young generation collectors and 3 tenured generation ones. When I look at some GC logs, how can I deduce which you and tenured generation collectors are used? Thanks again for the tremendous help provided here. Regards Alex A -----Original Message----- From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM] Sent: Tuesday, February 12, 2008 2:04 PM To: Alex Aisinzon Cc: hotspot-gc-use at openjdk.java.net Subject: Re: Back to back Full Garbage collections when conditions for Full GCdo not seem met Alex Aisinzon wrote: > Jon > > What you said about the different collectors used may actually be the > most enlightening piece. > The JVM options corresponding to the ProblematicFullGC.zip logs do not > have "-server" as the first parameter. Therefore, we have reasons to > believe that, in that case, the JVM is running in client mode. > The JVM options corresponding to the OKGC.zip used the "-server" option > as the first parameter. > Is Serial GC the default for the client VM and Parallel GC the default > for the server VM? > The serial GC is definitely the default collector for the client VM. I'm a little puzzle how the UseParallelGC collector is being in the OKGC log. I don't think it is used by default. It's an older release so something might have changed since I worked with it but I thought that UseParallelGC would explicitly have to be turned on for it to run. Does the customer have AggressiveHeap turned on? That would turn on UseParallelGC. > In that light, we would ask our customer to use the -server option > properly aka as a first parameter. > Our application is actually certified on Server mode only. > > Thanks for letting us know. > > Regards > > Alex A > > > > -----Original Message----- > From: Alex Aisinzon > Sent: Tuesday, February 12, 2008 1:26 PM > To: 'Jon.Masamitsu at Sun.COM' > Cc: hotspot-gc-use at openjdk.java.net > Subject: RE: Back to back Full Garbage collections when conditions for > Full GC do not seem met > > Jon > > Thanks for the amazingly quick answer. > Would you want me to gather additional information? If so, please > provide me details on how to gather that additional information. > One theory that I am testing is that, in the customer settings, the > "-server" option was not in front. > I was told that, in that case, the JVM would not pick that option. > To check this further, I just restarted a perf run with the -server not > in front, like the customer. > > Regards > > Alex Aisinzon > > -----Original Message----- > From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM] > Sent: Tuesday, February 12, 2008 1:22 PM > To: Alex Aisinzon > Cc: hotspot-gc-use at openjdk.java.net > Subject: Re: Back to back Full Garbage collections when conditions for > Full GC do not seem met > > Alex, > > The log in ProblematicFullGC.zip is from the serial GC collector which > is subject to the "young generation guarantee". The log from OKGC.zip > is from the parallel (UseParallelGC) collector and that collector is > not subject to the "young generation guarantee". > > I don't see yet why the collections are starting in > ProblematicFullGC.zip. > > Jon > > Alex Aisinzon wrote: > >> Hi all >> >> A customer of ours is experiencing instances whereby the Sun JDK >> 1.4.2_16 on Windows goes through many Full GCs for no good reason. >> >> The Sun JDK parameters used are: -Xms1280m -Xmx1280m -XX:NewSize=500m >> -XX:MaxNewSize=500m -XX:PermSize=150m -XX:MaxPermSize=150m -server >> -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC >> >> I have enclosed a snippet of the garbage collection logs with minor GC >> > > >> before and after (ProblematicFullGC.zip). >> >> From timestamp 21928.044 to 21958.226 + 5 seconds aka 35 seconds, the >> JVM goes through 8 Full GCs with hardly any time in between. >> >> To the users, the JVM is frozen. The JVM needs to get restarted to get >> > > >> back to a good state. >> >> The most surprising is that the new generation is almost empty (1 or 2 >> > > >> MB max used out of 460MB) in all but the first Full GC and that there >> is a lot of free space in the tenured generation (more than 400MB). >> >> Our thinking is that the JVM may be triggering a Full GC because of >> the "Young Generation Guarantee". >> >> If such is the case, the behaviour is not consistent across platforms: >> > > >> we have run a similar load test in house with Sun JDK 1.4.2_16 on >> Linux with the identical parameters ("-server -Xms1280m -Xmx1280m >> -XX:NewSize=500m -XX:MaxNewSize=500m -XX:PermSize=150m >> -XX:MaxPermSize=150m -XX:+PrintGCDetails -XX:+PrintHeapAtGC >> -XX:+PrintGCTimeStamps -verbose:gc". In that case, the Full GCs occur >> very normally though the tenured space is much fuller (140-180MB free >> only). A snippet of the GC logs is enclosed at OKGC.zip. It shows >> several Full GCs with a healthy dose of minor GC in between. >> >> So far, we have recommended to the customer that they use a smaller >> New Generation, as this may make it less likely to get closer to the >> "Young Generation Guarantee" threshold. Nevertheless, as the math is >> not there for that "Young Generation Guarantee", we want to dig >> further into this. >> >> Thanks in advance for any help. >> >> Alex Aisinzon >> >> >> > ------------------------------------------------------------------------ > >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From Jon.Masamitsu at Sun.COM Wed Feb 13 14:19:24 2008 From: Jon.Masamitsu at Sun.COM (Jon Masamitsu) Date: Wed, 13 Feb 2008 06:19:24 -0800 Subject: Back to back Full Garbage collections when conditions for Full GC do not seem met In-Reply-To: <545E8962B7529546962672A564039F990B903874@exchange.guidewire.com> References: <545E8962B7529546962672A564039F990B90349A@exchange.guidewire.com> <47B20E13.7020604@Sun.COM> <545E8962B7529546962672A564039F990B903577@exchange.guidewire.com> <47B217E6.5000605@Sun.COM> <545E8962B7529546962672A564039F990B903874@exchange.guidewire.com> Message-ID: <47B2FC6C.30009@Sun.COM> RESEND: Sorry if this is a duplicate. I got a warning that the original message could not be sent. Alex, Much of this information is in http://java.sun.com/docs/hotspot/gc1.4.2/ In your logs "def new generation" implies the serial collector of the young generation "tenured generation" implies the serial collector of the tenured generation "PSYoungGen" implies the parallel young generation collector for UseParallelGC "PSOldGen" implies the serial collector for the tenured generation (basically the same as "tenured generation" UseParallelOldGC is not implemented in 1.4.2 so you won't see it. Comments below. Alex Aisinzon wrote: > Jon > > The theory that the customer was using the client mode seems not to be > true. > We realize that some nodes were using client and some other were using > server but that the excessive Full GC seems to happen also on the JVMs > using the server mode. > It would be useful to know if the excessive full GC's are occuring with the UseParallelGC collector ("PSYoungGen" and "PSOldGen" in the logs). I also think (as you have suggested) that the size of the young gen (500m) is too large for the size of the tenured generation and the amount of live data in the tenured generation (the amount of free space is approaching the size of the young gen). Do try a smaller young gen or larger overall heap. It looks like only full collections are being attempted. I don't recall anything that would cause this type of behavior. If you have a Sun support contract, request some help on this. The support organization is more familar with the current behavior of the earlier releases (i.e., what bugs have been fixed and what enhancements have been made) and might be able to help. > The point about the usage of different GC policies is interesting > though: > Could you share with me the GC log pattern to GC policy mapping? > In other words, as you indicate in your blog > (http://blogs.sun.com/jonthecollector/), there are 3 young generation > collectors and 3 tenured generation ones. > When I look at some GC logs, how can I deduce which you and tenured > generation collectors are used? > > Thanks again for the tremendous help provided here. > > Regards > > Alex A > > > _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From Jon.Masamitsu at sun.com Wed Feb 13 05:16:44 2008 From: Jon.Masamitsu at sun.com (Jon Masamitsu) Date: Tue, 12 Feb 2008 21:16:44 -0800 Subject: Back to back Full Garbage collections when conditions for Full GC do not seem met In-Reply-To: <545E8962B7529546962672A564039F990B903874@exchange.guidewire.com> References: <545E8962B7529546962672A564039F990B90349A@exchange.guidewire.com> <47B20E13.7020604@Sun.COM> <545E8962B7529546962672A564039F990B903577@exchange.guidewire.com> <47B217E6.5000605@Sun.COM> <545E8962B7529546962672A564039F990B903874@exchange.guidewire.com> Message-ID: <47B27D3C.7060508@sun.com> Alex, Much of this information is in http://java.sun.com/docs/hotspot/gc1.4.2/ In your logs "def new generation" implies the serial collector of the young generation "tenured generation" implies the serial collector of the tenured generation "PSYoungGen" implies the parallel young generation collector for UseParallelGC "PSOldGen" implies the serial collector for the tenured generation (basically the same as "tenured generation" UseParallelOldGC is not implemented in 1.4.2 so you won't see it. Comments below. Alex Aisinzon wrote On 02/12/08 16:15,: >Jon > >The theory that the customer was using the client mode seems not to be >true. >We realize that some nodes were using client and some other were using >server but that the excessive Full GC seems to happen also on the JVMs >using the server mode. > > It would be useful to know if the excessive full GC's are occuring with the UseParallelGC collector ("PSYoungGen" and "PSOldGen" in the logs). I also think (as you have suggested) that the size of the young gen (500m) is too large for the size of the tenured generation and the amount of live data in the tenured generation (the amount of free space is approaching the size of the young gen). Do try a smaller young gen or larger overall heap. It looks like only full collections are being attempted. I don't recall anything that would cause this type of behavior. If you have a Sun support contract, request some help on this. The support organization is more familar with the current behavior of the earlier releases (i.e., what bugs have been fixed and what enhancements have been made) and might be able to help. >The point about the usage of different GC policies is interesting >though: >Could you share with me the GC log pattern to GC policy mapping? >In other words, as you indicate in your blog >(http://blogs.sun.com/jonthecollector/), there are 3 young generation >collectors and 3 tenured generation ones. >When I look at some GC logs, how can I deduce which you and tenured >generation collectors are used? > >Thanks again for the tremendous help provided here. > >Regards > >Alex A > >-----Original Message----- >From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM] >Sent: Tuesday, February 12, 2008 2:04 PM >To: Alex Aisinzon >Cc: hotspot-gc-use at openjdk.java.net >Subject: Re: Back to back Full Garbage collections when conditions for >Full GCdo not seem met > > > >Alex Aisinzon wrote: > > >>Jon >> >>What you said about the different collectors used may actually be the >>most enlightening piece. >>The JVM options corresponding to the ProblematicFullGC.zip logs do not >>have "-server" as the first parameter. Therefore, we have reasons to >>believe that, in that case, the JVM is running in client mode. >>The JVM options corresponding to the OKGC.zip used the "-server" >> >> >option > > >>as the first parameter. >>Is Serial GC the default for the client VM and Parallel GC the default >>for the server VM? >> >> >> > >The serial GC is definitely the default collector for the client VM. >I'm a little puzzle how >the UseParallelGC collector is being in the OKGC log. I don't think it >is used by default. >It's an older release so something might have changed since I worked >with it but I thought >that UseParallelGC would explicitly have to be turned on for it to run. > >Does the customer >have AggressiveHeap turned on? That would turn on UseParallelGC. > > > > >>In that light, we would ask our customer to use the -server option >>properly aka as a first parameter. >>Our application is actually certified on Server mode only. >> >>Thanks for letting us know. >> >>Regards >> >>Alex A >> >> >> >>-----Original Message----- >>From: Alex Aisinzon >>Sent: Tuesday, February 12, 2008 1:26 PM >>To: 'Jon.Masamitsu at Sun.COM' >>Cc: hotspot-gc-use at openjdk.java.net >>Subject: RE: Back to back Full Garbage collections when conditions for >>Full GC do not seem met >> >>Jon >> >>Thanks for the amazingly quick answer. >>Would you want me to gather additional information? If so, please >>provide me details on how to gather that additional information. >>One theory that I am testing is that, in the customer settings, the >>"-server" option was not in front. >>I was told that, in that case, the JVM would not pick that option. >>To check this further, I just restarted a perf run with the -server >> >> >not > > >>in front, like the customer. >> >>Regards >> >>Alex Aisinzon >> >>-----Original Message----- >>From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM] >>Sent: Tuesday, February 12, 2008 1:22 PM >>To: Alex Aisinzon >>Cc: hotspot-gc-use at openjdk.java.net >>Subject: Re: Back to back Full Garbage collections when conditions for >>Full GC do not seem met >> >>Alex, >> >>The log in ProblematicFullGC.zip is from the serial GC collector which >>is subject to the "young generation guarantee". The log from OKGC.zip >>is from the parallel (UseParallelGC) collector and that collector is >>not subject to the "young generation guarantee". >> >>I don't see yet why the collections are starting in >>ProblematicFullGC.zip. >> >>Jon >> >>Alex Aisinzon wrote: >> >> >> >>>Hi all >>> >>>A customer of ours is experiencing instances whereby the Sun JDK >>>1.4.2_16 on Windows goes through many Full GCs for no good reason. >>> >>>The Sun JDK parameters used are: -Xms1280m -Xmx1280m -XX:NewSize=500m >>> >>> > > > >>>-XX:MaxNewSize=500m -XX:PermSize=150m -XX:MaxPermSize=150m -server >>>-XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC >>> >>>I have enclosed a snippet of the garbage collection logs with minor >>> >>> >GC > > >>> >>> >>> >> >> >> >>>before and after (ProblematicFullGC.zip). >>> >>>From timestamp 21928.044 to 21958.226 + 5 seconds aka 35 seconds, the >>> >>> > > > >>>JVM goes through 8 Full GCs with hardly any time in between. >>> >>>To the users, the JVM is frozen. The JVM needs to get restarted to >>> >>> >get > > >>> >>> >>> >> >> >> >>>back to a good state. >>> >>>The most surprising is that the new generation is almost empty (1 or >>> >>> >2 > > >>> >>> >>> >> >> >> >>>MB max used out of 460MB) in all but the first Full GC and that there >>> >>> > > > >>>is a lot of free space in the tenured generation (more than 400MB). >>> >>>Our thinking is that the JVM may be triggering a Full GC because of >>>the "Young Generation Guarantee". >>> >>>If such is the case, the behaviour is not consistent across >>> >>> >platforms: > > >>> >>> >>> >> >> >> >>>we have run a similar load test in house with Sun JDK 1.4.2_16 on >>>Linux with the identical parameters ("-server -Xms1280m -Xmx1280m >>>-XX:NewSize=500m -XX:MaxNewSize=500m -XX:PermSize=150m >>>-XX:MaxPermSize=150m -XX:+PrintGCDetails -XX:+PrintHeapAtGC >>>-XX:+PrintGCTimeStamps -verbose:gc". In that case, the Full GCs occur >>> >>> > > > >>>very normally though the tenured space is much fuller (140-180MB free >>> >>> > > > >>>only). A snippet of the GC logs is enclosed at OKGC.zip. It shows >>>several Full GCs with a healthy dose of minor GC in between. >>> >>>So far, we have recommended to the customer that they use a smaller >>>New Generation, as this may make it less likely to get closer to the >>>"Young Generation Guarantee" threshold. Nevertheless, as the math is >>>not there for that "Young Generation Guarantee", we want to dig >>>further into this. >>> >>>Thanks in advance for any help. >>> >>>Alex Aisinzon >>> >>> >>> >>> >>> >------------------------------------------------------------------------ > > >> >> >> >>>_______________________________________________ >>>hotspot-gc-use mailing list >>>hotspot-gc-use at openjdk.java.net >>>http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>> >>> >>> >>> >>_______________________________________________ >>hotspot-gc-use mailing list >>hotspot-gc-use at openjdk.java.net >>http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> >> >_______________________________________________ >hotspot-gc-use mailing list >hotspot-gc-use at openjdk.java.net >http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From aaisinzon at guidewire.com Wed Feb 13 19:00:54 2008 From: aaisinzon at guidewire.com (Alex Aisinzon) Date: Wed, 13 Feb 2008 11:00:54 -0800 Subject: Back to back Full Garbage collections when conditions for Full GCdo not seem met In-Reply-To: <47B2FC6C.30009@Sun.COM> References: <545E8962B7529546962672A564039F990B90349A@exchange.guidewire.com> <47B20E13.7020604@Sun.COM> <545E8962B7529546962672A564039F990B903577@exchange.guidewire.com> <47B217E6.5000605@Sun.COM> <545E8962B7529546962672A564039F990B903874@exchange.guidewire.com> <47B2FC6C.30009@Sun.COM> Message-ID: <545E8962B7529546962672A564039F990B974239@exchange.guidewire.com> Jon The first message did go through. The customer changed some JVM tunings, which led the JVM to use parallel young generation and serial collector for the tenured old. The issue still occurrs (snippet enclosed). I am going to follow your recommendation about contact Sun Support. I fully understand that you worked on JDK 1.5 and 1.6 and that you are not so familiar with 1.4.2. I am already amazed at the level of assistance provided through the mailing list. Thanks again for all the help provided. Regards Alex A -----Original Message----- From: Jon.Masamitsu at Sun.COM [mailto:Jon.Masamitsu at Sun.COM] Sent: Wednesday, February 13, 2008 6:19 AM To: Alex Aisinzon Cc: hotspot-gc-use at openjdk.java.net Subject: Re: Back to back Full Garbage collections when conditions for Full GCdo not seem met RESEND: Sorry if this is a duplicate. I got a warning that the original message could not be sent. Alex, Much of this information is in http://java.sun.com/docs/hotspot/gc1.4.2/ In your logs "def new generation" implies the serial collector of the young generation "tenured generation" implies the serial collector of the tenured generation "PSYoungGen" implies the parallel young generation collector for UseParallelGC "PSOldGen" implies the serial collector for the tenured generation (basically the same as "tenured generation" UseParallelOldGC is not implemented in 1.4.2 so you won't see it. Comments below. Alex Aisinzon wrote: > Jon > > The theory that the customer was using the client mode seems not to be > true. > We realize that some nodes were using client and some other were using > server but that the excessive Full GC seems to happen also on the JVMs > using the server mode. > It would be useful to know if the excessive full GC's are occuring with the UseParallelGC collector ("PSYoungGen" and "PSOldGen" in the logs). I also think (as you have suggested) that the size of the young gen (500m) is too large for the size of the tenured generation and the amount of live data in the tenured generation (the amount of free space is approaching the size of the young gen). Do try a smaller young gen or larger overall heap. It looks like only full collections are being attempted. I don't recall anything that would cause this type of behavior. If you have a Sun support contract, request some help on this. The support organization is more familar with the current behavior of the earlier releases (i.e., what bugs have been fixed and what enhancements have been made) and might be able to help. > The point about the usage of different GC policies is interesting > though: > Could you share with me the GC log pattern to GC policy mapping? > In other words, as you indicate in your blog > (http://blogs.sun.com/jonthecollector/), there are 3 young generation > collectors and 3 tenured generation ones. > When I look at some GC logs, how can I deduce which you and tenured > generation collectors are used? > > Thanks again for the tremendous help provided here. > > Regards > > Alex A > > > -------------- next part -------------- A non-text attachment was scrubbed... Name: ProblematicGC.zip Type: application/x-zip-compressed Size: 1168 bytes Desc: ProblematicGC.zip URL: -------------- next part -------------- _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From walters at verbum.org Thu Feb 28 14:33:05 2008 From: walters at verbum.org (Colin Walters) Date: Thu, 28 Feb 2008 09:33:05 -0500 Subject: no-administration heap? Message-ID: Hi, Essentially, my question is: Why do I have to set a maximum heap size (and PermGen space)? Practically speaking, it means I have to become a system administrator for my Eclipse installation (and other JVM apps). I kept having it abort, until I tracked it down to it hitting the maximum heap size and not being able to allocate any more. It looks like the heuristics are improved in J2SE 5 (http://java.sun.com/j2se/1.5.0/docs/guide/vm/gc-ergonomics.html), but it's still a fixed limit. At least for desktop applications such as Eclipse, what I would like is a much more user-friendly system where the application uses as much heap as it needs. From what I can tell, this is how the Boehm collector works, which is used by a number of projects like Python, Mono, etc. I understand there could be performance impact, but it makes more sense to me to have a default policy where VM options are a tuning tweak, not a required change if your application loads a large file, plugins, etc. From Jon.Masamitsu at Sun.COM Thu Feb 28 15:13:07 2008 From: Jon.Masamitsu at Sun.COM (Jon Masamitsu) Date: Thu, 28 Feb 2008 07:13:07 -0800 Subject: no-administration heap? In-Reply-To: References: Message-ID: <47C6CF83.4090801@Sun.COM> Colin, There was a discussion on this alias last July under "-XX:+MaximizeHeapSize". http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2007-July/000036.html Is that the type of thing you are thinking about? Jon Colin Walters wrote: > Hi, > > Essentially, my question is: Why do I have to set a maximum heap size > (and PermGen space)? Practically speaking, it means I have to become > a system administrator for my Eclipse installation (and other JVM > apps). I kept having it abort, until I tracked it down to it hitting > the maximum heap size and not being able to allocate any more. > > It looks like the heuristics are improved in J2SE 5 > (http://java.sun.com/j2se/1.5.0/docs/guide/vm/gc-ergonomics.html), but > it's still a fixed limit. > > At least for desktop applications such as Eclipse, what I would like > is a much more user-friendly system where the application uses as much > heap as it needs. From what I can tell, this is how the Boehm > collector works, which is used by a number of projects like Python, > Mono, etc. I understand there could be performance impact, but it > makes more sense to me to have a default policy where VM options are a > tuning tweak, not a required change if your application loads a large > file, plugins, etc. From walters at verbum.org Thu Feb 28 16:01:11 2008 From: walters at verbum.org (Colin Walters) Date: Thu, 28 Feb 2008 11:01:11 -0500 Subject: no-administration heap? In-Reply-To: <47C6CF83.4090801@Sun.COM> References: <47C6CF83.4090801@Sun.COM> Message-ID: Hi Jon, On Thu, Feb 28, 2008 at 10:13 AM, Jon Masamitsu wrote: > Colin, > > There was a discussion on this alias last July > under "-XX:+MaximizeHeapSize". Yes - that sounds like what I want (along with -XX:+CMSPermGenSweepingEnabled). Not technically unlimited, but if a JVM is using anywhere close to your maximum physical memory you're probably in trouble anyways. The question then is what are the default collection goals. What interests me the most are the defaults for the JVM on the desktop. At a high level, as free operating systems like Fedora move to integrating OpenJDK, I expect more applications will make use of it. Before, because the JVM was proprietary it required a manual installation process through a web page, and so having to follow instructions on another web page to tweak parameters such as heap size for the few JVM-based applications one installed wasn't a big deal. But having OpenJDK installed by default will remove this manual process; there should be no distinction between JVM-based applications vs not. From fw at deneb.enyo.de Thu Feb 28 21:20:14 2008 From: fw at deneb.enyo.de (Florian Weimer) Date: Thu, 28 Feb 2008 22:20:14 +0100 Subject: no-administration heap? In-Reply-To: (Colin Walters's message of "Thu, 28 Feb 2008 09:33:05 -0500") References: Message-ID: <87d4qgrd69.fsf@mid.deneb.enyo.de> * Colin Walters: > At least for desktop applications such as Eclipse, what I would like > is a much more user-friendly system where the application uses as much > heap as it needs. From what I can tell, this is how the Boehm > collector works, which is used by a number of projects like Python, > Mono, etc. The Boehm collector does not require a continuous heap segment, unlike the Hotspot collector (AFAIK). This makes incremental allocation from the operating system much easier. (Same for the Ocaml collector, BTW.) At least on Linux, the problem is that there is no documented API to reserve address space (so that no other mappings end up there) without actually allocating backing store. MAP_NORESERVE does not work as documented. From walters at verbum.org Thu Feb 28 21:58:58 2008 From: walters at verbum.org (Colin Walters) Date: Thu, 28 Feb 2008 16:58:58 -0500 Subject: no-administration heap? In-Reply-To: <87d4qgrd69.fsf@mid.deneb.enyo.de> References: <87d4qgrd69.fsf@mid.deneb.enyo.de> Message-ID: On Thu, Feb 28, 2008 at 4:20 PM, Florian Weimer wrote: > > At least on Linux, the problem is that there is no documented API to > reserve address space (so that no other mappings end up there) without > actually allocating backing store. MAP_NORESERVE does not work as > documented. Ah, I see. So this problem is essentially blocked on the lack of a sufficient/documented Linux kernel interface? Do other operating systems (Windows, Solaris) provide a good enough interface for OpenJDK? I'm assuming changing the collector to not require a continuous address space would be too invasive? From Keith.Holdaway at sas.com Thu Feb 28 22:44:32 2008 From: Keith.Holdaway at sas.com (Keith Holdaway) Date: Thu, 28 Feb 2008 17:44:32 -0500 Subject: no-administration heap? In-Reply-To: References: <87d4qgrd69.fsf@mid.deneb.enyo.de> Message-ID: <304E9E55F6A4BE4B910C2437D4D1B4960907C40874@MERCMBX14.na.sas.com> Gentlemen, I am trying to remotely connect to a running VM with visualgc. Here is the DOS command from my shell on the box containing visualgc 3.0_b07 C:\Documents and Settings\kehold>visualgc 4664 at AMDSRV4.na.sas.com 1000 Exception in thread "main" java.lang.NullPointerException at com.sun.jvmstat.tools.visualgc.MonitoredVmModel.getOsFrequency(Monito redVmModel.java:485) at com.sun.jvmstat.tools.visualgc.GCSample.initStaticCounters(GCSample.j ava:104) at com.sun.jvmstat.tools.visualgc.GCSample.(GCSample.java:129) at com.sun.jvmstat.tools.visualgc.VisualGC.main(VisualGC.java:128) I have kicked off a VM with jps - LVMID 4664 on the remote box AMDSRV4. I can run visualgc locally - I have run rmiregistry and established a jstatd.all.policy file on the remote box: grant codebase "file:${java.home}/../lib/tools.jar" { permission java.security.AllPermission; }; Any ideas? The local box is 32 bit Hotspot; the remote box a 64 bit running Hotspot. Keith R Holdaway Java Development Technologies SAS The Power to Know Carpe Diem -----Original Message----- From: hotspot-gc-dev-bounces at openjdk.java.net [mailto:hotspot-gc-dev-bounces at openjdk.java.net] On Behalf Of Colin Walters Sent: Thursday, February 28, 2008 4:59 PM To: Florian Weimer Cc: hotspot-gc-dev at openjdk.java.net Subject: Re: no-administration heap? On Thu, Feb 28, 2008 at 4:20 PM, Florian Weimer wrote: > > At least on Linux, the problem is that there is no documented API to > reserve address space (so that no other mappings end up there) without > actually allocating backing store. MAP_NORESERVE does not work as > documented. Ah, I see. So this problem is essentially blocked on the lack of a sufficient/documented Linux kernel interface? Do other operating systems (Windows, Solaris) provide a good enough interface for OpenJDK? I'm assuming changing the collector to not require a continuous address space would be too invasive? From Peter.Kessler at Sun.COM Thu Feb 28 22:46:09 2008 From: Peter.Kessler at Sun.COM (Peter B. Kessler) Date: Thu, 28 Feb 2008 14:46:09 -0800 Subject: no-administration heap? In-Reply-To: References: <87d4qgrd69.fsf@mid.deneb.enyo.de> Message-ID: <47C739B1.3010605@Sun.COM> Colin Walters wrote: > On Thu, Feb 28, 2008 at 4:20 PM, Florian Weimer wrote: >> At least on Linux, the problem is that there is no documented API to >> reserve address space (so that no other mappings end up there) without >> actually allocating backing store. MAP_NORESERVE does not work as >> documented. > > Ah, I see. So this problem is essentially blocked on the lack of a > sufficient/documented Linux kernel interface? > Do other operating systems (Windows, Solaris) provide a good enough > interface for OpenJDK? > > I'm assuming changing the collector to not require a continuous > address space would be too invasive? Nothing is "too invasive"! We have a sketch of a "chunked heap"; it's just a lot of work. Things like the card table that used to be indexed by offsets in the heap become part of each heap chunk and are indexed by an offset in the chunk. A lot of the underlying data structures have to be rewritten, but that's why they call it software. We would *love* to have that kind of heap. Then there are "details" like that once you chunk the heap you limit the maximum size of an object. I was thinking that 64MB chunks might be one possibility in that they would let you have (somewhat less than) a 64MB array, which is the largest you can have now on the client unless you use a larger -Xmx. Few people have individual objects larger than 64MB, but there are a few. Of course you'd want to allow people to specify a larger chunk size on the command line. An interesting observation (by Andrey) is that chunking doesn't make sense in a 64-bit JVM: there you just reserve enough contiguous virtual memory (but not swap space) for the amount of physical memory you have, and grow into it as you need it. That works because the virtual address space you have is guaranteed to be larger than the physical memory you have, so you don't have to worry about shutting out thread stacks, space for code, etc. There are details in there too: as in that the JVM should not expand the heap if it the total process footprint would exceed the physical memory, but that's just a matter of getting the policy right. ... peter From Keith.Holdaway at sas.com Thu Feb 28 22:47:00 2008 From: Keith.Holdaway at sas.com (Keith Holdaway) Date: Thu, 28 Feb 2008 17:47:00 -0500 Subject: no-administration heap? In-Reply-To: <304E9E55F6A4BE4B910C2437D4D1B4960907C40874@MERCMBX14.na.sas.com> References: <87d4qgrd69.fsf@mid.deneb.enyo.de> <304E9E55F6A4BE4B910C2437D4D1B4960907C40874@MERCMBX14.na.sas.com> Message-ID: <304E9E55F6A4BE4B910C2437D4D1B4960907C4087A@MERCMBX14.na.sas.com> Just discovered that I was running JDK 6.0 locally and JDK 5.0 remotely - thanks - it works now. keith Keith R Holdaway Java Development Technologies SAS The Power to Know Carpe Diem -----Original Message----- From: hotspot-gc-dev-bounces at openjdk.java.net [mailto:hotspot-gc-dev-bounces at openjdk.java.net] On Behalf Of Keith Holdaway Sent: Thursday, February 28, 2008 5:45 PM To: hotspot-gc-dev at openjdk.java.net Subject: RE: no-administration heap? Gentlemen, I am trying to remotely connect to a running VM with visualgc. Here is the DOS command from my shell on the box containing visualgc 3.0_b07 C:\Documents and Settings\kehold>visualgc 4664 at AMDSRV4.na.sas.com 1000 Exception in thread "main" java.lang.NullPointerException at com.sun.jvmstat.tools.visualgc.MonitoredVmModel.getOsFrequency(Monito redVmModel.java:485) at com.sun.jvmstat.tools.visualgc.GCSample.initStaticCounters(GCSample.j ava:104) at com.sun.jvmstat.tools.visualgc.GCSample.(GCSample.java:129) at com.sun.jvmstat.tools.visualgc.VisualGC.main(VisualGC.java:128) I have kicked off a VM with jps - LVMID 4664 on the remote box AMDSRV4. I can run visualgc locally - I have run rmiregistry and established a jstatd.all.policy file on the remote box: grant codebase "file:${java.home}/../lib/tools.jar" { permission java.security.AllPermission; }; Any ideas? The local box is 32 bit Hotspot; the remote box a 64 bit running Hotspot. Keith R Holdaway Java Development Technologies SAS The Power to Know Carpe Diem -----Original Message----- From: hotspot-gc-dev-bounces at openjdk.java.net [mailto:hotspot-gc-dev-bounces at openjdk.java.net] On Behalf Of Colin Walters Sent: Thursday, February 28, 2008 4:59 PM To: Florian Weimer Cc: hotspot-gc-dev at openjdk.java.net Subject: Re: no-administration heap? On Thu, Feb 28, 2008 at 4:20 PM, Florian Weimer wrote: > > At least on Linux, the problem is that there is no documented API to > reserve address space (so that no other mappings end up there) without > actually allocating backing store. MAP_NORESERVE does not work as > documented. Ah, I see. So this problem is essentially blocked on the lack of a sufficient/documented Linux kernel interface? Do other operating systems (Windows, Solaris) provide a good enough interface for OpenJDK? I'm assuming changing the collector to not require a continuous address space would be too invasive? From walters at verbum.org Thu Feb 28 23:09:05 2008 From: walters at verbum.org (Colin Walters) Date: Thu, 28 Feb 2008 18:09:05 -0500 Subject: no-administration heap? In-Reply-To: <47C739B1.3010605@Sun.COM> References: <87d4qgrd69.fsf@mid.deneb.enyo.de> <47C739B1.3010605@Sun.COM> Message-ID: On Thu, Feb 28, 2008 at 5:46 PM, Peter B. Kessler wrote: > > Nothing is "too invasive"! We have a sketch of a "chunked heap"; > it's just a lot of work. Things like the card table that used to > be indexed by offsets in the heap become part of each heap chunk > and are indexed by an offset in the chunk. A lot of the underlying > data structures have to be rewritten, but that's why they call it > software. We would *love* to have that kind of heap. Great to hear there's work in this area! I'll be keeping an eye on the progress; I'm planning to convert a CPython application to OpenJDK+Jython, and this issue is one thing that would be nice to have solved. > Then there are "details" like that once you chunk the heap you > limit the maximum size of an object. I was thinking that 64MB > chunks might be one possibility in that they would let you have > (somewhat less than) a 64MB array, which is the largest you can > have now on the client unless you use a larger -Xmx. Large byte arrays might be interesting to support, but otherwise a 64MB limit sounds reasonable.