From jamesnichols3 at gmail.com Fri Feb 20 10:00:06 2009 From: jamesnichols3 at gmail.com (James Nichols) Date: Fri, 20 Feb 2009 13:00:06 -0500 Subject: Really short GC/application time cycle. Message-ID: <83a51e120902201000n6b2ab990k61f8f9d2c34c90fb@mail.gmail.com> I was looking through a recent GC log and saw a recurring pattern in the log output that looked like what is shown below. The application was stopped for a very short period of time and the GC's were very short. What is perplexing is that my combination of JVM arguments should cause the heap before/after snapshot to occur when garbage collection is occuring. I'm not sure what the JVM is doing that causes the application/gc to cycle this rapidly and why there are no deatils being printed. Is this a bug in the log output or am I missing a setting? Total time for which application threads were stopped: 0.1968050 seconds Application time: 5.0923120 seconds Total time for which application threads were stopped: 0.0223190 seconds Application time: 0.0031100 seconds Total time for which application threads were stopped: 0.0096730 seconds Application time: 0.0006320 seconds Total time for which application threads were stopped: 0.0135390 seconds Application time: 0.0001600 seconds Total time for which application threads were stopped: 0.0420840 seconds Application time: 0.0009400 seconds Total time for which application threads were stopped: 0.0222590 seconds Application time: 0.0008400 seconds My JVM arguments are: -server -Xms4096m -Xmx4096m -XX:NewSize=1228M -XX:MaxNewSize=1228M -XX:MaxTenuringThreshold=4 -XX:SurvivorRatio=6 -XX:+ScavengeBeforeFullGC -XX:PermSize=256M -XX:MaxPermSize=256M -XX:+HandlePromotionFailure -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ParallelGCThreads=3 -XX:+CMSParallelRemarkEnabled -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -verbosegc -XX:+DisableExplicitGC -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -XX:+PrintClassHistogram -Xloggc:/var/log/jboss/gc.dat These configuration settings cause the normal heap before/after snapshot to show up at other times in the log, show below: 5043.055: [GC {Heap before gc invocations=145: par new generation total 1100288K, used 984350K [0x00002aaab21b0000, 0x00002aaafedb0000, 0x00002aaafedb0000) eden space 943104K, 100% used [0x00002aaab21b0000, 0x00002aaaebab0000, 0x00002aaaebab0000) from space 157184K, 26% used [0x00002aaaf5430000, 0x00002aaaf7c77a60, 0x00002aaafedb0000) to space 157184K, 0% used [0x00002aaaebab0000, 0x00002aaaebab0000, 0x00002aaaf5430000) concurrent mark-sweep generation total 2936832K, used 1051213K [0x00002aaafedb0000, 0x00002aabb21b0000, 0x00002aabb21b0000) concurrent-mark-sweep perm gen total 262144K, used 136182K [0x00002aabb21b0000, 0x00002aabc21b0000, 0x00002aabc21b0000) 5043.056: [ParNew Desired survivor size 80478208 bytes, new threshold 4 (max 4) - age 1: 29564048 bytes, 29564048 total - age 2: 1292208 bytes, 30856256 total - age 3: 8619864 bytes, 39476120 total - age 4: 818880 bytes, 40295000 total : 984350K->39396K(1100288K), 0.1633940 secs] 2035563K->1090727K(4037120K)Heap after gc invocations=146: par new generation total 1100288K, used 39396K [0x00002aaab21b0000, 0x00002aaafedb0000, 0x00002aaafedb0000) eden space 943104K, 0% used [0x00002aaab21b0000, 0x00002aaab21b0000, 0x00002aaaebab0000) from space 157184K, 25% used [0x00002aaaebab0000, 0x00002aaaee1290e8, 0x00002aaaf5430000) to space 157184K, 0% used [0x00002aaaf5430000, 0x00002aaaf5430000, 0x00002aaafedb0000) concurrent mark-sweep generation total 2936832K, used 1051331K [0x00002aaafedb0000, 0x00002aabb21b0000, 0x00002aabb21b0000) concurrent-mark-sweep perm gen total 262144K, used 136182K [0x00002aabb21b0000, 0x00002aabc21b0000, 0x00002aabc21b0000) } , 0.1644490 secs] -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20090220/c1159405/attachment.html From Y.S.Ramakrishna at Sun.COM Fri Feb 20 10:02:07 2009 From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna) Date: Fri, 20 Feb 2009 10:02:07 -0800 Subject: Really short GC/application time cycle. In-Reply-To: <83a51e120902201000n6b2ab990k61f8f9d2c34c90fb@mail.gmail.com> References: <83a51e120902201000n6b2ab990k61f8f9d2c34c90fb@mail.gmail.com> Message-ID: <6b40db956848.499e7f9f@sun.com> Try -XX:+PrintSafepointStatistics to get some more detail on what might be causing these frequent safepoints. -- ramki ----- Original Message ----- From: James Nichols Date: Friday, February 20, 2009 10:00 am Subject: Really short GC/application time cycle. To: "hotspot-gc-use at openjdk.java.net" > I was looking through a recent GC log and saw a recurring pattern in > the log > output that looked like what is shown below. The application was stopped > for a very short period of time and the GC's were very short. What is > perplexing is that my combination of JVM arguments should cause the heap > before/after snapshot to occur when garbage collection is occuring. > I'm not > sure what the JVM is doing that causes the application/gc to cycle this > rapidly and why there are no deatils being printed. Is this a bug in > the > log output or am I missing a setting? > > > Total time for which application threads were stopped: 0.1968050 seconds > Application time: 5.0923120 seconds > Total time for which application threads were stopped: 0.0223190 seconds > Application time: 0.0031100 seconds > Total time for which application threads were stopped: 0.0096730 seconds > Application time: 0.0006320 seconds > Total time for which application threads were stopped: 0.0135390 seconds > Application time: 0.0001600 seconds > Total time for which application threads were stopped: 0.0420840 seconds > Application time: 0.0009400 seconds > Total time for which application threads were stopped: 0.0222590 seconds > Application time: 0.0008400 seconds > > > > My JVM arguments are: > -server -Xms4096m -Xmx4096m -XX:NewSize=1228M -XX:MaxNewSize=1228M > -XX:MaxTenuringThreshold=4 -XX:SurvivorRatio=6 -XX:+ScavengeBeforeFullGC > -XX:PermSize=256M -XX:MaxPermSize=256M -XX:+HandlePromotionFailure > -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ParallelGCThreads=3 > -XX:+CMSParallelRemarkEnabled -XX:+CMSClassUnloadingEnabled > -XX:+CMSPermGenSweepingEnabled -verbosegc -XX:+DisableExplicitGC > -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+PrintGCTimeStamps > -XX:+PrintHeapAtGC -XX:+PrintGCApplicationConcurrentTime > -XX:+PrintGCApplicationStoppedTime -XX:+PrintClassHistogram > -Xloggc:/var/log/jboss/gc.dat > > > > These configuration settings cause the normal heap before/after > snapshot to > show up at other times in the log, show below: > > 5043.055: [GC {Heap before gc invocations=145: > par new generation total 1100288K, used 984350K [0x00002aaab21b0000, > 0x00002aaafedb0000, 0x00002aaafedb0000) > eden space 943104K, 100% used [0x00002aaab21b0000, 0x00002aaaebab0000, > 0x00002aaaebab0000) > from space 157184K, 26% used [0x00002aaaf5430000, 0x00002aaaf7c77a60, > 0x00002aaafedb0000) > to space 157184K, 0% used [0x00002aaaebab0000, 0x00002aaaebab0000, > 0x00002aaaf5430000) > concurrent mark-sweep generation total 2936832K, used 1051213K > [0x00002aaafedb0000, 0x00002aabb21b0000, 0x00002aabb21b0000) > concurrent-mark-sweep perm gen total 262144K, used 136182K > [0x00002aabb21b0000, 0x00002aabc21b0000, 0x00002aabc21b0000) > 5043.056: [ParNew > Desired survivor size 80478208 bytes, new threshold 4 (max 4) > - age 1: 29564048 bytes, 29564048 total > - age 2: 1292208 bytes, 30856256 total > - age 3: 8619864 bytes, 39476120 total > - age 4: 818880 bytes, 40295000 total > : 984350K->39396K(1100288K), 0.1633940 secs] > 2035563K->1090727K(4037120K)Heap after gc invocations=146: > par new generation total 1100288K, used 39396K [0x00002aaab21b0000, > 0x00002aaafedb0000, 0x00002aaafedb0000) > eden space 943104K, 0% used [0x00002aaab21b0000, 0x00002aaab21b0000, > 0x00002aaaebab0000) > from space 157184K, 25% used [0x00002aaaebab0000, 0x00002aaaee1290e8, > 0x00002aaaf5430000) > to space 157184K, 0% used [0x00002aaaf5430000, 0x00002aaaf5430000, > 0x00002aaafedb0000) > concurrent mark-sweep generation total 2936832K, used 1051331K > [0x00002aaafedb0000, 0x00002aabb21b0000, 0x00002aabb21b0000) > concurrent-mark-sweep perm gen total 262144K, used 136182K > [0x00002aabb21b0000, 0x00002aabc21b0000, 0x00002aabc21b0000) > } > , 0.1644490 secs] > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From Y.S.Ramakrishna at Sun.COM Fri Feb 20 17:00:25 2009 From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna) Date: Fri, 20 Feb 2009 17:00:25 -0800 Subject: Really short GC/application time cycle. In-Reply-To: <499F3EC7.7030502@kippdata.de> References: <83a51e120902201000n6b2ab990k61f8f9d2c34c90fb@mail.gmail.com> <6b40db956848.499e7f9f@sun.com> <499F3EC7.7030502@kippdata.de> Message-ID: <7320c7bf4a5a.499ee1a9@sun.com> Hmm, I believe Jim is seeing these messages on 5u12 where biased locking is not enabled by default as far as i can tell (and he's not explicitly enabled it at least from the set of JVM options he listed below). So the pauses might be for some other reason, perhaps deoptimization because of class loading etc. maybe? Unfortunately, 5u12 does not support the +PrintSafepointStatistics flag that would have been useful to get more visibility into what was going on. --ramki ----- Original Message ----- From: Rainer Jung Date: Friday, February 20, 2009 3:41 pm Subject: Re: Really short GC/application time cycle. To: Y Srinivas Ramakrishna Cc: James Nichols , "hotspot-gc-use at openjdk.java.net" > I expect the answer is the same I got from Tony Printezis in October > last year. See > > http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2008-October/000478.html > > http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2008-October/000532.html > > Regards, > > Rainer > > On 20.02.2009 19:02, Y Srinivas Ramakrishna wrote: > >Try -XX:+PrintSafepointStatistics to get some more detail > >on what might be causing these frequent safepoints. > > > >-- ramki > > > >----- Original Message ----- > >From: James Nichols > >Date: Friday, February 20, 2009 10:00 am > >Subject: Really short GC/application time cycle. > >To: "hotspot-gc-use at openjdk.java.net" > > > > > >>I was looking through a recent GC log and saw a recurring pattern in > >>the log > >>output that looked like what is shown below. The application was stopped > >>for a very short period of time and the GC's were very short. What > is > >>perplexing is that my combination of JVM arguments should cause the > heap > >>before/after snapshot to occur when garbage collection is occuring. > >>I'm not > >>sure what the JVM is doing that causes the application/gc to cycle this > >>rapidly and why there are no deatils being printed. Is this a bug in > >>the > >>log output or am I missing a setting? > >> > >> > >>Total time for which application threads were stopped: 0.1968050 seconds > >>Application time: 5.0923120 seconds > >>Total time for which application threads were stopped: 0.0223190 seconds > >>Application time: 0.0031100 seconds > >>Total time for which application threads were stopped: 0.0096730 seconds > >>Application time: 0.0006320 seconds > >>Total time for which application threads were stopped: 0.0135390 seconds > >>Application time: 0.0001600 seconds > >>Total time for which application threads were stopped: 0.0420840 seconds > >>Application time: 0.0009400 seconds > >>Total time for which application threads were stopped: 0.0222590 seconds > >>Application time: 0.0008400 seconds > >> > >> > >> > >>My JVM arguments are: > >>-server -Xms4096m -Xmx4096m -XX:NewSize=1228M -XX:MaxNewSize=1228M > >>-XX:MaxTenuringThreshold=4 -XX:SurvivorRatio=6 -XX:+ScavengeBeforeFullGC > >>-XX:PermSize=256M -XX:MaxPermSize=256M -XX:+HandlePromotionFailure > >>-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ParallelGCThreads=3 > >>-XX:+CMSParallelRemarkEnabled -XX:+CMSClassUnloadingEnabled > >>-XX:+CMSPermGenSweepingEnabled -verbosegc -XX:+DisableExplicitGC > >>-XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+PrintGCTimeStamps > >>-XX:+PrintHeapAtGC -XX:+PrintGCApplicationConcurrentTime > >>-XX:+PrintGCApplicationStoppedTime -XX:+PrintClassHistogram > >>-Xloggc:/var/log/jboss/gc.dat > >> > >> > >> > >>These configuration settings cause the normal heap before/after > >>snapshot to > >>show up at other times in the log, show below: > >> > >>5043.055: [GC {Heap before gc invocations=145: > >> par new generation total 1100288K, used 984350K [0x00002aaab21b0000, > >>0x00002aaafedb0000, 0x00002aaafedb0000) > >> eden space 943104K, 100% used [0x00002aaab21b0000, 0x00002aaaebab0000, > >>0x00002aaaebab0000) > >> from space 157184K, 26% used [0x00002aaaf5430000, 0x00002aaaf7c77a60, > >>0x00002aaafedb0000) > >> to space 157184K, 0% used [0x00002aaaebab0000, 0x00002aaaebab0000, > >>0x00002aaaf5430000) > >> concurrent mark-sweep generation total 2936832K, used 1051213K > >>[0x00002aaafedb0000, 0x00002aabb21b0000, 0x00002aabb21b0000) > >> concurrent-mark-sweep perm gen total 262144K, used 136182K > >>[0x00002aabb21b0000, 0x00002aabc21b0000, 0x00002aabc21b0000) > >>5043.056: [ParNew > >>Desired survivor size 80478208 bytes, new threshold 4 (max 4) > >>- age 1: 29564048 bytes, 29564048 total > >>- age 2: 1292208 bytes, 30856256 total > >>- age 3: 8619864 bytes, 39476120 total > >>- age 4: 818880 bytes, 40295000 total > >>: 984350K->39396K(1100288K), 0.1633940 secs] > >>2035563K->1090727K(4037120K)Heap after gc invocations=146: > >> par new generation total 1100288K, used 39396K [0x00002aaab21b0000, > >>0x00002aaafedb0000, 0x00002aaafedb0000) > >> eden space 943104K, 0% used [0x00002aaab21b0000, 0x00002aaab21b0000, > >>0x00002aaaebab0000) > >> from space 157184K, 25% used [0x00002aaaebab0000, 0x00002aaaee1290e8, > >>0x00002aaaf5430000) > >> to space 157184K, 0% used [0x00002aaaf5430000, 0x00002aaaf5430000, > >>0x00002aaafedb0000) > >> concurrent mark-sweep generation total 2936832K, used 1051331K > >>[0x00002aaafedb0000, 0x00002aabb21b0000, 0x00002aabb21b0000) > >> concurrent-mark-sweep perm gen total 262144K, used 136182K > >>[0x00002aabb21b0000, 0x00002aabc21b0000, 0x00002aabc21b0000) > >>} > >>, 0.1644490 secs] From rainer.jung at kippdata.de Fri Feb 20 15:37:43 2009 From: rainer.jung at kippdata.de (Rainer Jung) Date: Sat, 21 Feb 2009 00:37:43 +0100 Subject: Really short GC/application time cycle. In-Reply-To: <6b40db956848.499e7f9f@sun.com> References: <83a51e120902201000n6b2ab990k61f8f9d2c34c90fb@mail.gmail.com> <6b40db956848.499e7f9f@sun.com> Message-ID: <499F3EC7.7030502@kippdata.de> I expect the answer is the same I got from Tony Printezis in October last year. See http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2008-October/000478.html http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2008-October/000532.html Regards, Rainer On 20.02.2009 19:02, Y Srinivas Ramakrishna wrote: > Try -XX:+PrintSafepointStatistics to get some more detail > on what might be causing these frequent safepoints. > > -- ramki > > ----- Original Message ----- > From: James Nichols > Date: Friday, February 20, 2009 10:00 am > Subject: Really short GC/application time cycle. > To: "hotspot-gc-use at openjdk.java.net" > > >> I was looking through a recent GC log and saw a recurring pattern in >> the log >> output that looked like what is shown below. The application was stopped >> for a very short period of time and the GC's were very short. What is >> perplexing is that my combination of JVM arguments should cause the heap >> before/after snapshot to occur when garbage collection is occuring. >> I'm not >> sure what the JVM is doing that causes the application/gc to cycle this >> rapidly and why there are no deatils being printed. Is this a bug in >> the >> log output or am I missing a setting? >> >> >> Total time for which application threads were stopped: 0.1968050 seconds >> Application time: 5.0923120 seconds >> Total time for which application threads were stopped: 0.0223190 seconds >> Application time: 0.0031100 seconds >> Total time for which application threads were stopped: 0.0096730 seconds >> Application time: 0.0006320 seconds >> Total time for which application threads were stopped: 0.0135390 seconds >> Application time: 0.0001600 seconds >> Total time for which application threads were stopped: 0.0420840 seconds >> Application time: 0.0009400 seconds >> Total time for which application threads were stopped: 0.0222590 seconds >> Application time: 0.0008400 seconds >> >> >> >> My JVM arguments are: >> -server -Xms4096m -Xmx4096m -XX:NewSize=1228M -XX:MaxNewSize=1228M >> -XX:MaxTenuringThreshold=4 -XX:SurvivorRatio=6 -XX:+ScavengeBeforeFullGC >> -XX:PermSize=256M -XX:MaxPermSize=256M -XX:+HandlePromotionFailure >> -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ParallelGCThreads=3 >> -XX:+CMSParallelRemarkEnabled -XX:+CMSClassUnloadingEnabled >> -XX:+CMSPermGenSweepingEnabled -verbosegc -XX:+DisableExplicitGC >> -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+PrintGCTimeStamps >> -XX:+PrintHeapAtGC -XX:+PrintGCApplicationConcurrentTime >> -XX:+PrintGCApplicationStoppedTime -XX:+PrintClassHistogram >> -Xloggc:/var/log/jboss/gc.dat >> >> >> >> These configuration settings cause the normal heap before/after >> snapshot to >> show up at other times in the log, show below: >> >> 5043.055: [GC {Heap before gc invocations=145: >> par new generation total 1100288K, used 984350K [0x00002aaab21b0000, >> 0x00002aaafedb0000, 0x00002aaafedb0000) >> eden space 943104K, 100% used [0x00002aaab21b0000, 0x00002aaaebab0000, >> 0x00002aaaebab0000) >> from space 157184K, 26% used [0x00002aaaf5430000, 0x00002aaaf7c77a60, >> 0x00002aaafedb0000) >> to space 157184K, 0% used [0x00002aaaebab0000, 0x00002aaaebab0000, >> 0x00002aaaf5430000) >> concurrent mark-sweep generation total 2936832K, used 1051213K >> [0x00002aaafedb0000, 0x00002aabb21b0000, 0x00002aabb21b0000) >> concurrent-mark-sweep perm gen total 262144K, used 136182K >> [0x00002aabb21b0000, 0x00002aabc21b0000, 0x00002aabc21b0000) >> 5043.056: [ParNew >> Desired survivor size 80478208 bytes, new threshold 4 (max 4) >> - age 1: 29564048 bytes, 29564048 total >> - age 2: 1292208 bytes, 30856256 total >> - age 3: 8619864 bytes, 39476120 total >> - age 4: 818880 bytes, 40295000 total >> : 984350K->39396K(1100288K), 0.1633940 secs] >> 2035563K->1090727K(4037120K)Heap after gc invocations=146: >> par new generation total 1100288K, used 39396K [0x00002aaab21b0000, >> 0x00002aaafedb0000, 0x00002aaafedb0000) >> eden space 943104K, 0% used [0x00002aaab21b0000, 0x00002aaab21b0000, >> 0x00002aaaebab0000) >> from space 157184K, 25% used [0x00002aaaebab0000, 0x00002aaaee1290e8, >> 0x00002aaaf5430000) >> to space 157184K, 0% used [0x00002aaaf5430000, 0x00002aaaf5430000, >> 0x00002aaafedb0000) >> concurrent mark-sweep generation total 2936832K, used 1051331K >> [0x00002aaafedb0000, 0x00002aabb21b0000, 0x00002aabb21b0000) >> concurrent-mark-sweep perm gen total 262144K, used 136182K >> [0x00002aabb21b0000, 0x00002aabc21b0000, 0x00002aabc21b0000) >> } >> , 0.1644490 secs] From shane.cox at gmail.com Mon Feb 23 06:15:48 2009 From: shane.cox at gmail.com (Shane Cox) Date: Mon, 23 Feb 2009 09:15:48 -0500 Subject: incomplete dump? Message-ID: Last week a JVM crashed due to OutOfMemory and produced a core dump. GC logs indicate that the Heap filled to capacity with live objects. Here is the end of the GC log: 70804.138: [Full GC 70804.138: [CMS70807.169: [CMS-concurrent-mark: 3.037/3.040 secs] (concurrent mode failure): 2031615K->2031615K(2031616K), 10.1815366 secs] 2084095K->2034933K(2090624K), [CMS Perm : 18628K->18628K(31244K)], 10.1817359 secs] 70814.404: [GC [1 CMS-initial-mark: 2031615K(2031616K)] 2084391K(2090624K), 0.0489370 secs] 70814.453: [CMS-concurrent-mark-start] 70814.604: [Full GC 70814.604: [CMS As you can see, the GC log abnormally terminated at the time of the crash. We attempted to use JMAP to analyze the core file to determine what type of objects filled the heap. The output is below for your review. Unfortunately, the CMS generation is reported to be empty whereas the GC logs indicate that it was 100% full. Any insight into why the CMS generation is reported to be empty or what we might be doing wrong in our usage of JMAP would be much appreciated. Complete GC log is attached (gzipped) as well as the hotspot error log. Attaching to core core from executable /usr/jdk/instances/jdk1.6.0/bin/java, please wait... Debugger attached successfully. Server compiler detected. JVM version is 1.6.0_03-b05 using parallel threads in the new generation. using thread-local object allocation. Concurrent Mark-Sweep GC Heap Configuration: MinHeapFreeRatio = 40 MaxHeapFreeRatio = 70 MaxHeapSize = 2147483648 (2048.0MB) NewSize = 16777216 (16.0MB) MaxNewSize = 67108864 (64.0MB) OldSize = 50331648 (48.0MB) NewRatio = 8 SurvivorRatio = 8 PermSize = 16777216 (16.0MB) MaxPermSize = 67108864 (64.0MB) Heap Usage: New Generation (Eden + 1 Survivor Space): capacity = 60424192 (57.625MB) used = 60424184 (57.62499237060547MB) free = 8 (7.62939453125E-6MB) 99.99998676026979% used Eden Space: capacity = 53739520 (51.25MB) used = 53739520 (51.25MB) free = 0 (0.0MB) 100.0% used >From Space: capacity = 6684672 (6.375MB) used = 6684664 (6.374992370605469MB) free = 8 (7.62939453125E-6MB) 99.99988032322304% used To Space: capacity = 6684672 (6.375MB) used = 0 (0.0MB) free = 6684672 (6.375MB) 0.0% used *concurrent mark-sweep generation: capacity = 2080374784 (1984.0MB) used = 0 (0.0MB) free = 2080374784 (1984.0MB) 0.0% used* Perm Generation: capacity = 31993856 (30.51171875MB) Free chunk in CMS heap, size=12914368 used = 19079488 (18.19561767578125MB) Free chunk in CMS heap, size=12914368 free = 12914368 (12.31610107421875MB) Free chunk in CMS heap, size=12914368 59.63484989117911% used Attaching to core core from executable /usr/jdk/instances/jdk1.6.0/bin/java, please wait... Debugger attached successfully. Server compiler detected. JVM version is 1.6.0_03-b05 Iterating over heap. This may take a while... Free chunk in CMS heap, size=12914368 Object Histogram: Size Count Class description ------------------------------------------------------- 11801952 184641 char[] 10920360 22387 byte[] 7334880 26196 com.theice.mdf.adapter.comm.MDEvent 5325024 110938 java.nio.HeapByteBuffer 5039328 104986 java.nio.HeapCharBuffer 4779456 42169 * ConstMethodKlass 4276248 178177 java.lang.String 3377808 42169 * MethodKlass 2827968 65216 * SymbolKlass 2070232 34661 java.lang.Object[] 2033184 3744 * ConstantPoolKlass 1886040 26195 com.theice.mdf.adapter.comm.MDEventStatisticsCollector ........... -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20090223/c1863a4b/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: 2009-02-19-RouterPF221-gc-output.log.gz Type: application/x-gzip Size: 471662 bytes Desc: not available Url : http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20090223/c1863a4b/attachment.bin -------------- next part -------------- A non-text attachment was scrubbed... Name: hs_err_pid11438.log Type: application/octet-stream Size: 33818 bytes Desc: not available Url : http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20090223/c1863a4b/attachment.obj From jamesnichols3 at gmail.com Tue Feb 24 14:43:56 2009 From: jamesnichols3 at gmail.com (James Nichols) Date: Tue, 24 Feb 2009 17:43:56 -0500 Subject: Unexplained long stop when using CMS Message-ID: <83a51e120902241443x42065ef8rff9548696bf255f7@mail.gmail.com> Greetings. I've suddenly started seeing several long stop the world pauses using the CMS collector that I can't explain. Below is the output from my gc log file. As you can see, the application threads were stopped for 50 seconds, but I can't see where that time is being sent in the gc timing output. 9872.975: [CMS-concurrent-abortable-preclean: 6.885/9.607 secs] Application time: 7.7035080 seconds 9923.339: [GC[YG occupancy: 563247 K (1100288 K)]9923.341: [Rescan (parallel) , 0.3760170 secs]9923.717: [weak refs processing, 0.0496640 secs]9923.767: [class unloading, 0.1167260 secs]9923.884: [scrub symbol & string tables, 0.0241920 secs] [1 CMS-remark: 1657038K(2936832K)] 2220286K(4037120K), 0.5834530 secs] Total time for which application threads were stopped: 50.9487390 seconds Due to some unrelated application problems, I had submitted a case to JBoss support. They had noticed the phenomenon in the GC log and suggest an upgrade to JDK 1.5.0_14, pointing out the following bug: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6546278 It's crazy to take anything JBoss support ever gives you on faith, so I thought I would check with the experts. Unfortunately, the methodology for measuring if this is impacting my application described in the bug is really hard for me to implement, even though this is only occuring in a test environment. I am seeing really high load for short periods of time on this particular box, but it's hard for me to correlate the long application pause to the load average measurement. Is there anyway I can verify that this is the problem I'm seeing? Or even any insight into why there is this discrepency between the "application stopped" message and the rescan timings? Or should I just jump on rev 14? We've been on 12 for a while now. I have attached the gc.dat file containing this particular long GC. I'm using Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_12-b04, mixed mode), and here are the JVM arguments: /usr/java/jdk1.5.0_12/bin/java -Dprogram.name=run.sh -server -Xms4096m -Xmx4096m -XX:NewSize=1228M -XX:MaxNewSize=1228M -XX:MaxTenuringThreshold=4 -XX:SurvivorRatio=6 -XX:+ScavengeBeforeFullGC -XX:PermSize=256M -XX:MaxPermSize=256M -XX:+HandlePromotionFailure -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ParallelGCThreads=3 -XX:+CMSParallelRemarkEnabled -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -verbosegc -XX:+DisableExplicitGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -Xloggc:/var/log/jboss/gc.dat -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20090224/734d6e71/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: gc.dat.gz Type: application/x-gzip Size: 29617 bytes Desc: not available Url : http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20090224/734d6e71/attachment.bin From Y.S.Ramakrishna at Sun.COM Tue Feb 24 14:57:18 2009 From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna) Date: Tue, 24 Feb 2009 14:57:18 -0800 Subject: Unexplained long stop when using CMS In-Reply-To: <83a51e120902241443x42065ef8rff9548696bf255f7@mail.gmail.com> References: <83a51e120902241443x42065ef8rff9548696bf255f7@mail.gmail.com> Message-ID: <7350c2563888.49a40ace@sun.com> Hi Jim -- Yes, i'd try 5u14 (or later), or the -XX:+UseMembar workaround. It's the most likely explanation for the pause you are seeing. In more modern JVM's there are dtrace probes you could leverage for this kind of debugging. -- ramki ----- Original Message ----- From: James Nichols Date: Tuesday, February 24, 2009 2:44 pm Subject: Unexplained long stop when using CMS To: "hotspot-gc-use at openjdk.java.net" > Greetings. > > I've suddenly started seeing several long stop the world pauses using > the > CMS collector that I can't explain. Below is the output from my gc log > file. As you can see, the application threads were stopped for 50 seconds, > but I can't see where that time is being sent in the gc timing output. > > 9872.975: [CMS-concurrent-abortable-preclean: 6.885/9.607 secs] > Application time: 7.7035080 seconds > 9923.339: [GC[YG occupancy: 563247 K (1100288 K)]9923.341: [Rescan > (parallel) , 0.3760170 secs]9923.717: [weak refs processing, 0.0496640 > secs]9923.767: [class unloading, > 0.1167260 secs]9923.884: [scrub symbol & string tables, 0.0241920 > secs] [1 > CMS-remark: 1657038K(2936832K)] 2220286K(4037120K), 0.5834530 secs] > Total time for which application threads were stopped: 50.9487390 seconds > > > Due to some unrelated application problems, I had submitted a case to > JBoss > support. They had noticed the phenomenon in the GC log and suggest an > upgrade to JDK 1.5.0_14, pointing out the following bug: > http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6546278 > > It's crazy to take anything JBoss support ever gives you on faith, so > I > thought I would check with the experts. Unfortunately, the > methodology for > measuring if this is impacting my application described in the bug is > really > hard for me to implement, even though this is only occuring in a test > environment. I am seeing really high load for short periods of time > on this > particular box, but it's hard for me to correlate the long application > pause > to the load average measurement. Is there anyway I can verify that > this is > the problem I'm seeing? Or even any insight into why there is this > discrepency between the "application stopped" message and the rescan > timings? Or should I just jump on rev 14? We've been on 12 for a while > now. > > > I have attached the gc.dat file containing this particular long GC. > > I'm using Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_12-b04, mixed > mode), and here are the JVM arguments: > > /usr/java/jdk1.5.0_12/bin/java -Dprogram.name=run.sh -server -Xms4096m > -Xmx4096m -XX:NewSize=1228M -XX:MaxNewSize=1228M -XX:MaxTenuringThreshold=4 > -XX:SurvivorRatio=6 -XX:+ScavengeBeforeFullGC -XX:PermSize=256M > -XX:MaxPermSize=256M -XX:+HandlePromotionFailure -XX:+UseConcMarkSweepGC > -XX:+UseParNewGC -XX:ParallelGCThreads=3 -XX:+CMSParallelRemarkEnabled > -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -verbosegc > -XX:+DisableExplicitGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps > -XX:+PrintHeapAtGC -XX:+PrintGCApplicationConcurrentTime > -XX:+PrintGCApplicationStoppedTime -Xloggc:/var/log/jboss/gc.dat > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From Y.S.Ramakrishna at Sun.COM Wed Feb 25 00:38:58 2009 From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna) Date: Wed, 25 Feb 2009 00:38:58 -0800 Subject: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64 platform within data processing application In-Reply-To: <49A4DE63.5030704@sun.com> References: <499CA7D9.9040903@nuix.com> <4E0D3F42-021A-4D48-888E-594A5980D0E1@sun.com> <49A33600.7000004@nuix.com> <49A3454A.4010507@nuix.com> <49A34B69.2000608@sun.com> <49A34DF2.2010800@sun.com> <49A48B14.8070307@nuix.com> <49A4DE63.5030704@sun.com> Message-ID: <6ed08c5e5b2a.49a49322@sun.com> Doesn't the heap look too full? If a 64-bit JVM why use such an oversubscribed and small heap? Either make the old gen bigger or make the young gen smaller (giving that space to the older gen) so that each scavenge does not degenerate to a full gc as in your trace below. This discussion probably belongs on hotspot-gc-use at o.j.n list so i have cross-posted over to that list with a bcc to the hotspot-dev list. Also the GC tuning guides to be found here might be useful reading:- http://java.sun.com/javase/technologies/hotspot/gc/index.jsp -- ramki ----- Original Message ----- From: Jon Masamitsu Date: Tuesday, February 24, 2009 10:00 pm Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64 platform within data processing application To: David Sitsky Cc: hotspot-dev at openjdk.java.net, Tom Rodriguez > David, > > Can you also send a GC log from a run where there > is not a problem? As I understand it, that would > be a 32bit run. > > Jon > > David Sitsky wrote On 02/24/09 16:04,: > > >Jon Masamitsu wrote: > > > > > >>Jon Masamitsu wrote On 02/23/09 17:20,: > >> > >> > >> > >>>... > >>> > >>>Increase the heap by 30%. Also increase the the perm gen size > >>>(-XX:MaxPermSize=). > >>> > >>>Please use -XX:+PrintGCDetails -XX:+PrintGCTimeStamps when > gathering the > >>>GC logs. > >>>If you've already gathering some, send those but in future runs, > use the > >>>above. > >>> > >>> > > > >Here is a sample of output from a stuck process. You can see its > doing > >a full GC about every 3 seconds, and it seems as if there is little > >progress.. > > > >Please let me know if you need more information. > > > >Cheers, > >David > > > >62402.320: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8063178 secs] [Times: user=2.81 sys=0.00, > > >real=2.81 secs] > >62405.128: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8029997 secs] [Times: user=2.79 sys=0.00, > > >real=2.81 secs] > >62407.932: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.7917325 secs] [Times: user=2.79 sys=0.00, > > >real=2.79 secs] > >62410.725: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.7891387 secs] [Times: user=2.79 sys=0.00, > > >real=2.79 secs] > >62413.515: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.7649110 secs] [Times: user=2.76 sys=0.00, > > >real=2.76 secs] > >62416.281: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.7803983 secs] [Times: user=2.78 sys=0.00, > > >real=2.78 secs] > >62419.063: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.7643979 secs] [Times: user=2.76 sys=0.00, > > >real=2.76 secs] > >62421.828: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8114336 secs] [Times: user=2.81 sys=0.00, > > >real=2.81 secs] > >62424.640: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.7964912 secs] [Times: user=2.79 sys=0.00, > > >real=2.79 secs] > >62427.438: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8107278 secs] [Times: user=2.81 sys=0.00, > > >real=2.81 secs] > >62430.249: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 3.2345212 secs] [Times: user=2.84 sys=0.00, > > >real=3.24 secs] > >62433.484: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8341520 secs] [Times: user=2.82 sys=0.00, > > >real=2.82 secs] > >62436.319: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8698768 secs] [Times: user=2.87 sys=0.00, > > >real=2.87 secs] > >62439.190: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.9323230 secs] [Times: user=2.90 sys=0.00, > > >real=2.92 secs] > >62442.124: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.9644960 secs] [Times: user=2.96 sys=0.00, > > >real=2.96 secs] > >62445.089: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 3.0059221 secs] [Times: user=3.00 sys=0.00, > > >real=3.00 secs] > >62448.095: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.9832815 secs] [Times: user=3.00 sys=0.00, > > >real=2.99 secs] > >62451.079: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.9587156 secs] [Times: user=2.93 sys=0.00, > > >real=2.95 secs] > >62454.039: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.9488345 secs] [Times: user=2.92 sys=0.00, > > >real=2.95 secs] > >62456.988: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8969788 secs] [Times: user=2.90 sys=0.00, > > >real=2.90 secs] > >62459.886: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8794991 secs] [Times: user=2.89 sys=0.00, > > >real=2.89 secs] > >62462.766: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8842411 secs] [Times: user=2.87 sys=0.00, > > >real=2.89 secs] > >62465.651: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683676K->683675K(699072K)] 781916K->781915K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8669173 secs] [Times: user=2.85 sys=0.00, > > >real=2.85 secs] > >62468.519: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8664429 secs] [Times: user=2.85 sys=0.00, > > >real=2.86 secs] > >62471.386: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8844494 secs] [Times: user=2.87 sys=0.00, > > >real=2.89 secs] > >62474.271: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8648398 secs] [Times: user=2.87 sys=0.00, > > >real=2.87 secs] > >62477.137: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8971068 secs] [Times: user=2.87 sys=0.00, > > >real=2.90 secs] > >62480.034: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8655618 secs] [Times: user=2.86 sys=0.00, > > >real=2.86 secs] > >62482.901: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 3.0366140 secs] [Times: user=2.78 sys=0.00, > > >real=3.04 secs] > >62485.939: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8541753 secs] [Times: user=2.85 sys=0.00, > > >real=2.85 secs] > >62488.794: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8582816 secs] [Times: user=2.86 sys=0.00, > > >real=2.86 secs] > >62491.653: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8673218 secs] [Times: user=2.85 sys=0.00, > > >real=2.86 secs] > >62494.521: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.9014120 secs] [Times: user=2.87 sys=0.00, > > >real=2.90 secs] > >62497.424: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8805843 secs] [Times: user=2.89 sys=0.00, > > >real=2.89 secs] > >62500.305: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8905128 secs] [Times: user=2.89 sys=0.00, > > >real=2.89 secs] > >62503.196: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.9052007 secs] [Times: user=2.90 sys=0.00, > > >real=2.92 secs] > >62506.102: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.9004575 secs] [Times: user=2.89 sys=0.00, > > >real=2.90 secs] > >62509.003: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.9160655 secs] [Times: user=2.90 sys=0.00, > > >real=2.92 secs] > >62511.920: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.9013277 secs] [Times: user=2.90 sys=0.00, > > >real=2.90 secs] > >62514.822: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8982061 secs] [Times: user=2.89 sys=0.00, > > >real=2.89 secs] > >62517.721: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8922437 secs] [Times: user=2.89 sys=0.00, > > >real=2.89 secs] > >62520.614: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8873520 secs] [Times: user=2.87 sys=0.00, > > >real=2.89 secs] > >62523.502: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8805296 secs] [Times: user=2.89 sys=0.00, > > >real=2.89 secs] > >62526.383: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8958714 secs] [Times: user=2.85 sys=0.00, > > >real=2.89 secs] > >62529.279: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8735384 secs] [Times: user=2.89 sys=0.00, > > >real=2.89 secs] > >62532.154: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8705676 secs] [Times: user=2.87 sys=0.00, > > >real=2.87 secs] > >62535.025: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8723947 secs] [Times: user=2.85 sys=0.00, > > >real=2.87 secs] > >62537.898: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8624400 secs] [Times: user=2.86 sys=0.00, > > >real=2.86 secs] > >62540.761: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8245748 secs] [Times: user=2.84 sys=0.00, > > >real=2.84 secs] > >62543.587: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8432269 secs] [Times: user=2.84 sys=0.00, > > >real=2.84 secs] > >62546.432: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8394157 secs] [Times: user=2.84 sys=0.00, > > >real=2.84 secs] > >62549.272: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8471951 secs] [Times: user=2.85 sys=0.00, > > >real=2.85 secs] > >62552.121: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8584107 secs] [Times: user=2.85 sys=0.00, > > >real=2.86 secs] > >62554.981: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8376807 secs] [Times: user=2.84 sys=0.00, > > >real=2.84 secs] > >62557.820: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8402486 secs] [Times: user=2.84 sys=0.00, > > >real=2.84 secs] > >62560.661: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8482704 secs] [Times: user=2.86 sys=0.00, > > >real=2.85 secs] > >62563.511: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8115973 secs] [Times: user=2.81 sys=0.00, > > >real=2.81 secs] > >62566.324: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8523278 secs] [Times: user=2.85 sys=0.00, > > >real=2.86 secs] > >62569.177: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8128563 secs] [Times: user=2.81 sys=0.00, > > >real=2.81 secs] > >62571.990: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.7830644 secs] [Times: user=2.79 sys=0.00, > > >real=2.79 secs] > >62574.774: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8065106 secs] [Times: user=2.79 sys=0.00, > > >real=2.81 secs] > >62577.582: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.7892171 secs] [Times: user=2.79 sys=0.00, > > >real=2.79 secs] > >62580.372: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8059306 secs] [Times: user=2.79 sys=0.00, > > >real=2.79 secs] > >62583.179: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8641470 secs] [Times: user=2.82 sys=0.00, > > >real=2.86 secs] > >62586.044: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8421364 secs] [Times: user=2.84 sys=0.00, > > >real=2.84 secs] > >62588.887: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8852699 secs] [Times: user=2.89 sys=0.00, > > >real=2.89 secs] > >62591.773: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.9164279 secs] [Times: user=2.90 sys=0.00, > > >real=2.92 secs] > >62594.690: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.9450010 secs] [Times: user=2.95 sys=0.00, > > >real=2.95 secs] > >62597.636: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.9744636 secs] [Times: user=2.98 sys=0.00, > > >real=2.98 secs] > >62600.611: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.9900849 secs] [Times: user=2.99 sys=0.00, > > >real=3.00 secs] > >62603.602: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.6332370 secs] [Times: user=2.62 sys=0.00, > > >real=2.62 secs] > >62606.236: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.9801260 secs] [Times: user=2.95 sys=0.00, > > >real=2.98 secs] > >62609.226: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.9166374 secs] [Times: user=2.89 sys=0.00, > > >real=2.92 secs] > >62612.150: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.9475729 secs] [Times: user=2.95 sys=0.00, > > >real=2.95 secs] > >62615.098: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.9328670 secs] [Times: user=2.90 sys=0.00, > > >real=2.93 secs] > >62618.040: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8963825 secs] [Times: user=2.89 sys=0.00, > > >real=2.90 secs] > >62620.937: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8834715 secs] [Times: user=2.89 sys=0.00, > > >real=2.89 secs] > >62623.821: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8800691 secs] [Times: user=2.86 sys=0.00, > > >real=2.87 secs] > >62626.701: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683678K->683677K(699072K)] 781918K->781917K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8642587 secs] [Times: user=2.87 sys=0.00, > > >real=2.87 secs] > >62629.566: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8574615 secs] [Times: user=2.84 sys=0.00, > > >real=2.86 secs] > >62632.424: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8383412 secs] [Times: user=2.84 sys=0.00, > > >real=2.84 secs] > >62635.264: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8409891 secs] [Times: user=2.82 sys=0.00, > > >real=2.84 secs] > >62638.106: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.7906216 secs] [Times: user=2.79 sys=0.00, > > >real=2.79 secs] > >62640.898: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.7891730 secs] [Times: user=2.79 sys=0.00, > > >real=2.79 secs] > >62643.688: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.7892940 secs] [Times: user=2.79 sys=0.00, > > >real=2.79 secs] > >62646.479: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.7766807 secs] [Times: user=2.78 sys=0.00, > > >real=2.78 secs] > >62649.257: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.7796531 secs] [Times: user=2.78 sys=0.00, > > >real=2.78 secs] > >62652.037: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.7687240 secs] [Times: user=2.76 sys=0.00, > > >real=2.76 secs] > >62654.807: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.7613769 secs] [Times: user=2.76 sys=0.00, > > >real=2.76 secs] > >62657.570: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.7712254 secs] [Times: user=2.78 sys=0.00, > > >real=2.78 secs] > >62660.342: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.7968108 secs] [Times: user=2.79 sys=0.00, > > >real=2.79 secs] > >62663.139: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.7924173 secs] [Times: user=2.79 sys=0.00, > > >real=2.79 secs] > >62665.933: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8002912 secs] [Times: user=2.79 sys=0.00, > > >real=2.81 secs] > >62668.736: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8291434 secs] [Times: user=2.82 sys=0.00, > > >real=2.82 secs] > >62671.566: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8527186 secs] [Times: user=2.86 sys=0.00, > > >real=2.85 secs] > >62674.419: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8982825 secs] [Times: user=2.90 sys=0.00, > > >real=2.90 secs] > >62677.318: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.9254483 secs] [Times: user=2.93 sys=0.00, > > >real=2.93 secs] > >62680.244: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.9707015 secs] [Times: user=2.95 sys=0.00, > > >real=2.96 secs] > >62683.216: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.9894145 secs] [Times: user=3.00 sys=0.00, > > >real=3.00 secs] > >62686.206: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.9870305 secs] [Times: user=2.98 sys=0.00, > > >real=2.98 secs] > >62689.193: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.9884647 secs] [Times: user=2.98 sys=0.00, > > >real=3.00 secs] > >62692.183: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.9635276 secs] [Times: user=2.96 sys=0.00, > > >real=2.96 secs] > >62695.147: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.9407559 secs] [Times: user=2.93 sys=0.00, > > >real=2.93 secs] > >62698.088: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.9299386 secs] [Times: user=2.93 sys=0.00, > > >real=2.93 secs] > >62701.019: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8994903 secs] [Times: user=2.90 sys=0.00, > > >real=2.90 secs] > >62703.919: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.9163417 secs] [Times: user=2.92 sys=0.00, > > >real=2.92 secs] > >62706.836: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.9216473 secs] [Times: user=2.93 sys=0.00, > > >real=2.93 secs] > >62709.758: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.9052547 secs] [Times: user=2.89 sys=0.00, > > >real=2.90 secs] > >62712.664: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8902824 secs] [Times: user=2.85 sys=0.00, > > >real=2.89 secs] > >62715.555: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8865932 secs] [Times: user=2.87 sys=0.00, > > >real=2.89 secs] > >62718.442: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8605445 secs] [Times: user=2.87 sys=0.00, > > >real=2.87 secs] > >62721.304: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8662771 secs] [Times: user=2.84 sys=0.00, > > >real=2.86 secs] > >62724.171: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: > > >683679K->683679K(699072K)] 781919K->781919K(913792K) [PSPermGen: > >49694K->49694K(49984K)], 2.8369076 secs] [Times: user=2.84 sys=0.00, > > >real=2.84 secs] > > > > > > > From sits at nuix.com Wed Feb 25 13:59:07 2009 From: sits at nuix.com (David Sitsky) Date: Thu, 26 Feb 2009 08:59:07 +1100 Subject: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64 platform within data processing application In-Reply-To: <6ed08c5e5b2a.49a49322@sun.com> References: <499CA7D9.9040903@nuix.com> <4E0D3F42-021A-4D48-888E-594A5980D0E1@sun.com> <49A33600.7000004@nuix.com> <49A3454A.4010507@nuix.com> <49A34B69.2000608@sun.com> <49A34DF2.2010800@sun.com> <49A48B14.8070307@nuix.com> <49A4DE63.5030704@sun.com> <6ed08c5e5b2a.49a49322@sun.com> Message-ID: <49A5BF2B.3030407@nuix.com> Hi Ramki, The next message I posted to the hotspot list showed a GC trace when I allocated more memory (an extra 300 megs), where everything worked fine. I am well aware that the heap size for this particular application and data set was too small. My run from last night with the extra heap is still running nicely. I reported this issue, because it seemed to me no progress was being made and no OutOfMemoryErrors were being generated. The application was effectively "stuck" making no progress at all. I couldn't even connect to it with jconsole, although jstack worked fine. My understanding is this condition is meant to be detected, and OutOfMemoryError is meant to be thrown, but perhaps I am mistaken? This is with the parallel GC. Cheers, David Y Srinivas Ramakrishna wrote: > Doesn't the heap look too full? > If a 64-bit JVM why use such an oversubscribed > and small heap? Either make the old gen bigger or make the > young gen smaller (giving that space to the older gen) > so that each scavenge does not degenerate to a full gc > as in your trace below. > > This discussion probably belongs on hotspot-gc-use at o.j.n list > so i have cross-posted over to that list with a bcc to > the hotspot-dev list. > > Also the GC tuning guides to be found here might be useful > reading:- > > http://java.sun.com/javase/technologies/hotspot/gc/index.jsp > > -- ramki > > ----- Original Message ----- > From: Jon Masamitsu > Date: Tuesday, February 24, 2009 10:00 pm > Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64 platform within data processing application > To: David Sitsky > Cc: hotspot-dev at openjdk.java.net, Tom Rodriguez > > >> David, >> >> Can you also send a GC log from a run where there >> is not a problem? As I understand it, that would >> be a 32bit run. >> >> Jon >> >> David Sitsky wrote On 02/24/09 16:04,: >> >>> Jon Masamitsu wrote: >>> >>> >>>> Jon Masamitsu wrote On 02/23/09 17:20,: >>>> >>>> >>>> >>>>> ... >>>>> >>>>> Increase the heap by 30%. Also increase the the perm gen size >>>>> (-XX:MaxPermSize=). >>>>> >>>>> Please use -XX:+PrintGCDetails -XX:+PrintGCTimeStamps when >> gathering the >>>>> GC logs. >>>>> If you've already gathering some, send those but in future runs, >> use the >>>>> above. >>>>> >>>>> >>> Here is a sample of output from a stuck process. You can see its >> doing >>> a full GC about every 3 seconds, and it seems as if there is little >>> progress.. >>> >>> Please let me know if you need more information. >>> >>> Cheers, >>> David >>> >>> 62402.320: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8063178 secs] [Times: user=2.81 sys=0.00, >>> real=2.81 secs] >>> 62405.128: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8029997 secs] [Times: user=2.79 sys=0.00, >>> real=2.81 secs] >>> 62407.932: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.7917325 secs] [Times: user=2.79 sys=0.00, >>> real=2.79 secs] >>> 62410.725: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.7891387 secs] [Times: user=2.79 sys=0.00, >>> real=2.79 secs] >>> 62413.515: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.7649110 secs] [Times: user=2.76 sys=0.00, >>> real=2.76 secs] >>> 62416.281: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.7803983 secs] [Times: user=2.78 sys=0.00, >>> real=2.78 secs] >>> 62419.063: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.7643979 secs] [Times: user=2.76 sys=0.00, >>> real=2.76 secs] >>> 62421.828: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8114336 secs] [Times: user=2.81 sys=0.00, >>> real=2.81 secs] >>> 62424.640: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.7964912 secs] [Times: user=2.79 sys=0.00, >>> real=2.79 secs] >>> 62427.438: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8107278 secs] [Times: user=2.81 sys=0.00, >>> real=2.81 secs] >>> 62430.249: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 3.2345212 secs] [Times: user=2.84 sys=0.00, >>> real=3.24 secs] >>> 62433.484: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8341520 secs] [Times: user=2.82 sys=0.00, >>> real=2.82 secs] >>> 62436.319: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8698768 secs] [Times: user=2.87 sys=0.00, >>> real=2.87 secs] >>> 62439.190: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.9323230 secs] [Times: user=2.90 sys=0.00, >>> real=2.92 secs] >>> 62442.124: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.9644960 secs] [Times: user=2.96 sys=0.00, >>> real=2.96 secs] >>> 62445.089: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 3.0059221 secs] [Times: user=3.00 sys=0.00, >>> real=3.00 secs] >>> 62448.095: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.9832815 secs] [Times: user=3.00 sys=0.00, >>> real=2.99 secs] >>> 62451.079: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.9587156 secs] [Times: user=2.93 sys=0.00, >>> real=2.95 secs] >>> 62454.039: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.9488345 secs] [Times: user=2.92 sys=0.00, >>> real=2.95 secs] >>> 62456.988: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8969788 secs] [Times: user=2.90 sys=0.00, >>> real=2.90 secs] >>> 62459.886: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8794991 secs] [Times: user=2.89 sys=0.00, >>> real=2.89 secs] >>> 62462.766: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683675K->683675K(699072K)] 781915K->781915K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8842411 secs] [Times: user=2.87 sys=0.00, >>> real=2.89 secs] >>> 62465.651: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683676K->683675K(699072K)] 781916K->781915K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8669173 secs] [Times: user=2.85 sys=0.00, >>> real=2.85 secs] >>> 62468.519: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8664429 secs] [Times: user=2.85 sys=0.00, >>> real=2.86 secs] >>> 62471.386: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8844494 secs] [Times: user=2.87 sys=0.00, >>> real=2.89 secs] >>> 62474.271: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8648398 secs] [Times: user=2.87 sys=0.00, >>> real=2.87 secs] >>> 62477.137: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8971068 secs] [Times: user=2.87 sys=0.00, >>> real=2.90 secs] >>> 62480.034: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8655618 secs] [Times: user=2.86 sys=0.00, >>> real=2.86 secs] >>> 62482.901: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 3.0366140 secs] [Times: user=2.78 sys=0.00, >>> real=3.04 secs] >>> 62485.939: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8541753 secs] [Times: user=2.85 sys=0.00, >>> real=2.85 secs] >>> 62488.794: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8582816 secs] [Times: user=2.86 sys=0.00, >>> real=2.86 secs] >>> 62491.653: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8673218 secs] [Times: user=2.85 sys=0.00, >>> real=2.86 secs] >>> 62494.521: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.9014120 secs] [Times: user=2.87 sys=0.00, >>> real=2.90 secs] >>> 62497.424: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8805843 secs] [Times: user=2.89 sys=0.00, >>> real=2.89 secs] >>> 62500.305: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8905128 secs] [Times: user=2.89 sys=0.00, >>> real=2.89 secs] >>> 62503.196: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.9052007 secs] [Times: user=2.90 sys=0.00, >>> real=2.92 secs] >>> 62506.102: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.9004575 secs] [Times: user=2.89 sys=0.00, >>> real=2.90 secs] >>> 62509.003: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.9160655 secs] [Times: user=2.90 sys=0.00, >>> real=2.92 secs] >>> 62511.920: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.9013277 secs] [Times: user=2.90 sys=0.00, >>> real=2.90 secs] >>> 62514.822: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8982061 secs] [Times: user=2.89 sys=0.00, >>> real=2.89 secs] >>> 62517.721: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8922437 secs] [Times: user=2.89 sys=0.00, >>> real=2.89 secs] >>> 62520.614: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8873520 secs] [Times: user=2.87 sys=0.00, >>> real=2.89 secs] >>> 62523.502: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8805296 secs] [Times: user=2.89 sys=0.00, >>> real=2.89 secs] >>> 62526.383: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8958714 secs] [Times: user=2.85 sys=0.00, >>> real=2.89 secs] >>> 62529.279: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8735384 secs] [Times: user=2.89 sys=0.00, >>> real=2.89 secs] >>> 62532.154: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8705676 secs] [Times: user=2.87 sys=0.00, >>> real=2.87 secs] >>> 62535.025: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8723947 secs] [Times: user=2.85 sys=0.00, >>> real=2.87 secs] >>> 62537.898: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8624400 secs] [Times: user=2.86 sys=0.00, >>> real=2.86 secs] >>> 62540.761: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8245748 secs] [Times: user=2.84 sys=0.00, >>> real=2.84 secs] >>> 62543.587: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8432269 secs] [Times: user=2.84 sys=0.00, >>> real=2.84 secs] >>> 62546.432: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8394157 secs] [Times: user=2.84 sys=0.00, >>> real=2.84 secs] >>> 62549.272: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8471951 secs] [Times: user=2.85 sys=0.00, >>> real=2.85 secs] >>> 62552.121: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8584107 secs] [Times: user=2.85 sys=0.00, >>> real=2.86 secs] >>> 62554.981: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683676K->683676K(699072K)] 781916K->781916K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8376807 secs] [Times: user=2.84 sys=0.00, >>> real=2.84 secs] >>> 62557.820: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8402486 secs] [Times: user=2.84 sys=0.00, >>> real=2.84 secs] >>> 62560.661: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8482704 secs] [Times: user=2.86 sys=0.00, >>> real=2.85 secs] >>> 62563.511: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8115973 secs] [Times: user=2.81 sys=0.00, >>> real=2.81 secs] >>> 62566.324: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8523278 secs] [Times: user=2.85 sys=0.00, >>> real=2.86 secs] >>> 62569.177: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8128563 secs] [Times: user=2.81 sys=0.00, >>> real=2.81 secs] >>> 62571.990: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.7830644 secs] [Times: user=2.79 sys=0.00, >>> real=2.79 secs] >>> 62574.774: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8065106 secs] [Times: user=2.79 sys=0.00, >>> real=2.81 secs] >>> 62577.582: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.7892171 secs] [Times: user=2.79 sys=0.00, >>> real=2.79 secs] >>> 62580.372: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8059306 secs] [Times: user=2.79 sys=0.00, >>> real=2.79 secs] >>> 62583.179: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8641470 secs] [Times: user=2.82 sys=0.00, >>> real=2.86 secs] >>> 62586.044: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8421364 secs] [Times: user=2.84 sys=0.00, >>> real=2.84 secs] >>> 62588.887: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8852699 secs] [Times: user=2.89 sys=0.00, >>> real=2.89 secs] >>> 62591.773: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.9164279 secs] [Times: user=2.90 sys=0.00, >>> real=2.92 secs] >>> 62594.690: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.9450010 secs] [Times: user=2.95 sys=0.00, >>> real=2.95 secs] >>> 62597.636: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.9744636 secs] [Times: user=2.98 sys=0.00, >>> real=2.98 secs] >>> 62600.611: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.9900849 secs] [Times: user=2.99 sys=0.00, >>> real=3.00 secs] >>> 62603.602: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.6332370 secs] [Times: user=2.62 sys=0.00, >>> real=2.62 secs] >>> 62606.236: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.9801260 secs] [Times: user=2.95 sys=0.00, >>> real=2.98 secs] >>> 62609.226: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.9166374 secs] [Times: user=2.89 sys=0.00, >>> real=2.92 secs] >>> 62612.150: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.9475729 secs] [Times: user=2.95 sys=0.00, >>> real=2.95 secs] >>> 62615.098: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.9328670 secs] [Times: user=2.90 sys=0.00, >>> real=2.93 secs] >>> 62618.040: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8963825 secs] [Times: user=2.89 sys=0.00, >>> real=2.90 secs] >>> 62620.937: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8834715 secs] [Times: user=2.89 sys=0.00, >>> real=2.89 secs] >>> 62623.821: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8800691 secs] [Times: user=2.86 sys=0.00, >>> real=2.87 secs] >>> 62626.701: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683678K->683677K(699072K)] 781918K->781917K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8642587 secs] [Times: user=2.87 sys=0.00, >>> real=2.87 secs] >>> 62629.566: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683677K->683677K(699072K)] 781917K->781917K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8574615 secs] [Times: user=2.84 sys=0.00, >>> real=2.86 secs] >>> 62632.424: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8383412 secs] [Times: user=2.84 sys=0.00, >>> real=2.84 secs] >>> 62635.264: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8409891 secs] [Times: user=2.82 sys=0.00, >>> real=2.84 secs] >>> 62638.106: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.7906216 secs] [Times: user=2.79 sys=0.00, >>> real=2.79 secs] >>> 62640.898: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.7891730 secs] [Times: user=2.79 sys=0.00, >>> real=2.79 secs] >>> 62643.688: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.7892940 secs] [Times: user=2.79 sys=0.00, >>> real=2.79 secs] >>> 62646.479: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.7766807 secs] [Times: user=2.78 sys=0.00, >>> real=2.78 secs] >>> 62649.257: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.7796531 secs] [Times: user=2.78 sys=0.00, >>> real=2.78 secs] >>> 62652.037: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.7687240 secs] [Times: user=2.76 sys=0.00, >>> real=2.76 secs] >>> 62654.807: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.7613769 secs] [Times: user=2.76 sys=0.00, >>> real=2.76 secs] >>> 62657.570: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.7712254 secs] [Times: user=2.78 sys=0.00, >>> real=2.78 secs] >>> 62660.342: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.7968108 secs] [Times: user=2.79 sys=0.00, >>> real=2.79 secs] >>> 62663.139: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.7924173 secs] [Times: user=2.79 sys=0.00, >>> real=2.79 secs] >>> 62665.933: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8002912 secs] [Times: user=2.79 sys=0.00, >>> real=2.81 secs] >>> 62668.736: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8291434 secs] [Times: user=2.82 sys=0.00, >>> real=2.82 secs] >>> 62671.566: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8527186 secs] [Times: user=2.86 sys=0.00, >>> real=2.85 secs] >>> 62674.419: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8982825 secs] [Times: user=2.90 sys=0.00, >>> real=2.90 secs] >>> 62677.318: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.9254483 secs] [Times: user=2.93 sys=0.00, >>> real=2.93 secs] >>> 62680.244: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.9707015 secs] [Times: user=2.95 sys=0.00, >>> real=2.96 secs] >>> 62683.216: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.9894145 secs] [Times: user=3.00 sys=0.00, >>> real=3.00 secs] >>> 62686.206: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.9870305 secs] [Times: user=2.98 sys=0.00, >>> real=2.98 secs] >>> 62689.193: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.9884647 secs] [Times: user=2.98 sys=0.00, >>> real=3.00 secs] >>> 62692.183: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.9635276 secs] [Times: user=2.96 sys=0.00, >>> real=2.96 secs] >>> 62695.147: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.9407559 secs] [Times: user=2.93 sys=0.00, >>> real=2.93 secs] >>> 62698.088: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.9299386 secs] [Times: user=2.93 sys=0.00, >>> real=2.93 secs] >>> 62701.019: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8994903 secs] [Times: user=2.90 sys=0.00, >>> real=2.90 secs] >>> 62703.919: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.9163417 secs] [Times: user=2.92 sys=0.00, >>> real=2.92 secs] >>> 62706.836: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.9216473 secs] [Times: user=2.93 sys=0.00, >>> real=2.93 secs] >>> 62709.758: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.9052547 secs] [Times: user=2.89 sys=0.00, >>> real=2.90 secs] >>> 62712.664: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8902824 secs] [Times: user=2.85 sys=0.00, >>> real=2.89 secs] >>> 62715.555: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8865932 secs] [Times: user=2.87 sys=0.00, >>> real=2.89 secs] >>> 62718.442: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8605445 secs] [Times: user=2.87 sys=0.00, >>> real=2.87 secs] >>> 62721.304: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8662771 secs] [Times: user=2.84 sys=0.00, >>> real=2.86 secs] >>> 62724.171: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: >>> 683679K->683679K(699072K)] 781919K->781919K(913792K) [PSPermGen: >>> 49694K->49694K(49984K)], 2.8369076 secs] [Times: user=2.84 sys=0.00, >>> real=2.84 secs] >>> >>> >>> From Jon.Masamitsu at Sun.COM Wed Feb 25 14:42:32 2009 From: Jon.Masamitsu at Sun.COM (Jon Masamitsu) Date: Wed, 25 Feb 2009 14:42:32 -0800 Subject: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64 platform within data processing application In-Reply-To: <49A5BF2B.3030407@nuix.com> References: <499CA7D9.9040903@nuix.com> <4E0D3F42-021A-4D48-888E-594A5980D0E1@sun.com> <49A33600.7000004@nuix.com> <49A3454A.4010507@nuix.com> <49A34B69.2000608@sun.com> <49A34DF2.2010800@sun.com> <49A48B14.8070307@nuix.com> <49A4DE63.5030704@sun.com> <6ed08c5e5b2a.49a49322@sun.com> <49A5BF2B.3030407@nuix.com> Message-ID: <49A5C958.8070704@sun.com> David, This is an educated guess but I would say that no out-of-memory was thrown because there was still significant space in the young gen after a collection. 62712.664: [Full GC [PSYoungGen: 98240K->98240K(214720K)] [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) [PSPermGen: The young gen capacity being 214720K and the used space in the young gen being 98240K. That missing space may be in the survivor spaces which are not directly available to the application with the UseParallelGC collector. The logic that would throw the out-of-memory is very conservative and probably does not allow for such a case. Jon David Sitsky wrote On 02/25/09 13:59,: >Hi Ramki, > >The next message I posted to the hotspot list showed a GC trace when I >allocated more memory (an extra 300 megs), where everything worked fine. > I am well aware that the heap size for this particular application and >data set was too small. My run from last night with the extra heap is >still running nicely. > >I reported this issue, because it seemed to me no progress was being >made and no OutOfMemoryErrors were being generated. The application was >effectively "stuck" making no progress at all. I couldn't even connect >to it with jconsole, although jstack worked fine. > >My understanding is this condition is meant to be detected, and >OutOfMemoryError is meant to be thrown, but perhaps I am mistaken? This >is with the parallel GC. > >Cheers, >David > >Y Srinivas Ramakrishna wrote: > > >>Doesn't the heap look too full? >>If a 64-bit JVM why use such an oversubscribed >>and small heap? Either make the old gen bigger or make the >>young gen smaller (giving that space to the older gen) >>so that each scavenge does not degenerate to a full gc >>as in your trace below. >> >>This discussion probably belongs on hotspot-gc-use at o.j.n list >>so i have cross-posted over to that list with a bcc to >>the hotspot-dev list. >> >>Also the GC tuning guides to be found here might be useful >>reading:- >> >>http://java.sun.com/javase/technologies/hotspot/gc/index.jsp >> >>-- ramki >> >>----- Original Message ----- >>From: Jon Masamitsu >>Date: Tuesday, February 24, 2009 10:00 pm >>Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64 platform within data processing application >>To: David Sitsky >>Cc: hotspot-dev at openjdk.java.net, Tom Rodriguez >> >> >> >> >>>David, >>> >>>Can you also send a GC log from a run where there >>>is not a problem? As I understand it, that would >>>be a 32bit run. >>> >>>Jon >>> >>>David Sitsky wrote On 02/24/09 16:04,: >>> >>> >>> >>>>Jon Masamitsu wrote: >>>> >>>> >>>> >>>> >>>>>Jon Masamitsu wrote On 02/23/09 17:20,: >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>>... >>>>>> >>>>>>Increase the heap by 30%. Also increase the the perm gen size >>>>>>(-XX:MaxPermSize=). >>>>>> >>>>>>Please use -XX:+PrintGCDetails -XX:+PrintGCTimeStamps when >>>>>> >>>>>> >>>gathering the >>> >>> >>>>>>GC logs. >>>>>>If you've already gathering some, send those but in future runs, >>>>>> >>>>>> >>>use the >>> >>> >>>>>>above. >>>>>> >>>>>> >>>>>> >>>>>> >>>>Here is a sample of output from a stuck process. You can see its >>>> >>>> >>>doing >>> >>> >>>>a full GC about every 3 seconds, and it seems as if there is little >>>>progress.. >>>> >>>>Please let me know if you need more information. >>>> >>>>Cheers, >>>>David >>>> >>>> >>>> >>>> >>>> >>>> >>>> > >_______________________________________________ >hotspot-gc-use mailing list >hotspot-gc-use at openjdk.java.net >http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > From Y.S.Ramakrishna at Sun.COM Wed Feb 25 20:41:10 2009 From: Y.S.Ramakrishna at Sun.COM (Y Srinivas Ramakrishna) Date: Wed, 25 Feb 2009 20:41:10 -0800 Subject: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64 platform within data processing application In-Reply-To: <49A5BF2B.3030407@nuix.com> References: <499CA7D9.9040903@nuix.com> <4E0D3F42-021A-4D48-888E-594A5980D0E1@sun.com> <49A33600.7000004@nuix.com> <49A3454A.4010507@nuix.com> <49A34B69.2000608@sun.com> <49A34DF2.2010800@sun.com> <49A48B14.8070307@nuix.com> <49A4DE63.5030704@sun.com> <6ed08c5e5b2a.49a49322@sun.com> <49A5BF2B.3030407@nuix.com> Message-ID: <71e0b0454a6.49a5ace6@sun.com> Yes, sorry, my bad. I think Jon can explain that the trace you included will show metrics (GC overhead and Space free(d)) that somehow fall below the thresholds that trigger the GCOverhead related OOM. These thresholds can of course be modified via suitable JVM options. -- ramki ----- Original Message ----- From: David Sitsky Date: Wednesday, February 25, 2009 1:59 pm Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64 platform within data processing application To: Y Srinivas Ramakrishna Cc: hotspot-gc-use at openjdk.java.net > Hi Ramki, > > The next message I posted to the hotspot list showed a GC trace when I > allocated more memory (an extra 300 megs), where everything worked > fine. I am well aware that the heap size for this particular > application and data set was too small. My run from last night with > the extra heap is still running nicely. > > I reported this issue, because it seemed to me no progress was being > made and no OutOfMemoryErrors were being generated. The application > was effectively "stuck" making no progress at all. I couldn't even > connect to it with jconsole, although jstack worked fine. > > My understanding is this condition is meant to be detected, and > OutOfMemoryError is meant to be thrown, but perhaps I am mistaken? > This is with the parallel GC. > > Cheers, > David > > Y Srinivas Ramakrishna wrote: > >Doesn't the heap look too full? > >If a 64-bit JVM why use such an oversubscribed > >and small heap? Either make the old gen bigger or make the > >young gen smaller (giving that space to the older gen) > >so that each scavenge does not degenerate to a full gc > >as in your trace below. > > > >This discussion probably belongs on hotspot-gc-use at o.j.n list > >so i have cross-posted over to that list with a bcc to > >the hotspot-dev list. > > > >Also the GC tuning guides to be found here might be useful > >reading:- > > > >http://java.sun.com/javase/technologies/hotspot/gc/index.jsp > > > >-- ramki > > > >----- Original Message ----- > >From: Jon Masamitsu > >Date: Tuesday, February 24, 2009 10:00 pm > >Subject: Re: 100% CPU usage in "VM Thread" for Hotspot 10/11 on x64 > platform within data processing application > >To: David Sitsky > >Cc: hotspot-dev at openjdk.java.net, Tom Rodriguez > > > > > >>David, > >> > >>Can you also send a GC log from a run where there > >>is not a problem? As I understand it, that would > >>be a 32bit run. > >> > >>Jon > >> > >>David Sitsky wrote On 02/24/09 16:04,: > >> > >>>Jon Masamitsu wrote: > >>> > >>>>Jon Masamitsu wrote On 02/23/09 17:20,: > >>>> > >>>> > >>>>>... > >>>>> > >>>>>Increase the heap by 30%. Also increase the the perm gen size > >>>>>(-XX:MaxPermSize=). > >>>>> > >>>>>Please use -XX:+PrintGCDetails -XX:+PrintGCTimeStamps when > >>gathering the > >>>>>GC logs. > >>>>>If you've already gathering some, send those but in future runs, > > >>use the > >>>>>above. > >>>>> > >>>Here is a sample of output from a stuck process. You can see its > >>doing > >>>a full GC about every 3 seconds, and it seems as if there is little > progress.. > >>> > >>>Please let me know if you need more information. > >>> > >>>Cheers, > >>>David > >>> > >>>62402.320: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8063178 secs] [Times: user=2.81 > sys=0.00, real=2.81 secs] > >>>62405.128: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8029997 secs] [Times: user=2.79 > sys=0.00, real=2.81 secs] > >>>62407.932: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.7917325 secs] [Times: user=2.79 > sys=0.00, real=2.79 secs] > >>>62410.725: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.7891387 secs] [Times: user=2.79 > sys=0.00, real=2.79 secs] > >>>62413.515: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.7649110 secs] [Times: user=2.76 > sys=0.00, real=2.76 secs] > >>>62416.281: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.7803983 secs] [Times: user=2.78 > sys=0.00, real=2.78 secs] > >>>62419.063: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.7643979 secs] [Times: user=2.76 > sys=0.00, real=2.76 secs] > >>>62421.828: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8114336 secs] [Times: user=2.81 > sys=0.00, real=2.81 secs] > >>>62424.640: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.7964912 secs] [Times: user=2.79 > sys=0.00, real=2.79 secs] > >>>62427.438: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8107278 secs] [Times: user=2.81 > sys=0.00, real=2.81 secs] > >>>62430.249: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 3.2345212 secs] [Times: user=2.84 > sys=0.00, real=3.24 secs] > >>>62433.484: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8341520 secs] [Times: user=2.82 > sys=0.00, real=2.82 secs] > >>>62436.319: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8698768 secs] [Times: user=2.87 > sys=0.00, real=2.87 secs] > >>>62439.190: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.9323230 secs] [Times: user=2.90 > sys=0.00, real=2.92 secs] > >>>62442.124: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.9644960 secs] [Times: user=2.96 > sys=0.00, real=2.96 secs] > >>>62445.089: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 3.0059221 secs] [Times: user=3.00 > sys=0.00, real=3.00 secs] > >>>62448.095: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.9832815 secs] [Times: user=3.00 > sys=0.00, real=2.99 secs] > >>>62451.079: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.9587156 secs] [Times: user=2.93 > sys=0.00, real=2.95 secs] > >>>62454.039: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.9488345 secs] [Times: user=2.92 > sys=0.00, real=2.95 secs] > >>>62456.988: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8969788 secs] [Times: user=2.90 > sys=0.00, real=2.90 secs] > >>>62459.886: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8794991 secs] [Times: user=2.89 > sys=0.00, real=2.89 secs] > >>>62462.766: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683675K->683675K(699072K)] 781915K->781915K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8842411 secs] [Times: user=2.87 > sys=0.00, real=2.89 secs] > >>>62465.651: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683676K->683675K(699072K)] 781916K->781915K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8669173 secs] [Times: user=2.85 > sys=0.00, real=2.85 secs] > >>>62468.519: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8664429 secs] [Times: user=2.85 > sys=0.00, real=2.86 secs] > >>>62471.386: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8844494 secs] [Times: user=2.87 > sys=0.00, real=2.89 secs] > >>>62474.271: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8648398 secs] [Times: user=2.87 > sys=0.00, real=2.87 secs] > >>>62477.137: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8971068 secs] [Times: user=2.87 > sys=0.00, real=2.90 secs] > >>>62480.034: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8655618 secs] [Times: user=2.86 > sys=0.00, real=2.86 secs] > >>>62482.901: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 3.0366140 secs] [Times: user=2.78 > sys=0.00, real=3.04 secs] > >>>62485.939: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8541753 secs] [Times: user=2.85 > sys=0.00, real=2.85 secs] > >>>62488.794: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8582816 secs] [Times: user=2.86 > sys=0.00, real=2.86 secs] > >>>62491.653: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8673218 secs] [Times: user=2.85 > sys=0.00, real=2.86 secs] > >>>62494.521: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.9014120 secs] [Times: user=2.87 > sys=0.00, real=2.90 secs] > >>>62497.424: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8805843 secs] [Times: user=2.89 > sys=0.00, real=2.89 secs] > >>>62500.305: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8905128 secs] [Times: user=2.89 > sys=0.00, real=2.89 secs] > >>>62503.196: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.9052007 secs] [Times: user=2.90 > sys=0.00, real=2.92 secs] > >>>62506.102: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.9004575 secs] [Times: user=2.89 > sys=0.00, real=2.90 secs] > >>>62509.003: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.9160655 secs] [Times: user=2.90 > sys=0.00, real=2.92 secs] > >>>62511.920: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.9013277 secs] [Times: user=2.90 > sys=0.00, real=2.90 secs] > >>>62514.822: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8982061 secs] [Times: user=2.89 > sys=0.00, real=2.89 secs] > >>>62517.721: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8922437 secs] [Times: user=2.89 > sys=0.00, real=2.89 secs] > >>>62520.614: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8873520 secs] [Times: user=2.87 > sys=0.00, real=2.89 secs] > >>>62523.502: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8805296 secs] [Times: user=2.89 > sys=0.00, real=2.89 secs] > >>>62526.383: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8958714 secs] [Times: user=2.85 > sys=0.00, real=2.89 secs] > >>>62529.279: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8735384 secs] [Times: user=2.89 > sys=0.00, real=2.89 secs] > >>>62532.154: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8705676 secs] [Times: user=2.87 > sys=0.00, real=2.87 secs] > >>>62535.025: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8723947 secs] [Times: user=2.85 > sys=0.00, real=2.87 secs] > >>>62537.898: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8624400 secs] [Times: user=2.86 > sys=0.00, real=2.86 secs] > >>>62540.761: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8245748 secs] [Times: user=2.84 > sys=0.00, real=2.84 secs] > >>>62543.587: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8432269 secs] [Times: user=2.84 > sys=0.00, real=2.84 secs] > >>>62546.432: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8394157 secs] [Times: user=2.84 > sys=0.00, real=2.84 secs] > >>>62549.272: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8471951 secs] [Times: user=2.85 > sys=0.00, real=2.85 secs] > >>>62552.121: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8584107 secs] [Times: user=2.85 > sys=0.00, real=2.86 secs] > >>>62554.981: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683676K->683676K(699072K)] 781916K->781916K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8376807 secs] [Times: user=2.84 > sys=0.00, real=2.84 secs] > >>>62557.820: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8402486 secs] [Times: user=2.84 > sys=0.00, real=2.84 secs] > >>>62560.661: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8482704 secs] [Times: user=2.86 > sys=0.00, real=2.85 secs] > >>>62563.511: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8115973 secs] [Times: user=2.81 > sys=0.00, real=2.81 secs] > >>>62566.324: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8523278 secs] [Times: user=2.85 > sys=0.00, real=2.86 secs] > >>>62569.177: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8128563 secs] [Times: user=2.81 > sys=0.00, real=2.81 secs] > >>>62571.990: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.7830644 secs] [Times: user=2.79 > sys=0.00, real=2.79 secs] > >>>62574.774: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8065106 secs] [Times: user=2.79 > sys=0.00, real=2.81 secs] > >>>62577.582: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.7892171 secs] [Times: user=2.79 > sys=0.00, real=2.79 secs] > >>>62580.372: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8059306 secs] [Times: user=2.79 > sys=0.00, real=2.79 secs] > >>>62583.179: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8641470 secs] [Times: user=2.82 > sys=0.00, real=2.86 secs] > >>>62586.044: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8421364 secs] [Times: user=2.84 > sys=0.00, real=2.84 secs] > >>>62588.887: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8852699 secs] [Times: user=2.89 > sys=0.00, real=2.89 secs] > >>>62591.773: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.9164279 secs] [Times: user=2.90 > sys=0.00, real=2.92 secs] > >>>62594.690: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.9450010 secs] [Times: user=2.95 > sys=0.00, real=2.95 secs] > >>>62597.636: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.9744636 secs] [Times: user=2.98 > sys=0.00, real=2.98 secs] > >>>62600.611: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.9900849 secs] [Times: user=2.99 > sys=0.00, real=3.00 secs] > >>>62603.602: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.6332370 secs] [Times: user=2.62 > sys=0.00, real=2.62 secs] > >>>62606.236: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.9801260 secs] [Times: user=2.95 > sys=0.00, real=2.98 secs] > >>>62609.226: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.9166374 secs] [Times: user=2.89 > sys=0.00, real=2.92 secs] > >>>62612.150: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.9475729 secs] [Times: user=2.95 > sys=0.00, real=2.95 secs] > >>>62615.098: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.9328670 secs] [Times: user=2.90 > sys=0.00, real=2.93 secs] > >>>62618.040: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8963825 secs] [Times: user=2.89 > sys=0.00, real=2.90 secs] > >>>62620.937: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8834715 secs] [Times: user=2.89 > sys=0.00, real=2.89 secs] > >>>62623.821: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8800691 secs] [Times: user=2.86 > sys=0.00, real=2.87 secs] > >>>62626.701: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683678K->683677K(699072K)] 781918K->781917K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8642587 secs] [Times: user=2.87 > sys=0.00, real=2.87 secs] > >>>62629.566: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683677K->683677K(699072K)] 781917K->781917K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8574615 secs] [Times: user=2.84 > sys=0.00, real=2.86 secs] > >>>62632.424: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8383412 secs] [Times: user=2.84 > sys=0.00, real=2.84 secs] > >>>62635.264: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8409891 secs] [Times: user=2.82 > sys=0.00, real=2.84 secs] > >>>62638.106: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.7906216 secs] [Times: user=2.79 > sys=0.00, real=2.79 secs] > >>>62640.898: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.7891730 secs] [Times: user=2.79 > sys=0.00, real=2.79 secs] > >>>62643.688: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.7892940 secs] [Times: user=2.79 > sys=0.00, real=2.79 secs] > >>>62646.479: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.7766807 secs] [Times: user=2.78 > sys=0.00, real=2.78 secs] > >>>62649.257: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.7796531 secs] [Times: user=2.78 > sys=0.00, real=2.78 secs] > >>>62652.037: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.7687240 secs] [Times: user=2.76 > sys=0.00, real=2.76 secs] > >>>62654.807: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.7613769 secs] [Times: user=2.76 > sys=0.00, real=2.76 secs] > >>>62657.570: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.7712254 secs] [Times: user=2.78 > sys=0.00, real=2.78 secs] > >>>62660.342: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.7968108 secs] [Times: user=2.79 > sys=0.00, real=2.79 secs] > >>>62663.139: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.7924173 secs] [Times: user=2.79 > sys=0.00, real=2.79 secs] > >>>62665.933: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8002912 secs] [Times: user=2.79 > sys=0.00, real=2.81 secs] > >>>62668.736: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8291434 secs] [Times: user=2.82 > sys=0.00, real=2.82 secs] > >>>62671.566: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8527186 secs] [Times: user=2.86 > sys=0.00, real=2.85 secs] > >>>62674.419: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8982825 secs] [Times: user=2.90 > sys=0.00, real=2.90 secs] > >>>62677.318: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.9254483 secs] [Times: user=2.93 > sys=0.00, real=2.93 secs] > >>>62680.244: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.9707015 secs] [Times: user=2.95 > sys=0.00, real=2.96 secs] > >>>62683.216: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.9894145 secs] [Times: user=3.00 > sys=0.00, real=3.00 secs] > >>>62686.206: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.9870305 secs] [Times: user=2.98 > sys=0.00, real=2.98 secs] > >>>62689.193: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.9884647 secs] [Times: user=2.98 > sys=0.00, real=3.00 secs] > >>>62692.183: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.9635276 secs] [Times: user=2.96 > sys=0.00, real=2.96 secs] > >>>62695.147: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.9407559 secs] [Times: user=2.93 > sys=0.00, real=2.93 secs] > >>>62698.088: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.9299386 secs] [Times: user=2.93 > sys=0.00, real=2.93 secs] > >>>62701.019: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8994903 secs] [Times: user=2.90 > sys=0.00, real=2.90 secs] > >>>62703.919: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.9163417 secs] [Times: user=2.92 > sys=0.00, real=2.92 secs] > >>>62706.836: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.9216473 secs] [Times: user=2.93 > sys=0.00, real=2.93 secs] > >>>62709.758: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.9052547 secs] [Times: user=2.89 > sys=0.00, real=2.90 secs] > >>>62712.664: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8902824 secs] [Times: user=2.85 > sys=0.00, real=2.89 secs] > >>>62715.555: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8865932 secs] [Times: user=2.87 > sys=0.00, real=2.89 secs] > >>>62718.442: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8605445 secs] [Times: user=2.87 > sys=0.00, real=2.87 secs] > >>>62721.304: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683678K->683678K(699072K)] 781918K->781918K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8662771 secs] [Times: user=2.84 > sys=0.00, real=2.86 secs] > >>>62724.171: [Full GC [PSYoungGen: 98240K->98240K(214720K)] > [PSOldGen: 683679K->683679K(699072K)] 781919K->781919K(913792K) > [PSPermGen: 49694K->49694K(49984K)], 2.8369076 secs] [Times: user=2.84 > sys=0.00, real=2.84 secs] > >>> > >>> >