From fengtao04 at gmail.com Wed May 11 06:30:20 2016 From: fengtao04 at gmail.com (Tao Feng) Date: Tue, 10 May 2016 23:30:20 -0700 Subject: Long GC pause(promotion failure) in CMS when heap is just half used Message-ID: Hi, We encounter a long GC pause in one of our production service and would like to seek your suggestion on what's happening here. I understand that heap could be very fragmented which object could encounter promote failure even the heap still has plenty of space. But in this case the heap is just about one half usage which I kinda doubt the heap could be that fragmented... Any suggetions of how to further debug would be appreciated. Thanks, -Tao ///////////JVM flag///////// CommandLine flags: -XX:+AlwaysPreTouch -XX:CMSInitiatingOccupancyFraction=85 -XX:ErrorFile=logs/hs_err.log -XX:GCLogFileSize=26214400 -XX:+HeapDumpOnOutOfMemoryError -XX:InitialHeapSize=17179869184 -XX:+ManagementServer -XX:MaxHeapSize=17179869184 -XX:MaxMetaspaceSize=134217728 -XX:MaxNewSize=805306368 -XX:MaxTenuringThreshold=15 -XX:MetaspaceSize=134217728 -XX:MinHeapDeltaBytes=196608 -XX:NewSize=805306368 -XX:NumberOfGCLogFiles=40 -XX:OldPLABSize=16 -XX:OldSize=16374562816 -XX:+ParallelRefProcEnabled -XX:+PrintGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:SurvivorRatio=8 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseGCLogFileRotation -XX:+UseParNewGC ////////////GC Log////////////// 2016-05-09T13:20:43.984+0000: 420349.714: [GC (Allocation Failure) 420349.714: [ParNew Desired survivor size 40239104 bytes, new threshold 2 (max 15) - age 1: 26070160 bytes, 26070160 total - age 2: 15543856 bytes, 41614016 total - age 3: 10532152 bytes, 52146168 total : 686280K->57692K(707840K), 0.0375021 secs] 7482403K->6863648K(16698624K), 0.0378490 secs] [Times: user=0.38 sys=0.03, real=0.03 secs] 2016-05-09T13:20:44.022+0000: 420349.753: Total time for which application threads were stopped: 0.0427562 seconds 2016-05-09T13:20:44.939+0000: 420350.669: [GC (Allocation Failure) 420350.670: [ParNew Desired survivor size 40239104 bytes, new threshold 15 (max 15) - age 1: 27280264 bytes, 27280264 total - age 2: 11179688 bytes, 38459952 total : 686569K->50544K(707840K), 0.0644130 secs] 7492525K->6881668K(16698624K), 0.0648059 secs] [Times: user=0.50 sys=0.09, real=0.07 secs] 2016-05-09T13:20:45.004+0000: 420350.735: Total time for which application threads were stopped: 0.0700694 seconds 2016-05-09T13:20:45.653+0000: 420351.383: [GC (Allocation Failure) 420351.383: [ParNew Desired survivor size 40239104 bytes, new threshold 3 (max 15) - age 1: 25456392 bytes, 25456392 total - age 2: 11812240 bytes, 37268632 total - age 3: 10727648 bytes, 47996280 total : 679424K->58946K(707840K), 0.0252806 secs] 7510548K->6890070K(16698624K), 0.0256222 secs] [Times: user=0.30 sys=0.01, real=0.02 secs] 2016-05-09T13:20:45.679+0000: 420351.409: Total time for which application threads were stopped: 0.0302293 seconds 2016-05-09T13:20:46.632+0000: 420352.363: [GC (Allocation Failure) 420352.363: [ParNew (*promotion failed*): 687696K->681421K(707840K), 0.3959862 secs]420352.759: [CMS: *6833071K*->751692K(*15990784K*), 28.4273576 secs] 7518819K->751692K(16698624K), [Metaspace: 73074K->73074K(75776K)], 28.8237795 secs] [Times: user=7.49 sys=4.13, real= *28.82* secs] 2016-05-09T13:21:15.456+0000: 420381.187: Total time for which application threads were stopped: *28.8286392* seconds -------------- next part -------------- An HTML attachment was scrubbed... URL: From Corbin.Hughes at Cerner.com Fri May 27 20:07:35 2016 From: Corbin.Hughes at Cerner.com (Hughes,Corbin) Date: Fri, 27 May 2016 20:07:35 +0000 Subject: Unexpected Java 6 vs 8 GC performance difference Message-ID: <2542380347364af1917a71d63f4b00b1@CERNMSGLS5EX02B.cerner.net> Hello, We're having some issues with large GC performance discrepancies between Java 6 and Java 7 or 8, and I'm hoping someone can help shed some light. The application in question loads in about 1.5 GB of immutable caches, then, while this sits in the OldGen, we have requests come in that generate huge amounts of very short lived objects (~1 GB or so per request) that fill up the eden space and then get cleaned up. It's very rare that anything will last more than 1 GC cycle. Usually 2-10MB or so goes into the survivor space, and then nothing makes it past survivor through enough GC cycles to get into the OldGen. This current setup in Java 6 is pretty much ideal for us: young gen collections are very fast, and other than the initial cache loading, we never do full GC. The problem is that we're seeing ~10 ms PSYoungGen collections in Java 6, but ~200 ms PSYoungGen collections in Java 7 and 8. I thought for a while that it might have been due to extra garbage being generated via the String.substring changes or something of that nature, but according to the GC logs, over 1500 requests, Java 6 and Java 8 are collecting equal amounts (or rather less than a fraction of a percent different) of memory. Furthermore, the two different versions of Java are even making very similar amounts of PSYoungGen collections: 759 for Java 6, 744 for Java 8. Everything I can see seems to be the same: number of GCs, amount of garbage collected, etc. The only difference is that the collections take longer in Java 7 and Java 8, and I'm struggling to figure out why. Though other versions of Java exhibit the same behavior, we're using JDK 6u101 and JDK 8u92 with the following settings: Java 6: -Xms5g -Xmx5g -XX:NewSize=2g -XX:MaxNewSize=2g -XX:PermSize=800m \ -XX:MaxPermSize=1g -XX:StringTableSize=5834701 Java 8: -Xms6g -Xmx6g -XX:NewSize=2g -XX:MaxNewSize=2g -XX:MaxPermSize=1g \ -Djava.util.Arrays.useLegacyMergeSort=true -XX:StringTableSize=5834701 Since interned strings no longer go in the PermGen, the OldGen size is upped a bit in Java 8 to compensate. We have tested with the exact same settings though since the OldGen can technically fit the interned strings, and results are the same. We've also tested various young gen sizes for Java 8, and oddly enough, massively increasing the young gen practically makes no difference in collection time. Whether the young gen is 1 GB, 2 GB or 16 GB, the collections still take 200 ms. We've also noticed that the bottleneck appears to be on a single thread since the real time and the user time are very nearly equal. This implies to us that something serial is happening that is not proportional to the size of the young gen but of something else, something that's staying constant. To analyze GC logs, we've been using GCViewer along with the following settings: -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution \ -XX:+PrintHeapAtGC -Xloggc:$TOMCAT_HOME/logs/gc-$(date +%s).log \ -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M In short, we're hoping someone might be able to help us ascertain what this high cost thing is that we're seeing in the Java 8 collections. No matter what settings we've tried or how many attempts at tuning we've made, we just can't get PSYoungGen to run in less than 200 ms, and we're stumped. A few GCs from each Java version are included below. ---------- Java 6 ---------- {Heap before GC invocations=612 (full 1): PSYoungGen total 2090240K, used 2086987K [0x0000000780000000, 0x0000000800000000, 0x0000000800000000) eden space 2083008K, 100% used [0x0000000780000000,0x00000007ff230000,0x00000007ff230000) from space 7232K, 55% used [0x00000007ff8f0000,0x00000007ffcd2cc8,0x0000000800000000) to space 6912K, 0% used [0x00000007ff230000,0x00000007ff230000,0x00000007ff8f0000) PSOldGen total 3145728K, used 1438842K [0x00000006c0000000, 0x0000000780000000, 0x0000000780000000) object space 3145728K, 45% used [0x00000006c0000000,0x0000000717d1e8f8,0x0000000780000000) PSPermGen total 1048576K, used 596997K [0x0000000680000000, 0x00000006c0000000, 0x00000006c0000000) object space 1048576K, 56% used [0x0000000680000000,0x00000006a4701508,0x00000006c0000000) 1426.874: [GC Desired survivor size 7077888 bytes, new threshold 15 (max 15) [PSYoungGen: 2086987K->2558K(2089920K)] 3525829K->1442904K(5235648K), 0.0079930 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] Heap after GC invocations=612 (full 1): PSYoungGen total 2089920K, used 2558K [0x0000000780000000, 0x0000000800000000, 0x0000000800000000) eden space 2083008K, 0% used [0x0000000780000000,0x0000000780000000,0x00000007ff230000) from space 6912K, 37% used [0x00000007ff230000,0x00000007ff4af9e8,0x00000007ff8f0000) to space 6912K, 0% used [0x00000007ff940000,0x00000007ff940000,0x0000000800000000) PSOldGen total 3145728K, used 1440346K [0x00000006c0000000, 0x0000000780000000, 0x0000000780000000) object space 3145728K, 45% used [0x00000006c0000000,0x0000000717e969a0,0x0000000780000000) PSPermGen total 1048576K, used 596997K [0x0000000680000000, 0x00000006c0000000, 0x00000006c0000000) object space 1048576K, 56% used [0x0000000680000000,0x00000006a4701508,0x00000006c0000000) } {Heap before GC invocations=613 (full 1): PSYoungGen total 2089920K, used 2085566K [0x0000000780000000, 0x0000000800000000, 0x0000000800000000) eden space 2083008K, 100% used [0x0000000780000000,0x00000007ff230000,0x00000007ff230000) from space 6912K, 37% used [0x00000007ff230000,0x00000007ff4af9e8,0x00000007ff8f0000) to space 6912K, 0% used [0x00000007ff940000,0x00000007ff940000,0x0000000800000000) PSOldGen total 3145728K, used 1440346K [0x00000006c0000000, 0x0000000780000000, 0x0000000780000000) object space 3145728K, 45% used [0x00000006c0000000,0x0000000717e969a0,0x0000000780000000) PSPermGen total 1048576K, used 596997K [0x0000000680000000, 0x00000006c0000000, 0x00000006c0000000) object space 1048576K, 56% used [0x0000000680000000,0x00000006a4701570,0x00000006c0000000) 1429.269: [GC Desired survivor size 6815744 bytes, new threshold 15 (max 15) [PSYoungGen: 2085566K->3488K(2090496K)] 3525912K->1443866K(5236224K), 0.0050560 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] Heap after GC invocations=613 (full 1): PSYoungGen total 2090496K, used 3488K [0x0000000780000000, 0x0000000800000000, 0x0000000800000000) eden space 2083584K, 0% used [0x0000000780000000,0x0000000780000000,0x00000007ff2c0000) from space 6912K, 50% used [0x00000007ff940000,0x00000007ffca8000,0x0000000800000000) to space 6656K, 0% used [0x00000007ff2c0000,0x00000007ff2c0000,0x00000007ff940000) PSOldGen total 3145728K, used 1440378K [0x00000006c0000000, 0x0000000780000000, 0x0000000780000000) object space 3145728K, 45% used [0x00000006c0000000,0x0000000717e9e9a0,0x0000000780000000) PSPermGen total 1048576K, used 596997K [0x0000000680000000, 0x00000006c0000000, 0x00000006c0000000) object space 1048576K, 56% used [0x0000000680000000,0x00000006a4701570,0x00000006c0000000) } {Heap before GC invocations=614 (full 1): PSYoungGen total 2090496K, used 2086981K [0x0000000780000000, 0x0000000800000000, 0x0000000800000000) eden space 2083584K, 99% used [0x0000000780000000,0x00000007ff2a9640,0x00000007ff2c0000) from space 6912K, 50% used [0x00000007ff940000,0x00000007ffca8000,0x0000000800000000) to space 6656K, 0% used [0x00000007ff2c0000,0x00000007ff2c0000,0x00000007ff940000) PSOldGen total 3145728K, used 1440378K [0x00000006c0000000, 0x0000000780000000, 0x0000000780000000) object space 3145728K, 45% used [0x00000006c0000000,0x0000000717e9e9a0,0x0000000780000000) PSPermGen total 1048576K, used 596997K [0x0000000680000000, 0x00000006c0000000, 0x00000006c0000000) object space 1048576K, 56% used [0x0000000680000000,0x00000006a4701570,0x00000006c0000000) 1431.623: [GC Desired survivor size 7012352 bytes, new threshold 15 (max 15) [PSYoungGen: 2086981K->1344K(2090240K)] 3527359K->1441738K(5235968K), 0.0034930 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] Heap after GC invocations=614 (full 1): PSYoungGen total 2090240K, used 1344K [0x0000000780000000, 0x0000000800000000, 0x0000000800000000) eden space 2083584K, 0% used [0x0000000780000000,0x0000000780000000,0x00000007ff2c0000) from space 6656K, 20% used [0x00000007ff2c0000,0x00000007ff410000,0x00000007ff940000) to space 6848K, 0% used [0x00000007ff950000,0x00000007ff950000,0x0000000800000000) PSOldGen total 3145728K, used 1440394K [0x00000006c0000000, 0x0000000780000000, 0x0000000780000000) object space 3145728K, 45% used [0x00000006c0000000,0x0000000717ea29a0,0x0000000780000000) PSPermGen total 1048576K, used 596997K [0x0000000680000000, 0x00000006c0000000, 0x00000006c0000000) object space 1048576K, 56% used [0x0000000680000000,0x00000006a4701570,0x00000006c0000000) } ---------- Java 8 ---------- {Heap before GC invocations=139 (full 2): PSYoungGen total 2092032K, used 2088832K [0x0000000740000000, 0x00000007c0000000, 0x00000007c0000000) eden space 2086912K, 100% used [0x0000000740000000,0x00000007bf600000,0x00000007bf600000) from space 5120K, 37% used [0x00000007bfb00000,0x00000007bfce0000,0x00000007c0000000) to space 5120K, 0% used [0x00000007bf600000,0x00000007bf600000,0x00000007bfb00000) ParOldGen total 4194304K, used 2114679K [0x0000000640000000, 0x0000000740000000, 0x0000000740000000) object space 4194304K, 50% used [0x0000000640000000,0x00000006c111dc38,0x0000000740000000) Metaspace used 58033K, capacity 58834K, committed 59056K, reserved 1101824K class space used 6754K, capacity 6987K, committed 7088K, reserved 1048576K 1378.385: [GC (Allocation Failure) Desired survivor size 4718592 bytes, new threshold 1 (max 15) [PSYoungGen: 2088832K->1632K(2092032K)] 4203511K->2116327K(6286336K), 0.1997693 secs] [Times: user=0.22 sys=0.00, real=0.20 secs] Heap after GC invocations=139 (full 2): PSYoungGen total 2092032K, used 1632K [0x0000000740000000, 0x00000007c0000000, 0x00000007c0000000) eden space 2086912K, 0% used [0x0000000740000000,0x0000000740000000,0x00000007bf600000) from space 5120K, 31% used [0x00000007bf600000,0x00000007bf798000,0x00000007bfb00000) to space 4608K, 0% used [0x00000007bfb80000,0x00000007bfb80000,0x00000007c0000000) ParOldGen total 4194304K, used 2114695K [0x0000000640000000, 0x0000000740000000, 0x0000000740000000) object space 4194304K, 50% used [0x0000000640000000,0x00000006c1121c38,0x0000000740000000) Metaspace used 58033K, capacity 58834K, committed 59056K, reserved 1101824K class space used 6754K, capacity 6987K, committed 7088K, reserved 1048576K } {Heap before GC invocations=140 (full 2): PSYoungGen total 2092032K, used 2088544K [0x0000000740000000, 0x00000007c0000000, 0x00000007c0000000) eden space 2086912K, 100% used [0x0000000740000000,0x00000007bf600000,0x00000007bf600000) from space 5120K, 31% used [0x00000007bf600000,0x00000007bf798000,0x00000007bfb00000) to space 4608K, 0% used [0x00000007bfb80000,0x00000007bfb80000,0x00000007c0000000) ParOldGen total 4194304K, used 2114695K [0x0000000640000000, 0x0000000740000000, 0x0000000740000000) object space 4194304K, 50% used [0x0000000640000000,0x00000006c1121c38,0x0000000740000000) Metaspace used 58037K, capacity 58834K, committed 59056K, reserved 1101824K class space used 6754K, capacity 6987K, committed 7088K, reserved 1048576K 1380.588: [GC (Allocation Failure) Desired survivor size 4718592 bytes, new threshold 1 (max 15) [PSYoungGen: 2088544K->1216K(2092544K)] 4203239K->2115935K(6286848K), 0.1992978 secs] [Times: user=0.22 sys=0.00, real=0.20 secs] Heap after GC invocations=140 (full 2): PSYoungGen total 2092544K, used 1216K [0x0000000740000000, 0x00000007c0000000, 0x00000007c0000000) eden space 2087936K, 0% used [0x0000000740000000,0x0000000740000000,0x00000007bf700000) from space 4608K, 26% used [0x00000007bfb80000,0x00000007bfcb0000,0x00000007c0000000) to space 4608K, 0% used [0x00000007bf700000,0x00000007bf700000,0x00000007bfb80000) ParOldGen total 4194304K, used 2114719K [0x0000000640000000, 0x0000000740000000, 0x0000000740000000) object space 4194304K, 50% used [0x0000000640000000,0x00000006c1127c38,0x0000000740000000) Metaspace used 58037K, capacity 58834K, committed 59056K, reserved 1101824K class space used 6754K, capacity 6987K, committed 7088K, reserved 1048576K } {Heap before GC invocations=141 (full 2): PSYoungGen total 2092544K, used 2089152K [0x0000000740000000, 0x00000007c0000000, 0x00000007c0000000) eden space 2087936K, 100% used [0x0000000740000000,0x00000007bf700000,0x00000007bf700000) from space 4608K, 26% used [0x00000007bfb80000,0x00000007bfcb0000,0x00000007c0000000) to space 4608K, 0% used [0x00000007bf700000,0x00000007bf700000,0x00000007bfb80000) ParOldGen total 4194304K, used 2114719K [0x0000000640000000, 0x0000000740000000, 0x0000000740000000) object space 4194304K, 50% used [0x0000000640000000,0x00000006c1127c38,0x0000000740000000) Metaspace used 58038K, capacity 58834K, committed 59056K, reserved 1101824K class space used 6754K, capacity 6987K, committed 7088K, reserved 1048576K 1382.754: [GC (Allocation Failure) Desired survivor size 4718592 bytes, new threshold 1 (max 15) [PSYoungGen: 2089152K->2260K(2092544K)] 4203871K->2117011K(6286848K), 0.2010010 secs] [Times: user=0.23 sys=0.00, real=0.20 secs] Heap after GC invocations=141 (full 2): PSYoungGen total 2092544K, used 2260K [0x0000000740000000, 0x00000007c0000000, 0x00000007c0000000) eden space 2087936K, 0% used [0x0000000740000000,0x0000000740000000,0x00000007bf700000) from space 4608K, 49% used [0x00000007bf700000,0x00000007bf9351e0,0x00000007bfb80000) to space 4608K, 0% used [0x00000007bfb80000,0x00000007bfb80000,0x00000007c0000000) ParOldGen total 4194304K, used 2114751K [0x0000000640000000, 0x0000000740000000, 0x0000000740000000) object space 4194304K, 50% used [0x0000000640000000,0x00000006c112fc38,0x0000000740000000) Metaspace used 58038K, capacity 58834K, committed 59056K, reserved 1101824K class space used 6754K, capacity 6987K, committed 7088K, reserved 1048576K } CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jon.masamitsu at oracle.com Fri May 27 21:04:31 2016 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Fri, 27 May 2016 14:04:31 -0700 Subject: Unexpected Java 6 vs 8 GC performance difference In-Reply-To: <2542380347364af1917a71d63f4b00b1@CERNMSGLS5EX02B.cerner.net> References: <2542380347364af1917a71d63f4b00b1@CERNMSGLS5EX02B.cerner.net> Message-ID: <893d89a1-1c18-fcb4-c624-06dc7aec9891@oracle.com> Corbin, A difference I can see in the log output is that with jdk6 you are using the serial Old GC (which is the default) and with jdk8 your are using the parallel Old GC (which was changed to be the default in jdk7). On jdk8 add -XX:-UseParallelOldGC and see if it changes anything. I don't know why it would but if you run jdk8 with the serial Old GC it will give us a little better apples-to-apples comparison. Also, could you send logs jdk6 vs jdk7 next time. Lots changed in jdk8 and your problem occurs when going to jdk7 so not having to think about the jdk8 changes will help. Jon On 5/27/2016 1:07 PM, Hughes,Corbin wrote: > Hello, > We're having some issues with large GC performance discrepancies > between Java 6 > and Java 7 or 8, and I'm hoping someone can help shed some light. > The application in question loads in about 1.5 GB of immutable caches, > then, while this sits in the OldGen, we have requests come in that > generate > huge amounts of very short lived objects (~1 GB or so per request) that > fill up the eden space and then get cleaned up. It's very rare that > anything > will last more than 1 GC cycle. Usually 2-10MB or so goes into the > survivor > space, and then nothing makes it past survivor through enough GC > cycles to > get into the OldGen. > This current setup in Java 6 is pretty much ideal for us: young gen > collections > are very fast, and other than the initial cache loading, we never do > full GC. > The problem is that we're seeing ~10 ms PSYoungGen collections in > Java 6, but > ~200 ms PSYoungGen collections in Java 7 and 8. > I thought for a while that it might have been due to extra garbage being > generated via the String.substring changes or something of that > nature, but > according to the GC logs, over 1500 requests, Java 6 and Java 8 are > collecting equal amounts (or rather less than a fraction of a percent > different) of memory. Furthermore, the two different versions of Java are > even making very similar amounts of PSYoungGen collections: 759 for > Java 6, 744 > for Java 8. > Everything I can see seems to be the same: number of GCs, amount of > garbage > collected, etc. The only difference is that the collections take > longer in Java 7 > and Java 8, and I'm struggling to figure out why. > Though other versions of Java exhibit the same behavior, we're using > JDK 6u101 > and JDK 8u92 with the following settings: > Java 6: > -Xms5g -Xmx5g -XX:NewSize=2g -XX:MaxNewSize=2g -XX:PermSize=800m \ > -XX:MaxPermSize=1g -XX:StringTableSize=5834701 > Java 8: > -Xms6g -Xmx6g -XX:NewSize=2g -XX:MaxNewSize=2g -XX:MaxPermSize=1g \ > -Djava.util.Arrays.useLegacyMergeSort=true -XX:StringTableSize=5834701 > Since interned strings no longer go in the PermGen, the OldGen size is > upped a bit in Java 8 to compensate. We have tested with the exact same > settings though since the OldGen can technically fit the interned strings, > and results are the same. > We've also tested various young gen sizes for Java 8, and oddly enough, > massively increasing the young gen practically makes no difference in > collection time. Whether the young gen is 1 GB, 2 GB or 16 GB, the > collections > still take 200 ms. We've also noticed that the bottleneck appears to > be on a > single thread since the real time and the user time are very nearly > equal. This > implies to us that something serial is happening that is not > proportional to > the size of the young gen but of something else, something that's staying > constant. > To analyze GC logs, we've been using GCViewer along with the following > settings: > -XX:+PrintGCDetails -XX:+PrintGCTimeStamps > -XX:+PrintTenuringDistribution \ > -XX:+PrintHeapAtGC -Xloggc:$TOMCAT_HOME/logs/gc-$(date +%s).log \ > -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M > In short, we're hoping someone might be able to help us ascertain what > this > high cost thing is that we're seeing in the Java 8 collections. No > matter what > settings we've tried or how many attempts at tuning we've made, we > just can't > get PSYoungGen to run in less than 200 ms, and we're stumped. > A few GCs from each Java version are included below. > ---------- Java 6 ---------- > {Heap before GC invocations=612 (full 1): > PSYoungGen total 2090240K, used 2086987K [0x0000000780000000, > 0x0000000800000000, 0x0000000800000000) > eden space 2083008K, 100% used > [0x0000000780000000,0x00000007ff230000,0x00000007ff230000) > from space 7232K, 55% used > [0x00000007ff8f0000,0x00000007ffcd2cc8,0x0000000800000000) > to space 6912K, 0% used > [0x00000007ff230000,0x00000007ff230000,0x00000007ff8f0000) > PSOldGen total 3145728K, used 1438842K [0x00000006c0000000, > 0x0000000780000000, 0x0000000780000000) > object space 3145728K, 45% used > [0x00000006c0000000,0x0000000717d1e8f8,0x0000000780000000) > PSPermGen total 1048576K, used 596997K [0x0000000680000000, > 0x00000006c0000000, 0x00000006c0000000) > object space 1048576K, 56% used > [0x0000000680000000,0x00000006a4701508,0x00000006c0000000) > 1426.874: [GC > Desired survivor size 7077888 bytes, new threshold 15 (max 15) > [PSYoungGen: 2086987K->2558K(2089920K)] 3525829K->1442904K(5235648K), > 0.0079930 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] > Heap after GC invocations=612 (full 1): > PSYoungGen total 2089920K, used 2558K [0x0000000780000000, > 0x0000000800000000, 0x0000000800000000) > eden space 2083008K, 0% used > [0x0000000780000000,0x0000000780000000,0x00000007ff230000) > from space 6912K, 37% used > [0x00000007ff230000,0x00000007ff4af9e8,0x00000007ff8f0000) > to space 6912K, 0% used > [0x00000007ff940000,0x00000007ff940000,0x0000000800000000) > PSOldGen total 3145728K, used 1440346K [0x00000006c0000000, > 0x0000000780000000, 0x0000000780000000) > object space 3145728K, 45% used > [0x00000006c0000000,0x0000000717e969a0,0x0000000780000000) > PSPermGen total 1048576K, used 596997K [0x0000000680000000, > 0x00000006c0000000, 0x00000006c0000000) > object space 1048576K, 56% used > [0x0000000680000000,0x00000006a4701508,0x00000006c0000000) > } > {Heap before GC invocations=613 (full 1): > PSYoungGen total 2089920K, used 2085566K [0x0000000780000000, > 0x0000000800000000, 0x0000000800000000) > eden space 2083008K, 100% used > [0x0000000780000000,0x00000007ff230000,0x00000007ff230000) > from space 6912K, 37% used > [0x00000007ff230000,0x00000007ff4af9e8,0x00000007ff8f0000) > to space 6912K, 0% used > [0x00000007ff940000,0x00000007ff940000,0x0000000800000000) > PSOldGen total 3145728K, used 1440346K [0x00000006c0000000, > 0x0000000780000000, 0x0000000780000000) > object space 3145728K, 45% used > [0x00000006c0000000,0x0000000717e969a0,0x0000000780000000) > PSPermGen total 1048576K, used 596997K [0x0000000680000000, > 0x00000006c0000000, 0x00000006c0000000) > object space 1048576K, 56% used > [0x0000000680000000,0x00000006a4701570,0x00000006c0000000) > 1429.269: [GC > Desired survivor size 6815744 bytes, new threshold 15 (max 15) > [PSYoungGen: 2085566K->3488K(2090496K)] 3525912K->1443866K(5236224K), > 0.0050560 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] > Heap after GC invocations=613 (full 1): > PSYoungGen total 2090496K, used 3488K [0x0000000780000000, > 0x0000000800000000, 0x0000000800000000) > eden space 2083584K, 0% used > [0x0000000780000000,0x0000000780000000,0x00000007ff2c0000) > from space 6912K, 50% used > [0x00000007ff940000,0x00000007ffca8000,0x0000000800000000) > to space 6656K, 0% used > [0x00000007ff2c0000,0x00000007ff2c0000,0x00000007ff940000) > PSOldGen total 3145728K, used 1440378K [0x00000006c0000000, > 0x0000000780000000, 0x0000000780000000) > object space 3145728K, 45% used > [0x00000006c0000000,0x0000000717e9e9a0,0x0000000780000000) > PSPermGen total 1048576K, used 596997K [0x0000000680000000, > 0x00000006c0000000, 0x00000006c0000000) > object space 1048576K, 56% used > [0x0000000680000000,0x00000006a4701570,0x00000006c0000000) > } > {Heap before GC invocations=614 (full 1): > PSYoungGen total 2090496K, used 2086981K [0x0000000780000000, > 0x0000000800000000, 0x0000000800000000) > eden space 2083584K, 99% used > [0x0000000780000000,0x00000007ff2a9640,0x00000007ff2c0000) > from space 6912K, 50% used > [0x00000007ff940000,0x00000007ffca8000,0x0000000800000000) > to space 6656K, 0% used > [0x00000007ff2c0000,0x00000007ff2c0000,0x00000007ff940000) > PSOldGen total 3145728K, used 1440378K [0x00000006c0000000, > 0x0000000780000000, 0x0000000780000000) > object space 3145728K, 45% used > [0x00000006c0000000,0x0000000717e9e9a0,0x0000000780000000) > PSPermGen total 1048576K, used 596997K [0x0000000680000000, > 0x00000006c0000000, 0x00000006c0000000) > object space 1048576K, 56% used > [0x0000000680000000,0x00000006a4701570,0x00000006c0000000) > 1431.623: [GC > Desired survivor size 7012352 bytes, new threshold 15 (max 15) > [PSYoungGen: 2086981K->1344K(2090240K)] 3527359K->1441738K(5235968K), > 0.0034930 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] > Heap after GC invocations=614 (full 1): > PSYoungGen total 2090240K, used 1344K [0x0000000780000000, > 0x0000000800000000, 0x0000000800000000) > eden space 2083584K, 0% used > [0x0000000780000000,0x0000000780000000,0x00000007ff2c0000) > from space 6656K, 20% used > [0x00000007ff2c0000,0x00000007ff410000,0x00000007ff940000) > to space 6848K, 0% used > [0x00000007ff950000,0x00000007ff950000,0x0000000800000000) > PSOldGen total 3145728K, used 1440394K [0x00000006c0000000, > 0x0000000780000000, 0x0000000780000000) > object space 3145728K, 45% used > [0x00000006c0000000,0x0000000717ea29a0,0x0000000780000000) > PSPermGen total 1048576K, used 596997K [0x0000000680000000, > 0x00000006c0000000, 0x00000006c0000000) > object space 1048576K, 56% used > [0x0000000680000000,0x00000006a4701570,0x00000006c0000000) > } > ---------- Java 8 ---------- > {Heap before GC invocations=139 (full 2): > PSYoungGen total 2092032K, used 2088832K [0x0000000740000000, > 0x00000007c0000000, 0x00000007c0000000) > eden space 2086912K, 100% used > [0x0000000740000000,0x00000007bf600000,0x00000007bf600000) > from space 5120K, 37% used > [0x00000007bfb00000,0x00000007bfce0000,0x00000007c0000000) > to space 5120K, 0% used > [0x00000007bf600000,0x00000007bf600000,0x00000007bfb00000) > ParOldGen total 4194304K, used 2114679K [0x0000000640000000, > 0x0000000740000000, 0x0000000740000000) > object space 4194304K, 50% used > [0x0000000640000000,0x00000006c111dc38,0x0000000740000000) > Metaspace used 58033K, capacity 58834K, committed 59056K, > reserved 1101824K > class space used 6754K, capacity 6987K, committed 7088K, reserved > 1048576K > 1378.385: [GC (Allocation Failure) > Desired survivor size 4718592 bytes, new threshold 1 (max 15) > [PSYoungGen: 2088832K->1632K(2092032K)] 4203511K->2116327K(6286336K), > 0.1997693 secs] [Times: user=0.22 sys=0.00, real=0.20 secs] > Heap after GC invocations=139 (full 2): > PSYoungGen total 2092032K, used 1632K [0x0000000740000000, > 0x00000007c0000000, 0x00000007c0000000) > eden space 2086912K, 0% used > [0x0000000740000000,0x0000000740000000,0x00000007bf600000) > from space 5120K, 31% used > [0x00000007bf600000,0x00000007bf798000,0x00000007bfb00000) > to space 4608K, 0% used > [0x00000007bfb80000,0x00000007bfb80000,0x00000007c0000000) > ParOldGen total 4194304K, used 2114695K [0x0000000640000000, > 0x0000000740000000, 0x0000000740000000) > object space 4194304K, 50% used > [0x0000000640000000,0x00000006c1121c38,0x0000000740000000) > Metaspace used 58033K, capacity 58834K, committed 59056K, > reserved 1101824K > class space used 6754K, capacity 6987K, committed 7088K, reserved > 1048576K > } > {Heap before GC invocations=140 (full 2): > PSYoungGen total 2092032K, used 2088544K [0x0000000740000000, > 0x00000007c0000000, 0x00000007c0000000) > eden space 2086912K, 100% used > [0x0000000740000000,0x00000007bf600000,0x00000007bf600000) > from space 5120K, 31% used > [0x00000007bf600000,0x00000007bf798000,0x00000007bfb00000) > to space 4608K, 0% used > [0x00000007bfb80000,0x00000007bfb80000,0x00000007c0000000) > ParOldGen total 4194304K, used 2114695K [0x0000000640000000, > 0x0000000740000000, 0x0000000740000000) > object space 4194304K, 50% used > [0x0000000640000000,0x00000006c1121c38,0x0000000740000000) > Metaspace used 58037K, capacity 58834K, committed 59056K, > reserved 1101824K > class space used 6754K, capacity 6987K, committed 7088K, reserved > 1048576K > 1380.588: [GC (Allocation Failure) > Desired survivor size 4718592 bytes, new threshold 1 (max 15) > [PSYoungGen: 2088544K->1216K(2092544K)] 4203239K->2115935K(6286848K), > 0.1992978 secs] [Times: user=0.22 sys=0.00, real=0.20 secs] > Heap after GC invocations=140 (full 2): > PSYoungGen total 2092544K, used 1216K [0x0000000740000000, > 0x00000007c0000000, 0x00000007c0000000) > eden space 2087936K, 0% used > [0x0000000740000000,0x0000000740000000,0x00000007bf700000) > from space 4608K, 26% used > [0x00000007bfb80000,0x00000007bfcb0000,0x00000007c0000000) > to space 4608K, 0% used > [0x00000007bf700000,0x00000007bf700000,0x00000007bfb80000) > ParOldGen total 4194304K, used 2114719K [0x0000000640000000, > 0x0000000740000000, 0x0000000740000000) > object space 4194304K, 50% used > [0x0000000640000000,0x00000006c1127c38,0x0000000740000000) > Metaspace used 58037K, capacity 58834K, committed 59056K, > reserved 1101824K > class space used 6754K, capacity 6987K, committed 7088K, reserved > 1048576K > } > {Heap before GC invocations=141 (full 2): > PSYoungGen total 2092544K, used 2089152K [0x0000000740000000, > 0x00000007c0000000, 0x00000007c0000000) > eden space 2087936K, 100% used > [0x0000000740000000,0x00000007bf700000,0x00000007bf700000) > from space 4608K, 26% used > [0x00000007bfb80000,0x00000007bfcb0000,0x00000007c0000000) > to space 4608K, 0% used > [0x00000007bf700000,0x00000007bf700000,0x00000007bfb80000) > ParOldGen total 4194304K, used 2114719K [0x0000000640000000, > 0x0000000740000000, 0x0000000740000000) > object space 4194304K, 50% used > [0x0000000640000000,0x00000006c1127c38,0x0000000740000000) > Metaspace used 58038K, capacity 58834K, committed 59056K, > reserved 1101824K > class space used 6754K, capacity 6987K, committed 7088K, reserved > 1048576K > 1382.754: [GC (Allocation Failure) > Desired survivor size 4718592 bytes, new threshold 1 (max 15) > [PSYoungGen: 2089152K->2260K(2092544K)] 4203871K->2117011K(6286848K), > 0.2010010 secs] [Times: user=0.23 sys=0.00, real=0.20 secs] > Heap after GC invocations=141 (full 2): > PSYoungGen total 2092544K, used 2260K [0x0000000740000000, > 0x00000007c0000000, 0x00000007c0000000) > eden space 2087936K, 0% used > [0x0000000740000000,0x0000000740000000,0x00000007bf700000) > from space 4608K, 49% used > [0x00000007bf700000,0x00000007bf9351e0,0x00000007bfb80000) > to space 4608K, 0% used > [0x00000007bfb80000,0x00000007bfb80000,0x00000007c0000000) > ParOldGen total 4194304K, used 2114751K [0x0000000640000000, > 0x0000000740000000, 0x0000000740000000) > object space 4194304K, 50% used > [0x0000000640000000,0x00000006c112fc38,0x0000000740000000) > Metaspace used 58038K, capacity 58834K, committed 59056K, > reserved 1101824K > class space used 6754K, capacity 6987K, committed 7088K, reserved > 1048576K > } > CONFIDENTIALITY NOTICE This message and any included attachments are > from Cerner Corporation and are intended only for the addressee. The > information contained in this message is confidential and may > constitute inside or non-public information under international, > federal, or state securities laws. Unauthorized forwarding, printing, > copying, distribution, or use of such information is strictly > prohibited and may be unlawful. If you are not the addressee, please > promptly delete this message and notify the sender of the delivery > error by e-mail or you may call Cerner's corporate offices in Kansas > City, Missouri, U.S.A at (+1) (816)221-1024. > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: From Corbin.Hughes at Cerner.com Sat May 28 02:24:24 2016 From: Corbin.Hughes at Cerner.com (Hughes,Corbin) Date: Sat, 28 May 2016 02:24:24 +0000 Subject: Unexpected Java 6 vs 8 GC performance difference Message-ID: <360b3157768640ed9b2eeedd1dca2b2c@CERNMSGLS5EX02B.cerner.net> Jon, Thanks for the interest. Hopefully this log will be more helpful. This is Java 7 with the -XX:-UseParallelOldGC flag. Perhaps notably, the PSYoungGen collections are even slower in Java 7 (regardless of whether or not the serial old collector is used). Thanks, Corbin {Heap before GC invocations=456 (full 0): PSYoungGen total 2091008K, used 2087456K [0x0000000780000000, 0x0000000800000000, 0x0000000800000000) eden space 2084864K, 100% used [0x0000000780000000,0x00000007ff400000,0x00000007ff400000) from space 6144K, 42% used [0x00000007ff400000,0x00000007ff688000,0x00000007ffa00000) to space 6144K, 0% used [0x00000007ffa00000,0x00000007ffa00000,0x0000000800000000) PSOldGen total 3145728K, used 2267941K [0x00000006c0000000, 0x0000000780000000, 0x0000000780000000) object space 3145728K, 72% used [0x00000006c0000000,0x000000074a6c9508,0x0000000780000000) PSPermGen total 819200K, used 57100K [0x0000000680000000, 0x00000006b2000000, 0x00000006c0000000) object space 819200K, 6% used [0x0000000680000000,0x00000006837c3130,0x00000006b2000000) 1461.284: [GC Desired survivor size 6291456 bytes, new threshold 1 (max 15) [PSYoungGen: 2087456K->1482K(2091008K)] 4355397K->2269463K(5236736K), 0.2986350 secs] [Times: user=0.34 sys=0.00, real=0.30 secs] Heap after GC invocations=456 (full 0): PSYoungGen total 2091008K, used 1482K [0x0000000780000000, 0x0000000800000000, 0x0000000800000000) eden space 2084864K, 0% used [0x0000000780000000,0x0000000780000000,0x00000007ff400000) from space 6144K, 24% used [0x00000007ffa00000,0x00000007ffb72928,0x0000000800000000) to space 6144K, 0% used [0x00000007ff400000,0x00000007ff400000,0x00000007ffa00000) PSOldGen total 3145728K, used 2267981K [0x00000006c0000000, 0x0000000780000000, 0x0000000780000000) object space 3145728K, 72% used [0x00000006c0000000,0x000000074a6d3528,0x0000000780000000) PSPermGen total 819200K, used 57100K [0x0000000680000000, 0x00000006b2000000, 0x00000006c0000000) object space 819200K, 6% used [0x0000000680000000,0x00000006837c3130,0x00000006b2000000) } {Heap before GC invocations=457 (full 0): PSYoungGen total 2091008K, used 2086346K [0x0000000780000000, 0x0000000800000000, 0x0000000800000000) eden space 2084864K, 100% used [0x0000000780000000,0x00000007ff400000,0x00000007ff400000) from space 6144K, 24% used [0x00000007ffa00000,0x00000007ffb72928,0x0000000800000000) to space 6144K, 0% used [0x00000007ff400000,0x00000007ff400000,0x00000007ffa00000) PSOldGen total 3145728K, used 2267981K [0x00000006c0000000, 0x0000000780000000, 0x0000000780000000) object space 3145728K, 72% used [0x00000006c0000000,0x000000074a6d3528,0x0000000780000000) PSPermGen total 819200K, used 57100K [0x0000000680000000, 0x00000006b2000000, 0x00000006c0000000) object space 819200K, 6% used [0x0000000680000000,0x00000006837c3130,0x00000006b2000000) 1463.560: [GC Desired survivor size 6291456 bytes, new threshold 1 (max 15) [PSYoungGen: 2086346K->3177K(2091008K)] 4354327K->2271183K(5236736K), 0.2974600 secs] [Times: user=0.33 sys=0.00, real=0.30 secs] Heap after GC invocations=457 (full 0): PSYoungGen total 2091008K, used 3177K [0x0000000780000000, 0x0000000800000000, 0x0000000800000000) eden space 2084864K, 0% used [0x0000000780000000,0x0000000780000000,0x00000007ff400000) from space 6144K, 51% used [0x00000007ff400000,0x00000007ff71a730,0x00000007ffa00000) to space 6144K, 0% used [0x00000007ffa00000,0x00000007ffa00000,0x0000000800000000) PSOldGen total 3145728K, used 2268005K [0x00000006c0000000, 0x0000000780000000, 0x0000000780000000) object space 3145728K, 72% used [0x00000006c0000000,0x000000074a6d9528,0x0000000780000000) PSPermGen total 819200K, used 57100K [0x0000000680000000, 0x00000006b2000000, 0x00000006c0000000) object space 819200K, 6% used [0x0000000680000000,0x00000006837c3130,0x00000006b2000000) } {Heap before GC invocations=458 (full 0): PSYoungGen total 2091008K, used 2088041K [0x0000000780000000, 0x0000000800000000, 0x0000000800000000) eden space 2084864K, 100% used [0x0000000780000000,0x00000007ff400000,0x00000007ff400000) from space 6144K, 51% used [0x00000007ff400000,0x00000007ff71a730,0x00000007ffa00000) to space 6144K, 0% used [0x00000007ffa00000,0x00000007ffa00000,0x0000000800000000) PSOldGen total 3145728K, used 2268005K [0x00000006c0000000, 0x0000000780000000, 0x0000000780000000) object space 3145728K, 72% used [0x00000006c0000000,0x000000074a6d9528,0x0000000780000000) PSPermGen total 819200K, used 57100K [0x0000000680000000, 0x00000006b2000000, 0x00000006c0000000) object space 819200K, 6% used [0x0000000680000000,0x00000006837c31b0,0x00000006b2000000) 1465.998: [GC Desired survivor size 6291456 bytes, new threshold 1 (max 15) [PSYoungGen: 2088041K->1456K(2091008K)] 4356047K->2269485K(5236736K), 0.2940590 secs] [Times: user=0.33 sys=0.00, real=0.29 secs] Heap after GC invocations=458 (full 0): PSYoungGen total 2091008K, used 1456K [0x0000000780000000, 0x0000000800000000, 0x0000000800000000) eden space 2084864K, 0% used [0x0000000780000000,0x0000000780000000,0x00000007ff400000) from space 6144K, 23% used [0x00000007ffa00000,0x00000007ffb6c188,0x0000000800000000) to space 6144K, 0% used [0x00000007ff400000,0x00000007ff400000,0x00000007ffa00000) PSOldGen total 3145728K, used 2268029K [0x00000006c0000000, 0x0000000780000000, 0x0000000780000000) object space 3145728K, 72% used [0x00000006c0000000,0x000000074a6df528,0x0000000780000000) PSPermGen total 819200K, used 57100K [0x0000000680000000, 0x00000006b2000000, 0x00000006c0000000) object space 819200K, 6% used [0x0000000680000000,0x00000006837c31b0,0x00000006b2000000) } CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Corbin.Hughes at Cerner.com Tue May 31 22:36:25 2016 From: Corbin.Hughes at Cerner.com (Hughes,Corbin) Date: Tue, 31 May 2016 22:36:25 +0000 Subject: Unexpected Java 6 vs 8 GC performance difference In-Reply-To: <360b3157768640ed9b2eeedd1dca2b2c@CERNMSGLS5EX02B.cerner.net> References: <360b3157768640ed9b2eeedd1dca2b2c@CERNMSGLS5EX02B.cerner.net> Message-ID: After coming across -XX:+PrintGCTaskTimeStamps, I noticed that the listed tasks don't add up to the listed PSYoungGen times, so I ended up adding more "printf" style timing outputs to the hotspot source, and ended up tracking it down. Turns out it's the interned strings. The way they're GC'd isn't particularly efficient for large numbers of strings: the call to StringTable::unlink_or_oops_do(&_is_alive_closure, &root_closure); was taking ~185 ms of the ~200 ms. I wish there were a flag to disable GC of interned strings, but I think we'll be able to get away with using something like a Guava Interner instead of the built in StringTable, so mystery solved. From: hotspot-gc-use [mailto:hotspot-gc-use-bounces at openjdk.java.net] On Behalf Of Hughes,Corbin Sent: Friday, May 27, 2016 9:24 PM To: hotspot-gc-use at openjdk.java.net Subject: Re: Unexpected Java 6 vs 8 GC performance difference Jon, Thanks for the interest. Hopefully this log will be more helpful. This is Java 7 with the -XX:-UseParallelOldGC flag. Perhaps notably, the PSYoungGen collections are even slower in Java 7 (regardless of whether or not the serial old collector is used). Thanks, Corbin {Heap before GC invocations=456 (full 0): PSYoungGen total 2091008K, used 2087456K [0x0000000780000000, 0x0000000800000000, 0x0000000800000000) eden space 2084864K, 100% used [0x0000000780000000,0x00000007ff400000,0x00000007ff400000) from space 6144K, 42% used [0x00000007ff400000,0x00000007ff688000,0x00000007ffa00000) to space 6144K, 0% used [0x00000007ffa00000,0x00000007ffa00000,0x0000000800000000) PSOldGen total 3145728K, used 2267941K [0x00000006c0000000, 0x0000000780000000, 0x0000000780000000) object space 3145728K, 72% used [0x00000006c0000000,0x000000074a6c9508,0x0000000780000000) PSPermGen total 819200K, used 57100K [0x0000000680000000, 0x00000006b2000000, 0x00000006c0000000) object space 819200K, 6% used [0x0000000680000000,0x00000006837c3130,0x00000006b2000000) 1461.284: [GC Desired survivor size 6291456 bytes, new threshold 1 (max 15) [PSYoungGen: 2087456K->1482K(2091008K)] 4355397K->2269463K(5236736K), 0.2986350 secs] [Times: user=0.34 sys=0.00, real=0.30 secs] Heap after GC invocations=456 (full 0): PSYoungGen total 2091008K, used 1482K [0x0000000780000000, 0x0000000800000000, 0x0000000800000000) eden space 2084864K, 0% used [0x0000000780000000,0x0000000780000000,0x00000007ff400000) from space 6144K, 24% used [0x00000007ffa00000,0x00000007ffb72928,0x0000000800000000) to space 6144K, 0% used [0x00000007ff400000,0x00000007ff400000,0x00000007ffa00000) PSOldGen total 3145728K, used 2267981K [0x00000006c0000000, 0x0000000780000000, 0x0000000780000000) object space 3145728K, 72% used [0x00000006c0000000,0x000000074a6d3528,0x0000000780000000) PSPermGen total 819200K, used 57100K [0x0000000680000000, 0x00000006b2000000, 0x00000006c0000000) object space 819200K, 6% used [0x0000000680000000,0x00000006837c3130,0x00000006b2000000) } {Heap before GC invocations=457 (full 0): PSYoungGen total 2091008K, used 2086346K [0x0000000780000000, 0x0000000800000000, 0x0000000800000000) eden space 2084864K, 100% used [0x0000000780000000,0x00000007ff400000,0x00000007ff400000) from space 6144K, 24% used [0x00000007ffa00000,0x00000007ffb72928,0x0000000800000000) to space 6144K, 0% used [0x00000007ff400000,0x00000007ff400000,0x00000007ffa00000) PSOldGen total 3145728K, used 2267981K [0x00000006c0000000, 0x0000000780000000, 0x0000000780000000) object space 3145728K, 72% used [0x00000006c0000000,0x000000074a6d3528,0x0000000780000000) PSPermGen total 819200K, used 57100K [0x0000000680000000, 0x00000006b2000000, 0x00000006c0000000) object space 819200K, 6% used [0x0000000680000000,0x00000006837c3130,0x00000006b2000000) 1463.560: [GC Desired survivor size 6291456 bytes, new threshold 1 (max 15) [PSYoungGen: 2086346K->3177K(2091008K)] 4354327K->2271183K(5236736K), 0.2974600 secs] [Times: user=0.33 sys=0.00, real=0.30 secs] Heap after GC invocations=457 (full 0): PSYoungGen total 2091008K, used 3177K [0x0000000780000000, 0x0000000800000000, 0x0000000800000000) eden space 2084864K, 0% used [0x0000000780000000,0x0000000780000000,0x00000007ff400000) from space 6144K, 51% used [0x00000007ff400000,0x00000007ff71a730,0x00000007ffa00000) to space 6144K, 0% used [0x00000007ffa00000,0x00000007ffa00000,0x0000000800000000) PSOldGen total 3145728K, used 2268005K [0x00000006c0000000, 0x0000000780000000, 0x0000000780000000) object space 3145728K, 72% used [0x00000006c0000000,0x000000074a6d9528,0x0000000780000000) PSPermGen total 819200K, used 57100K [0x0000000680000000, 0x00000006b2000000, 0x00000006c0000000) object space 819200K, 6% used [0x0000000680000000,0x00000006837c3130,0x00000006b2000000) } {Heap before GC invocations=458 (full 0): PSYoungGen total 2091008K, used 2088041K [0x0000000780000000, 0x0000000800000000, 0x0000000800000000) eden space 2084864K, 100% used [0x0000000780000000,0x00000007ff400000,0x00000007ff400000) from space 6144K, 51% used [0x00000007ff400000,0x00000007ff71a730,0x00000007ffa00000) to space 6144K, 0% used [0x00000007ffa00000,0x00000007ffa00000,0x0000000800000000) PSOldGen total 3145728K, used 2268005K [0x00000006c0000000, 0x0000000780000000, 0x0000000780000000) object space 3145728K, 72% used [0x00000006c0000000,0x000000074a6d9528,0x0000000780000000) PSPermGen total 819200K, used 57100K [0x0000000680000000, 0x00000006b2000000, 0x00000006c0000000) object space 819200K, 6% used [0x0000000680000000,0x00000006837c31b0,0x00000006b2000000) 1465.998: [GC Desired survivor size 6291456 bytes, new threshold 1 (max 15) [PSYoungGen: 2088041K->1456K(2091008K)] 4356047K->2269485K(5236736K), 0.2940590 secs] [Times: user=0.33 sys=0.00, real=0.29 secs] Heap after GC invocations=458 (full 0): PSYoungGen total 2091008K, used 1456K [0x0000000780000000, 0x0000000800000000, 0x0000000800000000) eden space 2084864K, 0% used [0x0000000780000000,0x0000000780000000,0x00000007ff400000) from space 6144K, 23% used [0x00000007ffa00000,0x00000007ffb6c188,0x0000000800000000) to space 6144K, 0% used [0x00000007ff400000,0x00000007ff400000,0x00000007ffa00000) PSOldGen total 3145728K, used 2268029K [0x00000006c0000000, 0x0000000780000000, 0x0000000780000000) object space 3145728K, 72% used [0x00000006c0000000,0x000000074a6df528,0x0000000780000000) PSPermGen total 819200K, used 57100K [0x0000000680000000, 0x00000006b2000000, 0x00000006c0000000) object space 819200K, 6% used [0x0000000680000000,0x00000006837c31b0,0x00000006b2000000) } CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024. -------------- next part -------------- An HTML attachment was scrubbed... URL: