From barak.yaish at gmail.com Mon Feb 2 15:39:40 2015 From: barak.yaish at gmail.com (Barak Yaish) Date: Mon, 2 Feb 2015 17:39:40 +0200 Subject: Would like to understand why gc time grows Message-ID: Hi, I'm having my application running on java8 with g1gc enabled. I've enabled gc logs, and I can see that gc time is growing ( at least this is what I think..) VM flags: -Xms6g -Xmx6g -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX:G1HeapRegionSize=32M -XX:+PrintAdaptiveSizePolicy A snippet from the gc log: {Heap before GC invocations=342 (full 1): garbage-first heap total 6291456K, used 5257482K [0x0000000640000000, 0x00000007c0000000, 0x00000007c0000000) region size 32768K, 104 young (3407872K), 10 survivors (327680K) Metaspace used 31200K, capacity 31664K, committed 31872K, reserved 1077248K class space used 3496K, capacity 3596K, committed 3712K, reserved 1048576K 2015-02-02T17:36:48.237+0200: 4067.021: [GC pause (G1 Evacuation Pause) (young) 4067.021: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 23499, predicted base time: 11.30 ms, remaining time: 188.70 ms, target pause time: 200.00 ms] 4067.021: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 94 regions, survivors: 10 regions, predicted young region time: 159.99 ms] 4067.021: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 94 regions, survivors: 10 regions, old: 0 regions, predicted pause time: 171.29 ms, target pause time: 200.00 ms] , 0.2319995 secs] [Parallel Time: 229.0 ms, GC Workers: 13] [GC Worker Start (ms): Min: 4067021.3, Avg: 4067024.1, Max: 4067029.4, Diff: 8.0] [Ext Root Scanning (ms): Min: 0.0, Avg: 0.5, Max: 1.3, Diff: 1.3, Sum: 6.9] [Update RS (ms): Min: 0.6, Avg: 5.3, Max: 7.5, Diff: 6.9, Sum: 69.2] [Processed Buffers: Min: 1, Avg: 10.2, Max: 33, Diff: 32, Sum: 132] [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 1.0] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 219.7, Avg: 219.9, Max: 220.0, Diff: 0.3, Sum: 2858.1] [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.9] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 1.2] [GC Worker Total (ms): Min: 220.7, Avg: 225.9, Max: 228.8, Diff: 8.0, Sum: 2937.3] [GC Worker End (ms): Min: 4067250.0, Avg: 4067250.1, Max: 4067250.2, Diff: 0.2] [Code Root Fixup: 0.2 ms] [Code Root Migration: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 1.4 ms] [Other: 1.4 ms] [Choose CSet: 0.0 ms] [Ref Proc: 0.8 ms] [Ref Enq: 0.0 ms] [Redirty Cards: 0.2 ms] [Free CSet: 0.1 ms] [Eden: 3008.0M(3008.0M)->0.0B(2976.0M) Survivors: 320.0M->320.0M Heap: 5134.3M(6144.0M)->2168.1M(6144.0M)] Heap after GC invocations=343 (full 1): garbage-first heap total 6291456K, used 2220163K [0x0000000640000000, 0x00000007c0000000, 0x00000007c0000000) region size 32768K, 10 young (327680K), 10 survivors (327680K) Metaspace used 31200K, capacity 31664K, committed 31872K, reserved 1077248K class space used 3496K, capacity 3596K, committed 3712K, reserved 1048576K } [Times: user=2.94 sys=0.00, real=0.23 secs] And the "user" part is keep growing... Can you explain what this actually means? What exact process in the gc in contributing to this? And what can be done in order to minimize this time? Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yu.zhang at oracle.com Mon Feb 2 16:23:23 2015 From: yu.zhang at oracle.com (Yu Zhang) Date: Mon, 02 Feb 2015 08:23:23 -0800 Subject: Would like to understand why gc time grows In-Reply-To: References: Message-ID: <54CFA47B.4000503@oracle.com> Barak, If you can share the entire gc log, we can understand what is going on better. From the snip, user/read ratio is 12.8, and you have 13 gc threads. It looks normal. Thanks, Jenny On 2/2/2015 7:39 AM, Barak Yaish wrote: > Hi, > > I'm having my application running on java8 with g1gc enabled. I've > enabled gc logs, and I can see that gc time is growing ( at least this > is what I think..) > > VM flags: > -Xms6g -Xmx6g > -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX:G1HeapRegionSize=32M -XX:+PrintAdaptiveSizePolicy > > > A snippet from the gc log: > > {Heap before GC invocations=342 (full 1): > garbage-first heap total 6291456K, used 5257482K > [0x0000000640000000, 0x00000007c0000000, 0x00000007c0000000) > region size 32768K, 104 young (3407872K), 10 survivors (327680K) > Metaspace used 31200K, capacity 31664K, committed 31872K, > reserved 1077248K > class space used 3496K, capacity 3596K, committed 3712K, reserved > 1048576K > 2015-02-02T17:36:48.237+0200: 4067.021: [GC pause (G1 Evacuation > Pause) (young) 4067.021: [G1Ergonomics (CSet Construction) start > choosing CSet, _pending_cards: 23499, predicted base time: 11.30 ms, > remaining time: 188.70 ms, target pause time: 200.00 ms] > 4067.021: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 94 regions, survivors: 10 regions, predicted young region > time: 159.99 ms] > 4067.021: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 94 regions, survivors: 10 regions, old: 0 regions, predicted > pause time: 171.29 ms, target pause time: 200.00 ms] > , 0.2319995 secs] > [Parallel Time: 229.0 ms, GC Workers: 13] > [GC Worker Start (ms): Min: 4067021.3, Avg: 4067024.1, Max: > 4067029.4, Diff: 8.0] > [Ext Root Scanning (ms): Min: 0.0, Avg: 0.5, Max: 1.3, Diff: > 1.3, Sum: 6.9] > [Update RS (ms): Min: 0.6, Avg: 5.3, Max: 7.5, Diff: 6.9, Sum: 69.2] > [Processed Buffers: Min: 1, Avg: 10.2, Max: 33, Diff: 32, > Sum: 132] > [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 1.0] > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > [Object Copy (ms): Min: 219.7, Avg: 219.9, Max: 220.0, Diff: > 0.3, Sum: 2858.1] > [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: > 0.9] > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, > Sum: 1.2] > [GC Worker Total (ms): Min: 220.7, Avg: 225.9, Max: 228.8, Diff: > 8.0, Sum: 2937.3] > [GC Worker End (ms): Min: 4067250.0, Avg: 4067250.1, Max: > 4067250.2, Diff: 0.2] > [Code Root Fixup: 0.2 ms] > [Code Root Migration: 0.0 ms] > [Code Root Purge: 0.0 ms] > [Clear CT: 1.4 ms] > [Other: 1.4 ms] > [Choose CSet: 0.0 ms] > [Ref Proc: 0.8 ms] > [Ref Enq: 0.0 ms] > [Redirty Cards: 0.2 ms] > [Free CSet: 0.1 ms] > [Eden: 3008.0M(3008.0M)->0.0B(2976.0M) Survivors: 320.0M->320.0M > Heap: 5134.3M(6144.0M)->2168.1M(6144.0M)] > Heap after GC invocations=343 (full 1): > garbage-first heap total 6291456K, used 2220163K > [0x0000000640000000, 0x00000007c0000000, 0x00000007c0000000) > region size 32768K, 10 young (327680K), 10 survivors (327680K) > Metaspace used 31200K, capacity 31664K, committed 31872K, > reserved 1077248K > class space used 3496K, capacity 3596K, committed 3712K, reserved > 1048576K > } > [Times: user=2.94 sys=0.00, real=0.23 secs] > > And the "user" part is keep growing... Can you explain what this > actually means? What exact process in the gc in contributing to this? > And what can be done in order to minimize this time? > > Thanks. > > > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: From jon.masamitsu at oracle.com Mon Feb 2 18:06:22 2015 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Mon, 02 Feb 2015 10:06:22 -0800 Subject: longer ParNew collection in JDK8? In-Reply-To: <1905701844.2549870.1422556039324.JavaMail.yahoo@mail.yahoo.com> References: <1905701844.2549870.1422556039324.JavaMail.yahoo@mail.yahoo.com> Message-ID: <54CFBC9E.9070700@oracle.com> Joy, Is your application such that most everything dies in the young generation and very little is promoted? If yes, do you know if there is a difference in how much is promoted between the two releases? Jon On 01/29/2015 10:27 AM, Joy Xiong wrote: > > Hi, > > We recently move our services from JDK6 to JDK8, but observe longer > ParNew GC pause in JDK8. For example, the pause time increase from > 27ms to 43ms in one service. The service has the JVM parameters as below: > -Xms32684m -Xmx32684m -XX:NewSize=2048m -XX:MaxNewSize=2048m > -XX:+UseConcMarkSweepGC -XX:+UseParNewGC > -XX:CMSInitiatingOccupancyFraction=70 -XX:SurvivorRatio=2 > -XX:+AlwaysPreTouch -XX:+UseCompressedOops > -XX:+PrintTenuringDistribution -XX:+PrintGCDetails > -XX:+PrintGCDateStamps -Xloggc:logs/gc.log > -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime > > Is this longer ParNew pause expected? Any suggestions to mitigate the > longer pause? > > thanks, > -Joy > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: From joyxiong at yahoo.com Mon Feb 2 18:38:16 2015 From: joyxiong at yahoo.com (Joy Xiong) Date: Mon, 2 Feb 2015 18:38:16 +0000 (UTC) Subject: longer ParNew collection in JDK8? In-Reply-To: <54CFBC9E.9070700@oracle.com> References: <54CFBC9E.9070700@oracle.com> Message-ID: <12224176.1048123.1422902296117.JavaMail.yahoo@mail.yahoo.com> Hi Jon, Yes, most objects in my application die young and very little promoted. As you can see from the attached figure (the occupied heap size from Sep 28 to Nov 11), CMS happened very rare. However, it seems two releases do not have much difference in promotion rate, as they have same number of CMS in the log. We have the same JVM parameters for both releases as below: CommandLine flags: -XX:+AlwaysPreTouch -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=logs/hs_err.log -XX:InitialHeapSize=34271657984 -XX:+ManagementServer -XX:MaxHeapSize=34271657984 -XX:MaxNewSize=2147483648 -XX:NewSize=2147483648 -XX:OldPLABSize=16 -XX:+PrintGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:SurvivorRatio=2 -XX:+UnlockDiagnosticVMOptions -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC thanks,-Joy On Monday, February 2, 2015 10:06 AM, Jon Masamitsu wrote: Joy, Is your application such that most everything dies in the young generation and very little is promoted?? If yes, do you know if there is a difference in how much is promoted between the two releases? Jon On 01/29/2015 10:27 AM, Joy Xiong wrote: Hi, We recently move our services from JDK6 to JDK8, but observe longer ParNew GC pause in JDK8. For example, the pause time increase from 27ms to 43ms in one service. The service has the JVM parameters as below: -Xms32684m -Xmx32684m -XX:NewSize=2048m -XX:MaxNewSize=2048m -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=70 -XX:SurvivorRatio=2 -XX:+AlwaysPreTouch -XX:+UseCompressedOops -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:logs/gc.log -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime Is this longer ParNew pause expected? Any suggestions to mitigate the longer pause? thanks, -Joy _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2014-11-07 at 2.01.07 PM.png Type: image/png Size: 42316 bytes Desc: not available URL: From barak.yaish at gmail.com Mon Feb 2 18:58:33 2015 From: barak.yaish at gmail.com (Barak Yaish) Date: Mon, 2 Feb 2015 20:58:33 +0200 Subject: Would like to understand why gc time grows In-Reply-To: <54CFA47B.4000503@oracle.com> References: <54CFA47B.4000503@oracle.com> Message-ID: Sure, upload the file to https://www.dropbox.com/sh/kinl4zo8apg6kh2/AADlh9b-X_Zk6L_tgPL_lyyma?dl=0 Can you explain which messages indicates the time the app was paused due to gc? Thanks! On Mon, Feb 2, 2015 at 6:23 PM, Yu Zhang wrote: > Barak, > > If you can share the entire gc log, we can understand what is going on > better. From the snip, user/read ratio is 12.8, and you have 13 gc > threads. It looks normal. > > Thanks, > Jenny > > On 2/2/2015 7:39 AM, Barak Yaish wrote: > > Hi, > > I'm having my application running on java8 with g1gc enabled. I've > enabled gc logs, and I can see that gc time is growing ( at least this is > what I think..) > > VM flags: > -Xms6g -Xmx6g > -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX:G1HeapRegionSize=32M -XX:+PrintAdaptiveSizePolicy > > A snippet from the gc log: > > {Heap before GC invocations=342 (full 1): > garbage-first heap total 6291456K, used 5257482K [0x0000000640000000, > 0x00000007c0000000, 0x00000007c0000000) > region size 32768K, 104 young (3407872K), 10 survivors (327680K) > Metaspace used 31200K, capacity 31664K, committed 31872K, reserved > 1077248K > class space used 3496K, capacity 3596K, committed 3712K, reserved > 1048576K > 2015-02-02T17:36:48.237+0200: 4067.021: [GC pause (G1 Evacuation Pause) > (young) 4067.021: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 23499, predicted base time: 11.30 ms, remaining time: > 188.70 ms, target pause time: 200.00 ms] > 4067.021: [G1Ergonomics (CSet Construction) add young regions to CSet, > eden: 94 regions, survivors: 10 regions, predicted young region time: > 159.99 ms] > 4067.021: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: > 94 regions, survivors: 10 regions, old: 0 regions, predicted pause time: > 171.29 ms, target pause time: 200.00 ms] > , 0.2319995 secs] > [Parallel Time: 229.0 ms, GC Workers: 13] > [GC Worker Start (ms): Min: 4067021.3, Avg: 4067024.1, Max: > 4067029.4, Diff: 8.0] > [Ext Root Scanning (ms): Min: 0.0, Avg: 0.5, Max: 1.3, Diff: 1.3, > Sum: 6.9] > [Update RS (ms): Min: 0.6, Avg: 5.3, Max: 7.5, Diff: 6.9, Sum: 69.2] > [Processed Buffers: Min: 1, Avg: 10.2, Max: 33, Diff: 32, Sum: > 132] > [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 1.0] > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, > Sum: 0.0] > [Object Copy (ms): Min: 219.7, Avg: 219.9, Max: 220.0, Diff: 0.3, > Sum: 2858.1] > [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.9] > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: > 1.2] > [GC Worker Total (ms): Min: 220.7, Avg: 225.9, Max: 228.8, Diff: > 8.0, Sum: 2937.3] > [GC Worker End (ms): Min: 4067250.0, Avg: 4067250.1, Max: 4067250.2, > Diff: 0.2] > [Code Root Fixup: 0.2 ms] > [Code Root Migration: 0.0 ms] > [Code Root Purge: 0.0 ms] > [Clear CT: 1.4 ms] > [Other: 1.4 ms] > [Choose CSet: 0.0 ms] > [Ref Proc: 0.8 ms] > [Ref Enq: 0.0 ms] > [Redirty Cards: 0.2 ms] > [Free CSet: 0.1 ms] > [Eden: 3008.0M(3008.0M)->0.0B(2976.0M) Survivors: 320.0M->320.0M Heap: > 5134.3M(6144.0M)->2168.1M(6144.0M)] > Heap after GC invocations=343 (full 1): > garbage-first heap total 6291456K, used 2220163K [0x0000000640000000, > 0x00000007c0000000, 0x00000007c0000000) > region size 32768K, 10 young (327680K), 10 survivors (327680K) > Metaspace used 31200K, capacity 31664K, committed 31872K, reserved > 1077248K > class space used 3496K, capacity 3596K, committed 3712K, reserved > 1048576K > } > [Times: user=2.94 sys=0.00, real=0.23 secs] > > And the "user" part is keep growing... Can you explain what this > actually means? What exact process in the gc in contributing to this? And > what can be done in order to minimize this time? > > Thanks. > > > > > _______________________________________________ > hotspot-gc-use mailing listhotspot-gc-use at openjdk.java.nethttp://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yu.zhang at oracle.com Mon Feb 2 19:40:21 2015 From: yu.zhang at oracle.com (Yu Zhang) Date: Mon, 02 Feb 2015 11:40:21 -0800 Subject: Would like to understand why gc time grows In-Reply-To: References: <54CFA47B.4000503@oracle.com> Message-ID: <54CFD2A5.8010501@oracle.com> Barak, Thanks for the gc log. The stop the world pause in your log includes, young, mixed, remark and clean up phases. For details on how to under stand gc logs, please see https://blogs.oracle.com/g1gc/date/20140801 https://blogs.oracle.com/g1gc/date/20140807 Here is a chart of the gc pauses from your log. Can not conclude gc pause is getting longer. The default MaxGCPauseMillis is 200 (ms), does it work for you? Or you need lower pause time? Thanks, Jenny On 2/2/2015 10:58 AM, Barak Yaish wrote: > Sure, upload the file to > https://www.dropbox.com/sh/kinl4zo8apg6kh2/AADlh9b-X_Zk6L_tgPL_lyyma?dl=0 > > Can you explain which messages indicates the time the app was paused > due to gc? > > Thanks! > > On Mon, Feb 2, 2015 at 6:23 PM, Yu Zhang > wrote: > > Barak, > > If you can share the entire gc log, we can understand what is > going on better. From the snip, user/read ratio is 12.8, and you > have 13 gc threads. It looks normal. > > Thanks, > Jenny > > On 2/2/2015 7:39 AM, Barak Yaish wrote: >> Hi, >> >> I'm having my application running on java8 with g1gc enabled. >> I've enabled gc logs, and I can see that gc time is growing ( at >> least this is what I think..) >> >> VM flags: >> -Xms6g -Xmx6g >> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX:G1HeapRegionSize=32M -XX:+PrintAdaptiveSizePolicy >> >> >> A snippet from the gc log: >> >> {Heap before GC invocations=342 (full 1): >> garbage-first heap total 6291456K, used 5257482K >> [0x0000000640000000, 0x00000007c0000000, 0x00000007c0000000) >> region size 32768K, 104 young (3407872K), 10 survivors (327680K) >> Metaspace used 31200K, capacity 31664K, committed 31872K, >> reserved 1077248K >> class space used 3496K, capacity 3596K, committed 3712K, >> reserved 1048576K >> 2015-02-02T17:36:48.237+0200: 4067.021: [GC pause (G1 Evacuation >> Pause) (young) 4067.021: [G1Ergonomics (CSet Construction) start >> choosing CSet, _pending_cards: 23499, predicted base time: 11.30 >> ms, remaining time: 188.70 ms, target pause time: 200.00 ms] >> 4067.021: [G1Ergonomics (CSet Construction) add young regions to >> CSet, eden: 94 regions, survivors: 10 regions, predicted young >> region time: 159.99 ms] >> 4067.021: [G1Ergonomics (CSet Construction) finish choosing >> CSet, eden: 94 regions, survivors: 10 regions, old: 0 regions, >> predicted pause time: 171.29 ms, target pause time: 200.00 ms] >> , 0.2319995 secs] >> [Parallel Time: 229.0 ms, GC Workers: 13] >> [GC Worker Start (ms): Min: 4067021.3, Avg: 4067024.1, Max: >> 4067029.4, Diff: 8.0] >> [Ext Root Scanning (ms): Min: 0.0, Avg: 0.5, Max: 1.3, >> Diff: 1.3, Sum: 6.9] >> [Update RS (ms): Min: 0.6, Avg: 5.3, Max: 7.5, Diff: 6.9, >> Sum: 69.2] >> [Processed Buffers: Min: 1, Avg: 10.2, Max: 33, Diff: >> 32, Sum: 132] >> [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, >> Sum: 1.0] >> [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, >> Diff: 0.0, Sum: 0.0] >> [Object Copy (ms): Min: 219.7, Avg: 219.9, Max: 220.0, >> Diff: 0.3, Sum: 2858.1] >> [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, >> Sum: 0.9] >> [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: >> 0.2, Sum: 1.2] >> [GC Worker Total (ms): Min: 220.7, Avg: 225.9, Max: 228.8, >> Diff: 8.0, Sum: 2937.3] >> [GC Worker End (ms): Min: 4067250.0, Avg: 4067250.1, Max: >> 4067250.2, Diff: 0.2] >> [Code Root Fixup: 0.2 ms] >> [Code Root Migration: 0.0 ms] >> [Code Root Purge: 0.0 ms] >> [Clear CT: 1.4 ms] >> [Other: 1.4 ms] >> [Choose CSet: 0.0 ms] >> [Ref Proc: 0.8 ms] >> [Ref Enq: 0.0 ms] >> [Redirty Cards: 0.2 ms] >> [Free CSet: 0.1 ms] >> [Eden: 3008.0M(3008.0M)->0.0B(2976.0M) Survivors: >> 320.0M->320.0M Heap: 5134.3M(6144.0M)->2168.1M(6144.0M)] >> Heap after GC invocations=343 (full 1): >> garbage-first heap total 6291456K, used 2220163K >> [0x0000000640000000, 0x00000007c0000000, 0x00000007c0000000) >> region size 32768K, 10 young (327680K), 10 survivors (327680K) >> Metaspace used 31200K, capacity 31664K, committed 31872K, >> reserved 1077248K >> class space used 3496K, capacity 3596K, committed 3712K, >> reserved 1048576K >> } >> [Times: user=2.94 sys=0.00, real=0.23 secs] >> >> And the "user" part is keep growing... Can you explain what this >> actually means? What exact process in the gc in contributing to >> this? And what can be done in order to minimize this time? >> >> Thanks. >> >> >> >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: hihgjffh.png Type: image/png Size: 15756 bytes Desc: not available URL: From barak.yaish at gmail.com Mon Feb 2 19:53:12 2015 From: barak.yaish at gmail.com (Barak Yaish) Date: Mon, 2 Feb 2015 21:53:12 +0200 Subject: Would like to understand why gc time grows In-Reply-To: <54CFD2A5.8010501@oracle.com> References: <54CFA47B.4000503@oracle.com> <54CFD2A5.8010501@oracle.com> Message-ID: Thanks for the quick response :-) My app supposed to serve as a simple web server, responding to requests in less than 30ms, no long lived objects... Doesn't the increasing value of "user" time indicates STW pauses? And which tool you use to parse the log? Most tools I was trying to use are failing to parse correctly g1 logs.. On Mon, Feb 2, 2015 at 9:40 PM, Yu Zhang wrote: > Barak, > > Thanks for the gc log. > > The stop the world pause in your log includes, young, mixed, remark and > clean up phases. > For details on how to under stand gc logs, please see > https://blogs.oracle.com/g1gc/date/20140801 > https://blogs.oracle.com/g1gc/date/20140807 > > Here is a chart of the gc pauses from your log. Can not conclude gc pause > is getting longer. > The default MaxGCPauseMillis is 200 (ms), does it work for you? Or you > need lower pause time? > > Thanks, > Jenny > > On 2/2/2015 10:58 AM, Barak Yaish wrote: > > Sure, upload the file to > https://www.dropbox.com/sh/kinl4zo8apg6kh2/AADlh9b-X_Zk6L_tgPL_lyyma?dl=0 > > Can you explain which messages indicates the time the app was paused due > to gc? > > Thanks! > > On Mon, Feb 2, 2015 at 6:23 PM, Yu Zhang wrote: > >> Barak, >> >> If you can share the entire gc log, we can understand what is going on >> better. From the snip, user/read ratio is 12.8, and you have 13 gc >> threads. It looks normal. >> >> Thanks, >> Jenny >> >> On 2/2/2015 7:39 AM, Barak Yaish wrote: >> >> Hi, >> >> I'm having my application running on java8 with g1gc enabled. I've >> enabled gc logs, and I can see that gc time is growing ( at least this is >> what I think..) >> >> VM flags: >> -Xms6g -Xmx6g >> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX:G1HeapRegionSize=32M -XX:+PrintAdaptiveSizePolicy >> >> A snippet from the gc log: >> >> {Heap before GC invocations=342 (full 1): >> garbage-first heap total 6291456K, used 5257482K [0x0000000640000000, >> 0x00000007c0000000, 0x00000007c0000000) >> region size 32768K, 104 young (3407872K), 10 survivors (327680K) >> Metaspace used 31200K, capacity 31664K, committed 31872K, reserved >> 1077248K >> class space used 3496K, capacity 3596K, committed 3712K, reserved >> 1048576K >> 2015-02-02T17:36:48.237+0200: 4067.021: [GC pause (G1 Evacuation Pause) >> (young) 4067.021: [G1Ergonomics (CSet Construction) start choosing CSet, >> _pending_cards: 23499, predicted base time: 11.30 ms, remaining time: >> 188.70 ms, target pause time: 200.00 ms] >> 4067.021: [G1Ergonomics (CSet Construction) add young regions to CSet, >> eden: 94 regions, survivors: 10 regions, predicted young region time: >> 159.99 ms] >> 4067.021: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: >> 94 regions, survivors: 10 regions, old: 0 regions, predicted pause time: >> 171.29 ms, target pause time: 200.00 ms] >> , 0.2319995 secs] >> [Parallel Time: 229.0 ms, GC Workers: 13] >> [GC Worker Start (ms): Min: 4067021.3, Avg: 4067024.1, Max: >> 4067029.4, Diff: 8.0] >> [Ext Root Scanning (ms): Min: 0.0, Avg: 0.5, Max: 1.3, Diff: 1.3, >> Sum: 6.9] >> [Update RS (ms): Min: 0.6, Avg: 5.3, Max: 7.5, Diff: 6.9, Sum: 69.2] >> [Processed Buffers: Min: 1, Avg: 10.2, Max: 33, Diff: 32, Sum: >> 132] >> [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 1.0] >> [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, >> Sum: 0.0] >> [Object Copy (ms): Min: 219.7, Avg: 219.9, Max: 220.0, Diff: 0.3, >> Sum: 2858.1] >> [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: >> 0.9] >> [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, >> Sum: 1.2] >> [GC Worker Total (ms): Min: 220.7, Avg: 225.9, Max: 228.8, Diff: >> 8.0, Sum: 2937.3] >> [GC Worker End (ms): Min: 4067250.0, Avg: 4067250.1, Max: >> 4067250.2, Diff: 0.2] >> [Code Root Fixup: 0.2 ms] >> [Code Root Migration: 0.0 ms] >> [Code Root Purge: 0.0 ms] >> [Clear CT: 1.4 ms] >> [Other: 1.4 ms] >> [Choose CSet: 0.0 ms] >> [Ref Proc: 0.8 ms] >> [Ref Enq: 0.0 ms] >> [Redirty Cards: 0.2 ms] >> [Free CSet: 0.1 ms] >> [Eden: 3008.0M(3008.0M)->0.0B(2976.0M) Survivors: 320.0M->320.0M Heap: >> 5134.3M(6144.0M)->2168.1M(6144.0M)] >> Heap after GC invocations=343 (full 1): >> garbage-first heap total 6291456K, used 2220163K [0x0000000640000000, >> 0x00000007c0000000, 0x00000007c0000000) >> region size 32768K, 10 young (327680K), 10 survivors (327680K) >> Metaspace used 31200K, capacity 31664K, committed 31872K, reserved >> 1077248K >> class space used 3496K, capacity 3596K, committed 3712K, reserved >> 1048576K >> } >> [Times: user=2.94 sys=0.00, real=0.23 secs] >> >> And the "user" part is keep growing... Can you explain what this >> actually means? What exact process in the gc in contributing to this? And >> what can be done in order to minimize this time? >> >> Thanks. >> >> >> >> >> _______________________________________________ >> hotspot-gc-use mailing listhotspot-gc-use at openjdk.java.nethttp://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: hihgjffh.png Type: image/png Size: 15756 bytes Desc: not available URL: From yu.zhang at oracle.com Mon Feb 2 20:12:35 2015 From: yu.zhang at oracle.com (Yu Zhang) Date: Mon, 02 Feb 2015 12:12:35 -0800 Subject: Would like to understand why gc time grows In-Reply-To: References: <54CFA47B.4000503@oracle.com> <54CFD2A5.8010501@oracle.com> Message-ID: <54CFDA33.2050605@oracle.com> Barak, From the chart, the user time goes from ~250ms to 3250ms. Can not tell it is increasing. BTW, you should look at real time. As the user-time is the cpu time. If your pause target is 30ms, can you try MaxGCPauseMillis=30 and see how it goes? Your heap is under utilized, only 3g out of 6g. I think after adding this parameter, the Eden size and young gc pause would go down. The mixed gc may go up, but we can start from here. Thanks, Jenny On 2/2/2015 11:53 AM, Barak Yaish wrote: > Thanks for the quick response :-) > > My app supposed to serve as a simple web server, responding to > requests in less than 30ms, no long lived objects... Doesn't the > increasing value of "user" time indicates STW pauses? And which tool > you use to parse the log? Most tools I was trying to use are failing > to parse correctly g1 logs.. > > On Mon, Feb 2, 2015 at 9:40 PM, Yu Zhang > wrote: > > Barak, > > Thanks for the gc log. > > The stop the world pause in your log includes, young, mixed, > remark and clean up phases. > For details on how to under stand gc logs, please see > https://blogs.oracle.com/g1gc/date/20140801 > https://blogs.oracle.com/g1gc/date/20140807 > > Here is a chart of the gc pauses from your log. Can not conclude > gc pause is getting longer. > The default MaxGCPauseMillis is 200 (ms), does it work for you? Or > you need lower pause time? > > Thanks, > Jenny > > On 2/2/2015 10:58 AM, Barak Yaish wrote: >> Sure, upload the file to >> https://www.dropbox.com/sh/kinl4zo8apg6kh2/AADlh9b-X_Zk6L_tgPL_lyyma?dl=0 >> >> >> Can you explain which messages indicates the time the app was >> paused due to gc? >> >> Thanks! >> >> On Mon, Feb 2, 2015 at 6:23 PM, Yu Zhang > > wrote: >> >> Barak, >> >> If you can share the entire gc log, we can understand what is >> going on better. From the snip, user/read ratio is 12.8, and >> you have 13 gc threads. It looks normal. >> >> Thanks, >> Jenny >> >> On 2/2/2015 7:39 AM, Barak Yaish wrote: >>> Hi, >>> >>> I'm having my application running on java8 with g1gc >>> enabled. I've enabled gc logs, and I can see that gc time is >>> growing ( at least this is what I think..) >>> >>> VM flags: >>> -Xms6g -Xmx6g >>> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX:G1HeapRegionSize=32M -XX:+PrintAdaptiveSizePolicy >>> >>> >>> A snippet from the gc log: >>> >>> {Heap before GC invocations=342 (full 1): >>> garbage-first heap total 6291456K, used 5257482K >>> [0x0000000640000000, 0x00000007c0000000, 0x00000007c0000000) >>> region size 32768K, 104 young (3407872K), 10 survivors >>> (327680K) >>> Metaspace used 31200K, capacity 31664K, committed >>> 31872K, reserved 1077248K >>> class space used 3496K, capacity 3596K, committed >>> 3712K, reserved 1048576K >>> 2015-02-02T17:36:48.237+0200: 4067.021: [GC pause (G1 >>> Evacuation Pause) (young) 4067.021: [G1Ergonomics (CSet >>> Construction) start choosing CSet, _pending_cards: 23499, >>> predicted base time: 11.30 ms, remaining time: 188.70 ms, >>> target pause time: 200.00 ms] >>> 4067.021: [G1Ergonomics (CSet Construction) add young >>> regions to CSet, eden: 94 regions, survivors: 10 regions, >>> predicted young region time: 159.99 ms] >>> 4067.021: [G1Ergonomics (CSet Construction) finish choosing >>> CSet, eden: 94 regions, survivors: 10 regions, old: 0 >>> regions, predicted pause time: 171.29 ms, target pause time: >>> 200.00 ms] >>> , 0.2319995 secs] >>> [Parallel Time: 229.0 ms, GC Workers: 13] >>> [GC Worker Start (ms): Min: 4067021.3, Avg: 4067024.1, >>> Max: 4067029.4, Diff: 8.0] >>> [Ext Root Scanning (ms): Min: 0.0, Avg: 0.5, Max: 1.3, >>> Diff: 1.3, Sum: 6.9] >>> [Update RS (ms): Min: 0.6, Avg: 5.3, Max: 7.5, Diff: >>> 6.9, Sum: 69.2] >>> [Processed Buffers: Min: 1, Avg: 10.2, Max: 33, >>> Diff: 32, Sum: 132] >>> [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: >>> 0.1, Sum: 1.0] >>> [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: >>> 0.0, Diff: 0.0, Sum: 0.0] >>> [Object Copy (ms): Min: 219.7, Avg: 219.9, Max: 220.0, >>> Diff: 0.3, Sum: 2858.1] >>> [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: >>> 0.1, Sum: 0.9] >>> [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, >>> Diff: 0.2, Sum: 1.2] >>> [GC Worker Total (ms): Min: 220.7, Avg: 225.9, Max: >>> 228.8, Diff: 8.0, Sum: 2937.3] >>> [GC Worker End (ms): Min: 4067250.0, Avg: 4067250.1, >>> Max: 4067250.2, Diff: 0.2] >>> [Code Root Fixup: 0.2 ms] >>> [Code Root Migration: 0.0 ms] >>> [Code Root Purge: 0.0 ms] >>> [Clear CT: 1.4 ms] >>> [Other: 1.4 ms] >>> [Choose CSet: 0.0 ms] >>> [Ref Proc: 0.8 ms] >>> [Ref Enq: 0.0 ms] >>> [Redirty Cards: 0.2 ms] >>> [Free CSet: 0.1 ms] >>> [Eden: 3008.0M(3008.0M)->0.0B(2976.0M) Survivors: >>> 320.0M->320.0M Heap: 5134.3M(6144.0M)->2168.1M(6144.0M)] >>> Heap after GC invocations=343 (full 1): >>> garbage-first heap total 6291456K, used 2220163K >>> [0x0000000640000000, 0x00000007c0000000, 0x00000007c0000000) >>> region size 32768K, 10 young (327680K), 10 survivors (327680K) >>> Metaspace used 31200K, capacity 31664K, committed >>> 31872K, reserved 1077248K >>> class space used 3496K, capacity 3596K, committed >>> 3712K, reserved 1048576K >>> } >>> [Times: user=2.94 sys=0.00, real=0.23 secs] >>> >>> And the "user" part is keep growing... Can you explain what >>> this actually means? What exact process in the gc in >>> contributing to this? And what can be done in order to >>> minimize this time? >>> >>> Thanks. >>> >>> >>> >>> >>> _______________________________________________ >>> hotspot-gc-use mailing list >>> hotspot-gc-use at openjdk.java.net >>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 15756 bytes Desc: not available URL: From jon.masamitsu at oracle.com Mon Feb 2 20:39:17 2015 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Mon, 02 Feb 2015 12:39:17 -0800 Subject: longer ParNew collection in JDK8? In-Reply-To: <12224176.1048123.1422902296117.JavaMail.yahoo@mail.yahoo.com> References: <54CFBC9E.9070700@oracle.com> <12224176.1048123.1422902296117.JavaMail.yahoo@mail.yahoo.com> Message-ID: <54CFE075.2020900@oracle.com> Joy, If you look at the tenuring distribution (-XX:+PrintTenuringDistribution), do they look similar? Jon On 02/02/2015 10:38 AM, Joy Xiong wrote: > Hi Jon, > > Yes, most objects in my application die young and very little > promoted. As you can see from the attached figure (the occupied heap > size from Sep 28 to Nov 11), CMS happened very rare. However, it seems > two releases do not have much difference in promotion rate, as they > have same number of CMS in the log. > > We have the same JVM parameters for both releases as below: > > CommandLine flags: -XX:+AlwaysPreTouch > -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=logs/hs_err.log > -XX:InitialHeapSize=34271657984 -XX:+ManagementServer > -XX:MaxHeapSize=34271657984 -XX:MaxNewSize=2147483648 > -XX:NewSize=2147483648 -XX:OldPLABSize=16 -XX:+PrintGC > -XX:+PrintGCApplicationConcurrentTime > -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps > -XX:+PrintGCDetails -XX:+PrintGCTimeStamps > -XX:+PrintTenuringDistribution -XX:SurvivorRatio=2 > -XX:+UnlockDiagnosticVMOptions -XX:+UseCompressedClassPointers > -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC > > thanks, > -Joy > > > On Monday, February 2, 2015 10:06 AM, Jon Masamitsu > wrote: > > > Joy, > > Is your application such that most everything dies in the > young generation and very little is promoted? If yes, > do you know if there is a difference in how much is > promoted between the two releases? > > Jon > > On 01/29/2015 10:27 AM, Joy Xiong wrote: >> >> Hi, >> >> We recently move our services from JDK6 to JDK8, but observe longer >> ParNew GC pause in JDK8. For example, the pause time increase from >> 27ms to 43ms in one service. The service has the JVM parameters as below: >> -Xms32684m -Xmx32684m -XX:NewSize=2048m -XX:MaxNewSize=2048m >> -XX:+UseConcMarkSweepGC -XX:+UseParNewGC >> -XX:CMSInitiatingOccupancyFraction=70 -XX:SurvivorRatio=2 >> -XX:+AlwaysPreTouch -XX:+UseCompressedOops >> -XX:+PrintTenuringDistribution -XX:+PrintGCDetails >> -XX:+PrintGCDateStamps -Xloggc:logs/gc.log >> -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime >> >> Is this longer ParNew pause expected? Any suggestions to mitigate the >> longer pause? >> >> thanks, >> -Joy >> >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Andreas.Mueller at mgm-tp.com Tue Feb 3 17:33:53 2015 From: Andreas.Mueller at mgm-tp.com (=?utf-8?B?QW5kcmVhcyBNw7xsbGVy?=) Date: Tue, 3 Feb 2015 17:33:53 +0000 Subject: AW: AW: Minor GCs grow longer after Full GC In-Reply-To: <1422466566.3211.4.camel@oracle.com> References: <46FF8393B58AD84D95E444264805D98F01A047FCA6@edata01.mgm-edv.de> <1421935690.3426.19.camel@oracle.com> <46FF8393B58AD84D95E444264805D98F01A047FCF4@edata01.mgm-edv.de> <1422466566.3211.4.camel@oracle.com> Message-ID: <46FF8393B58AD84D95E444264805D98F01A04805A9@edata01.mgm-edv.de> Hi Thomas, >Please report back any experience. This helps us a lot to further improve G1. Preferably, if there are issues, with a log :) We had an issue in our Tomcat configuration which caused the problem in the end (see below), but G1 suffered more from that problem than other collectors in Java 7-72. I'll send you the detailed GC log in separate mail because it is too large for the mailing list. >> > Yes, but it seems that in your case the perm gen (or one of those additional roots) seems to be the problem. >> Good hint, but from what do you conclude that? >From that your application is a tomcat application server, the huge perm gen in conjunction with your comments about perm gen, and some guessing :) >We have worked together with people on another application server where the same behavior has been seen, and we found that there has been a similar problem with code roots and class unloading. We have now identified and fixed the root problem: - yes, growing minor GC times were related to a growing perm heap occupation - perm heap occupation grew because JSPs were recompiled and regenerated servlet classes reloaded (cause: maxLoadedJsps was set to some value in the web.xml and exceeded over and over again) Regards and thanks Andreas From jon.masamitsu at oracle.com Wed Feb 4 16:40:56 2015 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Wed, 04 Feb 2015 08:40:56 -0800 Subject: longer ParNew collection in JDK8? In-Reply-To: <1756968588.1798357.1423003069327.JavaMail.yahoo@mail.yahoo.com> References: <54CFE075.2020900@oracle.com> <1756968588.1798357.1423003069327.JavaMail.yahoo@mail.yahoo.com> Message-ID: <54D24B98.3080600@oracle.com> Joy, Do the ParNew pauses you see have a shape similar to the occupancy seen at the different ages? Do you have a test setup where you can do some experiments? Can you send a gc log from jdk8 and jdk6? I'm not looking for anything in particular but might see something. Jon On 2/3/2015 2:37 PM, Joy Xiong wrote: > Hi Jon, > > The tenuring distribution looks similar to me. Pls see the attached > graphs, which show the occupied size in age 1, age 2, age 14 and age > 15 after the young GC. I have graphs showing occupied size in other > ages too, and they all look similar to me. Let me know you'd like to > have all of them. > > thanks, > -Joy > > > On Monday, February 2, 2015 12:46 PM, Jon Masamitsu > wrote: > > > Joy, > > If you look at the tenuring distribution > (-XX:+PrintTenuringDistribution), do > they look similar? > > Jon > > On 02/02/2015 10:38 AM, Joy Xiong wrote: >> Hi Jon, >> >> Yes, most objects in my application die young and very little >> promoted. As you can see from the attached figure (the occupied heap >> size from Sep 28 to Nov 11), CMS happened very rare. However, it >> seems two releases do not have much difference in promotion rate, as >> they have same number of CMS in the log. >> >> We have the same JVM parameters for both releases as below: >> >> CommandLine flags: -XX:+AlwaysPreTouch >> -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=logs/hs_err.log >> -XX:InitialHeapSize=34271657984 -XX:+ManagementServer >> -XX:MaxHeapSize=34271657984 -XX:MaxNewSize=2147483648 >> -XX:NewSize=2147483648 -XX:OldPLABSize=16 -XX:+PrintGC >> -XX:+PrintGCApplicationConcurrentTime >> -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps >> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps >> -XX:+PrintTenuringDistribution -XX:SurvivorRatio=2 >> -XX:+UnlockDiagnosticVMOptions -XX:+UseCompressedClassPointers >> -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC >> >> thanks, >> -Joy >> >> >> On Monday, February 2, 2015 10:06 AM, Jon Masamitsu >> wrote: >> >> >> Joy, >> >> Is your application such that most everything dies in the >> young generation and very little is promoted? If yes, >> do you know if there is a difference in how much is >> promoted between the two releases? >> >> Jon >> >> On 01/29/2015 10:27 AM, Joy Xiong wrote: >>> >>> Hi, >>> >>> We recently move our services from JDK6 to JDK8, but observe longer >>> ParNew GC pause in JDK8. For example, the pause time increase from >>> 27ms to 43ms in one service. The service has the JVM parameters as >>> below: >>> -Xms32684m -Xmx32684m -XX:NewSize=2048m -XX:MaxNewSize=2048m >>> -XX:+UseConcMarkSweepGC -XX:+UseParNewGC >>> -XX:CMSInitiatingOccupancyFraction=70 -XX:SurvivorRatio=2 >>> -XX:+AlwaysPreTouch -XX:+UseCompressedOops >>> -XX:+PrintTenuringDistribution -XX:+PrintGCDetails >>> -XX:+PrintGCDateStamps -Xloggc:logs/gc.log >>> -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime >>> >>> Is this longer ParNew pause expected? Any suggestions to mitigate >>> the longer pause? >>> >>> thanks, >>> -Joy >>> >>> >>> _______________________________________________ >>> hotspot-gc-use mailing list >>> hotspot-gc-use at openjdk.java.net >>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From staffan.friberg at oracle.com Wed Feb 4 17:57:00 2015 From: staffan.friberg at oracle.com (Staffan Friberg) Date: Wed, 04 Feb 2015 09:57:00 -0800 Subject: longer ParNew collection in JDK8? In-Reply-To: <12224176.1048123.1422902296117.JavaMail.yahoo@mail.yahoo.com> References: <54CFBC9E.9070700@oracle.com> <12224176.1048123.1422902296117.JavaMail.yahoo@mail.yahoo.com> Message-ID: <54D25D6C.7010805@oracle.com> Hi Joy, Could you try with the following extra option, it would change the setting to be similar in JDK 7 as they are in JDK 6. -XX:+UnlockExperimentalVMOptions -XX:WorkStealingYieldsBeforeSleep=1000 //Staffan On 02/02/2015 10:38 AM, Joy Xiong wrote: > Hi Jon, > > Yes, most objects in my application die young and very little > promoted. As you can see from the attached figure (the occupied heap > size from Sep 28 to Nov 11), CMS happened very rare. However, it seems > two releases do not have much difference in promotion rate, as they > have same number of CMS in the log. > > We have the same JVM parameters for both releases as below: > > CommandLine flags: -XX:+AlwaysPreTouch > -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=logs/hs_err.log > -XX:InitialHeapSize=34271657984 -XX:+ManagementServer > -XX:MaxHeapSize=34271657984 -XX:MaxNewSize=2147483648 > -XX:NewSize=2147483648 -XX:OldPLABSize=16 -XX:+PrintGC > -XX:+PrintGCApplicationConcurrentTime > -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps > -XX:+PrintGCDetails -XX:+PrintGCTimeStamps > -XX:+PrintTenuringDistribution -XX:SurvivorRatio=2 > -XX:+UnlockDiagnosticVMOptions -XX:+UseCompressedClassPointers > -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC > > thanks, > -Joy > > > On Monday, February 2, 2015 10:06 AM, Jon Masamitsu > wrote: > > > Joy, > > Is your application such that most everything dies in the > young generation and very little is promoted? If yes, > do you know if there is a difference in how much is > promoted between the two releases? > > Jon > > On 01/29/2015 10:27 AM, Joy Xiong wrote: >> >> Hi, >> >> We recently move our services from JDK6 to JDK8, but observe longer >> ParNew GC pause in JDK8. For example, the pause time increase from >> 27ms to 43ms in one service. The service has the JVM parameters as below: >> -Xms32684m -Xmx32684m -XX:NewSize=2048m -XX:MaxNewSize=2048m >> -XX:+UseConcMarkSweepGC -XX:+UseParNewGC >> -XX:CMSInitiatingOccupancyFraction=70 -XX:SurvivorRatio=2 >> -XX:+AlwaysPreTouch -XX:+UseCompressedOops >> -XX:+PrintTenuringDistribution -XX:+PrintGCDetails >> -XX:+PrintGCDateStamps -Xloggc:logs/gc.log >> -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime >> >> Is this longer ParNew pause expected? Any suggestions to mitigate the >> longer pause? >> >> thanks, >> -Joy >> >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: From joyxiong at yahoo.com Wed Feb 4 18:45:15 2015 From: joyxiong at yahoo.com (Joy Xiong) Date: Wed, 4 Feb 2015 18:45:15 +0000 (UTC) Subject: longer ParNew collection in JDK8? In-Reply-To: <1835019261.2277885.1423074877584.JavaMail.yahoo@mail.yahoo.com> References: <54D24B98.3080600@oracle.com> <324050522.2273966.1423074334977.JavaMail.yahoo@mail.yahoo.com> <1835019261.2277885.1423074877584.JavaMail.yahoo@mail.yahoo.com> Message-ID: <1184526080.2266751.1423075515289.JavaMail.yahoo@mail.yahoo.com> Hi Jon, The graph showing ParNew pauses are as attached. Yes, it has a similar shape as the occupancy in the different ages. What experiments would you like me to try? I don't have experimental set up at hand, but I may be able to try some tests on prod machines. The logs are as attached in the previous email. thanks,-Joy On Wednesday, February 4, 2015 10:34 AM, Joy Xiong wrote: It seems the files are too big. Let me try with compressed version On Wednesday, February 4, 2015 10:25 AM, Joy Xiong wrote: Hi Jon, The graph showing ParNew pauses are as attached. Yes, it has a similar shape as the occupancy in the different ages. What experiments would you like me to try? I don't have experimental set up at hand, but I may be able to try some tests on prod machines. The logs are as attached. thanks,-Joy On Wednesday, February 4, 2015 8:41 AM, Jon Masamitsu wrote: Joy, Do the ParNew pauses you see have a shape similar to the occupancy seen at the different ages? Do you have a test setup where you can do some experiments? Can you send a gc log from jdk8 and jdk6?? I'm not looking for anything in particular but might see something. Jon On 2/3/2015 2:37 PM, Joy Xiong wrote: Hi Jon, The tenuring distribution looks similar to me. ?Pls see the attached graphs, which show the occupied size in age 1, age 2, age 14 and age 15 after the young GC. I have graphs showing occupied size in other ages too, and they all look similar to me. Let me know you'd like to have all of them. thanks, -Joy? On Monday, February 2, 2015 12:46 PM, Jon Masamitsu wrote: Joy, If you look at the tenuring distribution (-XX:+PrintTenuringDistribution), do they look similar? Jon On 02/02/2015 10:38 AM, Joy Xiong wrote: Hi Jon, Yes, most objects in my application die young and very little promoted. As you can see from the attached figure (the occupied heap size from Sep 28 to Nov 11), CMS happened very rare. However, it seems two releases do not have much difference in promotion rate, as they have same number of CMS in the log. We have the same JVM parameters for both releases as below: CommandLine flags: -XX:+AlwaysPreTouch -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=logs/hs_err.log -XX:InitialHeapSize=34271657984 -XX:+ManagementServer -XX:MaxHeapSize=34271657984 -XX:MaxNewSize=2147483648 -XX:NewSize=2147483648 -XX:OldPLABSize=16 -XX:+PrintGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:SurvivorRatio=2 -XX:+UnlockDiagnosticVMOptions -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC thanks, -Joy On Monday, February 2, 2015 10:06 AM, Jon Masamitsu wrote: Joy, Is your application such that most everything dies in the young generation and very little is promoted?? If yes, do you know if there is a difference in how much is promoted between the two releases? Jon On 01/29/2015 10:27 AM, Joy Xiong wrote: Hi, We recently move our services from JDK6 to JDK8, but observe longer ParNew GC pause in JDK8. For example, the pause time increase from 27ms to 43ms in one service. The service has the JVM parameters as below: -Xms32684m -Xmx32684m -XX:NewSize=2048m -XX:MaxNewSize=2048m -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=70 -XX:SurvivorRatio=2 -XX:+AlwaysPreTouch -XX:+UseCompressedOops -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:logs/gc.log -XX:+PrintGCApplicationStoppedTime-XX:+PrintGCApplicationConcurrentTime Is this longer ParNew pause expected? Any suggestions to mitigate the longer pause? thanks, -Joy _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2015-02-04 at 10.15.12 AM.png Type: image/png Size: 31045 bytes Desc: not available URL: From joyxiong at yahoo.com Wed Feb 4 22:45:43 2015 From: joyxiong at yahoo.com (Joy Xiong) Date: Wed, 4 Feb 2015 22:45:43 +0000 (UTC) Subject: longer ParNew collection in JDK8? In-Reply-To: <151238641.2407918.1423086278034.JavaMail.yahoo@mail.yahoo.com> References: <54D25D6C.7010805@oracle.com> <151238641.2407918.1423086278034.JavaMail.yahoo@mail.yahoo.com> Message-ID: <215756293.2414130.1423089943668.JavaMail.yahoo@mail.yahoo.com> Hi Staffan, Thanks for the suggestion. I will try to find an experimental testbed and give it a try. Also for you info, it seems to me the longer ParNew pause is introduced in JDK7, as I saw similar increase when I used JDK7. thanks,-Joy On Wednesday, February 4, 2015 9:57 AM, Staffan Friberg wrote: Hi Joy, Could you try with the following extra option, it would change the setting to be similar in JDK 7 as they are in JDK 6. -XX:+UnlockExperimentalVMOptions -XX:WorkStealingYieldsBeforeSleep=1000 //Staffan On 02/02/2015 10:38 AM, Joy Xiong wrote: Hi Jon, Yes, most objects in my application die young and very little promoted. As you can see from the attached figure (the occupied heap size from Sep 28 to Nov 11), CMS happened very rare. However, it seems two releases do not have much difference in promotion rate, as they have same number of CMS in the log. We have the same JVM parameters for both releases as below: CommandLine flags: -XX:+AlwaysPreTouch -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=logs/hs_err.log -XX:InitialHeapSize=34271657984 -XX:+ManagementServer -XX:MaxHeapSize=34271657984 -XX:MaxNewSize=2147483648 -XX:NewSize=2147483648 -XX:OldPLABSize=16 -XX:+PrintGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:SurvivorRatio=2 -XX:+UnlockDiagnosticVMOptions -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC thanks, -Joy On Monday, February 2, 2015 10:06 AM, Jon Masamitsu wrote: Joy, Is your application such that most everything dies in the young generation and very little is promoted?? If yes, do you know if there is a difference in how much is promoted between the two releases? Jon On 01/29/2015 10:27 AM, Joy Xiong wrote: Hi, We recently move our services from JDK6 to JDK8, but observe longer ParNew GC pause in JDK8. For example, the pause time increase from 27ms to 43ms in one service. The service has the JVM parameters as below: -Xms32684m -Xmx32684m -XX:NewSize=2048m -XX:MaxNewSize=2048m -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=70 -XX:SurvivorRatio=2 -XX:+AlwaysPreTouch -XX:+UseCompressedOops -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:logs/gc.log -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime Is this longer ParNew pause expected? Any suggestions to mitigate the longer pause? thanks, -Joy _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: From joyxiong at yahoo.com Fri Feb 6 19:17:57 2015 From: joyxiong at yahoo.com (Joy Xiong) Date: Fri, 6 Feb 2015 19:17:57 +0000 (UTC) Subject: longer ParNew collection in JDK8? In-Reply-To: <215756293.2414130.1423089943668.JavaMail.yahoo@mail.yahoo.com> References: <215756293.2414130.1423089943668.JavaMail.yahoo@mail.yahoo.com> Message-ID: <1480901824.951743.1423250277694.JavaMail.yahoo@mail.yahoo.com> I tried the experimental flag:?-XX:+UnlockExperimentalVMOptions -XX:WorkStealingYieldsBeforeSleep=1000but the ParNew pause time is very close to JDK8 without the change. Here is the comparison data of ParNew pause in millisecond:JDK6: 13JDK8: 36JDK8 (with WorkStealingYieldsBeforeSleep=1000): 38 I notice there are 4 tunables related with WorkStealingXXX, I am not clear what these tunables are. But is it worthwhile to try a combination of these?WorkStealingYieldsBeforeSleep,?WorkStealingHardSpins??WorkStealingSleepMillis?WorkStealingSpinToYieldRatio ? ? ?? thanks,-Joy On Wednesday, February 4, 2015 2:45 PM, Joy Xiong wrote: Hi Staffan, Thanks for the suggestion. I will try to find an experimental testbed and give it a try. Also for you info, it seems to me the longer ParNew pause is introduced in JDK7, as I saw similar increase when I used JDK7. thanks,-Joy On Wednesday, February 4, 2015 9:57 AM, Staffan Friberg wrote: Hi Joy, Could you try with the following extra option, it would change the setting to be similar in JDK 7 as they are in JDK 6. -XX:+UnlockExperimentalVMOptions -XX:WorkStealingYieldsBeforeSleep=1000 //Staffan On 02/02/2015 10:38 AM, Joy Xiong wrote: Hi Jon, Yes, most objects in my application die young and very little promoted. As you can see from the attached figure (the occupied heap size from Sep 28 to Nov 11), CMS happened very rare. However, it seems two releases do not have much difference in promotion rate, as they have same number of CMS in the log. We have the same JVM parameters for both releases as below: CommandLine flags: -XX:+AlwaysPreTouch -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=logs/hs_err.log -XX:InitialHeapSize=34271657984 -XX:+ManagementServer -XX:MaxHeapSize=34271657984 -XX:MaxNewSize=2147483648 -XX:NewSize=2147483648 -XX:OldPLABSize=16 -XX:+PrintGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:SurvivorRatio=2 -XX:+UnlockDiagnosticVMOptions -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC thanks, -Joy On Monday, February 2, 2015 10:06 AM, Jon Masamitsu wrote: Joy, Is your application such that most everything dies in the young generation and very little is promoted?? If yes, do you know if there is a difference in how much is promoted between the two releases? Jon On 01/29/2015 10:27 AM, Joy Xiong wrote: Hi, We recently move our services from JDK6 to JDK8, but observe longer ParNew GC pause in JDK8. For example, the pause time increase from 27ms to 43ms in one service. The service has the JVM parameters as below: -Xms32684m -Xmx32684m -XX:NewSize=2048m -XX:MaxNewSize=2048m -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=70 -XX:SurvivorRatio=2 -XX:+AlwaysPreTouch -XX:+UseCompressedOops -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:logs/gc.log -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime Is this longer ParNew pause expected? Any suggestions to mitigate the longer pause? thanks, -Joy _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: From staffan.friberg at oracle.com Fri Feb 6 21:04:13 2015 From: staffan.friberg at oracle.com (Staffan Friberg) Date: Fri, 06 Feb 2015 13:04:13 -0800 Subject: longer ParNew collection in JDK8? In-Reply-To: <1480901824.951743.1423250277694.JavaMail.yahoo@mail.yahoo.com> References: <215756293.2414130.1423089943668.JavaMail.yahoo@mail.yahoo.com> <1480901824.951743.1423250277694.JavaMail.yahoo@mail.yahoo.com> Message-ID: <54D52C4D.7090309@oracle.com> Hi Joy, I don't think it is worth while exploring this further. The reason for me asking was that on a particular workload I saw that the increased work stealing before yielding caused longer young collections times. The updated value generally improves or doesn't affect the YC times, but have seen at least one case and wanted to make sure it was not more common than we believed when updating it. Thanks for taking the time to verify. //Staffan On 02/06/2015 11:17 AM, Joy Xiong wrote: > I tried the experimental flag: > -XX:+UnlockExperimentalVMOptions -XX:WorkStealingYieldsBeforeSleep=1000 > but the ParNew pause time is very close to JDK8 without the change. > > Here is the comparison data of ParNew pause in millisecond: > JDK6: 13 > JDK8: 36 > JDK8 (with WorkStealingYieldsBeforeSleep=1000): 38 > > I notice there are 4 tunables related with WorkStealingXXX, I am not > clear what these tunables are. But is it worthwhile to try a > combination of these? > /WorkStealingYieldsBeforeSleep/, > /WorkStealingHardSpins / > /WorkStealingSleepMillis / > /WorkStealingSpinToYieldRatio / > > thanks, > -Joy > > > On Wednesday, February 4, 2015 2:45 PM, Joy Xiong > wrote: > > > Hi Staffan, > > Thanks for the suggestion. I will try to find an experimental testbed > and give it a try. Also for you info, it seems to me the longer ParNew > pause is introduced in JDK7, as I saw similar increase when I used JDK7. > > thanks, > -Joy > > > On Wednesday, February 4, 2015 9:57 AM, Staffan Friberg > wrote: > > > Hi Joy, > > Could you try with the following extra option, it would change the > setting to be similar in JDK 7 as they are in JDK 6. > > -XX:+UnlockExperimentalVMOptions -XX:WorkStealingYieldsBeforeSleep=1000 > > //Staffan > > On 02/02/2015 10:38 AM, Joy Xiong wrote: >> Hi Jon, >> >> Yes, most objects in my application die young and very little >> promoted. As you can see from the attached figure (the occupied heap >> size from Sep 28 to Nov 11), CMS happened very rare. However, it >> seems two releases do not have much difference in promotion rate, as >> they have same number of CMS in the log. >> >> We have the same JVM parameters for both releases as below: >> >> CommandLine flags: -XX:+AlwaysPreTouch >> -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=logs/hs_err.log >> -XX:InitialHeapSize=34271657984 -XX:+ManagementServer >> -XX:MaxHeapSize=34271657984 -XX:MaxNewSize=2147483648 >> -XX:NewSize=2147483648 -XX:OldPLABSize=16 -XX:+PrintGC >> -XX:+PrintGCApplicationConcurrentTime >> -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps >> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps >> -XX:+PrintTenuringDistribution -XX:SurvivorRatio=2 >> -XX:+UnlockDiagnosticVMOptions -XX:+UseCompressedClassPointers >> -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC >> >> thanks, >> -Joy >> >> >> On Monday, February 2, 2015 10:06 AM, Jon Masamitsu >> wrote: >> >> >> Joy, >> >> Is your application such that most everything dies in the >> young generation and very little is promoted? If yes, >> do you know if there is a difference in how much is >> promoted between the two releases? >> >> Jon >> >> On 01/29/2015 10:27 AM, Joy Xiong wrote: >>> >>> Hi, >>> >>> We recently move our services from JDK6 to JDK8, but observe longer >>> ParNew GC pause in JDK8. For example, the pause time increase from >>> 27ms to 43ms in one service. The service has the JVM parameters as >>> below: >>> -Xms32684m -Xmx32684m -XX:NewSize=2048m -XX:MaxNewSize=2048m >>> -XX:+UseConcMarkSweepGC -XX:+UseParNewGC >>> -XX:CMSInitiatingOccupancyFraction=70 -XX:SurvivorRatio=2 >>> -XX:+AlwaysPreTouch -XX:+UseCompressedOops >>> -XX:+PrintTenuringDistribution -XX:+PrintGCDetails >>> -XX:+PrintGCDateStamps -Xloggc:logs/gc.log >>> -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime >>> >>> Is this longer ParNew pause expected? Any suggestions to mitigate >>> the longer pause? >>> >>> thanks, >>> -Joy >>> >>> >>> _______________________________________________ >>> hotspot-gc-use mailing list >>> hotspot-gc-use at openjdk.java.net >>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> >> >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From michal at frajt.eu Fri Feb 6 22:06:47 2015 From: michal at frajt.eu (Michal Frajt) Date: Fri, 6 Feb 2015 23:06:47 +0100 Subject: longer ParNew collection in JDK8? In-Reply-To: <54CFBC9E.9070700@oracle.com> References: =?iso-8859-1?q?=3C1905701844=2E2549870=2E1422556039324=2EJavaMail=2Ey?= =?iso-8859-1?q?ahoo=40mail=2Eyahoo=2Ecom=3E_=3C54CFBC9E=2E9070700=40o?= =?iso-8859-1?q?racle=2Ecom=3E?= Message-ID: Joy, The ParNew pause time is just one aspect of comparing. What is the frequency of your ParNew invocation in JDK6/7/8? It can be the JDK7/8 ParNew pause takes longer but it is invoked less frequently. Just the JDK7/8 might produce less garbage due to a better implementation of JDK classes (String does not contain offset, substring works differently, autoboxing of ints, etc) or the JVM escape analysis provides better inputs for the scalar allocation. Less garbage being produced means that filling the eden space takes longer, eden space contains more objects which survive first age, which finally makes ParNew to take longer. The promotion rate to the old generation is not impacted or can be even smaller as more objects have chance to die. You can control the ParNew invocation frequency by the size of the new generation and the survivor ratio. Btw, SurvivorRatio of 2 is not very common for an application where objects die young, but correct parameter depends on many other factors. We are currently testing JDK8-b40 where we similarly observe a bit less frequent ParNew invocations with a bit increased pause time. Regards, Michal Od: "hotspot-gc-use" hotspot-gc-use-bounces at openjdk.java.net Komu: hotspot-gc-use at openjdk.java.net Kopie: Datum: Mon, 02 Feb 2015 10:06:22 -0800 P?edmet: Re: longer ParNew collection in JDK8? > Joy, > > Is your application such that most everything dies in the > young generation and very little is promoted?? If yes, > do you know if there is a difference in how much is > promoted between the two releases? > > Jon > > > On 01/29/2015 10:27 AM, Joy Xiong wrote: > > > > > > Hi, > > We recently move our services from JDK6 to JDK8, but observe longer ParNew GC pause in JDK8. For example, the pause time increase from 27ms to 43ms in one service. The service has the JVM parameters as below: > > -Xms32684m -Xmx32684m -XX:NewSize=2048m -XX:MaxNewSize=2048m -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=70 -XX:SurvivorRatio=2 -XX:+AlwaysPreTouch -XX:+UseCompressedOops -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:logs/gc.log -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime > > Is this longer ParNew pause expected? Any suggestions to mitigate the longer pause? > > thanks, > -Joy > > > _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > -------------- next part -------------- An HTML attachment was scrubbed... URL: From simone.bordet at gmail.com Tue Feb 10 18:34:52 2015 From: simone.bordet at gmail.com (Simone Bordet) Date: Tue, 10 Feb 2015 19:34:52 +0100 Subject: WeakReference processing: long times during G1 remark Message-ID: Hi, I just want to double check with the experts whether what I see is normal or needs more investigation. The application uses one ThreadLocal, which eventually generates WeakReferences, lots of them. Below you can find 2 G1 remark GC log lines that are 39s apart. -XX:+ParallelRefProcEnabled is on, JDK 8u25. What I am interested in is understanding if ~35s of CPU time is considered normal to process 3M of weak references, since seems a lot of time (about 12 micros each). Also, the weak refs number reported in the GC log line is the number of weak refs that have been GC marked, or the number of weak refs that have been enqueued (their referent is weakly reachable) ? I ask because we have instrumented WeakReference allocation, and the numbers shows an allocation of about 2.5M weak refs per hour, while here seems 3M in 39s. Furthermore, we have no evidence that 3M weak refs survive for a long period of time, and we see a big difference in the number of weak refs processed during a young GC (at most around 1K, more often in the tens or hundreds), and the 3M processed during remark. Am I right in assuming that a young GC processes the weak refs from the young generation, while a remark processes the weak refs that have been found during marking (and therefore those that are in old generation) ? Another thing I noticed is that the weak refs processed during the mixed GC do not sum up to the 3M processed during remark: at most few thousands vs 3M. Not sure they should, I just wanted to report this information in case it matters. Is there any known issue about ref processing during the remark phase ? Thanks ! 2015-02-09T12:07:00.327-0800: 31671.525: [GC remark 31671.530: [GC ref-proc31671.530: [SoftReference, 2 refs, 0.0052296 secs]31671.535: [WeakReference, 3264308 refs, 2.0524238 secs]31673.588: [FinalReference, 215 refs, 0.0028225 secs]31673.591: [PhantomReference, 1787 refs, 0.0050046 secs]31673.596: [JNI Weak Reference, 0.0007776 secs], 2.0932150 secs], 2.1138048 secs] [Times: user=36.39 sys=0.68, real=2.11 secs] 2015-02-09T12:07:39.210-0800: 31710.408: [GC remark 31710.413: [GC ref-proc31710.413: [SoftReference, 3 refs, 0.0064721 secs]31710.420: [WeakReference, 3168257 refs, 1.7807508 secs]31712.201: [FinalReference, 268 refs, 0.0025569 secs]31712.203: [PhantomReference, 1702 refs, 0.0038438 secs]31712.207: [JNI Weak Reference, 0.0007997 secs], 1.8847662 secs], 1.9052513 secs] [Times: user=34.54 sys=0.63, real=1.90 secs] -- Simone Bordet http://bordet.blogspot.com --- Finally, no matter how good the architecture and design are, to deliver bug-free software with optimal performance and reliability, the implementation technique must be flawless. Victoria Livschitz From dvdeepankar.reddy at gmail.com Thu Feb 12 00:12:58 2015 From: dvdeepankar.reddy at gmail.com (D vd Reddy) Date: Wed, 11 Feb 2015 16:12:58 -0800 Subject: Question about RS Set size Message-ID: Hi, we are using G1 GC for heap size of about 92 - 96 GB and we observed in the logs that Scan RS times were too high (comparable to Object Copy time, Is this normal ?, I have checked that there are no coarsening(s)). So in order to counter that we plan to decrease the region count, currently we set it explicitly to 4096 with regionSize to 32MB. My question is that if we drop setting regionSize and decrease the regions to 2048 what will happen to region size, Can it go above 32 MB and keep the regions to 2048 or is it capped at 32MB ? Thanks in advance -------------- next part -------------- An HTML attachment was scrubbed... URL: From yu.zhang at oracle.com Thu Feb 12 01:31:16 2015 From: yu.zhang at oracle.com (Yu Zhang) Date: Wed, 11 Feb 2015 17:31:16 -0800 Subject: Question about RS Set size In-Reply-To: References: Message-ID: <54DC0264.7020605@oracle.com> Hi, Currently the region size is capped at 32MB. It is good that you confirmed there is no coarsening. Is the high scanRS in mixed gc? If it is, then there are some tunings you can try to reduce it. Thanks, Jenny On 2/11/2015 4:12 PM, D vd Reddy wrote: > > Hi, > we are using G1 GC for heap size of about 92 - 96 GB and we > observed in the logs that Scan RS times were too high (comparable to > Object Copy time, Is this normal ?, I have checked that there are no > coarsening(s)). So in order to counter that we plan to decrease the > region count, currently we set it explicitly to 4096 with regionSize > to 32MB. My question is that if we drop setting regionSize and > decrease the regions to 2048 what will happen to region size, Can it > go above 32 MB and keep the regions to 2048 or is it capped at 32MB ? > > > Thanks in advance > > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: From dvdeepankar.reddy at gmail.com Thu Feb 12 04:00:14 2015 From: dvdeepankar.reddy at gmail.com (D vd Reddy) Date: Wed, 11 Feb 2015 20:00:14 -0800 Subject: Question about RS Set size In-Reply-To: <54DC0264.7020605@oracle.com> References: <54DC0264.7020605@oracle.com> Message-ID: Hi, Thanks for the reply.If the region size is capped then GC will not be adhering to the regionCount mentioned right? Also yes the high scanRS is in mixed GC, Can you suggest the tunings / experiments so that we can reduce this. The parameters we run with are (we also ran with MaxGCPause to 200 we saw similar things) *-XX:MaxGCPauseMillis=100 -XX:G1HeapRegionSize=32m -XX:InitiatingHeapOccupancyPercent=65 -XX:G1ReservePercent=20 -XX:G1HeapWastePercent=5 -XX:G1MixedGCLiveThresholdPercent=75 -XX:G1RSetRegionEntries=4096* I am attaching a part of the log that covers the run for around 6 min (Bigger log for a day here https://raw.githubusercontent.com/dvdreddy/gc-logs/master/g1-gc.log) . I am also pasting in the mail a sample mixed GC run here for quick review, you can see that ScanRS and Object copy are at the same level 2015-02-09T10:10:25.491-0500: 5383952.213: [GC pause (mixed) Desired survivor size 285212672 bytes, new threshold 3 (max 15) - age 1: 65081144 bytes, 65081144 total - age 2: 136289896 bytes, 201371040 total - age 3: 122469928 bytes, 323840968 total 5383952.213: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 23323, predicted base time: 42.39 ms, remaining time: 57.61 ms, target pause time:\ 100.00 ms] 5383952.213: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 114 regions, survivors: 17 regions, predicted young region time: 28.59 ms] 5383952.214: [G1Ergonomics (CSet Construction) finish adding old regions to CSet, reason: predicted time is too high, predicted time: 1.30 ms, remaining time: 0.00 ms\ , old: 107 regions, min: 107 regions] 5383952.214: [G1Ergonomics (CSet Construction) added expensive regions to CSet, reason: old CSet region num not reached min, old: 107 regions, expensive: 60 regions, \ min: 107 regions, remaining time: 0.00 ms] 5383952.214: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 114 regions, survivors: 17 regions, old: 107 regions, predicted pause time: 170.47 ms, targ\ et pause time: 100.00 ms] 5383952.388: [G1Ergonomics (Mixed GCs) continue mixed GCs, reason: candidate old regions available, candidate old regions: 743 regions, reclaimable: 12156413304 bytes\ (13.82 %), threshold: 5.00 %] , 0.1751220 secs] [Parallel Time: 131.1 ms, GC Workers: 23] [GC Worker Start (ms): Min: 5383952214.8, Avg: 5383952215.2, Max: 5383952215.5, Diff: 0.7] [Ext Root Scanning (ms): Min: 5.4, Avg: 6.1, Max: 9.6, Diff: 4.3, Sum: 140.5] [Update RS (ms): Min: 3.7, Avg: 6.7, Max: 7.7, Diff: 4.0, Sum: 154.8] [Processed Buffers: Min: 9, Avg: 14.6, Max: 22, Diff: 13, Sum: 335] [Scan RS (ms): Min: 50.9, Avg: 51.2, Max: 51.4, Diff: 0.5, Sum: 1178.2] [Object Copy (ms): Min: 65.3, Avg: 65.6, Max: 66.1, Diff: 0.8, Sum: 1507.9] [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.5] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 1.5] [GC Worker Total (ms): Min: 129.4, Avg: 129.7, Max: 130.2, Diff: 0.8, Sum: 2983.3] [GC Worker End (ms): Min: 5383952344.9, Avg: 5383952344.9, Max: 5383952345.1, Diff: 0.2] [Code Root Fixup: 0.0 ms] [Clear CT: 7.8 ms] [Other: 36.2 ms] [Choose CSet: 1.6 ms] [Ref Proc: 6.1 ms] [Ref Enq: 0.1 ms] [Free CSet: 3.8 ms] [Eden: 3648.0M(3648.0M)->0.0B(3808.0M) Survivors: 544.0M->384.0M Heap: 56.6G(81.9G)->50.6G(81.9G)] [Times: user=3.10 sys=0.00, real=0.18 secs] On Wed, Feb 11, 2015 at 5:31 PM, Yu Zhang wrote: > Hi, > > Currently the region size is capped at 32MB. > > It is good that you confirmed there is no coarsening. Is the high scanRS > in mixed gc? If it is, then there are some tunings you can try to reduce > it. > > Thanks, > Jenny > > On 2/11/2015 4:12 PM, D vd Reddy wrote: > > > Hi, > we are using G1 GC for heap size of about 92 - 96 GB and we observed > in the logs that Scan RS times were too high (comparable to Object Copy > time, Is this normal ?, I have checked that there are no coarsening(s)). So > in order to counter that we plan to decrease the region count, currently we > set it explicitly to 4096 with regionSize to 32MB. My question is that if > we drop setting regionSize and decrease the regions to 2048 what will > happen to region size, Can it go above 32 MB and keep the regions to 2048 > or is it capped at 32MB ? > > > Thanks in advance > > > > _______________________________________________ > hotspot-gc-use mailing listhotspot-gc-use at openjdk.java.nethttp://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: g1-gc.log Type: application/octet-stream Size: 54199 bytes Desc: not available URL: From wolfgang.pedot at finkzeit.at Fri Feb 13 13:36:39 2015 From: wolfgang.pedot at finkzeit.at (Wolfgang Pedot) Date: Fri, 13 Feb 2015 14:36:39 +0100 Subject: G1GC, Java8u40ea, Metaspace questions Message-ID: <821215C9-36AC-41BB-A9A6-1E136341778F@finkzeit.at> Hello, my last mail regarding this topic seems to have been lost somewhere but thats not really a problem because my questions have changed: I am currently evaluation 8u40 early-access (b21 and b23) for an application server which, unfortunately, generates a fair amount of short-lived dynamic classes. So far it runs on 7u67 and has enough PermGen space to survive most days with 1 FullGC to get rid of the classes. This situation is somewhat acceptable but I am really looking forward to not requiring FullGCs for class unloading, hence my tests with 8u40ea. I am testing on a smaller system with much less headroom in PermGen and as a first shot I just renamed MaxPermSize to MaxMetaspaceSize and everything looked OK at first, classes where unloaded without FullGCs and I was happy. When the Metaspace baseline-usage grew I noticed that there where quite some FullGCs after all and when looking closer I noticed that they came in pairs: 276129.049: [Full GC (Metadata GC Threshold) 2015-02-05T17:03:04.509+0100: 276129.049: [GC concurrent-root-region-scan-end, 0.0001449 secs] 2015-02-05T17:03:04.509+0100: 276129.049: [GC concurrent-mark-start] 492M->476M(904M), 1.5020455 secs] [Eden: 0.0B(336.0M)->0.0B(352.0M) Survivors: 16.0M->0.0B Heap: 492.2M(904.0M)->476.4M(904.0M)], [Metaspace: 183332K->183332K(1247232K)] [Times: user=2.74 sys=0.01, real=1.50 secs] 2015-02-05T17:03:06.011+0100: 276130.551: [Full GC (Last ditch collection) 476M->468M(904M), 1.4614149 secs] [Eden: 0.0B(352.0M)->0.0B(360.0M) Survivors: 0.0B->0.0B Heap: 476.4M(904.0M)->468.0M(904.0M)], [Metaspace: 183332K->178688K(1247232K)] [Times: user=2.62 sys=0.01, real=1.46 secs] MaxMetaspaceSize was set to 228M at the time so it was not completely full yet the JVM threw in a "Last ditch collection" after every (quite frequent) FullGC to desperately try and free some Metaspace. I even got to the point where the JVM threw OOME:Metaspace errors even though according to the numbers Metaspace was nowhere near full (different settings though). After banging my head against various settings regarding Metaspace I decided to run some worst-case tests on my laptop with different MaxMetaspaceSize-values (no other metaspace related settings). My test generates lots of classes while causing only minor increases in old-gen so the JVM runs into FullGC (Metaspace threshold) consistently. I discovered that the reported Metaspace-Size in the gclog (and jconsole) never actually reaches the configured maximum but instead tops out at about 85% of that value. "Commited" will be the Maxsize but "capacity" never goes that high and since my test-system did not have enough headroom to cover that it was collecting because of Metaspace way to often. This brings me to my first question: What is the reason why the JVM (appearantly) does not use the whole space it is allowed to use? Is other stuff stored in there but not reported or is there some kind of overhead? After giving the test-system more headroom it appears to work OK but it always takes a couple of days use before Metaspace-baseline evens out so I will see next week if that did the trick, I dont really want to go "no limit". Now a more general question: I noticed that G1 also triggers a concurrent marking cycle because of "Metaspace threshold", is it using the same threshold that would also trigger a FullGC or is it slightly lower? Is that treshold visible somewhere? The reason I am asking is that I want to keep FullGCs down to a minimum so if the class count rises reasonably slow even without causing much oldgen-increase should the concurrent cycle be triggered before the JVM throws in a FullGC most of the time? Quite a long Mail for two questions, any hints would be greatly appreciated! Wolfgang From yu.zhang at oracle.com Fri Feb 13 17:20:20 2015 From: yu.zhang at oracle.com (Yu Zhang) Date: Fri, 13 Feb 2015 09:20:20 -0800 Subject: G1GC, Java8u40ea, Metaspace questions In-Reply-To: <821215C9-36AC-41BB-A9A6-1E136341778F@finkzeit.at> References: <821215C9-36AC-41BB-A9A6-1E136341778F@finkzeit.at> Message-ID: <54DE3254.9030503@oracle.com> Wolfgang, Please see my comments in line. Thanks, Jenny On 2/13/2015 5:36 AM, Wolfgang Pedot wrote: > Hello, > > my last mail regarding this topic seems to have been lost somewhere but thats not really a problem because my questions have changed: > > I am currently evaluation 8u40 early-access (b21 and b23) for an application server which, unfortunately, generates a fair amount of short-lived dynamic classes. So far it runs on 7u67 and has enough PermGen space to survive most days with 1 FullGC to get rid of the classes. This situation is somewhat acceptable but I am really looking forward to not requiring FullGCs for class unloading, hence my tests with 8u40ea. > I am testing on a smaller system with much less headroom in PermGen and as a first shot I just renamed MaxPermSize to MaxMetaspaceSize and everything looked OK at first, classes where unloaded without FullGCs and I was happy. When the Metaspace baseline-usage grew I noticed that there where quite some FullGCs after all and when looking closer I noticed that they came in pairs: > > 276129.049: [Full GC (Metadata GC Threshold) 2015-02-05T17:03:04.509+0100: 276129.049: [GC concurrent-root-region-scan-end, 0.0001449 secs] > 2015-02-05T17:03:04.509+0100: 276129.049: [GC concurrent-mark-start] > 492M->476M(904M), 1.5020455 secs] > [Eden: 0.0B(336.0M)->0.0B(352.0M) Survivors: 16.0M->0.0B Heap: 492.2M(904.0M)->476.4M(904.0M)], [Metaspace: 183332K->183332K(1247232K)] > [Times: user=2.74 sys=0.01, real=1.50 secs] > 2015-02-05T17:03:06.011+0100: 276130.551: [Full GC (Last ditch collection) 476M->468M(904M), 1.4614149 secs] > [Eden: 0.0B(352.0M)->0.0B(360.0M) Survivors: 0.0B->0.0B Heap: 476.4M(904.0M)->468.0M(904.0M)], [Metaspace: 183332K->178688K(1247232K)] > [Times: user=2.62 sys=0.01, real=1.46 secs] > > MaxMetaspaceSize was set to 228M at the time so it was not completely full yet the JVM threw in a "Last ditch collection" after every (quite frequent) FullGC to desperately try and free some Metaspace. > I even got to the point where the JVM threw OOME:Metaspace errors even though according to the numbers Metaspace was nowhere near full (different settings though). MaxMetaspaceSize is the maximum size Metadata space can expand to. GC will start with metadata space defined by MetaspaceSize, then expand/shrink it. When it needs to expand it, a 'Metadata GC Threshold' full gc is called. To avoid this kind of full gc, you can increase MetaspaceSize. In jdk8u40, ClassUnloadingWithConcurrentMark(default true) so that the classes can be unloaded with concurrent marking. This will reduce the chance of running into Metadata GC Threshold' full gc > > After banging my head against various settings regarding Metaspace I decided to run some worst-case tests on my laptop with different MaxMetaspaceSize-values (no other metaspace related settings). My test generates lots of classes while causing only minor increases in old-gen so the JVM runs into FullGC (Metaspace threshold) consistently. I discovered that the reported Metaspace-Size in the gclog (and jconsole) never actually reaches the configured maximum but instead tops out at about 85% of that value. "Commited" will be the Maxsize but "capacity" never goes that high and since my test-system did not have enough headroom to cover that it was collecting because of Metaspace way to often. > > This brings me to my first question: What is the reason why the JVM (appearantly) does not use the whole space it is allowed to use? Is other stuff stored in there but not reported or is there some kind of overhead? > > After giving the test-system more headroom it appears to work OK but it always takes a couple of days use before Metaspace-baseline evens out so I will see next week if that did the trick, I dont really want to go "no limit". > > Now a more general question: > I noticed that G1 also triggers a concurrent marking cycle because of "Metaspace threshold", is it using the same threshold that would also trigger a FullGC or is it slightly lower? Is that treshold visible somewhere? > The reason I am asking is that I want to keep FullGCs down to a minimum so if the class count rises reasonably slow even without causing much oldgen-increase should the concurrent cycle be triggered before the JVM throws in a FullGC most of the time? You are talking about Full GC caused by 2 reasons: * Metaspace threshold: we can avoid this my increasing MetadataspaceSize * Allocation Failure: usually this happens after a few 'to-space exhausted'. I think this is not what you see in your log. A lot of time this is caused by poor tuning. The marking cycles are triggered after initial marking. And when initial marking is triggered, is decided by InitiatingHeapOccupancyPercent > > Quite a long Mail for two questions, any hints would be greatly appreciated! > > Wolfgang > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: From yu.zhang at oracle.com Fri Feb 13 17:51:57 2015 From: yu.zhang at oracle.com (Yu Zhang) Date: Fri, 13 Feb 2015 09:51:57 -0800 Subject: Question about RS Set size In-Reply-To: References: <54DC0264.7020605@oracle.com> Message-ID: <54DE39BD.2020508@oracle.com> Hi, -XX:G1RSetRegionEntries=4096 does not control the number of regions. The number of regions is calculated based on G1HeapRegionSize. G1RSetRegionEntries decides the entries in RSet fine grain table. Since you set it very big, you do not see coarsening. On the other hand, this could increase the memory foot print a lot. The heap size is 80g, but only used maximum 60g. Reading the log you attached, the 1st mixed gc after marking cycle is 400ms, 2nd 458ms, 3rd 537ms, 4th 744ms. Since you have a lot of unused heap, you can reduce the mixed gcs. There are several ways to do this: 1. increase G1HeapWastePercent=15 2. decrease G1MixedGCLiveThresholdPercent=65 You can try those in the order listed, or a combination. For you reference about those parameters please see https://blogs.oracle.com/g1gc/date/20140807 Thanks, Jenny On 2/11/2015 7:54 PM, D vd Reddy wrote: > Hi, > Thanks for the reply.If the region size is capped then GC will not > be adhering to the regionCount mentioned right? > > Also yes the high scanRS is in mixed GC, Can you suggest the tunings / > experiments so that we can reduce this. > > The parameters we run with are (we also ran with MaxGCPause to 200 we > saw similar things) > > *-XX:MaxGCPauseMillis=100 -XX:G1HeapRegionSize=32m -XX:InitiatingHeapOccupancyPercent=65 -XX:G1ReservePercent=20 -XX:G1HeapWastePercent=5 -XX:G1MixedGCLiveThresholdPercent=75 -XX:G1RSetRegionEntries=4096* > > I am attaching a part of the log that covers the run for around 10 min > (Bigger log for a day here > https://raw.githubusercontent.com/dvdreddy/gc-logs/master/g1-gc.log) > . I am also pasting in the mail a sample mixed GC run here for quick > review, you can see that ScanRS and Object copy are at the same level > > 2015-02-09T10:10:25.491-0500: 5383952.213: [GC pause (mixed) > Desired survivor size 285212672 bytes, new threshold 3 (max 15) > - age 1: 65081144 bytes, 65081144 total > - age 2: 136289896 bytes, 201371040 total > - age 3: 122469928 bytes, 323840968 total > 5383952.213: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 23323, predicted base time: 42.39 ms, remaining time: > 57.61 ms, target pause time:\ > 100.00 ms] > 5383952.213: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 114 regions, survivors: 17 regions, predicted young region > time: 28.59 ms] > 5383952.214: [G1Ergonomics (CSet Construction) finish adding old > regions to CSet, reason: predicted time is too high, predicted time: > 1.30 ms, remaining time: 0.00 ms\ > , old: 107 regions, min: 107 regions] > 5383952.214: [G1Ergonomics (CSet Construction) added expensive > regions to CSet, reason: old CSet region num not reached min, old: 107 > regions, expensive: 60 regions, \ > min: 107 regions, remaining time: 0.00 ms] > 5383952.214: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 114 regions, survivors: 17 regions, old: 107 regions, predicted > pause time: 170.47 ms, targ\ > et pause time: 100.00 ms] > 5383952.388: [G1Ergonomics (Mixed GCs) continue mixed GCs, reason: > candidate old regions available, candidate old regions: 743 regions, > reclaimable: 12156413304 bytes\ > (13.82 %), threshold: 5.00 %] > , 0.1751220 secs] > [Parallel Time: 131.1 ms, GC Workers: 23] > [GC Worker Start (ms): Min: 5383952214.8, Avg: 5383952215.2, > Max: 5383952215.5, Diff: 0.7] > [Ext Root Scanning (ms): Min: 5.4, Avg: 6.1, Max: 9.6, Diff: > 4.3, Sum: 140.5] > [Update RS (ms): Min: 3.7, Avg: 6.7, Max: 7.7, Diff: 4.0, Sum: > 154.8] > [Processed Buffers: Min: 9, Avg: 14.6, Max: 22, Diff: 13, > Sum: 335] > [Scan RS (ms): Min: 50.9, Avg: 51.2, Max: 51.4, Diff: 0.5, Sum: > 1178.2] > [Object Copy (ms): Min: 65.3, Avg: 65.6, Max: 66.1, Diff: 0.8, > Sum: 1507.9] > [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: > 0.5] > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, > Sum: 1.5] > [GC Worker Total (ms): Min: 129.4, Avg: 129.7, Max: 130.2, Diff: > 0.8, Sum: 2983.3] > [GC Worker End (ms): Min: 5383952344.9, Avg: 5383952344.9, Max: > 5383952345.1, Diff: 0.2] > [Code Root Fixup: 0.0 ms] > [Clear CT: 7.8 ms] > [Other: 36.2 ms] > [Choose CSet: 1.6 ms] > [Ref Proc: 6.1 ms] > [Ref Enq: 0.1 ms] > [Free CSet: 3.8 ms] > [Eden: 3648.0M(3648.0M)->0.0B(3808.0M) Survivors: 544.0M->384.0M > Heap: 56.6G(81.9G)->50.6G(81.9G)] > [Times: user=3.10 sys=0.00, real=0.18 secs] > > On Wed, Feb 11, 2015 at 5:31 PM, Yu Zhang > wrote: > > Hi, > > Currently the region size is capped at 32MB. > > It is good that you confirmed there is no coarsening. Is the high > scanRS in mixed gc? If it is, then there are some tunings you can > try to reduce it. > > Thanks, > Jenny > > On 2/11/2015 4:12 PM, D vd Reddy wrote: >> >> Hi, >> we are using G1 GC for heap size of about 92 - 96 GB and we >> observed in the logs that Scan RS times were too high (comparable >> to Object Copy time, Is this normal ?, I have checked that there >> are no coarsening(s)). So in order to counter that we plan to >> decrease the region count, currently we set it explicitly to 4096 >> with regionSize to 32MB. My question is that if we drop setting >> regionSize and decrease the regions to 2048 what will happen to >> region size, Can it go above 32 MB and keep the regions to 2048 >> or is it capped at 32MB ? >> >> >> Thanks in advance >> >> >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wolfgang.pedot at finkzeit.at Fri Feb 13 18:25:43 2015 From: wolfgang.pedot at finkzeit.at (Wolfgang Pedot) Date: Fri, 13 Feb 2015 19:25:43 +0100 Subject: G1GC, Java8u40ea, Metaspace questions In-Reply-To: <54DE3254.9030503@oracle.com> References: <821215C9-36AC-41BB-A9A6-1E136341778F@finkzeit.at> <54DE3254.9030503@oracle.com> Message-ID: <54DE41A7.6050004@finkzeit.at> Thanks for your reply, I was under the impression that the MetaspaceSize is only used as a starting value for the high-water mark and adjusting it will only change the behaviour at startup. Correct me if I am wrong but after the first collect ergonomics would take over and adjust that mark according to Min/MaxMetaspaceFreeRatio, right? My problem is that I have to deal with lots of short lived generated classes and it would be a step back to have the garbage-collector set a low mark right before a class-generation spike and then do a Full-GC before raising it again (Full-GC currently takes ~10sec on the real system). Although with MaxMetaSpaceFreeRation being 70% by default that would probably not happen that often. As for my first question, it is still not clear to me why Metaspace "capacity" never seemed to reach "commited" during my first tests. I have checked the logs for the current instance and with the higher maximum-setting "capacity" and "commited" actually can get quite close now but both are nowhere near the allowed maximum even after I have driven the JVM into multiple FullGCs by creating classes like crazy. Can I assume that RAM that is not "commited" is not actually used by the JVM and can still be used by the OS for filesystem cache and the such? Now to my second question: I was not talking about Full-GC there, what I was referring to is such a log-line: [G1Ergonomics (Concurrent Cycles) request concurrent cycle initiation, reason: requested by GC cause, GC cause: Metadata GC Threshold] this tells me that G1 will trigger a concurrent cycle when the Metaspace threshold is reached and sometimes that leads to classes getting unloaded BEFORE there is a FullGC for the very same reason. I would very much prefer this happening more often and FullGC only being used as a last-resort. Unfortunately most of the time there is a FullGC triggered right after I see the request for a concurrent cycle, this means that I can only benefit from concurrent class unloading if there is enough activity in old-gen to trigger concurrent cycles because of heap-occupation. If G1 would use a lower Metaspace threshold to initiate a concurrent cycle there would be a higher chance for classes to get unloaded without a FullGC even if heap occupation is currently low. That would be similar to what is going on in the heap, concurrent cycles are started before the limit is reached and a FullGC is required. Am I wrong here? Wolfgang Am 13.02.2015 18:20, schrieb Yu Zhang: > Wolfgang, > > Please see my comments in line. > Thanks, > Jenny > On 2/13/2015 5:36 AM, Wolfgang Pedot wrote: >> Hello, >> >> my last mail regarding this topic seems to have been lost somewhere but thats not really a problem because my questions have changed: >> >> I am currently evaluation 8u40 early-access (b21 and b23) for an application server which, unfortunately, generates a fair amount of short-lived dynamic classes. So far it runs on 7u67 and has enough PermGen space to survive most days with 1 FullGC to get rid of the classes. This situation is somewhat acceptable but I am really looking forward to not requiring FullGCs for class unloading, hence my tests with 8u40ea. >> I am testing on a smaller system with much less headroom in PermGen and as a first shot I just renamed MaxPermSize to MaxMetaspaceSize and everything looked OK at first, classes where unloaded without FullGCs and I was happy. When the Metaspace baseline-usage grew I noticed that there where quite some FullGCs after all and when looking closer I noticed that they came in pairs: >> >> 276129.049: [Full GC (Metadata GC Threshold) 2015-02-05T17:03:04.509+0100: 276129.049: [GC concurrent-root-region-scan-end, 0.0001449 secs] >> 2015-02-05T17:03:04.509+0100: 276129.049: [GC concurrent-mark-start] >> 492M->476M(904M), 1.5020455 secs] >> [Eden: 0.0B(336.0M)->0.0B(352.0M) Survivors: 16.0M->0.0B Heap: 492.2M(904.0M)->476.4M(904.0M)], [Metaspace: 183332K->183332K(1247232K)] >> [Times: user=2.74 sys=0.01, real=1.50 secs] >> 2015-02-05T17:03:06.011+0100: 276130.551: [Full GC (Last ditch collection) 476M->468M(904M), 1.4614149 secs] >> [Eden: 0.0B(352.0M)->0.0B(360.0M) Survivors: 0.0B->0.0B Heap: 476.4M(904.0M)->468.0M(904.0M)], [Metaspace: 183332K->178688K(1247232K)] >> [Times: user=2.62 sys=0.01, real=1.46 secs] >> >> MaxMetaspaceSize was set to 228M at the time so it was not completely full yet the JVM threw in a "Last ditch collection" after every (quite frequent) FullGC to desperately try and free some Metaspace. >> I even got to the point where the JVM threw OOME:Metaspace errors even though according to the numbers Metaspace was nowhere near full (different settings though). > MaxMetaspaceSize is the maximum size Metadata space can expand to. GC > will start with metadata space defined by MetaspaceSize, then > expand/shrink it. When it needs to expand it, a 'Metadata GC > Threshold' full gc is called. To avoid this kind of full gc, you can > increase MetaspaceSize. > > In jdk8u40, ClassUnloadingWithConcurrentMark(default true) so that the > classes can be unloaded with concurrent marking. This will reduce the > chance of running into Metadata GC Threshold' full gc > >> After banging my head against various settings regarding Metaspace I decided to run some worst-case tests on my laptop with different MaxMetaspaceSize-values (no other metaspace related settings). My test generates lots of classes while causing only minor increases in old-gen so the JVM runs into FullGC (Metaspace threshold) consistently. I discovered that the reported Metaspace-Size in the gclog (and jconsole) never actually reaches the configured maximum but instead tops out at about 85% of that value. "Commited" will be the Maxsize but "capacity" never goes that high and since my test-system did not have enough headroom to cover that it was collecting because of Metaspace way to often. >> >> This brings me to my first question: What is the reason why the JVM (appearantly) does not use the whole space it is allowed to use? Is other stuff stored in there but not reported or is there some kind of overhead? >> >> After giving the test-system more headroom it appears to work OK but it always takes a couple of days use before Metaspace-baseline evens out so I will see next week if that did the trick, I dont really want to go "no limit". >> >> Now a more general question: >> I noticed that G1 also triggers a concurrent marking cycle because of "Metaspace threshold", is it using the same threshold that would also trigger a FullGC or is it slightly lower? Is that treshold visible somewhere? >> The reason I am asking is that I want to keep FullGCs down to a minimum so if the class count rises reasonably slow even without causing much oldgen-increase should the concurrent cycle be triggered before the JVM throws in a FullGC most of the time? > You are talking about Full GC caused by 2 reasons: > > * > Metaspace threshold: we can avoid this my increasing MetadataspaceSize > > * Allocation Failure: usually this happens after a few 'to-space > exhausted'. I think this is not what you see in your log. A lot > of time this is caused by poor tuning. The marking cycles are > triggered after initial marking. And when initial marking is > triggered, is decided by InitiatingHeapOccupancyPercent > >> Quite a long Mail for two questions, any hints would be greatly appreciated! >> >> Wolfgang >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > -- Mit freundlichen Gr??en Wolfgang Pedot F&E ????????????????? Fink Zeitsysteme GmbH | M?slestra?e 19-21 | 6844 Altach | ?sterreich Tel: +43 5576 72388 | Fax: +43 5576 72388 14 wolfgang.pedot at finkzeit.at | www.finkzeit.at Landesgericht Feldkirch, 72223k | USt.ld: ATU36401407 Wir erbringen unsere Leistungen ausschlie?lich auf Basis unserer AGB und Leistungs- und Nutzungsvereinbarung, die wir auf unserer Webseite unter www.finkzeit.at/rechtliches ver?ffentlicht haben. From dvdeepankar.reddy at gmail.com Fri Feb 13 19:12:18 2015 From: dvdeepankar.reddy at gmail.com (D vd Reddy) Date: Fri, 13 Feb 2015 11:12:18 -0800 Subject: Question about RS Set size In-Reply-To: <54DE39BD.2020508@oracle.com> References: <54DC0264.7020605@oracle.com> <54DE39BD.2020508@oracle.com> Message-ID: Thanks for the reply, will try the parameters you suggested and report On Fri, Feb 13, 2015 at 9:51 AM, Yu Zhang wrote: > Hi, > > -XX:G1RSetRegionEntries=4096 does not control the number of regions. The > number of regions is calculated based on G1HeapRegionSize. > G1RSetRegionEntries decides the entries in RSet fine grain table. Since > you set it very big, you do not see coarsening. On the other hand, this > could increase the memory foot print a lot. > > The heap size is 80g, but only used maximum 60g. Reading the log you > attached, the 1st mixed gc after marking cycle is 400ms, 2nd 458ms, 3rd > 537ms, 4th 744ms. Since you have a lot of unused heap, you can reduce the > mixed gcs. There are several ways to do this: > 1. increase G1HeapWastePercent=15 > 2. decrease G1MixedGCLiveThresholdPercent=65 > > You can try those in the order listed, or a combination. > For you reference about those parameters please see > https://blogs.oracle.com/g1gc/date/20140807 > > Thanks, > Jenny > > On 2/11/2015 7:54 PM, D vd Reddy wrote: > > Hi, > Thanks for the reply.If the region size is capped then GC will not be > adhering to the regionCount mentioned right? > > Also yes the high scanRS is in mixed GC, Can you suggest the tunings / > experiments so that we can reduce this. > > The parameters we run with are (we also ran with MaxGCPause to 200 we > saw similar things) > > *-XX:MaxGCPauseMillis=100 -XX:G1HeapRegionSize=32m -XX:InitiatingHeapOccupancyPercent=65 -XX:G1ReservePercent=20 -XX:G1HeapWastePercent=5 -XX:G1MixedGCLiveThresholdPercent=75 -XX:G1RSetRegionEntries=4096* > > > I am attaching a part of the log that covers the run for around 10 min > (Bigger log for a day here > https://raw.githubusercontent.com/dvdreddy/gc-logs/master/g1-gc.log) . I > am also pasting in the mail a sample mixed GC run here for quick review, > you can see that ScanRS and Object copy are at the same level > > 2015-02-09T10:10:25.491-0500: 5383952.213: [GC pause (mixed) > Desired survivor size 285212672 bytes, new threshold 3 (max 15) > - age 1: 65081144 bytes, 65081144 total > - age 2: 136289896 bytes, 201371040 total > - age 3: 122469928 bytes, 323840968 total > 5383952.213: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 23323, predicted base time: 42.39 ms, remaining time: 57.61 > ms, target pause time:\ > 100.00 ms] > 5383952.213: [G1Ergonomics (CSet Construction) add young regions to CSet, > eden: 114 regions, survivors: 17 regions, predicted young region time: > 28.59 ms] > 5383952.214: [G1Ergonomics (CSet Construction) finish adding old regions > to CSet, reason: predicted time is too high, predicted time: 1.30 ms, > remaining time: 0.00 ms\ > , old: 107 regions, min: 107 regions] > 5383952.214: [G1Ergonomics (CSet Construction) added expensive regions to > CSet, reason: old CSet region num not reached min, old: 107 regions, > expensive: 60 regions, \ > min: 107 regions, remaining time: 0.00 ms] > 5383952.214: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 114 regions, survivors: 17 regions, old: 107 regions, predicted pause > time: 170.47 ms, targ\ > et pause time: 100.00 ms] > 5383952.388: [G1Ergonomics (Mixed GCs) continue mixed GCs, reason: > candidate old regions available, candidate old regions: 743 regions, > reclaimable: 12156413304 bytes\ > (13.82 %), threshold: 5.00 %] > , 0.1751220 secs] > [Parallel Time: 131.1 ms, GC Workers: 23] > [GC Worker Start (ms): Min: 5383952214.8, Avg: 5383952215.2, Max: > 5383952215.5, Diff: 0.7] > [Ext Root Scanning (ms): Min: 5.4, Avg: 6.1, Max: 9.6, Diff: 4.3, > Sum: 140.5] > [Update RS (ms): Min: 3.7, Avg: 6.7, Max: 7.7, Diff: 4.0, Sum: 154.8] > [Processed Buffers: Min: 9, Avg: 14.6, Max: 22, Diff: 13, Sum: > 335] > [Scan RS (ms): Min: 50.9, Avg: 51.2, Max: 51.4, Diff: 0.5, Sum: > 1178.2] > [Object Copy (ms): Min: 65.3, Avg: 65.6, Max: 66.1, Diff: 0.8, Sum: > 1507.9] > [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.5] > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: > 1.5] > [GC Worker Total (ms): Min: 129.4, Avg: 129.7, Max: 130.2, Diff: > 0.8, Sum: 2983.3] > [GC Worker End (ms): Min: 5383952344.9, Avg: 5383952344.9, Max: > 5383952345.1, Diff: 0.2] > [Code Root Fixup: 0.0 ms] > [Clear CT: 7.8 ms] > [Other: 36.2 ms] > [Choose CSet: 1.6 ms] > [Ref Proc: 6.1 ms] > [Ref Enq: 0.1 ms] > [Free CSet: 3.8 ms] > [Eden: 3648.0M(3648.0M)->0.0B(3808.0M) Survivors: 544.0M->384.0M Heap: > 56.6G(81.9G)->50.6G(81.9G)] > [Times: user=3.10 sys=0.00, real=0.18 secs] > > On Wed, Feb 11, 2015 at 5:31 PM, Yu Zhang wrote: > >> Hi, >> >> Currently the region size is capped at 32MB. >> >> It is good that you confirmed there is no coarsening. Is the high scanRS >> in mixed gc? If it is, then there are some tunings you can try to reduce >> it. >> >> Thanks, >> Jenny >> >> On 2/11/2015 4:12 PM, D vd Reddy wrote: >> >> >> Hi, >> we are using G1 GC for heap size of about 92 - 96 GB and we >> observed in the logs that Scan RS times were too high (comparable to Object >> Copy time, Is this normal ?, I have checked that there are no >> coarsening(s)). So in order to counter that we plan to decrease the region >> count, currently we set it explicitly to 4096 with regionSize to 32MB. My >> question is that if we drop setting regionSize and decrease the regions to >> 2048 what will happen to region size, Can it go above 32 MB and keep the >> regions to 2048 or is it capped at 32MB ? >> >> >> Thanks in advance >> >> >> >> _______________________________________________ >> hotspot-gc-use mailing listhotspot-gc-use at openjdk.java.nethttp://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yu.zhang at oracle.com Fri Feb 13 19:20:40 2015 From: yu.zhang at oracle.com (Yu Zhang) Date: Fri, 13 Feb 2015 11:20:40 -0800 Subject: G1GC, Java8u40ea, Metaspace questions In-Reply-To: <54DE41A7.6050004@finkzeit.at> References: <821215C9-36AC-41BB-A9A6-1E136341778F@finkzeit.at> <54DE3254.9030503@oracle.com> <54DE41A7.6050004@finkzeit.at> Message-ID: <54DE4E88.2000601@oracle.com> On 2/13/2015 10:25 AM, Wolfgang Pedot wrote: > Thanks for your reply, > > I was under the impression that the MetaspaceSize is only used as a > starting value for the high-water mark and adjusting it will only > change the behaviour at startup. Correct me if I am wrong but after > the first collect ergonomics would take over and adjust that mark > according to Min/MaxMetaspaceFreeRatio, right? My problem is that I > have to deal with lots of short lived generated classes and it would > be a step back to have the garbage-collector set a low mark right > before a class-generation spike and then do a Full-GC before raising > it again (Full-GC currently takes ~10sec on the real system). Although > with MaxMetaSpaceFreeRation being 70% by default that would probably > not happen that often. Your understanding is correct. MetaspaceSize is the starting value, as well as minimum metadata space size. When there is a need to expand metaspace, it only expands between MinMetaspaceExpansion and MaxMetaspaceExpansion; or if the requested space > MaxMetaspaceExpansion, it expands by the requested space + MinMetaspaceExpansion. The GC ergonomics decides the expansion, but every time it expanses, a full gc happens. Do you see metaspace shrink a lot? > > As for my first question, it is still not clear to me why Metaspace > "capacity" never seemed to reach "commited" during my first tests. I > have checked the logs for the current instance and with the higher > maximum-setting "capacity" and "commited" actually can get quite close > now but both are nowhere near the allowed maximum even after I have > driven the JVM into multiple FullGCs by creating classes like crazy. > Can I assume that RAM that is not "commited" is not actually used by > the JVM and can still be used by the OS for filesystem cache and the > such? Yes, if it is not committed, OS can use it. As why committed/capacity not reach max before full gc, I can think of several possible reasons: 1. The full gc is caused by expansion, by this expansion, we have not reached the max limit. 2. There are some Auxiliary data structure, but should not take too much memory I still think if you raise MetaspaceSize, you can avoid a lot of expansion. > > Now to my second question: > > I was not talking about Full-GC there, what I was referring to is such > a log-line: > > [G1Ergonomics (Concurrent Cycles) request concurrent cycle initiation, > reason: requested by GC cause, GC cause: Metadata GC Threshold] > > this tells me that G1 will trigger a concurrent cycle when the > Metaspace threshold is reached and sometimes that leads to classes > getting unloaded BEFORE there is a FullGC for the very same reason. I > would very much prefer this happening more often and FullGC only being > used as a last-resort. Unfortunately most of the time there is a > FullGC triggered right after I see the request for a concurrent cycle, > this means that I can only benefit from concurrent class unloading if > there is enough activity in old-gen to trigger concurrent cycles > because of heap-occupation. If G1 would use a lower Metaspace > threshold to initiate a concurrent cycle there would be a higher > chance for classes to get unloaded without a FullGC even if heap > occupation is currently low. That would be similar to what is going on > in the heap, concurrent cycles are started before the limit is reached > and a FullGC is required. Am I wrong here? I will forward your question to the developer. IIRC, after ClassUnloadingWithConcurrentMark is in place there is some changes in how metadata threshold is handled. But I am not quite sure. I will follow up on this later. > > Wolfgang > > > > Am 13.02.2015 18:20, schrieb Yu Zhang: >> Wolfgang, >> >> Please see my comments in line. >> Thanks, >> Jenny >> On 2/13/2015 5:36 AM, Wolfgang Pedot wrote: >>> Hello, >>> >>> my last mail regarding this topic seems to have been lost somewhere >>> but thats not really a problem because my questions have changed: >>> >>> I am currently evaluation 8u40 early-access (b21 and b23) for an >>> application server which, unfortunately, generates a fair amount of >>> short-lived dynamic classes. So far it runs on 7u67 and has enough >>> PermGen space to survive most days with 1 FullGC to get rid of the >>> classes. This situation is somewhat acceptable but I am really >>> looking forward to not requiring FullGCs for class unloading, hence >>> my tests with 8u40ea. >>> I am testing on a smaller system with much less headroom in PermGen >>> and as a first shot I just renamed MaxPermSize to MaxMetaspaceSize >>> and everything looked OK at first, classes where unloaded without >>> FullGCs and I was happy. When the Metaspace baseline-usage grew I >>> noticed that there where quite some FullGCs after all and when >>> looking closer I noticed that they came in pairs: >>> >>> 276129.049: [Full GC (Metadata GC Threshold) >>> 2015-02-05T17:03:04.509+0100: 276129.049: [GC >>> concurrent-root-region-scan-end, 0.0001449 secs] >>> 2015-02-05T17:03:04.509+0100: 276129.049: [GC concurrent-mark-start] >>> 492M->476M(904M), 1.5020455 secs] >>> [Eden: 0.0B(336.0M)->0.0B(352.0M) Survivors: 16.0M->0.0B Heap: >>> 492.2M(904.0M)->476.4M(904.0M)], [Metaspace: >>> 183332K->183332K(1247232K)] >>> [Times: user=2.74 sys=0.01, real=1.50 secs] >>> 2015-02-05T17:03:06.011+0100: 276130.551: [Full GC (Last ditch >>> collection) 476M->468M(904M), 1.4614149 secs] >>> [Eden: 0.0B(352.0M)->0.0B(360.0M) Survivors: 0.0B->0.0B Heap: >>> 476.4M(904.0M)->468.0M(904.0M)], [Metaspace: >>> 183332K->178688K(1247232K)] >>> [Times: user=2.62 sys=0.01, real=1.46 secs] >>> >>> MaxMetaspaceSize was set to 228M at the time so it was not >>> completely full yet the JVM threw in a "Last ditch collection" after >>> every (quite frequent) FullGC to desperately try and free some >>> Metaspace. >>> I even got to the point where the JVM threw OOME:Metaspace errors >>> even though according to the numbers Metaspace was nowhere near full >>> (different settings though). >> MaxMetaspaceSize is the maximum size Metadata space can expand to. >> GC will start with metadata space defined by MetaspaceSize, then >> expand/shrink it. When it needs to expand it, a 'Metadata GC >> Threshold' full gc is called. To avoid this kind of full gc, you can >> increase MetaspaceSize. >> >> In jdk8u40, ClassUnloadingWithConcurrentMark(default true) so that >> the classes can be unloaded with concurrent marking. This will >> reduce the chance of running into Metadata GC Threshold' full gc >> >>> After banging my head against various settings regarding Metaspace I >>> decided to run some worst-case tests on my laptop with different >>> MaxMetaspaceSize-values (no other metaspace related settings). My >>> test generates lots of classes while causing only minor increases in >>> old-gen so the JVM runs into FullGC (Metaspace threshold) >>> consistently. I discovered that the reported Metaspace-Size in the >>> gclog (and jconsole) never actually reaches the configured maximum >>> but instead tops out at about 85% of that value. "Commited" will be >>> the Maxsize but "capacity" never goes that high and since my >>> test-system did not have enough headroom to cover that it was >>> collecting because of Metaspace way to often. >>> >>> This brings me to my first question: What is the reason why the JVM >>> (appearantly) does not use the whole space it is allowed to use? Is >>> other stuff stored in there but not reported or is there some kind >>> of overhead? >>> >>> After giving the test-system more headroom it appears to work OK but >>> it always takes a couple of days use before Metaspace-baseline evens >>> out so I will see next week if that did the trick, I dont really >>> want to go "no limit". >>> >>> Now a more general question: >>> I noticed that G1 also triggers a concurrent marking cycle because >>> of "Metaspace threshold", is it using the same threshold that would >>> also trigger a FullGC or is it slightly lower? Is that treshold >>> visible somewhere? >>> The reason I am asking is that I want to keep FullGCs down to a >>> minimum so if the class count rises reasonably slow even without >>> causing much oldgen-increase should the concurrent cycle be >>> triggered before the JVM throws in a FullGC most of the time? >> You are talking about Full GC caused by 2 reasons: >> >> * >> Metaspace threshold: we can avoid this my increasing >> MetadataspaceSize >> >> * Allocation Failure: usually this happens after a few 'to-space >> exhausted'. I think this is not what you see in your log. A lot >> of time this is caused by poor tuning. The marking cycles are >> triggered after initial marking. And when initial marking is >> triggered, is decided by InitiatingHeapOccupancyPercent >> >>> Quite a long Mail for two questions, any hints would be greatly >>> appreciated! >>> >>> Wolfgang >>> _______________________________________________ >>> hotspot-gc-use mailing list >>> hotspot-gc-use at openjdk.java.net >>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> > > From jon.masamitsu at oracle.com Fri Feb 13 19:46:30 2015 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Fri, 13 Feb 2015 11:46:30 -0800 Subject: G1GC, Java8u40ea, Metaspace questions In-Reply-To: <54DE41A7.6050004@finkzeit.at> References: <821215C9-36AC-41BB-A9A6-1E136341778F@finkzeit.at> <54DE3254.9030503@oracle.com> <54DE41A7.6050004@finkzeit.at> Message-ID: <54DE5495.2010501@oracle.com> On 2/13/2015 10:25 AM, Wolfgang Pedot wrote: > Thanks for your reply, > > I was under the impression that the MetaspaceSize is only used as a > starting value for the high-water mark and That's correct. > adjusting it will only change the behaviour at startup. Correct me if > I am wrong but after the first collect ergonomics If "at startup" includes the first time the metaspace reaches MetaspaceSize, yes that is also correct. > would take over and adjust that mark according to > Min/MaxMetaspaceFreeRatio, right? My problem is that I have to Right. > deal with lots of short lived generated classes and it would be a step > back to have the garbage-collector set a low mark right before a > class-generation spike and then do a Full-GC before raising it again > (Full-GC currently takes ~10sec on the real system). Although with > MaxMetaSpaceFreeRation being 70% by default that would probably not > happen that often. > > As for my first question, it is still not clear to me why Metaspace > "capacity" never seemed to reach "commited" during my first tests. I > have checked the logs for the current instance and with the higher > maximum-setting "capacity" and "commited" actually can get quite close > now but both are nowhere near the allowed maximum even after I have > driven the JVM into multiple FullGCs by creating classes like crazy. > Can I assume that RAM that is not You're running on a 32 bit system, right? Can you run this and let us know what comes out. java -XX:+UseG1GC -XX:+PrintGCDetails -version I expect something like java version "1.9.0-ea" Java(TM) SE Runtime Environment (build 1.9.0-ea-b10) Java HotSpot(TM) Server VM (build 25.0-b62, mixed mode) Heap garbage-first heap total 65536K, used 0K [0xa1400000, 0xa5400000, 0xe1400000) region size 1024K, 1 young (1024K), 0 survivors (0K) Metaspace used 1418K, capacity 2200K, committed 2200K, reserved 4400K > "commited" is not actually used by the JVM and can still be used by > the OS for filesystem cache and the such? If not committed, it is not being used by the JVM but the OS might not use the space for anything else since the memory has been reserved. Depends on the OS. > > Now to my second question: > > I was not talking about Full-GC there, what I was referring to is such > a log-line: > > [G1Ergonomics (Concurrent Cycles) request concurrent cycle initiation, > reason: requested by GC cause, GC cause: Metadata GC Threshold] > > this tells me that G1 will trigger a concurrent cycle when the > Metaspace threshold is reached and sometimes that leads to classes > getting unloaded BEFORE there is a FullGC for the very same reason. I > would very much prefer this happening more often and FullGC only being > used as a last-resort. Unfortunately most of the time there is a > FullGC triggered right after I see the request for a concurrent cycle, > this means that I can only benefit from concurrent class unloading if > there is enough activity in old-gen to trigger concurrent cycles > because of heap-occupation. If G1 would use a lower Metaspace > threshold to initiate a concurrent cycle there would be a higher > chance for classes to get unloaded without a FullGC even if heap > occupation is currently low. That would be similar to what is going on > in the heap, concurrent cycles are started before the limit is reached > and a FullGC is required. Am I wrong here? I think you are right again. One explanation for what you are seeing is that the high-water-mark has grown too high. The concurrent collection for metadata has been started but there isn't enough metadata available for the needed allocations (runs out and then the full GC happens). I don't understand yet why you have so much metaspace reserved but not committed yet so don't know if this is actually what you are seeing. Jon > > Wolfgang > > > > Am 13.02.2015 18:20, schrieb Yu Zhang: >> Wolfgang, >> >> Please see my comments in line. >> Thanks, >> Jenny >> On 2/13/2015 5:36 AM, Wolfgang Pedot wrote: >>> Hello, >>> >>> my last mail regarding this topic seems to have been lost somewhere >>> but thats not really a problem because my questions have changed: >>> >>> I am currently evaluation 8u40 early-access (b21 and b23) for an >>> application server which, unfortunately, generates a fair amount of >>> short-lived dynamic classes. So far it runs on 7u67 and has enough >>> PermGen space to survive most days with 1 FullGC to get rid of the >>> classes. This situation is somewhat acceptable but I am really >>> looking forward to not requiring FullGCs for class unloading, hence >>> my tests with 8u40ea. >>> I am testing on a smaller system with much less headroom in PermGen >>> and as a first shot I just renamed MaxPermSize to MaxMetaspaceSize >>> and everything looked OK at first, classes where unloaded without >>> FullGCs and I was happy. When the Metaspace baseline-usage grew I >>> noticed that there where quite some FullGCs after all and when >>> looking closer I noticed that they came in pairs: >>> >>> 276129.049: [Full GC (Metadata GC Threshold) >>> 2015-02-05T17:03:04.509+0100: 276129.049: [GC >>> concurrent-root-region-scan-end, 0.0001449 secs] >>> 2015-02-05T17:03:04.509+0100: 276129.049: [GC concurrent-mark-start] >>> 492M->476M(904M), 1.5020455 secs] >>> [Eden: 0.0B(336.0M)->0.0B(352.0M) Survivors: 16.0M->0.0B Heap: >>> 492.2M(904.0M)->476.4M(904.0M)], [Metaspace: >>> 183332K->183332K(1247232K)] >>> [Times: user=2.74 sys=0.01, real=1.50 secs] >>> 2015-02-05T17:03:06.011+0100: 276130.551: [Full GC (Last ditch >>> collection) 476M->468M(904M), 1.4614149 secs] >>> [Eden: 0.0B(352.0M)->0.0B(360.0M) Survivors: 0.0B->0.0B Heap: >>> 476.4M(904.0M)->468.0M(904.0M)], [Metaspace: >>> 183332K->178688K(1247232K)] >>> [Times: user=2.62 sys=0.01, real=1.46 secs] >>> >>> MaxMetaspaceSize was set to 228M at the time so it was not >>> completely full yet the JVM threw in a "Last ditch collection" after >>> every (quite frequent) FullGC to desperately try and free some >>> Metaspace. >>> I even got to the point where the JVM threw OOME:Metaspace errors >>> even though according to the numbers Metaspace was nowhere near full >>> (different settings though). >> MaxMetaspaceSize is the maximum size Metadata space can expand to. >> GC will start with metadata space defined by MetaspaceSize, then >> expand/shrink it. When it needs to expand it, a 'Metadata GC >> Threshold' full gc is called. To avoid this kind of full gc, you can >> increase MetaspaceSize. >> >> In jdk8u40, ClassUnloadingWithConcurrentMark(default true) so that >> the classes can be unloaded with concurrent marking. This will >> reduce the chance of running into Metadata GC Threshold' full gc >> >>> After banging my head against various settings regarding Metaspace I >>> decided to run some worst-case tests on my laptop with different >>> MaxMetaspaceSize-values (no other metaspace related settings). My >>> test generates lots of classes while causing only minor increases in >>> old-gen so the JVM runs into FullGC (Metaspace threshold) >>> consistently. I discovered that the reported Metaspace-Size in the >>> gclog (and jconsole) never actually reaches the configured maximum >>> but instead tops out at about 85% of that value. "Commited" will be >>> the Maxsize but "capacity" never goes that high and since my >>> test-system did not have enough headroom to cover that it was >>> collecting because of Metaspace way to often. >>> >>> This brings me to my first question: What is the reason why the JVM >>> (appearantly) does not use the whole space it is allowed to use? Is >>> other stuff stored in there but not reported or is there some kind >>> of overhead? >>> >>> After giving the test-system more headroom it appears to work OK but >>> it always takes a couple of days use before Metaspace-baseline evens >>> out so I will see next week if that did the trick, I dont really >>> want to go "no limit". >>> >>> Now a more general question: >>> I noticed that G1 also triggers a concurrent marking cycle because >>> of "Metaspace threshold", is it using the same threshold that would >>> also trigger a FullGC or is it slightly lower? Is that treshold >>> visible somewhere? >>> The reason I am asking is that I want to keep FullGCs down to a >>> minimum so if the class count rises reasonably slow even without >>> causing much oldgen-increase should the concurrent cycle be >>> triggered before the JVM throws in a FullGC most of the time? >> You are talking about Full GC caused by 2 reasons: >> >> * >> Metaspace threshold: we can avoid this my increasing >> MetadataspaceSize >> >> * Allocation Failure: usually this happens after a few 'to-space >> exhausted'. I think this is not what you see in your log. A lot >> of time this is caused by poor tuning. The marking cycles are >> triggered after initial marking. And when initial marking is >> triggered, is decided by InitiatingHeapOccupancyPercent >> >>> Quite a long Mail for two questions, any hints would be greatly >>> appreciated! >>> >>> Wolfgang >>> _______________________________________________ >>> hotspot-gc-use mailing list >>> hotspot-gc-use at openjdk.java.net >>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> > > From wolfgang.pedot at finkzeit.at Fri Feb 13 20:19:49 2015 From: wolfgang.pedot at finkzeit.at (Wolfgang Pedot) Date: Fri, 13 Feb 2015 21:19:49 +0100 Subject: G1GC, Java8u40ea, Metaspace questions In-Reply-To: <54DE546B.8030607@oracle.com> References: <821215C9-36AC-41BB-A9A6-1E136341778F@finkzeit.at> <54DE3254.9030503@oracle.com> <54DE41A7.6050004@finkzeit.at> <54DE546B.8030607@oracle.com> Message-ID: <54DE5C65.7020006@finkzeit.at> Not at all, unfortunately I have not kept many logs so far because I was playing with lots of different configurations. I also have some new Information: I ran a test with 500M MetaspaceSize and a max of 600M (Test-setup needs ~100MB after Full-GC) and the first 3 collects in Metaspace where beautifully handled by G1 but every collect afterwards was a Full-GC. The log of that test was lost because I terminated it to start my current test, which is basically the same but without a maximum. My test represents a worst-case scenario in that it generates lots of short-lived classes and objects, heap occupation stays well below the configured threshold. Here is what happened so far: The first concurrent cycle was requested right as commited Metaspace hit 500M exactly Metaspace used 430049K, capacity 511992K, committed 512000K, reserved 1435648K Afterwards: Metaspace used 174458K, capacity 196210K, committed 352588K, reserved 1275904K As you can see "used" is higher than I expected but I guess its possible that a FullGC can clean up more than a concurrent cycle while my test is still running. Now on to the second cycle: Metaspace used 491388K, capacity 587512K, committed 587648K, reserved 1490944K Metaspace used 172063K, capacity 193180K, committed 387200K, reserved 1290240K third: Metaspace used 538141K, capacity 645280K, committed 645336K, reserved 1531904K Metaspace used 170096K, capacity 190548K, committed 410148K, reserved 1296384K fourth: Metaspace used 569051K, capacity 683142K, committed 683580K, reserved 1558528K Metaspace used 168038K, capacity 187908K, committed 421768K, reserved 1296384K Skipping ahead to number seven: Metaspace used 639341K, capacity 770728K, committed 770820K, reserved 1622016K Metaspace used 165744K, capacity 185838K, committed 476240K, reserved 1327104K Right after that collect my loop was done, generating about 500000 classes without a single FullGC. What is alarming though is that the high-water mark appears to be rising because even though usage after the collects is more or less stable the collects happen later and later. If I try to limit this by setting a maximum I begin to see FullGCs.... I restarted my test-loop (same JVM still) and this is cycle eight: Metaspace used 657814K, capacity 793582K, committed 793736K, reserved 1638400K Metaspace used 163159K, capacity 182612K, committed 478472K, reserved 1323008K I will send you the full log, its slightly to big for the list. Wolfgang From wolfgang.pedot at finkzeit.at Mon Feb 16 14:08:02 2015 From: wolfgang.pedot at finkzeit.at (Wolfgang Pedot) Date: Mon, 16 Feb 2015 15:08:02 +0100 Subject: G1GC, Java8u40ea, Metaspace questions In-Reply-To: <54DE5C65.7020006@finkzeit.at> References: <821215C9-36AC-41BB-A9A6-1E136341778F@finkzeit.at> <54DE3254.9030503@oracle.com> <54DE41A7.6050004@finkzeit.at> <54DE546B.8030607@oracle.com> <54DE5C65.7020006@finkzeit.at> Message-ID: <54E1F9C2.3080904@finkzeit.at> Appearantly Metaspace is ALWAYS grown by MinMetaspaceExpansion even if it would not be required. I ran a test with MinMetaspaceFreeRatio=0 and MetaspaceSize=500M which should prevent growth of Metaspace alltogether but I can see a 332k increase after every collection in Metaspace. 332k just happens to be the default-value of MinMetaspaceExpansion... regards Wolfgang From yu.zhang at oracle.com Tue Feb 17 19:22:35 2015 From: yu.zhang at oracle.com (Yu Zhang) Date: Tue, 17 Feb 2015 11:22:35 -0800 Subject: G1GC, Java8u40ea, Metaspace questions In-Reply-To: <54DE5495.2010501@oracle.com> References: <821215C9-36AC-41BB-A9A6-1E136341778F@finkzeit.at> <54DE3254.9030503@oracle.com> <54DE41A7.6050004@finkzeit.at> <54DE5495.2010501@oracle.com> Message-ID: <54E394FB.3040204@oracle.com> Jon, Here is the version: Java HotSpot(TM) 64-Bit Server VM (25.40-b25) for windows-amd64 JRE (1.8.0_40-ea-b23), built on Jan 27 2015 19:35:49 by "java_re" with MS VC++ 10.0 (VS2010) Wolfang, Thanks for the logs. But the one you sent me seems pretty well behaved. There is no full gc. The interval between initial-marking for metesapce is not getting longer. Do you have a log that you saw the high-warter mark is getting higher, so the concurrent marking is less frequent, then there is a full gc? Thanks, Jenny On 2/13/2015 11:46 AM, Jon Masamitsu wrote: > > On 2/13/2015 10:25 AM, Wolfgang Pedot wrote: >> Thanks for your reply, >> >> I was under the impression that the MetaspaceSize is only used as a >> starting value for the high-water mark and > That's correct. > >> adjusting it will only change the behaviour at startup. Correct me if >> I am wrong but after the first collect ergonomics > > If "at startup" includes the first time the metaspace reaches > MetaspaceSize, yes that is also correct. > >> would take over and adjust that mark according to >> Min/MaxMetaspaceFreeRatio, right? My problem is that I have to > Right. > >> deal with lots of short lived generated classes and it would be a >> step back to have the garbage-collector set a low mark right before a >> class-generation spike and then do a Full-GC before raising it again >> (Full-GC currently takes ~10sec on the real system). Although with >> MaxMetaSpaceFreeRation being 70% by default that would probably not >> happen that often. >> >> As for my first question, it is still not clear to me why Metaspace >> "capacity" never seemed to reach "commited" during my first tests. I >> have checked the logs for the current instance and with the higher >> maximum-setting "capacity" and "commited" actually can get quite >> close now but both are nowhere near the allowed maximum even after I >> have driven the JVM into multiple FullGCs by creating classes like >> crazy. Can I assume that RAM that is not > > You're running on a 32 bit system, right? Can you run this and let us > know what comes out. > > java -XX:+UseG1GC -XX:+PrintGCDetails -version > > I expect something like > > java version "1.9.0-ea" > Java(TM) SE Runtime Environment (build 1.9.0-ea-b10) > Java HotSpot(TM) Server VM (build 25.0-b62, mixed mode) > Heap > garbage-first heap total 65536K, used 0K [0xa1400000, 0xa5400000, > 0xe1400000) > region size 1024K, 1 young (1024K), 0 survivors (0K) > Metaspace used 1418K, capacity 2200K, committed 2200K, reserved > 4400K > > >> "commited" is not actually used by the JVM and can still be used by >> the OS for filesystem cache and the such? > > If not committed, it is not being used by the JVM but the OS might not > use > the space for anything else since the memory has been reserved. > Depends on > the OS. > >> >> Now to my second question: >> >> I was not talking about Full-GC there, what I was referring to is >> such a log-line: >> >> [G1Ergonomics (Concurrent Cycles) request concurrent cycle >> initiation, reason: requested by GC cause, GC cause: Metadata GC >> Threshold] >> >> this tells me that G1 will trigger a concurrent cycle when the >> Metaspace threshold is reached and sometimes that leads to classes >> getting unloaded BEFORE there is a FullGC for the very same reason. I >> would very much prefer this happening more often and FullGC only >> being used as a last-resort. Unfortunately most of the time there is >> a FullGC triggered right after I see the request for a concurrent >> cycle, this means that I can only benefit from concurrent class >> unloading if there is enough activity in old-gen to trigger >> concurrent cycles because of heap-occupation. If G1 would use a lower >> Metaspace threshold to initiate a concurrent cycle there would be a >> higher chance for classes to get unloaded without a FullGC even if >> heap occupation is currently low. That would be similar to what is >> going on in the heap, concurrent cycles are started before the limit >> is reached and a FullGC is required. Am I wrong here? > > I think you are right again. One explanation for what you are > seeing is that > the high-water-mark has grown too high. The concurrent collection for > metadata has been started but there isn't enough metadata available for > the needed allocations (runs out and then the full GC happens). I don't > understand yet why you have so much metaspace reserved but not > committed yet so don't know if this is actually what you are seeing. > > Jon > >> >> Wolfgang >> >> >> >> Am 13.02.2015 18:20, schrieb Yu Zhang: >>> Wolfgang, >>> >>> Please see my comments in line. >>> Thanks, >>> Jenny >>> On 2/13/2015 5:36 AM, Wolfgang Pedot wrote: >>>> Hello, >>>> >>>> my last mail regarding this topic seems to have been lost somewhere >>>> but thats not really a problem because my questions have changed: >>>> >>>> I am currently evaluation 8u40 early-access (b21 and b23) for an >>>> application server which, unfortunately, generates a fair amount of >>>> short-lived dynamic classes. So far it runs on 7u67 and has enough >>>> PermGen space to survive most days with 1 FullGC to get rid of the >>>> classes. This situation is somewhat acceptable but I am really >>>> looking forward to not requiring FullGCs for class unloading, hence >>>> my tests with 8u40ea. >>>> I am testing on a smaller system with much less headroom in PermGen >>>> and as a first shot I just renamed MaxPermSize to MaxMetaspaceSize >>>> and everything looked OK at first, classes where unloaded without >>>> FullGCs and I was happy. When the Metaspace baseline-usage grew I >>>> noticed that there where quite some FullGCs after all and when >>>> looking closer I noticed that they came in pairs: >>>> >>>> 276129.049: [Full GC (Metadata GC Threshold) >>>> 2015-02-05T17:03:04.509+0100: 276129.049: [GC >>>> concurrent-root-region-scan-end, 0.0001449 secs] >>>> 2015-02-05T17:03:04.509+0100: 276129.049: [GC concurrent-mark-start] >>>> 492M->476M(904M), 1.5020455 secs] >>>> [Eden: 0.0B(336.0M)->0.0B(352.0M) Survivors: 16.0M->0.0B Heap: >>>> 492.2M(904.0M)->476.4M(904.0M)], [Metaspace: >>>> 183332K->183332K(1247232K)] >>>> [Times: user=2.74 sys=0.01, real=1.50 secs] >>>> 2015-02-05T17:03:06.011+0100: 276130.551: [Full GC (Last ditch >>>> collection) 476M->468M(904M), 1.4614149 secs] >>>> [Eden: 0.0B(352.0M)->0.0B(360.0M) Survivors: 0.0B->0.0B Heap: >>>> 476.4M(904.0M)->468.0M(904.0M)], [Metaspace: >>>> 183332K->178688K(1247232K)] >>>> [Times: user=2.62 sys=0.01, real=1.46 secs] >>>> >>>> MaxMetaspaceSize was set to 228M at the time so it was not >>>> completely full yet the JVM threw in a "Last ditch collection" >>>> after every (quite frequent) FullGC to desperately try and free >>>> some Metaspace. >>>> I even got to the point where the JVM threw OOME:Metaspace errors >>>> even though according to the numbers Metaspace was nowhere near >>>> full (different settings though). >>> MaxMetaspaceSize is the maximum size Metadata space can expand to. >>> GC will start with metadata space defined by MetaspaceSize, then >>> expand/shrink it. When it needs to expand it, a 'Metadata GC >>> Threshold' full gc is called. To avoid this kind of full gc, you >>> can increase MetaspaceSize. >>> >>> In jdk8u40, ClassUnloadingWithConcurrentMark(default true) so that >>> the classes can be unloaded with concurrent marking. This will >>> reduce the chance of running into Metadata GC Threshold' full gc >>> >>>> After banging my head against various settings regarding Metaspace >>>> I decided to run some worst-case tests on my laptop with different >>>> MaxMetaspaceSize-values (no other metaspace related settings). My >>>> test generates lots of classes while causing only minor increases >>>> in old-gen so the JVM runs into FullGC (Metaspace threshold) >>>> consistently. I discovered that the reported Metaspace-Size in the >>>> gclog (and jconsole) never actually reaches the configured maximum >>>> but instead tops out at about 85% of that value. "Commited" will be >>>> the Maxsize but "capacity" never goes that high and since my >>>> test-system did not have enough headroom to cover that it was >>>> collecting because of Metaspace way to often. >>>> >>>> This brings me to my first question: What is the reason why the JVM >>>> (appearantly) does not use the whole space it is allowed to use? >>>> Is other stuff stored in there but not reported or is there some >>>> kind of overhead? >>>> >>>> After giving the test-system more headroom it appears to work OK >>>> but it always takes a couple of days use before Metaspace-baseline >>>> evens out so I will see next week if that did the trick, I dont >>>> really want to go "no limit". >>>> >>>> Now a more general question: >>>> I noticed that G1 also triggers a concurrent marking cycle because >>>> of "Metaspace threshold", is it using the same threshold that would >>>> also trigger a FullGC or is it slightly lower? Is that treshold >>>> visible somewhere? >>>> The reason I am asking is that I want to keep FullGCs down to a >>>> minimum so if the class count rises reasonably slow even without >>>> causing much oldgen-increase should the concurrent cycle be >>>> triggered before the JVM throws in a FullGC most of the time? >>> You are talking about Full GC caused by 2 reasons: >>> >>> * >>> Metaspace threshold: we can avoid this my increasing >>> MetadataspaceSize >>> >>> * Allocation Failure: usually this happens after a few 'to-space >>> exhausted'. I think this is not what you see in your log. A lot >>> of time this is caused by poor tuning. The marking cycles are >>> triggered after initial marking. And when initial marking is >>> triggered, is decided by InitiatingHeapOccupancyPercent >>> >>>> Quite a long Mail for two questions, any hints would be greatly >>>> appreciated! >>>> >>>> Wolfgang >>>> _______________________________________________ >>>> hotspot-gc-use mailing list >>>> hotspot-gc-use at openjdk.java.net >>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>> >> >> > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: From wolfgang.pedot at finkzeit.at Tue Feb 17 19:51:40 2015 From: wolfgang.pedot at finkzeit.at (Wolfgang Pedot) Date: Tue, 17 Feb 2015 20:51:40 +0100 Subject: G1GC, Java8u40ea, Metaspace questions In-Reply-To: <54E394FB.3040204@oracle.com> References: <821215C9-36AC-41BB-A9A6-1E136341778F@finkzeit.at> <54DE3254.9030503@oracle.com> <54DE41A7.6050004@finkzeit.at> <54DE5495.2010501@oracle.com> <54E394FB.3040204@oracle.com> Message-ID: <54E39BCC.7090802@finkzeit.at> Hi, the last log was with a MaxMetaspaceFreeRatio of 10% and that was quite well behaved. The log I have sent before (starting 2015-02-13T20:04:43.512+0100) shows a slow but continuous increase of the "commited" value for Metaspace even though "used" after a collect is quite stable and "commited" is shrunk after every collect. Once a configured MaxMetaspaceSize is reached the FullGCs start. I do have a log showing that but that is with smaller sizes (Metaspace=300MB, MaxMetaspace=305MB, MinMetaspaceFreeRatio=0) so I dont know if it will be a good case. I have found out that I need to set Min and MaxMetaspaceFreeRatio really really low (like <=10%) to keep the high-water-mark in check and I dont quite see why in the numbers. I can send you the 300/305MB log if it helps, otherwise do you want me to run another test with specific configuration-values? Wolfgang Am 17.02.2015 20:22, schrieb Yu Zhang: > Jon, > > Here is the version: > Java HotSpot(TM) 64-Bit Server VM (25.40-b25) for windows-amd64 JRE > (1.8.0_40-ea-b23), built on Jan 27 2015 19:35:49 by "java_re" with MS > VC++ 10.0 (VS2010) > > Wolfang, > Thanks for the logs. But the one you sent me seems pretty well > behaved. There is no full gc. The interval between initial-marking > for metesapce is not getting longer. Do you have a log that you saw > the high-warter mark is getting higher, so the concurrent marking is > less frequent, then there is a full gc? > Thanks, > Jenny > On 2/13/2015 11:46 AM, Jon Masamitsu wrote: >> >> On 2/13/2015 10:25 AM, Wolfgang Pedot wrote: >>> Thanks for your reply, >>> >>> I was under the impression that the MetaspaceSize is only used as a >>> starting value for the high-water mark and >> That's correct. >> >>> adjusting it will only change the behaviour at startup. Correct me >>> if I am wrong but after the first collect ergonomics >> >> If "at startup" includes the first time the metaspace reaches >> MetaspaceSize, yes that is also correct. >> >>> would take over and adjust that mark according to >>> Min/MaxMetaspaceFreeRatio, right? My problem is that I have to >> Right. >> >>> deal with lots of short lived generated classes and it would be a >>> step back to have the garbage-collector set a low mark right before >>> a class-generation spike and then do a Full-GC before raising it >>> again (Full-GC currently takes ~10sec on the real system). Although >>> with MaxMetaSpaceFreeRation being 70% by default that would probably >>> not happen that often. >>> >>> As for my first question, it is still not clear to me why Metaspace >>> "capacity" never seemed to reach "commited" during my first tests. I >>> have checked the logs for the current instance and with the higher >>> maximum-setting "capacity" and "commited" actually can get quite >>> close now but both are nowhere near the allowed maximum even after I >>> have driven the JVM into multiple FullGCs by creating classes like >>> crazy. Can I assume that RAM that is not >> >> From joyxiong at yahoo.com Tue Feb 17 22:40:20 2015 From: joyxiong at yahoo.com (Joy Xiong) Date: Tue, 17 Feb 2015 22:40:20 +0000 (UTC) Subject: G1 flag Message-ID: <80364472.5537869.1424212820954.JavaMail.yahoo@mail.yahoo.com> Hi,? I am trying JDK1.8 with G1. However, the java process does not start if I specify -XX:+UnlockDiagnosticVMOptions -XX:G1MixedGCLiveThresholdPercent=xx. Does the version support the flag -XX:G1MixedGCLiveThresholdPercent=xx? thanks,-Joy -------------- next part -------------- An HTML attachment was scrubbed... URL: From yu.zhang at oracle.com Tue Feb 17 22:46:39 2015 From: yu.zhang at oracle.com (Yu Zhang) Date: Tue, 17 Feb 2015 14:46:39 -0800 Subject: G1 flag In-Reply-To: <80364472.5537869.1424212820954.JavaMail.yahoo@mail.yahoo.com> References: <80364472.5537869.1424212820954.JavaMail.yahoo@mail.yahoo.com> Message-ID: <54E3C4CF.7000102@oracle.com> Joy, It is an experimental flag. You can unlock it by -XX:+UnlockExperimentalVMOptions Thanks, Jenny On 2/17/2015 2:40 PM, Joy Xiong wrote: > > Hi, > > I am trying JDK1.8 with G1. However, the java process does not start > if I specify -XX:+UnlockDiagnosticVMOptions > -XX:G1MixedGCLiveThresholdPercent=xx. > > Does the version support the flag -XX:G1MixedGCLiveThresholdPercent=xx? > > thanks, > -Joy > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: From joyxiong at yahoo.com Wed Feb 18 02:02:18 2015 From: joyxiong at yahoo.com (Joy Xiong) Date: Wed, 18 Feb 2015 02:02:18 +0000 (UTC) Subject: G1 flag In-Reply-To: <54E3C4CF.7000102@oracle.com> References: <54E3C4CF.7000102@oracle.com> Message-ID: <2096389830.140050.1424224938798.JavaMail.yahoo@mail.yahoo.com> It works!?thanks. -Joy On Tuesday, February 17, 2015 2:46 PM, Yu Zhang wrote: Joy, It is an experimental flag. You can unlock it by -XX:+UnlockExperimentalVMOptions Thanks, Jenny On 2/17/2015 2:40 PM, Joy Xiong wrote: Hi,? I am trying JDK1.8 with G1. However, the java process does not start if I specify -XX:+UnlockDiagnosticVMOptions -XX:G1MixedGCLiveThresholdPercent=xx. Does the version support the flag -XX:G1MixedGCLiveThresholdPercent=xx? thanks, -Joy _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: From yu.zhang at oracle.com Wed Feb 18 04:12:23 2015 From: yu.zhang at oracle.com (Yu Zhang) Date: Tue, 17 Feb 2015 20:12:23 -0800 Subject: G1GC, Java8u40ea, Metaspace questions In-Reply-To: <54E39BCC.7090802@finkzeit.at> References: <821215C9-36AC-41BB-A9A6-1E136341778F@finkzeit.at> <54DE3254.9030503@oracle.com> <54DE41A7.6050004@finkzeit.at> <54DE5495.2010501@oracle.com> <54E394FB.3040204@oracle.com> <54E39BCC.7090802@finkzeit.at> Message-ID: <54E41127.3040002@oracle.com> Hi, After discussing with the developers, here is the explanation: >shows a slow but continuous increase of the "commited" value for Metaspace even though "used" after a collect is quite stable and "commited" is shrunk after every collect. This might be due to fragmentation in metadata space. Even though the total used is stable, the memory is fragmented so it has to allocate more. If you have Flight Recorder(a commercial feature), you can see this information in JFR. >I have found out that I need to set Min and MaxMetaspaceFreeRatio really really low (like <=10%) to keep the high-water-mark in check and I dont quite see why in the numbers. This might be related to how the high water mark is calculated. There are some ways to improve this but not implemented yet. Thank you very much for your feedback. Thanks, Jenny On 2/17/2015 11:51 AM, Wolfgang Pedot wrote: > Hi, > > the last log was with a MaxMetaspaceFreeRatio of 10% and that was > quite well behaved. The log I have sent before (starting > 2015-02-13T20:04:43.512+0100) shows a slow but continuous increase of > the "commited" value for Metaspace even though "used" after a collect > is quite stable and "commited" is shrunk after every collect. Once a > configured MaxMetaspaceSize is reached the FullGCs start. I do have a > log showing that but that is with smaller sizes (Metaspace=300MB, > MaxMetaspace=305MB, MinMetaspaceFreeRatio=0) so I dont know if it will > be a good case. I have found out that I need to set Min and > MaxMetaspaceFreeRatio really really low (like <=10%) to keep the > high-water-mark in check and I dont quite see why in the numbers. > > I can send you the 300/305MB log if it helps, otherwise do you want me > to run another test with specific configuration-values? > > Wolfgang > > > Am 17.02.2015 20:22, schrieb Yu Zhang: >> Jon, >> >> Here is the version: >> Java HotSpot(TM) 64-Bit Server VM (25.40-b25) for windows-amd64 JRE >> (1.8.0_40-ea-b23), built on Jan 27 2015 19:35:49 by "java_re" with MS >> VC++ 10.0 (VS2010) >> >> Wolfang, >> Thanks for the logs. But the one you sent me seems pretty well >> behaved. There is no full gc. The interval between initial-marking >> for metesapce is not getting longer. Do you have a log that you saw >> the high-warter mark is getting higher, so the concurrent marking is >> less frequent, then there is a full gc? >> Thanks, >> Jenny >> On 2/13/2015 11:46 AM, Jon Masamitsu wrote: >>> >>> On 2/13/2015 10:25 AM, Wolfgang Pedot wrote: >>>> Thanks for your reply, >>>> >>>> I was under the impression that the MetaspaceSize is only used as a >>>> starting value for the high-water mark and >>> That's correct. >>> >>>> adjusting it will only change the behaviour at startup. Correct me >>>> if I am wrong but after the first collect ergonomics >>> >>> If "at startup" includes the first time the metaspace reaches >>> MetaspaceSize, yes that is also correct. >>> >>>> would take over and adjust that mark according to >>>> Min/MaxMetaspaceFreeRatio, right? My problem is that I have to >>> Right. >>> >>>> deal with lots of short lived generated classes and it would be a >>>> step back to have the garbage-collector set a low mark right before >>>> a class-generation spike and then do a Full-GC before raising it >>>> again (Full-GC currently takes ~10sec on the real system). Although >>>> with MaxMetaSpaceFreeRation being 70% by default that would >>>> probably not happen that often. >>>> >>>> As for my first question, it is still not clear to me why Metaspace >>>> "capacity" never seemed to reach "commited" during my first tests. >>>> I have checked the logs for the current instance and with the >>>> higher maximum-setting "capacity" and "commited" actually can get >>>> quite close now but both are nowhere near the allowed maximum even >>>> after I have driven the JVM into multiple FullGCs by creating >>>> classes like crazy. Can I assume that RAM that is not >>> >>> > From wolfgang.pedot at finkzeit.at Thu Feb 19 11:46:51 2015 From: wolfgang.pedot at finkzeit.at (Wolfgang Pedot) Date: Thu, 19 Feb 2015 12:46:51 +0100 Subject: G1GC, Java8u40ea, Metaspace questions In-Reply-To: <54E41127.3040002@oracle.com> References: <821215C9-36AC-41BB-A9A6-1E136341778F@finkzeit.at> <54DE3254.9030503@oracle.com> <54DE41A7.6050004@finkzeit.at> <54DE5495.2010501@oracle.com> <54E394FB.3040204@oracle.com> <54E39BCC.7090802@finkzeit.at> <54E41127.3040002@oracle.com> Message-ID: <54E5CD2B.7030201@finkzeit.at> One more, something just came to me: Class unloading happens during the concurrent marking-cycle so the mixed collects that would free up unused classloaders in oldGen happen after that, right? That would mean the classes can only be cleaned up at the next cycle and stay in Metaspace until then. My test causes only Metaspace-triggered concurrent cycles so the garbage-collector is always behind by one cycle and therefor the amount of classes that can be unloaded can be different each time, irregardless of the percentage of wasted heap. I guess I have to extend my test-scenario in a way that also causes at least some heap-driven concurrent cycles and see what happens then. Still does not explain why I hardly ever see HWM go down but it explains some of my more confusing test-results... regards Wolfgang From joyxiong at yahoo.com Thu Feb 19 22:14:55 2015 From: joyxiong at yahoo.com (Joy Xiong) Date: Thu, 19 Feb 2015 22:14:55 +0000 (UTC) Subject: G1 guidelines/suggestions Message-ID: <1730454645.3548827.1424384095334.JavaMail.yahoo@mail.yahoo.com> Hi, We have hundreds of services all running with CMS, and I am very interested in moving the services from CMS to G1. For that efforts to be convincing, I'd like to start services which has the most potential to beat CMS in terms of performance. I understand JVM performance is very application-specific, but still I want to check if anybody has guidelines/suggestions on which type of applications can benefit from G1 the best? Appreciate any comments you may have.-Joy -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlie.hunt at oracle.com Fri Feb 20 15:59:06 2015 From: charlie.hunt at oracle.com (charlie hunt) Date: Fri, 20 Feb 2015 09:59:06 -0600 Subject: G1 guidelines/suggestions In-Reply-To: <1730454645.3548827.1424384095334.JavaMail.yahoo@mail.yahoo.com> References: <1730454645.3548827.1424384095334.JavaMail.yahoo@mail.yahoo.com> Message-ID: Hi Joy, The obvious case where G1 will do better is where CMS is experiencing promotion failures due to old generation heap fragmentation. It might be easier to suggest where G1 currently may not do better than CMS, and also realize that this is a moving target. I would also suggest that if you can wait a few more weeks until JDK 8u40 comes out, you will likely have a better G1 experience than with previous JDK 8 (or JDK 7) releases due to the amount of G1 improvements being made. To date, the areas where G1 may not do as well as CMS are: 1.) Frequent large / humongous object allocations. If you have an application is doing frequent large object allocations, 1/2 a G1 region size or greater, a tuned CMS will probably do better out of the box than G1 with a min / max heap size and pause time target. There are ways to tune G1 that will help with frequent large object allocations, and there is a specific enhancement in JDK 8u40 that should help too. 2.) In general, if you are expecting to consistently see GC pause times at or less than 50 ms. G1 in its current state may have difficulty in beating a tuned CMS that consistently does less than 50 ms pauses. Until recently there has not been much focus on pause times less than 50 ms for G1. Most of its focus to date has been on larger Java heaps and a little less pause time constraint, say in that 100ms to 500 ms range, Realize that 100 ms to 500 ms time range is very general statement! I would also suggest that you start with JDK 8u40 (when it is generally available) on one of your applications, and if you can share any of your observations, there are folks on this mailing who would be very interested. A presentation that might help you with migrating from CMS to G1, especially with JVM command line options: http://www.slideshare.net/MonicaBeckwith/garbage-first-garbage-collector-g1-gc-migration-to-expectations-and-advanced-tuning * There is a section in this slide deck specifically on CMS to G1 migration, (starts on slide 47). One thing I think you will find is that tuning G1 is much, much easier than tuning CMS. :-) hths, charlie > On Feb 19, 2015, at 4:14 PM, Joy Xiong wrote: > > Hi, > > We have hundreds of services all running with CMS, and I am very interested in moving the services from CMS to G1. For that efforts to be convincing, I'd like to start services which has the most potential to beat CMS in terms of performance. I understand JVM performance is very application-specific, but still I want to check if anybody has guidelines/suggestions on which type of applications can benefit from G1 the best? > > Appreciate any comments you may have. > -Joy > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: From joyxiong at yahoo.com Fri Feb 20 21:09:29 2015 From: joyxiong at yahoo.com (Joy Xiong) Date: Fri, 20 Feb 2015 21:09:29 +0000 (UTC) Subject: G1 guidelines/suggestions In-Reply-To: References: Message-ID: <1306770865.5225183.1424466569177.JavaMail.yahoo@mail.yahoo.com> Hi Charlie, Thank you for the info! The slide deck is great. -Joy On Friday, February 20, 2015 7:59 AM, charlie hunt wrote: Hi Joy, The obvious case where G1 will do better is where CMS is experiencing promotion failures due to old generation heap fragmentation. It might be easier to suggest where G1 currently may not do better than CMS, and also realize that this is a moving target. I would also suggest that if you can wait a few more weeks until JDK 8u40 comes out, you will likely have a better G1 experience than with previous JDK 8 (or JDK 7) releases due to the amount of G1 improvements being made. To date, the areas where G1 may not do as well as CMS are:1.) ?Frequent large / humongous object allocations. ?If you have an application is doing frequent large object allocations, 1/2 a G1 region size or greater, a tuned CMS will probably do better out of the box than G1 with a min / max heap size and pause time target. There are ways to tune G1 that will help with frequent large object allocations, and there is a specific enhancement in JDK 8u40 that should help too.2.) ?In general, if you are expecting to consistently see GC pause times at or less than 50 ms. G1 in its current state may have difficulty in beating a tuned CMS that consistently does less than 50 ms pauses. Until recently there has not been much focus on pause times less than 50 ms for G1. Most of its focus to date has been on larger Java heaps and a little less pause time constraint, say in that 100ms to 500 ms range, Realize that 100 ms to 500 ms time range is very general statement! I would also suggest that you start with JDK 8u40 (when it is generally available) on one of your applications, and if you can share any of your observations, there are folks on this mailing who would be very interested. A presentation that might help you with migrating from CMS to G1, especially with JVM command line options:http://www.slideshare.net/MonicaBeckwith/garbage-first-garbage-collector-g1-gc-migration-to-expectations-and-advanced-tuning* There is a section in this slide deck specifically on CMS to G1 migration, (starts on slide 47). One thing I think you will find is that tuning G1 is much, much easier than tuning CMS. :-) hths, charlie On Feb 19, 2015, at 4:14 PM, Joy Xiong wrote: Hi, We have hundreds of services all running with CMS, and I am very interested in moving the services from CMS to G1. For that efforts to be convincing, I'd like to start services which has the most potential to beat CMS in terms of performance. I understand JVM performance is very application-specific, but still I want to check if anybody has guidelines/suggestions on which type of applications can benefit from G1 the best? Appreciate any comments you may have.-Joy_______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: From junjie.qian at yahoo.com Mon Feb 23 22:28:40 2015 From: junjie.qian at yahoo.com (Junjie Qian) Date: Mon, 23 Feb 2015 22:28:40 +0000 (UTC) Subject: DFS or BFS traversal in Parallel Scavenge (or Parallel new) GC Message-ID: <2045381094.8663658.1424730520340.JavaMail.yahoo@mail.yahoo.com> Dear all, I am reading the Parallel scavenge GC implementation in OpenJDK1.7, and having a question regarding the way how the parallel GC threads traverse the heap graph to mark the live objects. The question is, does one GC thread traverse the graph in DFS or BFS way in Parallel scavenge (or parallel new)? There are two conflicting sources suggesting the answer:1. the paper on ASPLOS'13, "A study of the scalability of stop-the-world garbage collectors on multicores", in Section 2.3.3. the author says "A GC thread performs a breadth-first traversal (BFT) of the graph of live objects". 2. the reply in this mailist, "Request for review (S): 8005972: ParNew should not update the tenuring threshold when promotion failed has occurred" , Ramki said "ParNew's slightly more DFS-like evacuation compared with DefNew's considerably more BFS-like evacuation because of the latter's use of a pure Cheney scan compared with the use of (a) marking stack(s) in the former" A lot of references online explains Parnew uses the same parallel GC algorithm as ParallelScavenge, please correct me if I made a mistake on this one.?Thanks!BestJunjie -------------- next part -------------- An HTML attachment was scrubbed... URL: From jon.masamitsu at oracle.com Tue Feb 24 21:45:20 2015 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Tue, 24 Feb 2015 13:45:20 -0800 Subject: longer ParNew collection in JDK8? In-Reply-To: <1480901824.951743.1423250277694.JavaMail.yahoo@mail.yahoo.com> References: <215756293.2414130.1423089943668.JavaMail.yahoo@mail.yahoo.com> <1480901824.951743.1423250277694.JavaMail.yahoo@mail.yahoo.com> Message-ID: <54ECF0F0.3040303@oracle.com> Joy, Try running with -XX:+UnlockDiagnosticVMOptions -XX:+BlockOffsetArrayUseUnallocatedBlock It may not be safe to run that way (see https://bugs.openjdk.java.net/browse/JDK-6948538 ) I haven't read through the bug trail to see if this has been fixed in a later release. Please let me know if this is the cause of the slowdown. Jon On 02/06/2015 11:17 AM, Joy Xiong wrote: > I tried the experimental flag: > -XX:+UnlockExperimentalVMOptions -XX:WorkStealingYieldsBeforeSleep=1000 > but the ParNew pause time is very close to JDK8 without the change. > > Here is the comparison data of ParNew pause in millisecond: > JDK6: 13 > JDK8: 36 > JDK8 (with WorkStealingYieldsBeforeSleep=1000): 38 > > I notice there are 4 tunables related with WorkStealingXXX, I am not > clear what these tunables are. But is it worthwhile to try a > combination of these? > /WorkStealingYieldsBeforeSleep/, > /WorkStealingHardSpins / > /WorkStealingSleepMillis / > /WorkStealingSpinToYieldRatio / > > thanks, > -Joy > > > On Wednesday, February 4, 2015 2:45 PM, Joy Xiong > wrote: > > > Hi Staffan, > > Thanks for the suggestion. I will try to find an experimental testbed > and give it a try. Also for you info, it seems to me the longer ParNew > pause is introduced in JDK7, as I saw similar increase when I used JDK7. > > thanks, > -Joy > > > On Wednesday, February 4, 2015 9:57 AM, Staffan Friberg > wrote: > > > Hi Joy, > > Could you try with the following extra option, it would change the > setting to be similar in JDK 7 as they are in JDK 6. > > -XX:+UnlockExperimentalVMOptions -XX:WorkStealingYieldsBeforeSleep=1000 > > //Staffan > > On 02/02/2015 10:38 AM, Joy Xiong wrote: >> Hi Jon, >> >> Yes, most objects in my application die young and very little >> promoted. As you can see from the attached figure (the occupied heap >> size from Sep 28 to Nov 11), CMS happened very rare. However, it >> seems two releases do not have much difference in promotion rate, as >> they have same number of CMS in the log. >> >> We have the same JVM parameters for both releases as below: >> >> CommandLine flags: -XX:+AlwaysPreTouch >> -XX:CMSInitiatingOccupancyFraction=70 -XX:ErrorFile=logs/hs_err.log >> -XX:InitialHeapSize=34271657984 -XX:+ManagementServer >> -XX:MaxHeapSize=34271657984 -XX:MaxNewSize=2147483648 >> -XX:NewSize=2147483648 -XX:OldPLABSize=16 -XX:+PrintGC >> -XX:+PrintGCApplicationConcurrentTime >> -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps >> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps >> -XX:+PrintTenuringDistribution -XX:SurvivorRatio=2 >> -XX:+UnlockDiagnosticVMOptions -XX:+UseCompressedClassPointers >> -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC >> >> thanks, >> -Joy >> >> >> On Monday, February 2, 2015 10:06 AM, Jon Masamitsu >> wrote: >> >> >> Joy, >> >> Is your application such that most everything dies in the >> young generation and very little is promoted? If yes, >> do you know if there is a difference in how much is >> promoted between the two releases? >> >> Jon >> >> On 01/29/2015 10:27 AM, Joy Xiong wrote: >>> >>> Hi, >>> >>> We recently move our services from JDK6 to JDK8, but observe longer >>> ParNew GC pause in JDK8. For example, the pause time increase from >>> 27ms to 43ms in one service. The service has the JVM parameters as >>> below: >>> -Xms32684m -Xmx32684m -XX:NewSize=2048m -XX:MaxNewSize=2048m >>> -XX:+UseConcMarkSweepGC -XX:+UseParNewGC >>> -XX:CMSInitiatingOccupancyFraction=70 -XX:SurvivorRatio=2 >>> -XX:+AlwaysPreTouch -XX:+UseCompressedOops >>> -XX:+PrintTenuringDistribution -XX:+PrintGCDetails >>> -XX:+PrintGCDateStamps -Xloggc:logs/gc.log >>> -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime >>> >>> Is this longer ParNew pause expected? Any suggestions to mitigate >>> the longer pause? >>> >>> thanks, >>> -Joy >>> >>> >>> _______________________________________________ >>> hotspot-gc-use mailing list >>> hotspot-gc-use at openjdk.java.net >>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> >> >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jon.masamitsu at oracle.com Wed Feb 25 00:04:24 2015 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Tue, 24 Feb 2015 16:04:24 -0800 Subject: DFS or BFS traversal in Parallel Scavenge (or Parallel new) GC In-Reply-To: <2045381094.8663658.1424730520340.JavaMail.yahoo@mail.yahoo.com> References: <2045381094.8663658.1424730520340.JavaMail.yahoo@mail.yahoo.com> Message-ID: <54ED1188.7090704@oracle.com> Junjie, If you want a definitive answer, you really have to go look at the code. If you're willing to take my best recollection then read on. On 02/23/2015 02:28 PM, Junjie Qian wrote: > Dear all, > > I am reading the Parallel scavenge GC implementation in OpenJDK1.7, > and having a question regarding the way how the parallel GC threads > traverse the heap graph to mark the live objects. > > The question is, does one GC thread traverse the graph in DFS or BFS > way in Parallel scavenge (or parallel new)? UseParallelGC had both DFS and BFS under a flag. I think the BFS was removed in 7. > > There are two conflicting sources suggesting the answer: > 1. the paper on ASPLOS'13, "A study of the scalability of > stop-the-world garbage collectors on multicores", in Section 2.3.3. > the author says "A GC thread performs a breadth-first traversal (BFT) > of the graph of live objects". > > 2. the reply in this mailist, "Request for review (S): 8005972: ParNew > should not update the tenuring threshold when promotion failed has > occurred" , Ramki said "ParNew's slightly more DFS-like evacuation > compared with DefNew's considerably more BFS-like evacuation because > of the latter's use of a pure Cheney scan compared with the use of (a) > marking stack(s) in the former" Trust Ramki's description at that time, but I would have called UseParNewGC predominantly BFS. > > A lot of references online explains Parnew uses the same parallel GC > algorithm as ParallelScavenge, please correct me if I made a mistake > on this one. UseParallelGC and UseParNewGC are both copying collectors that use multiple threads so in that since they use the same algorithm. They share very little code. UseParNewGC works with CMS and makes some accommodations for CMS. Kinda of like a Ford and a Chevy are both cars. Jon > Thanks! > Best > Junjie > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhelbing at icubic.de Wed Feb 25 11:41:58 2015 From: rhelbing at icubic.de (Ralf Helbing) Date: Wed, 25 Feb 2015 12:41:58 +0100 Subject: SPECjvm2008 Message-ID: <54EDB506.4060708@icubic.de> I already asked this a long time ago on java.net. Based on the non-answers I got, I assumed SPECjvm2008 passed away quietly, but I read here and on gc-dev that some folks still use it. So I asked again at community.oracle.com/thread/3676919. How can I run SPECjvm2008 on windows machines? It seems to get stuck in startup.sunflow when using 1.8.0_40 and possibly other 1.8 jvms under Linux and Windows. Windows runs also hang at any point during the startup runs, even with a 1.7 vm. There seems to be no support for Java 9. Are there any updates for SPECjvm2008? What am I doing wrong? What magic is required? thanks in advance From rhelbing at icubic.de Wed Feb 25 11:53:31 2015 From: rhelbing at icubic.de (Ralf Helbing) Date: Wed, 25 Feb 2015 12:53:31 +0100 Subject: access restriction: GcInfo is not API (access restriction on required library rt.jar) Message-ID: <54EDB7BB.6020206@icubic.de> For quite a while I used code based on http://www.fasterj.com/articles/gcnotifs.shtml to monitor GC operation. With 1.7 jvms, this had no issues. Java 8 however, considers GcInfo and some related classes "no API" anymore. I can let Eclipse turn this error into a warning and continue, but maybe there is a kosher way to obtain this information. Or maybe restricting access has a reason that developers should be aware of? From yu.zhang at oracle.com Wed Feb 25 18:00:36 2015 From: yu.zhang at oracle.com (Yu Zhang) Date: Wed, 25 Feb 2015 10:00:36 -0800 Subject: SPECjvm2008 In-Reply-To: <54EDB506.4060708@icubic.de> References: <54EDB506.4060708@icubic.de> Message-ID: <54EE0DC4.5010606@oracle.com> This might be related to the deprecated javac in the benchmark. SPECjvm2008 bundles and early jdk7 javac compiler, which is incompatible with jdk8 Thanks, Jenny On 2/25/2015 3:41 AM, Ralf Helbing wrote: > I already asked this a long time ago on java.net. Based on the > non-answers I got, I assumed SPECjvm2008 passed away quietly, but I > read here and on gc-dev that some folks still use it. So I asked again > at community.oracle.com/thread/3676919. > > How can I run SPECjvm2008 on windows machines? It seems to get stuck > in startup.sunflow when using 1.8.0_40 and possibly other 1.8 jvms > under Linux and Windows. Windows runs also hang at any point during > the startup runs, even with a 1.7 vm. There seems to be no support for > Java 9. > > Are there any updates for SPECjvm2008? What am I doing wrong? What > magic is required? > > thanks in advance > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: From staffan.friberg at oracle.com Wed Feb 25 19:32:26 2015 From: staffan.friberg at oracle.com (Staffan Friberg) Date: Wed, 25 Feb 2015 11:32:26 -0800 Subject: SPECjvm2008 In-Reply-To: <54EE0DC4.5010606@oracle.com> References: <54EDB506.4060708@icubic.de> <54EE0DC4.5010606@oracle.com> Message-ID: <54EE234A.5080405@oracle.com> SPECjvm2008 does not support JDK 8, which is due to the issue mentioned by Jenny. ---- http://www.spec.org/jvm2008/docs/RunRules.html > The system must include a complete implementation of those classes that are referenced by this benchmark as in the Java SE 1.5.0, Java SE 6 or Java SE 7 specifications. Other Java SE specifications are not supported. ---- Regards, Staffan On 02/25/2015 10:00 AM, Yu Zhang wrote: > This might be related to the deprecated javac in the benchmark. > SPECjvm2008 bundles and early jdk7 javac compiler, which is > incompatible with jdk8 > > Thanks, > Jenny > On 2/25/2015 3:41 AM, Ralf Helbing wrote: >> I already asked this a long time ago on java.net. Based on the >> non-answers I got, I assumed SPECjvm2008 passed away quietly, but I >> read here and on gc-dev that some folks still use it. So I asked >> again at community.oracle.com/thread/3676919. >> >> How can I run SPECjvm2008 on windows machines? It seems to get stuck >> in startup.sunflow when using 1.8.0_40 and possibly other 1.8 jvms >> under Linux and Windows. Windows runs also hang at any point during >> the startup runs, even with a 1.7 vm. There seems to be no support >> for Java 9. >> >> Are there any updates for SPECjvm2008? What am I doing wrong? What >> magic is required? >> >> thanks in advance >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: From Vishal.Bhasin at sabre.com Wed Feb 25 19:38:59 2015 From: Vishal.Bhasin at sabre.com (Bhasin, Vishal) Date: Wed, 25 Feb 2015 13:38:59 -0600 Subject: unsubscribe Message-ID: <47CD97E925B8E848A5B20396887A4C450936FDC2BF@SGTULMMP002.Global.ad.sabre.com> unsubscribe -------------- next part -------------- An HTML attachment was scrubbed... URL: From bharat.2511 at gmail.com Wed Feb 4 00:32:11 2015 From: bharat.2511 at gmail.com (bharat naik) Date: Wed, 04 Feb 2015 00:32:11 -0000 Subject: Debugging high JNI Weak Reference times Message-ID: Hi all, We used to see a high amount of time spent during the rescan phase of GC, i..e the weak refs processing part used to take around 140 seconds. This would occur about 3-4 times for every 70 runs. We set the following options: -XX:+ParallelRefProcEnabled and -XX:+PrintReferenceGC to get more details and as a workaround. The time reduced to ~10 seconds and we see that most of it is spent in resolving JNI weak references. I did see a few similar problems on the net but they were all related to the G1 GC and we use CMS. How does one usually go about debugging further in this scenario and are there any other options that we could try to help mitigate this problem? Java version is: java version "1.7.0_45" Java(TM) SE Runtime Environment (build 1.7.0_45-b18) Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode) These are the relevant GC options: -XX:+UseCompressedOops -XX:SurvivorRatio=3 -XX:+UseLargePages -XX:TargetSurvivorRatio=90 -XX:+AlwaysPreTouch -XX:+AggressiveOpts -XX:+UseNUMA -XX:+UseTLAB -XX:+UseParNewGC -XX:+UseConcMarkSweepGC - -XX:CMSWaitDuration=90000 -XX:+ParallelRefProcEnabled -XX:+PrintReferenceGC -XX:+PrintJNIGCStalls -XX:ParallelCMSThreads=16 -XX:+CMSScavengeBeforeRemark -XX:-CMSConcurrentMTEnabled -XX:CMSInitiatingOccupancyFraction=10 -XX:+UseCMSInitiatingOccupancyOnly Any help is appreciated! Thanks, Bharat. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dvdeepankar.reddy at gmail.com Thu Feb 12 00:02:54 2015 From: dvdeepankar.reddy at gmail.com (D vd Reddy) Date: Wed, 11 Feb 2015 16:02:54 -0800 Subject: Question about RS Set size Message-ID: Hi, we are using G1 GC for heap size of about 92 - 96 GB and we observed in the logs that Scan RS times were too high (comparable to Object Copy time, Is this normal ?, I have checked that there are no coarsening(s)). So in order to counter that we plan to decrease the region count, currently we set it explicitly to 4096 with regionSize to 32MB. My question is that if we drop setting regionSize and decrease the regions to 2048 what will happen to region size, Can it go above 32 MB and keep the regions to 2048 or is it capped at 32MB ? Thanks in advance -------------- next part -------------- An HTML attachment was scrubbed... URL: From wolfgang.pedot at finkzeit.at Thu Feb 19 10:37:03 2015 From: wolfgang.pedot at finkzeit.at (Wolfgang Pedot) Date: Thu, 19 Feb 2015 11:37:03 +0100 Subject: G1GC, Java8u40ea, Metaspace questions In-Reply-To: <54E41127.3040002@oracle.com> References: <821215C9-36AC-41BB-A9A6-1E136341778F@finkzeit.at> <54DE3254.9030503@oracle.com> <54DE41A7.6050004@finkzeit.at> <54DE5495.2010501@oracle.com> <54E394FB.3040204@oracle.com> <54E39BCC.7090802@finkzeit.at> <54E41127.3040002@oracle.com> Message-ID: <54E5BCCF.8050902@finkzeit.at> Hi Am 18.02.2015 05:12, schrieb Yu Zhang: > Hi, > > After discussing with the developers, here is the explanation: > >shows a slow but continuous increase of the "commited" value for > Metaspace even though "used" after a collect is quite stable and > "commited" is shrunk after every collect. > > This might be due to fragmentation in metadata space. Even though the > total used is stable, the memory is fragmented so it has to allocate > more. If you have Flight Recorder(a commercial feature), you can see > this information in JFR. Unfortunately I do not have JFR available, however I did notice that the number of classes that fit into Metaspace before the collect also kept increasing so it cant all be fragmentation. > >I have found out that I need to set Min and MaxMetaspaceFreeRatio > really really low (like <=10%) to keep the high-water-mark in check > and I dont quite see why in the numbers. > > This might be related to how the high water mark is calculated. There > are some ways to improve this but not implemented yet. The more I test the more I come to the conclusion that there is something strange about that calculation (or my understanding of it). I am currently running a very long test on 64bit linux (>9million classes loaded so far) and there is a definite upwards trend of both the HWM and used-after-GC. Class unloading does not seem to be as successful when a concurrent cycle is triggered by Metaspace threshold compared to a Full-GC and that gap is growing over time. I guess thats because the heap of the test-JVM is rather large and empty and since G1 is allowed to waste 5% if it that means some classloaders stay alive longer although that should stabilize at some point. Increasing the usage in oldGen from ~0.5GB to ~1GB made class unloading "better" for one collect. The high-point of Metaspace usage slowly increased from ~400MB to ~600MB during the first part of the test and used-after-GC started at 200MB and crept up to around 400MB. I decided to throw in a manual FullGC during the test to see what happens and that brought the usage down to ~110MB as it should be so there is no real class-leak. Since MaxMetaspaceFreeRatio is configured to 30% (Min is 10%) I would have expected the HWM to go down after that collect but appearantly it did not, it stayed at ~600MB for the next collect and then went up another ~50MB. I dont really see the HWM-value so when I write "HWM" I am referring to the highest point of Metaspace-usage before a collect is initiated. I have attached a screenshot of jconsole showing the part with the FullGC (~9:40), I reconnected to the JVM at ~8:55 so it does not cover the whole uptime. Is this expected or explainable behaviour? The way I see it the high points of that curve may keep on climbing until they hit the limit and after that its FullGCs all the time. regards Wolfgang -------------- next part -------------- A non-text attachment was scrubbed... Name: jconsole_8u40_2048_30_10_fullgc.png Type: image/png Size: 102718 bytes Desc: not available URL: From yiyeguhu at gmail.com Wed Feb 25 23:01:27 2015 From: yiyeguhu at gmail.com (Tao Mao) Date: Wed, 25 Feb 2015 15:01:27 -0800 Subject: Debugging high JNI Weak Reference times In-Reply-To: References: Message-ID: Hi Bharat, Please attach your gc log with at least the following options "-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" Regards, Tao Mao On Tue, Feb 3, 2015 at 4:31 PM, bharat naik wrote: > Hi all, > > We used to see a high amount of time spent during the rescan phase of GC, > i..e the weak refs processing part used to take around 140 seconds. This > would occur about 3-4 times for every 70 runs. > > We set the following options: -XX:+ParallelRefProcEnabled and > -XX:+PrintReferenceGC to get more details and as a workaround. The time > reduced to ~10 seconds and we see that most of it is spent in resolving JNI > weak references. > > I did see a few similar problems on the net but they were all related to > the G1 GC and we use CMS. How does one usually go about debugging further > in this scenario and are there any other options that we could try to help > mitigate this problem? > > Java version is: > java version "1.7.0_45" > > Java(TM) SE Runtime Environment (build 1.7.0_45-b18) > > Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode) > > These are the relevant GC options: > > -XX:+UseCompressedOops > > -XX:SurvivorRatio=3 > > -XX:+UseLargePages > > -XX:TargetSurvivorRatio=90 > > -XX:+AlwaysPreTouch > > -XX:+AggressiveOpts > > -XX:+UseNUMA > > -XX:+UseTLAB > > -XX:+UseParNewGC > > -XX:+UseConcMarkSweepGC - > > -XX:CMSWaitDuration=90000 > > -XX:+ParallelRefProcEnabled > > -XX:+PrintReferenceGC > > -XX:+PrintJNIGCStalls > > -XX:ParallelCMSThreads=16 > > -XX:+CMSScavengeBeforeRemark > > -XX:-CMSConcurrentMTEnabled > > -XX:CMSInitiatingOccupancyFraction=10 > > -XX:+UseCMSInitiatingOccupancyOnly > > > Any help is appreciated! > > > Thanks, > > Bharat. > > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yiyeguhu at gmail.com Wed Feb 25 23:24:47 2015 From: yiyeguhu at gmail.com (Tao Mao) Date: Wed, 25 Feb 2015 15:24:47 -0800 Subject: Debugging high JNI Weak Reference times In-Reply-To: References: Message-ID: Hi Bharat, Then, the weird thing is: -XX:+ParallelRefProcEnabled did help reduce JNI weak references processing to ~10 sec in your case while the ticket says that JNI weak ref processing is single-threaded even with ParallelRefProcEnabled on. Thanks. On Wed, Feb 25, 2015 at 3:19 PM, bharat naik wrote: > Hi Tao, > > Thanks for the reply,but this occurrence was because JNI weak reference > processing isn't done in parallel currently. > A bug has been filed regarding this: JDK-8072498 > > > Thanks, > Bharat. > > On Wed, Feb 25, 2015 at 3:01 PM, Tao Mao wrote: > >> Hi Bharat, >> >> Please attach your gc log with at least the following options >> "-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" >> >> Regards, >> Tao Mao >> >> >> On Tue, Feb 3, 2015 at 4:31 PM, bharat naik >> wrote: >> >>> Hi all, >>> >>> We used to see a high amount of time spent during the rescan phase of >>> GC, i..e the weak refs processing part used to take around 140 seconds. >>> This would occur about 3-4 times for every 70 runs. >>> >>> We set the following options: -XX:+ParallelRefProcEnabled and >>> -XX:+PrintReferenceGC to get more details and as a workaround. The >>> time reduced to ~10 seconds and we see that most of it is spent in >>> resolving JNI weak references. >>> >>> I did see a few similar problems on the net but they were all related to >>> the G1 GC and we use CMS. How does one usually go about debugging further >>> in this scenario and are there any other options that we could try to help >>> mitigate this problem? >>> >>> Java version is: >>> java version "1.7.0_45" >>> >>> Java(TM) SE Runtime Environment (build 1.7.0_45-b18) >>> >>> Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode) >>> >>> These are the relevant GC options: >>> >>> -XX:+UseCompressedOops >>> >>> -XX:SurvivorRatio=3 >>> >>> -XX:+UseLargePages >>> >>> -XX:TargetSurvivorRatio=90 >>> >>> -XX:+AlwaysPreTouch >>> >>> -XX:+AggressiveOpts >>> >>> -XX:+UseNUMA >>> >>> -XX:+UseTLAB >>> >>> -XX:+UseParNewGC >>> >>> -XX:+UseConcMarkSweepGC - >>> >>> -XX:CMSWaitDuration=90000 >>> >>> -XX:+ParallelRefProcEnabled >>> >>> -XX:+PrintReferenceGC >>> >>> -XX:+PrintJNIGCStalls >>> >>> -XX:ParallelCMSThreads=16 >>> >>> -XX:+CMSScavengeBeforeRemark >>> >>> -XX:-CMSConcurrentMTEnabled >>> >>> -XX:CMSInitiatingOccupancyFraction=10 >>> >>> -XX:+UseCMSInitiatingOccupancyOnly >>> >>> >>> Any help is appreciated! >>> >>> >>> Thanks, >>> >>> Bharat. >>> >>> >>> >>> _______________________________________________ >>> hotspot-gc-use mailing list >>> hotspot-gc-use at openjdk.java.net >>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bharat.2511 at gmail.com Wed Feb 25 23:19:13 2015 From: bharat.2511 at gmail.com (bharat naik) Date: Wed, 25 Feb 2015 15:19:13 -0800 Subject: Debugging high JNI Weak Reference times In-Reply-To: References: Message-ID: Hi Tao, Thanks for the reply,but this occurrence was because JNI weak reference processing isn't done in parallel currently. A bug has been filed regarding this: JDK-8072498 Thanks, Bharat. On Wed, Feb 25, 2015 at 3:01 PM, Tao Mao wrote: > Hi Bharat, > > Please attach your gc log with at least the following options "-verbose:gc > -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" > > Regards, > Tao Mao > > > On Tue, Feb 3, 2015 at 4:31 PM, bharat naik wrote: > >> Hi all, >> >> We used to see a high amount of time spent during the rescan phase of GC, >> i..e the weak refs processing part used to take around 140 seconds. This >> would occur about 3-4 times for every 70 runs. >> >> We set the following options: -XX:+ParallelRefProcEnabled and >> -XX:+PrintReferenceGC to get more details and as a workaround. The time >> reduced to ~10 seconds and we see that most of it is spent in resolving JNI >> weak references. >> >> I did see a few similar problems on the net but they were all related to >> the G1 GC and we use CMS. How does one usually go about debugging further >> in this scenario and are there any other options that we could try to help >> mitigate this problem? >> >> Java version is: >> java version "1.7.0_45" >> >> Java(TM) SE Runtime Environment (build 1.7.0_45-b18) >> >> Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode) >> >> These are the relevant GC options: >> >> -XX:+UseCompressedOops >> >> -XX:SurvivorRatio=3 >> >> -XX:+UseLargePages >> >> -XX:TargetSurvivorRatio=90 >> >> -XX:+AlwaysPreTouch >> >> -XX:+AggressiveOpts >> >> -XX:+UseNUMA >> >> -XX:+UseTLAB >> >> -XX:+UseParNewGC >> >> -XX:+UseConcMarkSweepGC - >> >> -XX:CMSWaitDuration=90000 >> >> -XX:+ParallelRefProcEnabled >> >> -XX:+PrintReferenceGC >> >> -XX:+PrintJNIGCStalls >> >> -XX:ParallelCMSThreads=16 >> >> -XX:+CMSScavengeBeforeRemark >> >> -XX:-CMSConcurrentMTEnabled >> >> -XX:CMSInitiatingOccupancyFraction=10 >> >> -XX:+UseCMSInitiatingOccupancyOnly >> >> >> Any help is appreciated! >> >> >> Thanks, >> >> Bharat. >> >> >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jon.masamitsu at oracle.com Thu Feb 26 00:02:33 2015 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Wed, 25 Feb 2015 16:02:33 -0800 Subject: G1GC, Java8u40ea, Metaspace questions In-Reply-To: <54E5BCCF.8050902@finkzeit.at> References: <821215C9-36AC-41BB-A9A6-1E136341778F@finkzeit.at> <54DE3254.9030503@oracle.com> <54DE41A7.6050004@finkzeit.at> <54DE5495.2010501@oracle.com> <54E394FB.3040204@oracle.com> <54E39BCC.7090802@finkzeit.at> <54E41127.3040002@oracle.com> <54E5BCCF.8050902@finkzeit.at> Message-ID: <54EE6299.4010705@oracle.com> Wolfgang, Can you run you test program with UseParallelGC and see if you see the same growth in the Metaspace? Jon On 2/19/2015 2:37 AM, Wolfgang Pedot wrote: > Hi > > Am 18.02.2015 05:12, schrieb Yu Zhang: >> Hi, >> >> After discussing with the developers, here is the explanation: >> >shows a slow but continuous increase of the "commited" value for >> Metaspace even though "used" after a collect is quite stable and >> "commited" is shrunk after every collect. >> >> This might be due to fragmentation in metadata space. Even though >> the total used is stable, the memory is fragmented so it has to >> allocate more. If you have Flight Recorder(a commercial feature), >> you can see this information in JFR. > > Unfortunately I do not have JFR available, however I did notice that > the number of classes that fit into Metaspace before the collect also > kept increasing so it cant all be fragmentation. > >> >I have found out that I need to set Min and MaxMetaspaceFreeRatio >> really really low (like <=10%) to keep the high-water-mark in check >> and I dont quite see why in the numbers. >> >> This might be related to how the high water mark is calculated. There >> are some ways to improve this but not implemented yet. > > The more I test the more I come to the conclusion that there is > something strange about that calculation (or my understanding of it). > I am currently running a very long test on 64bit linux (>9million > classes loaded so far) and there is a definite upwards trend of both > the HWM and used-after-GC. Class unloading does not seem to be as > successful when a concurrent cycle is triggered by Metaspace threshold > compared to a Full-GC and that gap is growing over time. I guess thats > because the heap of the test-JVM is rather large and empty and since > G1 is allowed to waste 5% if it that means some classloaders stay > alive longer although that should stabilize at some point. Increasing > the usage in oldGen from ~0.5GB to ~1GB made class unloading "better" > for one collect. > > The high-point of Metaspace usage slowly increased from ~400MB to > ~600MB during the first part of the test and used-after-GC started at > 200MB and crept up to around 400MB. I decided to throw in a manual > FullGC during the test to see what happens and that brought the usage > down to ~110MB as it should be so there is no real class-leak. Since > MaxMetaspaceFreeRatio is configured to 30% (Min is 10%) I would have > expected the HWM to go down after that collect but appearantly it did > not, it stayed at ~600MB for the next collect and then went up another > ~50MB. > I dont really see the HWM-value so when I write "HWM" I am referring > to the highest point of Metaspace-usage before a collect is initiated. > > I have attached a screenshot of jconsole showing the part with the > FullGC (~9:40), I reconnected to the JVM at ~8:55 so it does not cover > the whole uptime. > > Is this expected or explainable behaviour? The way I see it the high > points of that curve may keep on climbing until they hit the limit and > after that its FullGCs all the time. > > regards > Wolfgang > From wolfgang.pedot at finkzeit.at Thu Feb 26 09:23:01 2015 From: wolfgang.pedot at finkzeit.at (Wolfgang Pedot) Date: Thu, 26 Feb 2015 10:23:01 +0100 Subject: G1GC, Java8u40ea, Metaspace questions In-Reply-To: <54EE6299.4010705@oracle.com> References: <821215C9-36AC-41BB-A9A6-1E136341778F@finkzeit.at> <54DE3254.9030503@oracle.com> <54DE41A7.6050004@finkzeit.at> <54DE5495.2010501@oracle.com> <54E394FB.3040204@oracle.com> <54E39BCC.7090802@finkzeit.at> <54E41127.3040002@oracle.com> <54E5BCCF.8050902@finkzeit.at> <54EE6299.4010705@oracle.com> Message-ID: <54EEE5F5.6010305@finkzeit.at> Jon, Am 26.02.2015 01:02, schrieb Jon Masamitsu: > Wolfgang, > > Can you run you test program with UseParallelGC and see if you > see the same growth in the Metaspace? > test is now running for a good 30min and it does not look like there is growth. The configuration is the same as before, I just changed +UseG1GC to -UseG1GC and added +UseParallelGC. This collector is more successful in getting rid of classes right away so Metaspace usage after GC stays around ~110MB. MetaspaceSize is configured to 300MB (Max is 2000MB) and the maximum values for commited Metaspace in the gclog are 307200K. Since usage stays lower than with G1 I am not sure if this is a "fair" test regarding HWM calculation. Wolfgang From jon.masamitsu at oracle.com Thu Feb 26 19:11:28 2015 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Thu, 26 Feb 2015 11:11:28 -0800 Subject: G1GC, Java8u40ea, Metaspace questions In-Reply-To: <54EEE504.2050502@finkzeit.at> References: <821215C9-36AC-41BB-A9A6-1E136341778F@finkzeit.at> <54DE3254.9030503@oracle.com> <54DE41A7.6050004@finkzeit.at> <54DE5495.2010501@oracle.com> <54E394FB.3040204@oracle.com> <54E39BCC.7090802@finkzeit.at> <54E41127.3040002@oracle.com> <54E5BCCF.8050902@finkzeit.at> <54EE6299.4010705@oracle.com> <54EEE504.2050502@finkzeit.at> Message-ID: <54EF6FE0.4000907@oracle.com> On 02/26/2015 01:19 AM, Wolfgang Pedot wrote: > Jon, > > Am 26.02.2015 01:02, schrieb Jon Masamitsu: >> Wolfgang, >> >> Can you run you test program with UseParallelGC and see if you >> see the same growth in the Metaspace? >> > > test is now running for a good 30min and it does not look like there > is growth. > The configuration is the same as before, I just changed +UseG1GC to > -UseG1GC and added +UseParallelGC. This collector is more successful > in getting rid of classes right away so Metaspace usage after GC stays > around ~110MB. > MetaspaceSize is configured to 300MB (Max is 2000MB) and the maximum > values for commited Metaspace in the gclog are 307200K, I have > attached a screenshot of jconsole showing Metaspace usage. > Since usage stays lower than with G1 I am not sure if this is a "fair" > test regarding HWM calculation. Thanks for running the tests. The difference could be the result of UseParallelGC doing a STW full collection when the HWM is reached and G1 starting a concurrent collection when the HWM is reached. I could accept the HWM with G1 being higher but having it grow is a bug. Jon > > Wolfgang > From yu.zhang at oracle.com Thu Feb 26 22:28:27 2015 From: yu.zhang at oracle.com (Yu Zhang) Date: Thu, 26 Feb 2015 14:28:27 -0800 Subject: WeakReference processing: long times during G1 remark In-Reply-To: References: Message-ID: <54EF9E0B.20301@oracle.com> Hi, Simone, Sorry for the very late response. I am posting my understanding how references are processed in G1. I might misunderstand it, I am hoping others can correct me if I am wrong. G1 has 2 reference processors: stop the world (_ref_processor_stw) and concurrent_marking (_ref_processor_cm). _ref_processor_stw is used during STW pauses(young/mixed/full gc) ; _ref_processor_cm is used during concurrent marking. These 2 both span the heap. During a STW pause, _ref_processor_cm discovery is temporally disabled. _ref_processor_stw handles the reference object discovery and processing. At the end of the evacuation, _ref_processor_stw process the discovered references and en queue the rest. _ref_processor_cm discovery is restored. ref_processor_cm handles reference object discovery and processing during the concurrent marking cycle. The discovery is enabled during initial-marking. The reference is processed during remark. It is strange that if your weak reference allocation is 2.5M per hour, but 3M in 39s during remark. >Am I right in assuming that a young GC processes the weak refs from the young generation, while a remark processes the weak refs that have been found during marking (and therefore those that are in old generation) ? The references during young gc are discovered during young gc pause. While the references processed during remark are discovered during marking cycle. > Another thing I noticed is that the weak refs processed during the mixed GC do not sum up to the 3M processed during remark: at most few thousands vs 3M. Not sure they should, I just wanted to report this information in case it matters. They are not related. Thanks, Jenny On 2/10/2015 10:34 AM, Simone Bordet wrote: > Hi, > > I just want to double check with the experts whether what I see is > normal or needs more investigation. > > The application uses one ThreadLocal, which eventually generates > WeakReferences, lots of them. > Below you can find 2 G1 remark GC log lines that are 39s apart. > -XX:+ParallelRefProcEnabled is on, JDK 8u25. > > What I am interested in is understanding if ~35s of CPU time is > considered normal to process 3M of weak references, since seems a lot > of time (about 12 micros each). > > Also, the weak refs number reported in the GC log line is the number > of weak refs that have been GC marked, or the number of weak refs that > have been enqueued (their referent is weakly reachable) ? > I ask because we have instrumented WeakReference allocation, and the > numbers shows an allocation of about 2.5M weak refs per hour, while > here seems 3M in 39s. > Furthermore, we have no evidence that 3M weak refs survive for a long > period of time, and we see a big difference in the number of weak refs > processed during a young GC (at most around 1K, more often in the tens > or hundreds), and the 3M processed during remark. > > Am I right in assuming that a young GC processes the weak refs from > the young generation, while a remark processes the weak refs that have > been found during marking (and therefore those that are in old > generation) ? > > Another thing I noticed is that the weak refs processed during the > mixed GC do not sum up to the 3M processed during remark: at most few > thousands vs 3M. Not sure they should, I just wanted to report this > information in case it matters. > > Is there any known issue about ref processing during the remark phase ? > > Thanks ! > > > 2015-02-09T12:07:00.327-0800: 31671.525: [GC remark 31671.530: [GC > ref-proc31671.530: [SoftReference, 2 refs, 0.0052296 secs]31671.535: > [WeakReference, 3264308 refs, 2.0524238 secs]31673.588: > [FinalReference, 215 refs, 0.0028225 secs]31673.591: > [PhantomReference, 1787 refs, 0.0050046 secs]31673.596: [JNI Weak > Reference, 0.0007776 secs], 2.0932150 secs], 2.1138048 secs] > [Times: user=36.39 sys=0.68, real=2.11 secs] > > 2015-02-09T12:07:39.210-0800: 31710.408: [GC remark 31710.413: [GC > ref-proc31710.413: [SoftReference, 3 refs, 0.0064721 secs]31710.420: > [WeakReference, 3168257 refs, 1.7807508 secs]31712.201: > [FinalReference, 268 refs, 0.0025569 secs]31712.203: > [PhantomReference, 1702 refs, 0.0038438 secs]31712.207: [JNI Weak > Reference, 0.0007997 secs], 1.8847662 secs], 1.9052513 secs] > [Times: user=34.54 sys=0.63, real=1.90 secs] >