From naman.jain1894 at yahoo.in Tue Oct 6 08:58:09 2015 From: naman.jain1894 at yahoo.in (Naman Jain) Date: Tue, 6 Oct 2015 08:58:09 +0000 (UTC) Subject: Huge difference in GC pause time and total application pause time. Message-ID: <287580904.383951.1444121889025.JavaMail.yahoo@mail.yahoo.com> Hey Everyone, G1 GC log:2015-10-05T14:54:35.518+0000: 549629.568: [GC pause (G1 Evacuation Pause) (young)Desired survivor size 322961408 bytes, new threshold 15 (max 15)- age ? 1:? ? 5292232 bytes,? ? 5292232 total- age ? 2:? ? 1201264 bytes,? ? 6493496 total- age ? 3: ? ? 377376 bytes,? ? 6870872 total- age ? 4: ? ? 440752 bytes,? ? 7311624 total- age ? 5: ? ? 201976 bytes,? ? 7513600 total- age ? 6:? ? ? 77616 bytes,? ? 7591216 total- age ? 7:? ? ? 28824 bytes,? ? 7620040 total- age ? 8:? ? ? 55312 bytes,? ? 7675352 total- age ? 9:? ? ? 55432 bytes,? ? 7730784 total- age? 10:? ? ? 14272 bytes,? ? 7745056 total- age? 11:? ? ? 16872 bytes,? ? 7761928 total- age? 12:? ? ? 18208 bytes,? ? 7780136 total- age? 13:? ? ? 11656 bytes,? ? 7791792 total- age? 14:? ? ? 15800 bytes,? ? 7807592 total- age? 15:? ? ? 33960 bytes,? ? 7841552 total?549629.569: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 14904, predicted base time: 20.98 ms, remaining time: 19.02 ms, target pause time: 40.00 ms]?549629.569: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 1224 regions, survivors: 4 regions, predicted young region time: 10.73 ms]?549629.569: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 1224 regions, survivors: 4 regions, old: 0 regions, predicted pause time: 31.70 ms, target pause time: 40.00 ms], 0.0355102 secs]?? [Parallel Time: 21.7 ms, GC Workers: 18]? ? ? [GC Worker Start (ms): Min: 549629569.6, Avg: 549629569.8, Max: 549629569.9, Diff: 0.3]? ? ? [Ext Root Scanning (ms): Min: 3.6, Avg: 4.3, Max: 12.7, Diff: 9.0, Sum: 78.1]? ? ? [Update RS (ms): Min: 0.0, Avg: 4.1, Max: 4.7, Diff: 4.7, Sum: 74.6]?? ? ? ? [Processed Buffers: Min: 0, Avg: 60.3, Max: 79, Diff: 79, Sum: 1086]? ? ? [Scan RS (ms): Min: 0.1, Avg: 2.2, Max: 2.7, Diff: 2.5, Sum: 40.4]? ? ? [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]? ? ? [Object Copy (ms): Min: 8.0, Avg: 10.2, Max: 10.5, Diff: 2.5, Sum: 183.2]? ? ? [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.5]? ? ? [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.0]? ? ? [GC Worker Total (ms): Min: 20.9, Avg: 21.2, Max: 21.4, Diff: 0.5, Sum: 381.0]? ? ? [GC Worker End (ms): Min: 549629590.9, Avg: 549629590.9, Max: 549629591.1, Diff: 0.2]?? [Code Root Fixup: 0.3 ms]?? [Code Root Purge: 0.0 ms]?? [Clear CT: 0.8 ms]?? [Other: 12.6 ms]? ? ? [Choose CSet: 0.0 ms]? ? ? [Ref Proc: 6.2 ms]? ? ? [Ref Enq: 0.5 ms]? ? ? [Redirty Cards: 0.4 ms]? ? ? [Humongous Reclaim: 0.0 ms]? ? ? [Free CSet: 2.9 ms]?? [Eden: 4896.0M(4896.0M)->0.0B(4892.0M) Survivors: 16.0M->20.0M Heap: 6505.9M(8192.0M)->1611.1M(8192.0M)]?[Times: user=0.38 sys=0.02, real=0.75 secs]?2015-10-05T14:54:36.272+0000: 549630.322: Total time for which application threads were stopped: 0.7695049 seconds, Stopping threads took: 0.0004596 seconds As you can see that young GC took around 35ms but the total time for which application threads were stopped is 769ms. Can anyone tell me what other pauses are included in this total time ?Also why pauses other then GC pauses were included in GC logs ?Earlier I was using CMS. But in CMS I never saw such differences in GC time and Application thread pause time. Can anyone tell me what is the reason behind such behaviour ? Thanks -Naman -------------- next part -------------- An HTML attachment was scrubbed... URL: From yu.zhang at oracle.com Tue Oct 6 15:30:31 2015 From: yu.zhang at oracle.com (Yu Zhang) Date: Tue, 6 Oct 2015 08:30:31 -0700 Subject: Huge difference in GC pause time and total application pause time. In-Reply-To: <287580904.383951.1444121889025.JavaMail.yahoo@mail.yahoo.com> References: <287580904.383951.1444121889025.JavaMail.yahoo@mail.yahoo.com> Message-ID: <5613E917.4080305@oracle.com> Naman, This entry is very strange, it has 0.0355102 sec for gc pause, but the time stamp at the bottom has [Times: user=0.38 sys=0.02, real=0.75 secs] The real time matches PrintGCApplicationStoppedTime. Since there are 18 gc threads, if not blocked by something, the real time should be smaller than user time. can you check if the system is swapping or waiting on i/o? Which JVM are you using? Thanks, Jenny On 10/6/2015 1:58 AM, Naman Jain wrote: > Hey Everyone, > > G1 GC log: > 2015-10-05T14:54:35.518+0000: 549629.568: [GC pause (G1 Evacuation > Pause) (young) > Desired survivor size 322961408 bytes, new threshold 15 (max 15) > - age 1: 5292232 bytes, 5292232 total > - age 2: 1201264 bytes, 6493496 total > - age 3: 377376 bytes, 6870872 total > - age 4: 440752 bytes, 7311624 total > - age 5: 201976 bytes, 7513600 total > - age 6: 77616 bytes, 7591216 total > - age 7: 28824 bytes, 7620040 total > - age 8: 55312 bytes, 7675352 total > - age 9: 55432 bytes, 7730784 total > - age 10: 14272 bytes, 7745056 total > - age 11: 16872 bytes, 7761928 total > - age 12: 18208 bytes, 7780136 total > - age 13: 11656 bytes, 7791792 total > - age 14: 15800 bytes, 7807592 total > - age 15: 33960 bytes, 7841552 total > 549629.569: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 14904, predicted base time: 20.98 ms, remaining time: > 19.02 ms, target pause time: 40.00 ms] > 549629.569: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 1224 regions, survivors: 4 regions, predicted young region > time: 10.73 ms] > 549629.569: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 1224 regions, survivors: 4 regions, old: 0 regions, predicted > pause time: 31.70 ms, target pause time: 40.00 ms] > , *0.0355102 secs*] > [Parallel Time: 21.7 ms, GC Workers: 18] > [GC Worker Start (ms): Min: 549629569.6, Avg: 549629569.8, Max: > 549629569.9, Diff: 0.3] > [Ext Root Scanning (ms): Min: 3.6, Avg: 4.3, Max: 12.7, Diff: > 9.0, Sum: 78.1] > [Update RS (ms): Min: 0.0, Avg: 4.1, Max: 4.7, Diff: 4.7, Sum: 74.6] > [Processed Buffers: Min: 0, Avg: 60.3, Max: 79, Diff: 79, Sum: 1086] > [Scan RS (ms): Min: 0.1, Avg: 2.2, Max: 2.7, Diff: 2.5, Sum: 40.4] > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.1] > [Object Copy (ms): Min: 8.0, Avg: 10.2, Max: 10.5, Diff: 2.5, > Sum: 183.2] > [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.5] > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, > Sum: 2.0] > [GC Worker Total (ms): Min: 20.9, Avg: 21.2, Max: 21.4, Diff: > 0.5, Sum: 381.0] > [GC Worker End (ms): Min: 549629590.9, Avg: 549629590.9, Max: > 549629591.1, Diff: 0.2] > [Code Root Fixup: 0.3 ms] > [Code Root Purge: 0.0 ms] > [Clear CT: 0.8 ms] > [Other: 12.6 ms] > [Choose CSet: 0.0 ms] > [Ref Proc: 6.2 ms] > [Ref Enq: 0.5 ms] > [Redirty Cards: 0.4 ms] > [Humongous Reclaim: 0.0 ms] > [Free CSet: 2.9 ms] > [Eden: 4896.0M(4896.0M)->0.0B(4892.0M) Survivors: 16.0M->20.0M > Heap: 6505.9M(8192.0M)->1611.1M(8192.0M)] > [Times: user=0.38 sys=0.02, real=0.75 secs] > 2015-10-05T14:54:36.272+0000: 549630.322: Total time for which > application threads were stopped:*0.7695049* seconds, Stopping threads > took: 0.0004596 seconds > > As you can see that young GC took around 35ms but the total time for > which application threads were stopped is 769ms. Can anyone tell me > what other pauses are included in this total time ? > Also why pauses other then GC pauses were included in GC logs ? > Earlier I was using CMS. But in CMS I never saw such differences in GC > time and Application thread pause time. Can anyone tell me what is the > reason behind such behaviour ? > > Thanks > > -Naman > > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: From naman.jain1894 at yahoo.in Tue Oct 6 16:26:10 2015 From: naman.jain1894 at yahoo.in (Naman Jain) Date: Tue, 6 Oct 2015 16:26:10 +0000 (UTC) Subject: Huge difference in GC pause time and total application pause time. In-Reply-To: <5613E917.4080305@oracle.com> References: <5613E917.4080305@oracle.com> Message-ID: <1166625427.1248273.1444148770333.JavaMail.yahoo@mail.yahoo.com> Hey Jenny, I checked the sar files, there was no swapping or waiting on i/o.The jvm I am using is:Java HotSpot(TM) 64-Bit Server VM (25.40-b25) for linux-amd64 JRE (1.8.0_40-b26)Can you tell me any other reason which might have caused this ?Also, does safepoint have anything do with this ?Thanks? Naman On Tuesday, 6 October 2015 9:00 PM, Yu Zhang wrote: Naman, This entry is very strange, it has 0.0355102 sec for gc pause, but the time stamp at the bottom has ?[Times: user=0.38 sys=0.02, real=0.75 secs] The real time matches PrintGCApplicationStoppedTime. Since there are 18 gc threads, if not blocked by something, the real time should be smaller than user time. can you check if the system is swapping or waiting on i/o? Which JVM are you using? Thanks, Jenny On 10/6/2015 1:58 AM, Naman Jain wrote: Hey Everyone, G1 GC log: 2015-10-05T14:54:35.518+0000: 549629.568: [GC pause (G1 Evacuation Pause) (young) Desired survivor size 322961408 bytes, new threshold 15 (max 15) - age ? 1:? ? 5292232 bytes,? ? 5292232 total - age ? 2:? ? 1201264 bytes,? ? 6493496 total - age ? 3: ? ? 377376 bytes,? ? 6870872 total - age ? 4: ? ? 440752 bytes,? ? 7311624 total - age ? 5: ? ? 201976 bytes,? ? 7513600 total - age ? 6:? ? ? 77616 bytes,? ? 7591216 total - age ? 7:? ? ? 28824 bytes,? ? 7620040 total - age ? 8:? ? ? 55312 bytes,? ? 7675352 total - age ? 9:? ? ? 55432 bytes,? ? 7730784 total - age? 10:? ? ? 14272 bytes,? ? 7745056 total - age? 11:? ? ? 16872 bytes,? ? 7761928 total - age? 12:? ? ? 18208 bytes,? ? 7780136 total - age? 13:? ? ? 11656 bytes,? ? 7791792 total - age? 14:? ? ? 15800 bytes,? ? 7807592 total - age? 15:? ? ? 33960 bytes,? ? 7841552 total ?549629.569: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 14904, predicted base time: 20.98 ms, remaining time: 19.02 ms, target pause time: 40.00 ms] ?549629.569: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 1224 regions, survivors: 4 regions, predicted young region time: 10.73 ms] ?549629.569: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 1224 regions, survivors: 4 regions, old: 0 regions, predicted pause time: 31.70 ms, target pause time: 40.00 ms] , 0.0355102 secs] ?? [Parallel Time: 21.7 ms, GC Workers: 18] ? ? ? [GC Worker Start (ms): Min: 549629569.6, Avg: 549629569.8, Max: 549629569.9, Diff: 0.3] ? ? ? [Ext Root Scanning (ms): Min: 3.6, Avg: 4.3, Max: 12.7, Diff: 9.0, Sum: 78.1] ? ? ? [Update RS (ms): Min: 0.0, Avg: 4.1, Max: 4.7, Diff: 4.7, Sum: 74.6] ?? ? ? ? [Processed Buffers: Min: 0, Avg: 60.3, Max: 79, Diff: 79, Sum: 1086] ? ? ? [Scan RS (ms): Min: 0.1, Avg: 2.2, Max: 2.7, Diff: 2.5, Sum: 40.4] ? ? ? [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] ? ? ? [Object Copy (ms): Min: 8.0, Avg: 10.2, Max: 10.5, Diff: 2.5, Sum: 183.2] ? ? ? [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.5] ? ? ? [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.0] ? ? ? [GC Worker Total (ms): Min: 20.9, Avg: 21.2, Max: 21.4, Diff: 0.5, Sum: 381.0] ? ? ? [GC Worker End (ms): Min: 549629590.9, Avg: 549629590.9, Max: 549629591.1, Diff: 0.2] ?? [Code Root Fixup: 0.3 ms] ?? [Code Root Purge: 0.0 ms] ?? [Clear CT: 0.8 ms] ?? [Other: 12.6 ms] ? ? ? [Choose CSet: 0.0 ms] ? ? ? [Ref Proc: 6.2 ms] ? ? ? [Ref Enq: 0.5 ms] ? ? ? [Redirty Cards: 0.4 ms] ? ? ? [Humongous Reclaim: 0.0 ms] ? ? ? [Free CSet: 2.9 ms] ?? [Eden: 4896.0M(4896.0M)->0.0B(4892.0M) Survivors: 16.0M->20.0M Heap: 6505.9M(8192.0M)->1611.1M(8192.0M)] ?[Times: user=0.38 sys=0.02, real=0.75 secs]? 2015-10-05T14:54:36.272+0000: 549630.322: Total time for which application threads were stopped: 0.7695049 seconds, Stopping threads took: 0.0004596 seconds As you can see that young GC took around 35ms but the total time for which application threads were stopped is 769ms. Can anyone tell me what other pauses are included in this total time ? Also why pauses other then GC pauses were included in GC logs ? Earlier I was using CMS. But in CMS I never saw such differences in GC time and Application thread pause time. Can anyone tell me what is the reason behind such behaviour ? Thanks -Naman _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: From naman.jain1894 at yahoo.in Tue Oct 6 16:32:53 2015 From: naman.jain1894 at yahoo.in (Naman Jain) Date: Tue, 6 Oct 2015 16:32:53 +0000 (UTC) Subject: Huge difference in GC pause time and total application pause time. In-Reply-To: <1166625427.1248273.1444148770333.JavaMail.yahoo@mail.yahoo.com> References: <5613E917.4080305@oracle.com> <1166625427.1248273.1444148770333.JavaMail.yahoo@mail.yahoo.com> Message-ID: <1234701443.1247746.1444149173316.JavaMail.yahoo@mail.yahoo.com> Hey jenny, JVM version used is:java version "1.8.0_40"Java(TM) SE Runtime Environment (build 1.8.0_40-b26)Java HotSpot(TM) 64-Bit Server VM (build 25.40-b25, mixed mode) -Naman On Tuesday, 6 October 2015 9:56 PM, Naman Jain wrote: Hey Jenny, I checked the sar files, there was no swapping or waiting on i/o.The jvm I am using is:Java HotSpot(TM) 64-Bit Server VM (25.40-b25) for linux-amd64 JRE (1.8.0_40-b26)Can you tell me any other reason which might have caused this ?Also, does safepoint have anything do with this ?Thanks? Naman On Tuesday, 6 October 2015 9:00 PM, Yu Zhang wrote: Naman, This entry is very strange, it has 0.0355102 sec for gc pause, but the time stamp at the bottom has ?[Times: user=0.38 sys=0.02, real=0.75 secs] The real time matches PrintGCApplicationStoppedTime. Since there are 18 gc threads, if not blocked by something, the real time should be smaller than user time. can you check if the system is swapping or waiting on i/o? Which JVM are you using? Thanks, Jenny On 10/6/2015 1:58 AM, Naman Jain wrote: Hey Everyone, G1 GC log: 2015-10-05T14:54:35.518+0000: 549629.568: [GC pause (G1 Evacuation Pause) (young) Desired survivor size 322961408 bytes, new threshold 15 (max 15) - age ? 1:? ? 5292232 bytes,? ? 5292232 total - age ? 2:? ? 1201264 bytes,? ? 6493496 total - age ? 3: ? ? 377376 bytes,? ? 6870872 total - age ? 4: ? ? 440752 bytes,? ? 7311624 total - age ? 5: ? ? 201976 bytes,? ? 7513600 total - age ? 6:? ? ? 77616 bytes,? ? 7591216 total - age ? 7:? ? ? 28824 bytes,? ? 7620040 total - age ? 8:? ? ? 55312 bytes,? ? 7675352 total - age ? 9:? ? ? 55432 bytes,? ? 7730784 total - age? 10:? ? ? 14272 bytes,? ? 7745056 total - age? 11:? ? ? 16872 bytes,? ? 7761928 total - age? 12:? ? ? 18208 bytes,? ? 7780136 total - age? 13:? ? ? 11656 bytes,? ? 7791792 total - age? 14:? ? ? 15800 bytes,? ? 7807592 total - age? 15:? ? ? 33960 bytes,? ? 7841552 total ?549629.569: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 14904, predicted base time: 20.98 ms, remaining time: 19.02 ms, target pause time: 40.00 ms] ?549629.569: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 1224 regions, survivors: 4 regions, predicted young region time: 10.73 ms] ?549629.569: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 1224 regions, survivors: 4 regions, old: 0 regions, predicted pause time: 31.70 ms, target pause time: 40.00 ms] , 0.0355102 secs] ?? [Parallel Time: 21.7 ms, GC Workers: 18] ? ? ? [GC Worker Start (ms): Min: 549629569.6, Avg: 549629569.8, Max: 549629569.9, Diff: 0.3] ? ? ? [Ext Root Scanning (ms): Min: 3.6, Avg: 4.3, Max: 12.7, Diff: 9.0, Sum: 78.1] ? ? ? [Update RS (ms): Min: 0.0, Avg: 4.1, Max: 4.7, Diff: 4.7, Sum: 74.6] ?? ? ? ? [Processed Buffers: Min: 0, Avg: 60.3, Max: 79, Diff: 79, Sum: 1086] ? ? ? [Scan RS (ms): Min: 0.1, Avg: 2.2, Max: 2.7, Diff: 2.5, Sum: 40.4] ? ? ? [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] ? ? ? [Object Copy (ms): Min: 8.0, Avg: 10.2, Max: 10.5, Diff: 2.5, Sum: 183.2] ? ? ? [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.5] ? ? ? [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.0] ? ? ? [GC Worker Total (ms): Min: 20.9, Avg: 21.2, Max: 21.4, Diff: 0.5, Sum: 381.0] ? ? ? [GC Worker End (ms): Min: 549629590.9, Avg: 549629590.9, Max: 549629591.1, Diff: 0.2] ?? [Code Root Fixup: 0.3 ms] ?? [Code Root Purge: 0.0 ms] ?? [Clear CT: 0.8 ms] ?? [Other: 12.6 ms] ? ? ? [Choose CSet: 0.0 ms] ? ? ? [Ref Proc: 6.2 ms] ? ? ? [Ref Enq: 0.5 ms] ? ? ? [Redirty Cards: 0.4 ms] ? ? ? [Humongous Reclaim: 0.0 ms] ? ? ? [Free CSet: 2.9 ms] ?? [Eden: 4896.0M(4896.0M)->0.0B(4892.0M) Survivors: 16.0M->20.0M Heap: 6505.9M(8192.0M)->1611.1M(8192.0M)] ?[Times: user=0.38 sys=0.02, real=0.75 secs]? 2015-10-05T14:54:36.272+0000: 549630.322: Total time for which application threads were stopped: 0.7695049 seconds, Stopping threads took: 0.0004596 seconds As you can see that young GC took around 35ms but the total time for which application threads were stopped is 769ms. Can anyone tell me what other pauses are included in this total time ? Also why pauses other then GC pauses were included in GC logs ? Earlier I was using CMS. But in CMS I never saw such differences in GC time and Application thread pause time. Can anyone tell me what is the reason behind such behaviour ? Thanks -Naman _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: From yu.zhang at oracle.com Tue Oct 6 17:52:27 2015 From: yu.zhang at oracle.com (Yu Zhang) Date: Tue, 6 Oct 2015 10:52:27 -0700 Subject: Huge difference in GC pause time and total application pause time. In-Reply-To: <1234701443.1247746.1444149173316.JavaMail.yahoo@mail.yahoo.com> References: <5613E917.4080305@oracle.com> <1166625427.1248273.1444148770333.JavaMail.yahoo@mail.yahoo.com> <1234701443.1247746.1444149173316.JavaMail.yahoo@mail.yahoo.com> Message-ID: <56140A5B.5090809@oracle.com> Naman, It seems to me not related to safe point. Because the entry has [Times: user=0.38 sys=0.02, real=0.75 secs] indicating it is 0.75 sec when collection at safe point is done. What I do not understand is the time 0.0355102 sec is printed after the evacuation is done. The only difference between 0.0355102 and last entry is some book keeping stuff. But we are using much more user and real time. The big difference between real and user time means the gc threads can not be scheduled to get cpu. But no explanation for 0.0355102 vs 0.38 sec. Do you see this happen a lot? Is your system cpu saturated? Thanks, Jenny On 10/6/2015 9:32 AM, Naman Jain wrote: > Hey jenny, > > JVM version used is: > java version "1.8.0_40" > Java(TM) SE Runtime Environment (build 1.8.0_40-b26) > Java HotSpot(TM) 64-Bit Server VM (build 25.40-b25, mixed mode) > > -Naman > > > > On Tuesday, 6 October 2015 9:56 PM, Naman Jain > wrote: > > > Hey Jenny, > > I checked the sar files, there was no swapping or waiting on i/o. > The jvm I am using is: > Java HotSpot(TM) 64-Bit Server VM (25.40-b25) for linux-amd64 JRE > (1.8.0_40-b26) > Can you tell me any other reason which might have caused this ? > Also, does safepoint have anything do with this ? > Thanks > > Naman > > > > On Tuesday, 6 October 2015 9:00 PM, Yu Zhang wrote: > > > Naman, > > This entry is very strange, > it has 0.0355102 sec for gc pause, but the time stamp at the bottom has > [Times: user=0.38 sys=0.02, real=0.75 secs] > The real time matches PrintGCApplicationStoppedTime. > Since there are 18 gc threads, if not blocked by something, the real > time should be smaller than user time. > can you check if the system is swapping or waiting on i/o? > Which JVM are you using? > > Thanks, > Jenny > On 10/6/2015 1:58 AM, Naman Jain wrote: >> Hey Everyone, >> >> G1 GC log: >> 2015-10-05T14:54:35.518+0000: 549629.568: [GC pause (G1 Evacuation >> Pause) (young) >> Desired survivor size 322961408 bytes, new threshold 15 (max 15) >> - age 1: 5292232 bytes, 5292232 total >> - age 2: 1201264 bytes, 6493496 total >> - age 3: 377376 bytes, 6870872 total >> - age 4: 440752 bytes, 7311624 total >> - age 5: 201976 bytes, 7513600 total >> - age 6: 77616 bytes, 7591216 total >> - age 7: 28824 bytes, 7620040 total >> - age 8: 55312 bytes, 7675352 total >> - age 9: 55432 bytes, 7730784 total >> - age 10: 14272 bytes, 7745056 total >> - age 11: 16872 bytes, 7761928 total >> - age 12: 18208 bytes, 7780136 total >> - age 13: 11656 bytes, 7791792 total >> - age 14: 15800 bytes, 7807592 total >> - age 15: 33960 bytes, 7841552 total >> 549629.569: [G1Ergonomics (CSet Construction) start choosing CSet, >> _pending_cards: 14904, predicted base time: 20.98 ms, remaining time: >> 19.02 ms, target pause time: 40.00 ms] >> 549629.569: [G1Ergonomics (CSet Construction) add young regions to >> CSet, eden: 1224 regions, survivors: 4 regions, predicted young >> region time: 10.73 ms] >> 549629.569: [G1Ergonomics (CSet Construction) finish choosing CSet, >> eden: 1224 regions, survivors: 4 regions, old: 0 regions, predicted >> pause time: 31.70 ms, target pause time: 40.00 ms] >> , *0.0355102 secs*] >> [Parallel Time: 21.7 ms, GC Workers: 18] >> [GC Worker Start (ms): Min: 549629569.6, Avg: 549629569.8, Max: >> 549629569.9, Diff: 0.3] >> [Ext Root Scanning (ms): Min: 3.6, Avg: 4.3, Max: 12.7, Diff: >> 9.0, Sum: 78.1] >> [Update RS (ms): Min: 0.0, Avg: 4.1, Max: 4.7, Diff: 4.7, Sum: 74.6] >> [Processed Buffers: Min: 0, Avg: 60.3, Max: 79, Diff: 79, Sum: >> 1086] >> [Scan RS (ms): Min: 0.1, Avg: 2.2, Max: 2.7, Diff: 2.5, Sum: 40.4] >> [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: >> 0.0, Sum: 0.1] >> [Object Copy (ms): Min: 8.0, Avg: 10.2, Max: 10.5, Diff: 2.5, >> Sum: 183.2] >> [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.5] >> [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, >> Sum: 2.0] >> [GC Worker Total (ms): Min: 20.9, Avg: 21.2, Max: 21.4, Diff: >> 0.5, Sum: 381.0] >> [GC Worker End (ms): Min: 549629590.9, Avg: 549629590.9, Max: >> 549629591.1, Diff: 0.2] >> [Code Root Fixup: 0.3 ms] >> [Code Root Purge: 0.0 ms] >> [Clear CT: 0.8 ms] >> [Other: 12.6 ms] >> [Choose CSet: 0.0 ms] >> [Ref Proc: 6.2 ms] >> [Ref Enq: 0.5 ms] >> [Redirty Cards: 0.4 ms] >> [Humongous Reclaim: 0.0 ms] >> [Free CSet: 2.9 ms] >> [Eden: 4896.0M(4896.0M)->0.0B(4892.0M) Survivors: 16.0M->20.0M Heap: >> 6505.9M(8192.0M)->1611.1M(8192.0M)] >> [Times: user=0.38 sys=0.02, real=0.75 secs] >> 2015-10-05T14:54:36.272+0000: 549630.322: Total time for which >> application threads were stopped:*0.7695049* seconds, Stopping >> threads took: 0.0004596 seconds >> >> As you can see that young GC took around 35ms but the total time for >> which application threads were stopped is 769ms. Can anyone tell me >> what other pauses are included in this total time ? >> Also why pauses other then GC pauses were included in GC logs ? >> Earlier I was using CMS. But in CMS I never saw such differences in >> GC time and Application thread pause time. Can anyone tell me what is >> the reason behind such behaviour ? >> >> Thanks >> >> -Naman >> >> >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yu.zhang at oracle.com Tue Oct 6 18:19:10 2015 From: yu.zhang at oracle.com (Yu Zhang) Date: Tue, 6 Oct 2015 11:19:10 -0700 Subject: Huge difference in GC pause time and total application pause time. In-Reply-To: <56140A5B.5090809@oracle.com> References: <5613E917.4080305@oracle.com> <1166625427.1248273.1444148770333.JavaMail.yahoo@mail.yahoo.com> <1234701443.1247746.1444149173316.JavaMail.yahoo@mail.yahoo.com> <56140A5B.5090809@oracle.com> Message-ID: <5614109E.6020807@oracle.com> Naman, Are you on Linux? Do you have Transparent Huge Page on from the system level? Thanks, Jenny On 10/6/2015 10:52 AM, Yu Zhang wrote: > Naman, > > It seems to me not related to safe point. Because the entry has > [Times: user=0.38 sys=0.02, real=0.75 secs] > indicating it is 0.75 sec when collection at safe point is done. > What I do not understand is the time 0.0355102 sec is printed after > the evacuation is done. The only difference between 0.0355102 and last > entry is some book keeping stuff. But we are using much more user and > real time. The big difference between real and user time means the gc > threads can not be scheduled to get cpu. But no explanation for > 0.0355102 vs 0.38 sec. > > Do you see this happen a lot? Is your system cpu saturated? > > Thanks, > Jenny > On 10/6/2015 9:32 AM, Naman Jain wrote: >> Hey jenny, >> >> JVM version used is: >> java version "1.8.0_40" >> Java(TM) SE Runtime Environment (build 1.8.0_40-b26) >> Java HotSpot(TM) 64-Bit Server VM (build 25.40-b25, mixed mode) >> >> -Naman >> >> >> >> On Tuesday, 6 October 2015 9:56 PM, Naman Jain >> wrote: >> >> >> Hey Jenny, >> >> I checked the sar files, there was no swapping or waiting on i/o. >> The jvm I am using is: >> Java HotSpot(TM) 64-Bit Server VM (25.40-b25) for linux-amd64 JRE >> (1.8.0_40-b26) >> Can you tell me any other reason which might have caused this ? >> Also, does safepoint have anything do with this ? >> Thanks >> >> Naman >> >> >> >> On Tuesday, 6 October 2015 9:00 PM, Yu Zhang wrote: >> >> >> Naman, >> >> This entry is very strange, >> it has 0.0355102 sec for gc pause, but the time stamp at the bottom has >> [Times: user=0.38 sys=0.02, real=0.75 secs] >> The real time matches PrintGCApplicationStoppedTime. >> Since there are 18 gc threads, if not blocked by something, the real >> time should be smaller than user time. >> can you check if the system is swapping or waiting on i/o? >> Which JVM are you using? >> >> Thanks, >> Jenny >> On 10/6/2015 1:58 AM, Naman Jain wrote: >>> Hey Everyone, >>> >>> G1 GC log: >>> 2015-10-05T14:54:35.518+0000: 549629.568: [GC pause (G1 Evacuation >>> Pause) (young) >>> Desired survivor size 322961408 bytes, new threshold 15 (max 15) >>> - age 1: 5292232 bytes, 5292232 total >>> - age 2: 1201264 bytes, 6493496 total >>> - age 3: 377376 bytes, 6870872 total >>> - age 4: 440752 bytes, 7311624 total >>> - age 5: 201976 bytes, 7513600 total >>> - age 6: 77616 bytes, 7591216 total >>> - age 7: 28824 bytes, 7620040 total >>> - age 8: 55312 bytes, 7675352 total >>> - age 9: 55432 bytes, 7730784 total >>> - age 10: 14272 bytes, 7745056 total >>> - age 11: 16872 bytes, 7761928 total >>> - age 12: 18208 bytes, 7780136 total >>> - age 13: 11656 bytes, 7791792 total >>> - age 14: 15800 bytes, 7807592 total >>> - age 15: 33960 bytes, 7841552 total >>> 549629.569: [G1Ergonomics (CSet Construction) start choosing CSet, >>> _pending_cards: 14904, predicted base time: 20.98 ms, remaining >>> time: 19.02 ms, target pause time: 40.00 ms] >>> 549629.569: [G1Ergonomics (CSet Construction) add young regions to >>> CSet, eden: 1224 regions, survivors: 4 regions, predicted young >>> region time: 10.73 ms] >>> 549629.569: [G1Ergonomics (CSet Construction) finish choosing CSet, >>> eden: 1224 regions, survivors: 4 regions, old: 0 regions, predicted >>> pause time: 31.70 ms, target pause time: 40.00 ms] >>> , *0.0355102 secs*] >>> [Parallel Time: 21.7 ms, GC Workers: 18] >>> [GC Worker Start (ms): Min: 549629569.6, Avg: 549629569.8, Max: >>> 549629569.9, Diff: 0.3] >>> [Ext Root Scanning (ms): Min: 3.6, Avg: 4.3, Max: 12.7, Diff: >>> 9.0, Sum: 78.1] >>> [Update RS (ms): Min: 0.0, Avg: 4.1, Max: 4.7, Diff: 4.7, Sum: 74.6] >>> [Processed Buffers: Min: 0, Avg: 60.3, Max: 79, Diff: 79, Sum: >>> 1086] >>> [Scan RS (ms): Min: 0.1, Avg: 2.2, Max: 2.7, Diff: 2.5, Sum: 40.4] >>> [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: >>> 0.0, Sum: 0.1] >>> [Object Copy (ms): Min: 8.0, Avg: 10.2, Max: 10.5, Diff: 2.5, >>> Sum: 183.2] >>> [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: >>> 2.5] >>> [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, >>> Sum: 2.0] >>> [GC Worker Total (ms): Min: 20.9, Avg: 21.2, Max: 21.4, Diff: >>> 0.5, Sum: 381.0] >>> [GC Worker End (ms): Min: 549629590.9, Avg: 549629590.9, Max: >>> 549629591.1, Diff: 0.2] >>> [Code Root Fixup: 0.3 ms] >>> [Code Root Purge: 0.0 ms] >>> [Clear CT: 0.8 ms] >>> [Other: 12.6 ms] >>> [Choose CSet: 0.0 ms] >>> [Ref Proc: 6.2 ms] >>> [Ref Enq: 0.5 ms] >>> [Redirty Cards: 0.4 ms] >>> [Humongous Reclaim: 0.0 ms] >>> [Free CSet: 2.9 ms] >>> [Eden: 4896.0M(4896.0M)->0.0B(4892.0M) Survivors: 16.0M->20.0M Heap: >>> 6505.9M(8192.0M)->1611.1M(8192.0M)] >>> [Times: user=0.38 sys=0.02, real=0.75 secs] >>> 2015-10-05T14:54:36.272+0000: 549630.322: Total time for which >>> application threads were stopped:*0.7695049* seconds, Stopping >>> threads took: 0.0004596 seconds >>> >>> As you can see that young GC took around 35ms but the total time for >>> which application threads were stopped is 769ms. Can anyone tell me >>> what other pauses are included in this total time ? >>> Also why pauses other then GC pauses were included in GC logs ? >>> Earlier I was using CMS. But in CMS I never saw such differences in >>> GC time and Application thread pause time. Can anyone tell me what >>> is the reason behind such behaviour ? >>> >>> Thanks >>> >>> -Naman >>> >>> >>> >>> _______________________________________________ >>> hotspot-gc-use mailing list >>> hotspot-gc-use at openjdk.java.net >>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> >> >> >> > > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: From kjkoster at gmail.com Tue Oct 6 18:43:45 2015 From: kjkoster at gmail.com (Kees Jan Koster) Date: Tue, 6 Oct 2015 20:43:45 +0200 Subject: Huge difference in GC pause time and total application pause time. In-Reply-To: <5613E917.4080305@oracle.com> References: <287580904.383951.1444121889025.JavaMail.yahoo@mail.yahoo.com> <5613E917.4080305@oracle.com> Message-ID: <9825E7B9-B8C4-44B5-AD44-44664AA291F9@gmail.com> Dear Naman, Is this JVM running on a virtual machine or on the bare iron? I have seen strange pauses in virtual machines where the host had memory overcommit, i.e. it had more memory allocated to guests than it had physical RAM. Not saying this is happening for you, just another avenue to explore. Kees Jan > On 6 Oct 2015, at 17:30, Yu Zhang wrote: > > Naman, > > This entry is very strange, > it has 0.0355102 sec for gc pause, but the time stamp at the bottom has > [Times: user=0.38 sys=0.02, real=0.75 secs] > The real time matches PrintGCApplicationStoppedTime. > Since there are 18 gc threads, if not blocked by something, the real time should be smaller than user time. > can you check if the system is swapping or waiting on i/o? > Which JVM are you using? > > > Thanks, > Jenny > > On 10/6/2015 1:58 AM, Naman Jain wrote: >> Hey Everyone, >> >> G1 GC log: >> 2015-10-05T14:54:35.518+0000: 549629.568: [GC pause (G1 Evacuation Pause) (young) >> Desired survivor size 322961408 bytes, new threshold 15 (max 15) >> - age 1: 5292232 bytes, 5292232 total >> - age 2: 1201264 bytes, 6493496 total >> - age 3: 377376 bytes, 6870872 total >> - age 4: 440752 bytes, 7311624 total >> - age 5: 201976 bytes, 7513600 total >> - age 6: 77616 bytes, 7591216 total >> - age 7: 28824 bytes, 7620040 total >> - age 8: 55312 bytes, 7675352 total >> - age 9: 55432 bytes, 7730784 total >> - age 10: 14272 bytes, 7745056 total >> - age 11: 16872 bytes, 7761928 total >> - age 12: 18208 bytes, 7780136 total >> - age 13: 11656 bytes, 7791792 total >> - age 14: 15800 bytes, 7807592 total >> - age 15: 33960 bytes, 7841552 total >> 549629.569: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 14904, predicted base time: 20.98 ms, remaining time: 19.02 ms, target pause time: 40.00 ms] >> 549629.569: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 1224 regions, survivors: 4 regions, predicted young region time: 10.73 ms] >> 549629.569: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 1224 regions, survivors: 4 regions, old: 0 regions, predicted pause time: 31.70 ms, target pause time: 40.00 ms] >> , 0.0355102 secs] >> [Parallel Time: 21.7 ms, GC Workers: 18] >> [GC Worker Start (ms): Min: 549629569.6, Avg: 549629569.8, Max: 549629569.9, Diff: 0.3] >> [Ext Root Scanning (ms): Min: 3.6, Avg: 4.3, Max: 12.7, Diff: 9.0, Sum: 78.1] >> [Update RS (ms): Min: 0.0, Avg: 4.1, Max: 4.7, Diff: 4.7, Sum: 74.6] >> [Processed Buffers: Min: 0, Avg: 60.3, Max: 79, Diff: 79, Sum: 1086] >> [Scan RS (ms): Min: 0.1, Avg: 2.2, Max: 2.7, Diff: 2.5, Sum: 40.4] >> [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] >> [Object Copy (ms): Min: 8.0, Avg: 10.2, Max: 10.5, Diff: 2.5, Sum: 183.2] >> [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.5] >> [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.0] >> [GC Worker Total (ms): Min: 20.9, Avg: 21.2, Max: 21.4, Diff: 0.5, Sum: 381.0] >> [GC Worker End (ms): Min: 549629590.9, Avg: 549629590.9, Max: 549629591.1, Diff: 0.2] >> [Code Root Fixup: 0.3 ms] >> [Code Root Purge: 0.0 ms] >> [Clear CT: 0.8 ms] >> [Other: 12.6 ms] >> [Choose CSet: 0.0 ms] >> [Ref Proc: 6.2 ms] >> [Ref Enq: 0.5 ms] >> [Redirty Cards: 0.4 ms] >> [Humongous Reclaim: 0.0 ms] >> [Free CSet: 2.9 ms] >> [Eden: 4896.0M(4896.0M)->0.0B(4892.0M) Survivors: 16.0M->20.0M Heap: 6505.9M(8192.0M)->1611.1M(8192.0M)] >> [Times: user=0.38 sys=0.02, real=0.75 secs] >> 2015-10-05T14:54:36.272+0000: 549630.322: Total time for which application threads were stopped: 0.7695049 seconds, Stopping threads took: 0.0004596 seconds >> >> As you can see that young GC took around 35ms but the total time for which application threads were stopped is 769ms. Can anyone tell me what other pauses are included in this total time ? >> Also why pauses other then GC pauses were included in GC logs ? >> Earlier I was using CMS. But in CMS I never saw such differences in GC time and Application thread pause time. Can anyone tell me what is the reason behind such behaviour ? >> >> Thanks >> >> -Naman >> >> >> >> _______________________________________________ >> hotspot-gc-use mailing list >> >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -- Kees Jan http://java-monitor.com/ kjkoster at kjkoster.org +31651838192 Human beings make life so interesting. Do you know that in a universe so full of wonders, they have managed to invent boredom. Quite astonishing... -- Terry Pratchett -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 455 bytes Desc: Message signed with OpenPGP using GPGMail URL: From naman.jain1894 at yahoo.in Wed Oct 7 07:09:49 2015 From: naman.jain1894 at yahoo.in (Naman Jain) Date: Wed, 7 Oct 2015 07:09:49 +0000 (UTC) Subject: Huge difference in GC pause time and total application pause time. In-Reply-To: <9825E7B9-B8C4-44B5-AD44-44664AA291F9@gmail.com> References: <9825E7B9-B8C4-44B5-AD44-44664AA291F9@gmail.com> Message-ID: <127321610.163750.1444201789382.JavaMail.yahoo@mail.yahoo.com> Hey Kees, It is running on bare iron.?Do you have any other reason for such behaviour? Thanks?Naman On Wednesday, 7 October 2015 12:13 AM, Kees Jan Koster wrote: Dear Naman, Is this JVM running on a virtual machine or on the bare iron? I have seen strange pauses in virtual machines where the host had memory overcommit, i.e. it had more memory allocated to guests than it had physical RAM. Not saying this is happening for you, just another avenue to explore. Kees Jan > On 6 Oct 2015, at 17:30, Yu Zhang wrote: > > Naman, > > This entry is very strange, > it has 0.0355102 sec for gc pause, but the time stamp at the bottom has >? [Times: user=0.38 sys=0.02, real=0.75 secs] > The real time matches PrintGCApplicationStoppedTime. > Since there are 18 gc threads, if not blocked by something, the real time should be smaller than user time. > can you check if the system is swapping or waiting on i/o? > Which JVM are you using? > > > Thanks, > Jenny > > On 10/6/2015 1:58 AM, Naman Jain wrote: >> Hey Everyone, >> >> G1 GC log: >> 2015-10-05T14:54:35.518+0000: 549629.568: [GC pause (G1 Evacuation Pause) (young) >> Desired survivor size 322961408 bytes, new threshold 15 (max 15) >> - age? 1:? ? 5292232 bytes,? ? 5292232 total >> - age? 2:? ? 1201264 bytes,? ? 6493496 total >> - age? 3:? ? 377376 bytes,? ? 6870872 total >> - age? 4:? ? 440752 bytes,? ? 7311624 total >> - age? 5:? ? 201976 bytes,? ? 7513600 total >> - age? 6:? ? ? 77616 bytes,? ? 7591216 total >> - age? 7:? ? ? 28824 bytes,? ? 7620040 total >> - age? 8:? ? ? 55312 bytes,? ? 7675352 total >> - age? 9:? ? ? 55432 bytes,? ? 7730784 total >> - age? 10:? ? ? 14272 bytes,? ? 7745056 total >> - age? 11:? ? ? 16872 bytes,? ? 7761928 total >> - age? 12:? ? ? 18208 bytes,? ? 7780136 total >> - age? 13:? ? ? 11656 bytes,? ? 7791792 total >> - age? 14:? ? ? 15800 bytes,? ? 7807592 total >> - age? 15:? ? ? 33960 bytes,? ? 7841552 total >>? 549629.569: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 14904, predicted base time: 20.98 ms, remaining time: 19.02 ms, target pause time: 40.00 ms] >>? 549629.569: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 1224 regions, survivors: 4 regions, predicted young region time: 10.73 ms] >>? 549629.569: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 1224 regions, survivors: 4 regions, old: 0 regions, predicted pause time: 31.70 ms, target pause time: 40.00 ms] >> , 0.0355102 secs] >>? ? [Parallel Time: 21.7 ms, GC Workers: 18] >>? ? ? [GC Worker Start (ms): Min: 549629569.6, Avg: 549629569.8, Max: 549629569.9, Diff: 0.3] >>? ? ? [Ext Root Scanning (ms): Min: 3.6, Avg: 4.3, Max: 12.7, Diff: 9.0, Sum: 78.1] >>? ? ? [Update RS (ms): Min: 0.0, Avg: 4.1, Max: 4.7, Diff: 4.7, Sum: 74.6] >>? ? ? ? ? [Processed Buffers: Min: 0, Avg: 60.3, Max: 79, Diff: 79, Sum: 1086] >>? ? ? [Scan RS (ms): Min: 0.1, Avg: 2.2, Max: 2.7, Diff: 2.5, Sum: 40.4] >>? ? ? [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] >>? ? ? [Object Copy (ms): Min: 8.0, Avg: 10.2, Max: 10.5, Diff: 2.5, Sum: 183.2] >>? ? ? [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.5] >>? ? ? [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.0] >>? ? ? [GC Worker Total (ms): Min: 20.9, Avg: 21.2, Max: 21.4, Diff: 0.5, Sum: 381.0] >>? ? ? [GC Worker End (ms): Min: 549629590.9, Avg: 549629590.9, Max: 549629591.1, Diff: 0.2] >>? ? [Code Root Fixup: 0.3 ms] >>? ? [Code Root Purge: 0.0 ms] >>? ? [Clear CT: 0.8 ms] >>? ? [Other: 12.6 ms] >>? ? ? [Choose CSet: 0.0 ms] >>? ? ? [Ref Proc: 6.2 ms] >>? ? ? [Ref Enq: 0.5 ms] >>? ? ? [Redirty Cards: 0.4 ms] >>? ? ? [Humongous Reclaim: 0.0 ms] >>? ? ? [Free CSet: 2.9 ms] >>? ? [Eden: 4896.0M(4896.0M)->0.0B(4892.0M) Survivors: 16.0M->20.0M Heap: 6505.9M(8192.0M)->1611.1M(8192.0M)] >>? [Times: user=0.38 sys=0.02, real=0.75 secs] >> 2015-10-05T14:54:36.272+0000: 549630.322: Total time for which application threads were stopped: 0.7695049 seconds, Stopping threads took: 0.0004596 seconds >> >> As you can see that young GC took around 35ms but the total time for which application threads were stopped is 769ms. Can anyone tell me what other pauses are included in this total time ? >> Also why pauses other then GC pauses were included in GC logs ? >> Earlier I was using CMS. But in CMS I never saw such differences in GC time and Application thread pause time. Can anyone tell me what is the reason behind such behaviour ? >> >> Thanks >> >> -Naman >> >> >> >> _______________________________________________ >> hotspot-gc-use mailing list >> >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -- Kees Jan http://java-monitor.com/ kjkoster at kjkoster.org +31651838192 Human beings make life so interesting. Do you know that in a universe so full of wonders, they have managed to invent boredom. Quite astonishing... -- Terry Pratchett -------------- next part -------------- An HTML attachment was scrubbed... URL: From naman.jain1894 at yahoo.in Wed Oct 7 07:11:15 2015 From: naman.jain1894 at yahoo.in (Naman Jain) Date: Wed, 7 Oct 2015 07:11:15 +0000 (UTC) Subject: Huge difference in GC pause time and total application pause time. In-Reply-To: <5614109E.6020807@oracle.com> References: <5614109E.6020807@oracle.com> Message-ID: <528491287.132585.1444201875382.JavaMail.yahoo@mail.yahoo.com> Hey Jenny, It happens frequently. And there is no CPU saturation at that time. -Naman On Tuesday, 6 October 2015 11:49 PM, Yu Zhang wrote: Naman, Are you on Linux? Do you have Transparent Huge Page on from the system level? Thanks, Jenny On 10/6/2015 10:52 AM, Yu Zhang wrote: Naman, It seems to me not related to safe point. Because the entry has ?[Times: user=0.38 sys=0.02, real=0.75 secs] indicating it is 0.75 sec when collection at safe point is done. What I do not understand is the time 0.0355102 sec is printed after the evacuation is done. The only difference between 0.0355102 and last entry is some book keeping stuff. But we are using much more user and real time. The big difference between real and user time means the gc threads can not be scheduled to get cpu. But no explanation for 0.0355102 vs 0.38 sec. Do you see this happen a lot? Is your system cpu saturated? Thanks, Jenny On 10/6/2015 9:32 AM, Naman Jain wrote: Hey jenny, JVM version used is: java version "1.8.0_40" Java(TM) SE Runtime Environment (build 1.8.0_40-b26) Java HotSpot(TM) 64-Bit Server VM (build 25.40-b25, mixed mode) -Naman On Tuesday, 6 October 2015 9:56 PM, Naman Jain wrote: Hey Jenny, I checked the sar files, there was no swapping or waiting on i/o. The jvm I am using is: Java HotSpot(TM) 64-Bit Server VM (25.40-b25) for linux-amd64 JRE (1.8.0_40-b26) Can you tell me any other reason which might have caused this ? Also, does safepoint have anything do with this ? Thanks? Naman On Tuesday, 6 October 2015 9:00 PM, Yu Zhang wrote: Naman, This entry is very strange, it has 0.0355102 sec for gc pause, but the time stamp at the bottom has ?[Times: user=0.38 sys=0.02, real=0.75 secs] The real time matches PrintGCApplicationStoppedTime. Since there are 18 gc threads, if not blocked by something, the real time should be smaller than user time. can you check if the system is swapping or waiting on i/o? Which JVM are you using? Thanks, Jenny On 10/6/2015 1:58 AM, Naman Jain wrote: Hey Everyone, G1 GC log: 2015-10-05T14:54:35.518+0000: 549629.568: [GC pause (G1 Evacuation Pause) (young) Desired survivor size 322961408 bytes, new threshold 15 (max 15) - age ? 1:? ? 5292232 bytes,? ? 5292232 total - age ? 2:? ? 1201264 bytes,? ? 6493496 total - age ? 3: ? ? 377376 bytes,? ? 6870872 total - age ? 4: ? ? 440752 bytes,? ? 7311624 total - age ? 5: ? ? 201976 bytes,? ? 7513600 total - age ? 6:? ? ? 77616 bytes,? ? 7591216 total - age ? 7:? ? ? 28824 bytes,? ? 7620040 total - age ? 8:? ? ? 55312 bytes,? ? 7675352 total - age ? 9:? ? ? 55432 bytes,? ? 7730784 total - age? 10:? ? ? 14272 bytes,? ? 7745056 total - age? 11:? ? ? 16872 bytes,? ? 7761928 total - age? 12:? ? ? 18208 bytes,? ? 7780136 total - age? 13:? ? ? 11656 bytes,? ? 7791792 total - age? 14:? ? ? 15800 bytes,? ? 7807592 total - age? 15:? ? ? 33960 bytes,? ? 7841552 total ?549629.569: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 14904, predicted base time: 20.98 ms, remaining time: 19.02 ms, target pause time: 40.00 ms] ?549629.569: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 1224 regions, survivors: 4 regions, predicted young region time: 10.73 ms] ?549629.569: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 1224 regions, survivors: 4 regions, old: 0 regions, predicted pause time: 31.70 ms, target pause time: 40.00 ms] , 0.0355102 secs] ?? [Parallel Time: 21.7 ms, GC Workers: 18] ? ? ? [GC Worker Start (ms): Min: 549629569.6, Avg: 549629569.8, Max: 549629569.9, Diff: 0.3] ? ? ? [Ext Root Scanning (ms): Min: 3.6, Avg: 4.3, Max: 12.7, Diff: 9.0, Sum: 78.1] ? ? ? [Update RS (ms): Min: 0.0, Avg: 4.1, Max: 4.7, Diff: 4.7, Sum: 74.6] ?? ? ? ? [Processed Buffers: Min: 0, Avg: 60.3, Max: 79, Diff: 79, Sum: 1086] ? ? ? [Scan RS (ms): Min: 0.1, Avg: 2.2, Max: 2.7, Diff: 2.5, Sum: 40.4] ? ? ? [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] ? ? ? [Object Copy (ms): Min: 8.0, Avg: 10.2, Max: 10.5, Diff: 2.5, Sum: 183.2] ? ? ? [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.5] ? ? ? [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.0] ? ? ? [GC Worker Total (ms): Min: 20.9, Avg: 21.2, Max: 21.4, Diff: 0.5, Sum: 381.0] ? ? ? [GC Worker End (ms): Min: 549629590.9, Avg: 549629590.9, Max: 549629591.1, Diff: 0.2] ?? [Code Root Fixup: 0.3 ms] ?? [Code Root Purge: 0.0 ms] ?? [Clear CT: 0.8 ms] ?? [Other: 12.6 ms] ? ? ? [Choose CSet: 0.0 ms] ? ? ? [Ref Proc: 6.2 ms] ? ? ? [Ref Enq: 0.5 ms] ? ? ? [Redirty Cards: 0.4 ms] ? ? ? [Humongous Reclaim: 0.0 ms] ? ? ? [Free CSet: 2.9 ms] ?? [Eden: 4896.0M(4896.0M)->0.0B(4892.0M) Survivors: 16.0M->20.0M Heap: 6505.9M(8192.0M)->1611.1M(8192.0M)] ?[Times: user=0.38 sys=0.02, real=0.75 secs]? 2015-10-05T14:54:36.272+0000: 549630.322: Total time for which application threads were stopped: 0.7695049 seconds, Stopping threads took: 0.0004596 seconds As you can see that young GC took around 35ms but the total time for which application threads were stopped is 769ms. Can anyone tell me what other pauses are included in this total time ? Also why pauses other then GC pauses were included in GC logs ? Earlier I was using CMS. But in CMS I never saw such differences in GC time and Application thread pause time. Can anyone tell me what is the reason behind such behaviour ? Thanks -Naman _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: From kelaban17 at gmail.com Wed Oct 7 18:40:57 2015 From: kelaban17 at gmail.com (Keith L) Date: Wed, 07 Oct 2015 18:40:57 +0000 Subject: [G1GC] Slow processing JNI Weak References Message-ID: Java: - JDK 1.8u40 - UseG1GC, ParallelRefProcEnabled, Xmx12g,Xms12g Application: - Solr 4.10.4 Application Details: Using Solr, which does not use JNI directly although it make heavy use of nio and MMaps. Possibly relevant is that we run our application with the yourkit agent with monitoring disabled by default. When using G1GC we are seeing very high processing times for JNI Weak Reference processing, in both GC Remak phase and during Mixed collections. Originally we saw high times during WeakReference processing but adding"-XX:+ParallelRefProcEnabled" flag did away with this. JNI Weak Reference processing we've seen take as much as 300+ seconds!, although more commonly we see it taking 20-60 seconds when it start becoming a problem. Currently JNI Weak Reference are not processed in parallel in HotSpot so adding the parallel flag has no affect. ( https://bugs.openjdk.java.net/browse/JDK-8072498) Generally we don't see the time for processing be a problem until a few days of uptime. At some point the JNI processing gets more expensive and time taken climbs up and is expensive nearly every there is a Remark phase. Possibly related: after a heap dump of a running application we noticed anywhere from a couple hundred to hundreds of thousands of unreferenced DirectByteBuffers. Doing a jstack I've seen as many 16 Million JNI Global References being reported. Although I haven't been able to find a way to get the JNI Weak Reference information to be displayed. Questions: 1) What is causing the JNI Weak References? - Is it from nio? Usage of MMaps, DirectBuffers, etc? 2) Why does it become worse during the lifetime of the application? - Is there a leak? - are medium life objects being moved too early - Tune G1NewSizePercent? - Tune Survivor settings? 3) Are the JNI Weak References being visited but not cleaned during GC? - Why not? tune G1HeapWastePercent? 4) How can more debugging be done to find where the references are coming from As of right now we are experimenting with different G1GC flags, however without understanding the root cause it sometimes requires days between experiments for the problem to present itself. We have not tuned any of the options above which disable the adaptive sizing policy, some options we are actively testing are: - Lowering InitiatingHeapOccupancyPercent - To force dereferenced objects in tenured to be cleaned earlier. - Increasing MaxGCPauseMillis - To encourage policy for the young generation size to be larger (discouraging premature promotion). Example of remark phases below (full gc log can be provided if required) One example log file (however we have seen worse), using flags in the beginning: 1348278.630: [GC pause (G1 Evacuation Pause) (young) (initial-mark) Desired survivor size 41943040 bytes, new threshold 15 (max 15) - age 1: 36138128 bytes, 36138128 total 1348278.676: [SoftReference, 0 refs, 0.0086523 secs]1348278.684: [WeakReference, 7128 refs, 0.0047846 secs]1348278.689: [FinalReference, 414 refs, 0.0063207 secs]1348278.695: [PhantomReference, 0 refs, 376 refs, 0.0129979 secs]1348278.709: [JNI Weak Reference, 0.1018689 secs], 0.1955299 secs] [Parallel Time: 39.7 ms, GC Workers: 23] [GC Worker Start (ms): Min: 1348278633.2, Avg: 1348278633.7, Max: 1348278634.2, Diff: 1.0] [Ext Root Scanning (ms): Min: 11.2, Avg: 11.7, Max: 14.9, Diff: 3.7, Sum: 269.2] [Update RS (ms): Min: 9.9, Avg: 13.3, Max: 14.0, Diff: 4.1, Sum: 304.8] [Processed Buffers: Min: 23, Avg: 66.9, Max: 117, Diff: 94, Sum: 1538] [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.6] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 11.8, Avg: 12.4, Max: 12.9, Diff: 1.1, Sum: 286.3] [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.3] [GC Worker Other (ms): Min: 0.1, Avg: 0.3, Max: 0.7, Diff: 0.6, Sum: 6.2] [GC Worker Total (ms): Min: 37.0, Avg: 37.7, Max: 38.6, Diff: 1.6, Sum: 867.4] [GC Worker End (ms): Min: 1348278671.2, Avg: 1348278671.4, Max: 1348278671.8, Diff: 0.6] [Code Root Fixup: 0.4 ms] [Code Root Purge: 0.0 ms] [Clear CT: 2.8 ms] [Other: 152.6 ms] [Choose CSet: 0.0 ms] [Ref Proc: 137.2 ms] [Ref Enq: 2.6 ms] [Redirty Cards: 3.2 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 0.6 ms] [Eden: 532.0M(532.0M)->0.0B(532.0M) Survivors: 80.0M->80.0M Heap: 6225.6M(12.0G)->5701.0M(12.0G)] [Times: user=0.80 sys=0.04, real=0.20 secs] 1348278.827: [GC concurrent-root-region-scan-start] 1348278.828: Total time for which application threads were stopped: 0.2145574 seconds, Stopping threads took: 0.0012976 seconds 1348278.844: [GC concurrent-root-region-scan-end, 0.0170768 secs] 1348278.844: [GC concurrent-mark-start] 1348279.075: [GC concurrent-mark-end, 0.2306171 secs] 1348279.088: [GC remark 1348279.088: [Finalize Marking, 0.0051946 secs] 1348279.093: [GC ref-proc1348279.093: [SoftReference, 186 refs, 0.0089505 secs]1348279.102: [WeakReference, 2668 refs, 0.0051839 secs]1348279.107: [FinalReference, 848 refs, 0.0088167 secs]1348279.116: [PhantomReference, 0 refs, 3836 refs, 0.0130326 secs]1348279.129: [JNI Weak Reference, 18.9741270 secs], 19.0130579 secs] 1348298.106: [Unloading, 0.0177559 secs], 19.0412558 secs] [Times: user=19.59 sys=0.00, real=19.04 secs] 1348298.132: Total time for which application threads were stopped: 19.0566871 seconds, Stopping threads took: 0.0011100 seconds 1348298.147: [GC cleanup 5836M->4256M(12G), 0.0124121 secs] [Times: user=0.11 sys=0.00, real=0.01 secs] ... 1348358.227: [GC remark 1348358.227: [Finalize Marking, 0.0042513 secs] 1348358.231: [GC ref-proc1348358.231: [SoftReference, 197 refs, 0.0085051 secs]1348358.240: [WeakReference, 5962 refs, 0.0082517 secs]1348358.248: [FinalReference, 1881 refs, 0.0074058 secs]1348358.255: [PhantomReference, 0 refs, 3802 refs, 0.0116230 secs]1348358.267: [JNI Weak Reference, 29.3914835 secs], 29.4310975 secs] 1348387.662: [Unloading, 0.0179274 secs], 29.4583167 secs] 1350907.355: [GC remark 1350907.355: [Finalize Marking, 0.0066394 secs] 1350907.362: [GC ref-proc1350907.362: [SoftReference, 189 refs, 0.0084702 secs]1350907.370: [WeakReference, 2379 refs, 0.0046033 secs]1350907.375: [FinalReference, 779 refs, 0.0086055 secs]1350907.383: [PhantomReference, 0 refs, 4349 refs, 0.0124574 secs]1350907.396: [JNI Weak Reference, 29.9550822 secs], 29.9923290 secs] 1350937.354: [Unloading, 0.0171185 secs], 30.0213910 secs] ... ... ... 1377495.992: [GC remark 1377495.992: [Finalize Marking, 0.0039972 secs] 1377495.996: [GC ref-proc1377495.996: [SoftReference, 192 refs, 0.0084780 secs]1377496.005: [WeakReference, 1171 refs, 0.0040591 secs]1377496.009: [FinalReference, 619 refs, 0.0068656 secs]1377496.016: [PhantomReference, 0 refs, 4886 refs, 0.0094143 secs]1377496.025: [JNI Weak Reference, 48.3669099 secs], 48.3980356 secs] 1377544.394: [Unloading, 0.0159881 secs], 48.4228951 secs] 1378034.868: [GC remark 1378034.868: [Finalize Marking, 0.0033629 secs] 1378034.871: [GC ref-proc1378034.871: [SoftReference, 193 refs, 0.0059063 secs]1378034.877: [WeakReference, 1332 refs, 0.0036461 secs]1378034.881: [FinalReference, 611 refs, 0.0046526 secs]1378034.885: [PhantomReference, 0 refs, 4254 refs, 0.0059437 secs]1378034.891: [JNI Weak Reference, 46.6392836 secs], 46.6624796 secs] 1378081.534: [Unloading, 0.0180802 secs], 46.6893835 secs] 1378677.326: [GC remark 1378677.326: [Finalize Marking, 0.0039769 secs] 1378677.330: [GC ref-proc1378677.330: [SoftReference, 193 refs, 0.0080377 secs]1378677.338: [WeakReference, 1719 refs, 0.0035972 secs]1378677.341: [FinalReference, 351 refs, 0.0063767 secs]1378677.348: [PhantomReference, 0 refs, 4753 refs, 0.0103290 secs]1378677.358: [JNI Weak Reference, 43.9230851 secs], 43.9537699 secs] 1378721.284: [Unloading, 0.0150821 secs], 43.9780128 secs] 1379626.009: [GC remark 1379626.009: [Finalize Marking, 0.0040175 secs] 1379626.013: [GC ref-proc1379626.013: [SoftReference, 186 refs, 0.0080680 secs]1379626.021: [WeakReference, 955 refs, 0.0064521 secs]1379626.028: [FinalReference, 567 refs, 0.0098877 secs]1379626.038: [PhantomReference, 0 refs, 4701 refs, 0.0104484 secs]1379626.048: [JNI Weak Reference, 51.3181908 secs], 51.3558998 secs] 1379677.369: [Unloading, 0.0207052 secs], 51.3856987 secs] ... 1380251.013: [GC pause (G1 Evacuation Pause) (young) (initial-mark) Desired survivor size 41943040 bytes, new threshold 1 (max 15) - age 1: 46612512 bytes, 46612512 total [SoftReference, 0 refs, 0.0074261 secs]1380251.048: [WeakReference, 5189 refs, 0.0041909 secs]1380251.052: [FinalReference, 393 refs, 0.0042160 secs]1380251.056: [PhantomReference, 0 refs, 83 refs, 0.0074198 secs]1380251.064: [JNI Weak Reference, 3.0041020 secs], 3.0678505 secs] [Parallel Time: 23.5 ms, GC Workers: 23] [GC Worker Start (ms): Min: 1380251014.8, Avg: 1380251015.3, Max: 1380251015.9, Diff: 1.1] [Ext Root Scanning (ms): Min: 6.7, Avg: 7.5, Max: 10.9, Diff: 4.2, Sum: 173.4] [Update RS (ms): Min: 3.7, Avg: 6.5, Max: 7.0, Diff: 3.3, Sum: 150.5] [Processed Buffers: Min: 27, Avg: 66.7, Max: 112, Diff: 85, Sum: 1534] [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.6] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 6.3, Avg: 6.7, Max: 7.3, Diff: 1.0, Sum: 154.9] [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] [GC Worker Other (ms): Min: 0.1, Avg: 0.4, Max: 0.8, Diff: 0.8, Sum: 8.3] [GC Worker Total (ms): Min: 20.4, Avg: 21.2, Max: 21.8, Diff: 1.4, Sum: 487.7] [GC Worker End (ms): Min: 1380251036.2, Avg: 1380251036.5, Max: 1380251037.0, Diff: 0.8] [Code Root Fixup: 0.4 ms] [Code Root Purge: 0.0 ms] [Clear CT: 2.9 ms] [Other: 3041.1 ms] [Choose CSet: 0.0 ms] [Ref Proc: 3029.5 ms] [Ref Enq: 2.7 ms] [Redirty Cards: 2.7 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 0.3 ms] [Eden: 532.0M(532.0M)->0.0B(548.0M) Survivors: 80.0M->64.0M Heap: 6142.5M(12.0G)->5620.7M(12.0G)] [Times: user=3.46 sys=0.00, real=3.07 secs] 1380254.081: [GC concurrent-root-region-scan-start] 1380254.082: Total time for which application threads were stopped: 3.0847197 seconds, Stopping threads took: 0.0011519 seconds 1380254.092: [GC concurrent-root-region-scan-end, 0.0105392 secs] 1380254.092: [GC concurrent-mark-start] 1380254.275: [GC concurrent-mark-end, 0.1832853 secs] 1380254.289: [GC remark 1380254.290: [Finalize Marking, 0.0039580 secs] 1380254.294: [GC ref-proc1380254.294: [SoftReference, 183 refs, 0.0068857 secs]1380254.300: [WeakReference, 1384 refs, 0.0043966 secs]1380254.305: [FinalReference, 579 refs, 0.0073469 secs]1380254.312: [PhantomReference, 0 refs, 4327 refs, 0.0077366 secs]1380254.320: [JNI Weak Reference, 38.7211438 secs], 38.7506058 secs] 1380293.044: [Unloading, 0.0174879 secs], 38.7765019 secs] [Times: user=39.37 sys=0.00, real=38.78 secs] 1380293.067: Total time for which application threads were stopped: 38.7917823 seconds, Stopping threads took: 0.0011561 seconds 1380293.077: [GC cleanup 5659M->4167M(12G), 0.0120391 secs] [Times: user=0.11 sys=0.00, real=0.01 secs] -------------- next part -------------- An HTML attachment was scrubbed... URL: From yu.zhang at oracle.com Fri Oct 9 18:09:24 2015 From: yu.zhang at oracle.com (Yu Zhang) Date: Fri, 9 Oct 2015 11:09:24 -0700 Subject: [G1GC] Slow processing JNI Weak References In-Reply-To: References: Message-ID: <561802D4.6070905@oracle.com> Keith, I am adding hotspot-gc-dev, hoping some one can give a better explanation. Yes, currently the JNI weak reference processing is single threaded. You mentioned JNI weak reference pause long for remark and mixed gc. For young gc, it is long as well. Just for young gc, the collected set is limited. My guess is you can not make much difference by tuning. There is a possibility for memory leak in the native code. A lot of Solr applications use native memory for cache. Thanks, Jenny On 10/7/2015 11:40 AM, Keith L wrote: > Java: > - JDK 1.8u40 > - UseG1GC, ParallelRefProcEnabled, Xmx12g,Xms12g > > Application: > - Solr 4.10.4 > > Application Details: > Using Solr, which does not use JNI directly although it make heavy use > of nio and MMaps. > Possibly relevant is that we run our application with the yourkit > agent with monitoring disabled by default. > > When using G1GC we are seeing very high processing times for JNI Weak > Reference processing, in both GC Remak phase and during Mixed > collections. Originally we saw high times during WeakReference > processing but adding"-XX:+ParallelRefProcEnabled" flag did away with > this. JNI Weak Reference processing we've seen take as much as 300+ > seconds!, although more commonly we see it taking 20-60 seconds when > it start becoming a problem. Currently JNI Weak Reference are not > processed in parallel in HotSpot so adding the parallel flag has no > affect. (https://bugs.openjdk.java.net/browse/JDK-8072498) > > Generally we don't see the time for processing be a problem until a > few days of uptime. At some point the JNI processing gets more > expensive and time taken climbs up and is expensive nearly every there > is a Remark phase. > > Possibly related: after a heap dump of a running application we > noticed anywhere from a couple hundred to hundreds of thousands of > unreferenced DirectByteBuffers. > > Doing a jstack I've seen as many 16 Million JNI Global References > being reported. Although I haven't been able to find a way to get the > JNI Weak Reference information to be displayed. > > Questions: > 1) What is causing the JNI Weak References? > - Is it from nio? Usage of MMaps, DirectBuffers, etc? > 2) Why does it become worse during the lifetime of the application? > - Is there a leak? > - are medium life objects being moved too early > - Tune G1NewSizePercent? > - Tune Survivor settings? > 3) Are the JNI Weak References being visited but not cleaned during GC? > - Why not? tune G1HeapWastePercent? > 4) How can more debugging be done to find where the references are > coming from > > > As of right now we are experimenting with different G1GC flags, > however without understanding the root cause it sometimes requires > days between experiments for the problem to present itself. > > > We have not tuned any of the options above which disable the adaptive > sizing policy, some options we are actively testing are: > > - Lowering InitiatingHeapOccupancyPercent > - To force dereferenced objects in tenured to be cleaned earlier. > > - Increasing MaxGCPauseMillis > - To encourage policy for the young generation size to be larger > (discouraging premature promotion). > > > Example of remark phases below (full gc log can be provided if required) > > One example log file (however we have seen worse), using flags in the > beginning: > > 1348278.630: [GC pause (G1 Evacuation Pause) (young) (initial-mark) > Desired survivor size 41943040 bytes, new threshold 15 (max 15) > - age 1: 36138128 bytes, 36138128 total > 1348278.676: [SoftReference, 0 refs, 0.0086523 secs]1348278.684: > [WeakReference, 7128 refs, 0.0047846 secs]1348278.689: > [FinalReference, 414 refs, 0.0063207 secs]1348278.695: > [PhantomReference, 0 refs, 376 refs, 0.0129979 secs]1348278.709: [JNI > Weak Reference, 0.1018689 secs], 0.1955299 secs] > [Parallel Time: 39.7 ms, GC Workers: 23] > [GC Worker Start (ms): Min: 1348278633.2, Avg: 1348278633.7, > Max: 1348278634.2, Diff: 1.0] > [Ext Root Scanning (ms): Min: 11.2, Avg: 11.7, Max: 14.9, Diff: > 3.7, Sum: 269.2] > [Update RS (ms): Min: 9.9, Avg: 13.3, Max: 14.0, Diff: 4.1, Sum: > 304.8] > [Processed Buffers: Min: 23, Avg: 66.9, Max: 117, Diff: 94, > Sum: 1538] > [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.6] > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > [Object Copy (ms): Min: 11.8, Avg: 12.4, Max: 12.9, Diff: 1.1, > Sum: 286.3] > [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: > 0.3] > [GC Worker Other (ms): Min: 0.1, Avg: 0.3, Max: 0.7, Diff: 0.6, > Sum: 6.2] > [GC Worker Total (ms): Min: 37.0, Avg: 37.7, Max: 38.6, Diff: > 1.6, Sum: 867.4] > [GC Worker End (ms): Min: 1348278671.2, Avg: 1348278671.4, Max: > 1348278671.8, Diff: 0.6] > [Code Root Fixup: 0.4 ms] > [Code Root Purge: 0.0 ms] > [Clear CT: 2.8 ms] > [Other: 152.6 ms] > [Choose CSet: 0.0 ms] > [Ref Proc: 137.2 ms] > [Ref Enq: 2.6 ms] > [Redirty Cards: 3.2 ms] > [Humongous Reclaim: 0.0 ms] > [Free CSet: 0.6 ms] > [Eden: 532.0M(532.0M)->0.0B(532.0M) Survivors: 80.0M->80.0M Heap: > 6225.6M(12.0G)->5701.0M(12.0G)] > [Times: user=0.80 sys=0.04, real=0.20 secs] > 1348278.827: [GC concurrent-root-region-scan-start] > 1348278.828: Total time for which application threads were stopped: > 0.2145574 seconds, Stopping threads took: 0.0012976 seconds > 1348278.844: [GC concurrent-root-region-scan-end, 0.0170768 secs] > 1348278.844: [GC concurrent-mark-start] > 1348279.075: [GC concurrent-mark-end, 0.2306171 secs] > 1348279.088: [GC remark 1348279.088: [Finalize Marking, 0.0051946 > secs] 1348279.093: [GC ref-proc1348279.093: [SoftReference, 186 refs, > 0.0089505 secs]1348279.102: [WeakReference, 2668 refs, 0.0051839 > secs]1348279.107: [FinalReference, 848 refs, 0.0088167 > secs]1348279.116: [PhantomReference, 0 refs, 3836 refs, 0.0130326 > secs]1348279.129: [JNI Weak Reference, 18.9741270 secs], 19.0130579 > secs] 1348298.106: [Unloading, 0.0177559 secs], 19.0412558 secs] > [Times: user=19.59 sys=0.00, real=19.04 secs] > 1348298.132: Total time for which application threads were stopped: > 19.0566871 seconds, Stopping threads took: 0.0011100 seconds > 1348298.147: [GC cleanup 5836M->4256M(12G), 0.0124121 secs] > [Times: user=0.11 sys=0.00, real=0.01 secs] > ... > 1348358.227: [GC remark 1348358.227: [Finalize Marking, 0.0042513 > secs] 1348358.231: [GC ref-proc1348358.231: [SoftReference, 197 refs, > 0.0085051 secs]1348358.240: [WeakReference, 5962 refs, 0.0082517 > secs]1348358.248: [FinalReference, 1881 refs, 0.0074058 > secs]1348358.255: [PhantomReference, 0 refs, 3802 refs, 0.0116230 > secs]1348358.267: [JNI Weak Reference, 29.3914835 secs], 29.4310975 > secs] 1348387.662: [Unloading, 0.0179274 secs], 29.4583167 secs] > 1350907.355: [GC remark 1350907.355: [Finalize Marking, 0.0066394 > secs] 1350907.362: [GC ref-proc1350907.362: [SoftReference, 189 refs, > 0.0084702 secs]1350907.370: [WeakReference, 2379 refs, 0.0046033 > secs]1350907.375: [FinalReference, 779 refs, 0.0086055 > secs]1350907.383: [PhantomReference, 0 refs, 4349 refs, 0.0124574 > secs]1350907.396: [JNI Weak Reference, 29.9550822 secs], 29.9923290 > secs] 1350937.354: [Unloading, 0.0171185 secs], 30.0213910 secs] > ... > ... > ... > 1377495.992: [GC remark 1377495.992: [Finalize Marking, 0.0039972 > secs] 1377495.996: [GC ref-proc1377495.996: [SoftReference, 192 refs, > 0.0084780 secs]1377496.005: [WeakReference, 1171 refs, 0.0040591 > secs]1377496.009: [FinalReference, 619 refs, 0.0068656 > secs]1377496.016: [PhantomReference, 0 refs, 4886 refs, 0.0094143 > secs]1377496.025: [JNI Weak Reference, 48.3669099 secs], 48.3980356 > secs] 1377544.394: [Unloading, 0.0159881 secs], 48.4228951 secs] > 1378034.868: [GC remark 1378034.868: [Finalize Marking, 0.0033629 > secs] 1378034.871: [GC ref-proc1378034.871: [SoftReference, 193 refs, > 0.0059063 secs]1378034.877: [WeakReference, 1332 refs, 0.0036461 > secs]1378034.881: [FinalReference, 611 refs, 0.0046526 > secs]1378034.885: [PhantomReference, 0 refs, 4254 refs, 0.0059437 > secs]1378034.891: [JNI Weak Reference, 46.6392836 secs], 46.6624796 > secs] 1378081.534: [Unloading, 0.0180802 secs], 46.6893835 secs] > 1378677.326: [GC remark 1378677.326: [Finalize Marking, 0.0039769 > secs] 1378677.330: [GC ref-proc1378677.330: [SoftReference, 193 refs, > 0.0080377 secs]1378677.338: [WeakReference, 1719 refs, 0.0035972 > secs]1378677.341: [FinalReference, 351 refs, 0.0063767 > secs]1378677.348: [PhantomReference, 0 refs, 4753 refs, 0.0103290 > secs]1378677.358: [JNI Weak Reference, 43.9230851 secs], 43.9537699 > secs] 1378721.284: [Unloading, 0.0150821 secs], 43.9780128 secs] > 1379626.009: [GC remark 1379626.009: [Finalize Marking, 0.0040175 > secs] 1379626.013: [GC ref-proc1379626.013: [SoftReference, 186 refs, > 0.0080680 secs]1379626.021: [WeakReference, 955 refs, 0.0064521 > secs]1379626.028: [FinalReference, 567 refs, 0.0098877 > secs]1379626.038: [PhantomReference, 0 refs, 4701 refs, 0.0104484 > secs]1379626.048: [JNI Weak Reference, 51.3181908 secs], 51.3558998 > secs] 1379677.369: [Unloading, 0.0207052 secs], 51.3856987 secs] > ... > 1380251.013: [GC pause (G1 Evacuation Pause) (young) (initial-mark) > Desired survivor size 41943040 bytes, new threshold 1 (max 15) > - age 1: 46612512 bytes, 46612512 total > [SoftReference, 0 refs, 0.0074261 secs]1380251.048: [WeakReference, > 5189 refs, 0.0041909 secs]1380251.052: [FinalReference, 393 refs, > 0.0042160 secs]1380251.056: [PhantomReference, 0 refs, 83 refs, > 0.0074198 secs]1380251.064: [JNI Weak Reference, 3.0041020 secs], > 3.0678505 secs] > [Parallel Time: 23.5 ms, GC Workers: 23] > [GC Worker Start (ms): Min: 1380251014.8, Avg: 1380251015.3, > Max: 1380251015.9, Diff: 1.1] > [Ext Root Scanning (ms): Min: 6.7, Avg: 7.5, Max: 10.9, Diff: > 4.2, Sum: 173.4] > [Update RS (ms): Min: 3.7, Avg: 6.5, Max: 7.0, Diff: 3.3, Sum: > 150.5] > [Processed Buffers: Min: 27, Avg: 66.7, Max: 112, Diff: 85, > Sum: 1534] > [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.6] > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > [Object Copy (ms): Min: 6.3, Avg: 6.7, Max: 7.3, Diff: 1.0, Sum: > 154.9] > [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: > 0.1] > [GC Worker Other (ms): Min: 0.1, Avg: 0.4, Max: 0.8, Diff: 0.8, > Sum: 8.3] > [GC Worker Total (ms): Min: 20.4, Avg: 21.2, Max: 21.8, Diff: > 1.4, Sum: 487.7] > [GC Worker End (ms): Min: 1380251036.2, Avg: 1380251036.5, Max: > 1380251037.0, Diff: 0.8] > [Code Root Fixup: 0.4 ms] > [Code Root Purge: 0.0 ms] > [Clear CT: 2.9 ms] > [Other: 3041.1 ms] > [Choose CSet: 0.0 ms] > [Ref Proc: 3029.5 ms] > [Ref Enq: 2.7 ms] > [Redirty Cards: 2.7 ms] > [Humongous Reclaim: 0.0 ms] > [Free CSet: 0.3 ms] > [Eden: 532.0M(532.0M)->0.0B(548.0M) Survivors: 80.0M->64.0M Heap: > 6142.5M(12.0G)->5620.7M(12.0G)] > [Times: user=3.46 sys=0.00, real=3.07 secs] > 1380254.081: [GC concurrent-root-region-scan-start] > 1380254.082: Total time for which application threads were stopped: > 3.0847197 seconds, Stopping threads took: 0.0011519 seconds > 1380254.092: [GC concurrent-root-region-scan-end, 0.0105392 secs] > 1380254.092: [GC concurrent-mark-start] > 1380254.275: [GC concurrent-mark-end, 0.1832853 secs] > 1380254.289: [GC remark 1380254.290: [Finalize Marking, 0.0039580 > secs] 1380254.294: [GC ref-proc1380254.294: [SoftReference, 183 refs, > 0.0068857 secs]1380254.300: [WeakReference, 1384 refs, 0.0043966 > secs]1380254.305: [FinalReference, 579 refs, 0.0073469 > secs]1380254.312: [PhantomReference, 0 refs, 4327 refs, 0.0077366 > secs]1380254.320: [JNI Weak Reference, 38.7211438 secs], 38.7506058 > secs] 1380293.044: [Unloading, 0.0174879 secs], 38.7765019 secs] > [Times: user=39.37 sys=0.00, real=38.78 secs] > 1380293.067: Total time for which application threads were stopped: > 38.7917823 seconds, Stopping threads took: 0.0011561 seconds > 1380293.077: [GC cleanup 5659M->4167M(12G), 0.0120391 secs] > [Times: user=0.11 sys=0.00, real=0.01 secs] > > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanbeast at gmail.com Mon Oct 12 09:14:29 2015 From: vanbeast at gmail.com (Ivan Shramko) Date: Mon, 12 Oct 2015 12:14:29 +0300 Subject: Very long duration of Finalize marking in the Remark stage of G1GC Message-ID: Application becomes unresponsive on high load while G1 garbage collector is in the Remark stage(Finalize marking). This is part of the gc log: {Heap before GC invocations=1558 (full 0): garbage-first heap total 94371840K, used 83001550K [0x00007f76d0000000, 0x00007f76d2005a00, 0x00007f8d50000000) region size 32768K, 1234 young (40435712K), 58 survivors (1900544K) Metaspace used 77160K, capacity 77951K, committed 78080K, reserved 79872K 204611.028: [G1Ergonomics (Concurrent Cycles) initiate concurrent cycle, reason: concurrent cycle initiation requested] 2015-10-04T15:00:05.197-0400: 204611.028: [GC pause (GCLocker Initiated GC) (young) (initial-mark) Desired survivor size 2600468480 bytes, new threshold 2 (max 2) - age 1: 621290376 bytes, 621290376 total - age 2: 245159208 bytes, 866449584 total 204611.029: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 184157, predicted base time: 57.32 ms, remaining time: 142.68 ms, target pause time: 200.00 ms] 204611.029: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 1176 regions, survivors: 58 regions, predicted young region time: 131.19 ms] 204611.029: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 1176 regions, survivors: 58 regions, old: 0 regions, predicted pause time: 188.51 ms, target pause time: 200.00 ms] 2015-10-04T15:00:05.416-0400: 204611.247: [SoftReference, 0 refs, 0.0023502 secs]2015-10-04T15:00:05.418-0400: 204611.249: [WeakReference, 394 refs, 0.0018755 secs]2015-10-04T15:00:05.420-0400: 204611.251: [FinalReference, 91704 refs, 0. 0081225 secs]2015-10-04T15:00:05.428-0400: 204611.259: [PhantomReference, 29 refs, 1 refs, 0.0048953 secs]2015-10-04T15:00:05.433-0400: 204611.264: [JNI Weak Reference, 0.0000229 secs], 0.2600007 secs] [Parallel Time: 215.1 ms, GC Workers: 32] [GC Worker Start (ms): Min: 204611030.3, Avg: 204611030.5, Max: 204611030.6, Diff: 0.4] [Ext Root Scanning (ms): Min: 2.5, Avg: 3.2, Max: 8.3, Diff: 5.8, Sum: 103.2] [Update RS (ms): Min: 19.0, Avg: 24.5, Max: 35.3, Diff: 16.3, Sum: 782.9] [Processed Buffers: Min: 7, Avg: 23.7, Max: 31, Diff: 24, Sum: 758] [Scan RS (ms): Min: 40.4, Avg: 42.2, Max: 43.0, Diff: 2.6, Sum: 1349.0] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.2] [Object Copy (ms): Min: 131.5, Avg: 140.0, Max: 142.4, Diff: 10.9, Sum: 4480.1] [Termination (ms): Min: 1.9, Avg: 4.1, Max: 4.6, Diff: 2.7, Sum: 130.8] [Termination Attempts: Min: 262, Avg: 544.9, Max: 599, Diff: 337, Sum: 17437] [GC Worker Other (ms): Min: 0.0, Avg: 0.2, Max: 0.4, Diff: 0.4, Sum: 7.1] [GC Worker Total (ms): Min: 213.8, Avg: 214.2, Max: 214.5, Diff: 0.7, Sum: 6853.3] [GC Worker End (ms): Min: 204611244.4, Avg: 204611244.6, Max: 204611244.8, Diff: 0.4] [Code Root Fixup: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 8.4 ms] [Other: 36.4 ms] [Choose CSet: 0.1 ms] [Ref Proc: 19.1 ms] [Ref Enq: 0.8 ms] [Redirty Cards: 2.0 ms] [Humongous Register: 0.0 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 11.5 ms] [Eden: 36.8G(36.7G)->0.0B(35.8G) Survivors: 1856.0M->1568.0M Heap: 79.2G(90.0G)->42.4G(90.0G)] Heap after GC invocations=1559 (full 0): garbage-first heap total 94371840K, used 44456740K [0x00007f76d0000000, 0x00007f76d2005a00, 0x00007f8d50000000) region size 32768K, 49 young (1605632K), 49 survivors (1605632K) Metaspace used 77160K, capacity 77951K, committed 78080K, reserved 79872K } [Times: user=7.11 sys=0.18, real=0.26 secs] 2015-10-04T15:00:05.457-0400: 204611.288: [GC concurrent-root-region-scan-start] 2015-10-04T15:00:05.457-0400: 204611.288: Total time for which application threads were stopped: 0.2607079 seconds, Stopping threads took: 0.0000623 seconds 2015-10-04T15:00:05.593-0400: 204611.424: [GC concurrent-root-region-scan-end, 0.1359037 secs] 2015-10-04T15:00:05.593-0400: 204611.424: [GC concurrent-mark-start] 2015-10-04T15:00:06.238-0400: 204612.069: [GC concurrent-mark-end, 0.6448793 secs] 2015-10-04T15:00:06.238-0400: 204612.069: [GC remark 2015-10-04T15:00:06.238-0400: 204612.069: [Finalize Marking, 82.5606577 secs] 2015-10-04T15:01:28.799-0400: 204694.630: [GC ref-proc2015-10-04T15:01:28.799-0400: 204694.630: [SoftRefer ence, 64 refs, 0.0031041 secs]2015-10-04T15:01:28.802-0400: 204694.633: [WeakReference, 383 refs, 0.0023197 secs]2015-10-04T15:01:28.805-0400: 204694.636: [FinalReference, 100288 refs, 0.0330312 secs]2015-10-04T15:01:28.838-0400: 204694. 669: [PhantomReference, 2 refs, 37 refs, 0.0030514 secs]2015-10-04T15:01:28.841-0400: 204694.672: [JNI Weak Reference, 0.0000915 secs], 0.0539037 secs] 2015-10-04T15:01:28.853-0400: 204694.684: [Unloading, 0.0130054 secs], 82.6649201 sec s] [Times: user=1139.19 sys=393.93, real=82.65 secs] 2015-10-04T15:01:28.903-0400: 204694.735: Total time for which application threads were stopped: 82.6655358 seconds, Stopping threads took: 0.0000918 seconds 2015-10-04T15:01:28.904-0400: 204694.735: [GC cleanup 42G->42G(90G), 0.1166196 secs] Java version: Java version "1.8.0_60"Java(TM) SE Runtime Environment (build 1.8.0_60-b27)Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode) Server has 128GB of RAM and 32 cores(with HyperThreading). GC-related JVM options: -XX:InitialHeapSize=107374182400 -XX:+ManagementServer -XX:MaxHeapSize=107374182400 -XX:-OmitStackTraceInFastThrow -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+TieredCompilation -XX:+UseCondCardMark -XX:+UseG1GC -XX:ParallelGCThreads=32 -XX:+UnlockDiagnosticVMOptions -XX:ParGCCardsPerStrideChunk=32678 -XX:+ParallelRefProcEnabled -J-XX:MaxGCPauseMillis=100 Any ideas what can be the reason? -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlie.hunt at oracle.com Mon Oct 12 12:40:19 2015 From: charlie.hunt at oracle.com (charlie hunt) Date: Mon, 12 Oct 2015 07:40:19 -0500 Subject: Very long duration of Finalize marking in the Remark stage of G1GC In-Reply-To: References: Message-ID: <8B24130B-764C-4594-B588-771235CDCF55@oracle.com> Hi Ivan, The remark pause (in addition to high ?Finalize Marking?) also has a rather high reported sys CPU time. Usually when I see high sys CPU time, it is often a symptom of paging to/from virtual memory, or if you are running on Linux it is a symptom of transparent huge pages being enabled. Fwiw, I don?t think I?ve seen a high ?Finalize Marking? time like what you have here. Could you confirm you are not paging to/from virtual memory, and if you are running on Linux, can you also confirm transparent huge pages are disabled? thanks, charlie > On Oct 12, 2015, at 4:14 AM, Ivan Shramko wrote: > > Application becomes unresponsive on high load while G1 garbage collector is in the Remark stage(Finalize marking). > This is part of the gc log: > {Heap before GC invocations=1558 (full 0): > garbage-first heap total 94371840K, used 83001550K [0x00007f76d0000000, 0x00007f76d2005a00, 0x00007f8d50000000) > region size 32768K, 1234 young (40435712K), 58 survivors (1900544K) > Metaspace used 77160K, capacity 77951K, committed 78080K, reserved 79872K > 204611.028: [G1Ergonomics (Concurrent Cycles) initiate concurrent cycle, reason: concurrent cycle initiation requested] > 2015-10-04T15:00:05.197-0400: 204611.028: [GC pause (GCLocker Initiated GC) (young) (initial-mark) > Desired survivor size 2600468480 bytes, new threshold 2 (max 2) > - age 1: 621290376 bytes, 621290376 total > - age 2: 245159208 bytes, 866449584 total > 204611.029: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 184157, predicted base time: 57.32 ms, remaining time: 142.68 ms, target pause time: 200.00 ms] > 204611.029: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 1176 regions, survivors: 58 regions, predicted young region time: 131.19 ms] > 204611.029: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 1176 regions, survivors: 58 regions, old: 0 regions, predicted pause time: 188.51 ms, target pause time: 200.00 ms] > 2015-10-04T15:00:05.416-0400: 204611.247: [SoftReference, 0 refs, 0.0023502 secs]2015-10-04T15:00:05.418-0400: 204611.249: [WeakReference, 394 refs, 0.0018755 secs]2015-10-04T15:00:05.420-0400: 204611.251: [FinalReference, 91704 refs, 0. > 0081225 secs]2015-10-04T15:00:05.428-0400: 204611.259: [PhantomReference, 29 refs, 1 refs, 0.0048953 secs]2015-10-04T15:00:05.433-0400: 204611.264: [JNI Weak Reference, 0.0000229 secs], 0.2600007 secs] > [Parallel Time: 215.1 ms, GC Workers: 32] > [GC Worker Start (ms): Min: 204611030.3, Avg: 204611030.5, Max: 204611030.6, Diff: 0.4] > [Ext Root Scanning (ms): Min: 2.5, Avg: 3.2, Max: 8.3, Diff: 5.8, Sum: 103.2] > [Update RS (ms): Min: 19.0, Avg: 24.5, Max: 35.3, Diff: 16.3, Sum: 782.9] > [Processed Buffers: Min: 7, Avg: 23.7, Max: 31, Diff: 24, Sum: 758] > [Scan RS (ms): Min: 40.4, Avg: 42.2, Max: 43.0, Diff: 2.6, Sum: 1349.0] > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.2] > [Object Copy (ms): Min: 131.5, Avg: 140.0, Max: 142.4, Diff: 10.9, Sum: 4480.1] > [Termination (ms): Min: 1.9, Avg: 4.1, Max: 4.6, Diff: 2.7, Sum: 130.8] > [Termination Attempts: Min: 262, Avg: 544.9, Max: 599, Diff: 337, Sum: 17437] > [GC Worker Other (ms): Min: 0.0, Avg: 0.2, Max: 0.4, Diff: 0.4, Sum: 7.1] > [GC Worker Total (ms): Min: 213.8, Avg: 214.2, Max: 214.5, Diff: 0.7, Sum: 6853.3] > [GC Worker End (ms): Min: 204611244.4, Avg: 204611244.6, Max: 204611244.8, Diff: 0.4] > [Code Root Fixup: 0.0 ms] > [Code Root Purge: 0.0 ms] > [Clear CT: 8.4 ms] > [Other: 36.4 ms] > [Choose CSet: 0.1 ms] > [Ref Proc: 19.1 ms] > [Ref Enq: 0.8 ms] > [Redirty Cards: 2.0 ms] > [Humongous Register: 0.0 ms] > [Humongous Reclaim: 0.0 ms] > [Free CSet: 11.5 ms] > [Eden: 36.8G(36.7G)->0.0B(35.8G) Survivors: 1856.0M->1568.0M Heap: 79.2G(90.0G)->42.4G(90.0G)] > Heap after GC invocations=1559 (full 0): > garbage-first heap total 94371840K, used 44456740K [0x00007f76d0000000, 0x00007f76d2005a00, 0x00007f8d50000000) > region size 32768K, 49 young (1605632K), 49 survivors (1605632K) > Metaspace used 77160K, capacity 77951K, committed 78080K, reserved 79872K > } > [Times: user=7.11 sys=0.18, real=0.26 secs] > 2015-10-04T15:00:05.457-0400: 204611.288: [GC concurrent-root-region-scan-start] > > 2015-10-04T15:00:05.457-0400: 204611.288: Total time for which application threads were stopped: 0.2607079 seconds, Stopping threads took: 0.0000623 seconds > > 2015-10-04T15:00:05.593-0400: 204611.424: [GC concurrent-root-region-scan-end, 0.1359037 secs] > > 2015-10-04T15:00:05.593-0400: 204611.424: [GC concurrent-mark-start] > > 2015-10-04T15:00:06.238-0400: 204612.069: [GC concurrent-mark-end, 0.6448793 secs] > > 2015-10-04T15:00:06.238-0400: 204612.069: [GC remark 2015-10-04T15:00:06.238-0400: 204612.069: [Finalize Marking, 82.5606577 secs] 2015-10-04T15:01:28.799-0400: 204694.630: [GC ref-proc2015-10-04T15:01:28.799-0400: 204694.630: [SoftRefer > ence, 64 refs, 0.0031041 secs]2015-10-04T15:01:28.802-0400: 204694.633: [WeakReference, 383 refs, 0.0023197 secs]2015-10-04T15:01:28.805-0400: 204694.636: [FinalReference, 100288 refs, 0.0330312 secs]2015-10-04T15:01:28.838-0400: 204694. > 669: [PhantomReference, 2 refs, 37 refs, 0.0030514 secs]2015-10-04T15:01:28.841-0400: 204694.672: [JNI Weak Reference, 0.0000915 secs], 0.0539037 secs] 2015-10-04T15:01:28.853-0400: 204694.684: [Unloading, 0.0130054 secs], 82.6649201 sec > s] > [Times: user=1139.19 sys=393.93, real=82.65 secs] > > 2015-10-04T15:01:28.903-0400: 204694.735: Total time for which application threads were stopped: 82.6655358 seconds, Stopping threads took: 0.0000918 seconds > > 2015-10-04T15:01:28.904-0400: 204694.735: [GC cleanup 42G->42G(90G), 0.1166196 secs] > > > Java version: > Java version "1.8.0_60" > Java(TM) SE Runtime Environment (build 1.8.0_60-b27) > Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode) > > Server has 128GB of RAM and 32 cores(with HyperThreading). > > GC-related JVM options: > > -XX:InitialHeapSize=107374182400 -XX:+ManagementServer -XX:MaxHeapSize=107374182400 -XX:-OmitStackTraceInFastThrow -XX:+PrintGC > -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+TieredCompilation -XX:+UseCondCardMark -XX:+UseG1GC > -XX:ParallelGCThreads=32 -XX:+UnlockDiagnosticVMOptions -XX:ParGCCardsPerStrideChunk=32678 -XX:+ParallelRefProcEnabled -J-XX:MaxGCPauseMillis=100 > > > Any ideas what can be the reason? > > > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanbeast at gmail.com Mon Oct 12 13:47:02 2015 From: vanbeast at gmail.com (Ivan Shramko) Date: Mon, 12 Oct 2015 16:47:02 +0300 Subject: Very long duration of Finalize marking in the Remark stage of G1GC In-Reply-To: <8B24130B-764C-4594-B588-771235CDCF55@oracle.com> References: <8B24130B-764C-4594-B588-771235CDCF55@oracle.com> Message-ID: Hi Charlie! Thanks for the reply. We use Linux (CentOS 6 64 bit). I didn't notice any swapping activity while server was running. But transparent huge pages option was enabled. I will try to disable it and report if it helps or not. Best regards, Ivan 2015-10-12 15:40 GMT+03:00 charlie hunt : > Hi Ivan, > > The remark pause (in addition to high ?Finalize Marking?) also has a > rather high reported sys CPU time. Usually when I see high sys CPU time, it > is often a symptom of paging to/from virtual memory, or if you are running > on Linux it is a symptom of transparent huge pages being enabled. Fwiw, I > don?t think I?ve seen a high ?Finalize Marking? time like what you have > here. > > Could you confirm you are not paging to/from virtual memory, and if you > are running on Linux, can you also confirm transparent huge pages are > disabled? > > thanks, > > charlie > > > > On Oct 12, 2015, at 4:14 AM, Ivan Shramko wrote: > > Application becomes unresponsive on high load while G1 garbage collector > is in the Remark stage(Finalize marking). > This is part of the gc log: > > {Heap before GC invocations=1558 (full 0): > garbage-first heap total 94371840K, used 83001550K [0x00007f76d0000000, 0x00007f76d2005a00, 0x00007f8d50000000) > region size 32768K, 1234 young (40435712K), 58 survivors (1900544K) > Metaspace used 77160K, capacity 77951K, committed 78080K, reserved 79872K > 204611.028: [G1Ergonomics (Concurrent Cycles) initiate concurrent cycle, reason: concurrent cycle initiation requested] > 2015-10-04T15:00:05.197-0400: 204611.028: [GC pause (GCLocker Initiated GC) (young) (initial-mark) > Desired survivor size 2600468480 bytes, new threshold 2 (max 2) > - age 1: 621290376 bytes, 621290376 total > - age 2: 245159208 bytes, 866449584 total > 204611.029: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 184157, predicted base time: 57.32 ms, remaining time: 142.68 ms, target pause time: 200.00 ms] > 204611.029: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 1176 regions, survivors: 58 regions, predicted young region time: 131.19 ms] > 204611.029: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 1176 regions, survivors: 58 regions, old: 0 regions, predicted pause time: 188.51 ms, target pause time: 200.00 ms] > 2015-10-04T15:00:05.416-0400: 204611.247: [SoftReference, 0 refs, 0.0023502 secs]2015-10-04T15:00:05.418-0400: 204611.249: [WeakReference, 394 refs, 0.0018755 secs]2015-10-04T15:00:05.420-0400: 204611.251: [FinalReference, 91704 refs, 0. > 0081225 secs]2015-10-04T15:00:05.428-0400: 204611.259: [PhantomReference, 29 refs, 1 refs, 0.0048953 secs]2015-10-04T15:00:05.433-0400: 204611.264: [JNI Weak Reference, 0.0000229 secs], 0.2600007 secs] > [Parallel Time: 215.1 ms, GC Workers: 32] > [GC Worker Start (ms): Min: 204611030.3, Avg: 204611030.5, Max: 204611030.6, Diff: 0.4] > [Ext Root Scanning (ms): Min: 2.5, Avg: 3.2, Max: 8.3, Diff: 5.8, Sum: 103.2] > [Update RS (ms): Min: 19.0, Avg: 24.5, Max: 35.3, Diff: 16.3, Sum: 782.9] > [Processed Buffers: Min: 7, Avg: 23.7, Max: 31, Diff: 24, Sum: 758] > [Scan RS (ms): Min: 40.4, Avg: 42.2, Max: 43.0, Diff: 2.6, Sum: 1349.0] > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.2] > [Object Copy (ms): Min: 131.5, Avg: 140.0, Max: 142.4, Diff: 10.9, Sum: 4480.1] > [Termination (ms): Min: 1.9, Avg: 4.1, Max: 4.6, Diff: 2.7, Sum: 130.8] > [Termination Attempts: Min: 262, Avg: 544.9, Max: 599, Diff: 337, Sum: 17437] > [GC Worker Other (ms): Min: 0.0, Avg: 0.2, Max: 0.4, Diff: 0.4, Sum: 7.1] > [GC Worker Total (ms): Min: 213.8, Avg: 214.2, Max: 214.5, Diff: 0.7, Sum: 6853.3] > [GC Worker End (ms): Min: 204611244.4, Avg: 204611244.6, Max: 204611244.8, Diff: 0.4] > [Code Root Fixup: 0.0 ms] > [Code Root Purge: 0.0 ms] > [Clear CT: 8.4 ms] > [Other: 36.4 ms] > [Choose CSet: 0.1 ms] > [Ref Proc: 19.1 ms] > [Ref Enq: 0.8 ms] > [Redirty Cards: 2.0 ms] > [Humongous Register: 0.0 ms] > [Humongous Reclaim: 0.0 ms] > [Free CSet: 11.5 ms] > [Eden: 36.8G(36.7G)->0.0B(35.8G) Survivors: 1856.0M->1568.0M Heap: 79.2G(90.0G)->42.4G(90.0G)] > Heap after GC invocations=1559 (full 0): > garbage-first heap total 94371840K, used 44456740K [0x00007f76d0000000, 0x00007f76d2005a00, 0x00007f8d50000000) > region size 32768K, 49 young (1605632K), 49 survivors (1605632K) > Metaspace used 77160K, capacity 77951K, committed 78080K, reserved 79872K > } > [Times: user=7.11 sys=0.18, real=0.26 secs] > 2015-10-04T15:00:05.457-0400: 204611.288: [GC concurrent-root-region-scan-start] > > 2015-10-04T15:00:05.457-0400: 204611.288: Total time for which application threads were stopped: 0.2607079 seconds, Stopping threads took: 0.0000623 seconds > > 2015-10-04T15:00:05.593-0400: 204611.424: [GC concurrent-root-region-scan-end, 0.1359037 secs] > > 2015-10-04T15:00:05.593-0400: 204611.424: [GC concurrent-mark-start] > > 2015-10-04T15:00:06.238-0400: 204612.069: [GC concurrent-mark-end, 0.6448793 secs] > > 2015-10-04T15:00:06.238-0400: 204612.069: [GC remark 2015-10-04T15:00:06.238-0400: 204612.069: [Finalize Marking, 82.5606577 secs] 2015-10-04T15:01:28.799-0400: 204694.630: [GC ref-proc2015-10-04T15:01:28.799-0400: 204694.630: [SoftRefer > ence, 64 refs, 0.0031041 secs]2015-10-04T15:01:28.802-0400: 204694.633: [WeakReference, 383 refs, 0.0023197 secs]2015-10-04T15:01:28.805-0400: 204694.636: [FinalReference, 100288 refs, 0.0330312 secs]2015-10-04T15:01:28.838-0400: 204694. > 669: [PhantomReference, 2 refs, 37 refs, 0.0030514 secs]2015-10-04T15:01:28.841-0400: 204694.672: [JNI Weak Reference, 0.0000915 secs], 0.0539037 secs] 2015-10-04T15:01:28.853-0400: 204694.684: [Unloading, 0.0130054 secs], 82.6649201 sec > s] > [Times: user=1139.19 sys=393.93, real=82.65 secs] > > 2015-10-04T15:01:28.903-0400: 204694.735: Total time for which application threads were stopped: 82.6655358 seconds, Stopping threads took: 0.0000918 seconds > > 2015-10-04T15:01:28.904-0400: 204694.735: [GC cleanup 42G->42G(90G), 0.1166196 secs] > > > Java version: > Java version "1.8.0_60"Java(TM) SE Runtime Environment (build 1.8.0_60-b27)Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode) > > Server has 128GB of RAM and 32 cores(with HyperThreading). > > GC-related JVM options: > > -XX:InitialHeapSize=107374182400 -XX:+ManagementServer -XX:MaxHeapSize=107374182400 -XX:-OmitStackTraceInFastThrow -XX:+PrintGC > -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+TieredCompilation -XX:+UseCondCardMark -XX:+UseG1GC > -XX:ParallelGCThreads=32 -XX:+UnlockDiagnosticVMOptions -XX:ParGCCardsPerStrideChunk=32678 -XX:+ParallelRefProcEnabled -J-XX:MaxGCPauseMillis=100 > > > Any ideas what can be the reason? > > > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yu.zhang at oracle.com Mon Oct 12 14:48:22 2015 From: yu.zhang at oracle.com (Yu Zhang) Date: Mon, 12 Oct 2015 07:48:22 -0700 Subject: [G1GC] Slow processing JNI Weak References In-Reply-To: References: Message-ID: <561BC836.60108@oracle.com> Keith, I am adding hotspot-gc-dev, hoping some one can give a better explanation. Yes, currently the JNI weak reference processing is single threaded. You mentioned JNI weak reference pause long for remark and mixed gc. For young gc, it is long as well. Just for young gc, the collected set is limited. My guess is you can not make much difference by tuning. There is a possibility for memory leak in the native code. A lot of Solr applications use native memory for cache. Thanks, Jenny On 10/7/2015 11:40 AM, Keith L wrote: > Java: > - JDK 1.8u40 > - UseG1GC, ParallelRefProcEnabled, Xmx12g,Xms12g > > Application: > - Solr 4.10.4 > > Application Details: > Using Solr, which does not use JNI directly although it make heavy use > of nio and MMaps. > Possibly relevant is that we run our application with the yourkit > agent with monitoring disabled by default. > > When using G1GC we are seeing very high processing times for JNI Weak > Reference processing, in both GC Remak phase and during Mixed > collections. Originally we saw high times during WeakReference > processing but adding"-XX:+ParallelRefProcEnabled" flag did away with > this. JNI Weak Reference processing we've seen take as much as 300+ > seconds!, although more commonly we see it taking 20-60 seconds when > it start becoming a problem. Currently JNI Weak Reference are not > processed in parallel in HotSpot so adding the parallel flag has no > affect. (https://bugs.openjdk.java.net/browse/JDK-8072498) > > Generally we don't see the time for processing be a problem until a > few days of uptime. At some point the JNI processing gets more > expensive and time taken climbs up and is expensive nearly every there > is a Remark phase. > > Possibly related: after a heap dump of a running application we > noticed anywhere from a couple hundred to hundreds of thousands of > unreferenced DirectByteBuffers. > > Doing a jstack I've seen as many 16 Million JNI Global References > being reported. Although I haven't been able to find a way to get the > JNI Weak Reference information to be displayed. > > Questions: > 1) What is causing the JNI Weak References? > - Is it from nio? Usage of MMaps, DirectBuffers, etc? > 2) Why does it become worse during the lifetime of the application? > - Is there a leak? > - are medium life objects being moved too early > - Tune G1NewSizePercent? > - Tune Survivor settings? > 3) Are the JNI Weak References being visited but not cleaned during GC? > - Why not? tune G1HeapWastePercent? > 4) How can more debugging be done to find where the references are > coming from > > > As of right now we are experimenting with different G1GC flags, > however without understanding the root cause it sometimes requires > days between experiments for the problem to present itself. > > > We have not tuned any of the options above which disable the adaptive > sizing policy, some options we are actively testing are: > > - Lowering InitiatingHeapOccupancyPercent > - To force dereferenced objects in tenured to be cleaned earlier. > > - Increasing MaxGCPauseMillis > - To encourage policy for the young generation size to be larger > (discouraging premature promotion). > > > Example of remark phases below (full gc log can be provided if required) > > One example log file (however we have seen worse), using flags in the > beginning: > > 1348278.630: [GC pause (G1 Evacuation Pause) (young) (initial-mark) > Desired survivor size 41943040 bytes, new threshold 15 (max 15) > - age 1: 36138128 bytes, 36138128 total > 1348278.676: [SoftReference, 0 refs, 0.0086523 secs]1348278.684: > [WeakReference, 7128 refs, 0.0047846 secs]1348278.689: > [FinalReference, 414 refs, 0.0063207 secs]1348278.695: > [PhantomReference, 0 refs, 376 refs, 0.0129979 secs]1348278.709: [JNI > Weak Reference, 0.1018689 secs], 0.1955299 secs] > [Parallel Time: 39.7 ms, GC Workers: 23] > [GC Worker Start (ms): Min: 1348278633.2, Avg: 1348278633.7, > Max: 1348278634.2, Diff: 1.0] > [Ext Root Scanning (ms): Min: 11.2, Avg: 11.7, Max: 14.9, Diff: > 3.7, Sum: 269.2] > [Update RS (ms): Min: 9.9, Avg: 13.3, Max: 14.0, Diff: 4.1, Sum: > 304.8] > [Processed Buffers: Min: 23, Avg: 66.9, Max: 117, Diff: 94, > Sum: 1538] > [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.6] > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > [Object Copy (ms): Min: 11.8, Avg: 12.4, Max: 12.9, Diff: 1.1, > Sum: 286.3] > [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: > 0.3] > [GC Worker Other (ms): Min: 0.1, Avg: 0.3, Max: 0.7, Diff: 0.6, > Sum: 6.2] > [GC Worker Total (ms): Min: 37.0, Avg: 37.7, Max: 38.6, Diff: > 1.6, Sum: 867.4] > [GC Worker End (ms): Min: 1348278671.2, Avg: 1348278671.4, Max: > 1348278671.8, Diff: 0.6] > [Code Root Fixup: 0.4 ms] > [Code Root Purge: 0.0 ms] > [Clear CT: 2.8 ms] > [Other: 152.6 ms] > [Choose CSet: 0.0 ms] > [Ref Proc: 137.2 ms] > [Ref Enq: 2.6 ms] > [Redirty Cards: 3.2 ms] > [Humongous Reclaim: 0.0 ms] > [Free CSet: 0.6 ms] > [Eden: 532.0M(532.0M)->0.0B(532.0M) Survivors: 80.0M->80.0M Heap: > 6225.6M(12.0G)->5701.0M(12.0G)] > [Times: user=0.80 sys=0.04, real=0.20 secs] > 1348278.827: [GC concurrent-root-region-scan-start] > 1348278.828: Total time for which application threads were stopped: > 0.2145574 seconds, Stopping threads took: 0.0012976 seconds > 1348278.844: [GC concurrent-root-region-scan-end, 0.0170768 secs] > 1348278.844: [GC concurrent-mark-start] > 1348279.075: [GC concurrent-mark-end, 0.2306171 secs] > 1348279.088: [GC remark 1348279.088: [Finalize Marking, 0.0051946 > secs] 1348279.093: [GC ref-proc1348279.093: [SoftReference, 186 refs, > 0.0089505 secs]1348279.102: [WeakReference, 2668 refs, 0.0051839 > secs]1348279.107: [FinalReference, 848 refs, 0.0088167 > secs]1348279.116: [PhantomReference, 0 refs, 3836 refs, 0.0130326 > secs]1348279.129: [JNI Weak Reference, 18.9741270 secs], 19.0130579 > secs] 1348298.106: [Unloading, 0.0177559 secs], 19.0412558 secs] > [Times: user=19.59 sys=0.00, real=19.04 secs] > 1348298.132: Total time for which application threads were stopped: > 19.0566871 seconds, Stopping threads took: 0.0011100 seconds > 1348298.147: [GC cleanup 5836M->4256M(12G), 0.0124121 secs] > [Times: user=0.11 sys=0.00, real=0.01 secs] > ... > 1348358.227: [GC remark 1348358.227: [Finalize Marking, 0.0042513 > secs] 1348358.231: [GC ref-proc1348358.231: [SoftReference, 197 refs, > 0.0085051 secs]1348358.240: [WeakReference, 5962 refs, 0.0082517 > secs]1348358.248: [FinalReference, 1881 refs, 0.0074058 > secs]1348358.255: [PhantomReference, 0 refs, 3802 refs, 0.0116230 > secs]1348358.267: [JNI Weak Reference, 29.3914835 secs], 29.4310975 > secs] 1348387.662: [Unloading, 0.0179274 secs], 29.4583167 secs] > 1350907.355: [GC remark 1350907.355: [Finalize Marking, 0.0066394 > secs] 1350907.362: [GC ref-proc1350907.362: [SoftReference, 189 refs, > 0.0084702 secs]1350907.370: [WeakReference, 2379 refs, 0.0046033 > secs]1350907.375: [FinalReference, 779 refs, 0.0086055 > secs]1350907.383: [PhantomReference, 0 refs, 4349 refs, 0.0124574 > secs]1350907.396: [JNI Weak Reference, 29.9550822 secs], 29.9923290 > secs] 1350937.354: [Unloading, 0.0171185 secs], 30.0213910 secs] > ... > ... > ... > 1377495.992: [GC remark 1377495.992: [Finalize Marking, 0.0039972 > secs] 1377495.996: [GC ref-proc1377495.996: [SoftReference, 192 refs, > 0.0084780 secs]1377496.005: [WeakReference, 1171 refs, 0.0040591 > secs]1377496.009: [FinalReference, 619 refs, 0.0068656 > secs]1377496.016: [PhantomReference, 0 refs, 4886 refs, 0.0094143 > secs]1377496.025: [JNI Weak Reference, 48.3669099 secs], 48.3980356 > secs] 1377544.394: [Unloading, 0.0159881 secs], 48.4228951 secs] > 1378034.868: [GC remark 1378034.868: [Finalize Marking, 0.0033629 > secs] 1378034.871: [GC ref-proc1378034.871: [SoftReference, 193 refs, > 0.0059063 secs]1378034.877: [WeakReference, 1332 refs, 0.0036461 > secs]1378034.881: [FinalReference, 611 refs, 0.0046526 > secs]1378034.885: [PhantomReference, 0 refs, 4254 refs, 0.0059437 > secs]1378034.891: [JNI Weak Reference, 46.6392836 secs], 46.6624796 > secs] 1378081.534: [Unloading, 0.0180802 secs], 46.6893835 secs] > 1378677.326: [GC remark 1378677.326: [Finalize Marking, 0.0039769 > secs] 1378677.330: [GC ref-proc1378677.330: [SoftReference, 193 refs, > 0.0080377 secs]1378677.338: [WeakReference, 1719 refs, 0.0035972 > secs]1378677.341: [FinalReference, 351 refs, 0.0063767 > secs]1378677.348: [PhantomReference, 0 refs, 4753 refs, 0.0103290 > secs]1378677.358: [JNI Weak Reference, 43.9230851 secs], 43.9537699 > secs] 1378721.284: [Unloading, 0.0150821 secs], 43.9780128 secs] > 1379626.009: [GC remark 1379626.009: [Finalize Marking, 0.0040175 > secs] 1379626.013: [GC ref-proc1379626.013: [SoftReference, 186 refs, > 0.0080680 secs]1379626.021: [WeakReference, 955 refs, 0.0064521 > secs]1379626.028: [FinalReference, 567 refs, 0.0098877 > secs]1379626.038: [PhantomReference, 0 refs, 4701 refs, 0.0104484 > secs]1379626.048: [JNI Weak Reference, 51.3181908 secs], 51.3558998 > secs] 1379677.369: [Unloading, 0.0207052 secs], 51.3856987 secs] > ... > 1380251.013: [GC pause (G1 Evacuation Pause) (young) (initial-mark) > Desired survivor size 41943040 bytes, new threshold 1 (max 15) > - age 1: 46612512 bytes, 46612512 total > [SoftReference, 0 refs, 0.0074261 secs]1380251.048: [WeakReference, > 5189 refs, 0.0041909 secs]1380251.052: [FinalReference, 393 refs, > 0.0042160 secs]1380251.056: [PhantomReference, 0 refs, 83 refs, > 0.0074198 secs]1380251.064: [JNI Weak Reference, 3.0041020 secs], > 3.0678505 secs] > [Parallel Time: 23.5 ms, GC Workers: 23] > [GC Worker Start (ms): Min: 1380251014.8, Avg: 1380251015.3, > Max: 1380251015.9, Diff: 1.1] > [Ext Root Scanning (ms): Min: 6.7, Avg: 7.5, Max: 10.9, Diff: > 4.2, Sum: 173.4] > [Update RS (ms): Min: 3.7, Avg: 6.5, Max: 7.0, Diff: 3.3, Sum: > 150.5] > [Processed Buffers: Min: 27, Avg: 66.7, Max: 112, Diff: 85, > Sum: 1534] > [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.6] > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > [Object Copy (ms): Min: 6.3, Avg: 6.7, Max: 7.3, Diff: 1.0, Sum: > 154.9] > [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: > 0.1] > [GC Worker Other (ms): Min: 0.1, Avg: 0.4, Max: 0.8, Diff: 0.8, > Sum: 8.3] > [GC Worker Total (ms): Min: 20.4, Avg: 21.2, Max: 21.8, Diff: > 1.4, Sum: 487.7] > [GC Worker End (ms): Min: 1380251036.2, Avg: 1380251036.5, Max: > 1380251037.0, Diff: 0.8] > [Code Root Fixup: 0.4 ms] > [Code Root Purge: 0.0 ms] > [Clear CT: 2.9 ms] > [Other: 3041.1 ms] > [Choose CSet: 0.0 ms] > [Ref Proc: 3029.5 ms] > [Ref Enq: 2.7 ms] > [Redirty Cards: 2.7 ms] > [Humongous Reclaim: 0.0 ms] > [Free CSet: 0.3 ms] > [Eden: 532.0M(532.0M)->0.0B(548.0M) Survivors: 80.0M->64.0M Heap: > 6142.5M(12.0G)->5620.7M(12.0G)] > [Times: user=3.46 sys=0.00, real=3.07 secs] > 1380254.081: [GC concurrent-root-region-scan-start] > 1380254.082: Total time for which application threads were stopped: > 3.0847197 seconds, Stopping threads took: 0.0011519 seconds > 1380254.092: [GC concurrent-root-region-scan-end, 0.0105392 secs] > 1380254.092: [GC concurrent-mark-start] > 1380254.275: [GC concurrent-mark-end, 0.1832853 secs] > 1380254.289: [GC remark 1380254.290: [Finalize Marking, 0.0039580 > secs] 1380254.294: [GC ref-proc1380254.294: [SoftReference, 183 refs, > 0.0068857 secs]1380254.300: [WeakReference, 1384 refs, 0.0043966 > secs]1380254.305: [FinalReference, 579 refs, 0.0073469 > secs]1380254.312: [PhantomReference, 0 refs, 4327 refs, 0.0077366 > secs]1380254.320: [JNI Weak Reference, 38.7211438 secs], 38.7506058 > secs] 1380293.044: [Unloading, 0.0174879 secs], 38.7765019 secs] > [Times: user=39.37 sys=0.00, real=38.78 secs] > 1380293.067: Total time for which application threads were stopped: > 38.7917823 seconds, Stopping threads took: 0.0011561 seconds > 1380293.077: [GC cleanup 5659M->4167M(12G), 0.0120391 secs] > [Times: user=0.11 sys=0.00, real=0.01 secs] > > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: From kelaban17 at gmail.com Mon Oct 12 14:54:42 2015 From: kelaban17 at gmail.com (Keith L) Date: Mon, 12 Oct 2015 14:54:42 +0000 Subject: [G1GC] Slow processing JNI Weak References In-Reply-To: <561BC836.60108@oracle.com> References: <561BC836.60108@oracle.com> Message-ID: I'm not too familiar with debugging native code, what is the best way to try and find what objects are being weak/global referenced? On Mon, Oct 12, 2015 at 10:48 AM Yu Zhang wrote: > Keith, > > I am adding hotspot-gc-dev, hoping some one can give a better explanation. > > Yes, currently the JNI weak reference processing is single threaded. > You mentioned JNI weak reference pause long for remark and mixed gc. For > young gc, it is long as well. Just for young gc, the collected set is > limited. My guess is you can not make much difference by tuning. > > There is a possibility for memory leak in the native code. A lot of Solr > applications use native memory for cache. > > Thanks, > Jenny > > On 10/7/2015 11:40 AM, Keith L wrote: > > Java: > - JDK 1.8u40 > - UseG1GC, ParallelRefProcEnabled, Xmx12g,Xms12g > > Application: > - Solr 4.10.4 > > Application Details: > Using Solr, which does not use JNI directly although it make heavy use of > nio and MMaps. > Possibly relevant is that we run our application with the yourkit agent > with monitoring disabled by default. > > When using G1GC we are seeing very high processing times for JNI Weak > Reference processing, in both GC Remak phase and during Mixed collections. > Originally we saw high times during WeakReference processing but > adding"-XX:+ParallelRefProcEnabled" flag did away with this. JNI Weak > Reference processing we've seen take as much as 300+ seconds!, although > more commonly we see it taking 20-60 seconds when it start becoming a > problem. Currently JNI Weak Reference are not processed in parallel in > HotSpot so adding the parallel flag has no affect. ( > https://bugs.openjdk.java.net/browse/JDK-8072498) > > Generally we don't see the time for processing be a problem until a few > days of uptime. At some point the JNI processing gets more expensive and > time taken climbs up and is expensive nearly every there is a Remark phase. > > Possibly related: after a heap dump of a running application we noticed > anywhere from a couple hundred to hundreds of thousands of unreferenced > DirectByteBuffers. > > Doing a jstack I've seen as many 16 Million JNI Global References being > reported. Although I haven't been able to find a way to get the JNI Weak > Reference information to be displayed. > > Questions: > 1) What is causing the JNI Weak References? > - Is it from nio? Usage of MMaps, DirectBuffers, etc? > 2) Why does it become worse during the lifetime of the application? > - Is there a leak? > - are medium life objects being moved too early > - Tune G1NewSizePercent? > - Tune Survivor settings? > 3) Are the JNI Weak References being visited but not cleaned during GC? > - Why not? tune G1HeapWastePercent? > 4) How can more debugging be done to find where the references are coming > from > > > As of right now we are experimenting with different G1GC flags, however > without understanding the root cause it sometimes requires days between > experiments for the problem to present itself. > > > We have not tuned any of the options above which disable the adaptive > sizing policy, some options we are actively testing are: > > - Lowering InitiatingHeapOccupancyPercent > - To force dereferenced objects in tenured to be cleaned earlier. > > - Increasing MaxGCPauseMillis > - To encourage policy for the young generation size to be larger > (discouraging premature promotion). > > > Example of remark phases below (full gc log can be provided if required) > > One example log file (however we have seen worse), using flags in the > beginning: > > 1348278.630: [GC pause (G1 Evacuation Pause) (young) (initial-mark) > Desired survivor size 41943040 bytes, new threshold 15 (max 15) > - age 1: 36138128 bytes, 36138128 total > 1348278.676: [SoftReference, 0 refs, 0.0086523 secs]1348278.684: > [WeakReference, 7128 refs, 0.0047846 secs]1348278.689: [FinalReference, 414 > refs, 0.0063207 secs]1348278.695: [PhantomReference, 0 refs, 376 refs, > 0.0129979 secs]1348278.709: [JNI Weak Reference, 0.1018689 secs], 0.1955299 > secs] > [Parallel Time: 39.7 ms, GC Workers: 23] > [GC Worker Start (ms): Min: 1348278633.2, Avg: 1348278633.7, Max: > 1348278634.2, Diff: 1.0] > [Ext Root Scanning (ms): Min: 11.2, Avg: 11.7, Max: 14.9, Diff: 3.7, > Sum: 269.2] > [Update RS (ms): Min: 9.9, Avg: 13.3, Max: 14.0, Diff: 4.1, Sum: > 304.8] > [Processed Buffers: Min: 23, Avg: 66.9, Max: 117, Diff: 94, Sum: > 1538] > [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.6] > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, > Sum: 0.0] > [Object Copy (ms): Min: 11.8, Avg: 12.4, Max: 12.9, Diff: 1.1, Sum: > 286.3] > [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.3] > [GC Worker Other (ms): Min: 0.1, Avg: 0.3, Max: 0.7, Diff: 0.6, Sum: > 6.2] > [GC Worker Total (ms): Min: 37.0, Avg: 37.7, Max: 38.6, Diff: 1.6, > Sum: 867.4] > [GC Worker End (ms): Min: 1348278671.2, Avg: 1348278671.4, Max: > 1348278671.8, Diff: 0.6] > [Code Root Fixup: 0.4 ms] > [Code Root Purge: 0.0 ms] > [Clear CT: 2.8 ms] > [Other: 152.6 ms] > [Choose CSet: 0.0 ms] > [Ref Proc: 137.2 ms] > [Ref Enq: 2.6 ms] > [Redirty Cards: 3.2 ms] > [Humongous Reclaim: 0.0 ms] > [Free CSet: 0.6 ms] > [Eden: 532.0M(532.0M)->0.0B(532.0M) Survivors: 80.0M->80.0M Heap: > 6225.6M(12.0G)->5701.0M(12.0G)] > [Times: user=0.80 sys=0.04, real=0.20 secs] > 1348278.827: [GC concurrent-root-region-scan-start] > 1348278.828: Total time for which application threads were stopped: > 0.2145574 seconds, Stopping threads took: 0.0012976 seconds > 1348278.844: [GC concurrent-root-region-scan-end, 0.0170768 secs] > 1348278.844: [GC concurrent-mark-start] > 1348279.075: [GC concurrent-mark-end, 0.2306171 secs] > 1348279.088: [GC remark 1348279.088: [Finalize Marking, 0.0051946 secs] > 1348279.093: [GC ref-proc1348279.093: [SoftReference, 186 refs, 0.0089505 > secs]1348279.102: [WeakReference, 2668 refs, 0.0051839 secs]1348279.107: > [FinalReference, 848 refs, 0.0088167 secs]1348279.116: [PhantomReference, 0 > refs, 3836 refs, 0.0130326 secs]1348279.129: [JNI Weak Reference, > 18.9741270 secs], 19.0130579 secs] 1348298.106: [Unloading, 0.0177559 > secs], 19.0412558 secs] > [Times: user=19.59 sys=0.00, real=19.04 secs] > 1348298.132: Total time for which application threads were stopped: > 19.0566871 seconds, Stopping threads took: 0.0011100 seconds > 1348298.147: [GC cleanup 5836M->4256M(12G), 0.0124121 secs] > [Times: user=0.11 sys=0.00, real=0.01 secs] > ... > 1348358.227: [GC remark 1348358.227: [Finalize Marking, 0.0042513 secs] > 1348358.231: [GC ref-proc1348358.231: [SoftReference, 197 refs, 0.0085051 > secs]1348358.240: [WeakReference, 5962 refs, 0.0082517 secs]1348358.248: > [FinalReference, 1881 refs, 0.0074058 secs]1348358.255: [PhantomReference, > 0 refs, 3802 refs, 0.0116230 secs]1348358.267: [JNI Weak Reference, > 29.3914835 secs], 29.4310975 secs] 1348387.662: [Unloading, 0.0179274 > secs], 29.4583167 secs] > 1350907.355: [GC remark 1350907.355: [Finalize Marking, 0.0066394 secs] > 1350907.362: [GC ref-proc1350907.362: [SoftReference, 189 refs, 0.0084702 > secs]1350907.370: [WeakReference, 2379 refs, 0.0046033 secs]1350907.375: > [FinalReference, 779 refs, 0.0086055 secs]1350907.383: [PhantomReference, 0 > refs, 4349 refs, 0.0124574 secs]1350907.396: [JNI Weak Reference, > 29.9550822 secs], 29.9923290 secs] 1350937.354: [Unloading, 0.0171185 > secs], 30.0213910 secs] > ... > ... > ... > 1377495.992: [GC remark 1377495.992: [Finalize Marking, 0.0039972 secs] > 1377495.996: [GC ref-proc1377495.996: [SoftReference, 192 refs, 0.0084780 > secs]1377496.005: [WeakReference, 1171 refs, 0.0040591 secs]1377496.009: > [FinalReference, 619 refs, 0.0068656 secs]1377496.016: [PhantomReference, 0 > refs, 4886 refs, 0.0094143 secs]1377496.025: [JNI Weak Reference, > 48.3669099 secs], 48.3980356 secs] 1377544.394: [Unloading, 0.0159881 > secs], 48.4228951 secs] > 1378034.868: [GC remark 1378034.868: [Finalize Marking, 0.0033629 secs] > 1378034.871: [GC ref-proc1378034.871: [SoftReference, 193 refs, 0.0059063 > secs]1378034.877: [WeakReference, 1332 refs, 0.0036461 secs]1378034.881: > [FinalReference, 611 refs, 0.0046526 secs]1378034.885: [PhantomReference, 0 > refs, 4254 refs, 0.0059437 secs]1378034.891: [JNI Weak Reference, > 46.6392836 secs], 46.6624796 secs] 1378081.534: [Unloading, 0.0180802 > secs], 46.6893835 secs] > 1378677.326: [GC remark 1378677.326: [Finalize Marking, 0.0039769 secs] > 1378677.330: [GC ref-proc1378677.330: [SoftReference, 193 refs, 0.0080377 > secs]1378677.338: [WeakReference, 1719 refs, 0.0035972 secs]1378677.341: > [FinalReference, 351 refs, 0.0063767 secs]1378677.348: [PhantomReference, 0 > refs, 4753 refs, 0.0103290 secs]1378677.358: [JNI Weak Reference, > 43.9230851 secs], 43.9537699 secs] 1378721.284: [Unloading, 0.0150821 > secs], 43.9780128 secs] > 1379626.009: [GC remark 1379626.009: [Finalize Marking, 0.0040175 secs] > 1379626.013: [GC ref-proc1379626.013: [SoftReference, 186 refs, 0.0080680 > secs]1379626.021: [WeakReference, 955 refs, 0.0064521 secs]1379626.028: > [FinalReference, 567 refs, 0.0098877 secs]1379626.038: [PhantomReference, 0 > refs, 4701 refs, 0.0104484 secs]1379626.048: [JNI Weak Reference, > 51.3181908 secs], 51.3558998 secs] 1379677.369: [Unloading, 0.0207052 > secs], 51.3856987 secs] > ... > 1380251.013: [GC pause (G1 Evacuation Pause) (young) (initial-mark) > Desired survivor size 41943040 bytes, new threshold 1 (max 15) > - age 1: 46612512 bytes, 46612512 total > [SoftReference, 0 refs, 0.0074261 secs]1380251.048: [WeakReference, 5189 > refs, 0.0041909 secs]1380251.052: [FinalReference, 393 refs, 0.0042160 > secs]1380251.056: [PhantomReference, 0 refs, 83 refs, 0.0074198 > secs]1380251.064: [JNI Weak Reference, 3.0041020 secs], 3.0678505 secs] > [Parallel Time: 23.5 ms, GC Workers: 23] > [GC Worker Start (ms): Min: 1380251014.8, Avg: 1380251015.3, Max: > 1380251015.9, Diff: 1.1] > [Ext Root Scanning (ms): Min: 6.7, Avg: 7.5, Max: 10.9, Diff: 4.2, > Sum: 173.4] > [Update RS (ms): Min: 3.7, Avg: 6.5, Max: 7.0, Diff: 3.3, Sum: 150.5] > [Processed Buffers: Min: 27, Avg: 66.7, Max: 112, Diff: 85, Sum: > 1534] > [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.6] > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, > Sum: 0.0] > [Object Copy (ms): Min: 6.3, Avg: 6.7, Max: 7.3, Diff: 1.0, Sum: > 154.9] > [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] > [GC Worker Other (ms): Min: 0.1, Avg: 0.4, Max: 0.8, Diff: 0.8, Sum: > 8.3] > [GC Worker Total (ms): Min: 20.4, Avg: 21.2, Max: 21.8, Diff: 1.4, > Sum: 487.7] > [GC Worker End (ms): Min: 1380251036.2, Avg: 1380251036.5, Max: > 1380251037.0, Diff: 0.8] > [Code Root Fixup: 0.4 ms] > [Code Root Purge: 0.0 ms] > [Clear CT: 2.9 ms] > [Other: 3041.1 ms] > [Choose CSet: 0.0 ms] > [Ref Proc: 3029.5 ms] > [Ref Enq: 2.7 ms] > [Redirty Cards: 2.7 ms] > [Humongous Reclaim: 0.0 ms] > [Free CSet: 0.3 ms] > [Eden: 532.0M(532.0M)->0.0B(548.0M) Survivors: 80.0M->64.0M Heap: > 6142.5M(12.0G)->5620.7M(12.0G)] > [Times: user=3.46 sys=0.00, real=3.07 secs] > 1380254.081: [GC concurrent-root-region-scan-start] > 1380254.082: Total time for which application threads were stopped: > 3.0847197 seconds, Stopping threads took: 0.0011519 seconds > 1380254.092: [GC concurrent-root-region-scan-end, 0.0105392 secs] > 1380254.092: [GC concurrent-mark-start] > 1380254.275: [GC concurrent-mark-end, 0.1832853 secs] > 1380254.289: [GC remark 1380254.290: [Finalize Marking, 0.0039580 secs] > 1380254.294: [GC ref-proc1380254.294: [SoftReference, 183 refs, 0.0068857 > secs]1380254.300: [WeakReference, 1384 refs, 0.0043966 secs]1380254.305: > [FinalReference, 579 refs, 0.0073469 secs]1380254.312: [PhantomReference, 0 > refs, 4327 refs, 0.0077366 secs]1380254.320: [JNI Weak Reference, > 38.7211438 secs], 38.7506058 secs] 1380293.044: [Unloading, 0.0174879 > secs], 38.7765019 secs] > [Times: user=39.37 sys=0.00, real=38.78 secs] > 1380293.067: Total time for which application threads were stopped: > 38.7917823 seconds, Stopping threads took: 0.0011561 seconds > 1380293.077: [GC cleanup 5659M->4167M(12G), 0.0120391 secs] > [Times: user=0.11 sys=0.00, real=0.01 secs] > > > > _______________________________________________ > hotspot-gc-use mailing listhotspot-gc-use at openjdk.java.nethttp://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanbeast at gmail.com Wed Oct 14 09:47:38 2015 From: vanbeast at gmail.com (Ivan Shramko) Date: Wed, 14 Oct 2015 12:47:38 +0300 Subject: Very long duration of Finalize marking in the Remark stage of G1GC In-Reply-To: References: <8B24130B-764C-4594-B588-771235CDCF55@oracle.com> Message-ID: Hi Charlie! I disabled transparent huge pages but it didn't help. We are still having repeating pauses in Finalize Marking. Best regards, Ivan 2015-10-12 16:47 GMT+03:00 Ivan Shramko : > Hi Charlie! > > Thanks for the reply. We use Linux (CentOS 6 64 bit). I didn't notice any > swapping activity while server was running. But transparent huge pages > option was enabled. I will try to disable it and report if it helps or not. > > Best regards, > Ivan > > 2015-10-12 15:40 GMT+03:00 charlie hunt : > >> Hi Ivan, >> >> The remark pause (in addition to high ?Finalize Marking?) also has a >> rather high reported sys CPU time. Usually when I see high sys CPU time, it >> is often a symptom of paging to/from virtual memory, or if you are running >> on Linux it is a symptom of transparent huge pages being enabled. Fwiw, I >> don?t think I?ve seen a high ?Finalize Marking? time like what you have >> here. >> >> Could you confirm you are not paging to/from virtual memory, and if you >> are running on Linux, can you also confirm transparent huge pages are >> disabled? >> >> thanks, >> >> charlie >> >> >> >> On Oct 12, 2015, at 4:14 AM, Ivan Shramko wrote: >> >> Application becomes unresponsive on high load while G1 garbage collector >> is in the Remark stage(Finalize marking). >> This is part of the gc log: >> >> {Heap before GC invocations=1558 (full 0): >> garbage-first heap total 94371840K, used 83001550K [0x00007f76d0000000, 0x00007f76d2005a00, 0x00007f8d50000000) >> region size 32768K, 1234 young (40435712K), 58 survivors (1900544K) >> Metaspace used 77160K, capacity 77951K, committed 78080K, reserved 79872K >> 204611.028: [G1Ergonomics (Concurrent Cycles) initiate concurrent cycle, reason: concurrent cycle initiation requested] >> 2015-10-04T15:00:05.197-0400: 204611.028: [GC pause (GCLocker Initiated GC) (young) (initial-mark) >> Desired survivor size 2600468480 bytes, new threshold 2 (max 2) >> - age 1: 621290376 bytes, 621290376 total >> - age 2: 245159208 bytes, 866449584 total >> 204611.029: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 184157, predicted base time: 57.32 ms, remaining time: 142.68 ms, target pause time: 200.00 ms] >> 204611.029: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 1176 regions, survivors: 58 regions, predicted young region time: 131.19 ms] >> 204611.029: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 1176 regions, survivors: 58 regions, old: 0 regions, predicted pause time: 188.51 ms, target pause time: 200.00 ms] >> 2015-10-04T15:00:05.416-0400: 204611.247: [SoftReference, 0 refs, 0.0023502 secs]2015-10-04T15:00:05.418-0400: 204611.249: [WeakReference, 394 refs, 0.0018755 secs]2015-10-04T15:00:05.420-0400: 204611.251: [FinalReference, 91704 refs, 0. >> 0081225 secs]2015-10-04T15:00:05.428-0400: 204611.259: [PhantomReference, 29 refs, 1 refs, 0.0048953 secs]2015-10-04T15:00:05.433-0400: 204611.264: [JNI Weak Reference, 0.0000229 secs], 0.2600007 secs] >> [Parallel Time: 215.1 ms, GC Workers: 32] >> [GC Worker Start (ms): Min: 204611030.3, Avg: 204611030.5, Max: 204611030.6, Diff: 0.4] >> [Ext Root Scanning (ms): Min: 2.5, Avg: 3.2, Max: 8.3, Diff: 5.8, Sum: 103.2] >> [Update RS (ms): Min: 19.0, Avg: 24.5, Max: 35.3, Diff: 16.3, Sum: 782.9] >> [Processed Buffers: Min: 7, Avg: 23.7, Max: 31, Diff: 24, Sum: 758] >> [Scan RS (ms): Min: 40.4, Avg: 42.2, Max: 43.0, Diff: 2.6, Sum: 1349.0] >> [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.2] >> [Object Copy (ms): Min: 131.5, Avg: 140.0, Max: 142.4, Diff: 10.9, Sum: 4480.1] >> [Termination (ms): Min: 1.9, Avg: 4.1, Max: 4.6, Diff: 2.7, Sum: 130.8] >> [Termination Attempts: Min: 262, Avg: 544.9, Max: 599, Diff: 337, Sum: 17437] >> [GC Worker Other (ms): Min: 0.0, Avg: 0.2, Max: 0.4, Diff: 0.4, Sum: 7.1] >> [GC Worker Total (ms): Min: 213.8, Avg: 214.2, Max: 214.5, Diff: 0.7, Sum: 6853.3] >> [GC Worker End (ms): Min: 204611244.4, Avg: 204611244.6, Max: 204611244.8, Diff: 0.4] >> [Code Root Fixup: 0.0 ms] >> [Code Root Purge: 0.0 ms] >> [Clear CT: 8.4 ms] >> [Other: 36.4 ms] >> [Choose CSet: 0.1 ms] >> [Ref Proc: 19.1 ms] >> [Ref Enq: 0.8 ms] >> [Redirty Cards: 2.0 ms] >> [Humongous Register: 0.0 ms] >> [Humongous Reclaim: 0.0 ms] >> [Free CSet: 11.5 ms] >> [Eden: 36.8G(36.7G)->0.0B(35.8G) Survivors: 1856.0M->1568.0M Heap: 79.2G(90.0G)->42.4G(90.0G)] >> Heap after GC invocations=1559 (full 0): >> garbage-first heap total 94371840K, used 44456740K [0x00007f76d0000000, 0x00007f76d2005a00, 0x00007f8d50000000) >> region size 32768K, 49 young (1605632K), 49 survivors (1605632K) >> Metaspace used 77160K, capacity 77951K, committed 78080K, reserved 79872K >> } >> [Times: user=7.11 sys=0.18, real=0.26 secs] >> 2015-10-04T15:00:05.457-0400: 204611.288: [GC concurrent-root-region-scan-start] >> >> 2015-10-04T15:00:05.457-0400: 204611.288: Total time for which application threads were stopped: 0.2607079 seconds, Stopping threads took: 0.0000623 seconds >> >> 2015-10-04T15:00:05.593-0400: 204611.424: [GC concurrent-root-region-scan-end, 0.1359037 secs] >> >> 2015-10-04T15:00:05.593-0400: 204611.424: [GC concurrent-mark-start] >> >> 2015-10-04T15:00:06.238-0400: 204612.069: [GC concurrent-mark-end, 0.6448793 secs] >> >> 2015-10-04T15:00:06.238-0400: 204612.069: [GC remark 2015-10-04T15:00:06.238-0400: 204612.069: [Finalize Marking, 82.5606577 secs] 2015-10-04T15:01:28.799-0400: 204694.630: [GC ref-proc2015-10-04T15:01:28.799-0400: 204694.630: [SoftRefer >> ence, 64 refs, 0.0031041 secs]2015-10-04T15:01:28.802-0400: 204694.633: [WeakReference, 383 refs, 0.0023197 secs]2015-10-04T15:01:28.805-0400: 204694.636: [FinalReference, 100288 refs, 0.0330312 secs]2015-10-04T15:01:28.838-0400: 204694. >> 669: [PhantomReference, 2 refs, 37 refs, 0.0030514 secs]2015-10-04T15:01:28.841-0400: 204694.672: [JNI Weak Reference, 0.0000915 secs], 0.0539037 secs] 2015-10-04T15:01:28.853-0400: 204694.684: [Unloading, 0.0130054 secs], 82.6649201 sec >> s] >> [Times: user=1139.19 sys=393.93, real=82.65 secs] >> >> 2015-10-04T15:01:28.903-0400: 204694.735: Total time for which application threads were stopped: 82.6655358 seconds, Stopping threads took: 0.0000918 seconds >> >> 2015-10-04T15:01:28.904-0400: 204694.735: [GC cleanup 42G->42G(90G), 0.1166196 secs] >> >> >> Java version: >> Java version "1.8.0_60"Java(TM) SE Runtime Environment (build 1.8.0_60-b27)Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode) >> >> Server has 128GB of RAM and 32 cores(with HyperThreading). >> >> GC-related JVM options: >> >> -XX:InitialHeapSize=107374182400 -XX:+ManagementServer -XX:MaxHeapSize=107374182400 -XX:-OmitStackTraceInFastThrow -XX:+PrintGC >> -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+TieredCompilation -XX:+UseCondCardMark -XX:+UseG1GC >> -XX:ParallelGCThreads=32 -XX:+UnlockDiagnosticVMOptions -XX:ParGCCardsPerStrideChunk=32678 -XX:+ParallelRefProcEnabled -J-XX:MaxGCPauseMillis=100 >> >> >> Any ideas what can be the reason? >> >> >> >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ceeaspb at gmail.com Wed Oct 14 10:10:27 2015 From: ceeaspb at gmail.com (Alex Bagehot) Date: Wed, 14 Oct 2015 11:10:27 +0100 Subject: Very long duration of Finalize marking in the Remark stage of G1GC In-Reply-To: References: <8B24130B-764C-4594-B588-771235CDCF55@oracle.com> Message-ID: Hi Ivan, Just to be sure can you confirm it is disabled *and* no longer in use? There's a risk it was disabled but still in use, see below: https://access.redhat.com/solutions/46111 https://access.redhat.com/solutions/422283 Thanks, Alex On Wed, Oct 14, 2015 at 10:47 AM, Ivan Shramko wrote: > Hi Charlie! > I disabled transparent huge pages but it didn't help. We are still having > repeating pauses in Finalize Marking. > Best regards, > Ivan > > 2015-10-12 16:47 GMT+03:00 Ivan Shramko : > >> Hi Charlie! >> >> Thanks for the reply. We use Linux (CentOS 6 64 bit). I didn't notice any >> swapping activity while server was running. But transparent huge pages >> option was enabled. I will try to disable it and report if it helps or not. >> >> Best regards, >> Ivan >> >> 2015-10-12 15:40 GMT+03:00 charlie hunt : >> >>> Hi Ivan, >>> >>> The remark pause (in addition to high ?Finalize Marking?) also has a >>> rather high reported sys CPU time. Usually when I see high sys CPU time, it >>> is often a symptom of paging to/from virtual memory, or if you are running >>> on Linux it is a symptom of transparent huge pages being enabled. Fwiw, I >>> don?t think I?ve seen a high ?Finalize Marking? time like what you have >>> here. >>> >>> Could you confirm you are not paging to/from virtual memory, and if you >>> are running on Linux, can you also confirm transparent huge pages are >>> disabled? >>> >>> thanks, >>> >>> charlie >>> >>> >>> >>> On Oct 12, 2015, at 4:14 AM, Ivan Shramko wrote: >>> >>> Application becomes unresponsive on high load while G1 garbage collector >>> is in the Remark stage(Finalize marking). >>> This is part of the gc log: >>> >>> {Heap before GC invocations=1558 (full 0): >>> garbage-first heap total 94371840K, used 83001550K [0x00007f76d0000000, 0x00007f76d2005a00, 0x00007f8d50000000) >>> region size 32768K, 1234 young (40435712K), 58 survivors (1900544K) >>> Metaspace used 77160K, capacity 77951K, committed 78080K, reserved 79872K >>> 204611.028: [G1Ergonomics (Concurrent Cycles) initiate concurrent cycle, reason: concurrent cycle initiation requested] >>> 2015-10-04T15:00:05.197-0400: 204611.028: [GC pause (GCLocker Initiated GC) (young) (initial-mark) >>> Desired survivor size 2600468480 bytes, new threshold 2 (max 2) >>> - age 1: 621290376 bytes, 621290376 total >>> - age 2: 245159208 bytes, 866449584 total >>> 204611.029: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 184157, predicted base time: 57.32 ms, remaining time: 142.68 ms, target pause time: 200.00 ms] >>> 204611.029: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 1176 regions, survivors: 58 regions, predicted young region time: 131.19 ms] >>> 204611.029: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 1176 regions, survivors: 58 regions, old: 0 regions, predicted pause time: 188.51 ms, target pause time: 200.00 ms] >>> 2015-10-04T15:00:05.416-0400: 204611.247: [SoftReference, 0 refs, 0.0023502 secs]2015-10-04T15:00:05.418-0400: 204611.249: [WeakReference, 394 refs, 0.0018755 secs]2015-10-04T15:00:05.420-0400: 204611.251: [FinalReference, 91704 refs, 0. >>> 0081225 secs]2015-10-04T15:00:05.428-0400: 204611.259: [PhantomReference, 29 refs, 1 refs, 0.0048953 secs]2015-10-04T15:00:05.433-0400: 204611.264: [JNI Weak Reference, 0.0000229 secs], 0.2600007 secs] >>> [Parallel Time: 215.1 ms, GC Workers: 32] >>> [GC Worker Start (ms): Min: 204611030.3, Avg: 204611030.5, Max: 204611030.6, Diff: 0.4] >>> [Ext Root Scanning (ms): Min: 2.5, Avg: 3.2, Max: 8.3, Diff: 5.8, Sum: 103.2] >>> [Update RS (ms): Min: 19.0, Avg: 24.5, Max: 35.3, Diff: 16.3, Sum: 782.9] >>> [Processed Buffers: Min: 7, Avg: 23.7, Max: 31, Diff: 24, Sum: 758] >>> [Scan RS (ms): Min: 40.4, Avg: 42.2, Max: 43.0, Diff: 2.6, Sum: 1349.0] >>> [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.2] >>> [Object Copy (ms): Min: 131.5, Avg: 140.0, Max: 142.4, Diff: 10.9, Sum: 4480.1] >>> [Termination (ms): Min: 1.9, Avg: 4.1, Max: 4.6, Diff: 2.7, Sum: 130.8] >>> [Termination Attempts: Min: 262, Avg: 544.9, Max: 599, Diff: 337, Sum: 17437] >>> [GC Worker Other (ms): Min: 0.0, Avg: 0.2, Max: 0.4, Diff: 0.4, Sum: 7.1] >>> [GC Worker Total (ms): Min: 213.8, Avg: 214.2, Max: 214.5, Diff: 0.7, Sum: 6853.3] >>> [GC Worker End (ms): Min: 204611244.4, Avg: 204611244.6, Max: 204611244.8, Diff: 0.4] >>> [Code Root Fixup: 0.0 ms] >>> [Code Root Purge: 0.0 ms] >>> [Clear CT: 8.4 ms] >>> [Other: 36.4 ms] >>> [Choose CSet: 0.1 ms] >>> [Ref Proc: 19.1 ms] >>> [Ref Enq: 0.8 ms] >>> [Redirty Cards: 2.0 ms] >>> [Humongous Register: 0.0 ms] >>> [Humongous Reclaim: 0.0 ms] >>> [Free CSet: 11.5 ms] >>> [Eden: 36.8G(36.7G)->0.0B(35.8G) Survivors: 1856.0M->1568.0M Heap: 79.2G(90.0G)->42.4G(90.0G)] >>> Heap after GC invocations=1559 (full 0): >>> garbage-first heap total 94371840K, used 44456740K [0x00007f76d0000000, 0x00007f76d2005a00, 0x00007f8d50000000) >>> region size 32768K, 49 young (1605632K), 49 survivors (1605632K) >>> Metaspace used 77160K, capacity 77951K, committed 78080K, reserved 79872K >>> } >>> [Times: user=7.11 sys=0.18, real=0.26 secs] >>> 2015-10-04T15:00:05.457-0400: 204611.288: [GC concurrent-root-region-scan-start] >>> >>> 2015-10-04T15:00:05.457-0400: 204611.288: Total time for which application threads were stopped: 0.2607079 seconds, Stopping threads took: 0.0000623 seconds >>> >>> 2015-10-04T15:00:05.593-0400: 204611.424: [GC concurrent-root-region-scan-end, 0.1359037 secs] >>> >>> 2015-10-04T15:00:05.593-0400: 204611.424: [GC concurrent-mark-start] >>> >>> 2015-10-04T15:00:06.238-0400: 204612.069: [GC concurrent-mark-end, 0.6448793 secs] >>> >>> 2015-10-04T15:00:06.238-0400: 204612.069: [GC remark 2015-10-04T15:00:06.238-0400: 204612.069: [Finalize Marking, 82.5606577 secs] 2015-10-04T15:01:28.799-0400: 204694.630: [GC ref-proc2015-10-04T15:01:28.799-0400: 204694.630: [SoftRefer >>> ence, 64 refs, 0.0031041 secs]2015-10-04T15:01:28.802-0400: 204694.633: [WeakReference, 383 refs, 0.0023197 secs]2015-10-04T15:01:28.805-0400: 204694.636: [FinalReference, 100288 refs, 0.0330312 secs]2015-10-04T15:01:28.838-0400: 204694. >>> 669: [PhantomReference, 2 refs, 37 refs, 0.0030514 secs]2015-10-04T15:01:28.841-0400: 204694.672: [JNI Weak Reference, 0.0000915 secs], 0.0539037 secs] 2015-10-04T15:01:28.853-0400: 204694.684: [Unloading, 0.0130054 secs], 82.6649201 sec >>> s] >>> [Times: user=1139.19 sys=393.93, real=82.65 secs] >>> >>> 2015-10-04T15:01:28.903-0400: 204694.735: Total time for which application threads were stopped: 82.6655358 seconds, Stopping threads took: 0.0000918 seconds >>> >>> 2015-10-04T15:01:28.904-0400: 204694.735: [GC cleanup 42G->42G(90G), 0.1166196 secs] >>> >>> >>> Java version: >>> Java version "1.8.0_60"Java(TM) SE Runtime Environment (build 1.8.0_60-b27)Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode) >>> >>> Server has 128GB of RAM and 32 cores(with HyperThreading). >>> >>> GC-related JVM options: >>> >>> -XX:InitialHeapSize=107374182400 -XX:+ManagementServer -XX:MaxHeapSize=107374182400 -XX:-OmitStackTraceInFastThrow -XX:+PrintGC >>> -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+TieredCompilation -XX:+UseCondCardMark -XX:+UseG1GC >>> -XX:ParallelGCThreads=32 -XX:+UnlockDiagnosticVMOptions -XX:ParGCCardsPerStrideChunk=32678 -XX:+ParallelRefProcEnabled -J-XX:MaxGCPauseMillis=100 >>> >>> >>> Any ideas what can be the reason? >>> >>> >>> >>> >>> _______________________________________________ >>> hotspot-gc-use mailing list >>> hotspot-gc-use at openjdk.java.net >>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>> >>> >>> >> > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanbeast at gmail.com Wed Oct 14 10:18:12 2015 From: vanbeast at gmail.com (Ivan Shramko) Date: Wed, 14 Oct 2015 13:18:12 +0300 Subject: Very long duration of Finalize marking in the Remark stage of G1GC In-Reply-To: References: <8B24130B-764C-4594-B588-771235CDCF55@oracle.com> Message-ID: I don't think it's in use. There is no process named "*khugepaged*". grep -i HugePages /proc/meminfo: AnonHugePages: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 2015-10-14 13:10 GMT+03:00 Alex Bagehot : > Hi Ivan, > Just to be sure can you confirm it is disabled *and* no longer in use? > There's a risk it was disabled but still in use, see below: > https://access.redhat.com/solutions/46111 > https://access.redhat.com/solutions/422283 > > Thanks, > Alex > > > > On Wed, Oct 14, 2015 at 10:47 AM, Ivan Shramko wrote: > >> Hi Charlie! >> I disabled transparent huge pages but it didn't help. We are still having >> repeating pauses in Finalize Marking. >> Best regards, >> Ivan >> >> 2015-10-12 16:47 GMT+03:00 Ivan Shramko : >> >>> Hi Charlie! >>> >>> Thanks for the reply. We use Linux (CentOS 6 64 bit). I didn't notice >>> any swapping activity while server was running. But transparent huge pages >>> option was enabled. I will try to disable it and report if it helps or not. >>> >>> Best regards, >>> Ivan >>> >>> 2015-10-12 15:40 GMT+03:00 charlie hunt : >>> >>>> Hi Ivan, >>>> >>>> The remark pause (in addition to high ?Finalize Marking?) also has a >>>> rather high reported sys CPU time. Usually when I see high sys CPU time, it >>>> is often a symptom of paging to/from virtual memory, or if you are running >>>> on Linux it is a symptom of transparent huge pages being enabled. Fwiw, I >>>> don?t think I?ve seen a high ?Finalize Marking? time like what you have >>>> here. >>>> >>>> Could you confirm you are not paging to/from virtual memory, and if you >>>> are running on Linux, can you also confirm transparent huge pages are >>>> disabled? >>>> >>>> thanks, >>>> >>>> charlie >>>> >>>> >>>> >>>> On Oct 12, 2015, at 4:14 AM, Ivan Shramko wrote: >>>> >>>> Application becomes unresponsive on high load while G1 garbage >>>> collector is in the Remark stage(Finalize marking). >>>> This is part of the gc log: >>>> >>>> {Heap before GC invocations=1558 (full 0): >>>> garbage-first heap total 94371840K, used 83001550K [0x00007f76d0000000, 0x00007f76d2005a00, 0x00007f8d50000000) >>>> region size 32768K, 1234 young (40435712K), 58 survivors (1900544K) >>>> Metaspace used 77160K, capacity 77951K, committed 78080K, reserved 79872K >>>> 204611.028: [G1Ergonomics (Concurrent Cycles) initiate concurrent cycle, reason: concurrent cycle initiation requested] >>>> 2015-10-04T15:00:05.197-0400: 204611.028: [GC pause (GCLocker Initiated GC) (young) (initial-mark) >>>> Desired survivor size 2600468480 bytes, new threshold 2 (max 2) >>>> - age 1: 621290376 bytes, 621290376 total >>>> - age 2: 245159208 bytes, 866449584 total >>>> 204611.029: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 184157, predicted base time: 57.32 ms, remaining time: 142.68 ms, target pause time: 200.00 ms] >>>> 204611.029: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 1176 regions, survivors: 58 regions, predicted young region time: 131.19 ms] >>>> 204611.029: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 1176 regions, survivors: 58 regions, old: 0 regions, predicted pause time: 188.51 ms, target pause time: 200.00 ms] >>>> 2015-10-04T15:00:05.416-0400: 204611.247: [SoftReference, 0 refs, 0.0023502 secs]2015-10-04T15:00:05.418-0400: 204611.249: [WeakReference, 394 refs, 0.0018755 secs]2015-10-04T15:00:05.420-0400: 204611.251: [FinalReference, 91704 refs, 0. >>>> 0081225 secs]2015-10-04T15:00:05.428-0400: 204611.259: [PhantomReference, 29 refs, 1 refs, 0.0048953 secs]2015-10-04T15:00:05.433-0400: 204611.264: [JNI Weak Reference, 0.0000229 secs], 0.2600007 secs] >>>> [Parallel Time: 215.1 ms, GC Workers: 32] >>>> [GC Worker Start (ms): Min: 204611030.3, Avg: 204611030.5, Max: 204611030.6, Diff: 0.4] >>>> [Ext Root Scanning (ms): Min: 2.5, Avg: 3.2, Max: 8.3, Diff: 5.8, Sum: 103.2] >>>> [Update RS (ms): Min: 19.0, Avg: 24.5, Max: 35.3, Diff: 16.3, Sum: 782.9] >>>> [Processed Buffers: Min: 7, Avg: 23.7, Max: 31, Diff: 24, Sum: 758] >>>> [Scan RS (ms): Min: 40.4, Avg: 42.2, Max: 43.0, Diff: 2.6, Sum: 1349.0] >>>> [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.2] >>>> [Object Copy (ms): Min: 131.5, Avg: 140.0, Max: 142.4, Diff: 10.9, Sum: 4480.1] >>>> [Termination (ms): Min: 1.9, Avg: 4.1, Max: 4.6, Diff: 2.7, Sum: 130.8] >>>> [Termination Attempts: Min: 262, Avg: 544.9, Max: 599, Diff: 337, Sum: 17437] >>>> [GC Worker Other (ms): Min: 0.0, Avg: 0.2, Max: 0.4, Diff: 0.4, Sum: 7.1] >>>> [GC Worker Total (ms): Min: 213.8, Avg: 214.2, Max: 214.5, Diff: 0.7, Sum: 6853.3] >>>> [GC Worker End (ms): Min: 204611244.4, Avg: 204611244.6, Max: 204611244.8, Diff: 0.4] >>>> [Code Root Fixup: 0.0 ms] >>>> [Code Root Purge: 0.0 ms] >>>> [Clear CT: 8.4 ms] >>>> [Other: 36.4 ms] >>>> [Choose CSet: 0.1 ms] >>>> [Ref Proc: 19.1 ms] >>>> [Ref Enq: 0.8 ms] >>>> [Redirty Cards: 2.0 ms] >>>> [Humongous Register: 0.0 ms] >>>> [Humongous Reclaim: 0.0 ms] >>>> [Free CSet: 11.5 ms] >>>> [Eden: 36.8G(36.7G)->0.0B(35.8G) Survivors: 1856.0M->1568.0M Heap: 79.2G(90.0G)->42.4G(90.0G)] >>>> Heap after GC invocations=1559 (full 0): >>>> garbage-first heap total 94371840K, used 44456740K [0x00007f76d0000000, 0x00007f76d2005a00, 0x00007f8d50000000) >>>> region size 32768K, 49 young (1605632K), 49 survivors (1605632K) >>>> Metaspace used 77160K, capacity 77951K, committed 78080K, reserved 79872K >>>> } >>>> [Times: user=7.11 sys=0.18, real=0.26 secs] >>>> 2015-10-04T15:00:05.457-0400: 204611.288: [GC concurrent-root-region-scan-start] >>>> >>>> 2015-10-04T15:00:05.457-0400: 204611.288: Total time for which application threads were stopped: 0.2607079 seconds, Stopping threads took: 0.0000623 seconds >>>> >>>> 2015-10-04T15:00:05.593-0400: 204611.424: [GC concurrent-root-region-scan-end, 0.1359037 secs] >>>> >>>> 2015-10-04T15:00:05.593-0400: 204611.424: [GC concurrent-mark-start] >>>> >>>> 2015-10-04T15:00:06.238-0400: 204612.069: [GC concurrent-mark-end, 0.6448793 secs] >>>> >>>> 2015-10-04T15:00:06.238-0400: 204612.069: [GC remark 2015-10-04T15:00:06.238-0400: 204612.069: [Finalize Marking, 82.5606577 secs] 2015-10-04T15:01:28.799-0400: 204694.630: [GC ref-proc2015-10-04T15:01:28.799-0400: 204694.630: [SoftRefer >>>> ence, 64 refs, 0.0031041 secs]2015-10-04T15:01:28.802-0400: 204694.633: [WeakReference, 383 refs, 0.0023197 secs]2015-10-04T15:01:28.805-0400: 204694.636: [FinalReference, 100288 refs, 0.0330312 secs]2015-10-04T15:01:28.838-0400: 204694. >>>> 669: [PhantomReference, 2 refs, 37 refs, 0.0030514 secs]2015-10-04T15:01:28.841-0400: 204694.672: [JNI Weak Reference, 0.0000915 secs], 0.0539037 secs] 2015-10-04T15:01:28.853-0400: 204694.684: [Unloading, 0.0130054 secs], 82.6649201 sec >>>> s] >>>> [Times: user=1139.19 sys=393.93, real=82.65 secs] >>>> >>>> 2015-10-04T15:01:28.903-0400: 204694.735: Total time for which application threads were stopped: 82.6655358 seconds, Stopping threads took: 0.0000918 seconds >>>> >>>> 2015-10-04T15:01:28.904-0400: 204694.735: [GC cleanup 42G->42G(90G), 0.1166196 secs] >>>> >>>> >>>> Java version: >>>> Java version "1.8.0_60"Java(TM) SE Runtime Environment (build 1.8.0_60-b27)Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode) >>>> >>>> Server has 128GB of RAM and 32 cores(with HyperThreading). >>>> >>>> GC-related JVM options: >>>> >>>> -XX:InitialHeapSize=107374182400 -XX:+ManagementServer -XX:MaxHeapSize=107374182400 -XX:-OmitStackTraceInFastThrow -XX:+PrintGC >>>> -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+TieredCompilation -XX:+UseCondCardMark -XX:+UseG1GC >>>> -XX:ParallelGCThreads=32 -XX:+UnlockDiagnosticVMOptions -XX:ParGCCardsPerStrideChunk=32678 -XX:+ParallelRefProcEnabled -J-XX:MaxGCPauseMillis=100 >>>> >>>> >>>> Any ideas what can be the reason? >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> hotspot-gc-use mailing list >>>> hotspot-gc-use at openjdk.java.net >>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>>> >>>> >>>> >>> >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.schatzl at oracle.com Wed Oct 14 10:38:57 2015 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Wed, 14 Oct 2015 12:38:57 +0200 Subject: Very long duration of Finalize marking in the Remark stage of G1GC In-Reply-To: References: <8B24130B-764C-4594-B588-771235CDCF55@oracle.com> Message-ID: <1444819137.10769.6.camel@oracle.com> Hi Ivan, just some random idea: do you know if your application creates very large arrays containing object references? I.e. in the range of a few hundred MB? We might run into one of them during remark sometimes, and they may prevent scaling, similar to https://bugs.openjdk.java.net/browse/JDK-8057003 . It would be nice if you could run your application with -XX: +UnlockExperimentalVMOptions -XX:G1LogLevel=finest -XX: +PrintAdaptiveSizePolicy -XX:+G1TraceEagerReclaimHumongousObjects to get an idea about that. It would be nice to have a log with a long remark pause and a few young gcs before that. This is really only a guess, so the problem might not be related at all. Thanks, Thomas From vanbeast at gmail.com Wed Oct 14 11:12:14 2015 From: vanbeast at gmail.com (Ivan Shramko) Date: Wed, 14 Oct 2015 14:12:14 +0300 Subject: Very long duration of Finalize marking in the Remark stage of G1GC In-Reply-To: <1444819137.10769.6.camel@oracle.com> References: <8B24130B-764C-4594-B588-771235CDCF55@oracle.com> <1444819137.10769.6.camel@oracle.com> Message-ID: Hi Thomas! Did you mean humongous objects? We don't have them. Grepping gc log with "humongous" doesn't show anything. I attached part of gc log with young gcs before long finalize marking. I have added JVM options you asked and will post log after next long finalize pause. Best regards, Ivan 2015-10-14 13:38 GMT+03:00 Thomas Schatzl : > Hi Ivan, > > just some random idea: do you know if your application creates very > large arrays containing object references? I.e. in the range of a few > hundred MB? > > We might run into one of them during remark sometimes, and they may > prevent scaling, similar to > https://bugs.openjdk.java.net/browse/JDK-8057003 . > > It would be nice if you could run your application with -XX: > +UnlockExperimentalVMOptions -XX:G1LogLevel=finest -XX: > +PrintAdaptiveSizePolicy -XX:+G1TraceEagerReclaimHumongousObjects to get > an idea about that. > > It would be nice to have a log with a long remark pause and a few young > gcs before that. > > This is really only a guess, so the problem might not be related at all. > > Thanks, > Thomas > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- {Heap before GC invocations=1556 (full 0): garbage-first heap total 94371840K, used 78351927K [0x00007f76d0000000, 0x00007f76d2005a00, 0x00007f8d50000000) region size 32768K, 1114 young (36503552K), 53 survivors (1736704K) Metaspace used 77160K, capacity 77951K, committed 78080K, reserved 79872K 2015-10-04T14:53:07.305-0400: 204193.136: [GC pause (G1 Evacuation Pause) (young) Desired survivor size 1325400064 bytes, new threshold 2 (max 2) - age 1: 523073840 bytes, 523073840 total - age 2: 229700672 bytes, 752774512 total 204193.136: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 138447, predicted base time: 48.81 ms, remaining time: 151.19 ms, target pause time: 200.00 ms] 204193.136: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 1061 regions, survivors: 53 regions, predicted young region time: 146.32 ms] 204193.136: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 1061 regions, survivors: 53 regions, old: 0 regions, predicted pause time: 195.13 ms, target pause time: 200.00 ms ] 2015-10-04T14:53:07.462-0400: 204193.293: [SoftReference, 0 refs, 0.0025515 secs]2015-10-04T14:53:07.464-0400: 204193.295: [WeakReference, 362 refs, 0.0015984 secs]2015-10-04T14:53:07.466-0 400: 204193.297: [FinalReference, 90835 refs, 0.0092076 secs]2015-10-04T14:53:07.475-0400: 204193.306: [PhantomReference, 32 refs, 1 refs, 0.0030970 secs]2015-10-04T14:53:07.478-0400: 20419 3.309: [JNI Weak Reference, 0.0000245 secs], 0.1913854 secs] [Parallel Time: 154.0 ms, GC Workers: 32] [GC Worker Start (ms): Min: 204193137.3, Avg: 204193137.8, Max: 204193144.9, Diff: 7.5] [Ext Root Scanning (ms): Min: 0.0, Avg: 0.6, Max: 2.4, Diff: 2.4, Sum: 17.8] [Update RS (ms): Min: 12.7, Avg: 19.4, Max: 23.3, Diff: 10.6, Sum: 619.5] [Processed Buffers: Min: 5, Avg: 18.2, Max: 27, Diff: 22, Sum: 581] [Scan RS (ms): Min: 38.0, Avg: 42.1, Max: 50.3, Diff: 12.3, Sum: 1348.6] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.2] [Object Copy (ms): Min: 80.0, Avg: 87.7, Max: 88.3, Diff: 8.3, Sum: 2807.9] [Termination (ms): Min: 0.0, Avg: 3.0, Max: 3.1, Diff: 3.1, Sum: 95.8] [Termination Attempts: Min: 1, Avg: 24.8, Max: 40, Diff: 39, Sum: 792] [GC Worker Other (ms): Min: 0.0, Avg: 0.2, Max: 0.3, Diff: 0.3, Sum: 4.8] [GC Worker Total (ms): Min: 145.9, Avg: 153.0, Max: 153.6, Diff: 7.7, Sum: 4894.8] [GC Worker End (ms): Min: 204193290.7, Avg: 204193290.8, Max: 204193291.0, Diff: 0.3] [Code Root Fixup: 0.1 ms] [Code Root Purge: 0.0 ms] [Clear CT: 7.7 ms] [Other: 29.5 ms] [Choose CSet: 0.0 ms] [Ref Proc: 17.9 ms] [Ref Enq: 0.9 ms] [Redirty Cards: 1.9 ms] [Humongous Register: 0.1 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 7.6 ms] [Eden: 33.2G(18.1G)->0.0B(37.0G) Survivors: 1696.0M->1856.0M Heap: 74.7G(90.0G)->42.1G(90.0G)] Heap after GC invocations=1557 (full 0): garbage-first heap total 94371840K, used 44189261K [0x00007f76d0000000, 0x00007f76d2005a00, 0x00007f8d50000000) region size 32768K, 58 young (1900544K), 58 survivors (1900544K) Metaspace used 77160K, capacity 77951K, committed 78080K, reserved 79872K } [Times: user=5.24 sys=0.11, real=0.19 secs] 2015-10-04T14:53:07.497-0400: 204193.328: Total time for which application threads were stopped: 0.1924941 seconds, Stopping threads took: 0.0003254 seconds 2015-10-04T14:53:07.536-0400: 204193.367: Total time for which application threads were stopped: 0.0024199 seconds, Stopping threads took: 0.0019603 seconds 2015-10-04T14:53:07.537-0400: 204193.368: Total time for which application threads were stopped: 0.0005090 seconds, Stopping threads took: 0.0003125 seconds 2015-10-04T14:53:07.539-0400: 204193.370: Total time for which application threads were stopped: 0.0011741 seconds, Stopping threads took: 0.0008785 seconds 2015-10-04T14:53:07.540-0400: 204193.371: Total time for which application threads were stopped: 0.0003168 seconds, Stopping threads took: 0.0000983 seconds 2015-10-04T14:53:07.542-0400: 204193.373: Total time for which application threads were stopped: 0.0012312 seconds, Stopping threads took: 0.0009565 seconds 2015-10-04T14:53:07.543-0400: 204193.374: Total time for which application threads were stopped: 0.0003502 seconds, Stopping threads took: 0.0001167 seconds 2015-10-04T14:53:07.545-0400: 204193.376: Total time for which application threads were stopped: 0.0013865 seconds, Stopping threads took: 0.0010004 seconds 2015-10-04T14:53:16.073-0400: 204201.904: Total time for which application threads were stopped: 0.0007099 seconds, Stopping threads took: 0.0000944 seconds 2015-10-04T14:53:20.731-0400: 204206.562: Total time for which application threads were stopped: 0.0006761 seconds, Stopping threads took: 0.0000973 seconds 2015-10-04T14:53:55.583-0400: 204241.414: Total time for which application threads were stopped: 0.0005850 seconds, Stopping threads took: 0.0001100 seconds 2015-10-04T14:53:55.583-0400: 204241.414: Total time for which application threads were stopped: 0.0002960 seconds, Stopping threads took: 0.0000804 seconds 2015-10-04T14:54:39.509-0400: 204285.340: Total time for which application threads were stopped: 0.0007971 seconds, Stopping threads took: 0.0002291 seconds 2015-10-04T14:54:39.510-0400: 204285.341: Total time for which application threads were stopped: 0.0002893 seconds, Stopping threads took: 0.0000419 seconds 2015-10-04T14:54:39.764-0400: 204285.595: Total time for which application threads were stopped: 0.0013350 seconds, Stopping threads took: 0.0008491 seconds 2015-10-04T14:54:41.851-0400: 204287.682: Total time for which application threads were stopped: 0.0012849 seconds, Stopping threads took: 0.0001466 seconds 2015-10-04T14:54:49.561-0400: 204295.392: Total time for which application threads were stopped: 0.0009442 seconds, Stopping threads took: 0.0002284 seconds 2015-10-04T14:54:51.683-0400: 204297.514: Total time for which application threads were stopped: 0.0005281 seconds, Stopping threads took: 0.0000984 seconds 2015-10-04T14:55:13.691-0400: 204319.522: Total time for which application threads were stopped: 0.0024420 seconds, Stopping threads took: 0.0017980 seconds 2015-10-04T14:55:34.964-0400: 204340.795: Total time for which application threads were stopped: 0.0004586 seconds, Stopping threads took: 0.0000642 seconds 2015-10-04T14:56:03.426-0400: 204369.257: Total time for which application threads were stopped: 0.0007487 seconds, Stopping threads took: 0.0001356 seconds 2015-10-04T14:56:33.823-0400: 204399.654: Total time for which application threads were stopped: 0.0005697 seconds, Stopping threads took: 0.0000907 seconds {Heap before GC invocations=1557 (full 0): garbage-first heap total 94371840K, used 82986573K [0x00007f76d0000000, 0x00007f76d2005a00, 0x00007f8d50000000) region size 32768K, 1243 young (40730624K), 58 survivors (1900544K) Metaspace used 77160K, capacity 77951K, committed 78080K, reserved 79872K 2015-10-04T14:56:33.824-0400: 204399.655: [GC pause (GCLocker Initiated GC) (young) Desired survivor size 2617245696 bytes, new threshold 2 (max 2) - age 1: 564311480 bytes, 564311480 total - age 2: 233691344 bytes, 798002824 total 204399.655: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 205743, predicted base time: 59.14 ms, remaining time: 140.86 ms, target pause time: 200.00 ms] 204399.655: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 1185 regions, survivors: 58 regions, predicted young region time: 133.39 ms] 204399.655: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 1185 regions, survivors: 58 regions, old: 0 regions, predicted pause time: 192.52 ms, target pause time: 200.00 ms ] 2015-10-04T14:56:34.006-0400: 204399.837: [SoftReference, 0 refs, 0.0025401 secs]2015-10-04T14:56:34.008-0400: 204399.840: [WeakReference, 440 refs, 0.0015279 secs]2015-10-04T14:56:34.010-0 400: 204399.841: [FinalReference, 92689 refs, 0.0102004 secs]2015-10-04T14:56:34.020-0400: 204399.851: [PhantomReference, 14 refs, 1 refs, 0.0031138 secs]2015-10-04T14:56:34.023-0400: 20439 9.855: [JNI Weak Reference, 0.0000505 secs] 204399.874: [G1Ergonomics (Concurrent Cycles) request concurrent cycle initiation, reason: occupancy higher than threshold, occupancy: 4365431603 2 bytes, allocation request: 0 bytes, threshold: 43486543845 bytes (45.00 %), source: end of GC] , 0.2197629 secs] [Parallel Time: 179.4 ms, GC Workers: 32] [GC Worker Start (ms): Min: 204399656.2, Avg: 204399657.1, Max: 204399663.9, Diff: 7.7] [Ext Root Scanning (ms): Min: 0.0, Avg: 0.6, Max: 2.5, Diff: 2.5, Sum: 18.0] [Update RS (ms): Min: 21.8, Avg: 28.3, Max: 29.7, Diff: 7.9, Sum: 905.3] [Processed Buffers: Min: 12, Avg: 26.4, Max: 33, Diff: 21, Sum: 844] [Scan RS (ms): Min: 39.4, Avg: 41.3, Max: 41.9, Diff: 2.5, Sum: 1321.5] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.2] [Object Copy (ms): Min: 107.2, Avg: 107.7, Max: 108.2, Diff: 1.0, Sum: 3446.6] [Termination (ms): Min: 0.0, Avg: 0.2, Max: 0.3, Diff: 0.3, Sum: 7.6] [Termination Attempts: Min: 1, Avg: 71.9, Max: 94, Diff: 93, Sum: 2302] [GC Worker Other (ms): Min: 0.0, Avg: 0.2, Max: 0.3, Diff: 0.3, Sum: 5.6] [GC Worker Total (ms): Min: 171.4, Avg: 178.3, Max: 179.3, Diff: 7.9, Sum: 5704.8] [GC Worker End (ms): Min: 204399835.2, Avg: 204399835.4, Max: 204399835.6, Diff: 0.3] [Code Root Fixup: 0.1 ms] [Code Root Purge: 0.0 ms] [Clear CT: 8.3 ms] [Other: 32.0 ms] [Choose CSet: 0.1 ms] [Ref Proc: 18.9 ms] [Ref Enq: 0.7 ms] [Redirty Cards: 1.7 ms] [Humongous Register: 0.1 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 8.9 ms] [Eden: 37.0G(37.0G)->0.0B(36.7G) Survivors: 1856.0M->1856.0M Heap: 79.2G(90.0G)->42.4G(90.0G)] Heap after GC invocations=1558 (full 0): garbage-first heap total 94371840K, used 44499150K [0x00007f76d0000000, 0x00007f76d2005a00, 0x00007f8d50000000) region size 32768K, 58 young (1900544K), 58 survivors (1900544K) Metaspace used 77160K, capacity 77951K, committed 78080K, reserved 79872K } [Times: user=6.07 sys=0.19, real=0.22 secs] 2015-10-04T14:56:34.044-0400: 204399.875: Total time for which application threads were stopped: 0.2204766 seconds, Stopping threads took: 0.0001223 seconds 2015-10-04T14:56:34.075-0400: 204399.907: Total time for which application threads were stopped: 0.0010407 seconds, Stopping threads took: 0.0006153 seconds 2015-10-04T14:56:34.077-0400: 204399.908: Total time for which application threads were stopped: 0.0005106 seconds, Stopping threads took: 0.0001712 seconds 2015-10-04T14:56:34.077-0400: 204399.908: Total time for which application threads were stopped: 0.0002922 seconds, Stopping threads took: 0.0000431 seconds 2015-10-04T14:56:34.079-0400: 204399.910: Total time for which application threads were stopped: 0.0007010 seconds, Stopping threads took: 0.0002168 seconds 2015-10-04T14:56:34.080-0400: 204399.911: Total time for which application threads were stopped: 0.0003869 seconds, Stopping threads took: 0.0001520 seconds 2015-10-04T14:56:34.081-0400: 204399.912: Total time for which application threads were stopped: 0.0015659 seconds, Stopping threads took: 0.0001263 seconds 2015-10-04T14:57:00.447-0400: 204426.278: Total time for which application threads were stopped: 0.0006533 seconds, Stopping threads took: 0.0000893 seconds 2015-10-04T14:57:22.384-0400: 204448.216: Total time for which application threads were stopped: 0.0007878 seconds, Stopping threads took: 0.0002240 seconds 2015-10-04T14:57:25.372-0400: 204451.203: Total time for which application threads were stopped: 0.0007839 seconds, Stopping threads took: 0.0001125 seconds 2015-10-04T14:57:48.038-0400: 204473.870: Total time for which application threads were stopped: 0.0006679 seconds, Stopping threads took: 0.0000809 seconds 2015-10-04T14:57:52.957-0400: 204478.788: Total time for which application threads were stopped: 0.0009843 seconds, Stopping threads took: 0.0003051 seconds 2015-10-04T14:58:07.151-0400: 204492.982: Total time for which application threads were stopped: 0.0007069 seconds, Stopping threads took: 0.0001032 seconds 2015-10-04T14:58:15.571-0400: 204501.402: Total time for which application threads were stopped: 0.0007719 seconds, Stopping threads took: 0.0001712 seconds 2015-10-04T14:58:30.156-0400: 204515.988: Total time for which application threads were stopped: 0.0007603 seconds, Stopping threads took: 0.0001050 seconds 2015-10-04T14:59:55.163-0400: 204600.995: Total time for which application threads were stopped: 0.0008706 seconds, Stopping threads took: 0.0001749 seconds 2015-10-04T14:59:55.201-0400: 204601.032: Total time for which application threads were stopped: 0.0005852 seconds, Stopping threads took: 0.0001521 seconds 2015-10-04T14:59:55.202-0400: 204601.033: Total time for which application threads were stopped: 0.0004847 seconds, Stopping threads took: 0.0001346 seconds 2015-10-04T14:59:55.519-0400: 204601.350: Total time for which application threads were stopped: 0.0007758 seconds, Stopping threads took: 0.0001170 seconds 2015-10-04T14:59:57.666-0400: 204603.497: Total time for which application threads were stopped: 0.0007808 seconds, Stopping threads took: 0.0001688 seconds 2015-10-04T15:00:05.193-0400: 204611.024: Total time for which application threads were stopped: 0.0005358 seconds, Stopping threads took: 0.0001350 seconds {Heap before GC invocations=1558 (full 0): garbage-first heap total 94371840K, used 83001550K [0x00007f76d0000000, 0x00007f76d2005a00, 0x00007f8d50000000) region size 32768K, 1234 young (40435712K), 58 survivors (1900544K) Metaspace used 77160K, capacity 77951K, committed 78080K, reserved 79872K 204611.028: [G1Ergonomics (Concurrent Cycles) initiate concurrent cycle, reason: concurrent cycle initiation requested] 2015-10-04T15:00:05.197-0400: 204611.028: [GC pause (GCLocker Initiated GC) (young) (initial-mark) Desired survivor size 2600468480 bytes, new threshold 2 (max 2) - age 1: 621290376 bytes, 621290376 total - age 2: 245159208 bytes, 866449584 total 204611.029: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 184157, predicted base time: 57.32 ms, remaining time: 142.68 ms, target pause time: 200.00 ms] 204611.029: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 1176 regions, survivors: 58 regions, predicted young region time: 131.19 ms] 204611.029: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 1176 regions, survivors: 58 regions, old: 0 regions, predicted pause time: 188.51 ms, target pause time: 200.00 ms] 2015-10-04T15:00:05.416-0400: 204611.247: [SoftReference, 0 refs, 0.0023502 secs]2015-10-04T15:00:05.418-0400: 204611.249: [WeakReference, 394 refs, 0.0018755 secs]2015-10-04T15:00:05.420-0400: 204611.251: [FinalReference, 91704 refs, 0. 0081225 secs]2015-10-04T15:00:05.428-0400: 204611.259: [PhantomReference, 29 refs, 1 refs, 0.0048953 secs]2015-10-04T15:00:05.433-0400: 204611.264: [JNI Weak Reference, 0.0000229 secs], 0.2600007 secs] [Parallel Time: 215.1 ms, GC Workers: 32] [GC Worker Start (ms): Min: 204611030.3, Avg: 204611030.5, Max: 204611030.6, Diff: 0.4] [Ext Root Scanning (ms): Min: 2.5, Avg: 3.2, Max: 8.3, Diff: 5.8, Sum: 103.2] [Update RS (ms): Min: 19.0, Avg: 24.5, Max: 35.3, Diff: 16.3, Sum: 782.9] [Processed Buffers: Min: 7, Avg: 23.7, Max: 31, Diff: 24, Sum: 758] [Scan RS (ms): Min: 40.4, Avg: 42.2, Max: 43.0, Diff: 2.6, Sum: 1349.0] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.2] [Object Copy (ms): Min: 131.5, Avg: 140.0, Max: 142.4, Diff: 10.9, Sum: 4480.1] [Termination (ms): Min: 1.9, Avg: 4.1, Max: 4.6, Diff: 2.7, Sum: 130.8] [Termination Attempts: Min: 262, Avg: 544.9, Max: 599, Diff: 337, Sum: 17437] [GC Worker Other (ms): Min: 0.0, Avg: 0.2, Max: 0.4, Diff: 0.4, Sum: 7.1] [GC Worker Total (ms): Min: 213.8, Avg: 214.2, Max: 214.5, Diff: 0.7, Sum: 6853.3] [GC Worker End (ms): Min: 204611244.4, Avg: 204611244.6, Max: 204611244.8, Diff: 0.4] [Code Root Fixup: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 8.4 ms] [Other: 36.4 ms] [Choose CSet: 0.1 ms] [Ref Proc: 19.1 ms] [Ref Enq: 0.8 ms] [Redirty Cards: 2.0 ms] [Humongous Register: 0.0 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 11.5 ms] [Eden: 36.8G(36.7G)->0.0B(35.8G) Survivors: 1856.0M->1568.0M Heap: 79.2G(90.0G)->42.4G(90.0G)] Heap after GC invocations=1559 (full 0): garbage-first heap total 94371840K, used 44456740K [0x00007f76d0000000, 0x00007f76d2005a00, 0x00007f8d50000000) region size 32768K, 49 young (1605632K), 49 survivors (1605632K) Metaspace used 77160K, capacity 77951K, committed 78080K, reserved 79872K } [Times: user=7.11 sys=0.18, real=0.26 secs] 2015-10-04T15:00:05.457-0400: 204611.288: [GC concurrent-root-region-scan-start] 2015-10-04T15:00:05.457-0400: 204611.288: Total time for which application threads were stopped: 0.2607079 seconds, Stopping threads took: 0.0000623 seconds 2015-10-04T15:00:05.593-0400: 204611.424: [GC concurrent-root-region-scan-end, 0.1359037 secs] 2015-10-04T15:00:05.593-0400: 204611.424: [GC concurrent-mark-start] 2015-10-04T15:00:06.238-0400: 204612.069: [GC concurrent-mark-end, 0.6448793 secs] 2015-10-04T15:00:06.238-0400: 204612.069: [GC remark 2015-10-04T15:00:06.238-0400: 204612.069: [Finalize Marking, 82.5606577 secs] 2015-10-04T15:01:28.799-0400: 204694.630: [GC ref-proc2015-10-04T15:01:28.799-0400: 204694.630: [SoftRefer ence, 64 refs, 0.0031041 secs]2015-10-04T15:01:28.802-0400: 204694.633: [WeakReference, 383 refs, 0.0023197 secs]2015-10-04T15:01:28.805-0400: 204694.636: [FinalReference, 100288 refs, 0.0330312 secs]2015-10-04T15:01:28.838-0400: 204694. 669: [PhantomReference, 2 refs, 37 refs, 0.0030514 secs]2015-10-04T15:01:28.841-0400: 204694.672: [JNI Weak Reference, 0.0000915 secs], 0.0539037 secs] 2015-10-04T15:01:28.853-0400: 204694.684: [Unloading, 0.0130054 secs], 82.6649201 sec s] [Times: user=1139.19 sys=393.93, real=82.65 secs] 2015-10-04T15:01:28.903-0400: 204694.735: Total time for which application threads were stopped: 82.6655358 seconds, Stopping threads took: 0.0000918 seconds 2015-10-04T15:01:28.904-0400: 204694.735: [GC cleanup 42G->42G(90G), 0.1166196 secs] From thomas.schatzl at oracle.com Wed Oct 14 11:22:15 2015 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Wed, 14 Oct 2015 13:22:15 +0200 Subject: Very long duration of Finalize marking in the Remark stage of G1GC In-Reply-To: References: <8B24130B-764C-4594-B588-771235CDCF55@oracle.com> <1444819137.10769.6.camel@oracle.com> Message-ID: <1444821735.10769.13.camel@oracle.com> Hi, On Wed, 2015-10-14 at 14:12 +0300, Ivan Shramko wrote: > Hi Thomas! > > Did you mean humongous objects? We don't have them. Grepping gc log > with "humongous" doesn't show anything. Yes, I meant humongous objects. In the log snippet you gave, it did not indicate any amount of these objects too, e.g. > [Humongous Register: 0.0 ms] > [Humongous Reclaim: 0.0 ms] which typically indicates none. However both actions are typically very fast, so 0.0ms is not really unusual even in presence of some. In your log snippet I saw that some Humongous Register phases are 0.1ms, but that may just be some random variation. The options I gave you will make that clear (although I saw that you already have PrintAdaptiveSizePolicy enabled) because they will print the number of humongous objects directly. If you already mention that you are sure you do not have them, the mentioned issue cannot be the cause anyway. > I attached part of gc log with young gcs before long finalize marking. > > I have added JVM options you asked and will post log after next long > finalize pause. Maybe the new log will yield some new information. Not sure. Thanks, Thomas From vanbeast at gmail.com Mon Oct 19 12:07:31 2015 From: vanbeast at gmail.com (Ivan Shramko) Date: Mon, 19 Oct 2015 15:07:31 +0300 Subject: Very long duration of Finalize marking in the Remark stage of G1GC In-Reply-To: <1444821735.10769.13.camel@oracle.com> References: <8B24130B-764C-4594-B588-771235CDCF55@oracle.com> <1444819137.10769.6.camel@oracle.com> <1444821735.10769.13.camel@oracle.com> Message-ID: Hi! Attached new log. Best regards, Ivan Shramko 2015-10-14 14:22 GMT+03:00 Thomas Schatzl : > Hi, > > On Wed, 2015-10-14 at 14:12 +0300, Ivan Shramko wrote: > > Hi Thomas! > > > > Did you mean humongous objects? We don't have them. Grepping gc log > > with "humongous" doesn't show anything. > > Yes, I meant humongous objects. In the log snippet you gave, it did not > indicate any amount of these objects too, e.g. > > > [Humongous Register: 0.0 ms] > > [Humongous Reclaim: 0.0 ms] > > which typically indicates none. However both actions are typically very > fast, so 0.0ms is not really unusual even in presence of some. In your > log snippet I saw that some Humongous Register phases are 0.1ms, but > that may just be some random variation. > > The options I gave you will make that clear (although I saw that you > already have PrintAdaptiveSizePolicy enabled) because they will print > the number of humongous objects directly. > > If you already mention that you are sure you do not have them, the > mentioned issue cannot be the cause anyway. > > > I attached part of gc log with young gcs before long finalize marking. > > > > I have added JVM options you asked and will post log after next long > > finalize pause. > > Maybe the new log will yield some new information. Not sure. > > Thanks, > Thomas > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- {Heap before GC invocations=10613 (full 0): garbage-first heap total 94371840K, used 46579346K [0x00007f017c000000, 0x00007f017e005a00, 0x00007f17fc000000) region size 32768K, 144 young (4718592K), 18 survivors (589824K) Metaspace used 78702K, capacity 79547K, committed 79588K, reserved 79872K 2015-10-18T07:06:27.974-0400: 346692.925: [GC pause (G1 Evacuation Pause) (young) Desired survivor size 301989888 bytes, new threshold 2 (max 15) - age 1: 260252672 bytes, 260252672 total - age 2: 123695976 bytes, 383948648 total - age 3: 67812800 bytes, 451761448 total 346692.925: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 93568, predicted base time: 37.35 ms, remaining time: 62.65 ms, target pause time: 100.00 ms] 346692.925: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 126 regions, survivors: 18 regions, predicted young region time: 69.19 ms] 346692.925: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 126 regions, survivors: 18 regions, old: 0 regions, predicted pause time: 106.54 ms, target pause time: 100.00 ms] 2015-10-18T07:06:28.091-0400: 346693.042: [SoftReference, 0 refs, 0.0032346 secs]2015-10-18T07:06:28.094-0400: 346693.045: [WeakReference, 108 refs, 0.0014270 secs]2015-10-18T07:06:28.096-0400: 346693.047: [FinalReference, 718 refs, 0.0017805 secs]2015-10-18T07:06:28.097-0400: 346693.049: [PhantomReference, 13 refs, 0 refs, 0.0036647 secs]2015-10-18T07:06:28.101-0400: 346693.052: [JNI Weak Reference, 0.0000342 secs], 0.1382317 secs] [Parallel Time: 115.1 ms, GC Workers: 32] [GC Worker Start (ms): 346692925.7 346692925.7 346692925.7 346692925.7 346692925.7 346692925.7 346692925.7 346692925.8 346692925.8 346692925.8 346692925.8 346692925.8 346692925.8 346692925.8 346692925.9 346692925.9 346692925.9 346692925.9 346692925.9 346692925.9 346692926.0 346692926.0 346692926.0 346692926.0 346692926.0 346692926.0 346692926.1 346692926.1 346692926.1 346692926.1 346692926.1 346692926.1 Min: 346692925.7, Avg: 346692925.9, Max: 346692926.1, Diff: 0.5] [Ext Root Scanning (ms): 1.7 0.6 0.6 0.6 0.6 0.6 0.5 0.5 0.5 0.5 0.5 0.5 0.8 0.5 0.4 0.4 0.4 0.4 0.4 0.4 0.3 0.3 0.3 0.3 0.3 0.3 0.2 0.2 0.2 0.2 0.2 0.2 Min: 0.2, Avg: 0.4, Max: 1.7, Diff: 1.6, Sum: 14.3] [Thread Roots (ms): 0.0 0.2 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.2, Diff: 0.2, Sum: 1.1] [StringTable Roots (ms): 0.0 0.5 0.5 0.5 0.4 0.4 0.5 0.4 0.4 0.5 0.5 0.5 0.0 0.4 0.4 0.4 0.4 0.4 0.4 0.4 0.3 0.3 0.3 0.3 0.3 0.3 0.2 0.2 0.2 0.2 0.2 0.2 Min: 0.0, Avg: 0.3, Max: 0.5, Diff: 0.5, Sum: 10.6] [Universe Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [JNI Handles Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [ObjectSynchronizer Roots (ms): 0.0 0.0 0.0 0.0 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.1] [FlatProfiler Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Management Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [SystemDictionary Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.8, Diff: 0.8, Sum: 0.8] [CLDG Roots (ms): 1.7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.1, Max: 1.7, Diff: 1.7, Sum: 1.7] [JVMTI Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [CodeCache Roots (ms): 41.4 42.6 42.4 42.6 42.5 42.5 42.5 42.5 42.5 42.5 42.6 42.5 42.2 42.5 42.5 42.6 42.5 42.6 42.6 42.5 42.6 42.6 42.6 42.6 42.5 42.5 42.6 43.2 42.6 42.5 42.5 42.5 Min: 41.4, Avg: 42.5, Max: 43.2, Diff: 1.7, Sum: 1360.6] [CM RefProcessor Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Wait For Strong CLD (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Weak CLD Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [SATB Filtering (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Update RS (ms): 10.5 11.6 11.5 12.0 11.6 11.9 11.8 11.7 13.2 11.7 12.0 11.9 11.8 12.1 11.9 11.9 11.6 11.7 11.8 11.6 11.6 11.6 12.0 12.1 11.9 11.6 11.7 11.8 11.6 12.3 11.9 11.9 Min: 10.5, Avg: 11.8, Max: 13.2, Diff: 2.7, Sum: 378.0] [Processed Buffers: 13 14 18 13 11 14 14 13 12 12 15 12 14 11 11 12 9 11 9 10 13 15 12 11 13 12 12 12 12 12 12 12 Min: 9, Avg: 12.4, Max: 18, Diff: 9, Sum: 396] [Scan RS (ms): 30.9 31.0 30.9 30.6 30.9 30.6 30.6 30.8 29.3 30.8 30.6 30.5 30.5 30.4 30.6 30.6 30.9 30.9 30.8 30.9 31.0 31.0 30.6 30.5 30.6 30.9 30.8 31.4 30.9 30.3 30.6 30.6 Min: 29.3, Avg: 30.7, Max: 31.4, Diff: 2.1, Sum: 982.4] [Code Root Scanning (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] [Object Copy (ms): 69.5 69.6 69.5 69.7 69.8 69.4 69.7 69.9 69.2 70.6 69.2 69.9 69.5 69.3 69.7 69.2 69.7 69.0 69.0 67.6 69.1 69.3 69.0 69.3 69.9 69.5 69.2 68.7 69.6 69.5 69.9 69.4 Min: 67.6, Avg: 69.4, Max: 70.6, Diff: 3.0, Sum: 2221.3] [Termination (ms): 1.8 1.7 1.9 1.7 1.6 1.9 1.6 1.4 2.1 0.7 2.0 1.5 1.9 2.1 1.6 2.1 1.7 2.3 2.2 3.8 2.2 2.0 2.3 2.0 1.4 1.9 2.1 2.0 1.7 1.9 1.5 1.9 Min: 0.7, Avg: 1.9, Max: 3.8, Diff: 3.1, Sum: 60.6] [Termination Attempts: 163 158 145 170 162 141 163 149 156 49 151 162 156 138 135 154 133 140 125 100 149 151 125 157 155 149 140 146 148 162 157 155 Min: 49, Avg: 145.1, Max: 170, Diff: 121, Sum: 4644] [GC Worker Other (ms): 0.0 0.1 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.1 0.2 0.0 0.1 0.1 0.0 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.1 0.1 0.2 0.0 0.0 0.0 Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 1.7] [GC Worker Total (ms): 114.5 114.6 114.7 114.5 114.5 114.5 114.5 114.4 114.4 114.4 114.5 114.4 114.5 114.4 114.3 114.4 114.3 114.3 114.3 114.3 114.2 114.2 114.2 114.2 114.2 114.2 114.2 114.2 114.2 114.1 114.1 114.1 Min: 114.1, Avg: 114.3, Max: 114.7, Diff: 0.6, Sum: 3658.5] [GC Worker End (ms): 346693040.2 346693040.3 346693040.3 346693040.2 346693040.2 346693040.2 346693040.2 346693040.2 346693040.2 346693040.2 346693040.3 346693040.2 346693040.3 346693040.2 346693040.2 346693040.2 346693040.2 346693040.2 346693040.2 346693040.2 346693040.2 346693040.2 346693040.2 346693040.2 346693040.2 346693040.2 346693040.2 346693040.2 346693040.3 346693040.2 346693040.2 346693040.2 Min: 346693040.2, Avg: 346693040.2, Max: 346693040.3, Diff: 0.2] [Code Root Fixup: 0.1 ms] [Code Root Purge: 0.0 ms] [Clear CT: 5.3 ms] [Other: 17.7 ms] [Choose CSet: 0.0 ms] [Ref Proc: 11.5 ms] [Ref Enq: 0.7 ms] [Redirty Cards: 2.0 ms] [Parallel Redirty: 1.6 1.5 1.5 1.5 1.5 1.5 1.5 1.5 1.5 1.5 1.5 1.5 1.5 1.5 1.4 1.4 1.4 1.4 1.4 1.4 0.5 0.5 0.5 0.4 0.4 0.4 0.4 0.4 0.4 0.3 0.0 0.0 Min: 0.0, Avg: 1.1, Max: 1.6, Diff: 1.6, Sum: 33.9] [Redirtied Cards: 23267 23417 22407 23635 23469 21895 21708 20324 19484 17776 20028 20004 6757 18913 17664 18916 17459 18511 16659 12800 5207 4473 4218 4216 4258 4204 4944 3891 3355 3009 0 0 Min: 0, Avg: 13339.6, Max: 23635, Diff: 23635, Sum: 426868] [Humongous Register: 0.2 ms] [Humongous Total: 0] [Humongous Candidate: 0] [Humongous Reclaim: 0.1 ms] [Humongous Reclaimed: 0] [Free CSet: 2.4 ms] [Young Free CSet: 2.4 ms] [Non-Young Free CSet: 0.0 ms] [Eden: 4032.0M(4032.0M)->0.0B(4032.0M) Survivors: 576.0M->576.0M Heap: 44.4G(90.0G)->41.0G(90.0G)] Heap after GC invocations=10614 (full 0): garbage-first heap total 94371840K, used 42977782K [0x00007f017c000000, 0x00007f017e005a00, 0x00007f17fc000000) region size 32768K, 18 young (589824K), 18 survivors (589824K) Metaspace used 78702K, capacity 79547K, committed 79588K, reserved 79872K } [Times: user=3.82 sys=0.08, real=0.14 secs] 2015-10-18T07:06:28.113-0400: 346693.065: Total time for which application threads were stopped: 0.1403935 seconds, Stopping threads took: 0.0000975 seconds {Heap before GC invocations=10614 (full 0): garbage-first heap total 94371840K, used 47106550K [0x00007f017c000000, 0x00007f017e005a00, 0x00007f17fc000000) region size 32768K, 144 young (4718592K), 18 survivors (589824K) Metaspace used 78702K, capacity 79547K, committed 79588K, reserved 79872K 2015-10-18T07:06:58.055-0400: 346723.006: [GC pause (G1 Evacuation Pause) (young) Desired survivor size 301989888 bytes, new threshold 2 (max 15) - age 1: 222862048 bytes, 222862048 total - age 2: 198816712 bytes, 421678760 total 346723.006: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 77058, predicted base time: 34.23 ms, remaining time: 65.77 ms, target pause time: 100.00 ms] 346723.006: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 126 regions, survivors: 18 regions, predicted young region time: 79.29 ms] 346723.006: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 126 regions, survivors: 18 regions, old: 0 regions, predicted pause time: 113.52 ms, target pause time: 100.00 ms] 2015-10-18T07:06:58.165-0400: 346723.116: [SoftReference, 0 refs, 0.0025743 secs]2015-10-18T07:06:58.167-0400: 346723.119: [WeakReference, 74 refs, 0.0015911 secs]2015-10-18T07:06:58.169-0400: 346723.120: [FinalReference, 1094 refs, 0.0012432 secs]2015-10-18T07:06:58.170-0400: 346723.121: [PhantomReference, 13 refs, 0 refs, 0.0029009 secs]2015-10-18T07:06:58.173-0400: 346723.124: [JNI Weak Reference, 0.0000343 secs] 346723.134: [G1Ergonomics (Concurrent Cycles) request concurrent cycle initiation, reason: occupancy higher than threshold, occupancy: 43822088192 bytes, allocation request: 0 bytes, threshold: 43486543845 bytes (45.00 %), source: end of GC] , 0.1279979 secs] [Parallel Time: 107.8 ms, GC Workers: 32] [GC Worker Start (ms): 346723006.7 346723006.7 346723006.8 346723006.8 346723006.8 346723006.9 346723006.9 346723006.9 346723006.9 346723006.9 346723006.9 346723006.9 346723006.9 346723006.9 346723006.9 346723007.0 346723007.0 346723007.0 346723007.0 346723007.0 346723007.0 346723007.0 346723007.1 346723007.1 346723007.1 346723007.1 346723007.1 346723007.2 346723007.2 346723007.2 346723007.2 346723007.3 Min: 346723006.7, Avg: 346723007.0, Max: 346723007.3, Diff: 0.5] [Ext Root Scanning (ms): 2.9 1.4 0.8 0.8 0.8 0.7 0.7 0.7 0.7 0.7 0.6 0.7 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.5 0.5 0.5 0.5 0.5 0.5 0.4 0.4 0.4 0.4 0.4 0.3 0.3 Min: 0.3, Avg: 0.7, Max: 2.9, Diff: 2.6, Sum: 21.1] [Thread Roots (ms): 0.0 0.2 0.2 0.2 0.2 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.2, Diff: 0.2, Sum: 1.5] [StringTable Roots (ms): 0.0 0.0 0.6 0.6 0.6 0.6 0.6 0.3 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.5 0.5 0.5 0.5 0.5 0.5 0.4 0.4 0.4 0.4 0.4 0.3 0.3 0.3 Min: 0.0, Avg: 0.5, Max: 0.6, Diff: 0.6, Sum: 15.2] [Universe Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [JNI Handles Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [ObjectSynchronizer Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.3, Diff: 0.3, Sum: 0.3] [FlatProfiler Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Management Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [SystemDictionary Roots (ms): 0.0 1.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 1.2, Diff: 1.2, Sum: 1.2] [CLDG Roots (ms): 2.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.1, Max: 2.9, Diff: 2.9, Sum: 2.9] [JVMTI Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [CodeCache Roots (ms): 36.2 37.7 38.3 38.4 38.2 38.3 38.2 38.2 38.2 38.2 37.9 38.2 38.3 38.3 38.2 38.5 38.3 38.3 38.3 38.5 38.2 38.2 38.2 38.2 38.2 38.3 38.3 38.4 38.3 38.3 38.2 38.2 Min: 36.2, Avg: 38.2, Max: 38.5, Diff: 2.3, Sum: 1221.7] [CM RefProcessor Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Wait For Strong CLD (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Weak CLD Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [SATB Filtering (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Update RS (ms): 10.4 12.0 12.4 12.0 12.1 12.3 18.8 12.0 12.4 12.1 12.2 12.0 12.4 12.7 12.0 12.7 12.0 12.4 12.1 12.0 12.3 12.1 12.0 12.5 12.3 12.1 12.2 13.4 12.5 12.8 12.5 12.6 Min: 10.4, Avg: 12.4, Max: 18.8, Diff: 8.4, Sum: 398.2] [Processed Buffers: 10 11 12 11 11 9 10 11 13 9 12 12 13 12 9 12 8 9 8 10 8 9 10 9 8 12 9 13 8 11 9 9 Min: 8, Avg: 10.2, Max: 13, Diff: 5, Sum: 327] [Scan RS (ms): 25.8 25.6 25.9 26.4 26.2 26.0 19.4 26.3 25.8 26.1 25.7 26.3 25.9 25.6 26.2 25.8 26.3 25.9 26.2 26.5 25.9 26.2 26.2 25.7 25.9 26.2 26.1 25.1 25.9 25.5 25.7 25.6 Min: 19.4, Avg: 25.7, Max: 26.5, Diff: 7.0, Sum: 823.3] [Code Root Scanning (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] [Object Copy (ms): 67.0 66.9 66.9 66.9 67.1 67.0 67.0 67.5 67.1 67.3 67.9 67.5 67.0 67.0 67.1 66.8 66.9 67.0 67.1 66.8 67.2 67.1 67.2 67.0 67.0 67.0 67.0 66.8 67.0 67.1 67.0 67.1 Min: 66.8, Avg: 67.1, Max: 67.9, Diff: 1.1, Sum: 2146.3] [Termination (ms): 0.8 1.0 0.9 0.8 0.8 0.8 0.9 0.4 0.9 0.6 0.4 0.4 0.9 0.8 0.8 0.9 0.9 0.9 0.7 0.8 0.6 0.8 0.7 0.9 0.9 0.9 0.8 0.9 0.8 0.7 0.9 0.9 Min: 0.4, Avg: 0.8, Max: 1.0, Diff: 0.6, Sum: 25.3] [Termination Attempts: 22 13 18 13 18 17 17 1 14 17 18 13 17 14 20 11 10 13 19 6 13 7 9 14 13 9 11 10 19 21 16 8 Min: 1, Avg: 13.8, Max: 22, Diff: 21, Sum: 441] [GC Worker Other (ms): 0.1 0.0 0.0 0.2 0.0 0.1 0.0 0.0 0.0 0.1 0.0 0.0 0.0 0.0 0.0 0.1 0.0 0.0 0.0 0.1 0.0 0.0 0.0 0.0 0.2 0.0 0.0 0.0 0.1 0.0 0.1 0.0 Min: 0.0, Avg: 0.0, Max: 0.2, Diff: 0.1, Sum: 1.5] [GC Worker Total (ms): 107.0 107.0 106.9 107.0 106.9 106.9 106.9 106.9 106.8 106.8 106.8 106.8 106.8 106.8 106.7 106.8 106.7 106.7 106.7 106.8 106.7 106.6 106.6 106.6 106.7 106.6 106.5 106.5 106.6 106.5 106.6 106.5 Min: 106.5, Avg: 106.7, Max: 107.0, Diff: 0.6, Sum: 3415.9] [GC Worker End (ms): 346723113.7 346723113.7 346723113.7 346723113.8 346723113.7 346723113.8 346723113.7 346723113.7 346723113.7 346723113.7 346723113.7 346723113.7 346723113.7 346723113.7 346723113.7 346723113.8 346723113.7 346723113.7 346723113.7 346723113.8 346723113.7 346723113.7 346723113.7 346723113.7 346723113.8 346723113.7 346723113.7 346723113.7 346723113.8 346723113.7 346723113.8 346723113.7 Min: 346723113.7, Avg: 346723113.7, Max: 346723113.8, Diff: 0.1] [Code Root Fixup: 0.1 ms] [Code Root Purge: 0.0 ms] [Clear CT: 4.5 ms] [Other: 15.5 ms] [Choose CSet: 0.0 ms] [Ref Proc: 9.8 ms] [Ref Enq: 0.6 ms] [Redirty Cards: 1.2 ms] [Parallel Redirty: 0.8 0.8 0.8 0.8 0.8 0.8 0.8 0.8 0.8 0.8 0.8 0.8 0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.6 0.6 0.6 0.6 0.6 0.0 Min: 0.0, Avg: 0.7, Max: 0.8, Diff: 0.8, Sum: 22.6] [Redirtied Cards: 16384 15531 15569 16437 16135 14488 13742 14069 14114 15427 9009 8192 7936 7168 8303 8082 7026 7059 7314 7483 7062 6641 11720 7022 10834 10024 6945 10672 10346 9770 3840 0 Min: 0, Avg: 10135.8, Max: 16437, Diff: 16437, Sum: 324344] [Humongous Register: 0.2 ms] [Humongous Total: 0] [Humongous Candidate: 0] [Humongous Reclaim: 0.1 ms] [Humongous Reclaimed: 0] [Free CSet: 2.8 ms] [Young Free CSet: 2.8 ms] [Non-Young Free CSet: 0.0 ms] [Eden: 4032.0M(4032.0M)->0.0B(4032.0M) Survivors: 576.0M->576.0M Heap: 44.9G(90.0G)->41.4G(90.0G)] Heap after GC invocations=10615 (full 0): garbage-first heap total 94371840K, used 43360155K [0x00007f017c000000, 0x00007f017e005a00, 0x00007f17fc000000) region size 32768K, 18 young (589824K), 18 survivors (589824K) Metaspace used 78702K, capacity 79547K, committed 79588K, reserved 79872K } [Times: user=3.57 sys=0.09, real=0.13 secs] 2015-10-18T07:06:58.184-0400: 346723.135: Total time for which application threads were stopped: 0.1313165 seconds, Stopping threads took: 0.0018426 seconds 2015-10-18T07:07:15.146-0400: 346740.097: Total time for which application threads were stopped: 0.0007456 seconds, Stopping threads took: 0.0000971 seconds 2015-10-18T07:07:16.141-0400: 346741.092: Total time for which application threads were stopped: 0.0011149 seconds, Stopping threads took: 0.0002058 seconds 2015-10-18T07:07:18.313-0400: 346743.264: Total time for which application threads were stopped: 0.0010722 seconds, Stopping threads took: 0.0001563 seconds {Heap before GC invocations=10615 (full 0): garbage-first heap total 94371840K, used 47488923K [0x00007f017c000000, 0x00007f017e005a00, 0x00007f17fc000000) region size 32768K, 144 young (4718592K), 18 survivors (589824K) Metaspace used 78702K, capacity 79547K, committed 79588K, reserved 79872K 346752.983: [G1Ergonomics (Concurrent Cycles) initiate concurrent cycle, reason: concurrent cycle initiation requested] 2015-10-18T07:07:28.032-0400: 346752.983: [GC pause (G1 Evacuation Pause) (young) (initial-mark) Desired survivor size 301989888 bytes, new threshold 2 (max 15) - age 1: 193262688 bytes, 193262688 total - age 2: 195192416 bytes, 388455104 total 346752.984: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 55586, predicted base time: 30.65 ms, remaining time: 69.35 ms, target pause time: 100.00 ms] 346752.984: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 126 regions, survivors: 18 regions, predicted young region time: 83.03 ms] 346752.984: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 126 regions, survivors: 18 regions, old: 0 regions, predicted pause time: 113.68 ms, target pause time: 100.00 ms] 2015-10-18T07:07:28.153-0400: 346753.104: [SoftReference, 0 refs, 0.0031514 secs]2015-10-18T07:07:28.156-0400: 346753.107: [WeakReference, 72 refs, 0.0012962 secs]2015-10-18T07:07:28.157-0400: 346753.108: [FinalReference, 489 refs, 0.0016356 secs]2015-10-18T07:07:28.159-0400: 346753.110: [PhantomReference, 6 refs, 1 refs, 0.0029412 secs]2015-10-18T07:07:28.162-0400: 346753.113: [JNI Weak Reference, 0.0000709 secs], 0.1410925 secs] [Parallel Time: 117.7 ms, GC Workers: 32] [GC Worker Start (ms): 346752984.8 346752984.8 346752984.8 346752984.8 346752984.9 346752984.9 346752984.9 346752984.9 346752984.9 346752985.0 346752985.0 346752985.0 346752985.0 346752985.0 346752985.0 346752985.0 346752985.1 346752985.1 346752985.1 346752985.1 346752985.1 346752985.1 346752985.1 346752985.2 346752985.2 346752985.2 346752985.2 346752985.2 346752985.2 346752985.2 346752985.2 346752985.2 Min: 346752984.8, Avg: 346752985.0, Max: 346752985.2, Diff: 0.4] [Ext Root Scanning (ms): 4.0 3.8 3.9 3.8 3.8 3.9 3.8 3.8 3.8 3.7 3.6 3.7 3.7 3.8 3.7 3.7 3.6 3.7 3.7 3.6 3.6 3.7 3.5 3.4 3.5 3.6 3.4 3.4 3.6 3.6 3.6 3.4 Min: 3.4, Avg: 3.7, Max: 4.0, Diff: 0.7, Sum: 117.2] [Thread Roots (ms): 0.0 3.7 0.2 0.2 0.2 0.2 0.1 0.1 0.1 0.1 0.1 0.2 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.2, Max: 3.7, Diff: 3.7, Sum: 5.5] [StringTable Roots (ms): 0.0 0.0 0.6 0.6 0.3 0.6 0.6 0.6 0.6 0.6 0.6 0.5 0.6 0.0 0.6 0.6 0.6 0.6 0.6 0.6 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.4 0.4 0.4 0.4 Min: 0.0, Avg: 0.5, Max: 0.6, Diff: 0.6, Sum: 15.5] [Universe Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [JNI Handles Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [ObjectSynchronizer Roots (ms): 0.0 0.0 0.0 0.0 0.3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.3, Diff: 0.3, Sum: 0.3] [FlatProfiler Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Management Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [SystemDictionary Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 1.1, Diff: 1.1, Sum: 1.1] [CLDG Roots (ms): 1.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 1.2, Diff: 1.2, Sum: 1.2] [JVMTI Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [CodeCache Roots (ms): 39.6 39.8 39.6 39.6 39.7 39.5 40.0 39.6 39.6 39.6 40.3 40.1 40.1 40.0 39.7 39.6 39.6 39.6 39.6 40.0 39.6 39.6 45.7 40.2 39.7 39.6 39.7 40.3 39.6 40.0 40.3 39.8 Min: 39.5, Avg: 40.0, Max: 45.7, Diff: 6.2, Sum: 1279.5] [CM RefProcessor Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Wait For Strong CLD (ms): 2.8 0.0 3.1 3.0 3.0 3.1 3.0 2.9 3.1 3.0 2.9 2.9 2.9 2.6 3.0 3.0 2.9 3.1 3.1 3.0 3.0 3.1 2.9 2.9 2.9 3.1 2.9 2.9 3.1 3.1 3.1 2.9 Min: 0.0, Avg: 2.9, Max: 3.1, Diff: 3.1, Sum: 92.1] [Weak CLD Roots (ms): 0.0 0.1 0.0 0.0 0.0 0.0 0.0 0.1 0.0 0.0 0.1 0.1 0.1 0.0 0.1 0.0 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.1 0.0 0.0 0.1 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 1.3] [SATB Filtering (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Update RS (ms): 10.0 9.0 8.6 9.1 9.2 8.9 8.6 9.2 9.1 8.8 9.4 9.0 8.9 8.7 9.2 8.8 9.4 9.1 8.8 9.1 9.1 8.8 8.9 8.8 9.7 9.2 9.3 8.9 9.0 8.7 8.6 9.0 Min: 8.6, Avg: 9.0, Max: 10.0, Diff: 1.4, Sum: 288.8] [Processed Buffers: 9 10 11 6 6 6 8 6 9 5 8 6 7 8 7 5 7 5 5 6 6 9 9 9 5 10 9 9 13 8 9 11 Min: 5, Avg: 7.7, Max: 13, Diff: 8, Sum: 247] [Scan RS (ms): 29.6 30.8 30.9 30.5 30.5 30.6 31.4 30.4 30.5 30.8 30.9 31.1 31.3 31.3 30.5 30.8 30.2 30.5 30.9 30.9 30.5 30.8 36.9 31.4 30.0 30.4 30.5 31.4 30.5 31.2 31.7 30.8 Min: 29.6, Avg: 31.0, Max: 36.9, Diff: 7.3, Sum: 990.5] [Code Root Scanning (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] [Object Copy (ms): 71.7 71.5 71.7 71.8 72.0 71.7 71.5 71.8 71.7 71.7 71.2 71.2 71.7 71.3 71.8 71.7 72.0 71.8 71.7 71.4 71.6 71.7 65.7 71.2 71.6 71.6 71.8 71.1 71.7 71.4 70.9 71.7 Min: 65.7, Avg: 71.4, Max: 72.0, Diff: 6.2, Sum: 2285.0] [Termination (ms): 1.4 1.6 1.5 1.4 1.1 1.5 1.2 1.4 1.3 1.4 1.4 1.5 0.9 1.4 1.3 1.4 1.2 1.3 1.3 1.4 1.5 1.4 1.4 1.5 1.5 1.5 1.3 1.4 1.5 1.2 1.5 1.3 Min: 0.9, Avg: 1.4, Max: 1.6, Diff: 0.6, Sum: 44.2] [Termination Attempts: 13 12 14 14 12 17 11 14 11 15 9 17 5 11 9 16 7 9 14 8 10 16 13 7 14 14 12 16 17 4 17 7 Min: 4, Avg: 12.0, Max: 17, Diff: 13, Sum: 385] [GC Worker Other (ms): 0.0 0.0 0.1 0.1 0.1 0.0 0.0 0.1 0.0 0.0 0.0 0.1 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.1 0.0 0.1 0.0 0.0 0.0 0.0 0.1 0.1 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 1.5] [GC Worker Total (ms): 116.7 116.7 116.7 116.6 116.7 116.6 116.6 116.6 116.6 116.5 116.5 116.6 116.6 116.5 116.5 116.5 116.4 116.4 116.4 116.4 116.5 116.4 116.4 116.3 116.3 116.3 116.3 116.4 116.3 116.3 116.3 116.2 Min: 116.2, Avg: 116.5, Max: 116.7, Diff: 0.5, Sum: 3727.2] [GC Worker End (ms): 346753101.5 346753101.5 346753101.6 346753101.5 346753101.6 346753101.5 346753101.5 346753101.5 346753101.5 346753101.5 346753101.5 346753101.5 346753101.6 346753101.5 346753101.5 346753101.5 346753101.5 346753101.5 346753101.5 346753101.5 346753101.6 346753101.5 346753101.5 346753101.5 346753101.5 346753101.5 346753101.5 346753101.6 346753101.6 346753101.5 346753101.5 346753101.5 Min: 346753101.5, Avg: 346753101.5, Max: 346753101.6, Diff: 0.1] [Code Root Fixup: 0.1 ms] [Code Root Purge: 0.0 ms] [Clear CT: 5.1 ms] [Other: 18.2 ms] [Choose CSet: 0.0 ms] [Ref Proc: 10.7 ms] [Ref Enq: 0.5 ms] [Redirty Cards: 2.3 ms] [Parallel Redirty: 1.8 1.8 1.8 1.7 1.7 1.7 1.7 1.7 1.7 1.7 1.7 1.7 1.7 1.7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.8, Max: 1.8, Diff: 1.8, Sum: 24.3] [Redirtied Cards: 20279 24852 24843 24186 19664 23217 23422 24870 25919 25178 23849 18080 21743 22024 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Min: 0, Avg: 10066.4, Max: 25919, Diff: 25919, Sum: 322126] [Humongous Register: 0.1 ms] [Humongous Total: 0] [Humongous Candidate: 0] [Humongous Reclaim: 0.1 ms] [Humongous Reclaimed: 0] [Free CSet: 2.5 ms] [Young Free CSet: 2.5 ms] [Non-Young Free CSet: 0.0 ms] [Eden: 4032.0M(4032.0M)->0.0B(4032.0M) Survivors: 576.0M->576.0M Heap: 45.3G(90.0G)->41.7G(90.0G)] Heap after GC invocations=10616 (full 0): garbage-first heap total 94371840K, used 43732854K [0x00007f017c000000, 0x00007f017e005a00, 0x00007f17fc000000) region size 32768K, 18 young (589824K), 18 survivors (589824K) Metaspace used 78702K, capacity 79547K, committed 79588K, reserved 79872K } [Times: user=3.84 sys=0.08, real=0.14 secs] 2015-10-18T07:07:28.174-0400: 346753.126: [GC concurrent-root-region-scan-start] 2015-10-18T07:07:28.175-0400: 346753.126: Total time for which application threads were stopped: 0.1434439 seconds, Stopping threads took: 0.0001640 seconds 2015-10-18T07:07:28.260-0400: 346753.211: [GC concurrent-root-region-scan-end, 0.0852149 secs] 2015-10-18T07:07:28.260-0400: 346753.211: [GC concurrent-mark-start] 2015-10-18T07:07:28.965-0400: 346753.916: [GC concurrent-mark-end, 0.7049345 secs] 2015-10-18T07:07:28.966-0400: 346753.917: [GC remark 2015-10-18T07:07:28.966-0400: 346753.917: [Finalize Marking, 115.4404545 secs] 2015-10-18T07:09:24.406-0400: 346869.357: [GC ref-proc2015-10-18T07:09:24.406-0400: 346869.357: [SoftReference, 54 refs, 0.0028579 secs]2015-10-18T07:09:24.409-0400: 346869.360: [WeakReference, 418 refs, 0.0020702 secs]2015-10-18T07:09:24.411-0400: 346869.362: [FinalReference, 1633 refs, 0.0025592 secs]2015-10-18T07:09:24.414-0400: 346869.365: [PhantomReference, 6 refs, 21 refs, 0.0040350 secs]2015-10-18T07:09:24.418-0400: 346869.369: [JNI Weak Reference, 0.0001088 secs], 0.0130411 secs] 2015-10-18T07:09:24.419-0400: 346869.370: [Unloading 2015-10-18T07:09:24.420-0400: 346869.371: [System Dictionary Unloading, 0.0001458 secs] 2015-10-18T07:09:24.420-0400: 346869.371: [Parallel Unloading, 0.0153302 secs] 2015-10-18T07:09:24.435-0400: 346869.386: [Deallocate Metadata, 0.0000526 secs], 0.0159394 secs], 115.5038062 secs] [Times: user=1641.24 sys=743.90, real=115.48 secs] 2015-10-18T07:09:24.469-0400: 346869.421: Total time for which application threads were stopped: 115.5047519 seconds, Stopping threads took: 0.0001246 seconds 2015-10-18T07:09:24.470-0400: 346869.421: [GC cleanup 41G->41G(90G), 0.1233809 secs] [Times: user=3.83 sys=0.00, real=0.13 secs] 2015-10-18T07:09:24.594-0400: 346869.545: Total time for which application threads were stopped: 0.1240198 seconds, Stopping threads took: 0.0001006 seconds 2015-10-18T07:09:24.594-0400: 346869.545: [GC concurrent-cleanup-start] 2015-10-18T07:09:24.594-0400: 346869.545: [GC concurrent-cleanup-end, 0.0003056 secs] 2015-10-18T07:09:24.594-0400: 346869.546: Total time for which application threads were stopped: 0.0006490 seconds, Stopping threads took: 0.0001584 seconds 2015-10-18T07:09:24.597-0400: 346869.548: Total time for which application threads were stopped: 0.0006890 seconds, Stopping threads took: 0.0002038 seconds 2015-10-18T07:09:37.098-0400: 346882.049: Total time for which application threads were stopped: 0.0015512 seconds, Stopping threads took: 0.0006916 seconds From thomas.schatzl at oracle.com Mon Oct 19 12:54:38 2015 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Mon, 19 Oct 2015 14:54:38 +0200 Subject: Very long duration of Finalize marking in the Remark stage of G1GC In-Reply-To: References: <8B24130B-764C-4594-B588-771235CDCF55@oracle.com> <1444819137.10769.6.camel@oracle.com> <1444821735.10769.13.camel@oracle.com> Message-ID: <1445259278.2280.21.camel@oracle.com> Hi Ivan, On Mon, 2015-10-19 at 15:07 +0300, Ivan Shramko wrote: > Hi! > Attached new log. > thanks. Thinking about it now, but no idea yet. One request, could you add one or two young gcs after that long remark pause? Thanks, Thomas From vanbeast at gmail.com Mon Oct 19 13:20:24 2015 From: vanbeast at gmail.com (Ivan Shramko) Date: Mon, 19 Oct 2015 16:20:24 +0300 Subject: Very long duration of Finalize marking in the Remark stage of G1GC In-Reply-To: <1445259278.2280.21.camel@oracle.com> References: <8B24130B-764C-4594-B588-771235CDCF55@oracle.com> <1444819137.10769.6.camel@oracle.com> <1444821735.10769.13.camel@oracle.com> <1445259278.2280.21.camel@oracle.com> Message-ID: Hi Thomas, Added young gc after pause Best regards, Ivan 2015-10-19 15:54 GMT+03:00 Thomas Schatzl : > Hi Ivan, > > On Mon, 2015-10-19 at 15:07 +0300, Ivan Shramko wrote: > > Hi! > > Attached new log. > > > thanks. Thinking about it now, but no idea yet. One request, could you > add one or two young gcs after that long remark pause? > > Thanks, > Thomas > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- {Heap before GC invocations=10613 (full 0): garbage-first heap total 94371840K, used 46579346K [0x00007f017c000000, 0x00007f017e005a00, 0x00007f17fc000000) region size 32768K, 144 young (4718592K), 18 survivors (589824K) Metaspace used 78702K, capacity 79547K, committed 79588K, reserved 79872K 2015-10-18T07:06:27.974-0400: 346692.925: [GC pause (G1 Evacuation Pause) (young) Desired survivor size 301989888 bytes, new threshold 2 (max 15) - age 1: 260252672 bytes, 260252672 total - age 2: 123695976 bytes, 383948648 total - age 3: 67812800 bytes, 451761448 total 346692.925: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 93568, predicted base time: 37.35 ms, remaining time: 62.65 ms, target pause time: 100.00 ms] 346692.925: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 126 regions, survivors: 18 regions, predicted young region time: 69.19 ms] 346692.925: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 126 regions, survivors: 18 regions, old: 0 regions, predicted pause time: 106.54 ms, target pause time: 100.00 ms] 2015-10-18T07:06:28.091-0400: 346693.042: [SoftReference, 0 refs, 0.0032346 secs]2015-10-18T07:06:28.094-0400: 346693.045: [WeakReference, 108 refs, 0.0014270 secs]2015-10-18T07:06:28.096-0400: 346693.047: [FinalReference, 718 refs, 0.0017805 secs]2015-10-18T07:06:28.097-0400: 346693.049: [PhantomReference, 13 refs, 0 refs, 0.0036647 secs]2015-10-18T07:06:28.101-0400: 346693.052: [JNI Weak Reference, 0.0000342 secs], 0.1382317 secs] [Parallel Time: 115.1 ms, GC Workers: 32] [GC Worker Start (ms): 346692925.7 346692925.7 346692925.7 346692925.7 346692925.7 346692925.7 346692925.7 346692925.8 346692925.8 346692925.8 346692925.8 346692925.8 346692925.8 346692925.8 346692925.9 346692925.9 346692925.9 346692925.9 346692925.9 346692925.9 346692926.0 346692926.0 346692926.0 346692926.0 346692926.0 346692926.0 346692926.1 346692926.1 346692926.1 346692926.1 346692926.1 346692926.1 Min: 346692925.7, Avg: 346692925.9, Max: 346692926.1, Diff: 0.5] [Ext Root Scanning (ms): 1.7 0.6 0.6 0.6 0.6 0.6 0.5 0.5 0.5 0.5 0.5 0.5 0.8 0.5 0.4 0.4 0.4 0.4 0.4 0.4 0.3 0.3 0.3 0.3 0.3 0.3 0.2 0.2 0.2 0.2 0.2 0.2 Min: 0.2, Avg: 0.4, Max: 1.7, Diff: 1.6, Sum: 14.3] [Thread Roots (ms): 0.0 0.2 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.2, Diff: 0.2, Sum: 1.1] [StringTable Roots (ms): 0.0 0.5 0.5 0.5 0.4 0.4 0.5 0.4 0.4 0.5 0.5 0.5 0.0 0.4 0.4 0.4 0.4 0.4 0.4 0.4 0.3 0.3 0.3 0.3 0.3 0.3 0.2 0.2 0.2 0.2 0.2 0.2 Min: 0.0, Avg: 0.3, Max: 0.5, Diff: 0.5, Sum: 10.6] [Universe Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [JNI Handles Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [ObjectSynchronizer Roots (ms): 0.0 0.0 0.0 0.0 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.1] [FlatProfiler Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Management Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [SystemDictionary Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.8, Diff: 0.8, Sum: 0.8] [CLDG Roots (ms): 1.7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.1, Max: 1.7, Diff: 1.7, Sum: 1.7] [JVMTI Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [CodeCache Roots (ms): 41.4 42.6 42.4 42.6 42.5 42.5 42.5 42.5 42.5 42.5 42.6 42.5 42.2 42.5 42.5 42.6 42.5 42.6 42.6 42.5 42.6 42.6 42.6 42.6 42.5 42.5 42.6 43.2 42.6 42.5 42.5 42.5 Min: 41.4, Avg: 42.5, Max: 43.2, Diff: 1.7, Sum: 1360.6] [CM RefProcessor Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Wait For Strong CLD (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Weak CLD Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [SATB Filtering (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Update RS (ms): 10.5 11.6 11.5 12.0 11.6 11.9 11.8 11.7 13.2 11.7 12.0 11.9 11.8 12.1 11.9 11.9 11.6 11.7 11.8 11.6 11.6 11.6 12.0 12.1 11.9 11.6 11.7 11.8 11.6 12.3 11.9 11.9 Min: 10.5, Avg: 11.8, Max: 13.2, Diff: 2.7, Sum: 378.0] [Processed Buffers: 13 14 18 13 11 14 14 13 12 12 15 12 14 11 11 12 9 11 9 10 13 15 12 11 13 12 12 12 12 12 12 12 Min: 9, Avg: 12.4, Max: 18, Diff: 9, Sum: 396] [Scan RS (ms): 30.9 31.0 30.9 30.6 30.9 30.6 30.6 30.8 29.3 30.8 30.6 30.5 30.5 30.4 30.6 30.6 30.9 30.9 30.8 30.9 31.0 31.0 30.6 30.5 30.6 30.9 30.8 31.4 30.9 30.3 30.6 30.6 Min: 29.3, Avg: 30.7, Max: 31.4, Diff: 2.1, Sum: 982.4] [Code Root Scanning (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] [Object Copy (ms): 69.5 69.6 69.5 69.7 69.8 69.4 69.7 69.9 69.2 70.6 69.2 69.9 69.5 69.3 69.7 69.2 69.7 69.0 69.0 67.6 69.1 69.3 69.0 69.3 69.9 69.5 69.2 68.7 69.6 69.5 69.9 69.4 Min: 67.6, Avg: 69.4, Max: 70.6, Diff: 3.0, Sum: 2221.3] [Termination (ms): 1.8 1.7 1.9 1.7 1.6 1.9 1.6 1.4 2.1 0.7 2.0 1.5 1.9 2.1 1.6 2.1 1.7 2.3 2.2 3.8 2.2 2.0 2.3 2.0 1.4 1.9 2.1 2.0 1.7 1.9 1.5 1.9 Min: 0.7, Avg: 1.9, Max: 3.8, Diff: 3.1, Sum: 60.6] [Termination Attempts: 163 158 145 170 162 141 163 149 156 49 151 162 156 138 135 154 133 140 125 100 149 151 125 157 155 149 140 146 148 162 157 155 Min: 49, Avg: 145.1, Max: 170, Diff: 121, Sum: 4644] [GC Worker Other (ms): 0.0 0.1 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.1 0.2 0.0 0.1 0.1 0.0 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.1 0.1 0.2 0.0 0.0 0.0 Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 1.7] [GC Worker Total (ms): 114.5 114.6 114.7 114.5 114.5 114.5 114.5 114.4 114.4 114.4 114.5 114.4 114.5 114.4 114.3 114.4 114.3 114.3 114.3 114.3 114.2 114.2 114.2 114.2 114.2 114.2 114.2 114.2 114.2 114.1 114.1 114.1 Min: 114.1, Avg: 114.3, Max: 114.7, Diff: 0.6, Sum: 3658.5] [GC Worker End (ms): 346693040.2 346693040.3 346693040.3 346693040.2 346693040.2 346693040.2 346693040.2 346693040.2 346693040.2 346693040.2 346693040.3 346693040.2 346693040.3 346693040.2 346693040.2 346693040.2 346693040.2 346693040.2 346693040.2 346693040.2 346693040.2 346693040.2 346693040.2 346693040.2 346693040.2 346693040.2 346693040.2 346693040.2 346693040.3 346693040.2 346693040.2 346693040.2 Min: 346693040.2, Avg: 346693040.2, Max: 346693040.3, Diff: 0.2] [Code Root Fixup: 0.1 ms] [Code Root Purge: 0.0 ms] [Clear CT: 5.3 ms] [Other: 17.7 ms] [Choose CSet: 0.0 ms] [Ref Proc: 11.5 ms] [Ref Enq: 0.7 ms] [Redirty Cards: 2.0 ms] [Parallel Redirty: 1.6 1.5 1.5 1.5 1.5 1.5 1.5 1.5 1.5 1.5 1.5 1.5 1.5 1.5 1.4 1.4 1.4 1.4 1.4 1.4 0.5 0.5 0.5 0.4 0.4 0.4 0.4 0.4 0.4 0.3 0.0 0.0 Min: 0.0, Avg: 1.1, Max: 1.6, Diff: 1.6, Sum: 33.9] [Redirtied Cards: 23267 23417 22407 23635 23469 21895 21708 20324 19484 17776 20028 20004 6757 18913 17664 18916 17459 18511 16659 12800 5207 4473 4218 4216 4258 4204 4944 3891 3355 3009 0 0 Min: 0, Avg: 13339.6, Max: 23635, Diff: 23635, Sum: 426868] [Humongous Register: 0.2 ms] [Humongous Total: 0] [Humongous Candidate: 0] [Humongous Reclaim: 0.1 ms] [Humongous Reclaimed: 0] [Free CSet: 2.4 ms] [Young Free CSet: 2.4 ms] [Non-Young Free CSet: 0.0 ms] [Eden: 4032.0M(4032.0M)->0.0B(4032.0M) Survivors: 576.0M->576.0M Heap: 44.4G(90.0G)->41.0G(90.0G)] Heap after GC invocations=10614 (full 0): garbage-first heap total 94371840K, used 42977782K [0x00007f017c000000, 0x00007f017e005a00, 0x00007f17fc000000) region size 32768K, 18 young (589824K), 18 survivors (589824K) Metaspace used 78702K, capacity 79547K, committed 79588K, reserved 79872K } [Times: user=3.82 sys=0.08, real=0.14 secs] 2015-10-18T07:06:28.113-0400: 346693.065: Total time for which application threads were stopped: 0.1403935 seconds, Stopping threads took: 0.0000975 seconds {Heap before GC invocations=10614 (full 0): garbage-first heap total 94371840K, used 47106550K [0x00007f017c000000, 0x00007f017e005a00, 0x00007f17fc000000) region size 32768K, 144 young (4718592K), 18 survivors (589824K) Metaspace used 78702K, capacity 79547K, committed 79588K, reserved 79872K 2015-10-18T07:06:58.055-0400: 346723.006: [GC pause (G1 Evacuation Pause) (young) Desired survivor size 301989888 bytes, new threshold 2 (max 15) - age 1: 222862048 bytes, 222862048 total - age 2: 198816712 bytes, 421678760 total 346723.006: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 77058, predicted base time: 34.23 ms, remaining time: 65.77 ms, target pause time: 100.00 ms] 346723.006: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 126 regions, survivors: 18 regions, predicted young region time: 79.29 ms] 346723.006: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 126 regions, survivors: 18 regions, old: 0 regions, predicted pause time: 113.52 ms, target pause time: 100.00 ms] 2015-10-18T07:06:58.165-0400: 346723.116: [SoftReference, 0 refs, 0.0025743 secs]2015-10-18T07:06:58.167-0400: 346723.119: [WeakReference, 74 refs, 0.0015911 secs]2015-10-18T07:06:58.169-0400: 346723.120: [FinalReference, 1094 refs, 0.0012432 secs]2015-10-18T07:06:58.170-0400: 346723.121: [PhantomReference, 13 refs, 0 refs, 0.0029009 secs]2015-10-18T07:06:58.173-0400: 346723.124: [JNI Weak Reference, 0.0000343 secs] 346723.134: [G1Ergonomics (Concurrent Cycles) request concurrent cycle initiation, reason: occupancy higher than threshold, occupancy: 43822088192 bytes, allocation request: 0 bytes, threshold: 43486543845 bytes (45.00 %), source: end of GC] , 0.1279979 secs] [Parallel Time: 107.8 ms, GC Workers: 32] [GC Worker Start (ms): 346723006.7 346723006.7 346723006.8 346723006.8 346723006.8 346723006.9 346723006.9 346723006.9 346723006.9 346723006.9 346723006.9 346723006.9 346723006.9 346723006.9 346723006.9 346723007.0 346723007.0 346723007.0 346723007.0 346723007.0 346723007.0 346723007.0 346723007.1 346723007.1 346723007.1 346723007.1 346723007.1 346723007.2 346723007.2 346723007.2 346723007.2 346723007.3 Min: 346723006.7, Avg: 346723007.0, Max: 346723007.3, Diff: 0.5] [Ext Root Scanning (ms): 2.9 1.4 0.8 0.8 0.8 0.7 0.7 0.7 0.7 0.7 0.6 0.7 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.5 0.5 0.5 0.5 0.5 0.5 0.4 0.4 0.4 0.4 0.4 0.3 0.3 Min: 0.3, Avg: 0.7, Max: 2.9, Diff: 2.6, Sum: 21.1] [Thread Roots (ms): 0.0 0.2 0.2 0.2 0.2 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.2, Diff: 0.2, Sum: 1.5] [StringTable Roots (ms): 0.0 0.0 0.6 0.6 0.6 0.6 0.6 0.3 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.5 0.5 0.5 0.5 0.5 0.5 0.4 0.4 0.4 0.4 0.4 0.3 0.3 0.3 Min: 0.0, Avg: 0.5, Max: 0.6, Diff: 0.6, Sum: 15.2] [Universe Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [JNI Handles Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [ObjectSynchronizer Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.3, Diff: 0.3, Sum: 0.3] [FlatProfiler Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Management Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [SystemDictionary Roots (ms): 0.0 1.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 1.2, Diff: 1.2, Sum: 1.2] [CLDG Roots (ms): 2.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.1, Max: 2.9, Diff: 2.9, Sum: 2.9] [JVMTI Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [CodeCache Roots (ms): 36.2 37.7 38.3 38.4 38.2 38.3 38.2 38.2 38.2 38.2 37.9 38.2 38.3 38.3 38.2 38.5 38.3 38.3 38.3 38.5 38.2 38.2 38.2 38.2 38.2 38.3 38.3 38.4 38.3 38.3 38.2 38.2 Min: 36.2, Avg: 38.2, Max: 38.5, Diff: 2.3, Sum: 1221.7] [CM RefProcessor Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Wait For Strong CLD (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Weak CLD Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [SATB Filtering (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Update RS (ms): 10.4 12.0 12.4 12.0 12.1 12.3 18.8 12.0 12.4 12.1 12.2 12.0 12.4 12.7 12.0 12.7 12.0 12.4 12.1 12.0 12.3 12.1 12.0 12.5 12.3 12.1 12.2 13.4 12.5 12.8 12.5 12.6 Min: 10.4, Avg: 12.4, Max: 18.8, Diff: 8.4, Sum: 398.2] [Processed Buffers: 10 11 12 11 11 9 10 11 13 9 12 12 13 12 9 12 8 9 8 10 8 9 10 9 8 12 9 13 8 11 9 9 Min: 8, Avg: 10.2, Max: 13, Diff: 5, Sum: 327] [Scan RS (ms): 25.8 25.6 25.9 26.4 26.2 26.0 19.4 26.3 25.8 26.1 25.7 26.3 25.9 25.6 26.2 25.8 26.3 25.9 26.2 26.5 25.9 26.2 26.2 25.7 25.9 26.2 26.1 25.1 25.9 25.5 25.7 25.6 Min: 19.4, Avg: 25.7, Max: 26.5, Diff: 7.0, Sum: 823.3] [Code Root Scanning (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] [Object Copy (ms): 67.0 66.9 66.9 66.9 67.1 67.0 67.0 67.5 67.1 67.3 67.9 67.5 67.0 67.0 67.1 66.8 66.9 67.0 67.1 66.8 67.2 67.1 67.2 67.0 67.0 67.0 67.0 66.8 67.0 67.1 67.0 67.1 Min: 66.8, Avg: 67.1, Max: 67.9, Diff: 1.1, Sum: 2146.3] [Termination (ms): 0.8 1.0 0.9 0.8 0.8 0.8 0.9 0.4 0.9 0.6 0.4 0.4 0.9 0.8 0.8 0.9 0.9 0.9 0.7 0.8 0.6 0.8 0.7 0.9 0.9 0.9 0.8 0.9 0.8 0.7 0.9 0.9 Min: 0.4, Avg: 0.8, Max: 1.0, Diff: 0.6, Sum: 25.3] [Termination Attempts: 22 13 18 13 18 17 17 1 14 17 18 13 17 14 20 11 10 13 19 6 13 7 9 14 13 9 11 10 19 21 16 8 Min: 1, Avg: 13.8, Max: 22, Diff: 21, Sum: 441] [GC Worker Other (ms): 0.1 0.0 0.0 0.2 0.0 0.1 0.0 0.0 0.0 0.1 0.0 0.0 0.0 0.0 0.0 0.1 0.0 0.0 0.0 0.1 0.0 0.0 0.0 0.0 0.2 0.0 0.0 0.0 0.1 0.0 0.1 0.0 Min: 0.0, Avg: 0.0, Max: 0.2, Diff: 0.1, Sum: 1.5] [GC Worker Total (ms): 107.0 107.0 106.9 107.0 106.9 106.9 106.9 106.9 106.8 106.8 106.8 106.8 106.8 106.8 106.7 106.8 106.7 106.7 106.7 106.8 106.7 106.6 106.6 106.6 106.7 106.6 106.5 106.5 106.6 106.5 106.6 106.5 Min: 106.5, Avg: 106.7, Max: 107.0, Diff: 0.6, Sum: 3415.9] [GC Worker End (ms): 346723113.7 346723113.7 346723113.7 346723113.8 346723113.7 346723113.8 346723113.7 346723113.7 346723113.7 346723113.7 346723113.7 346723113.7 346723113.7 346723113.7 346723113.7 346723113.8 346723113.7 346723113.7 346723113.7 346723113.8 346723113.7 346723113.7 346723113.7 346723113.7 346723113.8 346723113.7 346723113.7 346723113.7 346723113.8 346723113.7 346723113.8 346723113.7 Min: 346723113.7, Avg: 346723113.7, Max: 346723113.8, Diff: 0.1] [Code Root Fixup: 0.1 ms] [Code Root Purge: 0.0 ms] [Clear CT: 4.5 ms] [Other: 15.5 ms] [Choose CSet: 0.0 ms] [Ref Proc: 9.8 ms] [Ref Enq: 0.6 ms] [Redirty Cards: 1.2 ms] [Parallel Redirty: 0.8 0.8 0.8 0.8 0.8 0.8 0.8 0.8 0.8 0.8 0.8 0.8 0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.6 0.6 0.6 0.6 0.6 0.0 Min: 0.0, Avg: 0.7, Max: 0.8, Diff: 0.8, Sum: 22.6] [Redirtied Cards: 16384 15531 15569 16437 16135 14488 13742 14069 14114 15427 9009 8192 7936 7168 8303 8082 7026 7059 7314 7483 7062 6641 11720 7022 10834 10024 6945 10672 10346 9770 3840 0 Min: 0, Avg: 10135.8, Max: 16437, Diff: 16437, Sum: 324344] [Humongous Register: 0.2 ms] [Humongous Total: 0] [Humongous Candidate: 0] [Humongous Reclaim: 0.1 ms] [Humongous Reclaimed: 0] [Free CSet: 2.8 ms] [Young Free CSet: 2.8 ms] [Non-Young Free CSet: 0.0 ms] [Eden: 4032.0M(4032.0M)->0.0B(4032.0M) Survivors: 576.0M->576.0M Heap: 44.9G(90.0G)->41.4G(90.0G)] Heap after GC invocations=10615 (full 0): garbage-first heap total 94371840K, used 43360155K [0x00007f017c000000, 0x00007f017e005a00, 0x00007f17fc000000) region size 32768K, 18 young (589824K), 18 survivors (589824K) Metaspace used 78702K, capacity 79547K, committed 79588K, reserved 79872K } [Times: user=3.57 sys=0.09, real=0.13 secs] 2015-10-18T07:06:58.184-0400: 346723.135: Total time for which application threads were stopped: 0.1313165 seconds, Stopping threads took: 0.0018426 seconds 2015-10-18T07:07:15.146-0400: 346740.097: Total time for which application threads were stopped: 0.0007456 seconds, Stopping threads took: 0.0000971 seconds 2015-10-18T07:07:16.141-0400: 346741.092: Total time for which application threads were stopped: 0.0011149 seconds, Stopping threads took: 0.0002058 seconds 2015-10-18T07:07:18.313-0400: 346743.264: Total time for which application threads were stopped: 0.0010722 seconds, Stopping threads took: 0.0001563 seconds {Heap before GC invocations=10615 (full 0): garbage-first heap total 94371840K, used 47488923K [0x00007f017c000000, 0x00007f017e005a00, 0x00007f17fc000000) region size 32768K, 144 young (4718592K), 18 survivors (589824K) Metaspace used 78702K, capacity 79547K, committed 79588K, reserved 79872K 346752.983: [G1Ergonomics (Concurrent Cycles) initiate concurrent cycle, reason: concurrent cycle initiation requested] 2015-10-18T07:07:28.032-0400: 346752.983: [GC pause (G1 Evacuation Pause) (young) (initial-mark) Desired survivor size 301989888 bytes, new threshold 2 (max 15) - age 1: 193262688 bytes, 193262688 total - age 2: 195192416 bytes, 388455104 total 346752.984: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 55586, predicted base time: 30.65 ms, remaining time: 69.35 ms, target pause time: 100.00 ms] 346752.984: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 126 regions, survivors: 18 regions, predicted young region time: 83.03 ms] 346752.984: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 126 regions, survivors: 18 regions, old: 0 regions, predicted pause time: 113.68 ms, target pause time: 100.00 ms] 2015-10-18T07:07:28.153-0400: 346753.104: [SoftReference, 0 refs, 0.0031514 secs]2015-10-18T07:07:28.156-0400: 346753.107: [WeakReference, 72 refs, 0.0012962 secs]2015-10-18T07:07:28.157-0400: 346753.108: [FinalReference, 489 refs, 0.0016356 secs]2015-10-18T07:07:28.159-0400: 346753.110: [PhantomReference, 6 refs, 1 refs, 0.0029412 secs]2015-10-18T07:07:28.162-0400: 346753.113: [JNI Weak Reference, 0.0000709 secs], 0.1410925 secs] [Parallel Time: 117.7 ms, GC Workers: 32] [GC Worker Start (ms): 346752984.8 346752984.8 346752984.8 346752984.8 346752984.9 346752984.9 346752984.9 346752984.9 346752984.9 346752985.0 346752985.0 346752985.0 346752985.0 346752985.0 346752985.0 346752985.0 346752985.1 346752985.1 346752985.1 346752985.1 346752985.1 346752985.1 346752985.1 346752985.2 346752985.2 346752985.2 346752985.2 346752985.2 346752985.2 346752985.2 346752985.2 346752985.2 Min: 346752984.8, Avg: 346752985.0, Max: 346752985.2, Diff: 0.4] [Ext Root Scanning (ms): 4.0 3.8 3.9 3.8 3.8 3.9 3.8 3.8 3.8 3.7 3.6 3.7 3.7 3.8 3.7 3.7 3.6 3.7 3.7 3.6 3.6 3.7 3.5 3.4 3.5 3.6 3.4 3.4 3.6 3.6 3.6 3.4 Min: 3.4, Avg: 3.7, Max: 4.0, Diff: 0.7, Sum: 117.2] [Thread Roots (ms): 0.0 3.7 0.2 0.2 0.2 0.2 0.1 0.1 0.1 0.1 0.1 0.2 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.2, Max: 3.7, Diff: 3.7, Sum: 5.5] [StringTable Roots (ms): 0.0 0.0 0.6 0.6 0.3 0.6 0.6 0.6 0.6 0.6 0.6 0.5 0.6 0.0 0.6 0.6 0.6 0.6 0.6 0.6 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.4 0.4 0.4 0.4 Min: 0.0, Avg: 0.5, Max: 0.6, Diff: 0.6, Sum: 15.5] [Universe Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [JNI Handles Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [ObjectSynchronizer Roots (ms): 0.0 0.0 0.0 0.0 0.3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.3, Diff: 0.3, Sum: 0.3] [FlatProfiler Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Management Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [SystemDictionary Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 1.1, Diff: 1.1, Sum: 1.1] [CLDG Roots (ms): 1.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 1.2, Diff: 1.2, Sum: 1.2] [JVMTI Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [CodeCache Roots (ms): 39.6 39.8 39.6 39.6 39.7 39.5 40.0 39.6 39.6 39.6 40.3 40.1 40.1 40.0 39.7 39.6 39.6 39.6 39.6 40.0 39.6 39.6 45.7 40.2 39.7 39.6 39.7 40.3 39.6 40.0 40.3 39.8 Min: 39.5, Avg: 40.0, Max: 45.7, Diff: 6.2, Sum: 1279.5] [CM RefProcessor Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Wait For Strong CLD (ms): 2.8 0.0 3.1 3.0 3.0 3.1 3.0 2.9 3.1 3.0 2.9 2.9 2.9 2.6 3.0 3.0 2.9 3.1 3.1 3.0 3.0 3.1 2.9 2.9 2.9 3.1 2.9 2.9 3.1 3.1 3.1 2.9 Min: 0.0, Avg: 2.9, Max: 3.1, Diff: 3.1, Sum: 92.1] [Weak CLD Roots (ms): 0.0 0.1 0.0 0.0 0.0 0.0 0.0 0.1 0.0 0.0 0.1 0.1 0.1 0.0 0.1 0.0 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.1 0.0 0.0 0.1 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 1.3] [SATB Filtering (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Update RS (ms): 10.0 9.0 8.6 9.1 9.2 8.9 8.6 9.2 9.1 8.8 9.4 9.0 8.9 8.7 9.2 8.8 9.4 9.1 8.8 9.1 9.1 8.8 8.9 8.8 9.7 9.2 9.3 8.9 9.0 8.7 8.6 9.0 Min: 8.6, Avg: 9.0, Max: 10.0, Diff: 1.4, Sum: 288.8] [Processed Buffers: 9 10 11 6 6 6 8 6 9 5 8 6 7 8 7 5 7 5 5 6 6 9 9 9 5 10 9 9 13 8 9 11 Min: 5, Avg: 7.7, Max: 13, Diff: 8, Sum: 247] [Scan RS (ms): 29.6 30.8 30.9 30.5 30.5 30.6 31.4 30.4 30.5 30.8 30.9 31.1 31.3 31.3 30.5 30.8 30.2 30.5 30.9 30.9 30.5 30.8 36.9 31.4 30.0 30.4 30.5 31.4 30.5 31.2 31.7 30.8 Min: 29.6, Avg: 31.0, Max: 36.9, Diff: 7.3, Sum: 990.5] [Code Root Scanning (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] [Object Copy (ms): 71.7 71.5 71.7 71.8 72.0 71.7 71.5 71.8 71.7 71.7 71.2 71.2 71.7 71.3 71.8 71.7 72.0 71.8 71.7 71.4 71.6 71.7 65.7 71.2 71.6 71.6 71.8 71.1 71.7 71.4 70.9 71.7 Min: 65.7, Avg: 71.4, Max: 72.0, Diff: 6.2, Sum: 2285.0] [Termination (ms): 1.4 1.6 1.5 1.4 1.1 1.5 1.2 1.4 1.3 1.4 1.4 1.5 0.9 1.4 1.3 1.4 1.2 1.3 1.3 1.4 1.5 1.4 1.4 1.5 1.5 1.5 1.3 1.4 1.5 1.2 1.5 1.3 Min: 0.9, Avg: 1.4, Max: 1.6, Diff: 0.6, Sum: 44.2] [Termination Attempts: 13 12 14 14 12 17 11 14 11 15 9 17 5 11 9 16 7 9 14 8 10 16 13 7 14 14 12 16 17 4 17 7 Min: 4, Avg: 12.0, Max: 17, Diff: 13, Sum: 385] [GC Worker Other (ms): 0.0 0.0 0.1 0.1 0.1 0.0 0.0 0.1 0.0 0.0 0.0 0.1 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.1 0.0 0.1 0.0 0.0 0.0 0.0 0.1 0.1 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 1.5] [GC Worker Total (ms): 116.7 116.7 116.7 116.6 116.7 116.6 116.6 116.6 116.6 116.5 116.5 116.6 116.6 116.5 116.5 116.5 116.4 116.4 116.4 116.4 116.5 116.4 116.4 116.3 116.3 116.3 116.3 116.4 116.3 116.3 116.3 116.2 Min: 116.2, Avg: 116.5, Max: 116.7, Diff: 0.5, Sum: 3727.2] [GC Worker End (ms): 346753101.5 346753101.5 346753101.6 346753101.5 346753101.6 346753101.5 346753101.5 346753101.5 346753101.5 346753101.5 346753101.5 346753101.5 346753101.6 346753101.5 346753101.5 346753101.5 346753101.5 346753101.5 346753101.5 346753101.5 346753101.6 346753101.5 346753101.5 346753101.5 346753101.5 346753101.5 346753101.5 346753101.6 346753101.6 346753101.5 346753101.5 346753101.5 Min: 346753101.5, Avg: 346753101.5, Max: 346753101.6, Diff: 0.1] [Code Root Fixup: 0.1 ms] [Code Root Purge: 0.0 ms] [Clear CT: 5.1 ms] [Other: 18.2 ms] [Choose CSet: 0.0 ms] [Ref Proc: 10.7 ms] [Ref Enq: 0.5 ms] [Redirty Cards: 2.3 ms] [Parallel Redirty: 1.8 1.8 1.8 1.7 1.7 1.7 1.7 1.7 1.7 1.7 1.7 1.7 1.7 1.7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.8, Max: 1.8, Diff: 1.8, Sum: 24.3] [Redirtied Cards: 20279 24852 24843 24186 19664 23217 23422 24870 25919 25178 23849 18080 21743 22024 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Min: 0, Avg: 10066.4, Max: 25919, Diff: 25919, Sum: 322126] [Humongous Register: 0.1 ms] [Humongous Total: 0] [Humongous Candidate: 0] [Humongous Reclaim: 0.1 ms] [Humongous Reclaimed: 0] [Free CSet: 2.5 ms] [Young Free CSet: 2.5 ms] [Non-Young Free CSet: 0.0 ms] [Eden: 4032.0M(4032.0M)->0.0B(4032.0M) Survivors: 576.0M->576.0M Heap: 45.3G(90.0G)->41.7G(90.0G)] Heap after GC invocations=10616 (full 0): garbage-first heap total 94371840K, used 43732854K [0x00007f017c000000, 0x00007f017e005a00, 0x00007f17fc000000) region size 32768K, 18 young (589824K), 18 survivors (589824K) Metaspace used 78702K, capacity 79547K, committed 79588K, reserved 79872K } [Times: user=3.84 sys=0.08, real=0.14 secs] 2015-10-18T07:07:28.174-0400: 346753.126: [GC concurrent-root-region-scan-start] 2015-10-18T07:07:28.175-0400: 346753.126: Total time for which application threads were stopped: 0.1434439 seconds, Stopping threads took: 0.0001640 seconds 2015-10-18T07:07:28.260-0400: 346753.211: [GC concurrent-root-region-scan-end, 0.0852149 secs] 2015-10-18T07:07:28.260-0400: 346753.211: [GC concurrent-mark-start] 2015-10-18T07:07:28.965-0400: 346753.916: [GC concurrent-mark-end, 0.7049345 secs] 2015-10-18T07:07:28.966-0400: 346753.917: [GC remark 2015-10-18T07:07:28.966-0400: 346753.917: [Finalize Marking, 115.4404545 secs] 2015-10-18T07:09:24.406-0400: 346869.357: [GC ref-proc2015-10-18T07:09:24.406-0400: 346869.357: [SoftReference, 54 refs, 0.0028579 secs]2015-10-18T07:09:24.409-0400: 346869.360: [WeakReference, 418 refs, 0.0020702 secs]2015-10-18T07:09:24.411-0400: 346869.362: [FinalReference, 1633 refs, 0.0025592 secs]2015-10-18T07:09:24.414-0400: 346869.365: [PhantomReference, 6 refs, 21 refs, 0.0040350 secs]2015-10-18T07:09:24.418-0400: 346869.369: [JNI Weak Reference, 0.0001088 secs], 0.0130411 secs] 2015-10-18T07:09:24.419-0400: 346869.370: [Unloading 2015-10-18T07:09:24.420-0400: 346869.371: [System Dictionary Unloading, 0.0001458 secs] 2015-10-18T07:09:24.420-0400: 346869.371: [Parallel Unloading, 0.0153302 secs] 2015-10-18T07:09:24.435-0400: 346869.386: [Deallocate Metadata, 0.0000526 secs], 0.0159394 secs], 115.5038062 secs] [Times: user=1641.24 sys=743.90, real=115.48 secs] 2015-10-18T07:09:24.469-0400: 346869.421: Total time for which application threads were stopped: 115.5047519 seconds, Stopping threads took: 0.0001246 seconds 2015-10-18T07:09:24.470-0400: 346869.421: [GC cleanup 41G->41G(90G), 0.1233809 secs] [Times: user=3.83 sys=0.00, real=0.13 secs] 2015-10-18T07:09:24.594-0400: 346869.545: Total time for which application threads were stopped: 0.1240198 seconds, Stopping threads took: 0.0001006 seconds 2015-10-18T07:09:24.594-0400: 346869.545: [GC concurrent-cleanup-start] 2015-10-18T07:09:24.594-0400: 346869.545: [GC concurrent-cleanup-end, 0.0003056 secs] 2015-10-18T07:09:24.594-0400: 346869.546: Total time for which application threads were stopped: 0.0006490 seconds, Stopping threads took: 0.0001584 seconds 2015-10-18T07:09:24.597-0400: 346869.548: Total time for which application threads were stopped: 0.0006890 seconds, Stopping threads took: 0.0002038 seconds 2015-10-18T07:09:37.098-0400: 346882.049: Total time for which application threads were stopped: 0.0015512 seconds, Stopping threads took: 0.0006916 seconds {Heap before GC invocations=10617 (full 0): garbage-first heap total 94371840K, used 47042422K [0x00007f017c000000, 0x00007f017e005a00, 0x00007f17fc000000) region size 32768K, 145 young (4751360K), 18 survivors (589824K) Metaspace used 78704K, capacity 79547K, committed 79588K, reserved 79872K 2015-10-18T07:09:37.100-0400: 346882.051: [GC pause (GCLocker Initiated GC) (young) Desired survivor size 301989888 bytes, new threshold 2 (max 15) - age 1: 207377760 bytes, 207377760 total - age 2: 151159664 bytes, 358537424 total 346882.051: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 72161, predicted base time: 35.14 ms, remaining time: 64.86 ms, target pause time: 100.00 ms] 346882.051: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 127 regions, survivors: 18 regions, predicted young region time: 75.17 ms] 346882.051: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 127 regions, survivors: 18 regions, old: 0 regions, predicted pause time: 110.32 ms, target pause time: 100.00 ms] 2015-10-18T07:09:37.257-0400: 346882.208: [SoftReference, 0 refs, 0.0030782 secs]2015-10-18T07:09:37.260-0400: 346882.211: [WeakReference, 333 refs, 0.0017802 secs]2015-10-18T07:09:37.262-0400: 346882.213: [FinalReference, 1864 refs, 0.0013357 secs]2015-10-18T07:09:37.263-0400: 346882.214: [PhantomReference, 3 refs, 1 refs, 0.0037742 secs]2015-10-18T07:09:37.267-0400: 346882.218: [JNI Weak Reference, 0.0000415 secs] 346882.230: [G1Ergonomics (Concurrent Cycles) do not request concurrent cycle initiation, reason: still doing mixed collections, occupancy: 44828721152 bytes, allocation request: 0 bytes, threshold: 43486543845 bytes (45.00 %), source: end of GC] 346882.230: [G1Ergonomics (Mixed GCs) start mixed GCs, reason: candidate old regions available, candidate old regions: 1129 regions, reclaimable: 11728668024 bytes (12.14 %), threshold: 10.00 %] , 0.1783612 secs] [Parallel Time: 154.7 ms, GC Workers: 32] [GC Worker Start (ms): 346882051.9 346882052.0 346882052.0 346882052.0 346882052.0 346882052.0 346882052.0 346882052.1 346882052.1 346882052.1 346882052.1 346882052.1 346882052.1 346882052.1 346882052.1 346882052.2 346882052.2 346882052.2 346882052.2 346882052.2 346882052.2 346882052.2 346882052.2 346882052.3 346882052.3 346882052.3 346882052.3 346882052.3 346882052.3 346882052.3 346882052.3 346882052.3 Min: 346882051.9, Avg: 346882052.2, Max: 346882052.3, Diff: 0.4] [Ext Root Scanning (ms): 1.8 0.6 0.6 0.6 0.6 0.6 0.6 0.9 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.4 0.4 0.4 0.4 0.4 0.4 0.4 0.4 0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.3 Min: 0.3, Avg: 0.5, Max: 1.8, Diff: 1.6, Sum: 15.8] [Thread Roots (ms): 0.0 0.2 0.2 0.2 0.1 0.1 0.2 0.1 0.1 0.1 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.2, Diff: 0.2, Sum: 1.4] [StringTable Roots (ms): 0.0 0.4 0.4 0.4 0.5 0.4 0.4 0.0 0.5 0.5 0.5 0.1 0.4 0.4 0.4 0.4 0.4 0.4 0.4 0.4 0.4 0.4 0.4 0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.3 Min: 0.0, Avg: 0.4, Max: 0.5, Diff: 0.5, Sum: 11.3] [Universe Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [JNI Handles Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [ObjectSynchronizer Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.3, Diff: 0.3, Sum: 0.3] [FlatProfiler Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Management Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [SystemDictionary Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.8, Diff: 0.8, Sum: 0.8] [CLDG Roots (ms): 1.8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.1, Max: 1.8, Diff: 1.8, Sum: 1.8] [JVMTI Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [CodeCache Roots (ms): 47.5 48.8 48.4 48.6 48.7 47.5 47.6 48.3 48.7 48.6 48.5 48.5 48.8 48.8 48.6 48.6 48.5 48.7 47.8 48.7 48.6 48.8 48.7 48.6 48.5 48.5 48.5 48.7 48.5 48.5 48.6 48.8 Min: 47.5, Avg: 48.5, Max: 48.8, Diff: 1.2, Sum: 1551.6] [CM RefProcessor Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Wait For Strong CLD (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Weak CLD Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [SATB Filtering (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Update RS (ms): 9.2 10.4 10.1 10.5 10.6 10.3 10.7 10.4 10.3 10.2 10.6 10.1 11.1 10.7 10.4 10.3 10.4 10.4 10.8 10.5 10.2 10.3 10.5 10.3 10.2 10.5 10.3 11.0 10.1 10.3 10.3 10.7 Min: 9.2, Avg: 10.4, Max: 11.1, Diff: 1.9, Sum: 332.8] [Processed Buffers: 9 10 10 8 10 8 8 10 12 9 10 10 10 10 9 9 12 8 12 10 11 13 10 10 10 8 8 12 9 8 9 10 Min: 8, Avg: 9.8, Max: 13, Diff: 5, Sum: 312] [Scan RS (ms): 38.4 38.3 38.4 38.1 38.1 37.2 36.9 38.0 38.4 38.4 37.9 38.4 37.6 38.1 38.2 38.3 38.1 38.3 37.0 38.2 38.4 38.4 38.2 38.3 38.3 38.0 38.2 37.7 38.4 38.2 38.3 38.0 Min: 36.9, Avg: 38.1, Max: 38.4, Diff: 1.5, Sum: 1218.7] [Code Root Scanning (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): 104.3 104.3 104.6 104.4 104.3 105.5 105.5 104.3 104.4 104.5 104.5 104.6 104.3 104.3 104.5 104.4 104.6 104.4 105.3 104.4 104.5 104.3 104.3 104.5 104.6 104.6 104.5 104.4 104.6 104.6 104.5 104.3 Min: 104.3, Avg: 104.5, Max: 105.5, Diff: 1.2, Sum: 3345.0] [Termination (ms): 0.0 0.0 0.0 0.0 0.1 0.0 0.0 0.0 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.9] [Termination Attempts: 8 6 8 7 8 10 1 15 12 9 10 14 6 11 9 8 11 9 10 9 7 8 7 9 11 8 15 7 11 11 12 9 Min: 1, Avg: 9.2, Max: 15, Diff: 14, Sum: 296] [GC Worker Other (ms): 0.0 0.1 0.1 0.1 0.0 0.0 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.1 0.1 0.2 0.1 0.0 0.0 0.0 0.1 0.1 0.2 0.1 0.0 0.1 0.0 Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.0] [GC Worker Total (ms): 153.8 153.8 153.8 153.8 153.7 153.7 153.7 153.7 153.6 153.6 153.6 153.6 153.6 153.6 153.6 153.6 153.6 153.6 153.6 153.6 153.7 153.5 153.5 153.5 153.5 153.5 153.5 153.6 153.5 153.4 153.5 153.4 Min: 153.4, Avg: 153.6, Max: 153.8, Diff: 0.5, Sum: 4915.2] [GC Worker End (ms): 346882205.7 346882205.8 346882205.8 346882205.8 346882205.7 346882205.7 346882205.7 346882205.7 346882205.7 346882205.7 346882205.7 346882205.7 346882205.7 346882205.7 346882205.7 346882205.7 346882205.7 346882205.7 346882205.8 346882205.8 346882205.9 346882205.7 346882205.7 346882205.7 346882205.7 346882205.8 346882205.8 346882205.9 346882205.8 346882205.7 346882205.8 346882205.7 Min: 346882205.7, Avg: 346882205.8, Max: 346882205.9, Diff: 0.2] [Code Root Fixup: 0.1 ms] [Code Root Purge: 0.0 ms] [Clear CT: 5.0 ms] [Other: 18.6 ms] [Choose CSet: 0.0 ms] [Ref Proc: 11.2 ms] [Ref Enq: 0.6 ms] [Redirty Cards: 3.0 ms] [Parallel Redirty: 2.2 2.1 2.1 2.1 2.1 2.1 2.1 2.1 2.1 2.1 2.1 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 1.9 1.9 1.9 1.9 1.9 1.9 1.9 1.9 1.9 1.8 1.8 1.8 1.8 Min: 1.8, Avg: 2.0, Max: 2.2, Diff: 0.4, Sum: 63.5] [Redirtied Cards: 15014 13056 15083 13907 14366 13979 14002 13824 12032 14503 13240 13824 10240 13404 19006 19871 19294 23587 768 19920 23717 22881 23341 20632 19993 19645 20288 20548 21391 20430 9303 16760 Min: 768, Avg: 16620.3, Max: 23717, Diff: 22949, Sum: 531849] [Humongous Register: 0.2 ms] [Humongous Total: 0] [Humongous Candidate: 0] [Humongous Reclaim: 0.1 ms] [Humongous Reclaimed: 0] [Free CSet: 2.6 ms] [Young Free CSet: 2.6 ms] [Non-Young Free CSet: 0.0 ms] [Eden: 4064.0M(4032.0M)->0.0B(4032.0M) Survivors: 576.0M->576.0M Heap: 44.9G(90.0G)->42.3G(90.0G)] Heap after GC invocations=10618 (full 0): garbage-first heap total 94371840K, used 44344390K [0x00007f017c000000, 0x00007f017e005a00, 0x00007f17fc000000) region size 32768K, 18 young (589824K), 18 survivors (589824K) Metaspace used 78704K, capacity 79547K, committed 79588K, reserved 79872K } [Times: user=5.13 sys=0.04, real=0.18 secs] 2015-10-18T07:09:37.280-0400: 346882.231: Total time for which application threads were stopped: 0.1801263 seconds, Stopping threads took: 0.0000659 seconds 2015-10-18T07:09:44.274-0400: 346889.225: Total time for which application threads were stopped: 0.0008202 seconds, Stopping threads took: 0.0001214 seconds 2015-10-18T07:09:44.275-0400: 346889.226: Total time for which application threads were stopped: 0.0008439 seconds, Stopping threads took: 0.0003144 seconds 2015-10-18T07:09:44.276-0400: 346889.227: Total time for which application threads were stopped: 0.0005881 seconds, Stopping threads took: 0.0001411 seconds 2015-10-18T07:09:44.390-0400: 346889.341: Total time for which application threads were stopped: 0.0023666 seconds, Stopping threads took: 0.0016287 seconds 2015-10-18T07:09:46.565-0400: 346891.516: Total time for which application threads were stopped: 0.0025967 seconds, Stopping threads took: 0.0018072 seconds 2015-10-18T07:09:51.526-0400: 346896.477: Total time for which application threads were stopped: 0.0010744 seconds, Stopping threads took: 0.0003298 seconds 2015-10-18T07:09:51.526-0400: 346896.477: Total time for which application threads were stopped: 0.0004840 seconds, Stopping threads took: 0.0000811 seconds 2015-10-18T07:09:51.527-0400: 346896.478: Total time for which application threads were stopped: 0.0004491 seconds, Stopping threads took: 0.0000825 seconds 2015-10-18T07:09:51.922-0400: 346896.873: Total time for which application threads were stopped: 0.0016204 seconds, Stopping threads took: 0.0009132 seconds {Heap before GC invocations=10618 (full 0): garbage-first heap total 94371840K, used 48473158K [0x00007f017c000000, 0x00007f017e005a00, 0x00007f17fc000000) region size 32768K, 145 young (4751360K), 18 survivors (589824K) Metaspace used 78704K, capacity 79547K, committed 79588K, reserved 79872K 2015-10-18T07:09:51.922-0400: 346896.873: [GC pause (GCLocker Initiated GC) (mixed) Desired survivor size 301989888 bytes, new threshold 1 (max 15) - age 1: 392388224 bytes, 392388224 total - age 2: 21600448 bytes, 413988672 total 346896.873: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 59516, predicted base time: 32.78 ms, remaining time: 67.22 ms, target pause time: 100.00 ms] 346896.874: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 127 regions, survivors: 18 regions, predicted young region time: 88.09 ms] 346896.877: [G1Ergonomics (CSet Construction) finish adding old regions to CSet, reason: reclaimable percentage not over threshold, old: 118 regions, max: 288 regions, reclaimable: 9654274128 bytes (9.99 %), threshold: 10.00 %] 346896.877: [G1Ergonomics (CSet Construction) added expensive regions to CSet, reason: old CSet region num not reached min, old: 118 regions, expensive: 118 regions, min: 142 regions, remaining time: 0.00 ms] 346896.877: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 127 regions, survivors: 18 regions, old: 118 regions, predicted pause time: 317.68 ms, target pause time: 100.00 ms] 2015-10-18T07:09:52.282-0400: 346897.233: [SoftReference, 0 refs, 0.0029883 secs]2015-10-18T07:09:52.285-0400: 346897.236: [WeakReference, 38 refs, 0.0014392 secs]2015-10-18T07:09:52.286-0400: 346897.238: [FinalReference, 540 refs, 0.0019740 secs]2015-10-18T07:09:52.288-0400: 346897.240: [PhantomReference, 2 refs, 0 refs, 0.0037135 secs]2015-10-18T07:09:52.292-0400: 346897.243: [JNI Weak Reference, 0.0000363 secs] 346897.259: [G1Ergonomics (Concurrent Cycles) do not request concurrent cycle initiation, reason: still doing mixed collections, occupancy: 46103789568 bytes, allocation request: 0 bytes, threshold: 43486543845 bytes (45.00 %), source: end of GC] 346897.259: [G1Ergonomics (Mixed GCs) do not continue mixed GCs, reason: reclaimable percentage not over threshold, candidate old regions: 1011 regions, reclaimable: 9654274128 bytes (9.99 %), threshold: 10.00 %] , 0.3856805 secs] [Parallel Time: 353.5 ms, GC Workers: 32] [GC Worker Start (ms): 346896877.6 346896877.6 346896877.7 346896877.7 346896877.7 346896877.8 346896877.8 346896877.8 346896877.8 346896877.8 346896877.8 346896877.9 346896877.9 346896877.9 346896877.9 346896877.9 346896878.0 346896878.0 346896878.0 346896878.0 346896878.0 346896878.0 346896878.0 346896878.1 346896878.1 346896878.1 346896878.1 346896878.1 346896878.4 346896878.4 346896878.4 346896878.5 Min: 346896877.6, Avg: 346896878.0, Max: 346896878.5, Diff: 0.8] [Ext Root Scanning (ms): 2.7 0.7 0.7 0.7 0.6 0.9 0.6 0.6 0.6 0.6 0.5 0.5 0.5 0.5 0.4 0.4 0.4 0.4 0.4 0.4 0.4 0.4 0.4 0.3 0.3 0.3 0.3 0.3 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.5, Max: 2.7, Diff: 2.7, Sum: 15.7] [Thread Roots (ms): 0.0 0.3 0.3 0.3 0.2 0.2 0.1 0.2 0.1 0.2 0.1 0.1 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.3, Sum: 2.2] [StringTable Roots (ms): 0.0 0.5 0.4 0.3 0.4 0.0 0.4 0.4 0.5 0.3 0.4 0.4 0.4 0.7 0.4 7.8 0.7 0.4 0.4 0.4 0.4 0.4 0.3 0.3 0.3 0.3 0.3 0.3 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.6, Max: 7.8, Diff: 7.8, Sum: 17.9] [Universe Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [JNI Handles Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [ObjectSynchronizer Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.1] [FlatProfiler Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Management Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [SystemDictionary Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.7, Diff: 0.7, Sum: 0.7] [CLDG Roots (ms): 2.7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.1, Max: 2.7, Diff: 2.7, Sum: 2.7] [JVMTI Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [CodeCache Roots (ms): 80.1 81.4 81.9 74.4 81.9 81.7 81.6 73.9 76.1 83.2 76.1 81.6 81.8 81.8 82.0 74.6 90.8 81.8 81.6 81.8 81.9 81.8 81.8 81.8 81.7 81.9 81.6 81.0 81.9 82.0 81.9 81.9 Min: 73.9, Avg: 81.0, Max: 90.8, Diff: 16.9, Sum: 2591.1] [CM RefProcessor Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Wait For Strong CLD (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Weak CLD Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [SATB Filtering (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Update RS (ms): 7.0 9.0 8.8 1.7 8.6 8.3 8.2 1.2 1.4 8.4 1.3 8.3 8.5 8.4 8.9 1.4 8.8 8.3 8.2 9.0 8.4 8.8 8.7 8.5 8.7 9.3 8.4 14.8 9.2 8.6 8.6 8.6 Min: 1.2, Avg: 7.6, Max: 14.8, Diff: 13.5, Sum: 244.2] [Processed Buffers: 12 8 10 3 9 9 9 1 2 9 5 8 13 7 10 1 9 9 11 10 9 11 8 9 9 9 8 7 9 10 9 10 Min: 1, Avg: 8.2, Max: 13, Diff: 12, Sum: 263] [Scan RS (ms): 73.1 72.3 73.1 72.7 73.3 73.4 73.4 72.4 74.7 74.7 74.8 73.3 73.3 73.4 73.1 73.2 82.0 73.4 73.4 72.8 73.5 72.9 73.1 73.3 73.0 72.6 73.1 66.2 72.7 73.4 73.3 73.3 Min: 66.2, Avg: 73.3, Max: 82.0, Diff: 15.7, Sum: 2346.2] [Code Root Scanning (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.2 0.0 0.0 0.0 0.0 0.1 0.0 0.0 0.0 0.0 0.1 0.0 0.0 0.0 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.2, Diff: 0.2, Sum: 0.6] [Object Copy (ms): 269.6 270.3 269.8 277.2 269.7 269.6 269.9 277.7 275.4 268.4 275.5 269.9 269.8 269.8 269.6 277.0 260.8 269.8 270.0 269.9 269.8 270.1 269.8 269.9 269.9 269.7 270.0 270.6 269.7 269.6 269.6 269.8 Min: 260.8, Avg: 270.6, Max: 277.7, Diff: 16.9, Sum: 8658.4] [Termination (ms): 0.3 0.2 0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.4 0.3 0.2 0.0 0.3 0.2 0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.1 Min: 0.0, Avg: 0.3, Max: 0.4, Diff: 0.3, Sum: 9.0] [Termination Attempts: 41 38 49 39 45 35 39 50 38 35 31 39 47 50 49 37 35 41 47 41 3 7 31 39 44 37 45 35 46 40 45 38 Min: 3, Avg: 38.6, Max: 50, Diff: 47, Sum: 1236] [GC Worker Other (ms): 0.0 0.0 0.0 0.0 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.1 0.0 0.0 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.9] [GC Worker Total (ms): 352.7 352.6 352.7 352.6 352.6 352.6 352.5 352.5 352.5 352.5 352.5 352.4 352.4 352.4 352.4 352.4 352.3 352.3 352.3 352.3 352.4 352.3 352.3 352.3 352.2 352.2 352.2 352.2 351.9 351.9 351.9 351.8 Min: 351.8, Avg: 352.3, Max: 352.7, Diff: 0.8, Sum: 11275.0] [GC Worker End (ms): 346897230.3 346897230.3 346897230.3 346897230.3 346897230.4 346897230.3 346897230.3 346897230.3 346897230.3 346897230.3 346897230.3 346897230.3 346897230.3 346897230.3 346897230.3 346897230.3 346897230.3 346897230.3 346897230.3 346897230.3 346897230.4 346897230.3 346897230.3 346897230.3 346897230.3 346897230.3 346897230.3 346897230.3 346897230.4 346897230.3 346897230.3 346897230.3 Min: 346897230.3, Avg: 346897230.3, Max: 346897230.4, Diff: 0.1] [Code Root Fixup: 0.2 ms] [Code Root Purge: 0.0 ms] [Clear CT: 5.5 ms] [Other: 26.5 ms] [Choose CSet: 3.6 ms] [Ref Proc: 12.1 ms] [Ref Enq: 0.8 ms] [Redirty Cards: 4.8 ms] [Parallel Redirty: 2.5 2.5 2.5 2.5 2.5 2.5 2.4 2.4 2.4 2.4 2.4 2.4 2.4 2.3 2.3 2.3 2.3 2.3 2.3 4.5 2.2 2.2 2.2 2.2 2.2 2.2 2.2 2.2 2.2 2.2 2.2 0.0 Min: 0.0, Avg: 2.3, Max: 4.5, Diff: 4.5, Sum: 74.2] [Redirtied Cards: 11597 11264 55387 56233 53310 57466 51402 55606 55978 54953 8448 53657 6400 5632 7680 8704 5632 7168 7424 7424 8432 4352 50425 7936 6673 48363 48500 47506 48396 46998 49563 0 Min: 0, Avg: 29640.9, Max: 57466, Diff: 57466, Sum: 948509] [Humongous Register: 0.2 ms] [Humongous Total: 0] [Humongous Candidate: 0] [Humongous Reclaim: 0.1 ms] [Humongous Reclaimed: 0] [Free CSet: 4.3 ms] [Young Free CSet: 0.6 ms] [Non-Young Free CSet: 3.7 ms] [Eden: 4064.0M(4032.0M)->0.0B(4032.0M) Survivors: 576.0M->576.0M Heap: 46.2G(90.0G)->43.5G(90.0G)] Heap after GC invocations=10619 (full 0): garbage-first heap total 94371840K, used 45613056K [0x00007f017c000000, 0x00007f017e005a00, 0x00007f17fc000000) region size 32768K, 18 young (589824K), 18 survivors (589824K) Metaspace used 78704K, capacity 79547K, committed 79588K, reserved 79872K } [Times: user=11.25 sys=0.05, real=0.39 secs] 2015-10-18T07:09:52.309-0400: 346897.260: Total time for which application threads were stopped: 0.3868101 seconds, Stopping threads took: 0.0001002 seconds 2015-10-18T07:09:53.265-0400: 346898.217: Total time for which application threads were stopped: 0.0018434 seconds, Stopping threads took: 0.0010742 seconds 2015-10-18T07:09:53.788-0400: 346898.739: Total time for which application threads were stopped: 0.0024891 seconds, Stopping threads took: 0.0014010 seconds 2015-10-18T07:09:55.970-0400: 346900.922: Total time for which application threads were stopped: 0.0013719 seconds, Stopping threads took: 0.0006626 seconds 2015-10-18T07:10:03.606-0400: 346908.557: Total time for which application threads were stopped: 0.0009251 seconds, Stopping threads took: 0.0001622 seconds 2015-10-18T07:10:05.791-0400: 346910.742: Total time for which application threads were stopped: 0.0005516 seconds, Stopping threads took: 0.0000785 seconds 2015-10-18T07:10:10.965-0400: 346915.916: Total time for which application threads were stopped: 0.0013605 seconds, Stopping threads took: 0.0006394 seconds {Heap before GC invocations=10619 (full 0): garbage-first heap total 94371840K, used 49741824K [0x00007f017c000000, 0x00007f017e005a00, 0x00007f17fc000000) region size 32768K, 145 young (4751360K), 18 survivors (589824K) Metaspace used 78704K, capacity 79547K, committed 79588K, reserved 79872K 2015-10-18T07:10:10.971-0400: 346915.922: [GC pause (GCLocker Initiated GC) (young) Desired survivor size 301989888 bytes, new threshold 1 (max 15) - age 1: 447704504 bytes, 447704504 total 346915.923: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 76514, predicted base time: 36.05 ms, remaining time: 63.95 ms, target pause time: 100.00 ms] 346915.923: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 127 regions, survivors: 18 regions, predicted young region time: 138.94 ms] 346915.923: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 127 regions, survivors: 18 regions, old: 0 regions, predicted pause time: 174.98 ms, target pause time: 100.00 ms] 2015-10-18T07:10:11.148-0400: 346916.099: [SoftReference, 0 refs, 0.0038994 secs]2015-10-18T07:10:11.152-0400: 346916.103: [WeakReference, 52 refs, 0.0021611 secs]2015-10-18T07:10:11.154-0400: 346916.105: [FinalReference, 579 refs, 0.0024199 secs]2015-10-18T07:10:11.156-0400: 346916.107: [PhantomReference, 3 refs, 1 refs, 0.0044465 secs]2015-10-18T07:10:11.161-0400: 346916.112: [JNI Weak Reference, 0.0000439 secs] 346916.122: [G1Ergonomics (Concurrent Cycles) request concurrent cycle initiation, reason: occupancy higher than threshold, occupancy: 49291460608 bytes, allocation request: 0 bytes, threshold: 43486543845 bytes (45.00 %), source: end of GC] , 0.1993445 secs] [Parallel Time: 173.7 ms, GC Workers: 32] [GC Worker Start (ms): 346915923.4 346915923.4 346915923.4 346915923.5 346915923.5 346915923.5 346915923.5 346915923.5 346915923.5 346915923.5 346915923.6 346915923.6 346915923.6 346915923.6 346915923.6 346915923.7 346915923.7 346915923.7 346915923.7 346915923.7 346915923.8 346915923.8 346915923.8 346915923.9 346915923.9 346915923.9 346915923.9 346915930.7 346915930.7 346915930.7 346915930.7 346915930.7 Min: 346915923.4, Avg: 346915924.7, Max: 346915930.7, Diff: 7.3] [Ext Root Scanning (ms): 3.4 0.8 0.7 0.7 0.7 0.7 0.7 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.8 0.5 0.5 0.5 0.5 0.4 0.4 0.4 0.3 0.3 0.3 0.3 0.3 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.5, Max: 3.4, Diff: 3.4, Sum: 17.5] [Thread Roots (ms): 0.0 0.2 0.2 0.2 0.2 0.2 0.1 0.1 0.1 0.1 0.1 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 1.8] [StringTable Roots (ms): 0.0 0.5 0.5 0.5 0.6 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.6 0.4 0.0 0.5 0.5 0.5 0.4 0.4 0.4 0.4 0.3 0.3 0.3 0.3 0.3 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.4, Max: 0.6, Diff: 0.6, Sum: 11.3] [Universe Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [JNI Handles Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [ObjectSynchronizer Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.1] [FlatProfiler Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Management Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [SystemDictionary Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.8, Diff: 0.8, Sum: 0.8] [CLDG Roots (ms): 3.4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.1, Max: 3.4, Diff: 3.4, Sum: 3.4] [JVMTI Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [CodeCache Roots (ms): 17.8 20.2 20.3 20.4 20.1 20.1 20.4 20.4 20.4 20.1 20.2 20.3 20.3 20.4 20.2 20.4 20.4 20.4 20.4 20.3 20.4 20.2 20.4 20.4 20.4 20.4 20.4 14.0 14.0 14.0 14.0 13.9 Min: 13.9, Avg: 19.3, Max: 20.4, Diff: 6.5, Sum: 616.1] [CM RefProcessor Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Wait For Strong CLD (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Weak CLD Roots (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [SATB Filtering (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Update RS (ms): 9.2 11.7 11.7 12.3 12.1 11.9 11.8 12.5 12.1 11.8 12.5 11.8 11.8 11.9 11.8 12.3 14.1 11.8 12.2 11.9 12.2 11.9 12.2 12.2 11.9 12.3 11.9 5.3 5.8 5.4 5.9 5.3 Min: 5.3, Avg: 11.0, Max: 14.1, Diff: 8.8, Sum: 351.7] [Processed Buffers: 7 11 11 12 11 10 11 15 10 12 10 11 11 11 11 12 6 19 13 13 11 12 9 9 10 12 9 8 7 5 7 6 Min: 5, Avg: 10.4, Max: 19, Diff: 14, Sum: 332] [Scan RS (ms): 8.5 8.5 8.6 8.1 8.0 8.2 8.6 7.9 8.3 8.2 7.7 8.5 8.5 8.6 8.4 8.1 6.3 8.6 8.1 8.4 8.2 8.3 8.2 8.2 8.5 8.1 8.4 8.7 8.1 8.7 8.1 8.6 Min: 6.3, Avg: 8.3, Max: 8.7, Diff: 2.4, Sum: 264.2] [Code Root Scanning (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): 151.3 151.4 151.4 151.3 151.5 151.6 151.3 151.3 151.4 151.6 151.4 151.3 151.3 151.2 151.2 151.3 151.3 151.3 151.3 151.3 151.3 151.5 151.3 151.3 151.3 151.3 151.3 151.2 151.2 151.2 151.2 151.3 Min: 151.2, Avg: 151.3, Max: 151.6, Diff: 0.5, Sum: 4842.3] [Termination (ms): 0.1 0.1 0.1 0.2 0.1 0.2 0.2 0.2 0.0 0.1 0.2 0.2 0.1 0.1 0.1 0.2 0.2 0.2 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.2 0.2 0.2 Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 4.6] [Termination Attempts: 2 3 7 5 4 3 3 5 1 5 7 6 6 4 6 4 5 3 6 4 5 3 5 3 5 6 3 3 5 4 4 3 Min: 1, Avg: 4.3, Max: 7, Diff: 6, Sum: 138] [GC Worker Other (ms): 0.0 0.1 0.1 0.0 0.1 0.0 0.0 0.0 0.0 0.1 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.1 0.1 0.1 0.1 0.1 0.2 0.0 0.0 0.1 0.0 0.1 Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 1.9] [GC Worker Total (ms): 172.6 172.7 172.7 172.6 172.6 172.6 172.5 172.5 172.5 172.5 172.5 172.4 172.4 172.4 172.4 172.3 172.3 172.3 172.3 172.3 172.3 172.3 172.2 172.2 172.2 172.2 172.2 165.4 165.4 165.4 165.4 165.4 Min: 165.4, Avg: 171.3, Max: 172.7, Diff: 7.3, Sum: 5482.3] [GC Worker End (ms): 346916096.0 346916096.1 346916096.2 346916096.0 346916096.1 346916096.0 346916096.0 346916096.0 346916096.0 346916096.1 346916096.1 346916096.0 346916096.0 346916096.0 346916096.0 346916096.0 346916096.0 346916096.0 346916096.0 346916096.0 346916096.0 346916096.1 346916096.1 346916096.1 346916096.1 346916096.1 346916096.2 346916096.0 346916096.0 346916096.1 346916096.1 346916096.1 Min: 346916096.0, Avg: 346916096.1, Max: 346916096.2, Diff: 0.1] [Code Root Fixup: 0.1 ms] [Code Root Purge: 0.0 ms] [Clear CT: 5.3 ms] [Other: 20.2 ms] [Choose CSet: 0.0 ms] [Ref Proc: 14.7 ms] [Ref Enq: 0.9 ms] [Redirty Cards: 1.3 ms] [Parallel Redirty: 0.9 0.9 0.9 0.9 0.9 0.9 0.8 0.8 0.8 0.8 0.8 0.8 0.8 0.8 0.8 0.8 0.7 0.7 0.7 0.7 0.7 0.7 0.7 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.3 0.0 Min: 0.0, Avg: 0.7, Max: 0.9, Diff: 0.9, Sum: 22.7] [Redirtied Cards: 10545 10284 8960 10583 9724 9928 8987 11050 9047 8219 5376 8556 7135 5652 6488 5632 9483 4864 5828 3584 8346 5034 5116 4864 4450 7615 4096 6002 5270 1792 3577 0 Min: 0, Avg: 6752.7, Max: 11050, Diff: 11050, Sum: 216087] [Humongous Register: 0.2 ms] [Humongous Total: 0] [Humongous Candidate: 0] [Humongous Reclaim: 0.2 ms] [Humongous Reclaimed: 0] [Free CSet: 1.6 ms] [Young Free CSet: 1.6 ms] [Non-Young Free CSet: 0.0 ms] [Eden: 4064.0M(4032.0M)->0.0B(4032.0M) Survivors: 576.0M->576.0M Heap: 47.4G(90.0G)->46.5G(90.0G)] Heap after GC invocations=10620 (full 0): garbage-first heap total 94371840K, used 48709632K [0x00007f017c000000, 0x00007f017e005a00, 0x00007f17fc000000) region size 32768K, 18 young (589824K), 18 survivors (589824K) Metaspace used 78704K, capacity 79547K, committed 79588K, reserved 79872K } [Times: user=5.58 sys=0.03, real=0.20 secs] 2015-10-18T07:10:11.172-0400: 346916.123: Total time for which application threads were stopped: 0.2012083 seconds, Stopping threads took: 0.0000795 seconds From bengt.rutisson at oracle.com Mon Oct 26 12:47:02 2015 From: bengt.rutisson at oracle.com (Bengt Rutisson) Date: Mon, 26 Oct 2015 13:47:02 +0100 Subject: JEP 271: Unified GC Logging Message-ID: <562E20C6.5050703@oracle.com> Hi eveyone, The discussions regarding the new GC logging is handled on hotspot-gc-dev at openjdk.java.net. But since it concerns the usability of the GC I wanted to ping on this list as well. There is a first pre-review out now. See this email: http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2015-October/015070.html Please handle all comments and feedback on hotspot-gc-dev at openjdk.java.net. Thanks, Bengt From jun.zhuang at hobsons.com Mon Oct 26 19:33:36 2015 From: jun.zhuang at hobsons.com (Jun Zhuang) Date: Mon, 26 Oct 2015 19:33:36 +0000 Subject: Seeking answer to a GC pattern Message-ID: Hi, When running performance testing for a java web service running on JBOSS, I observed a clear saw-tooth pattern in CPU utilization that closely follows the GC cycles. see below: [cid:image002.jpg at 01D11003.AD2D2BA0] [cid:image010.jpg at 01D11003.AD2D2BA0] Java startup parameters used: -XX:+TieredCompilation -XX:+PrintTenuringDistribution -Xms2048m -Xmx4096m -XX:MaxPermSize=256m -XX:NewSize=128m -XX:MaxNewSize=128m -XX:SurvivorRatio=126 -XX:-UseAdaptiveSizePolicy -XX:+DisableExplicitGC -XX:+AlwaysTenure With this set of parameters, the young GC pause time ranged from 0.02 to 0.25 secs. When I used 256m for the young gen, the young GC pause time ranged from 0.02 to 0.5 secs. My understanding is that the young GC pause time normally stays fairly stable, I have spent quite some time researching but have yet to find an answer to this behavior. I wonder if people in this distribution list can help me out? Other related info * Server Specs: VM with 4 CPUs and 8 Gb mem * Test setup: * # of Vusers: 100 * Ramp up: 10 mins * Pacing: 5 - 7 secs * I tried with all other available GC algorithms, tenuring thresholds, various sizes of the generations, but the AlwaysTenure parameter seems to work the best so far. [cid:image011.jpg at 01D11003.AD2D2BA0] [cid:image012.jpg at 01D11003.AD2D2BA0] Any input will be highly appreciated. Sincerely yours, Jun Zhuang Sr. Performance QA Engineer | Hobsons 513-746-2288 (work) 513-227-7643 (mobile) -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 7083 bytes Desc: image002.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image010.jpg Type: image/jpeg Size: 11773 bytes Desc: image010.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image011.jpg Type: image/jpeg Size: 33498 bytes Desc: image011.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image012.jpg Type: image/jpeg Size: 19704 bytes Desc: image012.jpg URL: