From amit.mishra at redknee.com Thu Jan 19 13:05:00 2017 From: amit.mishra at redknee.com (Amit Mishra) Date: Thu, 19 Jan 2017 13:05:00 +0000 Subject: Need help on G1 GC young gen Update RS and Scan RS pause reduction Message-ID: Hello Friends, I am using G1 GC on Java 1.7.45 version in my solaris LAB environment but am getting very high UpdateRS time and few occasions high Scan RS time as well. OS version : Generic_150401-30 i86pc i386 i86pc Number of CPU # 80, A day back i have tried to play with below options hoping extra GC threads will help me but all of no avail as I had witnessed long pauses up to 7 seconds. Right now I have disabled these GC threads. argv[52]: -XX:ParallelGCThreads=70 argv[53]: -XX:ConcGCThreads=60 argv[54]: -XX:G1ConcRefinementThreads=70 I have already put -XX:G1RSetUpdatingPauseTimePercent=5 option and relaxed pause time goal by setting it to 500 ms but it doesn't seems to be helping me. Your urgent help will be highly appreciated as I need system where avg. pause time remain around 200-300 milli-second and Max pause remain around 500-600 ms. Current GC parameters are as below: argv[11]: -Xmx48g argv[12]: -Xms48g argv[13]: -XX:-EliminateLocks argv[24]: -Xss1m argv[25]: -Xoss1m argv[26]: -XX:PermSize=512m argv[27]: -XX:MaxPermSize=512m argv[28]: -XX:ReservedCodeCacheSize=128m argv[29]: -XX:+HeapDumpOnOutOfMemoryError argv[30]: -XX:+AggressiveOpts argv[35]: -XX:+UseG1GC argv[36]: -XX:MaxGCPauseMillis=500 argv[37]: -XX:+PrintFlagsFinal argv[38]: -XX:G1RSetUpdatingPauseTimePercent=5 argv[39]: -XX:+PrintGCTimeStamps argv[40]: -XX:+PrintGCDetails argv[42]: -verbose:gc argv[43]: -XX:+UseLargePages argv[44]: -XX:+MaxFDLimit argv[48]: -XX:+UnlockExperimentalVMOptions argv[49]: -XX:G1NewSizePercent=1 argv[50]: -XX:+ParallelRefProcEnabled argv[51]: -XX:+DisableExplicitGC argv[52]: -XX:G1MaxNewSizePercent=1 argv[53]: -XX:InitiatingHeapOccupancyPercent=40 argv[54]: -XX:+UnlockDiagnosticVMOptions argv[55]: -XX:+G1SummarizeRSetStats argv[56]: -XX:G1SummarizeRSetStatsPeriod=1 argv[57]: -XX:+PerfDisableSharedMem argv[58]: -XX:+AlwaysPreTouch Pause snapshots: Very high UpdateRS but low scan RS 2602.655: [GC pause (young), 7.8981373 secs] [Parallel Time: 7883.9 ms, GC Workers: 53] [GC Worker Start (ms): Min: 2602654.9, Avg: 2602655.3, Max: 2602655.7, Diff: 0.8] [Ext Root Scanning (ms): Min: 3.5, Avg: 4.5, Max: 34.2, Diff: 30.7, Sum: 240.0] [Update RS (ms): Min: 7796.5, Avg: 7826.2, Max: 7827.8, Diff: 31.4, Sum: 414789.9] [Processed Buffers: Min: 1, Avg: 3.8, Max: 12, Diff: 11, Sum: 199] [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.6] [Object Copy (ms): Min: 48.2, Avg: 48.8, Max: 49.4, Diff: 1.3, Sum: 2588.2] [Termination (ms): Min: 3.1, Avg: 3.5, Max: 3.9, Diff: 0.8, Sum: 183.8] [GC Worker Other (ms): Min: 0.0, Avg: 0.2, Max: 0.4, Diff: 0.3, Sum: 8.4] [GC Worker Total (ms): Min: 7882.7, Avg: 7883.2, Max: 7883.8, Diff: 1.1, Sum: 417810.9] [GC Worker End (ms): Min: 2610538.4, Avg: 2610538.5, Max: 2610538.7, Diff: 0.3] [Code Root Fixup: 0.0 ms] [Clear CT: 0.9 ms] [Other: 13.3 ms] [Choose CSet: 0.0 ms] [Ref Proc: 10.5 ms] [Ref Enq: 1.0 ms] [Free CSet: 0.5 ms] [Eden: 432.0M(432.0M)->0.0B(416.0M) Survivors: 48.0M->64.0M Heap: 19.3G(48.0G)->18.9G(48.0G)] [Times: user=3.24 sys=0.02, real=7.90 secs] Concurrent RS processed 48786945 cards Of 1661300 completed buffers: 1661300 (100.0%) by conc RS threads. 0 ( 0.0%) by mutator threads. Conc RS threads times(s) 17.55 15.01 12.67 11.26 10.14 9.56 8.13 7.26 6.61 6.18 5.25 4.64 3.97 3.16 2.92 2.76 2.60 2.54 2.36 2.30 2.16 2.12 2.00 1.92 1.83 1.82 1.75 1.69 1.64 1.64 1.55 1.50 1.45 1.48 1.41 1.36 1.32 1.29 1.26 1.22 1.15 1.18 1.12 1.07 1.05 1.03 1.01 0.97 0.96 0.95 0.89 0.89 0.87 0.33 Total heap region rem set sizes = 808374K. Max = 4135K. Static structures = 1596K, free_lists = 0K. 72965736 occupied cards represented. Max size region = 35:(O)[0xfffffd73e3000000,0xfffffd73e4000000,0xfffffd73e4000000], size = 4136K, occupied = 14130K. Did 252 coarsenings. High Update RS and high Scan RS : 301.125: [GC pause (young), 1.5623216 secs] [Parallel Time: 1542.0 ms, GC Workers: 53] [GC Worker Start (ms): Min: 301126.4, Avg: 301126.9, Max: 301127.4, Diff: 0.9] [Ext Root Scanning (ms): Min: 2.1, Avg: 2.8, Max: 3.9, Diff: 1.8, Sum: 146.3] [Update RS (ms): Min: 593.4, Avg: 647.2, Max: 723.0, Diff: 129.6, Sum: 34303.1] [Processed Buffers: Min: 10, Avg: 19.4, Max: 46, Diff: 36, Sum: 1026] [Scan RS (ms): Min: 504.4, Avg: 580.7, Max: 633.5, Diff: 129.0, Sum: 30775.8] [Object Copy (ms): Min: 293.4, Avg: 306.5, Max: 310.0, Diff: 16.6, Sum: 16245.9] [Termination (ms): Min: 0.0, Avg: 3.2, Max: 9.8, Diff: 9.7, Sum: 171.0] [GC Worker Other (ms): Min: 0.0, Avg: 0.3, Max: 0.5, Diff: 0.4, Sum: 13.8] [GC Worker Total (ms): Min: 1540.1, Avg: 1540.7, Max: 1541.3, Diff: 1.2, Sum: 81656.0] [GC Worker End (ms): Min: 302667.3, Avg: 302667.6, Max: 302667.8, Diff: 0.4] [Code Root Fixup: 0.0 ms] [Clear CT: 1.0 ms] [Other: 19.3 ms] [Choose CSet: 0.0 ms] [Ref Proc: 12.4 ms] [Ref Enq: 1.1 ms] [Free CSet: 0.6 ms] [Eden: 416.0M(416.0M)->0.0B(416.0M) Survivors: 64.0M->64.0M Heap: 14.8G(48.0G)->14.7G(48.0G)] [Times: user=12.90 sys=1.13, real=1.56 secs] Concurrent RS processed 9611680 cards Of 113955 completed buffers: 113955 (100.0%) by conc RS threads. 0 ( 0.0%) by mutator threads. Conc RS threads times(s) 5.63 5.17 4.67 4.39 4.03 3.81 3.46 3.21 3.01 2.88 2.67 2.47 2.27 2.08 1.89 1.76 1.64 1.58 1.45 1.40 1.31 1.27 1.17 1.12 1.06 1.04 0.99 0.95 0.93 0.92 0.86 0.83 0.82 0.82 0.77 0.74 0.71 0.69 0.67 0.64 0.60 0.61 0.57 0.55 0.53 0.52 0.51 0.48 0.47 0.48 0.43 0.43 0.42 0.02 Total heap region rem set sizes = 255924K. Max = 3260K. Static structures = 1596K, free_lists = 6K. 42009931 occupied cards represented. Max size region = 31:(O)[0xfffffd73df000000,0xfffffd73e0000000,0xfffffd73e0000000], size = 3261K, occupied = 11784K. Did 0 coarsenings. Thanks, Amit Mishra From: Amit Mishra Sent: Tuesday, January 17, 2017 18:53 To: 'yu.zhang at oracle.com' ; Poonam Bajaj Parhar Subject: RE: Need help on G1 GC young pause reduction Thank you very much Poonam and Zhang for your response. I have enabled below flags to summarize RS update sets as well as keep application un-impacted in case of high disk IO or swapping on Node. argv[58]: -XX:+G1SummarizeRSetStats argv[59]: -XX:G1SummarizeRSetStatsPeriod=10 argv[60]: -XX:+PerfDisableSharedMem argv[61]: -XX:+AlwaysPreTouch I am monitoring the situation and will raise my concern on hotspot mail list if I encounter any further pause. I just want to know apart from below 3 STW pauses if there is any other STW pause in G1 GC. a)Young generation evacuation pause b)Remark phase pause c) Cleanup phase pause Thanks, Amit Mishra From: yu.zhang at oracle.com [mailto:yu.zhang at oracle.com] Sent: Saturday, January 14, 2017 00:57 To: Poonam Bajaj Parhar >; Amit Mishra > Subject: Re: Need help on G1 GC young pause reduction Poonam, Thanks for forwarding this. As you suggested, I also recommend sending to hotspot-gc-use at openjdk.java.net To add to what Poonam said, With -XX:+G1SummarizeRSetStats you can check if there are a lot of coarsening, and adjust the remember set table entries if there a a lot. Another suggestion is to tune the Red/Yellow/Green zone definition for refinement. That requires getting the refinement details first. Thanks Jenny On 01/13/2017 10:13 AM, Poonam Bajaj Parhar wrote: Hello Amit, You could try to get more information with -XX:+G1SummarizeRSetStats and see what is taking more time. I am copying Jenny who would be able to help further. And I suggest you send your question with GC logs to hotspot-gc-use at openjdk.java.net to get GC experts opinion on this. Thanks, Poonam On 1/13/2017 5:02 AM, Amit Mishra wrote: Hello Poonam, I have updated the mentioned parameter but strangely young pause due to RSUpdate is going over 1 second while at most of time it remains around 30-50 ms. Please help me to decrease this RSupdate time. argv[35]: -XX:+UseG1GC argv[38]: -XX:G1RSetUpdatingPauseTimePercent=5 argv[49]: -XX:G1NewSizePercent=1 argv[50]: -XX:+ParallelRefProcEnabled argv[51]: -XX:+DisableExplicitGC argv[52]: -XX:G1MaxNewSizePercent=1 argv[53]: -XX:ParallelGCThreads=70 argv[54]: -XX:ConcGCThreads=60 argv[55]: -XX:G1ConcRefinementThreads=70 argv[56]: -XX:InitiatingHeapOccupancyPercent=30 4082.624: [GC pause (young), 0.0319047 secs] [Parallel Time: 13.9 ms, GC Workers: 70] [GC Worker Start (ms): Min: 4082624.8, Avg: 4082625.3, Max: 4082625.9, Diff: 1.1] [Ext Root Scanning (ms): Min: 1.3, Avg: 2.1, Max: 5.9, Diff: 4.6, Sum: 146.0] [Update RS (ms): Min: 0.0, Avg: 2.8, Max: 3.7, Diff: 3.7, Sum: 194.1] [Processed Buffers: Min: 0, Avg: 3.1, Max: 16, Diff: 16, Sum: 216] [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.9] [Object Copy (ms): Min: 6.0, Avg: 6.9, Max: 7.4, Diff: 1.4, Sum: 481.1] [Termination (ms): Min: 0.3, Avg: 0.6, Max: 0.8, Diff: 0.4, Sum: 38.8] [GC Worker Other (ms): Min: 0.0, Avg: 0.2, Max: 0.5, Diff: 0.5, Sum: 17.4] [GC Worker Total (ms): Min: 12.0, Avg: 12.5, Max: 13.2, Diff: 1.2, Sum: 878.3] [GC Worker End (ms): Min: 4082637.6, Avg: 4082637.9, Max: 4082638.1, Diff: 0.5] [Code Root Fixup: 0.0 ms] [Clear CT: 1.1 ms] [Other: 16.8 ms] [Choose CSet: 0.0 ms] [Ref Proc: 14.0 ms] [Ref Enq: 1.1 ms] [Free CSet: 0.5 ms] [Eden: 464.0M(464.0M)->0.0B(464.0M) Survivors: 16.0M->16.0M Heap: 14.7G(48.0G)->14.2G(48.0G)] [Times: user=1.04 sys=0.02, real=0.03 secs] 4083.465: [GC pause (young), 1.3382388 secs] [Parallel Time: 1319.7 ms, GC Workers: 70] [GC Worker Start (ms): Min: 4083465.7, Avg: 4083466.2, Max: 4083466.7, Diff: 1.0] [Ext Root Scanning (ms): Min: 1.6, Avg: 22.2, Max: 1306.9, Diff: 1305.3, Sum: 1557.0] [Update RS (ms): Min: 3.3, Avg: 1287.7, Max: 1308.7, Diff: 1305.4, Sum: 90139.6] [Processed Buffers: Min: 1, Avg: 2.8, Max: 11, Diff: 10, Sum: 198] [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 1.0] [Object Copy (ms): Min: 6.8, Avg: 8.0, Max: 8.6, Diff: 1.8, Sum: 558.6] [Termination (ms): Min: 0.5, Avg: 0.7, Max: 0.9, Diff: 0.4, Sum: 46.0] [GC Worker Other (ms): Min: 0.0, Avg: 0.2, Max: 0.4, Diff: 0.4, Sum: 11.2] [GC Worker Total (ms): Min: 1318.2, Avg: 1318.8, Max: 1319.5, Diff: 1.3, Sum: 92313.2] [GC Worker End (ms): Min: 4084784.8, Avg: 4084785.0, Max: 4084785.2, Diff: 0.4] [Code Root Fixup: 0.0 ms] [Clear CT: 1.3 ms] [Other: 17.3 ms] [Choose CSet: 0.0 ms] [Ref Proc: 14.3 ms] [Ref Enq: 1.3 ms] [Free CSet: 0.5 ms] [Eden: 464.0M(464.0M)->0.0B(464.0M) Survivors: 16.0M->16.0M Heap: 14.7G(48.0G)->14.2G(48.0G)] [Times: user=1.11 sys=0.02, real=1.34 secs] 4085.195: [GC pause (young), 0.0303773 secs] [Parallel Time: 13.4 ms, GC Workers: 70] [GC Worker Start (ms): Min: 4085194.9, Avg: 4085195.4, Max: 4085195.9, Diff: 1.0] [Ext Root Scanning (ms): Min: 1.6, Avg: 2.3, Max: 5.4, Diff: 3.8, Sum: 161.6] [Update RS (ms): Min: 0.0, Avg: 2.8, Max: 3.8, Diff: 3.8, Sum: 196.4] [Processed Buffers: Min: 0, Avg: 3.4, Max: 15, Diff: 15, Sum: 235] [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 1.0] [Object Copy (ms): Min: 5.4, Avg: 6.3, Max: 6.8, Diff: 1.4, Sum: 442.2] [Termination (ms): Min: 0.3, Avg: 0.4, Max: 0.6, Diff: 0.3, Sum: 28.6] [GC Worker Other (ms): Min: 0.0, Avg: 0.2, Max: 0.5, Diff: 0.4, Sum: 15.1] [GC Worker Total (ms): Min: 11.4, Avg: 12.1, Max: 12.7, Diff: 1.2, Sum: 844.9] [GC Worker End (ms): Min: 4085207.2, Avg: 4085207.4, Max: 4085207.7, Diff: 0.4] [Code Root Fixup: 0.0 ms] [Clear CT: 1.2 ms] [Other: 15.8 ms] [Choose CSet: 0.0 ms] [Ref Proc: 12.9 ms] [Ref Enq: 1.2 ms] [Free CSet: 0.4 ms] Regards, Amit Mishra From: Poonam Bajaj Parhar [mailto:poonam.bajaj at oracle.com] Sent: Wednesday, January 11, 2017 23:59 To: Amit Mishra Subject: Re: Need help on G1 GC young pause reduction Hello Amit, You should decrease the value of G1RSetUpdatingPauseTimePercent (default is 10) and not increase it. By decreasing the value of this parameter, you are instructing G1 to spend less time on RS update during evacuation pauses. The leftover work would be done concurrently by the refinement threads. To help speed up the concurrent RS update, increase the number of concurrent refinement threads with -XX:G1ConcRefinementThreads. You may first want to check with the help of -XX:+G1SummarizeRSetStats if the refinement threads are able to keep up with the update work. If they are not able to complete the RS update work, then increase the number of refinement threads. Hope this helps! Thanks, Poonam On 1/11/2017 5:04 AM, Amit Mishra wrote: Thank you very much Poonam, I have implemented particular option and added ConcGC thread as well but still I am getting RSUpdate time greater than 1 second which we cannot afford on our Production platform. I have already enabled all the parameters that I can think of use after reading your mentioned article but still pause value are beyond our expectations. Is there anyone from your team who can help us to alleviate such large pauses ? Does reducing region sizes or implementing any other flag will going to help us ? 15872.711: [GC pause (young), 1.4327308 secs] [Parallel Time: 1407.3 ms, GC Workers: 70] [GC Worker Start (ms): Min: 15872711.5, Avg: 15872712.1, Max: 15872712.7, Diff: 1.2] [Ext Root Scanning (ms): Min: 1.5, Avg: 42.1, Max: 1398.8, Diff: 1397.3, Sum: 2945.8] [Update RS (ms): Min: 0.0, Avg: 1224.4, Max: 1399.8, Diff: 1399.7, Sum: 85704.5] [Processed Buffers: Min: 0, Avg: 15.5, Max: 93, Diff: 93, Sum: 1083] [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.9] [Object Copy (ms): Min: 0.1, Avg: 4.7, Max: 33.1, Diff: 33.0, Sum: 326.9] [Termination (ms): Min: 0.2, Avg: 131.3, Max: 1316.1, Diff: 1315.9, Sum: 9188.6] [GC Worker Other (ms): Min: 0.0, Avg: 0.2, Max: 0.5, Diff: 0.5, Sum: 15.2] [GC Worker Total (ms): Min: 1401.8, Avg: 1402.6, Max: 1406.7, Diff: 4.8, Sum: 98181.9] [GC Worker End (ms): Min: 15874114.4, Avg: 15874114.7, Max: 15874118.4, Diff: 4.0] [Code Root Fixup: 0.0 ms] [Clear CT: 1.5 ms] [Other: 24.0 ms] [Choose CSet: 0.0 ms] [Ref Proc: 16.9 ms] [Ref Enq: 1.6 ms] [Free CSet: 1.5 ms] [Eden: 928.0M(928.0M)->0.0B(912.0M) Survivors: 48.0M->64.0M Heap: 20.4G(48.0G)->19.5G(48.0G)] [Times: user=9.37 sys=0.13, real=1.43 secs] argv[35]: -XX:+UseG1GC argv[36]: -XX:MaxGCPauseMillis=800 argv[37]: -XX:+UseCompressedOops argv[38]: -XX:+PrintFlagsFinal argv[39]: -XX:G1RSetUpdatingPauseTimePercent=20 argv[40]: -XX:+PrintGCTimeStamps argv[41]: -XX:+PrintGCDetails argv[42]: -Xloggc:/opt/redknee/product/cps/9_2_20/log/gcstats.log.17962 argv[43]: -verbose:gc argv[44]: -XX:+UseLargePages argv[45]: -XX:+MaxFDLimit argv[49]: -XX:+UnlockExperimentalVMOptions argv[50]: -XX:G1NewSizePercent=2 argv[51]: -XX:+ParallelRefProcEnabled argv[52]: -XX:+DisableExplicitGC argv[53]: -XX:G1MaxNewSizePercent=2 argv[54]: -XX:ParallelGCThreads=70 argv[55]: -XX:ConcGCThreads=60 Thanks, Amit Mishra From: Poonam Bajaj Parhar [mailto:poonam.bajaj at oracle.com] Sent: Tuesday, January 10, 2017 03:57 To: Amit Mishra Subject: Re: Need help on G1 GC young pause reduction Hello Amit, You can try limiting the amount of time that the GC threads should spend in updating the RS with the option -XX:G1RSetUpdatingPauseTimePercent. Its default value is 10. Please see this article here: https://www.infoq.com/articles/tuning-tips-G1-GC This talks in detail about how to get more diagnostic information about the RS update times and how it can be controlled. Thanks, Poonam On 1/9/2017 4:27 AM, Amit Mishra wrote: Hello Poonam, I had changed the MaxNewGen to 2% and parallel GC threads to 70 but even though I am getting random STW pauses greater than 1 second. This time it happened during updating Remembered set, can you help me to decrease this pause by enabling any additional flag. 37171.325: [GC pause (young), 3.3011446 secs] [Parallel Time: 3280.9 ms, GC Workers: 70] [GC Worker Start (ms): Min: 37171325.3, Avg: 37171325.9, Max: 37171326.4, Diff: 1.0] [Ext Root Scanning (ms): Min: 1.9, Avg: 2.6, Max: 6.5, Diff: 4.5, Sum: 179.9] [Update RS (ms): Min: 6.0, Avg: 56.3, Max: 3267.8, Diff: 3261.8, Sum: 3939.2] [Processed Buffers: Min: 4, Avg: 6.5, Max: 18, Diff: 14, Sum: 455] [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 1.2] [Object Copy (ms): Min: 0.2, Avg: 103.5, Max: 105.6, Diff: 105.4, Sum: 7245.4] [Termination (ms): Min: 0.0, Avg: 3108.7, Max: 3162.7, Diff: 3162.7, Sum: 217611.5] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.4, Diff: 0.4, Sum: 9.9] [GC Worker Total (ms): Min: 3270.0, Avg: 3271.2, Max: 3279.6, Diff: 9.7, Sum: 228987.2] [GC Worker End (ms): Min: 37174596.3, Avg: 37174597.1, Max: 37174605.9, Diff: 9.6] [Code Root Fixup: 0.0 ms] [Clear CT: 1.2 ms] [Other: 19.0 ms] [Choose CSet: 0.0 ms] [Ref Proc: 13.8 ms] [Ref Enq: 1.2 ms] [Free CSet: 1.6 ms] [Eden: 848.0M(848.0M)->0.0B(848.0M) Survivors: 128.0M->128.0M Heap: 22.0G(48.0G)->21.2G(48.0G)] [Times: user=185.76 sys=1.63, real=3.30 secs] Thanks, Amit Mishra From: Amit Mishra Sent: Thursday, January 5, 2017 13:07 To: 'Poonam Bajaj Parhar' Subject: RE: Need help on G1 GC young pause reduction Thank you very much Poonam, I had re -run the test by keeping new gen size as 2% which would be roughly 1 GB. One more thing I want to ask can we maximize number of parallel GC threads as default is 5/6 of number of cores so here we have 80 CPU on LAB so can we use 70 Parallel GC threads as there is no risk of CMS fragmentation in G1 GC. Thanks, Amit Mishra From: Poonam Bajaj Parhar [mailto:poonam.bajaj at oracle.com] Sent: Wednesday, January 4, 2017 19:21 To: Amit Mishra > Subject: Re: Need help on G1 GC young pause reduction Hello Amit, For some of the GC events, the sys time is higher than than the usr time: Timestamp User Time (secs) Sys Time (secs) Real Time (secs) 00:01:33 3.61 4.55 0.18 00:01:34 3.82 4.16 0.17 00:01:56 4.32 6.64 0.23 00:02:01 4.39 6.25 0.23 ..... Please check if there is something else going on at the system level when these GC events occur. Have you tried setting -XX:G1MaxNewSizePercent to set the max percentage size for the young gen? G1NewSizePercent would set the minimum size for the young gen, not the max limit on it. Making the young gen smaller might help in reducing the evacuation pause times. Let me know if this helps. Thanks, Poonam On 1/4/2017 4:24 AM, Amit Mishra wrote: Hello Poonam, We are testing one of our application for 48G heap size with G1 GC as there were issue with fragmentation with CMS. But we are seeing young pause greater than 1 second every day which we cannot afford on production system, recent pause is as high as 5.7 seconds. There are 64 virtual cores on platform, attaching GC file for your reference. Initially I tested with only default G1 GC parameter and then after with -XX:G1NewSizePercent=1 but in any case there are young gen pauses greater than 1 second which is not acceptable. Need your expert help on this topic to reduce young gen pause. 13531.695: [GC pause (young), 5.7629644 secs] [Parallel Time: 5714.4 ms, GC Workers: 53] [GC Worker Start (ms): Min: 13531695.5, Avg: 13531696.0, Max: 13531696.4, Diff: 0.9] [Ext Root Scanning (ms): Min: 2.2, Avg: 501.4, Max: 5286.9, Diff: 5284.7, Sum: 26572.0] [Update RS (ms): Min: 3.4, Avg: 4789.8, Max: 5289.7, Diff: 5286.3, Sum: 253861.2] [Processed Buffers: Min: 2, Avg: 10.4, Max: 32, Diff: 30, Sum: 552] [Scan RS (ms): Min: 5.0, Avg: 6.1, Max: 6.9, Diff: 1.9, Sum: 324.6] [Object Copy (ms): Min: 380.8, Avg: 382.7, Max: 384.4, Diff: 3.6, Sum: 20284.2] [Termination (ms): Min: 31.2, Avg: 33.0, Max: 34.9, Diff: 3.7, Sum: 1746.7] [GC Worker Other (ms): Min: 0.0, Avg: 0.3, Max: 0.7, Diff: 0.6, Sum: 17.7] [GC Worker Total (ms): Min: 5712.7, Avg: 5713.3, Max: 5714.1, Diff: 1.4, Sum: 302806.3] [GC Worker End (ms): Min: 13537409.0, Avg: 13537409.3, Max: 13537409.6, Diff: 0.6] [Code Root Fixup: 0.0 ms] [Clear CT: 7.0 ms] [Other: 41.6 ms] [Choose CSet: 0.0 ms] [Ref Proc: 25.7 ms] [Ref Enq: 1.0 ms] [Free CSet: 11.9 ms] [Eden: 20.9G(20.9G)->0.0B(128.0M) Survivors: 432.0M->352.0M Heap: 42.7G(48.0G)->21.8G(48.0G)] [Times: user=24.40 sys=0.02, real=5.76 secs] GC parameters are as below: argv[11]: -Xmx48g argv[12]: -Xms48g argv[13]: -XX:-EliminateLocks argv[18]: -server argv[24]: -Xss1m argv[25]: -Xoss1m argv[26]: -XX:PermSize=512m argv[27]: -XX:MaxPermSize=512m argv[28]: -XX:ReservedCodeCacheSize=128m argv[29]: -XX:+HeapDumpOnOutOfMemoryError argv[30]: -XX:+AggressiveOpts argv[31]: -Dnetworkaddress.cache.ttl=3600 argv[32]: -Dcom.sun.management.jmxremote.port=11883 argv[33]: -Dcom.sun.management.jmxremote.ssl=false argv[34]: -Dcom.sun.management.jmxremote.authenticate=false argv[35]: -XX:+UseG1GC argv[36]: -XX:MaxGCPauseMillis=200 argv[37]: -XX:+UseCompressedOops argv[38]: -XX:+PrintFlagsFinal argv[39]: -XX:+PrintGCTimeStamps argv[40]: -XX:+PrintGCDetails argv[41]: -Xloggc:/opt/redknee/product/cps/9_2_20/log/gcstats.log.27925 argv[42]: -verbose:gc argv[43]: -XX:+UseLargePages argv[44]: -XX:+MaxFDLimit argv[48]: -XX:+UnlockExperimentalVMOptions argv[49]: -XX:G1NewSizePercent=1 argv[50]: -XX:+ParallelRefProcEnabled argv[51]: -XX:+DisableExplicitGC Thanks, Amit Mishra -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.schatzl at oracle.com Thu Jan 19 19:03:24 2017 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Thu, 19 Jan 2017 20:03:24 +0100 Subject: Need help on G1 GC young gen Update RS and Scan RS pause reduction In-Reply-To: References: Message-ID: <1484852604.6579.27.camel@oracle.com> Hi Amit, On Thu, 2017-01-19 at 13:05 +0000, Amit Mishra wrote: > Hello Friends, > ? > I am using G1 GC on Java 1.7.45 version in my solaris LAB environment > but am getting very high UpdateRS time and few occasions high Scan RS > time as well. ? just as reminder, consider at least the latest version of 1.7. There were a few interesting changes to G1 later than 1.7.45. JDK8 would even be better of course :) > OS version : Generic_150401-30 i86pc i386 i86pc > Number of CPU # 80, A day back i have tried to play with below > options hoping extra GC threads will help me but all of no avail as I > had witnessed long pauses up to 7 seconds. Right now I have disabled > these GC threads. > ? > argv[52]: -XX:ParallelGCThreads=70 > argv[53]: -XX:ConcGCThreads=60 > argv[54]: -XX:G1ConcRefinementThreads=70 Unless I misunderstand what a virtual cpu in Solaris is, I do not think using more than there are (64 from an earlier email) gives you any advantage. > I have already put -XX:G1RSetUpdatingPauseTimePercent=5 option and > relaxed pause time goal by setting it to 500 ms but it doesn?t seems > to be helping me. There is one (compiler) optimization that may generate lots of work for the Update RS phase called?ReduceInitialCardMarks. Basically it tells g1 to revisit the entire array concurrently. If this allocation happens just before a GC, the gc needs to do all the work in the pause. You can try the option that disables this optimization, i.e. set -XX:- ReduceInitialCardMarks. Of course it is kind of a shot in the dark, and only if you know that your application allocates (a lot) of largish objects. However, this helped in other cases of spurious long Update RS pauses. Also note that I am not sure whether disabling this option causes crashes in 7u - there were some bugfixes in 9 for that. Another option (that should help at least a bit) to reduce update rs/scan rs times may be increasing heap region size to 32M (- XX:G1HeapRegionSize=32M). > Your urgent help will be highly appreciated as I need system where > ?avg. pause time remain around 200-300 milli-second and Max pause > remain around 500-600 ms. > ? > Current GC parameters are as below: > argv[11]: -Xmx48g > argv[12]: -Xms48g > argv[13]: -XX:-EliminateLocks > argv[24]: -Xss1m > argv[25]: -Xoss1m > argv[26]: -XX:PermSize=512m > argv[27]: -XX:MaxPermSize=512m > argv[28]: -XX:ReservedCodeCacheSize=128m > argv[29]: -XX:+HeapDumpOnOutOfMemoryError > argv[30]: -XX:+AggressiveOpts > argv[35]: -XX:+UseG1GC > argv[36]: -XX:MaxGCPauseMillis=500 > argv[37]: -XX:+PrintFlagsFinal > argv[38]: -XX:G1RSetUpdatingPauseTimePercent=5 > argv[39]: -XX:+PrintGCTimeStamps > argv[40]: -XX:+PrintGCDetails > argv[42]: -verbose:gc > argv[43]: -XX:+UseLargePages > argv[44]: -XX:+MaxFDLimit > argv[48]: -XX:+UnlockExperimentalVMOptions > argv[49]: -XX:G1NewSizePercent=1 > argv[50]: -XX:+ParallelRefProcEnabled > argv[51]: -XX:+DisableExplicitGC > argv[52]: -XX:G1MaxNewSizePercent=1 I am not sure if setting the maximum new size is a good idea (causing long Object Copy *mostly*). It might be useful to just remove this one. > argv[53]: -XX:InitiatingHeapOccupancyPercent=40 > argv[54]: -XX:+UnlockDiagnosticVMOptions > argv[55]: -XX:+G1SummarizeRSetStats > argv[56]: -XX:G1SummarizeRSetStatsPeriod=1 > argv[57]: -XX:+PerfDisableSharedMem > argv[58]: -XX:+AlwaysPreTouch > ??????????????? > ? > ? > Pause snapshots: > ? > Very high UpdateRS but low scan RS > ? > 2602.655: [GC pause (young), 7.8981373 secs] > ?? [Parallel Time: 7883.9 ms, GC Workers: 53] > ????? [GC Worker Start (ms): Min: 2602654.9, Avg: 2602655.3, Max: > 2602655.7, Diff: 0.8] > ????? [Ext Root Scanning (ms): Min: 3.5, Avg: 4.5, Max: 34.2, Diff: > 30.7, Sum: 240.0] > ????? [Update RS (ms): Min: 7796.5, Avg: 7826.2, Max: 7827.8, Diff: > 31.4, Sum: 414789.9] > ???????? [Processed Buffers: Min: 1, Avg: 3.8, Max: 12, Diff: 11, > Sum: 199] > ????? [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: > 0.6] > ????? [Object Copy (ms): Min: 48.2, Avg: 48.8, Max: 49.4, Diff: 1.3, > Sum: 2588.2] > ????? [Termination (ms): Min: 3.1, Avg: 3.5, Max: 3.9, Diff: 0.8, > Sum: 183.8] > ????? [GC Worker Other (ms): Min: 0.0, Avg: 0.2, Max: 0.4, Diff: 0.3, > Sum: 8.4] > ????? [GC Worker Total (ms): Min: 7882.7, Avg: 7883.2, Max: 7883.8, > Diff: 1.1, Sum: 417810.9] > ????? [GC Worker End (ms): Min: 2610538.4, Avg: 2610538.5, Max: > 2610538.7, Diff: 0.3] > ?? [Code Root Fixup: 0.0 ms] > ?? [Clear CT: 0.9 ms] > ?? [Other: 13.3 ms] > ????? [Choose CSet: 0.0 ms] > ????? [Ref Proc: 10.5 ms] > ????? [Ref Enq: 1.0 ms] > ????? [Free CSet: 0.5 ms] > ?? [Eden: 432.0M(432.0M)->0.0B(416.0M) Survivors: 48.0M->64.0M Heap: > 19.3G(48.0G)->18.9G(48.0G)] > [Times: user=3.24 sys=0.02, real=7.90 secs] Something is wrong here, i.e. a a wall time larger than user time is strange to say at least. Maybe that machine is overloaded with other applications? I recommend trying to get to the bottom of this first. > Concurrent RS processed 48786945 cards > ? Of 1661300 completed buffers: > ????? 1661300 (100.0%) by conc RS threads. > ??????????? 0 (? 0.0%) by mutator threads. > ? Conc RS threads times(s) > ???????? 17.55??? 15.01??? 12.67??? 11.26??? 10.14???? 9.56???? > 8.13???? 7.26???? 6.61???? 6.18???? 5.25???? 4.64???? 3.97???? > 3.16???? 2.92???? 2.76???? 2.60???? 2.54???? 2.36???? 2.30???? 2.16 > ? 2.12???? 2.00???? 1.92???? 1.83???? 1.82???? 1.75???? 1.69???? > 1.64???? 1.64???? 1.55???? 1.50???? 1.45???? 1.48???? 1.41???? > 1.36???? 1.32???? 1.29???? 1.26???? 1.22???? 1.15??? ?1.18???? 1.12 > ?? 1.07???? 1.05???? 1.03???? 1.01???? 0.97???? 0.96???? 0.95???? > 0.89???? 0.89???? 0.87???? 0.33 > ? Total heap region rem set sizes = 808374K.? Max = 4135K. > ? Static structures = 1596K, free_lists = 0K. > ??? 72965736 occupied cards represented. > ??? Max size region = > 35:(O)[0xfffffd73e3000000,0xfffffd73e4000000,0xfffffd73e4000000], > size = 4136K, occupied = 14130K. > ??? Did 252 coarsenings. This may or may not be what causes long scan rs times; you could try -XX:G1RSetRegionEntries=2048 to avoid coarsenings (and you should remove the g1SummarizeRSetStats* options again). (the 2048 is just a value higher than the number of regions in the heap, at least with 32m regions). Hth, ? Thomas From Milan.Mimica at infobip.com Fri Jan 20 20:24:34 2017 From: Milan.Mimica at infobip.com (Milan Mimica) Date: Fri, 20 Jan 2017 20:24:34 +0000 Subject: G1 native memory consumption Message-ID: <1484943874550.90103@infobip.com> Hi I'm inspecting memory consumption issues of a service running on java-8u102, linux. The service is running for a few days now, and in a few days more it would consume all of 32GB physical memory available, and get killed by OOM Killer. Questions: - If the code is not allocating any significant off-heap memory, neither by Unsafe.allocateMemory or by external library, isn't 7GB native memory overhead supposed to be enough for a 25GB heap? - Why so much memory spent on "Internal" category, apparently from G1 thread? Find the attached jemalloc heap profile, showing "live" allocations that happened in about 30 hour timespan, and a NMT profile of approximately same period. Profiling was done after some warm-up time time, and with a manually triggered Full GC in between just to give the JVM a chance to clean up everything. Relevant JVM options: -Xms25G -Xmx25G -Xss256k -XX:+UseG1GC -XX:MaxGCPauseMillis=250 -XX:InitiatingHeapOccupancyPercent=80 $ /opt/java/sun-jdk1.8.0-102_no_setcap/bin/jcmd 60028 VM.native_memory summary.diff scale=MB 60028: Native Memory Tracking: Total: reserved=30376MB +599MB, committed=29197MB +601MB - Java Heap (reserved=25600MB, committed=25600MB) (mmap: reserved=25600MB, committed=25600MB) - Class (reserved=1168MB, committed=161MB +1MB) (classes #24813 -590) (malloc=4MB #68786 +965) (mmap: reserved=1164MB, committed=157MB +1MB) - Thread (reserved=246MB -6MB, committed=246MB -6MB) (thread #824 -24) (stack: reserved=241MB -6MB, committed=241MB -6MB) (malloc=3MB #4130 -120) (arena=2MB #1646 -48) - Code (reserved=267MB, committed=145MB +1MB) (malloc=24MB #33869 +368) (mmap: reserved=244MB, committed=122MB +1MB) - GC (reserved=1151MB +26MB, committed=1151MB +26MB) (malloc=169MB +26MB #936843 +282881) (mmap: reserved=982MB, committed=982MB) - Compiler (reserved=2MB, committed=2MB) (malloc=2MB #2930 +31) - Internal (reserved=1817MB +571MB, committed=1817MB +571MB) (malloc=1817MB +571MB #906346 +289268) - Symbol (reserved=28MB, committed=28MB) (malloc=24MB #255887 -135) (arena=4MB #1) - Native Memory Tracking (reserved=35MB +9MB, committed=35MB +9MB) (tracking overhead=35MB +9MB) - Tracing (reserved=12MB, committed=12MB) (malloc=12MB #57235 -831) - Unknown (reserved=50MB, committed=0MB) (mmap: reserved=50MB, committed=0MB) Milan Mimica, Software Engineer / Team Leader [Visit Infobip website] [http://www.infobip.com/images/signatures/mwc17.jpg] It's time for Barcelona! Meet us in hall 8.1, stand F49, 27 February-02 March! Office: Mletacka 12/III, 52100 Pula, Croatia | Fax: +38552210979 | Mobile: +385993061692 Email: Milan.Mimica at infobip.com | Skype: mmimicaib www.infobip.com / GSMA Associate Member / Mobey Forum Member This message is private and confidential. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Infobip d.o.o. If you have received this message in error, please notify us immediately via email to customer.support at infobip.com or telephone +442032864235. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: iz_diff_all.svg Type: image/svg+xml Size: 55534 bytes Desc: iz_diff_all.svg URL: From thomas.schatzl at oracle.com Mon Jan 23 10:41:19 2017 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Mon, 23 Jan 2017 11:41:19 +0100 Subject: G1 native memory consumption In-Reply-To: <1484943874550.90103@infobip.com> References: <1484943874550.90103@infobip.com> Message-ID: <1485168079.2811.21.camel@oracle.com> Hi Milan, On Fri, 2017-01-20 at 20:24 +0000, Milan Mimica wrote: > Hi > > I'm inspecting memory consumption issues of a service running on > java-8u102, linux. The service is running for a few days now, and in > a few days more it would consume all of 32GB physical memory > available, and get killed by OOM Killer. > Questions: > - If the code is not allocating any significant off-heap memory, > neither by Unsafe.allocateMemory or by external library, isn't 7GB > native memory overhead supposed to be enough for a 25GB heap? > - Why so much memory spent on "Internal" category, apparently from G1 > thread? G1 remembered sets typically consume approximately 10% of java heap, depending on application, heap and your remembered set configuration. This remembered set contains information that is necessary to be able to do incremental and partial old generation compaction. So the 7GB should be sufficient. You can decrease remembered set overhead by e.g. increasing heap region size. Given the heap size you mentioned it seems it would be worth a try to go to 16M regions. > Find the attached jemalloc heap profile, showing "live" > allocations?that happened?in about 30 hour timespan, and?a > NMT?profile?of approximately same?period. > Profiling was done after?some?warm-up?time time, and with a > manually?triggered?Full GC in?between just to give the JVM a chance > to clean up everything.? Remembered set memory consumption should level out after some time, where "some" may be quite a bit of time. I.e. it may take much longer than for other collectors. It also depends on the memory allocator. Do you have any new measurements after this weekend? It should have levelled out by this time. JDK9 contains some improvements on memory usage in exactly this area. There will likely be further improvements in this area going forward. Thanks, ? Thomas From thomas.schatzl at oracle.com Tue Jan 24 08:02:46 2017 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Tue, 24 Jan 2017 09:02:46 +0100 Subject: Need help on G1 GC young gen Update RS and Scan RS pause reduction In-Reply-To: References: <1484852604.6579.27.camel@oracle.com> Message-ID: <1485244966.2883.8.camel@oracle.com> Hi Amit, On Fri, 2017-01-20 at 10:51 +0000, Amit Mishra wrote: > Thank you very much Thomas, after making said changes RSUpdate time > is in control but Scan RS is still high. > ? ? can you summarize your GC parameters again, and also, are these high Scan RS times you observe regular or not? Did you check with?-XX:+G1SummarizeRSetStats?- XX:G1SummarizeRSetStatsPeriod= whether there are still coarsenings? Could you also please provide a longer snippet of your log file containing a few gcs before and after this occurrence? Other worthwhile tries could be to fudge with -XX:ParallelGCThreads, either decreasing or increasing them (up to your vcpus) by ten or so to see whether that phase suffers from synchronization issues. The high scan rs times could maybe also a bit mitigated by increasing G1NewSizePercent to maybe two or three to see whether g1 can avoid large young remembered sets by that (Object copy might go up a bit then though). It may be easier to think of causes for the issues you are experiencing by having a little bit of more information. Thanks, ? Thomas From amit.mishra at redknee.com Fri Jan 20 10:51:28 2017 From: amit.mishra at redknee.com (Amit Mishra) Date: Fri, 20 Jan 2017 10:51:28 +0000 Subject: Need help on G1 GC young gen Update RS and Scan RS pause reduction In-Reply-To: <1484852604.6579.27.camel@oracle.com> References: <1484852604.6579.27.camel@oracle.com> Message-ID: Thank you very much Thomas, after making said changes RSUpdate time is in control but Scan RS is still high. I also observed that sys time is very high but my system has sufficient memory and there has not been a single swapping/paging happened on node, Not sure how to troubleshoot this high sys time. 244.719: [GC pause (young), 1.3896037 secs] [Parallel Time: 1356.6 ms, GC Workers: 53] [GC Worker Start (ms): Min: 244718.9, Avg: 244719.3, Max: 244719.8, Diff: 0.9] [Ext Root Scanning (ms): Min: 2.2, Avg: 3.0, Max: 3.8, Diff: 1.6, Sum: 156.7] [Update RS (ms): Min: 13.3, Avg: 16.3, Max: 47.0, Diff: 33.6, Sum: 866.5] [Processed Buffers: Min: 3, Avg: 6.8, Max: 17, Diff: 14, Sum: 361] [Scan RS (ms): Min: 972.9, Avg: 1002.8, Max: 1006.0, Diff: 33.1, Sum: 53149.3] [Object Copy (ms): Min: 332.1, Avg: 333.1, Max: 334.5, Diff: 2.4, Sum: 17656.0] [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 1.0] [GC Worker Other (ms): Min: 0.0, Avg: 0.2, Max: 0.5, Diff: 0.4, Sum: 11.5] [GC Worker Total (ms): Min: 1354.9, Avg: 1355.5, Max: 1356.1, Diff: 1.2, Sum: 71841.1] [GC Worker End (ms): Min: 246074.7, Avg: 246074.8, Max: 246075.1, Diff: 0.4] [Code Root Fixup: 0.0 ms] [Clear CT: 2.1 ms] [Other: 30.8 ms] [Choose CSet: 0.0 ms] [Ref Proc: 10.3 ms] [Ref Enq: 1.6 ms] [Free CSet: 1.6 ms] [Eden: 2848.0M(1856.0M)->0.0B(224.0M) Survivors: 192.0M->256.0M Heap: 10.3G(48.0G)->9324.6M(48.0G)] [Times: user=16.93 sys=1.07, real=1.39 secs] Thanks, Amit Mishra -----Original Message----- From: Thomas Schatzl [mailto:thomas.schatzl at oracle.com] Sent: Friday, January 20, 2017 00:33 To: Amit Mishra ; hotspot-gc-use at openjdk.java.net Subject: Re: Need help on G1 GC young gen Update RS and Scan RS pause reduction Hi Amit, On Thu, 2017-01-19 at 13:05 +0000, Amit Mishra wrote: > Hello Friends, > > I am using G1 GC on Java 1.7.45 version in my solaris LAB environment > but am getting very high UpdateRS time and few occasions high Scan RS > time as well. just as reminder, consider at least the latest version of 1.7. There were a few interesting changes to G1 later than 1.7.45. JDK8 would even be better of course :) > OS version : Generic_150401-30 i86pc i386 i86pc Number of CPU # 80, A > day back i have tried to play with below options hoping extra GC > threads will help me but all of no avail as I had witnessed long > pauses up to 7 seconds. Right now I have disabled these GC threads. > > argv[52]: -XX:ParallelGCThreads=70 > argv[53]: -XX:ConcGCThreads=60 > argv[54]: -XX:G1ConcRefinementThreads=70 Unless I misunderstand what a virtual cpu in Solaris is, I do not think using more than there are (64 from an earlier email) gives you any advantage. > I have already put -XX:G1RSetUpdatingPauseTimePercent=5 option and > relaxed pause time goal by setting it to 500 ms but it doesn?t seems > to be helping me. There is one (compiler) optimization that may generate lots of work for the Update RS phase called ReduceInitialCardMarks. Basically it tells g1 to revisit the entire array concurrently. If this allocation happens just before a GC, the gc needs to do all the work in the pause. You can try the option that disables this optimization, i.e. set -XX:- ReduceInitialCardMarks. Of course it is kind of a shot in the dark, and only if you know that your application allocates (a lot) of largish objects. However, this helped in other cases of spurious long Update RS pauses. Also note that I am not sure whether disabling this option causes crashes in 7u - there were some bugfixes in 9 for that. Another option (that should help at least a bit) to reduce update rs/scan rs times may be increasing heap region size to 32M (- XX:G1HeapRegionSize=32M). > Your urgent help will be highly appreciated as I need system where > avg. pause time remain around 200-300 milli-second and Max pause > remain around 500-600 ms. > > Current GC parameters are as below: > argv[11]: -Xmx48g > argv[12]: -Xms48g > argv[13]: -XX:-EliminateLocks > argv[24]: -Xss1m > argv[25]: -Xoss1m > argv[26]: -XX:PermSize=512m > argv[27]: -XX:MaxPermSize=512m > argv[28]: -XX:ReservedCodeCacheSize=128m > argv[29]: -XX:+HeapDumpOnOutOfMemoryError > argv[30]: -XX:+AggressiveOpts > argv[35]: -XX:+UseG1GC > argv[36]: -XX:MaxGCPauseMillis=500 > argv[37]: -XX:+PrintFlagsFinal > argv[38]: -XX:G1RSetUpdatingPauseTimePercent=5 > argv[39]: -XX:+PrintGCTimeStamps > argv[40]: -XX:+PrintGCDetails > argv[42]: -verbose:gc > argv[43]: -XX:+UseLargePages > argv[44]: -XX:+MaxFDLimit > argv[48]: -XX:+UnlockExperimentalVMOptions > argv[49]: -XX:G1NewSizePercent=1 > argv[50]: -XX:+ParallelRefProcEnabled > argv[51]: -XX:+DisableExplicitGC > argv[52]: -XX:G1MaxNewSizePercent=1 I am not sure if setting the maximum new size is a good idea (causing long Object Copy *mostly*). It might be useful to just remove this one. > argv[53]: -XX:InitiatingHeapOccupancyPercent=40 > argv[54]: -XX:+UnlockDiagnosticVMOptions > argv[55]: -XX:+G1SummarizeRSetStats > argv[56]: -XX:G1SummarizeRSetStatsPeriod=1 > argv[57]: -XX:+PerfDisableSharedMem > argv[58]: -XX:+AlwaysPreTouch > > > > Pause snapshots: > > Very high UpdateRS but low scan RS > > 2602.655: [GC pause (young), 7.8981373 secs] > [Parallel Time: 7883.9 ms, GC Workers: 53] > [GC Worker Start (ms): Min: 2602654.9, Avg: 2602655.3, Max: > 2602655.7, Diff: 0.8] > [Ext Root Scanning (ms): Min: 3.5, Avg: 4.5, Max: 34.2, Diff: > 30.7, Sum: 240.0] > [Update RS (ms): Min: 7796.5, Avg: 7826.2, Max: 7827.8, Diff: > 31.4, Sum: 414789.9] > [Processed Buffers: Min: 1, Avg: 3.8, Max: 12, Diff: 11, > Sum: 199] > [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: > 0.6] > [Object Copy (ms): Min: 48.2, Avg: 48.8, Max: 49.4, Diff: 1.3, > Sum: 2588.2] > [Termination (ms): Min: 3.1, Avg: 3.5, Max: 3.9, Diff: 0.8, > Sum: 183.8] > [GC Worker Other (ms): Min: 0.0, Avg: 0.2, Max: 0.4, Diff: 0.3, > Sum: 8.4] > [GC Worker Total (ms): Min: 7882.7, Avg: 7883.2, Max: 7883.8, > Diff: 1.1, Sum: 417810.9] > [GC Worker End (ms): Min: 2610538.4, Avg: 2610538.5, Max: > 2610538.7, Diff: 0.3] > [Code Root Fixup: 0.0 ms] > [Clear CT: 0.9 ms] > [Other: 13.3 ms] > [Choose CSet: 0.0 ms] > [Ref Proc: 10.5 ms] > [Ref Enq: 1.0 ms] > [Free CSet: 0.5 ms] > [Eden: 432.0M(432.0M)->0.0B(416.0M) Survivors: 48.0M->64.0M Heap: > 19.3G(48.0G)->18.9G(48.0G)] > [Times: user=3.24 sys=0.02, real=7.90 secs] Something is wrong here, i.e. a a wall time larger than user time is strange to say at least. Maybe that machine is overloaded with other applications? I recommend trying to get to the bottom of this first. > Concurrent RS processed 48786945 cards > Of 1661300 completed buffers: > 1661300 (100.0%) by conc RS threads. > 0 ( 0.0%) by mutator threads. > Conc RS threads times(s) > 17.55 15.01 12.67 11.26 10.14 9.56 > 8.13 7.26 6.61 6.18 5.25 4.64 3.97 > 3.16 2.92 2.76 2.60 2.54 2.36 2.30 2.16 > 2.12 2.00 1.92 1.83 1.82 1.75 1.69 > 1.64 1.64 1.55 1.50 1.45 1.48 1.41 > 1.36 1.32 1.29 1.26 1.22 1.15 1.18 1.12 > 1.07 1.05 1.03 1.01 0.97 0.96 0.95 > 0.89 0.89 0.87 0.33 > Total heap region rem set sizes = 808374K. Max = 4135K. > Static structures = 1596K, free_lists = 0K. > 72965736 occupied cards represented. > Max size region = > 35:(O)[0xfffffd73e3000000,0xfffffd73e4000000,0xfffffd73e4000000], > size = 4136K, occupied = 14130K. > Did 252 coarsenings. This may or may not be what causes long scan rs times; you could try -XX:G1RSetRegionEntries=2048 to avoid coarsenings (and you should remove the g1SummarizeRSetStats* options again). (the 2048 is just a value higher than the number of regions in the heap, at least with 32m regions). Hth, Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From amit.mishra at redknee.com Tue Jan 24 03:32:12 2017 From: amit.mishra at redknee.com (Amit Mishra) Date: Tue, 24 Jan 2017 03:32:12 +0000 Subject: Need help on G1 GC young gen Update RS and Scan RS pause reduction References: <1484852604.6579.27.camel@oracle.com> Message-ID: Hello Thomas/team, Could you please help me on this as per observation mentioned in trailing mail. Regards, Amit Mishra _____________________________________________ From: Amit Mishra Sent: Friday, January 20, 2017 16:21 To: 'Thomas Schatzl' ; hotspot-gc-use at openjdk.java.net Subject: RE: Need help on G1 GC young gen Update RS and Scan RS pause reduction Thank you very much Thomas, after making said changes RSUpdate time is in control but Scan RS is still high. I also observed that sys time is very high but my system has sufficient memory and there has not been a single swapping/paging happened on node, Not sure how to troubleshoot this high sys time. 244.719: [GC pause (young), 1.3896037 secs] [Parallel Time: 1356.6 ms, GC Workers: 53] [GC Worker Start (ms): Min: 244718.9, Avg: 244719.3, Max: 244719.8, Diff: 0.9] [Ext Root Scanning (ms): Min: 2.2, Avg: 3.0, Max: 3.8, Diff: 1.6, Sum: 156.7] [Update RS (ms): Min: 13.3, Avg: 16.3, Max: 47.0, Diff: 33.6, Sum: 866.5] [Processed Buffers: Min: 3, Avg: 6.8, Max: 17, Diff: 14, Sum: 361] [Scan RS (ms): Min: 972.9, Avg: 1002.8, Max: 1006.0, Diff: 33.1, Sum: 53149.3] [Object Copy (ms): Min: 332.1, Avg: 333.1, Max: 334.5, Diff: 2.4, Sum: 17656.0] [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 1.0] [GC Worker Other (ms): Min: 0.0, Avg: 0.2, Max: 0.5, Diff: 0.4, Sum: 11.5] [GC Worker Total (ms): Min: 1354.9, Avg: 1355.5, Max: 1356.1, Diff: 1.2, Sum: 71841.1] [GC Worker End (ms): Min: 246074.7, Avg: 246074.8, Max: 246075.1, Diff: 0.4] [Code Root Fixup: 0.0 ms] [Clear CT: 2.1 ms] [Other: 30.8 ms] [Choose CSet: 0.0 ms] [Ref Proc: 10.3 ms] [Ref Enq: 1.6 ms] [Free CSet: 1.6 ms] [Eden: 2848.0M(1856.0M)->0.0B(224.0M) Survivors: 192.0M->256.0M Heap: 10.3G(48.0G)->9324.6M(48.0G)] [Times: user=16.93 sys=1.07, real=1.39 secs] Thanks, Amit Mishra -----Original Message----- From: Thomas Schatzl [mailto:thomas.schatzl at oracle.com] Sent: Friday, January 20, 2017 00:33 To: Amit Mishra >; hotspot-gc-use at openjdk.java.net Subject: Re: Need help on G1 GC young gen Update RS and Scan RS pause reduction Hi Amit, On Thu, 2017-01-19 at 13:05 +0000, Amit Mishra wrote: > Hello Friends, > > I am using G1 GC on Java 1.7.45 version in my solaris LAB environment > but am getting very high UpdateRS time and few occasions high Scan RS > time as well. just as reminder, consider at least the latest version of 1.7. There were a few interesting changes to G1 later than 1.7.45. JDK8 would even be better of course :) > OS version : Generic_150401-30 i86pc i386 i86pc Number of CPU # 80, A > day back i have tried to play with below options hoping extra GC > threads will help me but all of no avail as I had witnessed long > pauses up to 7 seconds. Right now I have disabled these GC threads. > > argv[52]: -XX:ParallelGCThreads=70 > argv[53]: -XX:ConcGCThreads=60 > argv[54]: -XX:G1ConcRefinementThreads=70 Unless I misunderstand what a virtual cpu in Solaris is, I do not think using more than there are (64 from an earlier email) gives you any advantage. > I have already put -XX:G1RSetUpdatingPauseTimePercent=5 option and > relaxed pause time goal by setting it to 500 ms but it doesn?t seems > to be helping me. There is one (compiler) optimization that may generate lots of work for the Update RS phase called ReduceInitialCardMarks. Basically it tells g1 to revisit the entire array concurrently. If this allocation happens just before a GC, the gc needs to do all the work in the pause. You can try the option that disables this optimization, i.e. set -XX:- ReduceInitialCardMarks. Of course it is kind of a shot in the dark, and only if you know that your application allocates (a lot) of largish objects. However, this helped in other cases of spurious long Update RS pauses. Also note that I am not sure whether disabling this option causes crashes in 7u - there were some bugfixes in 9 for that. Another option (that should help at least a bit) to reduce update rs/scan rs times may be increasing heap region size to 32M (- XX:G1HeapRegionSize=32M). > Your urgent help will be highly appreciated as I need system where > avg. pause time remain around 200-300 milli-second and Max pause > remain around 500-600 ms. > > Current GC parameters are as below: > argv[11]: -Xmx48g > argv[12]: -Xms48g > argv[13]: -XX:-EliminateLocks > argv[24]: -Xss1m > argv[25]: -Xoss1m > argv[26]: -XX:PermSize=512m > argv[27]: -XX:MaxPermSize=512m > argv[28]: -XX:ReservedCodeCacheSize=128m > argv[29]: -XX:+HeapDumpOnOutOfMemoryError > argv[30]: -XX:+AggressiveOpts > argv[35]: -XX:+UseG1GC > argv[36]: -XX:MaxGCPauseMillis=500 > argv[37]: -XX:+PrintFlagsFinal > argv[38]: -XX:G1RSetUpdatingPauseTimePercent=5 > argv[39]: -XX:+PrintGCTimeStamps > argv[40]: -XX:+PrintGCDetails > argv[42]: -verbose:gc > argv[43]: -XX:+UseLargePages > argv[44]: -XX:+MaxFDLimit > argv[48]: -XX:+UnlockExperimentalVMOptions > argv[49]: -XX:G1NewSizePercent=1 > argv[50]: -XX:+ParallelRefProcEnabled > argv[51]: -XX:+DisableExplicitGC > argv[52]: -XX:G1MaxNewSizePercent=1 I am not sure if setting the maximum new size is a good idea (causing long Object Copy *mostly*). It might be useful to just remove this one. > argv[53]: -XX:InitiatingHeapOccupancyPercent=40 > argv[54]: -XX:+UnlockDiagnosticVMOptions > argv[55]: -XX:+G1SummarizeRSetStats > argv[56]: -XX:G1SummarizeRSetStatsPeriod=1 > argv[57]: -XX:+PerfDisableSharedMem > argv[58]: -XX:+AlwaysPreTouch > > > > Pause snapshots: > > Very high UpdateRS but low scan RS > > 2602.655: [GC pause (young), 7.8981373 secs] > [Parallel Time: 7883.9 ms, GC Workers: 53] > [GC Worker Start (ms): Min: 2602654.9, Avg: 2602655.3, Max: > 2602655.7, Diff: 0.8] > [Ext Root Scanning (ms): Min: 3.5, Avg: 4.5, Max: 34.2, Diff: > 30.7, Sum: 240.0] > [Update RS (ms): Min: 7796.5, Avg: 7826.2, Max: 7827.8, Diff: > 31.4, Sum: 414789.9] > [Processed Buffers: Min: 1, Avg: 3.8, Max: 12, Diff: 11, > Sum: 199] > [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: > 0.6] > [Object Copy (ms): Min: 48.2, Avg: 48.8, Max: 49.4, Diff: 1.3, > Sum: 2588.2] > [Termination (ms): Min: 3.1, Avg: 3.5, Max: 3.9, Diff: 0.8, > Sum: 183.8] > [GC Worker Other (ms): Min: 0.0, Avg: 0.2, Max: 0.4, Diff: 0.3, > Sum: 8.4] > [GC Worker Total (ms): Min: 7882.7, Avg: 7883.2, Max: 7883.8, > Diff: 1.1, Sum: 417810.9] > [GC Worker End (ms): Min: 2610538.4, Avg: 2610538.5, Max: > 2610538.7, Diff: 0.3] > [Code Root Fixup: 0.0 ms] > [Clear CT: 0.9 ms] > [Other: 13.3 ms] > [Choose CSet: 0.0 ms] > [Ref Proc: 10.5 ms] > [Ref Enq: 1.0 ms] > [Free CSet: 0.5 ms] > [Eden: 432.0M(432.0M)->0.0B(416.0M) Survivors: 48.0M->64.0M Heap: > 19.3G(48.0G)->18.9G(48.0G)] > [Times: user=3.24 sys=0.02, real=7.90 secs] Something is wrong here, i.e. a a wall time larger than user time is strange to say at least. Maybe that machine is overloaded with other applications? I recommend trying to get to the bottom of this first. > Concurrent RS processed 48786945 cards > Of 1661300 completed buffers: > 1661300 (100.0%) by conc RS threads. > 0 ( 0.0%) by mutator threads. > Conc RS threads times(s) > 17.55 15.01 12.67 11.26 10.14 9.56 > 8.13 7.26 6.61 6.18 5.25 4.64 3.97 > 3.16 2.92 2.76 2.60 2.54 2.36 2.30 2.16 > 2.12 2.00 1.92 1.83 1.82 1.75 1.69 > 1.64 1.64 1.55 1.50 1.45 1.48 1.41 > 1.36 1.32 1.29 1.26 1.22 1.15 1.18 1.12 > 1.07 1.05 1.03 1.01 0.97 0.96 0.95 > 0.89 0.89 0.87 0.33 > Total heap region rem set sizes = 808374K. Max = 4135K. > Static structures = 1596K, free_lists = 0K. > 72965736 occupied cards represented. > Max size region = > 35:(O)[0xfffffd73e3000000,0xfffffd73e4000000,0xfffffd73e4000000], > size = 4136K, occupied = 14130K. > Did 252 coarsenings. This may or may not be what causes long scan rs times; you could try -XX:G1RSetRegionEntries=2048 to avoid coarsenings (and you should remove the g1SummarizeRSetStats* options again). (the 2048 is just a value higher than the number of regions in the heap, at least with 32m regions). Hth, Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From amit.mishra at redknee.com Tue Jan 24 10:41:44 2017 From: amit.mishra at redknee.com (Amit Mishra) Date: Tue, 24 Jan 2017 10:41:44 +0000 Subject: Need help on G1 GC young gen Update RS and Scan RS pause reduction In-Reply-To: <1485244966.2883.8.camel@oracle.com> References: <1484852604.6579.27.camel@oracle.com> <1485244966.2883.8.camel@oracle.com> Message-ID: Hello Thomas/team, I have put parameters as per your suggestion and now update RS time is manageable but Scan RS and Object copy time are high causing pause time to go beyond 1 second while we do expect max pause time to not to be greater than 500 ms. Value of pause are as below(attaching complete gc file for you) grep -i young gcstats.log.3319|cut -d, -f2|awk -F" " '{print $1}'|awk '$1 > 1' 1.4668911 1.2109846 Note : I am observing pauses just after 4-5 minutes after Application restart post new GC parameters implementation. GC parameters are as: argv[11]: -Xmx48g argv[12]: -Xms48g argv[13]: -XX:-EliminateLocks argv[14]: -Dorg.omg.CORBA.ORBSingletonClass=org.jacorb.orb.ORBSingleton argv[15]: -Dorg.omg.CORBA.ORBClass=org.jacorb.orb.ORB argv[18]: -XX:-ReduceInitialCardMarks argv[19]: -server argv[21]: -classpath argv[24]: -Djava.io.tmpdir=/tmp argv[25]: -Xss1m argv[26]: -Xoss1m argv[27]: -XX:PermSize=512m argv[28]: -XX:MaxPermSize=512m argv[29]: -XX:ReservedCodeCacheSize=128m argv[30]: -XX:+HeapDumpOnOutOfMemoryError argv[31]: -XX:+AggressiveOpts argv[32]: -Dnetworkaddress.cache.ttl=3600 argv[33]: -Dcom.sun.management.jmxremote.port=11883 argv[34]: -Dcom.sun.management.jmxremote.ssl=false argv[35]: -Dcom.sun.management.jmxremote.authenticate=false argv[36]: -XX:+UseG1GC argv[37]: -XX:MaxGCPauseMillis=500 argv[38]: -XX:+PrintFlagsFinal argv[39]: -XX:G1RSetUpdatingPauseTimePercent=5 argv[40]: -XX:+PrintGCTimeStamps argv[41]: -XX:+PrintGCDetails argv[43]: -verbose:gc argv[44]: -XX:+UseLargePages argv[45]: -XX:+MaxFDLimit argv[49]: -XX:+UnlockExperimentalVMOptions argv[50]: -XX:G1NewSizePercent=2 argv[51]: -XX:+ParallelRefProcEnabled argv[52]: -XX:+DisableExplicitGC argv[53]: -XX:ParallelGCThreads=70 argv[54]: -XX:InitiatingHeapOccupancyPercent=40 argv[55]: -XX:+UnlockDiagnosticVMOptions argv[56]: -XX:+G1SummarizeRSetStats argv[57]: -XX:G1SummarizeRSetStatsPeriod=1 argv[58]: -XX:+PerfDisableSharedMem argv[59]: -XX:+AlwaysPreTouch argv[60]: -XX:G1HeapRegionSize=32M argv[61]: -XX:G1RSetRegionEntries=2048 argv[62]: -XX:+UnlockDiagnosticVMOptions Thanks, Amit Mishra -----Original Message----- From: Thomas Schatzl [mailto:thomas.schatzl at oracle.com] Sent: Tuesday, January 24, 2017 13:33 To: Amit Mishra ; hotspot-gc-use at openjdk.java.net Subject: Re: Need help on G1 GC young gen Update RS and Scan RS pause reduction Hi Amit, On Fri, 2017-01-20 at 10:51 +0000, Amit Mishra wrote: > Thank you very much Thomas, after making said changes RSUpdate time is > in control but Scan RS is still high. > ? ? can you summarize your GC parameters again, and also, are these high Scan RS times you observe regular or not? Did you check with?-XX:+G1SummarizeRSetStats?- XX:G1SummarizeRSetStatsPeriod= whether there are still coarsenings? Could you also please provide a longer snippet of your log file containing a few gcs before and after this occurrence? Other worthwhile tries could be to fudge with -XX:ParallelGCThreads, either decreasing or increasing them (up to your vcpus) by ten or so to see whether that phase suffers from synchronization issues. The high scan rs times could maybe also a bit mitigated by increasing G1NewSizePercent to maybe two or three to see whether g1 can avoid large young remembered sets by that (Object copy might go up a bit then though). It may be easier to think of causes for the issues you are experiencing by having a little bit of more information. Thanks, ? Thomas -------------- next part -------------- A non-text attachment was scrubbed... Name: gcstats.log.3319_cps_24jan Type: application/octet-stream Size: 439720 bytes Desc: gcstats.log.3319_cps_24jan URL: From tg at freigmbh.de Tue Jan 24 16:30:22 2017 From: tg at freigmbh.de (Thorsten Goetzke) Date: Tue, 24 Jan 2017 17:30:22 +0100 Subject: Unreachable Memory not freed Message-ID: Hello, I am working on a closed source software that uses a patched version of apache-batik to display SVG-Documents one by one. If you keep loading and discarding some specific SVG-Documents memory consumption keeps on growing until it reaches more than 95% of the available memory. The application becomes unusable slow, very little memory gets freed, although the gc is constantly running. Sometimes an OutOfMemoryError gets thrown. I took memory snaphsots and analyzed them using jvisualvm and yourkit. These tools report the majority of the memory as unreachable. I have tried different garbage collectors (G1,ConcurrentMarkSweep), and different Versions of Java: 1.8_60;1.8_65;1.8_102, the issue is always reproducible. Is there something i can do to debug this further? I am currently not able to reproduce this issue in a somewhat small self contained example. I could provide gc-logs and the heap dump (about 1gb zipped) or any other debuging info if someone is interested. Best Regards, Thorsten Goetzke From yu.zhang at oracle.com Tue Jan 24 16:45:27 2017 From: yu.zhang at oracle.com (Jenny Zhang) Date: Tue, 24 Jan 2017 08:45:27 -0800 Subject: Unreachable Memory not freed In-Reply-To: References: Message-ID: Thorsten, Can you please put the heap dump and gc logs somewhere for download? Thanks Jenny On 1/24/2017 8:30 AM, Thorsten Goetzke wrote: > Hello, > > I am working on a closed source software that uses a patched version > of apache-batik to display SVG-Documents one by one. If you keep > loading and discarding some specific SVG-Documents memory consumption > keeps on growing until it reaches more than 95% of the available > memory. The application becomes unusable slow, very little memory gets > freed, although the gc is constantly running. Sometimes an > OutOfMemoryError gets thrown. I took memory snaphsots and analyzed > them using jvisualvm and yourkit. These tools report the majority of > the memory as unreachable. > I have tried different garbage collectors (G1,ConcurrentMarkSweep), > and different Versions of Java: 1.8_60;1.8_65;1.8_102, the issue is > always reproducible. > Is there something i can do to debug this further? I am currently not > able to reproduce this issue in a somewhat small self contained example. > I could provide gc-logs and the heap dump (about 1gb zipped) or any > other debuging info if someone is interested. > > Best Regards, > Thorsten Goetzke > > > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From tg at freigmbh.de Wed Jan 25 09:15:42 2017 From: tg at freigmbh.de (Thorsten Goetzke) Date: Wed, 25 Jan 2017 10:15:42 +0100 Subject: Unreachable Memory not freed In-Reply-To: References: Message-ID: <83689bdf-7285-3493-e7ab-c46b597bc02a@freigmbh.de> Hello, You can find the heap dump and gc logs under https://filesender.freigmbh.de/filesender/?vid=10f1aeb6-82b4-3d09-c201-00002042c681 Link is valid until 28.01.2017 (1/28/2017). Feel free to message me if you need a reupload. Best Regards, Thorsten > Thorsten, > > Can you please put the heap dump and gc logs somewhere for download? > > Thanks > Jenny > > On 1/24/2017 8:30 AM, Thorsten Goetzke wrote: >> Hello, >> >> I am working on a closed source software that uses a patched version >> of apache-batik to display SVG-Documents one by one. If you keep >> loading and discarding some specific SVG-Documents memory consumption >> keeps on growing until it reaches more than 95% of the available >> memory. The application becomes unusable slow, very little memory gets >> freed, although the gc is constantly running. Sometimes an >> OutOfMemoryError gets thrown. I took memory snaphsots and analyzed >> them using jvisualvm and yourkit. These tools report the majority of >> the memory as unreachable. >> I have tried different garbage collectors (G1,ConcurrentMarkSweep), >> and different Versions of Java: 1.8_60;1.8_65;1.8_102, the issue is >> always reproducible. >> Is there something i can do to debug this further? I am currently not >> able to reproduce this issue in a somewhat small self contained example. >> I could provide gc-logs and the heap dump (about 1gb zipped) or any >> other debuging info if someone is interested. >> >> Best Regards, >> Thorsten Goetzke >> From thomas.schatzl at oracle.com Wed Jan 25 10:07:38 2017 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Wed, 25 Jan 2017 11:07:38 +0100 Subject: Need help on G1 GC young gen Update RS and Scan RS pause reduction In-Reply-To: References: <1484852604.6579.27.camel@oracle.com> <1485244966.2883.8.camel@oracle.com> Message-ID: <1485338858.3625.42.camel@oracle.com> Hi Amit, On Tue, 2017-01-24 at 10:41 +0000, Amit Mishra wrote: > Hello Thomas/team, > > I have put parameters as per your suggestion and now update RS time > is manageable but Scan RS and Object copy time are high causing pause > time to go beyond 1 second while we do expect max pause time to not > to be greater than 500 ms. >From the log it seems that the application, at least during startup time, has quite a bit of variance in the amount of objects that are held live from one garbage collection to the other. It is quite common for applications to behave differently in this area during startup actually. Since G1 sizes the young gen based on previous measurements, if there is a long stretch of garbage collections that do not need lots of work, it will increase the size of the young gen. However, at some point the application's behavior changes (i.e. the number of objects that need to be preserved during collection), and then these long pauses occur. If these long garbage collections are really not desired, the only way I see is to decrease the maximum young generation size, not the minimum one as I wrongly suggested. >From that log I can see that the issue only occurs at the beginning of the run; i.e. the second occurrence is at around 310s, all other ~4100s are fine, using a pretty large young gen (~25G). To avoid these long pauses you would need to set maximum young gen at (I would guess) around 2G. Since this is a global setting for the entire run, this will decrease throughput a lot. It's up to you to determine whether this is okay in your case. To set minimum (and maximum) young generation size there are two sets of options: Set ? -XX:G1NewSizePercent (back) to 1 and ? -XX:G1MaxNewSizePercent to something like 4-5, maybe 3. As you might have noticed, there is not much wiggle room with percentages in your case any more, but you can also set absolute min/max young gen sizes via ? -XX:NewSize=X , probably something around or above 1G seems good. ? -XX:MaxNewSize=Y , in your case something around 2-3G should work. [Please make sure that you set both NewSize/MaxNewSize, otherwise you might experience somewhat unexpected behavior] I also looked through your current settings below. > Value of pause are as below(attaching complete gc file for you) > grep -i young gcstats.log.3319|cut -d, -f2|awk -F" " '{print $1}'|awk > '$1 > 1' > 1.4668911 > 1.2109846 > > Note : I am observing pauses just after 4-5 minutes after Application > restart post new GC parameters implementation. > > GC parameters are as: > > > argv[11]: -Xmx48g > argv[12]: -Xms48g > argv[13]: -XX:-EliminateLocks > argv[14]: > -Dorg.omg.CORBA.ORBSingletonClass=org.jacorb.orb.ORBSingleton > argv[15]: -Dorg.omg.CORBA.ORBClass=org.jacorb.orb.ORB > argv[18]: -XX:-ReduceInitialCardMarks > argv[19]: -server > argv[21]: -classpath > argv[24]: -Djava.io.tmpdir=/tmp > argv[25]: -Xss1m > argv[26]: -Xoss1m > argv[27]: -XX:PermSize=512m > argv[28]: -XX:MaxPermSize=512m > argv[29]: -XX:ReservedCodeCacheSize=128m > argv[30]: -XX:+HeapDumpOnOutOfMemoryError > argv[31]: -XX:+AggressiveOpts > argv[32]: -Dnetworkaddress.cache.ttl=3600 > argv[33]: -Dcom.sun.management.jmxremote.port=11883 > argv[34]: -Dcom.sun.management.jmxremote.ssl=false > argv[35]: -Dcom.sun.management.jmxremote.authenticate=false > argv[36]: -XX:+UseG1GC > argv[37]: -XX:MaxGCPauseMillis=500 > argv[38]: -XX:+PrintFlagsFinal > argv[39]: -XX:G1RSetUpdatingPauseTimePercent=5 > argv[40]: -XX:+PrintGCTimeStamps > argv[41]: -XX:+PrintGCDetails > argv[43]: -verbose:gc > argv[44]: -XX:+UseLargePages > argv[45]: -XX:+MaxFDLimit > argv[49]: -XX:+UnlockExperimentalVMOptions > argv[50]: -XX:G1NewSizePercent=2 ^--- as suggested above, use either G1NewSizePercent/G1MaxNewSizePercent or NewSize/MaxNewSize with the suggested values. NewSize/MaxNewSize don't need the UnlockExperimentalVMOptions btw. > argv[51]: -XX:+ParallelRefProcEnabled > argv[52]: -XX:+DisableExplicitGC > argv[53]: -XX:ParallelGCThreads=70 ^--- again, the recommendation is to not set ParallelGCThreads to anything above the number of virtual cpus you have. You could try removing this again.? > argv[54]: -XX:InitiatingHeapOccupancyPercent=40 I think this is a bit too conservative, but I don't have a good value, and maybe it is required later in the run. Independent on whether you cap maximum new size or not, it might be useful for throughput to increase this a lot. Some plug for JDK9: G1 automatically determines rather good values for that with that version. :) > argv[55]: -XX:+UnlockDiagnosticVMOptions > argv[56]: -XX:+G1SummarizeRSetStats > argv[57]: -XX:G1SummarizeRSetStatsPeriod=1 ^--- again, don't use these two production after your testing completes. If you remove them, I think you can also remove UnlockDiagnosticVMOptions. > argv[58]: -XX:+PerfDisableSharedMem > argv[59]: -XX:+AlwaysPreTouch > argv[60]: -XX:G1HeapRegionSize=32M > argv[61]: -XX:G1RSetRegionEntries=2048 > argv[62]: -XX:+UnlockDiagnosticVMOptions ^--- no need to repeat that at the end. Hth, ? Thomas From amit.mishra at redknee.com Wed Jan 25 10:16:59 2017 From: amit.mishra at redknee.com (Amit Mishra) Date: Wed, 25 Jan 2017 10:16:59 +0000 Subject: Need help on G1 GC young gen Update RS and Scan RS pause reduction In-Reply-To: <1485338858.3625.42.camel@oracle.com> References: <1484852604.6579.27.camel@oracle.com> <1485244966.2883.8.camel@oracle.com> <1485338858.3625.42.camel@oracle.com> Message-ID: Thank you very much Thomas but based on our CMS experience we do also set NewGen Size same as Max New Gen to avoid shrinking and expansion of new gen in real time which sometimes results in unexplainable pauses, can we do the same thing here as well to set min/max size as 1 G and see if it improves overall situation , next thing I am going to do it to increase InitiatingHeapOccupancyPercent from 40 to 60% which we generally set for CMS.(CMSInitiatingOccupancyFraction) I am doing these changes and will let you know once again. Regards, Amit Mishra -----Original Message----- From: Thomas Schatzl [mailto:thomas.schatzl at oracle.com] Sent: Wednesday, January 25, 2017 15:38 To: Amit Mishra ; hotspot-gc-use at openjdk.java.net Subject: Re: Need help on G1 GC young gen Update RS and Scan RS pause reduction Hi Amit, On Tue, 2017-01-24 at 10:41 +0000, Amit Mishra wrote: > Hello Thomas/team, > > I have put parameters as per your suggestion and now update RS time is > manageable but Scan RS and Object copy time are high causing pause > time to go beyond 1 second while we do expect max pause time to not to > be greater than 500 ms. From the log it seems that the application, at least during startup time, has quite a bit of variance in the amount of objects that are held live from one garbage collection to the other. It is quite common for applications to behave differently in this area during startup actually. Since G1 sizes the young gen based on previous measurements, if there is a long stretch of garbage collections that do not need lots of work, it will increase the size of the young gen. However, at some point the application's behavior changes (i.e. the number of objects that need to be preserved during collection), and then these long pauses occur. If these long garbage collections are really not desired, the only way I see is to decrease the maximum young generation size, not the minimum one as I wrongly suggested. From that log I can see that the issue only occurs at the beginning of the run; i.e. the second occurrence is at around 310s, all other ~4100s are fine, using a pretty large young gen (~25G). To avoid these long pauses you would need to set maximum young gen at (I would guess) around 2G. Since this is a global setting for the entire run, this will decrease throughput a lot. It's up to you to determine whether this is okay in your case. To set minimum (and maximum) young generation size there are two sets of options: Set ? -XX:G1NewSizePercent (back) to 1 and ? -XX:G1MaxNewSizePercent to something like 4-5, maybe 3. As you might have noticed, there is not much wiggle room with percentages in your case any more, but you can also set absolute min/max young gen sizes via ? -XX:NewSize=X , probably something around or above 1G seems good. ? -XX:MaxNewSize=Y , in your case something around 2-3G should work. [Please make sure that you set both NewSize/MaxNewSize, otherwise you might experience somewhat unexpected behavior] I also looked through your current settings below. > Value of pause are as below(attaching complete gc file for you) grep > -i young gcstats.log.3319|cut -d, -f2|awk -F" " '{print $1}'|awk > '$1 > 1' > 1.4668911 > 1.2109846 > > Note : I am observing pauses just after 4-5 minutes after Application > restart post new GC parameters implementation. > > GC parameters are as: > > > argv[11]: -Xmx48g > argv[12]: -Xms48g > argv[13]: -XX:-EliminateLocks > argv[14]: > -Dorg.omg.CORBA.ORBSingletonClass=org.jacorb.orb.ORBSingleton > argv[15]: -Dorg.omg.CORBA.ORBClass=org.jacorb.orb.ORB > argv[18]: -XX:-ReduceInitialCardMarks > argv[19]: -server > argv[21]: -classpath > argv[24]: -Djava.io.tmpdir=/tmp > argv[25]: -Xss1m > argv[26]: -Xoss1m > argv[27]: -XX:PermSize=512m > argv[28]: -XX:MaxPermSize=512m > argv[29]: -XX:ReservedCodeCacheSize=128m > argv[30]: -XX:+HeapDumpOnOutOfMemoryError > argv[31]: -XX:+AggressiveOpts > argv[32]: -Dnetworkaddress.cache.ttl=3600 > argv[33]: -Dcom.sun.management.jmxremote.port=11883 > argv[34]: -Dcom.sun.management.jmxremote.ssl=false > argv[35]: -Dcom.sun.management.jmxremote.authenticate=false > argv[36]: -XX:+UseG1GC > argv[37]: -XX:MaxGCPauseMillis=500 > argv[38]: -XX:+PrintFlagsFinal > argv[39]: -XX:G1RSetUpdatingPauseTimePercent=5 > argv[40]: -XX:+PrintGCTimeStamps > argv[41]: -XX:+PrintGCDetails > argv[43]: -verbose:gc > argv[44]: -XX:+UseLargePages > argv[45]: -XX:+MaxFDLimit > argv[49]: -XX:+UnlockExperimentalVMOptions > argv[50]: -XX:G1NewSizePercent=2 ^--- as suggested above, use either G1NewSizePercent/G1MaxNewSizePercent or NewSize/MaxNewSize with the suggested values. NewSize/MaxNewSize don't need the UnlockExperimentalVMOptions btw. > argv[51]: -XX:+ParallelRefProcEnabled > argv[52]: -XX:+DisableExplicitGC > argv[53]: -XX:ParallelGCThreads=70 ^--- again, the recommendation is to not set ParallelGCThreads to anything above the number of virtual cpus you have. You could try removing this again.? > argv[54]: -XX:InitiatingHeapOccupancyPercent=40 I think this is a bit too conservative, but I don't have a good value, and maybe it is required later in the run. Independent on whether you cap maximum new size or not, it might be useful for throughput to increase this a lot. Some plug for JDK9: G1 automatically determines rather good values for that with that version. :) > argv[55]: -XX:+UnlockDiagnosticVMOptions > argv[56]: -XX:+G1SummarizeRSetStats > argv[57]: -XX:G1SummarizeRSetStatsPeriod=1 ^--- again, don't use these two production after your testing completes. If you remove them, I think you can also remove UnlockDiagnosticVMOptions. > argv[58]: -XX:+PerfDisableSharedMem > argv[59]: -XX:+AlwaysPreTouch > argv[60]: -XX:G1HeapRegionSize=32M > argv[61]: -XX:G1RSetRegionEntries=2048 > argv[62]: -XX:+UnlockDiagnosticVMOptions ^--- no need to repeat that at the end. Hth, ? Thomas From nanda.yugandhar at gmail.com Wed Jan 25 15:15:08 2017 From: nanda.yugandhar at gmail.com (yugandhar nanda) Date: Wed, 25 Jan 2017 15:15:08 +0000 Subject: Sudden increase in survivor space with G1GC Message-ID: Hi, I'm facing some issue with G1GC, it's suddenly increasing it's survivor space and then the old gen heap space. *JVM and command info:* Java HotSpot(TM) 64-Bit Server VM (25.121-b13) for linux-amd64 JRE (1.8.0_121-b13), built on Dec 12 2016 16:36:53 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8) Memory: 4k page, physical 61832044k(18159472k free), swap 0k(0k free) CommandLine flags: -XX:ConcGCThreads=8 -XX:G1MaxNewSizePercent=50 -XX:G1NewSizePercent=50 -XX:+HeapDumpAfterFullGC -XX:+HeapDumpBeforeFullGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=../app-thread_dump.hprof -XX:InitialHeapSize=8589934592 -XX:InitiatingHeapOccupancyPercent=75 -XX:MaxGCPauseMillis=150 -XX:MaxHeapSize=8589934592 -XX:ParallelGCThreads=20 -XX:+ParallelRefProcEnabled -XX:+PrintAdaptiveSizePolicy -XX:+PrintClassHistogramAfterFullGC -XX:+PrintClassHistogramBeforeFullGC -XX:+PrintGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTaskTimeStamps -XX:+PrintGCTimeStamps -XX:+PrintJNIGCStalls -XX:+PrintReferenceGC -XX:+PrintTLAB -XX:+PrintTenuringDistribution -XX:+UnlockExperimentalVMOptions -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC *In general GC log (without issue):* 2017-01-25T13:02:26.812+0530: 63991.265: Total time for which application threads were stopped: 0.0130009 seconds, Stopping threads took: 0.0000569 seconds 2017-01-25T13:02:26.812+0530: 63991.265: Application time: 0.0001108 seconds 2017-01-25T13:02:26.813+0530: 63991.265: Total time for which application threads were stopped: 0.0002348 seconds, Stopping threads took: 0.0000709 seconds 2017-01-25T13:02:26.813+0530: 63991.265: Application time: 0.0001053 seconds 2017-01-25T13:02:26.813+0530: 63991.265: Total time for which application threads were stopped: 0.0001487 seconds, Stopping threads took: 0.0000642 seconds 2017-01-25T13:02:26.813+0530: 63991.266: Application time: 0.0000308 seconds 2017-01-25T13:02:26.813+0530: 63991.266: Total time for which application threads were stopped: 0.0000995 seconds, Stopping threads took: 0.0000221 seconds 2017-01-25T13:02:26.813+0530: 63991.266: Application time: 0.0000918 seconds 2017-01-25T13:02:26.813+0530: 63991.266: Total time for which application threads were stopped: 0.0003873 seconds, Stopping threads took: 0.0000630 seconds 2017-01-25T13:02:30.548+0530: 63995.001: Application time: 3.7348000 seconds 2017-01-25T13:02:30.549+0530: 63995.001: Total time for which application threads were stopped: 0.0002501 seconds, Stopping threads took: 0.0000410 seconds 2017-01-25T13:02:35.549+0530: 64000.001: Application time: 5.0000879 seconds 2017-01-25T13:02:35.549+0530: 64000.001: Total time for which application threads were stopped: 0.0002537 seconds, Stopping threads took: 0.0000520 seconds 2017-01-25T13:02:40.549+0530: 64005.002: Application time: 5.0001048 seconds 2017-01-25T13:02:40.549+0530: 64005.002: Total time for which application threads were stopped: 0.0002789 seconds, Stopping threads took: 0.0000443 seconds 2017-01-25T13:02:45.549+0530: 64010.002: Application time: 5.0002048 seconds 2017-01-25T13:02:45.550+0530: 64010.002: Total time for which application threads were stopped: 0.0002770 seconds, Stopping threads took: 0.0000567 seconds 2017-01-25T13:02:50.550+0530: 64015.003: Application time: 5.0002496 seconds 2017-01-25T13:02:50.550+0530: 64015.003: Total time for which application threads were stopped: 0.0003006 seconds, Stopping threads took: 0.0000513 seconds 2017-01-25T13:02:55.551+0530: 64020.003: Application time: 5.0002249 seconds 2017-01-25T13:02:55.551+0530: 64020.003: Total time for which application threads were stopped: 0.0002606 seconds, Stopping threads took: 0.0000475 seconds 2017-01-25T13:03:00.551+0530: 64025.004: Application time: 5.0001728 seconds 2017-01-25T13:03:00.551+0530: 64025.004: Total time for which application threads were stopped: 0.0002634 seconds, Stopping threads took: 0.0000488 seconds 2017-01-25T13:03:05.551+0530: 64030.004: Application time: 5.0001301 seconds 2017-01-25T13:03:05.552+0530: 64030.004: Total time for which application threads were stopped: 0.0002513 seconds, Stopping threads took: 0.0000492 seconds 2017-01-25T13:03:06.552+0530: 64031.004: Application time: 1.0001395 seconds 2017-01-25T13:03:06.552+0530: 64031.005: Total time for which application threads were stopped: 0.0002258 seconds, Stopping threads took: 0.0000423 seconds 2017-01-25T13:03:10.552+0530: 64035.004: Application time: 3.9998560 seconds 2017-01-25T13:03:10.552+0530: 64035.005: Total time for which application threads were stopped: 0.0002408 seconds, Stopping threads took: 0.0000388 seconds 2017-01-25T13:03:15.552+0530: 64040.005: Application time: 5.0002342 seconds 2017-01-25T13:03:15.553+0530: 64040.005: Total time for which application threads were stopped: 0.0002513 seconds, Stopping threads took: 0.0000489 seconds 2017-01-25T13:03:20.553+0530: 64045.005: Application time: 5.0001538 seconds 2017-01-25T13:03:20.553+0530: 64045.006: Total time for which application threads were stopped: 0.0002752 seconds, Stopping threads took: 0.0000501 seconds 2017-01-25T13:03:25.553+0530: 64050.006: Application time: 5.0001211 seconds 2017-01-25T13:03:25.553+0530: 64050.006: Total time for which application threads were stopped: 0.0002745 seconds, Stopping threads took: 0.0000516 seconds 2017-01-25T13:03:30.554+0530: 64055.006: Application time: 5.0001992 seconds 2017-01-25T13:03:30.554+0530: 64055.006: Total time for which application threads were stopped: 0.0002515 seconds, Stopping threads took: 0.0000456 seconds 2017-01-25T13:03:35.554+0530: 64060.007: Application time: 5.0001805 seconds 2017-01-25T13:03:35.554+0530: 64060.007: Total time for which application threads were stopped: 0.0002972 seconds, Stopping threads took: 0.0000563 seconds 2017-01-25T13:03:40.555+0530: 64065.007: Application time: 5.0002179 seconds 2017-01-25T13:03:40.555+0530: 64065.007: Total time for which application threads were stopped: 0.0002622 seconds, Stopping threads took: 0.0000415 seconds 2017-01-25T13:03:45.555+0530: 64070.008: Application time: 5.0002244 seconds 2017-01-25T13:03:45.555+0530: 64070.008: Total time for which application threads were stopped: 0.0002557 seconds, Stopping threads took: 0.0000399 seconds 2017-01-25T13:03:50.556+0530: 64075.008: Application time: 5.0001938 seconds 2017-01-25T13:03:50.556+0530: 64075.008: Total time for which application threads were stopped: 0.0002542 seconds, Stopping threads took: 0.0000422 seconds 2017-01-25T13:03:55.556+0530: 64080.009: Application time: 5.0001925 seconds 2017-01-25T13:03:55.556+0530: 64080.009: Total time for which application threads were stopped: 0.0002727 seconds, Stopping threads took: 0.0000435 seconds 2017-01-25T13:04:00.556+0530: 64085.009: Application time: 5.0002240 seconds 2017-01-25T13:04:00.557+0530: 64085.009: Total time for which application threads were stopped: 0.0002573 seconds, Stopping threads took: 0.0000387 seconds 2017-01-25T13:04:05.557+0530: 64090.010: Application time: 5.0002203 seconds 2017-01-25T13:04:05.557+0530: 64090.010: Total time for which application threads were stopped: 0.0003009 seconds, Stopping threads took: 0.0000419 seconds 2017-01-25T13:04:10.557+0530: 64095.010: Application time: 5.0002308 seconds 2017-01-25T13:04:10.558+0530: 64095.010: Total time for which application threads were stopped: 0.0002892 seconds, Stopping threads took: 0.0000502 seconds 2017-01-25T13:04:15.558+0530: 64100.011: Application time: 5.0001824 seconds 2017-01-25T13:04:15.558+0530: 64100.011: Total time for which application threads were stopped: 0.0002421 seconds, Stopping threads took: 0.0000407 seconds 2017-01-25T13:04:20.558+0530: 64105.011: Application time: 5.0002074 seconds 2017-01-25T13:04:20.559+0530: 64105.011: Total time for which application threads were stopped: 0.0002598 seconds, Stopping threads took: 0.0000443 seconds 2017-01-25T13:04:25.559+0530: 64110.012: Application time: 5.0002581 seconds 2017-01-25T13:04:25.559+0530: 64110.012: Total time for which application threads were stopped: 0.0002521 seconds, Stopping threads took: 0.0000418 seconds 2017-01-25T13:04:30.559+0530: 64115.012: Application time: 5.0002090 seconds 2017-01-25T13:04:30.560+0530: 64115.012: Total time for which application threads were stopped: 0.0002510 seconds, Stopping threads took: 0.0000409 seconds 2017-01-25T13:04:35.560+0530: 64120.012: Application time: 5.0001457 seconds 2017-01-25T13:04:35.560+0530: 64120.013: Total time for which application threads were stopped: 0.0002465 seconds, Stopping threads took: 0.0000399 seconds 2017-01-25T13:04:37.135+0530: 64121.588: Application time: 1.5748912 seconds 2017-01-25T13:04:37.135+0530: 64121.588: [GC pause (G1 Evacuation Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] desired_size: 219KB slow allocs: 0 refill waste: 3504B alloc: 0.00262 10964KB refills: 46 waste 0.1% gc: 1536B slow: 7312B fast: 0B TLAB: gc thread: 0x00007f15f8114800 [id: 199] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.03023 126678KB refills: 55 waste 1.3% gc: 1534976B slow: 17184B fast: 0B TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 7KB slow allocs: 0 refill waste: 120B alloc: 0.00009 375KB refills: 45 waste 2.7% gc: 7512B slow: 1672B fast: 0B TLAB: gc thread: 0x00007f15906a8800 [id: 144] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00001 21KB refills: 8 waste 7.3% gc: 1008B slow: 200B fast: 0B TLAB: gc thread: 0x00007f15906a7000 [id: 143] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00001 21KB refills: 8 waste 6.3% gc: 880B slow: 168B fast: 0B TLAB: gc thread: 0x00007f15906a5000 [id: 142] desired_size: 2KB slow allocs: 2 refill waste: 32B alloc: 0.00001 21KB refills: 8 waste 7.0% gc: 928B slow: 224B fast: 0B TLAB: gc thread: 0x00007f15906a3800 [id: 141] desired_size: 2KB slow allocs: 7 refill waste: 32B alloc: 0.00001 56KB refills: 41 waste 1.5% gc: 40B slow: 1240B fast: 0B TLAB: gc thread: 0x00007f15906a1800 [id: 140] desired_size: 990KB slow allocs: 1 refill waste: 15848B alloc: 0.01182 49545KB refills: 42 waste 2.1% gc: 887440B slow: 5288B fast: 0B TLAB: gc thread: 0x00007f15906a0000 [id: 139] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00001 21KB refills: 8 waste 7.3% gc: 1008B slow: 200B fast: 0B TLAB: gc thread: 0x00007f159069e800 [id: 138] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00001 21KB refills: 8 waste 7.3% gc: 1008B slow: 200B fast: 0B TLAB: gc thread: 0x00007f159069c800 [id: 137] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00001 21KB refills: 8 waste 6.8% gc: 928B slow: 200B fast: 0B TLAB: gc thread: 0x00007f159069b000 [id: 136] desired_size: 2KB slow allocs: 0 refill waste: 40B alloc: 0.00004 147KB refills: 5 waste 4.8% gc: 568B slow: 152B fast: 0B TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 939KB slow allocs: 0 refill waste: 15032B alloc: 0.01122 46995KB refills: 49 waste 1.4% gc: 644224B slow: 10456B fast: 0B TLAB: gc thread: 0x00007f1590697800 [id: 134] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00001 21KB refills: 8 waste 6.8% gc: 928B slow: 200B fast: 0B TLAB: gc thread: 0x00007f1590695800 [id: 133] desired_size: 2KB slow allocs: 2 refill waste: 32B alloc: 0.00001 21KB refills: 8 waste 7.5% gc: 1008B slow: 224B fast: 0B TLAB: gc thread: 0x00007f1590693800 [id: 132] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00001 21KB refills: 8 waste 7.3% gc: 1008B slow: 200B fast: 0B TLAB: gc thread: 0x00007f1590692000 [id: 131] desired_size: 2KB slow allocs: 9 refill waste: 32B alloc: 0.00002 66KB refills: 28 waste 4.3% gc: 1656B slow: 832B fast: 0B TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 966KB slow allocs: 0 refill waste: 15456B alloc: 0.01153 48313KB refills: 47 waste 2.1% gc: 972808B slow: 7088B fast: 0B TLAB: gc thread: 0x00007f159068e800 [id: 129] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00001 21KB refills: 8 waste 6.8% gc: 928B slow: 200B fast: 0B TLAB: gc thread: 0x00007f159068d000 [id: 128] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00001 21KB refills: 8 waste 7.3% gc: 1008B slow: 200B fast: 0B TLAB: gc thread: 0x00007f159068b000 [id: 127] desired_size: 2KB slow allocs: 2 refill waste: 32B alloc: 0.00001 22KB refills: 7 waste 15.5% gc: 2064B slow: 176B fast: 0B TLAB: gc thread: 0x00007f1590689800 [id: 126] desired_size: 4KB slow allocs: 0 refill waste: 64B alloc: 0.00005 207KB refills: 5 waste 16.6% gc: 3376B slow: 152B fast: 0B TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 995KB slow allocs: 0 refill waste: 15920B alloc: 0.01188 49768KB refills: 46 waste 1.0% gc: 478608B slow: 11312B fast: 0B TLAB: gc thread: 0x00007f1590674800 [id: 124] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00001 21KB refills: 8 waste 6.8% gc: 928B slow: 200B fast: 0B TLAB: gc thread: 0x00007f1590673000 [id: 123] desired_size: 2KB slow allocs: 2 refill waste: 32B alloc: 0.00001 21KB refills: 8 waste 7.5% gc: 1008B slow: 224B fast: 0B TLAB: gc thread: 0x00007f1590671800 [id: 122] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00001 21KB refills: 8 waste 7.3% gc: 1008B slow: 200B fast: 0B TLAB: gc thread: 0x00007f159067d800 [id: 121] desired_size: 2KB slow allocs: 12 refill waste: 32B alloc: 0.00002 85KB refills: 73 waste 1.7% gc: 360B slow: 2264B fast: 0B TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 981KB slow allocs: 0 refill waste: 15696B alloc: 0.01171 49074KB refills: 46 waste 0.6% gc: 282728B slow: 4480B fast: 0B TLAB: gc thread: 0x00007f1590637000 [id: 117] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00001 21KB refills: 8 waste 9.5% gc: 1360B slow: 216B fast: 0B TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 125KB slow allocs: 0 refill waste: 2008B alloc: 0.00150 6286KB refills: 50 waste 1.2% gc: 74680B slow: 1784B fast: 0B TLAB: gc thread: 0x00007f159062b000 [id: 115] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 4KB refills: 2 waste 38.8% gc: 1568B slow: 32B fast: 0B TLAB: gc thread: 0x00007f159005b000 [id: 114] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 1KB refills: 1 waste 60.9% gc: 1256B slow: 0B fast: 0B TLAB: gc thread: 0x00007f158c010800 [id: 113] desired_size: 37KB slow allocs: 47 refill waste: 592B alloc: 0.00045 1870KB refills: 31 waste 1.8% gc: 19784B slow: 2160B fast: 0B TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.19981 837259KB refills: 1519 waste 0.0% gc: 0B slow: 91944B fast: 0B TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.08702 364643KB refills: 173 waste 0.2% gc: 650768B slow: 11936B fast: 0B TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 2047KB slow allocs: 40 refill waste: 32760B alloc: 0.14335 600674KB refills: 261 waste 0.1% gc: 158528B slow: 512864B fast: 0B TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 501KB slow allocs: 0 refill waste: 8024B alloc: 0.00598 25076KB refills: 45 waste 1.2% gc: 280640B slow: 4848B fast: 0B TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 586KB slow allocs: 0 refill waste: 9384B alloc: 0.00700 29348KB refills: 44 waste 1.8% gc: 484792B slow: 3672B fast: 0B TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 2047KB slow allocs: 112 refill waste: 32760B alloc: 0.45892 1922957KB refills: 799 waste 0.1% gc: 747928B slow: 1461464B fast: 0B TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 10KB refills: 4 waste 26.0% gc: 2016B slow: 128B fast: 0B TLAB: gc thread: 0x00007f170d35a000 [id: 85] desired_size: 165KB slow allocs: 0 refill waste: 2640B alloc: 0.00197 8251KB refills: 30 waste 0.5% gc: 22672B slow: 1704B fast: 0B TLAB: gc thread: 0x00007f170d2cf000 [id: 84] desired_size: 2KB slow allocs: 2 refill waste: 32B alloc: 0.00000 15KB refills: 5 waste 1.8% gc: 72B slow: 112B fast: 0B TLAB: gc thread: 0x00007f170cda3800 [id: 63] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 0KB refills: 1 waste 86.4% gc: 1784B slow: 0B fast: 0B TLAB totals: thrds: 43 refills: 3620 max: 1519 slow allocs: 249 max 112 waste: 0.2% gc: 7279328B max: 1534976B slow: 2165312B max: 1461464B fast: 0B max: 0B Desired survivor size 268435456 bytes, new threshold 15 (max 15) - age 1: 1800488 bytes, 1800488 total - age 2: 14336 bytes, 1814824 total - age 3: 464 bytes, 1815288 total - age 4: 17296 bytes, 1832584 total - age 5: 144 bytes, 1832728 total - age 6: 144 bytes, 1832872 total - age 7: 5744 bytes, 1838616 total - age 8: 72 bytes, 1838688 total - age 9: 864 bytes, 1839552 total - age 10: 48 bytes, 1839600 total - age 11: 48 bytes, 1839648 total - age 13: 80 bytes, 1839728 total - age 14: 656 bytes, 1840384 total - age 15: 48 bytes, 1840432 total 64121.588: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 4231, predicted base time: 9.67 ms, remaining time: 140.33 ms, target pause time: 150.00 ms] 64121.588: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 1023 regions, survivors: 1 regions, predicted young region time: 2.93 ms] 64121.588: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 1023 regions, survivors: 1 regions, old: 0 regions, predicted pause time: 12.60 ms, target pause time: 150.00 ms] 2017-01-25T13:04:37.139+0530: 64121.591: [SoftReference, 0 refs, 0.0052402 secs]2017-01-25T13:04:37.144+0530: 64121.597: [WeakReference, 3 refs, 0.0008352 secs]2017-01-25T13:04:37.145+0530: 64121.597: [FinalReference, 35 refs, 0.0012709 secs]2017-01-25T13:04:37.146+0530: 64121.599: [PhantomReference, 0 refs, 0 refs, 0.0017444 secs]2017-01-25T13:04:37.148+0530: 64121.601: [JNI Weak Reference, 0.0000165 secs], 0.0161273 secs] [Parallel Time: 2.7 ms, GC Workers: 20] [GC Worker Start (ms): Min: 64121588.1, Avg: 64121588.8, Max: 64121590.5, Diff: 2.4] [Ext Root Scanning (ms): Min: 0.0, Avg: 0.7, Max: 2.3, Diff: 2.3, Sum: 14.9] [Update RS (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.3, Sum: 1.7] [Processed Buffers: Min: 0, Avg: 2.8, Max: 8, Diff: 8, Sum: 56] [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.2, Sum: 2.4] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 0.0, Avg: 0.6, Max: 1.2, Diff: 1.2, Sum: 13.0] [Termination (ms): Min: 0.0, Avg: 0.2, Max: 0.3, Diff: 0.3, Sum: 3.7] [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 20] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 1.2] [GC Worker Total (ms): Min: 0.1, Avg: 1.8, Max: 2.5, Diff: 2.5, Sum: 36.8] [GC Worker End (ms): Min: 64121590.6, Avg: 64121590.6, Max: 64121590.7, Diff: 0.1] [Code Root Fixup: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.7 ms] [Other: 12.6 ms] [Choose CSet: 0.0 ms] [Ref Proc: 9.7 ms] [Ref Enq: 0.4 ms] [Redirty Cards: 0.3 ms] [Humongous Register: 0.1 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 1.9 ms] [Eden: 4092.0M(4092.0M)->0.0B(4088.0M) Survivors: 4096.0K->8192.0K Heap: 4179.6M(8192.0M)->90.7M(8192.0M)] [Times: user=0.16 sys=0.00, real=0.02 secs] 2017-01-25T13:04:37.151+0530: 64121.604: Total time for which application threads were stopped: 0.0165431 seconds, Stopping threads took: 0.0000626 seconds 2017-01-25T13:04:37.152+0530: 64121.604: Application time: 0.0000825 seconds 2017-01-25T13:04:37.152+0530: 64121.604: Total time for which application threads were stopped: 0.0002369 seconds, Stopping threads took: 0.0000639 seconds 2017-01-25T13:04:37.152+0530: 64121.604: Application time: 0.0000353 seconds 2017-01-25T13:04:37.152+0530: 64121.605: Total time for which application threads were stopped: 0.0001141 seconds, Stopping threads took: 0.0000393 seconds 2017-01-25T13:04:37.152+0530: 64121.605: Application time: 0.0000278 seconds 2017-01-25T13:04:37.152+0530: 64121.605: Total time for which application threads were stopped: 0.0001138 seconds, Stopping threads took: 0.0000388 seconds 2017-01-25T13:04:37.152+0530: 64121.605: Application time: 0.0000261 seconds 2017-01-25T13:04:37.152+0530: 64121.605: Total time for which application threads were stopped: 0.0001220 seconds, Stopping threads took: 0.0000420 seconds 2017-01-25T13:04:37.152+0530: 64121.605: Application time: 0.0000273 seconds 2017-01-25T13:04:37.152+0530: 64121.605: Total time for which application threads were stopped: 0.0001167 seconds, Stopping threads took: 0.0000382 seconds 2017-01-25T13:04:37.152+0530: 64121.605: Application time: 0.0000297 seconds 2017-01-25T13:04:37.153+0530: 64121.605: Total time for which application threads were stopped: 0.0001116 seconds, Stopping threads took: 0.0000387 seconds 2017-01-25T13:04:37.153+0530: 64121.605: Application time: 0.0000242 seconds 2017-01-25T13:04:37.153+0530: 64121.605: Total time for which application threads were stopped: 0.0001297 seconds, Stopping threads took: 0.0000483 seconds 2017-01-25T13:04:37.153+0530: 64121.605: Application time: 0.0000237 seconds 2017-01-25T13:04:37.153+0530: 64121.605: Total time for which application threads were stopped: 0.0001265 seconds, Stopping threads took: 0.0000466 seconds 2017-01-25T13:04:40.560+0530: 64125.013: Application time: 3.4073931 seconds 2017-01-25T13:04:40.561+0530: 64125.013: Total time for which application threads were stopped: 0.0002536 seconds, Stopping threads took: 0.0000435 seconds 2017-01-25T13:04:45.561+0530: 64130.013: Application time: 5.0002173 seconds 2017-01-25T13:04:45.561+0530: 64130.014: Total time for which application threads were stopped: 0.0002343 seconds, Stopping threads took: 0.0000381 seconds 2017-01-25T13:04:50.561+0530: 64135.014: Application time: 5.0002234 seconds 2017-01-25T13:04:50.561+0530: 64135.014: Total time for which application threads were stopped: 0.0002375 seconds, Stopping threads took: 0.0000394 seconds 2017-01-25T13:04:55.562+0530: 64140.014: Application time: 5.0002266 seconds 2017-01-25T13:04:55.562+0530: 64140.015: Total time for which application threads were stopped: 0.0002543 seconds, Stopping threads took: 0.0000466 seconds 2017-01-25T13:05:00.562+0530: 64145.015: Application time: 5.0001965 seconds 2017-01-25T13:05:00.562+0530: 64145.015: Total time for which application threads were stopped: 0.0002787 seconds, Stopping threads took: 0.0000461 seconds 2017-01-25T13:05:05.563+0530: 64150.015: Application time: 5.0002188 seconds 2017-01-25T13:05:05.563+0530: 64150.015: Total time for which application threads were stopped: 0.0002429 seconds, Stopping threads took: 0.0000443 seconds 2017-01-25T13:05:10.563+0530: 64155.016: Application time: 5.0001550 seconds 2017-01-25T13:05:10.563+0530: 64155.016: Total time for which application threads were stopped: 0.0002682 seconds, Stopping threads took: 0.0000528 seconds 2017-01-25T13:05:15.563+0530: 64160.016: Application time: 5.0001906 seconds 2017-01-25T13:05:15.564+0530: 64160.016: Total time for which application threads were stopped: 0.0002624 seconds, Stopping threads took: 0.0000533 seconds 2017-01-25T13:05:20.564+0530: 64165.016: Application time: 5.0001018 seconds 2017-01-25T13:05:20.564+0530: 64165.017: Total time for which application threads were stopped: 0.0003091 seconds, Stopping threads took: 0.0000802 seconds 2017-01-25T13:05:25.564+0530: 64170.017: Application time: 5.0001915 seconds 2017-01-25T13:05:25.565+0530: 64170.017: Total time for which application threads were stopped: 0.0002597 seconds, Stopping threads took: 0.0000489 seconds 2017-01-25T13:05:30.565+0530: 64175.017: Application time: 5.0001223 seconds 2017-01-25T13:05:30.565+0530: 64175.018: Total time for which application threads were stopped: 0.0002646 seconds, Stopping threads took: 0.0000521 seconds 2017-01-25T13:05:35.565+0530: 64180.018: Application time: 5.0001622 seconds 2017-01-25T13:05:35.565+0530: 64180.018: Total time for which application threads were stopped: 0.0002546 seconds, Stopping threads took: 0.0000495 seconds 2017-01-25T13:05:40.566+0530: 64185.018: Application time: 5.0002130 seconds 2017-01-25T13:05:40.566+0530: 64185.019: Total time for which application threads were stopped: 0.0002766 seconds, Stopping threads took: 0.0000417 seconds 2017-01-25T13:05:45.566+0530: 64190.019: Application time: 5.0002184 seconds 2017-01-25T13:05:45.566+0530: 64190.019: Total time for which application threads were stopped: 0.0002403 seconds, Stopping threads took: 0.0000375 seconds 2017-01-25T13:05:50.567+0530: 64195.019: Application time: 5.0002563 seconds 2017-01-25T13:05:50.567+0530: 64195.020: Total time for which application threads were stopped: 0.0002875 seconds, Stopping threads took: 0.0000455 seconds 2017-01-25T13:05:55.567+0530: 64200.020: Application time: 5.0002347 seconds 2017-01-25T13:05:55.567+0530: 64200.020: Total time for which application threads were stopped: 0.0002582 seconds, Stopping threads took: 0.0000465 seconds 2017-01-25T13:06:00.035+0530: 64204.488: Application time: 4.4678607 seconds 2017-01-25T13:06:00.036+0530: 64204.488: Total time for which application threads were stopped: 0.0002706 seconds, Stopping threads took: 0.0000526 seconds 2017-01-25T13:06:00.568+0530: 64205.020: Application time: 0.5320424 seconds 2017-01-25T13:06:00.568+0530: 64205.020: Total time for which application threads were stopped: 0.0002378 seconds, Stopping threads took: 0.0000537 seconds 2017-01-25T13:06:05.568+0530: 64210.021: Application time: 5.0002275 seconds 2017-01-25T13:06:05.568+0530: 64210.021: Total time for which application threads were stopped: 0.0002358 seconds, Stopping threads took: 0.0000394 seconds 2017-01-25T13:06:10.569+0530: 64215.021: Application time: 5.0002358 seconds 2017-01-25T13:06:10.569+0530: 64215.021: Total time for which application threads were stopped: 0.0002551 seconds, Stopping threads took: 0.0000409 seconds 2017-01-25T13:06:15.569+0530: 64220.022: Application time: 5.0002107 seconds 2017-01-25T13:06:15.569+0530: 64220.022: Total time for which application threads were stopped: 0.0002469 seconds, Stopping threads took: 0.0000403 seconds 2017-01-25T13:06:20.569+0530: 64225.022: Application time: 5.0001874 seconds 2017-01-25T13:06:20.570+0530: 64225.022: Total time for which application threads were stopped: 0.0002622 seconds, Stopping threads took: 0.0000424 seconds 2017-01-25T13:06:25.570+0530: 64230.023: Application time: 5.0002091 seconds 2017-01-25T13:06:25.570+0530: 64230.023: Total time for which application threads were stopped: 0.0002537 seconds, Stopping threads took: 0.0000461 seconds 2017-01-25T13:06:30.570+0530: 64235.023: Application time: 5.0002466 seconds 2017-01-25T13:06:30.571+0530: 64235.023: Total time for which application threads were stopped: 0.0002569 seconds, Stopping threads took: 0.0000517 seconds 2017-01-25T13:06:35.571+0530: 64240.024: Application time: 5.0002156 seconds 2017-01-25T13:06:35.571+0530: 64240.024: Total time for which application threads were stopped: 0.0002409 seconds, Stopping threads took: 0.0000435 seconds 2017-01-25T13:06:37.364+0530: 64241.817: Application time: 1.7927933 seconds 2017-01-25T13:06:37.364+0530: 64241.817: [GC pause (G1 Evacuation Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] desired_size: 188KB slow allocs: 0 refill waste: 3016B alloc: 0.00225 9430KB refills: 47 waste 1.0% gc: 79928B slow: 6736B fast: 0B TLAB: gc thread: 0x00007f15f8114800 [id: 199] desired_size: 2047KB slow allocs: 1 refill waste: 32760B alloc: 0.02598 108755KB refills: 60 waste 0.0% gc: 0B slow: 15240B fast: 0B TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 6KB slow allocs: 0 refill waste: 104B alloc: 0.00008 326KB refills: 43 waste 0.8% gc: 632B slow: 1752B fast: 0B TLAB: gc thread: 0x00007f14d4080000 [id: 197] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 0KB refills: 1 waste 95.0% gc: 1960B slow: 0B fast: 0B TLAB: gc thread: 0x00007f15906a8800 [id: 144] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00000 19KB refills: 7 waste 15.3% gc: 2064B slow: 152B fast: 0B TLAB: gc thread: 0x00007f15906a7000 [id: 143] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00000 19KB refills: 7 waste 15.3% gc: 2064B slow: 152B fast: 0B TLAB: gc thread: 0x00007f15906a5000 [id: 142] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00000 19KB refills: 7 waste 15.3% gc: 2064B slow: 152B fast: 0B TLAB: gc thread: 0x00007f15906a3800 [id: 141] desired_size: 2KB slow allocs: 2 refill waste: 32B alloc: 0.00002 67KB refills: 7 waste 15.5% gc: 2064B slow: 176B fast: 0B TLAB: gc thread: 0x00007f15906a1800 [id: 140] desired_size: 858KB slow allocs: 0 refill waste: 13736B alloc: 0.01026 42933KB refills: 42 waste 1.5% gc: 538704B slow: 2328B fast: 0B TLAB: gc thread: 0x00007f15906a0000 [id: 139] desired_size: 2KB slow allocs: 2 refill waste: 32B alloc: 0.00000 19KB refills: 7 waste 15.5% gc: 2064B slow: 176B fast: 0B TLAB: gc thread: 0x00007f159069e800 [id: 138] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00000 19KB refills: 7 waste 15.3% gc: 2064B slow: 152B fast: 0B TLAB: gc thread: 0x00007f159069c800 [id: 137] desired_size: 2KB slow allocs: 2 refill waste: 32B alloc: 0.00000 19KB refills: 7 waste 15.5% gc: 2064B slow: 176B fast: 0B TLAB: gc thread: 0x00007f159069b000 [id: 136] desired_size: 2KB slow allocs: 11 refill waste: 32B alloc: 0.00002 99KB refills: 54 waste 2.1% gc: 704B slow: 1688B fast: 0B TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 840KB slow allocs: 0 refill waste: 13448B alloc: 0.01004 42032KB refills: 45 waste 2.0% gc: 758712B slow: 3304B fast: 0B TLAB: gc thread: 0x00007f1590697800 [id: 134] desired_size: 2KB slow allocs: 2 refill waste: 32B alloc: 0.00000 19KB refills: 7 waste 15.5% gc: 2064B slow: 176B fast: 0B TLAB: gc thread: 0x00007f1590695800 [id: 133] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00000 19KB refills: 7 waste 15.3% gc: 2064B slow: 152B fast: 0B TLAB: gc thread: 0x00007f1590693800 [id: 132] desired_size: 2KB slow allocs: 2 refill waste: 32B alloc: 0.00000 19KB refills: 7 waste 15.5% gc: 2064B slow: 176B fast: 0B TLAB: gc thread: 0x00007f1590692000 [id: 131] desired_size: 2KB slow allocs: 7 refill waste: 32B alloc: 0.00002 63KB refills: 28 waste 4.4% gc: 1752B slow: 808B fast: 0B TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 860KB slow allocs: 1 refill waste: 13760B alloc: 0.01028 43013KB refills: 49 waste 1.6% gc: 682184B slow: 18672B fast: 0B TLAB: gc thread: 0x00007f159068e800 [id: 129] desired_size: 2KB slow allocs: 2 refill waste: 32B alloc: 0.00000 19KB refills: 7 waste 15.5% gc: 2064B slow: 176B fast: 0B TLAB: gc thread: 0x00007f159068d000 [id: 128] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00000 19KB refills: 7 waste 15.3% gc: 2064B slow: 152B fast: 0B TLAB: gc thread: 0x00007f159068b000 [id: 127] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00000 19KB refills: 7 waste 15.3% gc: 2064B slow: 152B fast: 0B TLAB: gc thread: 0x00007f1590689800 [id: 126] desired_size: 2KB slow allocs: 1 refill waste: 40B alloc: 0.00003 140KB refills: 8 waste 6.8% gc: 1360B slow: 208B fast: 0B TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 875KB slow allocs: 0 refill waste: 14000B alloc: 0.01046 43769KB refills: 49 waste 1.4% gc: 614224B slow: 1680B fast: 0B TLAB: gc thread: 0x00007f1590674800 [id: 124] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00000 19KB refills: 7 waste 14.3% gc: 1888B slow: 176B fast: 0B TLAB: gc thread: 0x00007f1590673000 [id: 123] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00000 19KB refills: 7 waste 15.3% gc: 2064B slow: 152B fast: 0B TLAB: gc thread: 0x00007f1590671800 [id: 122] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00000 19KB refills: 7 waste 15.3% gc: 2064B slow: 152B fast: 0B TLAB: gc thread: 0x00007f159067d800 [id: 121] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00003 111KB refills: 6 waste 3.7% gc: 376B slow: 128B fast: 0B TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 852KB slow allocs: 1 refill waste: 13640B alloc: 0.01018 42632KB refills: 44 waste 0.4% gc: 143120B slow: 9160B fast: 0B TLAB: gc thread: 0x00007f1590637000 [id: 117] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 19KB refills: 6 waste 2.3% gc: 136B slow: 144B fast: 0B TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 109KB slow allocs: 0 refill waste: 1744B alloc: 0.00131 5470KB refills: 53 waste 1.9% gc: 111032B slow: 2056B fast: 0B TLAB: gc thread: 0x00007f159062b000 [id: 115] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 4KB refills: 2 waste 45.7% gc: 1872B slow: 16B fast: 0B TLAB: gc thread: 0x00007f159005b000 [id: 114] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 1KB refills: 1 waste 68.6% gc: 1416B slow: 0B fast: 0B TLAB: gc thread: 0x00007f158c010800 [id: 113] desired_size: 36KB slow allocs: 26 refill waste: 584B alloc: 0.00044 1829KB refills: 21 waste 3.0% gc: 21936B slow: 1968B fast: 0B TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.30200 1264205KB refills: 1606 waste 0.0% gc: 143712B slow: 92912B fast: 0B TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.07563 316602KB refills: 169 waste 0.4% gc: 1284112B slow: 13088B fast: 0B TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 2047KB slow allocs: 11 refill waste: 32760B alloc: 0.12518 524007KB refills: 226 waste 0.2% gc: 799824B slow: 362304B fast: 0B TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 435KB slow allocs: 0 refill waste: 6968B alloc: 0.00520 21777KB refills: 46 waste 1.6% gc: 321056B slow: 4136B fast: 0B TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 509KB slow allocs: 0 refill waste: 8144B alloc: 0.00609 25475KB refills: 51 waste 1.9% gc: 487952B slow: 6592B fast: 0B TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 2047KB slow allocs: 83 refill waste: 32760B alloc: 0.40043 1676245KB refills: 741 waste 0.1% gc: 869336B slow: 1203552B fast: 0B TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 9KB refills: 3 waste 8.3% gc: 432B slow: 80B fast: 0B TLAB: gc thread: 0x00007f170d362000 [id: 86] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 0KB refills: 1 waste 96.9% gc: 2000B slow: 0B fast: 0B TLAB: gc thread: 0x00007f170d35a000 [id: 85] desired_size: 139KB slow allocs: 0 refill waste: 2224B alloc: 0.00166 6968KB refills: 38 waste 1.3% gc: 67248B slow: 2624B fast: 0B TLAB: gc thread: 0x00007f170d2cf000 [id: 84] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00000 12KB refills: 4 waste 11.2% gc: 840B slow: 88B fast: 0B TLAB: gc thread: 0x00007f170cda3800 [id: 63] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 0KB refills: 1 waste 91.5% gc: 1888B slow: 0B fast: 0B TLAB totals: thrds: 45 refills: 3557 max: 1606 slow allocs: 165 max 83 waste: 0.1% gc: 6971296B max: 1284112B slow: 1753864B max: 1203552B fast: 0B max: 0B Desired survivor size 268435456 bytes, new threshold 15 (max 15) - age 1: 4323832 bytes, 4323832 total - age 2: 14160 bytes, 4337992 total - age 3: 10912 bytes, 4348904 total - age 5: 176 bytes, 4349080 total - age 6: 144 bytes, 4349224 total - age 7: 144 bytes, 4349368 total - age 8: 5744 bytes, 4355112 total - age 9: 72 bytes, 4355184 total - age 10: 832 bytes, 4356016 total - age 11: 48 bytes, 4356064 total - age 12: 48 bytes, 4356112 total - age 14: 80 bytes, 4356192 total - age 15: 656 bytes, 4356848 total 64241.817: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 3714, predicted base time: 11.21 ms, remaining time: 138.79 ms, target pause time: 150.00 ms] 64241.817: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 1022 regions, survivors: 2 regions, predicted young region time: 2.25 ms] 64241.817: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 1022 regions, survivors: 2 regions, old: 0 regions, predicted pause time: 13.46 ms, target pause time: 150.00 ms] 2017-01-25T13:06:37.367+0530: 64241.820: [SoftReference, 0 refs, 0.0058087 secs]2017-01-25T13:06:37.373+0530: 64241.826: [WeakReference, 1 refs, 0.0028315 secs]2017-01-25T13:06:37.376+0530: 64241.828: [FinalReference, 36 refs, 0.0008956 secs]2017-01-25T13:06:37.377+0530: 64241.829: [PhantomReference, 0 refs, 0 refs, 0.0108982 secs]2017-01-25T13:06:37.388+0530: 64241.840: [JNI Weak Reference, 0.0000261 secs], 0.0263600 secs] [Parallel Time: 2.1 ms, GC Workers: 20] [GC Worker Start (ms): Min: 64241817.2, Avg: 64241817.3, Max: 64241817.5, Diff: 0.3] [Ext Root Scanning (ms): Min: 0.3, Avg: 1.0, Max: 1.8, Diff: 1.5, Sum: 19.4] [Update RS (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.3, Sum: 1.7] [Processed Buffers: Min: 0, Avg: 2.8, Max: 9, Diff: 9, Sum: 56] [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 1.7] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 0.0, Avg: 0.3, Max: 0.5, Diff: 0.5, Sum: 5.6] [Termination (ms): Min: 0.0, Avg: 0.3, Max: 0.4, Diff: 0.4, Sum: 6.0] [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 20] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 1.0] [GC Worker Total (ms): Min: 1.6, Avg: 1.8, Max: 1.9, Diff: 0.3, Sum: 35.4] [GC Worker End (ms): Min: 64241819.1, Avg: 64241819.1, Max: 64241819.2, Diff: 0.1] [Code Root Fixup: 0.1 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.6 ms] [Other: 23.7 ms] [Choose CSet: 0.0 ms] [Ref Proc: 21.0 ms] [Ref Enq: 0.3 ms] [Redirty Cards: 0.2 ms] [Humongous Register: 0.1 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 1.6 ms] [Eden: 4088.0M(4088.0M)->0.0B(4088.0M) Survivors: 8192.0K->8192.0K Heap: 4178.7M(8192.0M)->90.6M(8192.0M)] [Times: user=0.30 sys=0.00, real=0.03 secs] 2017-01-25T13:06:37.391+0530: 64241.843: Total time for which application threads were stopped: 0.0267672 seconds, Stopping threads took: 0.0000448 seconds 2017-01-25T13:06:37.391+0530: 64241.843: Application time: 0.0000947 seconds 2017-01-25T13:06:37.391+0530: 64241.844: Total time for which application threads were stopped: 0.0002803 seconds, Stopping threads took: 0.0000729 seconds 2017-01-25T13:06:37.391+0530: 64241.844: Application time: 0.0000418 seconds 2017-01-25T13:06:37.391+0530: 64241.844: Total time for which application threads were stopped: 0.0001173 seconds, Stopping threads took: 0.0000205 seconds 2017-01-25T13:06:37.391+0530: 64241.844: Application time: 0.0000254 seconds 2017-01-25T13:06:37.391+0530: 64241.844: Total time for which application threads were stopped: 0.0001434 seconds, Stopping threads took: 0.0000479 seconds 2017-01-25T13:06:37.391+0530: 64241.844: Application time: 0.0000188 seconds 2017-01-25T13:06:37.392+0530: 64241.844: Total time for which application threads were stopped: 0.0001591 seconds, Stopping threads took: 0.0000433 seconds 2017-01-25T13:06:37.392+0530: 64241.844: Application time: 0.0000286 seconds 2017-01-25T13:06:37.392+0530: 64241.844: Total time for which application threads were stopped: 0.0001666 seconds, Stopping threads took: 0.0000486 seconds 2017-01-25T13:06:37.392+0530: 64241.844: Application time: 0.0000257 seconds 2017-01-25T13:06:37.392+0530: 64241.845: Total time for which application threads were stopped: 0.0001308 seconds, Stopping threads took: 0.0000415 seconds 2017-01-25T13:06:37.392+0530: 64241.845: Application time: 0.0000210 seconds 2017-01-25T13:06:37.392+0530: 64241.845: Total time for which application threads were stopped: 0.0001181 seconds, Stopping threads took: 0.0000279 seconds 2017-01-25T13:06:37.392+0530: 64241.845: Application time: 0.0000209 seconds 2017-01-25T13:06:37.392+0530: 64241.845: Total time for which application threads were stopped: 0.0001475 seconds, Stopping threads took: 0.0000510 seconds 2017-01-25T13:06:40.571+0530: 64245.024: Application time: 3.1790511 seconds 2017-01-25T13:06:40.572+0530: 64245.024: Total time for which application threads were stopped: 0.0002598 seconds, Stopping threads took: 0.0000431 seconds 2017-01-25T13:06:45.572+0530: 64250.024: Application time: 5.0001376 seconds 2017-01-25T13:06:45.572+0530: 64250.025: Total time for which application threads were stopped: 0.0002687 seconds, Stopping threads took: 0.0000431 seconds 2017-01-25T13:06:50.572+0530: 64255.025: Application time: 5.0001946 seconds 2017-01-25T13:06:50.573+0530: 64255.025: Total time for which application threads were stopped: 0.0002942 seconds, Stopping threads took: 0.0000634 seconds 2017-01-25T13:06:55.573+0530: 64260.025: Application time: 5.0001895 seconds 2017-01-25T13:06:55.573+0530: 64260.026: Total time for which application threads were stopped: 0.0002684 seconds, Stopping threads took: 0.0000407 seconds 2017-01-25T13:07:00.573+0530: 64265.026: Application time: 5.0002077 seconds 2017-01-25T13:07:00.573+0530: 64265.026: Total time for which application threads were stopped: 0.0002506 seconds, Stopping threads took: 0.0000418 seconds 2017-01-25T13:07:05.574+0530: 64270.026: Application time: 5.0002046 seconds 2017-01-25T13:07:05.574+0530: 64270.026: Total time for which application threads were stopped: 0.0002517 seconds, Stopping threads took: 0.0000421 seconds 2017-01-25T13:07:10.574+0530: 64275.027: Application time: 5.0001865 seconds 2017-01-25T13:07:10.574+0530: 64275.027: Total time for which application threads were stopped: 0.0002522 seconds, Stopping threads took: 0.0000474 seconds 2017-01-25T13:07:15.575+0530: 64280.027: Application time: 5.0002286 seconds 2017-01-25T13:07:15.575+0530: 64280.027: Total time for which application threads were stopped: 0.0002538 seconds, Stopping threads took: 0.0000479 seconds 2017-01-25T13:07:20.575+0530: 64285.028: Application time: 5.0001932 seconds 2017-01-25T13:07:20.575+0530: 64285.028: Total time for which application threads were stopped: 0.0002462 seconds, Stopping threads took: 0.0000431 seconds 2017-01-25T13:07:25.575+0530: 64290.028: Application time: 5.0001952 seconds 2017-01-25T13:07:25.576+0530: 64290.028: Total time for which application threads were stopped: 0.0002328 seconds, Stopping threads took: 0.0000391 seconds 2017-01-25T13:07:30.576+0530: 64295.029: Application time: 5.0002049 seconds 2017-01-25T13:07:30.576+0530: 64295.029: Total time for which application threads were stopped: 0.0002359 seconds, Stopping threads took: 0.0000408 seconds 2017-01-25T13:07:35.576+0530: 64300.029: Application time: 5.0002040 seconds 2017-01-25T13:07:35.577+0530: 64300.029: Total time for which application threads were stopped: 0.0002753 seconds, Stopping threads took: 0.0000473 seconds 2017-01-25T13:07:40.577+0530: 64305.029: Application time: 5.0002046 seconds 2017-01-25T13:07:40.577+0530: 64305.030: Total time for which application threads were stopped: 0.0002560 seconds, Stopping threads took: 0.0000422 seconds 2017-01-25T13:07:45.577+0530: 64310.030: Application time: 5.0002068 seconds 2017-01-25T13:07:45.578+0530: 64310.030: Total time for which application threads were stopped: 0.0002618 seconds, Stopping threads took: 0.0000518 seconds 2017-01-25T13:07:50.578+0530: 64315.030: Application time: 5.0001895 seconds 2017-01-25T13:07:50.578+0530: 64315.031: Total time for which application threads were stopped: 0.0003045 seconds, Stopping threads took: 0.0000512 seconds 2017-01-25T13:07:55.578+0530: 64320.031: Application time: 5.0002394 seconds 2017-01-25T13:07:55.579+0530: 64320.031: Total time for which application threads were stopped: 0.0002506 seconds, Stopping threads took: 0.0000395 seconds 2017-01-25T13:08:00.579+0530: 64325.031: Application time: 5.0002288 seconds 2017-01-25T13:08:00.579+0530: 64325.032: Total time for which application threads were stopped: 0.0002531 seconds, Stopping threads took: 0.0000406 seconds 2017-01-25T13:08:05.579+0530: 64330.032: Application time: 5.0001996 seconds 2017-01-25T13:08:05.580+0530: 64330.032: Total time for which application threads were stopped: 0.0002486 seconds, Stopping threads took: 0.0000420 seconds 2017-01-25T13:08:10.580+0530: 64335.032: Application time: 5.0001750 seconds 2017-01-25T13:08:10.580+0530: 64335.033: Total time for which application threads were stopped: 0.0002557 seconds, Stopping threads took: 0.0000488 seconds 2017-01-25T13:08:15.580+0530: 64340.033: Application time: 5.0001817 seconds 2017-01-25T13:08:15.580+0530: 64340.033: Total time for which application threads were stopped: 0.0002597 seconds, Stopping threads took: 0.0000493 seconds 2017-01-25T13:08:20.581+0530: 64345.033: Application time: 5.0001852 seconds 2017-01-25T13:08:20.581+0530: 64345.033: Total time for which application threads were stopped: 0.0002825 seconds, Stopping threads took: 0.0000627 seconds 2017-01-25T13:08:25.581+0530: 64350.034: Application time: 5.0002295 seconds 2017-01-25T13:08:25.581+0530: 64350.034: Total time for which application threads were stopped: 0.0002499 seconds, Stopping threads took: 0.0000403 seconds 2017-01-25T13:08:30.582+0530: 64355.034: Application time: 5.0002247 seconds 2017-01-25T13:08:30.582+0530: 64355.034: Total time for which application threads were stopped: 0.0002803 seconds, Stopping threads took: 0.0000593 seconds 2017-01-25T13:08:35.582+0530: 64360.035: Application time: 5.0001862 seconds 2017-01-25T13:08:35.582+0530: 64360.035: Total time for which application threads were stopped: 0.0002972 seconds, Stopping threads took: 0.0000699 seconds 2017-01-25T13:08:40.133+0530: 64364.586: Application time: 4.5507335 seconds 2017-01-25T13:08:40.134+0530: 64364.586: Total time for which application threads were stopped: 0.0006192 seconds, Stopping threads took: 0.0000411 seconds 2017-01-25T13:08:40.583+0530: 64365.035: Application time: 0.4488614 seconds 2017-01-25T13:08:40.583+0530: 64365.035: Total time for which application threads were stopped: 0.0002583 seconds, Stopping threads took: 0.0000548 seconds 2017-01-25T13:08:45.583+0530: 64370.036: Application time: 5.0002011 seconds 2017-01-25T13:08:45.583+0530: 64370.036: Total time for which application threads were stopped: 0.0002594 seconds, Stopping threads took: 0.0000424 seconds 2017-01-25T13:08:50.583+0530: 64375.036: Application time: 5.0002066 seconds 2017-01-25T13:08:50.584+0530: 64375.036: Total time for which application threads were stopped: 0.0002680 seconds, Stopping threads took: 0.0000414 seconds 2017-01-25T13:08:55.584+0530: 64380.037: Application time: 5.0002392 seconds 2017-01-25T13:08:55.584+0530: 64380.037: Total time for which application threads were stopped: 0.0002646 seconds, Stopping threads took: 0.0000370 seconds 2017-01-25T13:09:00.584+0530: 64385.037: Application time: 5.0002253 seconds 2017-01-25T13:09:00.585+0530: 64385.037: Total time for which application threads were stopped: 0.0002727 seconds, Stopping threads took: 0.0000393 seconds 2017-01-25T13:09:05.585+0530: 64390.038: Application time: 5.0002276 seconds 2017-01-25T13:09:05.585+0530: 64390.038: Total time for which application threads were stopped: 0.0002608 seconds, Stopping threads took: 0.0000403 seconds 2017-01-25T13:09:10.585+0530: 64395.038: Application time: 5.0002179 seconds 2017-01-25T13:09:10.586+0530: 64395.038: Total time for which application threads were stopped: 0.0002685 seconds, Stopping threads took: 0.0000433 seconds 2017-01-25T13:09:15.586+0530: 64400.039: Application time: 5.0001999 seconds 2017-01-25T13:09:15.586+0530: 64400.039: Total time for which application threads were stopped: 0.0002479 seconds, Stopping threads took: 0.0000470 seconds 2017-01-25T13:09:20.586+0530: 64405.039: Application time: 5.0000955 seconds 2017-01-25T13:09:20.587+0530: 64405.039: Total time for which application threads were stopped: 0.0003038 seconds, Stopping threads took: 0.0000829 seconds 2017-01-25T13:09:25.587+0530: 64410.039: Application time: 5.0002022 seconds 2017-01-25T13:09:25.587+0530: 64410.040: Total time for which application threads were stopped: 0.0002422 seconds, Stopping threads took: 0.0000428 seconds 2017-01-25T13:09:30.587+0530: 64415.040: Application time: 5.0002011 seconds 2017-01-25T13:09:30.587+0530: 64415.040: Total time for which application threads were stopped: 0.0002440 seconds, Stopping threads took: 0.0000404 seconds 2017-01-25T13:09:35.588+0530: 64420.040: Application time: 5.0001995 seconds 2017-01-25T13:09:35.588+0530: 64420.041: Total time for which application threads were stopped: 0.0002617 seconds, Stopping threads took: 0.0000409 seconds 2017-01-25T13:09:40.588+0530: 64425.041: Application time: 5.0001848 seconds 2017-01-25T13:09:40.588+0530: 64425.041: Total time for which application threads were stopped: 0.0002620 seconds, Stopping threads took: 0.0000436 seconds 2017-01-25T13:09:42.834+0530: 64427.286: Application time: 2.2451421 seconds 2017-01-25T13:09:42.834+0530: 64427.286: [GC pause (G1 Evacuation Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] desired_size: 166KB slow allocs: 0 refill waste: 2664B alloc: 0.00199 8330KB refills: 77 waste 1.4% gc: 168776B slow: 14496B fast: 0B TLAB: gc thread: 0x00007f15f8114800 [id: 199] desired_size: 1999KB slow allocs: 1 refill waste: 31984B alloc: 0.02388 99965KB refills: 127 waste 0.2% gc: 484968B slow: 15936B fast: 0B TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 5KB slow allocs: 0 refill waste: 88B alloc: 0.00007 286KB refills: 71 waste 1.8% gc: 3464B slow: 4176B fast: 0B TLAB: gc thread: 0x00007f15906a8800 [id: 144] desired_size: 2KB slow allocs: 2 refill waste: 32B alloc: 0.00000 17KB refills: 10 waste 11.2% gc: 2064B slow: 248B fast: 0B TLAB: gc thread: 0x00007f15906a7000 [id: 143] desired_size: 2KB slow allocs: 3 refill waste: 32B alloc: 0.00000 17KB refills: 10 waste 11.3% gc: 2064B slow: 272B fast: 0B TLAB: gc thread: 0x00007f15906a5000 [id: 142] desired_size: 2KB slow allocs: 2 refill waste: 32B alloc: 0.00000 17KB refills: 10 waste 10.3% gc: 1888B slow: 248B fast: 0B TLAB: gc thread: 0x00007f15906a3800 [id: 141] desired_size: 2KB slow allocs: 42 refill waste: 32B alloc: 0.00001 48KB refills: 132 waste 2.0% gc: 1104B slow: 4328B fast: 0B TLAB: gc thread: 0x00007f15906a1800 [id: 140] desired_size: 750KB slow allocs: 1 refill waste: 12008B alloc: 0.00897 37540KB refills: 88 waste 0.9% gc: 625568B slow: 9488B fast: 0B TLAB: gc thread: 0x00007f15906a0000 [id: 139] desired_size: 2KB slow allocs: 3 refill waste: 32B alloc: 0.00000 17KB refills: 10 waste 11.3% gc: 2064B slow: 272B fast: 0B TLAB: gc thread: 0x00007f159069e800 [id: 138] desired_size: 2KB slow allocs: 2 refill waste: 32B alloc: 0.00000 17KB refills: 10 waste 11.2% gc: 2064B slow: 248B fast: 0B TLAB: gc thread: 0x00007f159069c800 [id: 137] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 17KB refills: 10 waste 10.3% gc: 1888B slow: 240B fast: 0B TLAB: gc thread: 0x00007f159069b000 [id: 136] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00003 105KB refills: 9 waste 3.8% gc: 488B slow: 240B fast: 0B TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 731KB slow allocs: 0 refill waste: 11696B alloc: 0.00873 36559KB refills: 88 waste 1.0% gc: 658072B slow: 5032B fast: 0B TLAB: gc thread: 0x00007f1590697800 [id: 134] desired_size: 2KB slow allocs: 2 refill waste: 32B alloc: 0.00000 17KB refills: 10 waste 11.2% gc: 2064B slow: 248B fast: 0B TLAB: gc thread: 0x00007f1590695800 [id: 133] desired_size: 2KB slow allocs: 3 refill waste: 32B alloc: 0.00000 17KB refills: 10 waste 11.3% gc: 2064B slow: 272B fast: 0B TLAB: gc thread: 0x00007f1590693800 [id: 132] desired_size: 2KB slow allocs: 2 refill waste: 32B alloc: 0.00000 17KB refills: 10 waste 10.7% gc: 1952B slow: 264B fast: 0B TLAB: gc thread: 0x00007f1590692000 [id: 131] desired_size: 2KB slow allocs: 63 refill waste: 32B alloc: 0.00001 61KB refills: 258 waste 1.8% gc: 1832B slow: 7936B fast: 0B TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 776KB slow allocs: 0 refill waste: 12416B alloc: 0.00927 38822KB refills: 84 waste 0.2% gc: 97352B slow: 8736B fast: 0B TLAB: gc thread: 0x00007f159068e800 [id: 129] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 17KB refills: 10 waste 10.1% gc: 1832B slow: 256B fast: 0B TLAB: gc thread: 0x00007f159068d000 [id: 128] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00000 17KB refills: 10 waste 10.6% gc: 1952B slow: 240B fast: 0B TLAB: gc thread: 0x00007f159068b000 [id: 127] desired_size: 2KB slow allocs: 2 refill waste: 32B alloc: 0.00000 17KB refills: 10 waste 11.2% gc: 2064B slow: 248B fast: 0B TLAB: gc thread: 0x00007f1590689800 [id: 126] desired_size: 2KB slow allocs: 2 refill waste: 32B alloc: 0.00002 98KB refills: 10 waste 11.2% gc: 2064B slow: 248B fast: 0B TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 775KB slow allocs: 0 refill waste: 12400B alloc: 0.00926 38755KB refills: 72 waste 0.1% gc: 36224B slow: 7680B fast: 0B TLAB: gc thread: 0x00007f1590674800 [id: 124] desired_size: 2KB slow allocs: 2 refill waste: 32B alloc: 0.00000 17KB refills: 10 waste 10.5% gc: 1888B slow: 272B fast: 0B TLAB: gc thread: 0x00007f1590673000 [id: 123] desired_size: 2KB slow allocs: 2 refill waste: 32B alloc: 0.00000 17KB refills: 10 waste 11.2% gc: 2064B slow: 248B fast: 0B TLAB: gc thread: 0x00007f1590671800 [id: 122] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00000 17KB refills: 10 waste 10.4% gc: 1888B slow: 264B fast: 0B TLAB: gc thread: 0x00007f159067d800 [id: 121] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00002 76KB refills: 10 waste 10.6% gc: 1952B slow: 240B fast: 0B TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 750KB slow allocs: 0 refill waste: 12008B alloc: 0.00897 37549KB refills: 79 waste 0.7% gc: 403168B slow: 11312B fast: 0B TLAB: gc thread: 0x00007f1590637000 [id: 117] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00000 16KB refills: 9 waste 3.3% gc: 376B slow: 232B fast: 0B TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 97KB slow allocs: 0 refill waste: 1560B alloc: 0.00117 4888KB refills: 82 waste 0.3% gc: 18624B slow: 2992B fast: 0B TLAB: gc thread: 0x00007f159062b000 [id: 115] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 3KB refills: 2 waste 12.0% gc: 480B slow: 16B fast: 0B TLAB: gc thread: 0x00007f159005b000 [id: 114] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 1KB refills: 1 waste 51.6% gc: 1064B slow: 0B fast: 0B TLAB: gc thread: 0x00007f158c010800 [id: 113] desired_size: 31KB slow allocs: 44 refill waste: 504B alloc: 0.00038 1575KB refills: 34 waste 1.9% gc: 18984B slow: 2256B fast: 0B TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.38170 1597857KB refills: 774 waste 0.1% gc: 1492120B slow: 81320B fast: 0B TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.06731 281777KB refills: 249 waste 0.3% gc: 1799984B slow: 18136B fast: 0B TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 2047KB slow allocs: 35 refill waste: 32760B alloc: 0.11055 462756KB refills: 376 waste 0.3% gc: 1727576B slow: 467192B fast: 0B TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 388KB slow allocs: 0 refill waste: 6216B alloc: 0.00464 19425KB refills: 81 waste 1.0% gc: 324512B slow: 7440B fast: 0B TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 451KB slow allocs: 0 refill waste: 7216B alloc: 0.00539 22558KB refills: 80 waste 0.9% gc: 320392B slow: 6880B fast: 0B TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 2047KB slow allocs: 131 refill waste: 32760B alloc: 0.35391 1481518KB refills: 1175 waste 0.1% gc: 0B slow: 2219392B fast: 0B TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 7KB refills: 5 waste 14.6% gc: 1344B slow: 160B fast: 0B TLAB: gc thread: 0x00007f170d35a000 [id: 85] desired_size: 122KB slow allocs: 0 refill waste: 1960B alloc: 0.00147 6144KB refills: 63 waste 0.7% gc: 51592B slow: 5096B fast: 0B TLAB: gc thread: 0x00007f170d2cf000 [id: 84] desired_size: 2KB slow allocs: 3 refill waste: 32B alloc: 0.00000 11KB refills: 6 waste 7.9% gc: 840B slow: 144B fast: 0B TLAB: gc thread: 0x00007f170cda3800 [id: 63] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 0KB refills: 1 waste 91.5% gc: 1888B slow: 0B fast: 0B TLAB totals: thrds: 43 refills: 4193 max: 1175 slow allocs: 351 max 131 waste: 0.2% gc: 8274608B max: 1799984B slow: 2904944B max: 2219392B fast: 0B max: 0B Desired survivor size 268435456 bytes, new threshold 15 (max 15) - age 1: 1828584 bytes, 1828584 total - age 2: 32728 bytes, 1861312 total - age 3: 3152 bytes, 1864464 total - age 4: 10128 bytes, 1874592 total - age 6: 176 bytes, 1874768 total - age 7: 144 bytes, 1874912 total - age 8: 144 bytes, 1875056 total - age 9: 5744 bytes, 1880800 total - age 10: 72 bytes, 1880872 total - age 11: 832 bytes, 1881704 total - age 12: 48 bytes, 1881752 total - age 13: 48 bytes, 1881800 total - age 15: 80 bytes, 1881880 total 64427.287: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 3542, predicted base time: 17.08 ms, remaining time: 132.92 ms, target pause time: 150.00 ms] 64427.287: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 1022 regions, survivors: 2 regions, predicted young region time: 2.13 ms] 64427.287: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 1022 regions, survivors: 2 regions, old: 0 regions, predicted pause time: 19.21 ms, target pause time: 150.00 ms] 2017-01-25T13:09:42.839+0530: 64427.292: [SoftReference, 0 refs, 0.0023408 secs]2017-01-25T13:09:42.841+0530: 64427.294: [WeakReference, 1 refs, 0.0008582 secs]2017-01-25T13:09:42.842+0530: 64427.295: [FinalReference, 67 refs, 0.0009552 secs]2017-01-25T13:09:42.843+0530: 64427.296: [PhantomReference, 0 refs, 0 refs, 0.0013949 secs]2017-01-25T13:09:42.845+0530: 64427.297: [JNI Weak Reference, 0.0000180 secs], 0.0139572 secs] [Parallel Time: 1.9 ms, GC Workers: 20] [GC Worker Start (ms): Min: 64427286.7, Avg: 64427286.9, Max: 64427288.1, Diff: 1.3] [Ext Root Scanning (ms): Min: 0.0, Avg: 0.7, Max: 1.5, Diff: 1.5, Sum: 14.9] [Update RS (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.3, Sum: 1.9] [Processed Buffers: Min: 0, Avg: 2.6, Max: 8, Diff: 8, Sum: 53] [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.3] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 0.0, Avg: 0.2, Max: 0.3, Diff: 0.3, Sum: 3.7] [Termination (ms): Min: 0.0, Avg: 0.3, Max: 0.4, Diff: 0.4, Sum: 6.1] [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 20] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 1.2] [GC Worker Total (ms): Min: 0.3, Avg: 1.5, Max: 1.7, Diff: 1.4, Sum: 30.1] [GC Worker End (ms): Min: 64427288.4, Avg: 64427288.4, Max: 64427288.5, Diff: 0.1] [Code Root Fixup: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.8 ms] [Other: 11.3 ms] [Choose CSet: 0.0 ms] [Ref Proc: 8.6 ms] [Ref Enq: 0.3 ms] [Redirty Cards: 0.2 ms] [Humongous Register: 0.1 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 1.6 ms] [Eden: 4088.0M(4088.0M)->0.0B(4092.0M) Survivors: 8192.0K->4096.0K Heap: 4178.6M(8192.0M)->87.1M(8192.0M)] [Times: user=0.10 sys=0.01, real=0.01 secs] *The GC with issue:* 2017-01-25T14:10:05.908+0530: 68050.361: [GC pause (G1 Evacuation Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] desired_size: 177KB slow allocs: 0 refill waste: 2840B alloc: 0.00212 8881KB refills: 70 waste 0.8% gc: 89264B slow: 10480B fast: 0B TLAB: gc thread: 0x00007f15f8114800 [id: 199] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.02722 113928KB refills: 83 waste 0.3% gc: 587872B slow: 3352B fast: 0B TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 6KB slow allocs: 0 refill waste: 96B alloc: 0.00007 304KB refills: 66 waste 0.9% gc: 1376B slow: 2384B fast: 0B TLAB: gc thread: 0x00007f15906a8800 [id: 144] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00000 18KB refills: 11 waste 5.0% gc: 880B slow: 264B fast: 0B TLAB: gc thread: 0x00007f15906a7000 [id: 143] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00000 18KB refills: 11 waste 5.0% gc: 880B slow: 264B fast: 0B TLAB: gc thread: 0x00007f15906a5000 [id: 142] desired_size: 2KB slow allocs: 3 refill waste: 32B alloc: 0.00000 18KB refills: 11 waste 5.5% gc: 928B slow: 320B fast: 0B TLAB: gc thread: 0x00007f15906a3800 [id: 141] desired_size: 3KB slow allocs: 1 refill waste: 48B alloc: 0.00004 172KB refills: 17 waste 6.6% gc: 3408B slow: 552B fast: 0B TLAB: gc thread: 0x00007f15906a1800 [id: 140] desired_size: 865KB slow allocs: 0 refill waste: 13840B alloc: 0.01033 43261KB refills: 72 waste 0.8% gc: 523960B slow: 3008B fast: 0B TLAB: gc thread: 0x00007f15906a0000 [id: 139] desired_size: 2KB slow allocs: 2 refill waste: 32B alloc: 0.00000 18KB refills: 11 waste 5.7% gc: 1008B slow: 296B fast: 0B TLAB: gc thread: 0x00007f159069e800 [id: 138] desired_size: 2KB slow allocs: 2 refill waste: 32B alloc: 0.00000 18KB refills: 11 waste 5.7% gc: 1008B slow: 296B fast: 0B TLAB: gc thread: 0x00007f159069c800 [id: 137] desired_size: 2KB slow allocs: 3 refill waste: 32B alloc: 0.00000 18KB refills: 11 waste 5.8% gc: 1008B slow: 320B fast: 0B TLAB: gc thread: 0x00007f159069b000 [id: 136] desired_size: 2KB slow allocs: 14 refill waste: 32B alloc: 0.00001 48KB refills: 110 waste 2.0% gc: 1200B slow: 3424B fast: 0B TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 751KB slow allocs: 0 refill waste: 12024B alloc: 0.00898 37577KB refills: 75 waste 0.2% gc: 134904B slow: 6808B fast: 0B TLAB: gc thread: 0x00007f1590697800 [id: 134] desired_size: 2KB slow allocs: 2 refill waste: 32B alloc: 0.00000 18KB refills: 11 waste 5.7% gc: 1008B slow: 296B fast: 0B TLAB: gc thread: 0x00007f1590695800 [id: 133] desired_size: 2KB slow allocs: 2 refill waste: 32B alloc: 0.00000 18KB refills: 11 waste 5.1% gc: 880B slow: 288B fast: 0B TLAB: gc thread: 0x00007f1590693800 [id: 132] desired_size: 2KB slow allocs: 2 refill waste: 32B alloc: 0.00000 18KB refills: 11 waste 4.7% gc: 776B slow: 296B fast: 0B TLAB: gc thread: 0x00007f1590692000 [id: 131] desired_size: 2KB slow allocs: 3 refill waste: 32B alloc: 0.00002 89KB refills: 14 waste 5.9% gc: 1264B slow: 440B fast: 0B TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 873KB slow allocs: 0 refill waste: 13976B alloc: 0.01044 43692KB refills: 68 waste 0.3% gc: 192240B slow: 11640B fast: 0B TLAB: gc thread: 0x00007f159068e800 [id: 129] desired_size: 2KB slow allocs: 3 refill waste: 32B alloc: 0.00000 18KB refills: 11 waste 5.8% gc: 1008B slow: 320B fast: 0B TLAB: gc thread: 0x00007f159068d000 [id: 128] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 18KB refills: 11 waste 4.6% gc: 776B slow: 264B fast: 0B TLAB: gc thread: 0x00007f159068b000 [id: 127] desired_size: 2KB slow allocs: 3 refill waste: 32B alloc: 0.00000 18KB refills: 10 waste 11.3% gc: 2064B slow: 272B fast: 0B TLAB: gc thread: 0x00007f1590689800 [id: 126] desired_size: 2KB slow allocs: 5 refill waste: 32B alloc: 0.00003 120KB refills: 24 waste 5.2% gc: 2272B slow: 800B fast: 0B TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 890KB slow allocs: 0 refill waste: 14248B alloc: 0.01064 44534KB refills: 59 waste 0.8% gc: 409872B slow: 11088B fast: 0B TLAB: gc thread: 0x00007f1590674800 [id: 124] desired_size: 2KB slow allocs: 2 refill waste: 32B alloc: 0.00000 18KB refills: 11 waste 5.7% gc: 1008B slow: 296B fast: 0B TLAB: gc thread: 0x00007f1590673000 [id: 123] desired_size: 2KB slow allocs: 2 refill waste: 32B alloc: 0.00000 18KB refills: 11 waste 5.4% gc: 928B slow: 296B fast: 0B TLAB: gc thread: 0x00007f1590671800 [id: 122] desired_size: 2KB slow allocs: 2 refill waste: 32B alloc: 0.00000 18KB refills: 11 waste 5.7% gc: 1008B slow: 296B fast: 0B TLAB: gc thread: 0x00007f159067d800 [id: 121] desired_size: 2KB slow allocs: 19 refill waste: 32B alloc: 0.00002 77KB refills: 116 waste 2.0% gc: 1392B slow: 3312B fast: 0B TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 807KB slow allocs: 0 refill waste: 12920B alloc: 0.00965 40392KB refills: 79 waste 0.8% gc: 509408B slow: 10496B fast: 0B TLAB: gc thread: 0x00007f1590637000 [id: 117] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00000 17KB refills: 11 waste 7.3% gc: 1360B slow: 304B fast: 0B TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 109KB slow allocs: 0 refill waste: 1752B alloc: 0.00131 5491KB refills: 82 waste 0.9% gc: 78864B slow: 2976B fast: 0B TLAB: gc thread: 0x00007f159062b000 [id: 115] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 3KB refills: 2 waste 12.0% gc: 480B slow: 16B fast: 0B TLAB: gc thread: 0x00007f159005b000 [id: 114] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 1KB refills: 1 waste 47.3% gc: 976B slow: 0B fast: 0B TLAB: gc thread: 0x00007f158c010800 [id: 113] desired_size: 35KB slow allocs: 42 refill waste: 560B alloc: 0.00042 1764KB refills: 31 waste 2.2% gc: 23272B slow: 1416B fast: 0B TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.30730 1286404KB refills: 879 waste 0.0% gc: 431048B slow: 78920B fast: 0B TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.07518 314707KB refills: 253 waste 0.2% gc: 1172880B slow: 17344B fast: 0B TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 2047KB slow allocs: 20 refill waste: 32760B alloc: 0.12527 524398KB refills: 370 waste 0.2% gc: 943176B slow: 444384B fast: 0B TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 433KB slow allocs: 0 refill waste: 6936B alloc: 0.00518 21700KB refills: 78 waste 0.5% gc: 177608B slow: 7000B fast: 0B TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 510KB slow allocs: 0 refill waste: 8160B alloc: 0.00610 25523KB refills: 75 waste 0.4% gc: 163696B slow: 6840B fast: 0B TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 2047KB slow allocs: 66 refill waste: 32760B alloc: 0.39765 1664620KB refills: 1099 waste 0.1% gc: 0B slow: 1894184B fast: 0B TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 8KB refills: 5 waste 15.0% gc: 1376B slow: 176B fast: 0B TLAB: gc thread: 0x00007f170d35a000 [id: 85] desired_size: 123KB slow allocs: 0 refill waste: 1976B alloc: 0.00148 6195KB refills: 80 waste 0.2% gc: 19776B slow: 4952B fast: 0B TLAB: gc thread: 0x00007f170d2cf000 [id: 84] desired_size: 2KB slow allocs: 3 refill waste: 32B alloc: 0.00000 12KB refills: 7 waste 1.3% gc: 24B slow: 160B fast: 0B TLAB: gc thread: 0x00007f170cda3800 [id: 63] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 0KB refills: 1 waste 91.5% gc: 1888B slow: 0B fast: 0B TLAB totals: thrds: 43 refills: 3991 max: 1099 slow allocs: 204 max 66 waste: 0.1% gc: 5490024B max: 1172880B slow: 2530840B max: 1894184B fast: 0B max: 0B Desired survivor size 268435456 bytes, new threshold 15 (max 15) - age 1: 1736992 bytes, 1736992 total - age 2: 3936 bytes, 1740928 total - age 3: 416 bytes, 1741344 total - age 4: 168 bytes, 1741512 total - age 5: 200 bytes, 1741712 total - age 8: 416 bytes, 1742128 total - age 11: 1816 bytes, 1743944 total - age 12: 24 bytes, 1743968 total - age 14: 24 bytes, 1743992 total 68050.361: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 4790, predicted base time: 18.67 ms, remaining time: 131.33 ms, target pause time: 150.00 ms] 68050.361: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 1022 regions, survivors: 2 regions, predicted young region time: 2.16 ms] 68050.361: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 1022 regions, survivors: 2 regions, old: 0 regions, predicted pause time: 20.83 ms, target pause time: 150.00 ms] 2017-01-25T14:10:05.914+0530: 68050.366: [SoftReference, 0 refs, 0.0045583 secs]2017-01-25T14:10:05.918+0530: 68050.371: [WeakReference, 1 refs, 0.0011573 secs]2017-01-25T14:10:05.920+0530: 68050.372: [FinalReference, 59 refs, 0.0008955 secs]2017-01-25T14:10:05.920+0530: 68050.373: [PhantomReference, 0 refs, 0 refs, 0.0060485 secs]2017-01-25T14:10:05.927+0530: 68050.379: [JNI Weak Reference, 0.0000168 secs], 0.0213658 secs] [Parallel Time: 2.9 ms, GC Workers: 20] [GC Worker Start (ms): Min: 68050361.3, Avg: 68050361.5, Max: 68050361.6, Diff: 0.3] [Ext Root Scanning (ms): Min: 0.4, Avg: 0.5, Max: 1.0, Diff: 0.6, Sum: 10.9] [Update RS (ms): Min: 0.0, Avg: 0.3, Max: 1.8, Diff: 1.8, Sum: 7.0] [Processed Buffers: Min: 0, Avg: 2.9, Max: 8, Diff: 8, Sum: 58] [Scan RS (ms): Min: 0.0, Avg: 0.2, Max: 0.3, Diff: 0.2, Sum: 3.9] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.1] [Object Copy (ms): Min: 0.0, Avg: 0.7, Max: 0.8, Diff: 0.8, Sum: 13.3] [Termination (ms): Min: 0.0, Avg: 0.6, Max: 1.0, Diff: 1.0, Sum: 12.3] [Termination Attempts: Min: 1, Avg: 26.3, Max: 36, Diff: 35, Sum: 526] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 1.4] [GC Worker Total (ms): Min: 2.3, Avg: 2.4, Max: 2.7, Diff: 0.4, Sum: 48.8] [GC Worker End (ms): Min: 68050363.9, Avg: 68050363.9, Max: 68050364.0, Diff: 0.2] [Code Root Fixup: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.6 ms] [Other: 17.9 ms] [Choose CSet: 0.0 ms] [Ref Proc: 15.0 ms] [Ref Enq: 0.3 ms] [Redirty Cards: 0.2 ms] [Humongous Register: 0.0 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 1.9 ms] [Eden: 4088.0M(4088.0M)->0.0B(4088.0M) Survivors: 8192.0K->8192.0K Heap: 4177.2M(8192.0M)->88.5M(8192.0M)] [Times: user=0.24 sys=0.00, real=0.03 secs] 2017-01-25T14:10:05.930+0530: 68050.382: Total time for which application threads were stopped: 0.0217941 seconds, Stopping threads took: 0.0000560 seconds 2017-01-25T14:10:05.930+0530: 68050.383: Application time: 0.0000388 seconds 2017-01-25T14:10:05.930+0530: 68050.383: Total time for which application threads were stopped: 0.0002600 seconds, Stopping threads took: 0.0000840 seconds 2017-01-25T14:10:05.930+0530: 68050.383: Application time: 0.0000432 seconds 2017-01-25T14:10:05.930+0530: 68050.383: Total time for which application threads were stopped: 0.0001275 seconds, Stopping threads took: 0.0000411 seconds 2017-01-25T14:10:05.930+0530: 68050.383: Application time: 0.0000422 seconds 2017-01-25T14:10:05.931+0530: 68050.383: Total time for which application threads were stopped: 0.0001408 seconds, Stopping threads took: 0.0000537 seconds 2017-01-25T14:10:05.931+0530: 68050.383: Application time: 0.0000273 seconds 2017-01-25T14:10:05.931+0530: 68050.383: Total time for which application threads were stopped: 0.0001279 seconds, Stopping threads took: 0.0000303 seconds 2017-01-25T14:10:05.931+0530: 68050.383: Application time: 0.0000279 seconds 2017-01-25T14:10:05.931+0530: 68050.383: Total time for which application threads were stopped: 0.0001316 seconds, Stopping threads took: 0.0000435 seconds 2017-01-25T14:10:05.931+0530: 68050.383: Application time: 0.0000296 seconds 2017-01-25T14:10:05.931+0530: 68050.384: Total time for which application threads were stopped: 0.0001133 seconds, Stopping threads took: 0.0000231 seconds 2017-01-25T14:10:05.931+0530: 68050.384: Application time: 0.0000331 seconds 2017-01-25T14:10:05.931+0530: 68050.384: Total time for which application threads were stopped: 0.0001241 seconds, Stopping threads took: 0.0000469 seconds 2017-01-25T14:10:05.931+0530: 68050.384: Application time: 0.0000244 seconds 2017-01-25T14:10:05.931+0530: 68050.384: Total time for which application threads were stopped: 0.0001041 seconds, Stopping threads took: 0.0000321 seconds 2017-01-25T14:10:05.931+0530: 68050.384: Application time: 0.0000256 seconds 2017-01-25T14:10:05.931+0530: 68050.384: Total time for which application threads were stopped: 0.0001178 seconds, Stopping threads took: 0.0000398 seconds 2017-01-25T14:10:05.931+0530: 68050.384: Application time: 0.0000258 seconds 2017-01-25T14:10:05.932+0530: 68050.384: Total time for which application threads were stopped: 0.0001089 seconds, Stopping threads took: 0.0000301 seconds 2017-01-25T14:10:05.932+0530: 68050.384: Application time: 0.0000252 seconds 2017-01-25T14:10:05.932+0530: 68050.384: Total time for which application threads were stopped: 0.0000885 seconds, Stopping threads took: 0.0000214 seconds 2017-01-25T14:10:05.932+0530: 68050.384: Application time: 0.0000271 seconds 2017-01-25T14:10:05.932+0530: 68050.384: Total time for which application threads were stopped: 0.0001099 seconds, Stopping threads took: 0.0000216 seconds 2017-01-25T14:10:05.932+0530: 68050.384: Application time: 0.0000299 seconds 2017-01-25T14:10:05.932+0530: 68050.385: Total time for which application threads were stopped: 0.0001135 seconds, Stopping threads took: 0.0000388 seconds 2017-01-25T14:10:05.932+0530: 68050.385: Application time: 0.0000268 seconds 2017-01-25T14:10:05.932+0530: 68050.385: Total time for which application threads were stopped: 0.0001216 seconds, Stopping threads took: 0.0000326 seconds 2017-01-25T14:10:05.932+0530: 68050.385: Application time: 0.0001340 seconds 2017-01-25T14:10:05.932+0530: 68050.385: Total time for which application threads were stopped: 0.0001468 seconds, Stopping threads took: 0.0000357 seconds 2017-01-25T14:10:10.930+0530: 68055.383: Application time: 4.9978906 seconds 2017-01-25T14:10:10.931+0530: 68055.383: Total time for which application threads were stopped: 0.0002560 seconds, Stopping threads took: 0.0000478 seconds 2017-01-25T14:10:15.931+0530: 68060.383: Application time: 5.0002051 seconds 2017-01-25T14:10:15.931+0530: 68060.384: Total time for which application threads were stopped: 0.0002450 seconds, Stopping threads took: 0.0000483 seconds 2017-01-25T14:10:20.931+0530: 68065.384: Application time: 5.0002026 seconds 2017-01-25T14:10:20.932+0530: 68065.384: Total time for which application threads were stopped: 0.0003329 seconds, Stopping threads took: 0.0000851 seconds 2017-01-25T14:10:23.115+0530: 68067.568: Application time: 2.1837577 seconds 2017-01-25T14:10:23.116+0530: 68067.568: [GC pause (G1 Evacuation Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] desired_size: 182KB slow allocs: 0 refill waste: 2912B alloc: 0.00218 9116KB refills: 8 waste 11.9% gc: 174904B slow: 2912B fast: 0B TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 6KB slow allocs: 0 refill waste: 96B alloc: 0.00007 311KB refills: 8 waste 2.2% gc: 888B slow: 256B fast: 0B TLAB: gc thread: 0x00007f15906a1800 [id: 140] desired_size: 860KB slow allocs: 0 refill waste: 13760B alloc: 0.01027 43002KB refills: 8 waste 6.4% gc: 449272B slow: 1056B fast: 0B TLAB: gc thread: 0x00007f159069b000 [id: 136] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00003 115KB refills: 2 waste 28.2% gc: 1288B slow: 32B fast: 0B TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 774KB slow allocs: 0 refill waste: 12392B alloc: 0.00925 38728KB refills: 5 waste 7.8% gc: 309640B slow: 104B fast: 0B TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 859KB slow allocs: 0 refill waste: 13744B alloc: 0.01026 42960KB refills: 8 waste 5.6% gc: 395744B slow: 176B fast: 0B TLAB: gc thread: 0x00007f1590689800 [id: 126] desired_size: 2KB slow allocs: 12 refill waste: 32B alloc: 0.00002 97KB refills: 39 waste 2.3% gc: 632B slow: 1256B fast: 0B TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 855KB slow allocs: 0 refill waste: 13680B alloc: 0.01021 42750KB refills: 6 waste 10.1% gc: 529496B slow: 360B fast: 0B TLAB: gc thread: 0x00007f159067d800 [id: 121] desired_size: 2KB slow allocs: 0 refill waste: 40B alloc: 0.00003 138KB refills: 1 waste 69.9% gc: 1968B slow: 0B fast: 0B TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 848KB slow allocs: 0 refill waste: 13576B alloc: 0.01014 42437KB refills: 5 waste 2.3% gc: 97552B slow: 240B fast: 0B TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 113KB slow allocs: 0 refill waste: 1808B alloc: 0.00135 5668KB refills: 8 waste 2.9% gc: 26872B slow: 216B fast: 0B TLAB: gc thread: 0x00007f159062b000 [id: 115] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 3KB refills: 1 waste 82.9% gc: 1712B slow: 0B fast: 0B TLAB: gc thread: 0x00007f159005b000 [id: 114] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 1KB refills: 1 waste 98.4% gc: 2032B slow: 0B fast: 0B TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.30038 1257411KB refills: 2876 waste 0.0% gc: 0B slow: 131144B fast: 0B TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.07714 322937KB refills: 24 waste 3.5% gc: 1750360B slow: 1128B fast: 0B TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 2047KB slow allocs: 5 refill waste: 32760B alloc: 0.12559 525720KB refills: 38 waste 1.8% gc: 1429064B slow: 11480B fast: 0B TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 444KB slow allocs: 0 refill waste: 7104B alloc: 0.00530 22206KB refills: 5 waste 7.5% gc: 170736B slow: 600B fast: 0B TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 527KB slow allocs: 0 refill waste: 8432B alloc: 0.00630 26360KB refills: 8 waste 2.8% gc: 119208B slow: 384B fast: 0B TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 2047KB slow allocs: 2 refill waste: 32760B alloc: 0.40114 1679210KB refills: 96 waste 0.6% gc: 1023536B slow: 163696B fast: 0B TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00000 9KB refills: 2 waste 19.4% gc: 728B slow: 72B fast: 0B TLAB: gc thread: 0x00007f170d35a000 [id: 85] desired_size: 141KB slow allocs: 0 refill waste: 2264B alloc: 0.00170 7097KB refills: 1 waste 38.5% gc: 55928B slow: 0B fast: 0B TLAB: gc thread: 0x00007f170cda3800 [id: 63] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 0KB refills: 1 waste 86.4% gc: 1784B slow: 0B fast: 0B TLAB totals: thrds: 22 refills: 3151 max: 2876 slow allocs: 20 max 12 waste: 0.1% gc: 6543344B max: 1750360B slow: 315112B max: 163696B fast: 0B max: 0B Desired survivor size 268435456 bytes, new threshold 15 (max 15) - age 1: 1746112 bytes, 1746112 total - age 2: 41776 bytes, 1787888 total - age 3: 408 bytes, 1788296 total - age 4: 152 bytes, 1788448 total - age 5: 168 bytes, 1788616 total - age 6: 200 bytes, 1788816 total - age 9: 416 bytes, 1789232 total - age 12: 1816 bytes, 1791048 total - age 13: 24 bytes, 1791072 total - age 15: 24 bytes, 1791096 total 68067.568: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 4943, predicted base time: 18.96 ms, remaining time: 131.04 ms, target pause time: 150.00 ms] 68067.568: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 1022 regions, survivors: 2 regions, predicted young region time: 2.82 ms] 68067.568: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 1022 regions, survivors: 2 regions, old: 0 regions, predicted pause time: 21.78 ms, target pause time: 150.00 ms] 2017-01-25T14:10:23.397+0530: 68067.850: [SoftReference, 0 refs, 0.0012874 secs]2017-01-25T14:10:23.398+0530: 68067.851: [WeakReference, 1 refs, 0.0008602 secs]2017-01-25T14:10:23.399+0530: 68067.852: [FinalReference, 0 refs, 0.0010149 secs]2017-01-25T14:10:23.400+0530: 68067.853: [PhantomReference, 0 refs, 0 refs, 0.0015731 secs]2017-01-25T14:10:23.402+0530: 68067.855: [JNI Weak Reference, 0.0000199 secs], 0.2898520 secs] [Parallel Time: 280.9 ms, GC Workers: 20] [GC Worker Start (ms): Min: 68067568.4, Avg: 68067568.5, Max: 68067568.7, Diff: 0.3] [Ext Root Scanning (ms): Min: 0.3, Avg: 1.5, Max: 5.3, Diff: 5.0, Sum: 30.9] [Update RS (ms): Min: 0.0, Avg: 0.8, Max: 13.2, Diff: 13.2, Sum: 15.0] [Processed Buffers: Min: 0, Avg: 1.9, Max: 7, Diff: 7, Sum: 39] [Scan RS (ms): Min: 0.0, Avg: 0.2, Max: 0.3, Diff: 0.2, Sum: 3.3] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 266.5, Avg: 277.1, Max: 279.5, Diff: 13.0, Sum: 5542.9] [Termination (ms): Min: 0.0, Avg: 0.9, Max: 1.2, Diff: 1.2, Sum: 17.0] [Termination Attempts: Min: 1, Avg: 2.5, Max: 10, Diff: 9, Sum: 49] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 1.9] [GC Worker Total (ms): Min: 280.4, Avg: 280.6, Max: 280.7, Diff: 0.4, Sum: 5611.1] [GC Worker End (ms): Min: 68067849.0, Avg: 68067849.1, Max: 68067849.2, Diff: 0.2] [Code Root Fixup: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.6 ms] [Other: 8.3 ms] [Choose CSet: 0.0 ms] [Ref Proc: 5.2 ms] [Ref Enq: 0.4 ms] [Redirty Cards: 0.3 ms] [Humongous Register: 0.1 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 1.9 ms] [Eden: 4088.0M(4088.0M)->0.0B(3584.0M) Survivors: 8192.0K->512.0M Heap: 4176.5M(8192.0M)->599.1M(8192.0M)] [Times: user=1.50 sys=3.93, real=0.29 secs] 2017-01-25T14:10:23.406+0530: 68067.858: Total time for which application threads were stopped: 0.2902731 seconds, Stopping threads took: 0.0000596 seconds 2017-01-25T14:10:25.461+0530: 68069.914: Application time: 2.0554312 seconds 2017-01-25T14:10:25.461+0530: 68069.914: [GC pause (G1 Evacuation Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] desired_size: 110KB slow allocs: 0 refill waste: 1760B alloc: 0.00150 5504KB refills: 2 waste 4.4% gc: 9864B slow: 96B fast: 0B TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 3KB slow allocs: 0 refill waste: 56B alloc: 0.00005 188KB refills: 1 waste 7.3% gc: 280B slow: 0B fast: 0B TLAB: gc thread: 0x00007f15906a3800 [id: 141] desired_size: 2KB slow allocs: 4 refill waste: 32B alloc: 0.00003 116KB refills: 35 waste 1.8% gc: 464B slow: 1064B fast: 0B TLAB: gc thread: 0x00007f15906a1800 [id: 140] desired_size: 513KB slow allocs: 0 refill waste: 8208B alloc: 0.00699 25671KB refills: 2 waste 26.5% gc: 279000B slow: 24B fast: 0B TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 459KB slow allocs: 0 refill waste: 7344B alloc: 0.00626 22973KB refills: 2 waste 34.2% gc: 321376B slow: 24B fast: 0B TLAB: gc thread: 0x00007f1590692000 [id: 131] desired_size: 2KB slow allocs: 3 refill waste: 32B alloc: 0.00002 59KB refills: 12 waste 6.7% gc: 1240B slow: 424B fast: 0B TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 517KB slow allocs: 0 refill waste: 8272B alloc: 0.00705 25874KB refills: 1 waste 16.2% gc: 86056B slow: 0B fast: 0B TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 509KB slow allocs: 0 refill waste: 8152B alloc: 0.00695 25493KB refills: 3 waste 21.2% gc: 332112B slow: 80B fast: 0B TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 501KB slow allocs: 0 refill waste: 8016B alloc: 0.00683 25060KB refills: 1 waste 18.4% gc: 94472B slow: 0B fast: 0B TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 67KB slow allocs: 0 refill waste: 1080B alloc: 0.00092 3381KB refills: 1 waste 0.9% gc: 624B slow: 0B fast: 0B TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.52461 1925342KB refills: 2664 waste 0.0% gc: 0B slow: 122088B fast: 0B TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.05268 193343KB refills: 4 waste 24.6% gc: 2066344B slow: 272B fast: 0B TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.08553 313904KB refills: 4 waste 2.4% gc: 204904B slow: 96B fast: 0B TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 264KB slow allocs: 0 refill waste: 4224B alloc: 0.00360 13203KB refills: 3 waste 32.1% gc: 260448B slow: 96B fast: 0B TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 315KB slow allocs: 0 refill waste: 5048B alloc: 0.00430 15778KB refills: 1 waste 2.8% gc: 9056B slow: 0B fast: 0B TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.27249 1000030KB refills: 9 waste 3.1% gc: 589904B slow: 944B fast: 0B TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 2KB slow allocs: 3 refill waste: 64B alloc: 0.00000 6KB refills: 2 waste 3.1% gc: 72B slow: 56B fast: 0B TLAB totals: thrds: 17 refills: 2747 max: 2664 slow allocs: 10 max 4 waste: 0.1% gc: 4256216B max: 2066344B slow: 125264B max: 122088B fast: 0B max: 0B Desired survivor size 268435456 bytes, new threshold 1 (max 15) - age 1: 530412208 bytes, 530412208 total - age 2: 1173416 bytes, 531585624 total - age 3: 40248 bytes, 531625872 total - age 4: 408 bytes, 531626280 total - age 5: 152 bytes, 531626432 total - age 6: 168 bytes, 531626600 total - age 7: 200 bytes, 531626800 total - age 10: 416 bytes, 531627216 total - age 13: 1816 bytes, 531629032 total - age 14: 24 bytes, 531629056 total 68069.914: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 6052, predicted base time: 17.09 ms, remaining time: 132.91 ms, target pause time: 150.00 ms] 68069.914: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 896 regions, survivors: 128 regions, predicted young region time: 75.53 ms] 68069.914: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 896 regions, survivors: 128 regions, old: 0 regions, predicted pause time: 92.62 ms, target pause time: 150.00 ms] 2017-01-25T14:10:25.952+0530: 68070.405: [SoftReference, 0 refs, 0.0013460 secs]2017-01-25T14:10:25.953+0530: 68070.406: [WeakReference, 1 refs, 0.0006541 secs]2017-01-25T14:10:25.954+0530: 68070.407: [FinalReference, 1 refs, 0.0008434 secs]2017-01-25T14:10:25.955+0530: 68070.408: [PhantomReference, 0 refs, 0 refs, 0.0014087 secs]2017-01-25T14:10:25.956+0530: 68070.409: [JNI Weak Reference, 0.0000171 secs], 0.4988131 secs] [Parallel Time: 490.1 ms, GC Workers: 20] [GC Worker Start (ms): Min: 68069914.2, Avg: 68069914.4, Max: 68069914.7, Diff: 0.6] [Ext Root Scanning (ms): Min: 0.0, Avg: 3.5, Max: 21.2, Diff: 21.1, Sum: 71.0] [Update RS (ms): Min: 0.0, Avg: 1.9, Max: 16.0, Diff: 16.0, Sum: 37.4] [Processed Buffers: Min: 0, Avg: 1.9, Max: 7, Diff: 7, Sum: 39] [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.5] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 468.1, Avg: 483.7, Max: 488.6, Diff: 20.6, Sum: 9673.4] [Termination (ms): Min: 0.0, Avg: 0.3, Max: 1.4, Diff: 1.4, Sum: 6.6] [Termination Attempts: Min: 1, Avg: 2.5, Max: 5, Diff: 4, Sum: 50] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 1.6] [GC Worker Total (ms): Min: 489.2, Avg: 489.6, Max: 489.9, Diff: 0.7, Sum: 9792.5] [GC Worker End (ms): Min: 68070403.9, Avg: 68070404.0, Max: 68070404.1, Diff: 0.2] [Code Root Fixup: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.7 ms] [Other: 7.9 ms] [Choose CSet: 0.1 ms] [Ref Proc: 4.7 ms] [Ref Enq: 0.4 ms] [Redirty Cards: 1.1 ms] [Humongous Register: 0.1 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 1.4 ms] [Eden: 3584.0M(3584.0M)->0.0B(3636.0M) Survivors: 512.0M->460.0M Heap: 4183.1M(8192.0M)->1063.5M(8192.0M)] [Times: user=3.32 sys=2.12, real=0.50 secs] 2017-01-25T14:10:25.960+0530: 68070.413: Total time for which application threads were stopped: 0.4992361 seconds, Stopping threads took: 0.0000544 seconds 2017-01-25T14:10:25.960+0530: 68070.413: Application time: 0.0002027 seconds 2017-01-25T14:10:25.961+0530: 68070.413: Total time for which application threads were stopped: 0.0002799 seconds, Stopping threads took: 0.0000997 seconds 2017-01-25T14:10:25.977+0530: 68070.429: Application time: 0.0160277 seconds 2017-01-25T14:10:25.977+0530: 68070.430: Total time for which application threads were stopped: 0.0003467 seconds, Stopping threads took: 0.0000823 seconds 2017-01-25T14:10:26.977+0530: 68071.429: Application time: 0.9997404 seconds 2017-01-25T14:10:26.977+0530: 68071.430: Total time for which application threads were stopped: 0.0003541 seconds, Stopping threads took: 0.0000769 seconds 2017-01-25T14:10:27.470+0530: 68071.923: Application time: 0.4927146 seconds 2017-01-25T14:10:27.470+0530: 68071.923: [GC pause (G1 Evacuation Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] desired_size: 72KB slow allocs: 0 refill waste: 1160B alloc: 0.00098 3634KB refills: 4 waste 16.2% gc: 47528B slow: 592B fast: 0B TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00003 125KB refills: 2 waste 30.9% gc: 1544B slow: 32B fast: 0B TLAB: gc thread: 0x00007f15906a3800 [id: 141] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00003 106KB refills: 1 waste 90.8% gc: 1968B slow: 0B fast: 0B TLAB: gc thread: 0x00007f15906a1800 [id: 140] desired_size: 342KB slow allocs: 0 refill waste: 5480B alloc: 0.00461 17150KB refills: 1 waste 84.3% gc: 296088B slow: 0B fast: 0B TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 306KB slow allocs: 0 refill waste: 4896B alloc: 0.00411 15318KB refills: 1 waste 65.3% gc: 204832B slow: 0B fast: 0B TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 344KB slow allocs: 0 refill waste: 5512B alloc: 0.00463 17230KB refills: 2 waste 10.6% gc: 75104B slow: 24B fast: 0B TLAB: gc thread: 0x00007f159068b000 [id: 127] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 16KB refills: 2 waste 25.2% gc: 1008B slow: 32B fast: 0B TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 342KB slow allocs: 0 refill waste: 5472B alloc: 0.00460 17113KB refills: 2 waste 47.1% gc: 330304B slow: 24B fast: 0B TLAB: gc thread: 0x00007f159067d800 [id: 121] desired_size: 2KB slow allocs: 4 refill waste: 32B alloc: 0.00002 80KB refills: 45 waste 2.6% gc: 1288B slow: 1136B fast: 0B TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 335KB slow allocs: 0 refill waste: 5368B alloc: 0.00451 16793KB refills: 3 waste 2.7% gc: 27960B slow: 48B fast: 0B TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 45KB slow allocs: 0 refill waste: 720B alloc: 0.00061 2261KB refills: 1 waste 56.2% gc: 25992B slow: 0B fast: 0B TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.68823 2562479KB refills: 2706 waste 0.0% gc: 0B slow: 121752B fast: 0B TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.03467 129069KB refills: 1 waste 40.0% gc: 839808B slow: 0B fast: 0B TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.05604 208660KB refills: 3 waste 10.5% gc: 658040B slow: 144B fast: 0B TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 177KB slow allocs: 0 refill waste: 2840B alloc: 0.00239 8900KB refills: 1 waste 42.7% gc: 77840B slow: 0B fast: 0B TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 211KB slow allocs: 0 refill waste: 3376B alloc: 0.00283 10551KB refills: 1 waste 57.8% gc: 124864B slow: 0B fast: 0B TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.17819 663460KB refills: 5 waste 17.5% gc: 1831760B slow: 3288B fast: 0B TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 2KB slow allocs: 83 refill waste: 32B alloc: 0.00000 5KB refills: 84 waste 2.2% gc: 608B slow: 3248B fast: 0B TLAB: gc thread: 0x00007f170d35a000 [id: 85] desired_size: 82KB slow allocs: 0 refill waste: 1320B alloc: 0.00111 4136KB refills: 1 waste 65.0% gc: 55048B slow: 0B fast: 0B TLAB totals: thrds: 19 refills: 2866 max: 2706 slow allocs: 87 max 83 waste: 0.1% gc: 4601584B max: 1831760B slow: 130320B max: 121752B fast: 0B max: 0B Desired survivor size 268435456 bytes, new threshold 1 (max 15) - age 1: 458894880 bytes, 458894880 total 68071.923: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 29154, predicted base time: 21.52 ms, remaining time: 128.48 ms, target pause time: 150.00 ms] 68071.923: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 909 regions, survivors: 115 regions, predicted young region time: 302.72 ms] 68071.923: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 909 regions, survivors: 115 regions, old: 0 regions, predicted pause time: 324.24 ms, target pause time: 150.00 ms] 2017-01-25T14:10:27.649+0530: 68072.102: [SoftReference, 0 refs, 0.0014567 secs]2017-01-25T14:10:27.650+0530: 68072.103: [WeakReference, 0 refs, 0.0007514 secs]2017-01-25T14:10:27.651+0530: 68072.104: [FinalReference, 0 refs, 0.0007490 secs]2017-01-25T14:10:27.652+0530: 68072.105: [PhantomReference, 0 refs, 0 refs, 0.0015445 secs]2017-01-25T14:10:27.654+0530: 68072.106: [JNI Weak Reference, 0.0000168 secs], 0.1878181 secs] [Parallel Time: 178.2 ms, GC Workers: 20] [GC Worker Start (ms): Min: 68071923.1, Avg: 68071923.9, Max: 68071926.9, Diff: 3.8] [Ext Root Scanning (ms): Min: 0.0, Avg: 0.5, Max: 0.9, Diff: 0.9, Sum: 9.6] [Update RS (ms): Min: 0.0, Avg: 2.1, Max: 3.0, Diff: 3.0, Sum: 42.9] [Processed Buffers: Min: 0, Avg: 6.6, Max: 15, Diff: 15, Sum: 132] [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.1] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 174.0, Avg: 174.3, Max: 174.4, Diff: 0.4, Sum: 3485.4] [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 20] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.6] [GC Worker Total (ms): Min: 174.3, Avg: 177.1, Max: 178.0, Diff: 3.7, Sum: 3542.6] [GC Worker End (ms): Min: 68072101.0, Avg: 68072101.1, Max: 68072101.2, Diff: 0.2] [Code Root Fixup: 0.1 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.6 ms] [Other: 8.9 ms] [Choose CSet: 0.0 ms] [Ref Proc: 4.8 ms] [Ref Enq: 0.4 ms] [Redirty Cards: 0.8 ms] [Humongous Register: 0.0 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 1.4 ms] [Eden: 3636.0M(3636.0M)->0.0B(3940.0M) Survivors: 460.0M->156.0M Heap: 4699.5M(8192.0M)->1224.0M(8192.0M)] [Times: user=2.04 sys=0.04, real=0.19 secs] 2017-01-25T14:10:27.658+0530: 68072.111: Total time for which application threads were stopped: 0.1882558 seconds, Stopping threads took: 0.0000569 seconds 2017-01-25T14:10:27.977+0530: 68072.430: Application time: 0.3188950 seconds 2017-01-25T14:10:27.978+0530: 68072.430: Total time for which application threads were stopped: 0.0004108 seconds, Stopping threads took: 0.0001408 seconds 2017-01-25T14:10:28.977+0530: 68073.430: Application time: 0.9998111 seconds 2017-01-25T14:10:28.978+0530: 68073.430: Total time for which application threads were stopped: 0.0004384 seconds, Stopping threads took: 0.0000808 seconds 2017-01-25T14:10:29.104+0530: 68073.556: Application time: 0.1258489 seconds 2017-01-25T14:10:29.104+0530: 68073.556: [GC pause (G1 Evacuation Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] desired_size: 52KB slow allocs: 0 refill waste: 840B alloc: 0.00065 2627KB refills: 3 waste 2.9% gc: 4568B slow: 72B fast: 0B TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00002 90KB refills: 2 waste 16.1% gc: 648B slow: 16B fast: 0B TLAB: gc thread: 0x00007f15906a1800 [id: 140] desired_size: 243KB slow allocs: 0 refill waste: 3888B alloc: 0.00302 12171KB refills: 2 waste 28.0% gc: 139424B slow: 24B fast: 0B TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 216KB slow allocs: 0 refill waste: 3464B alloc: 0.00269 10844KB refills: 2 waste 33.0% gc: 146744B slow: 16B fast: 0B TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 246KB slow allocs: 0 refill waste: 3936B alloc: 0.00305 12301KB refills: 4 waste 9.4% gc: 94712B slow: 80B fast: 0B TLAB: gc thread: 0x00007f1590689800 [id: 126] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00002 89KB refills: 1 waste 95.3% gc: 1968B slow: 0B fast: 0B TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 242KB slow allocs: 0 refill waste: 3872B alloc: 0.00301 12125KB refills: 2 waste 25.9% gc: 128608B slow: 40B fast: 0B TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 242KB slow allocs: 0 refill waste: 3872B alloc: 0.00300 12111KB refills: 4 waste 20.0% gc: 198160B slow: 56B fast: 0B TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 32KB slow allocs: 0 refill waste: 512B alloc: 0.00040 1618KB refills: 3 waste 32.5% gc: 32200B slow: 96B fast: 0B TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.79581 3210757KB refills: 2935 waste 0.0% gc: 0B slow: 125400B fast: 0B TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 1818KB slow allocs: 0 refill waste: 29088B alloc: 0.02254 90920KB refills: 3 waste 25.2% gc: 1406080B slow: 72B fast: 0B TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 2047KB slow allocs: 1 refill waste: 32760B alloc: 0.03673 148169KB refills: 4 waste 6.7% gc: 562664B slow: 96B fast: 0B TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 125KB slow allocs: 0 refill waste: 2000B alloc: 0.00155 6272KB refills: 3 waste 27.1% gc: 104384B slow: 80B fast: 0B TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 150KB slow allocs: 0 refill waste: 2408B alloc: 0.00187 7548KB refills: 2 waste 47.7% gc: 147464B slow: 32B fast: 0B TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 2047KB slow allocs: 5 refill waste: 32760B alloc: 0.11651 470065KB refills: 5 waste 0.2% gc: 21408B slow: 264B fast: 0B TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 2KB slow allocs: 94 refill waste: 32B alloc: 0.00002 77KB refills: 92 waste 2.2% gc: 608B slow: 3480B fast: 0B TLAB totals: thrds: 16 refills: 3067 max: 2935 slow allocs: 101 max 94 waste: 0.1% gc: 2989640B max: 1406080B slow: 129824B max: 125400B fast: 0B max: 0B Desired survivor size 268435456 bytes, new threshold 15 (max 15) - age 1: 142834624 bytes, 142834624 total 68073.557: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 32047, predicted base time: 19.88 ms, remaining time: 130.12 ms, target pause time: 150.00 ms] 68073.557: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 985 regions, survivors: 39 regions, predicted young region time: 172.21 ms] 68073.557: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 985 regions, survivors: 39 regions, old: 0 regions, predicted pause time: 192.09 ms, target pause time: 150.00 ms] 2017-01-25T14:10:29.209+0530: 68073.662: [SoftReference, 0 refs, 0.0009941 secs]2017-01-25T14:10:29.210+0530: 68073.663: [WeakReference, 1 refs, 0.0005778 secs]2017-01-25T14:10:29.211+0530: 68073.664: [FinalReference, 0 refs, 0.0007467 secs]2017-01-25T14:10:29.212+0530: 68073.664: [PhantomReference, 0 refs, 0 refs, 0.0019718 secs]2017-01-25T14:10:29.214+0530: 68073.666: [JNI Weak Reference, 0.0000231 secs], 0.1125066 secs] [Parallel Time: 104.9 ms, GC Workers: 20] [GC Worker Start (ms): Min: 68073556.7, Avg: 68073557.9, Max: 68073570.3, Diff: 13.6] [Ext Root Scanning (ms): Min: 0.0, Avg: 0.5, Max: 1.1, Diff: 1.1, Sum: 9.9] [Update RS (ms): Min: 0.0, Avg: 3.1, Max: 9.1, Diff: 9.1, Sum: 61.3] [Processed Buffers: Min: 0, Avg: 7.0, Max: 12, Diff: 12, Sum: 139] [Scan RS (ms): Min: 0.1, Avg: 0.2, Max: 0.3, Diff: 0.2, Sum: 4.2] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 83.1, Avg: 92.1, Max: 96.4, Diff: 13.3, Sum: 1842.9] [Termination (ms): Min: 4.0, Avg: 7.5, Max: 11.2, Diff: 7.3, Sum: 149.5] [Termination Attempts: Min: 1775, Avg: 2783.8, Max: 3057, Diff: 1282, Sum: 55675] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.3] [GC Worker Total (ms): Min: 91.2, Avg: 103.5, Max: 104.8, Diff: 13.6, Sum: 2070.1] [GC Worker End (ms): Min: 68073661.3, Avg: 68073661.4, Max: 68073661.5, Diff: 0.2] [Code Root Fixup: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.6 ms] [Other: 7.0 ms] [Choose CSet: 0.1 ms] [Ref Proc: 4.7 ms] [Ref Enq: 0.3 ms] [Redirty Cards: 0.2 ms] [Humongous Register: 0.0 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 1.4 ms] [Eden: 3940.0M(3940.0M)->0.0B(3936.0M) Survivors: 156.0M->160.0M Heap: 5164.0M(8192.0M)->1228.0M(8192.0M)] [Times: user=2.32 sys=0.00, real=0.11 secs] 2017-01-25T14:10:29.217+0530: 68073.669: Total time for which application threads were stopped: 0.1129090 seconds, Stopping threads took: 0.0000389 seconds 2017-01-25T14:10:29.978+0530: 68074.430: Application time: 0.7611185 seconds 2017-01-25T14:10:29.978+0530: 68074.431: Total time for which application threads were stopped: 0.0003596 seconds, Stopping threads took: 0.0000773 seconds 2017-01-25T14:10:30.655+0530: 68075.108: Application time: 0.6773986 seconds 2017-01-25T14:10:30.656+0530: 68075.108: [GC pause (G1 Evacuation Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] desired_size: 34KB slow allocs: 0 refill waste: 552B alloc: 0.00043 1734KB refills: 2 waste 26.2% gc: 18568B slow: 16B fast: 0B TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00001 60KB refills: 1 waste 14.0% gc: 288B slow: 0B fast: 0B TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 143KB slow allocs: 0 refill waste: 2288B alloc: 0.00178 7155KB refills: 1 waste 41.6% gc: 61008B slow: 0B fast: 0B TLAB: gc thread: 0x00007f1590692000 [id: 131] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00001 51KB refills: 1 waste 95.3% gc: 1968B slow: 0B fast: 0B TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 165KB slow allocs: 0 refill waste: 2640B alloc: 0.00205 8255KB refills: 1 waste 81.8% gc: 138304B slow: 0B fast: 0B TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 159KB slow allocs: 0 refill waste: 2544B alloc: 0.00198 7965KB refills: 8 waste 9.3% gc: 120768B slow: 264B fast: 0B TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 161KB slow allocs: 0 refill waste: 2576B alloc: 0.00200 8058KB refills: 4 waste 19.8% gc: 130920B slow: 104B fast: 0B TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 21KB slow allocs: 0 refill waste: 336B alloc: 0.00027 1069KB refills: 3 waste 29.5% gc: 19336B slow: 40B fast: 0B TLAB: gc thread: 0x00007f158c010800 [id: 113] desired_size: 33KB slow allocs: 7 refill waste: 536B alloc: 0.00042 1678KB refills: 13 waste 5.9% gc: 24736B slow: 1496B fast: 0B TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.86555 3488586KB refills: 2935 waste 0.0% gc: 0B slow: 129024B fast: 0B TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 1202KB slow allocs: 0 refill waste: 19232B alloc: 0.01491 60103KB refills: 2 waste 33.4% gc: 823272B slow: 40B fast: 0B TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 1948KB slow allocs: 0 refill waste: 31176B alloc: 0.02418 97446KB refills: 3 waste 13.0% gc: 777080B slow: 1624B fast: 0B TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 83KB slow allocs: 0 refill waste: 1328B alloc: 0.00103 4153KB refills: 2 waste 33.2% gc: 56096B slow: 320B fast: 0B TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 99KB slow allocs: 0 refill waste: 1592B alloc: 0.00124 4985KB refills: 4 waste 23.7% gc: 96856B slow: 80B fast: 0B TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.07632 307612KB refills: 3 waste 6.4% gc: 387864B slow: 12608B fast: 0B TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 2KB slow allocs: 33 refill waste: 40B alloc: 0.00003 125KB refills: 39 waste 4.0% gc: 2464B slow: 1544B fast: 0B TLAB totals: thrds: 16 refills: 3022 max: 2935 slow allocs: 40 max 33 waste: 0.0% gc: 2659528B max: 823272B slow: 147160B max: 129024B fast: 0B max: 0B Desired survivor size 268435456 bytes, new threshold 15 (max 15) - age 1: 760136 bytes, 760136 total - age 2: 142813000 bytes, 143573136 total 68075.108: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 4653, predicted base time: 13.35 ms, remaining time: 136.65 ms, target pause time: 150.00 ms] 68075.108: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 984 regions, survivors: 40 regions, predicted young region time: 183.84 ms] 68075.108: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 984 regions, survivors: 40 regions, old: 0 regions, predicted pause time: 197.19 ms, target pause time: 150.00 ms] 2017-01-25T14:10:30.735+0530: 68075.188: [SoftReference, 0 refs, 0.0011672 secs]2017-01-25T14:10:30.736+0530: 68075.189: [WeakReference, 0 refs, 0.0007628 secs]2017-01-25T14:10:30.737+0530: 68075.190: [FinalReference, 0 refs, 0.0009336 secs]2017-01-25T14:10:30.738+0530: 68075.190: [PhantomReference, 0 refs, 0 refs, 0.0015168 secs]2017-01-25T14:10:30.739+0530: 68075.192: [JNI Weak Reference, 0.0000164 secs], 0.0864694 secs] [Parallel Time: 78.7 ms, GC Workers: 20] [GC Worker Start (ms): Min: 68075108.5, Avg: 68075113.2, Max: 68075143.8, Diff: 35.3] [Ext Root Scanning (ms): Min: 0.0, Avg: 0.3, Max: 1.1, Diff: 1.1, Sum: 6.1] [Update RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 1.3] [Processed Buffers: Min: 0, Avg: 1.6, Max: 5, Diff: 5, Sum: 32] [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 2.1] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 35.8, Avg: 67.4, Max: 73.2, Diff: 37.4, Sum: 1348.8] [Termination (ms): Min: 2.7, Avg: 5.7, Max: 14.4, Diff: 11.7, Sum: 113.8] [Termination Attempts: Min: 1446, Avg: 2820.5, Max: 3166, Diff: 1720, Sum: 56410] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.6] [GC Worker Total (ms): Min: 43.2, Avg: 73.7, Max: 78.4, Diff: 35.2, Sum: 1474.7] [GC Worker End (ms): Min: 68075186.8, Avg: 68075186.9, Max: 68075187.0, Diff: 0.2] [Code Root Fixup: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.5 ms] [Other: 7.2 ms] [Choose CSet: 0.0 ms] [Ref Proc: 4.8 ms] [Ref Enq: 0.4 ms] [Redirty Cards: 0.3 ms] [Humongous Register: 0.0 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 1.4 ms] [Eden: 3936.0M(3936.0M)->0.0B(3936.0M) Survivors: 160.0M->160.0M Heap: 5164.0M(8192.0M)->1228.0M(8192.0M)] [Times: user=1.55 sys=0.00, real=0.09 secs] 2017-01-25T14:10:30.742+0530: 68075.195: Total time for which application threads were stopped: 0.0868569 seconds, Stopping threads took: 0.0000422 seconds 2017-01-25T14:10:30.961+0530: 68075.414: Application time: 0.2187346 seconds 2017-01-25T14:10:30.961+0530: 68075.414: Total time for which application threads were stopped: 0.0003312 seconds, Stopping threads took: 0.0001166 seconds 2017-01-25T14:10:30.978+0530: 68075.430: Application time: 0.0164710 seconds 2017-01-25T14:10:30.978+0530: 68075.431: Total time for which application threads were stopped: 0.0003836 seconds, Stopping threads took: 0.0000729 seconds 2017-01-25T14:10:31.978+0530: 68076.431: Application time: 0.9997933 seconds 2017-01-25T14:10:31.978+0530: 68076.431: Total time for which application threads were stopped: 0.0003692 seconds, Stopping threads took: 0.0000542 seconds 2017-01-25T14:10:32.213+0530: 68076.666: Application time: 0.2350334 seconds 2017-01-25T14:10:32.214+0530: 68076.666: [GC pause (G1 Evacuation Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] desired_size: 23KB slow allocs: 1 refill waste: 368B alloc: 0.00029 1156KB refills: 6 waste 11.5% gc: 16024B slow: 256B fast: 0B TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00001 39KB refills: 2 waste 16.1% gc: 648B slow: 16B fast: 0B TLAB: gc thread: 0x00007f1590695800 [id: 133] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 18KB refills: 2 waste 26.0% gc: 1056B slow: 16B fast: 0B TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 108KB slow allocs: 0 refill waste: 1736B alloc: 0.00135 5443KB refills: 1 waste 33.8% gc: 37696B slow: 0B fast: 0B TLAB: gc thread: 0x00007f1590674800 [id: 124] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 18KB refills: 2 waste 25.2% gc: 1008B slow: 32B fast: 0B TLAB: gc thread: 0x00007f159067d800 [id: 121] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00002 92KB refills: 1 waste 98.8% gc: 2040B slow: 0B fast: 0B TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 14KB slow allocs: 0 refill waste: 224B alloc: 0.00017 704KB refills: 1 waste 89.2% gc: 12856B slow: 0B fast: 0B TLAB: gc thread: 0x00007f159062b000 [id: 115] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 3KB refills: 1 waste 92.2% gc: 1904B slow: 0B fast: 0B TLAB: gc thread: 0x00007f159005b000 [id: 114] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 1KB refills: 1 waste 98.4% gc: 2032B slow: 0B fast: 0B TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.91131 3672984KB refills: 2943 waste 0.0% gc: 0B slow: 126608B fast: 0B TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 792KB slow allocs: 0 refill waste: 12680B alloc: 0.00984 39644KB refills: 1 waste 84.1% gc: 682600B slow: 0B fast: 0B TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 1288KB slow allocs: 0 refill waste: 20608B alloc: 0.01598 64410KB refills: 1 waste 87.7% gc: 1157424B slow: 0B fast: 0B TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 54KB slow allocs: 0 refill waste: 872B alloc: 0.00068 2737KB refills: 1 waste 88.4% gc: 49536B slow: 0B fast: 0B TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 65KB slow allocs: 0 refill waste: 1048B alloc: 0.00081 3279KB refills: 2 waste 46.4% gc: 62272B slow: 32B fast: 0B TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.05015 202117KB refills: 2 waste 41.0% gc: 1720200B slow: 16B fast: 0B TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 2KB slow allocs: 69 refill waste: 32B alloc: 0.00003 119KB refills: 85 waste 1.7% gc: 216B slow: 3280B fast: 0B TLAB: gc thread: 0x00007f170d35a000 [id: 85] desired_size: 58KB slow allocs: 0 refill waste: 928B alloc: 0.00072 2910KB refills: 2 waste 26.8% gc: 31920B slow: 16B fast: 0B TLAB totals: thrds: 17 refills: 3054 max: 2943 slow allocs: 71 max 69 waste: 0.1% gc: 3779432B max: 1720200B slow: 130272B max: 126608B fast: 0B max: 0B Desired survivor size 268435456 bytes, new threshold 15 (max 15) - age 1: 711456 bytes, 711456 total - age 2: 751240 bytes, 1462696 total - age 3: 142811400 bytes, 144274096 total 68076.666: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 3702, predicted base time: 13.05 ms, remaining time: 136.95 ms, target pause time: 150.00 ms] 68076.666: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 984 regions, survivors: 40 regions, predicted young region time: 156.79 ms] 68076.666: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 984 regions, survivors: 40 regions, old: 0 regions, predicted pause time: 169.84 ms, target pause time: 150.00 ms] 2017-01-25T14:10:32.310+0530: 68076.763: [SoftReference, 0 refs, 0.0012311 secs]2017-01-25T14:10:32.312+0530: 68076.764: [WeakReference, 1 refs, 0.0007000 secs]2017-01-25T14:10:32.312+0530: 68076.765: [FinalReference, 0 refs, 0.0006386 secs]2017-01-25T14:10:32.313+0530: 68076.766: [PhantomReference, 0 refs, 0 refs, 0.0012583 secs]2017-01-25T14:10:32.314+0530: 68076.767: [JNI Weak Reference, 0.0000173 secs], 0.1034113 secs] [Parallel Time: 96.2 ms, GC Workers: 20] [GC Worker Start (ms): Min: 68076666.5, Avg: 68076669.0, Max: 68076678.6, Diff: 12.1] [Ext Root Scanning (ms): Min: 0.0, Avg: 0.3, Max: 1.0, Diff: 1.0, Sum: 6.1] [Update RS (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.3, Sum: 1.4] [Processed Buffers: Min: 0, Avg: 1.5, Max: 4, Diff: 4, Sum: 30] [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 2.5] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 76.2, Avg: 86.0, Max: 91.6, Diff: 15.4, Sum: 1720.4] [Termination (ms): Min: 3.1, Avg: 6.8, Max: 9.0, Diff: 5.9, Sum: 136.7] [Termination Attempts: Min: 1634, Avg: 2592.2, Max: 2917, Diff: 1283, Sum: 51845] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.7] [GC Worker Total (ms): Min: 83.9, Avg: 93.5, Max: 96.1, Diff: 12.2, Sum: 1869.8] [GC Worker End (ms): Min: 68076762.4, Avg: 68076762.5, Max: 68076762.6, Diff: 0.2] [Code Root Fixup: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.7 ms] [Other: 6.5 ms] [Choose CSet: 0.0 ms] [Ref Proc: 4.2 ms] [Ref Enq: 0.3 ms] [Redirty Cards: 0.3 ms] [Humongous Register: 0.1 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 1.4 ms] [Eden: 3936.0M(3936.0M)->0.0B(3936.0M) Survivors: 160.0M->160.0M Heap: 5164.0M(8192.0M)->1230.0M(8192.0M)] [Times: user=2.15 sys=0.00, real=0.10 secs] 2017-01-25T14:10:32.317+0530: 68076.770: Total time for which application threads were stopped: 0.1038361 seconds, Stopping threads took: 0.0000385 seconds 2017-01-25T14:10:32.978+0530: 68077.431: Application time: 0.6609230 seconds 2017-01-25T14:10:32.978+0530: 68077.431: Total time for which application threads were stopped: 0.0003272 seconds, Stopping threads took: 0.0000550 seconds 2017-01-25T14:10:33.766+0530: 68078.219: Application time: 0.7878756 seconds 2017-01-25T14:10:33.767+0530: 68078.219: [GC pause (G1 Evacuation Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] desired_size: 15KB slow allocs: 1 refill waste: 248B alloc: 0.00020 791KB refills: 5 waste 19.1% gc: 15280B slow: 144B fast: 0B TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00001 27KB refills: 1 waste 14.0% gc: 288B slow: 0B fast: 0B TLAB: gc thread: 0x00007f15906a8800 [id: 144] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 18KB refills: 2 waste 26.0% gc: 1056B slow: 16B fast: 0B TLAB: gc thread: 0x00007f15906a7000 [id: 143] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 18KB refills: 2 waste 26.0% gc: 1056B slow: 16B fast: 0B TLAB: gc thread: 0x00007f15906a5000 [id: 142] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 18KB refills: 2 waste 26.0% gc: 1056B slow: 16B fast: 0B TLAB: gc thread: 0x00007f15906a0000 [id: 139] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 18KB refills: 2 waste 25.2% gc: 1008B slow: 32B fast: 0B TLAB: gc thread: 0x00007f159069e800 [id: 138] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 18KB refills: 2 waste 25.2% gc: 1008B slow: 32B fast: 0B TLAB: gc thread: 0x00007f159069c800 [id: 137] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 18KB refills: 2 waste 25.2% gc: 1008B slow: 32B fast: 0B TLAB: gc thread: 0x00007f1590697800 [id: 134] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 18KB refills: 2 waste 26.0% gc: 1056B slow: 16B fast: 0B TLAB: gc thread: 0x00007f1590693800 [id: 132] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 18KB refills: 2 waste 26.0% gc: 1056B slow: 16B fast: 0B TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 108KB slow allocs: 0 refill waste: 1728B alloc: 0.00134 5419KB refills: 1 waste 100.0% gc: 110960B slow: 0B fast: 0B TLAB: gc thread: 0x00007f159068e800 [id: 129] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 18KB refills: 2 waste 26.0% gc: 1056B slow: 16B fast: 0B TLAB: gc thread: 0x00007f159068d000 [id: 128] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 18KB refills: 2 waste 25.0% gc: 1016B slow: 16B fast: 0B TLAB: gc thread: 0x00007f1590673000 [id: 123] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 18KB refills: 2 waste 25.2% gc: 1008B slow: 32B fast: 0B TLAB: gc thread: 0x00007f1590671800 [id: 122] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 18KB refills: 2 waste 25.2% gc: 1008B slow: 32B fast: 0B TLAB: gc thread: 0x00007f1590637000 [id: 117] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 18KB refills: 2 waste 25.8% gc: 1048B slow: 16B fast: 0B TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 9KB slow allocs: 0 refill waste: 144B alloc: 0.00011 458KB refills: 2 waste 41.7% gc: 7816B slow: 16B fast: 0B TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.94192 3796378KB refills: 2942 waste 0.0% gc: 0B slow: 127600B fast: 0B TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.03274 131961KB refills: 1 waste 60.0% gc: 1257536B slow: 0B fast: 0B TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow allocs: 29 refill waste: 48B alloc: 0.00004 155KB refills: 31 waste 1.9% gc: 608B slow: 1296B fast: 0B TLAB totals: thrds: 20 refills: 3009 max: 2942 slow allocs: 30 max 29 waste: 0.0% gc: 1405928B max: 1257536B slow: 129344B max: 127600B fast: 0B max: 0B Desired survivor size 268435456 bytes, new threshold 15 (max 15) - age 1: 50800 bytes, 50800 total - age 2: 708304 bytes, 759104 total - age 3: 751208 bytes, 1510312 total - age 4: 142803240 bytes, 144313552 total 68078.219: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 4372, predicted base time: 12.18 ms, remaining time: 137.82 ms, target pause time: 150.00 ms] 68078.219: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 984 regions, survivors: 40 regions, predicted young region time: 148.25 ms] 68078.219: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 984 regions, survivors: 40 regions, old: 0 regions, predicted pause time: 160.43 ms, target pause time: 150.00 ms] 2017-01-25T14:10:33.847+0530: 68078.299: [SoftReference, 0 refs, 0.0010865 secs]2017-01-25T14:10:33.848+0530: 68078.300: [WeakReference, 0 refs, 0.0007960 secs]2017-01-25T14:10:33.848+0530: 68078.301: [FinalReference, 0 refs, 0.0007028 secs]2017-01-25T14:10:33.849+0530: 68078.302: [PhantomReference, 0 refs, 0 refs, 0.0015020 secs]2017-01-25T14:10:33.851+0530: 68078.303: [JNI Weak Reference, 0.0000198 secs], 0.0873865 secs] [Parallel Time: 79.2 ms, GC Workers: 20] [GC Worker Start (ms): Min: 68078219.5, Avg: 68078228.0, Max: 68078264.4, Diff: 44.9] [Ext Root Scanning (ms): Min: 0.0, Avg: 0.3, Max: 1.1, Diff: 1.1, Sum: 5.9] [Update RS (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 1.4] [Processed Buffers: Min: 0, Avg: 1.9, Max: 6, Diff: 6, Sum: 37] [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 2.3] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 29.0, Avg: 64.0, Max: 75.6, Diff: 46.6, Sum: 1280.6] [Termination (ms): Min: 2.1, Avg: 6.0, Max: 19.0, Diff: 16.9, Sum: 119.3] [Termination Attempts: Min: 1180, Avg: 2444.2, Max: 2973, Diff: 1793, Sum: 48883] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.3] [GC Worker Total (ms): Min: 34.1, Avg: 70.6, Max: 79.1, Diff: 45.0, Sum: 1411.8] [GC Worker End (ms): Min: 68078298.5, Avg: 68078298.6, Max: 68078298.7, Diff: 0.2] [Code Root Fixup: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.5 ms] [Other: 7.6 ms] [Choose CSet: 0.0 ms] [Ref Proc: 4.5 ms] [Ref Enq: 0.4 ms] [Redirty Cards: 0.3 ms] [Humongous Register: 0.0 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 2.0 ms] [Eden: 3936.0M(3936.0M)->0.0B(3932.0M) Survivors: 160.0M->164.0M Heap: 5166.0M(8192.0M)->1232.0M(8192.0M)] [Times: user=1.47 sys=0.00, real=0.09 secs] 2017-01-25T14:10:33.854+0530: 68078.307: Total time for which application threads were stopped: 0.0878013 seconds, Stopping threads took: 0.0000417 seconds 2017-01-25T14:10:33.978+0530: 68078.431: Application time: 0.1240628 seconds 2017-01-25T14:10:33.979+0530: 68078.431: Total time for which application threads were stopped: 0.0003893 seconds, Stopping threads took: 0.0000755 seconds 2017-01-25T14:10:34.978+0530: 68079.431: Application time: 0.9997692 seconds 2017-01-25T14:10:34.979+0530: 68079.431: Total time for which application threads were stopped: 0.0003924 seconds, Stopping threads took: 0.0000777 seconds 2017-01-25T14:10:35.281+0530: 68079.733: Application time: 0.3019245 seconds 2017-01-25T14:10:35.281+0530: 68079.734: [GC pause (G1 Evacuation Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] desired_size: 10KB slow allocs: 3 refill waste: 168B alloc: 0.00013 534KB refills: 11 waste 0.4% gc: 120B slow: 416B fast: 0B TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00000 18KB refills: 2 waste 16.1% gc: 648B slow: 16B fast: 0B TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 70KB slow allocs: 0 refill waste: 1120B alloc: 0.00087 3519KB refills: 4 waste 21.5% gc: 61856B slow: 120B fast: 0B TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 107KB slow allocs: 0 refill waste: 1712B alloc: 0.00133 5363KB refills: 2 waste 46.5% gc: 102184B slow: 24B fast: 0B TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 6KB slow allocs: 0 refill waste: 96B alloc: 0.00007 301KB refills: 4 waste 1.4% gc: 272B slow: 80B fast: 0B TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.96203 3873484KB refills: 2938 waste 0.0% gc: 0B slow: 126672B fast: 0B TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 517KB slow allocs: 0 refill waste: 8280B alloc: 0.00643 25881KB refills: 1 waste 50.0% gc: 265128B slow: 0B fast: 0B TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 844KB slow allocs: 0 refill waste: 13512B alloc: 0.01049 42241KB refills: 1 waste 47.6% gc: 412024B slow: 0B fast: 0B TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 35KB slow allocs: 0 refill waste: 568B alloc: 0.00044 1787KB refills: 2 waste 27.5% gc: 20088B slow: 16B fast: 0B TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 42KB slow allocs: 0 refill waste: 680B alloc: 0.00053 2132KB refills: 1 waste 16.5% gc: 7208B slow: 0B fast: 0B TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 1716KB slow allocs: 0 refill waste: 27456B alloc: 0.02131 85816KB refills: 1 waste 13.4% gc: 234760B slow: 0B fast: 0B TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 2KB slow allocs: 51 refill waste: 40B alloc: 0.00003 136KB refills: 68 waste 2.2% gc: 1336B slow: 2792B fast: 0B TLAB: gc thread: 0x00007f170d2cf000 [id: 84] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 12KB refills: 2 waste 49.6% gc: 2032B slow: 16B fast: 0B TLAB totals: thrds: 13 refills: 3037 max: 2938 slow allocs: 55 max 51 waste: 0.0% gc: 1107656B max: 412024B slow: 130152B max: 126672B fast: 0B max: 0B Desired survivor size 268435456 bytes, new threshold 15 (max 15) - age 1: 17912 bytes, 17912 total - age 2: 42776 bytes, 60688 total - age 3: 708128 bytes, 768816 total - age 4: 734632 bytes, 1503448 total - age 5: 142801080 bytes, 144304528 total 68079.734: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 3423, predicted base time: 13.52 ms, remaining time: 136.48 ms, target pause time: 150.00 ms] 68079.734: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 983 regions, survivors: 41 regions, predicted young region time: 127.04 ms] 68079.734: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 983 regions, survivors: 41 regions, old: 0 regions, predicted pause time: 140.56 ms, target pause time: 150.00 ms] 2017-01-25T14:10:35.378+0530: 68079.831: [SoftReference, 0 refs, 0.0009839 secs]2017-01-25T14:10:35.379+0530: 68079.832: [WeakReference, 0 refs, 0.0004729 secs]2017-01-25T14:10:35.380+0530: 68079.832: [FinalReference, 0 refs, 0.0013721 secs]2017-01-25T14:10:35.381+0530: 68079.834: [PhantomReference, 0 refs, 0 refs, 0.0016699 secs]2017-01-25T14:10:35.383+0530: 68079.835: [JNI Weak Reference, 0.0000267 secs], 0.1043019 secs] [Parallel Time: 96.5 ms, GC Workers: 20] [GC Worker Start (ms): Min: 68079733.9, Avg: 68079734.3, Max: 68079737.4, Diff: 3.6] [Ext Root Scanning (ms): Min: 0.0, Avg: 0.4, Max: 1.2, Diff: 1.2, Sum: 7.4] [Update RS (ms): Min: 0.0, Avg: 0.5, Max: 9.4, Diff: 9.4, Sum: 10.7] [Processed Buffers: Min: 0, Avg: 1.2, Max: 4, Diff: 4, Sum: 25] [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.8] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 79.1, Avg: 87.4, Max: 91.7, Diff: 12.7, Sum: 1748.5] [Termination (ms): Min: 3.2, Avg: 7.1, Max: 8.2, Diff: 5.0, Sum: 142.2] [Termination Attempts: Min: 1695, Avg: 2948.3, Max: 3320, Diff: 1625, Sum: 58966] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.2, Sum: 3.0] [GC Worker Total (ms): Min: 92.7, Avg: 95.7, Max: 96.2, Diff: 3.5, Sum: 1914.6] [GC Worker End (ms): Min: 68079829.9, Avg: 68079830.0, Max: 68079830.1, Diff: 0.2] [Code Root Fixup: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.5 ms] [Other: 7.3 ms] [Choose CSet: 0.0 ms] [Ref Proc: 4.9 ms] [Ref Enq: 0.4 ms] [Redirty Cards: 0.2 ms] [Humongous Register: 0.1 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 1.4 ms] [Eden: 3932.0M(3932.0M)->0.0B(3944.0M) Survivors: 164.0M->152.0M Heap: 5164.0M(8192.0M)->1222.0M(8192.0M)] [Times: user=2.28 sys=0.00, real=0.10 secs] 2017-01-25T14:10:35.385+0530: 68079.838: Total time for which application threads were stopped: 0.1047001 seconds, Stopping threads took: 0.0000510 seconds 2017-01-25T14:10:35.962+0530: 68080.414: Application time: 0.5761329 seconds 2017-01-25T14:10:35.962+0530: 68080.415: Total time for which application threads were stopped: 0.0003434 seconds, Stopping threads took: 0.0000783 seconds 2017-01-25T14:10:35.979+0530: 68080.431: Application time: 0.0166615 seconds 2017-01-25T14:10:35.979+0530: 68080.432: Total time for which application threads were stopped: 0.0004284 seconds, Stopping threads took: 0.0000705 seconds 2017-01-25T14:10:36.852+0530: 68081.305: Application time: 0.8734277 seconds 2017-01-25T14:10:36.853+0530: 68081.305: [GC pause (G1 Evacuation Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] desired_size: 7KB slow allocs: 4 refill waste: 120B alloc: 0.00009 380KB refills: 8 waste 3.4% gc: 1600B slow: 536B fast: 0B TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 13KB refills: 1 waste 14.0% gc: 288B slow: 0B fast: 0B TLAB: gc thread: 0x00007f15906a1800 [id: 140] desired_size: 158KB slow allocs: 0 refill waste: 2536B alloc: 0.00197 7938KB refills: 2 waste 3.3% gc: 10752B slow: 56B fast: 0B TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 93KB slow allocs: 0 refill waste: 1496B alloc: 0.00116 4686KB refills: 1 waste 71.5% gc: 68656B slow: 0B fast: 0B TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 46KB slow allocs: 0 refill waste: 744B alloc: 0.00058 2345KB refills: 5 waste 8.2% gc: 19568B slow: 112B fast: 0B TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 71KB slow allocs: 0 refill waste: 1136B alloc: 0.00088 3560KB refills: 3 waste 8.9% gc: 19320B slow: 64B fast: 0B TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 70KB slow allocs: 0 refill waste: 1128B alloc: 0.00088 3534KB refills: 3 waste 19.7% gc: 42520B slow: 152B fast: 0B TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 3KB slow allocs: 1 refill waste: 56B alloc: 0.00005 199KB refills: 11 waste 5.0% gc: 1952B slow: 272B fast: 0B TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.97497 3937562KB refills: 2941 waste 0.0% gc: 0B slow: 125192B fast: 0B TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 337KB slow allocs: 0 refill waste: 5392B alloc: 0.00418 16874KB refills: 7 waste 5.5% gc: 131656B slow: 376B fast: 0B TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 550KB slow allocs: 0 refill waste: 8808B alloc: 0.00682 27540KB refills: 5 waste 7.1% gc: 200160B slow: 768B fast: 0B TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 23KB slow allocs: 0 refill waste: 368B alloc: 0.00029 1166KB refills: 6 waste 12.3% gc: 17416B slow: 200B fast: 0B TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 27KB slow allocs: 0 refill waste: 440B alloc: 0.00034 1390KB refills: 7 waste 12.6% gc: 24792B slow: 392B fast: 0B TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 1124KB slow allocs: 0 refill waste: 17992B alloc: 0.01393 56239KB refills: 5 waste 2.1% gc: 117352B slow: 856B fast: 0B TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow allocs: 37 refill waste: 48B alloc: 0.00004 160KB refills: 30 waste 3.5% gc: 2016B slow: 1432B fast: 0B TLAB: gc thread: 0x00007f170d35a000 [id: 85] desired_size: 37KB slow allocs: 0 refill waste: 600B alloc: 0.00047 1896KB refills: 1 waste 23.6% gc: 9144B slow: 0B fast: 0B TLAB totals: thrds: 16 refills: 3036 max: 2941 slow allocs: 42 max 37 waste: 0.0% gc: 667192B max: 200160B slow: 130408B max: 125192B fast: 0B max: 0B Desired survivor size 268435456 bytes, new threshold 15 (max 15) - age 1: 123944 bytes, 123944 total - age 2: 14808 bytes, 138752 total - age 3: 42672 bytes, 181424 total - age 4: 691808 bytes, 873232 total - age 5: 731720 bytes, 1604952 total - age 6: 142801080 bytes, 144406032 total 68081.305: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 4530, predicted base time: 12.52 ms, remaining time: 137.48 ms, target pause time: 150.00 ms] 68081.305: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 986 regions, survivors: 38 regions, predicted young region time: 121.86 ms] 68081.305: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 986 regions, survivors: 38 regions, old: 0 regions, predicted pause time: 134.38 ms, target pause time: 150.00 ms] 2017-01-25T14:10:36.952+0530: 68081.405: [SoftReference, 0 refs, 0.0011154 secs]2017-01-25T14:10:36.954+0530: 68081.406: [WeakReference, 0 refs, 0.0007341 secs]2017-01-25T14:10:36.954+0530: 68081.407: [FinalReference, 0 refs, 0.0006622 secs]2017-01-25T14:10:36.955+0530: 68081.408: [PhantomReference, 0 refs, 0 refs, 0.0011375 secs]2017-01-25T14:10:36.956+0530: 68081.409: [JNI Weak Reference, 0.0000164 secs], 0.1062167 secs] [Parallel Time: 99.2 ms, GC Workers: 20] [GC Worker Start (ms): Min: 68081305.5, Avg: 68081305.7, Max: 68081306.1, Diff: 0.6] [Ext Root Scanning (ms): Min: 0.0, Avg: 0.6, Max: 5.4, Diff: 5.4, Sum: 11.8] [Update RS (ms): Min: 0.0, Avg: 0.5, Max: 8.3, Diff: 8.3, Sum: 9.8] [Processed Buffers: Min: 0, Avg: 1.6, Max: 5, Diff: 5, Sum: 32] [Scan RS (ms): Min: 0.0, Avg: 0.2, Max: 0.3, Diff: 0.2, Sum: 3.6] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 82.5, Avg: 90.3, Max: 94.8, Diff: 12.3, Sum: 1805.9] [Termination (ms): Min: 3.2, Avg: 7.0, Max: 8.2, Diff: 5.0, Sum: 140.6] [Termination Attempts: Min: 1726, Avg: 2931.2, Max: 3169, Diff: 1443, Sum: 58623] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.5] [GC Worker Total (ms): Min: 98.3, Avg: 98.7, Max: 99.0, Diff: 0.7, Sum: 1974.2] [GC Worker End (ms): Min: 68081404.3, Avg: 68081404.4, Max: 68081404.5, Diff: 0.2] [Code Root Fixup: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.5 ms] [Other: 6.5 ms] [Choose CSet: 0.0 ms] [Ref Proc: 4.1 ms] [Ref Enq: 0.4 ms] [Redirty Cards: 0.3 ms] [Humongous Register: 0.0 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 1.4 ms] [Eden: 3944.0M(3944.0M)->0.0B(3940.0M) Survivors: 152.0M->156.0M Heap: 5166.0M(8192.0M)->1224.0M(8192.0M)] [Times: user=2.14 sys=0.00, real=0.11 secs] 2017-01-25T14:10:36.959+0530: 68081.412: Total time for which application threads were stopped: 0.1066190 seconds, Stopping threads took: 0.0000391 seconds 2017-01-25T14:10:36.978+0530: 68081.430: Application time: 0.0187454 seconds 2017-01-25T14:10:36.978+0530: 68081.431: Total time for which application threads were stopped: 0.0003858 seconds, Stopping threads took: 0.0000614 seconds 2017-01-25T14:10:37.978+0530: 68082.431: Application time: 0.9999582 seconds 2017-01-25T14:10:37.979+0530: 68082.431: Total time for which application threads were stopped: 0.0004068 seconds, Stopping threads took: 0.0000728 seconds 2017-01-25T14:10:38.377+0530: 68082.829: Application time: 0.3980715 seconds 2017-01-25T14:10:38.377+0530: 68082.830: [GC pause (G1 Evacuation Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] desired_size: 5KB slow allocs: 8 refill waste: 80B alloc: 0.00007 269KB refills: 19 waste 5.3% gc: 4760B slow: 760B fast: 0B TLAB: gc thread: 0x00007f15f8114800 [id: 199] desired_size: 2047KB slow allocs: 1 refill waste: 32760B alloc: 0.02691 108566KB refills: 27 waste 3.6% gc: 2042560B slow: 840B fast: 0B TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00000 9KB refills: 2 waste 16.1% gc: 648B slow: 16B fast: 0B TLAB: gc thread: 0x00007f15906a1800 [id: 140] desired_size: 104KB slow allocs: 0 refill waste: 1664B alloc: 0.00129 5210KB refills: 3 waste 22.5% gc: 71936B slow: 40B fast: 0B TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 31KB slow allocs: 0 refill waste: 496B alloc: 0.00039 1560KB refills: 3 waste 28.5% gc: 27208B slow: 88B fast: 0B TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 46KB slow allocs: 0 refill waste: 736B alloc: 0.00058 2323KB refills: 1 waste 64.9% gc: 30880B slow: 0B fast: 0B TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 2KB slow allocs: 0 refill waste: 40B alloc: 0.00003 141KB refills: 3 waste 18.9% gc: 1560B slow: 80B fast: 0B TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.98287 3965429KB refills: 2917 waste 0.0% gc: 0B slow: 126104B fast: 0B TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 230KB slow allocs: 0 refill waste: 3680B alloc: 0.00286 11521KB refills: 3 waste 16.8% gc: 118472B slow: 184B fast: 0B TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 372KB slow allocs: 0 refill waste: 5952B alloc: 0.00461 18616KB refills: 2 waste 46.2% gc: 352504B slow: 104B fast: 0B TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 15KB slow allocs: 1 refill waste: 248B alloc: 0.00020 796KB refills: 3 waste 32.5% gc: 15528B slow: 360B fast: 0B TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 19KB slow allocs: 1 refill waste: 304B alloc: 0.00024 964KB refills: 2 waste 32.3% gc: 12440B slow: 304B fast: 0B TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 764KB slow allocs: 0 refill waste: 12224B alloc: 0.00947 38218KB refills: 1 waste 17.1% gc: 133552B slow: 0B fast: 0B TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 2KB slow allocs: 58 refill waste: 40B alloc: 0.00003 140KB refills: 73 waste 2.8% gc: 2416B slow: 3384B fast: 0B TLAB totals: thrds: 14 refills: 3059 max: 2917 slow allocs: 70 max 58 waste: 0.0% gc: 2814464B max: 2042560B slow: 132264B max: 126104B fast: 0B max: 0B Desired survivor size 268435456 bytes, new threshold 15 (max 15) - age 1: 397432 bytes, 397432 total - age 2: 112808 bytes, 510240 total - age 3: 14208 bytes, 524448 total - age 4: 41328 bytes, 565776 total - age 5: 689344 bytes, 1255120 total - age 6: 730968 bytes, 1986088 total - age 7: 142800480 bytes, 144786568 total 68082.830: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 6380, predicted base time: 11.57 ms, remaining time: 138.43 ms, target pause time: 150.00 ms] 68082.830: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 985 regions, survivors: 39 regions, predicted young region time: 119.81 ms] 68082.830: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 985 regions, survivors: 39 regions, old: 0 regions, predicted pause time: 131.38 ms, target pause time: 150.00 ms] 2017-01-25T14:10:38.474+0530: 68082.926: [SoftReference, 0 refs, 0.0013481 secs]2017-01-25T14:10:38.475+0530: 68082.928: [WeakReference, 0 refs, 0.0007696 secs]2017-01-25T14:10:38.476+0530: 68082.928: [FinalReference, 3 refs, 0.0007260 secs]2017-01-25T14:10:38.477+0530: 68082.929: [PhantomReference, 0 refs, 0 refs, 0.0014520 secs]2017-01-25T14:10:38.478+0530: 68082.931: [JNI Weak Reference, 0.0000179 secs], 0.1038195 secs] [Parallel Time: 95.8 ms, GC Workers: 20] [GC Worker Start (ms): Min: 68082829.8, Avg: 68082832.5, Max: 68082846.4, Diff: 16.6] [Ext Root Scanning (ms): Min: 0.0, Avg: 0.3, Max: 1.1, Diff: 1.1, Sum: 6.2] [Update RS (ms): Min: 0.0, Avg: 0.9, Max: 6.8, Diff: 6.8, Sum: 17.8] [Processed Buffers: Min: 0, Avg: 1.9, Max: 5, Diff: 5, Sum: 37] [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 1.8] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 68.4, Avg: 81.1, Max: 87.1, Diff: 18.7, Sum: 1622.2] [Termination (ms): Min: 2.9, Avg: 6.2, Max: 16.5, Diff: 13.6, Sum: 124.3] [Termination Attempts: Min: 1517, Avg: 2632.0, Max: 3026, Diff: 1509, Sum: 52640] [GC Worker Other (ms): Min: 0.0, Avg: 2.9, Max: 4.5, Diff: 4.4, Sum: 57.4] [GC Worker Total (ms): Min: 74.9, Avg: 91.5, Max: 95.7, Diff: 20.9, Sum: 1829.7] [GC Worker End (ms): Min: 68082921.1, Avg: 68082924.0, Max: 68082925.6, Diff: 4.4] [Code Root Fixup: 0.1 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.5 ms] [Other: 7.4 ms] [Choose CSet: 0.1 ms] [Ref Proc: 5.0 ms] [Ref Enq: 0.4 ms] [Redirty Cards: 0.3 ms] [Humongous Register: 0.1 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 1.4 ms] [Eden: 3940.0M(3940.0M)->0.0B(3940.0M) Survivors: 156.0M->156.0M Heap: 5164.0M(8192.0M)->1226.0M(8192.0M)] [Times: user=3.49 sys=0.00, real=0.11 secs] 2017-01-25T14:10:38.481+0530: 68082.933: Total time for which application threads were stopped: 0.1042423 seconds, Stopping threads took: 0.0000412 seconds 2017-01-25T14:10:38.481+0530: 68082.934: Application time: 0.0001033 seconds 2017-01-25T14:10:38.481+0530: 68082.934: Total time for which application threads were stopped: 0.0003928 seconds, Stopping threads took: 0.0000939 seconds 2017-01-25T14:10:38.978+0530: 68083.431: Application time: 0.4969123 seconds 2017-01-25T14:10:38.979+0530: 68083.431: Total time for which application threads were stopped: 0.0004010 seconds, Stopping threads took: 0.0000855 seconds 2017-01-25T14:10:39.933+0530: 68084.386: Application time: 0.9545832 seconds 2017-01-25T14:10:39.934+0530: 68084.386: [GC pause (G1 Evacuation Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] desired_size: 4KB slow allocs: 3 refill waste: 64B alloc: 0.00005 210KB refills: 13 waste 7.0% gc: 3512B slow: 384B fast: 0B TLAB: gc thread: 0x00007f15f8114800 [id: 199] desired_size: 1655KB slow allocs: 0 refill waste: 26488B alloc: 0.02052 82776KB refills: 2 waste 48.0% gc: 1626104B slow: 56B fast: 0B TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 7KB refills: 1 waste 14.0% gc: 288B slow: 0B fast: 0B TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.98542 3975742KB refills: 2952 waste 0.0% gc: 0B slow: 125280B fast: 0B TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 503KB slow allocs: 0 refill waste: 8056B alloc: 0.00625 25200KB refills: 1 waste 77.0% gc: 397600B slow: 0B fast: 0B TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow allocs: 26 refill waste: 48B alloc: 0.00004 169KB refills: 29 waste 2.2% gc: 1064B slow: 1120B fast: 0B TLAB totals: thrds: 6 refills: 2998 max: 2952 slow allocs: 29 max 26 waste: 0.0% gc: 2028568B max: 1626104B slow: 126840B max: 125280B fast: 0B max: 0B Desired survivor size 268435456 bytes, new threshold 15 (max 15) - age 1: 319896 bytes, 319896 total - age 2: 388232 bytes, 708128 total - age 3: 112808 bytes, 820936 total - age 4: 14208 bytes, 835144 total - age 5: 41328 bytes, 876472 total - age 6: 689344 bytes, 1565816 total - age 7: 730968 bytes, 2296784 total - age 8: 142800480 bytes, 145097264 total 68084.386: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 5401, predicted base time: 12.96 ms, remaining time: 137.04 ms, target pause time: 150.00 ms] 68084.386: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 985 regions, survivors: 39 regions, predicted young region time: 110.43 ms] 68084.386: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 985 regions, survivors: 39 regions, old: 0 regions, predicted pause time: 123.39 ms, target pause time: 150.00 ms] 2017-01-25T14:10:40.040+0530: 68084.492: [SoftReference, 0 refs, 0.0018914 secs]2017-01-25T14:10:40.042+0530: 68084.494: [WeakReference, 0 refs, 0.0009650 secs]2017-01-25T14:10:40.043+0530: 68084.495: [FinalReference, 1 refs, 0.0010309 secs]2017-01-25T14:10:40.044+0530: 68084.496: [PhantomReference, 0 refs, 0 refs, 0.0014095 secs]2017-01-25T14:10:40.045+0530: 68084.498: [JNI Weak Reference, 0.0000247 secs], 0.1142976 secs] [Parallel Time: 105.5 ms, GC Workers: 20] [GC Worker Start (ms): Min: 68084386.4, Avg: 68084388.9, Max: 68084402.4, Diff: 16.0] [Ext Root Scanning (ms): Min: 0.0, Avg: 0.3, Max: 1.1, Diff: 1.1, Sum: 6.4] [Update RS (ms): Min: 0.0, Avg: 0.2, Max: 0.4, Diff: 0.4, Sum: 3.7] [Processed Buffers: Min: 0, Avg: 1.4, Max: 7, Diff: 7, Sum: 29] [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 1.7] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 73.1, Avg: 92.3, Max: 97.6, Diff: 24.5, Sum: 1845.1] [Termination (ms): Min: 3.9, Avg: 9.6, Max: 20.0, Diff: 16.0, Sum: 192.4] [Termination Attempts: Min: 2087, Avg: 3497.8, Max: 3956, Diff: 1869, Sum: 69955] [GC Worker Other (ms): Min: 0.0, Avg: 0.2, Max: 0.3, Diff: 0.3, Sum: 3.2] [GC Worker Total (ms): Min: 89.1, Avg: 102.6, Max: 105.2, Diff: 16.2, Sum: 2052.5] [GC Worker End (ms): Min: 68084491.4, Avg: 68084491.5, Max: 68084491.7, Diff: 0.3] [Code Root Fixup: 0.1 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.5 ms] [Other: 8.3 ms] [Choose CSet: 0.0 ms] [Ref Proc: 5.9 ms] [Ref Enq: 0.4 ms] [Redirty Cards: 0.3 ms] [Humongous Register: 0.1 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 1.4 ms] [Eden: 3940.0M(3940.0M)->0.0B(3940.0M) Survivors: 156.0M->156.0M Heap: 5166.0M(8192.0M)->1224.0M(8192.0M)] [Times: user=3.16 sys=0.00, real=0.11 secs] 2017-01-25T14:10:40.048+0530: 68084.501: Total time for which application threads were stopped: 0.1147353 seconds, Stopping threads took: 0.0000420 seconds 2017-01-25T14:10:40.048+0530: 68084.501: Application time: 0.0001454 seconds 2017-01-25T14:10:40.049+0530: 68084.501: Total time for which application threads were stopped: 0.0003603 seconds, Stopping threads took: 0.0000723 seconds 2017-01-25T14:10:40.962+0530: 68085.415: Application time: 0.9135759 seconds 2017-01-25T14:10:40.962+0530: 68085.415: Total time for which application threads were stopped: 0.0002860 seconds, Stopping threads took: 0.0000639 seconds 2017-01-25T14:10:41.048+0530: 68085.501: Application time: 0.0858210 seconds 2017-01-25T14:10:41.049+0530: 68085.501: Total time for which application threads were stopped: 0.0003932 seconds, Stopping threads took: 0.0000801 seconds 2017-01-25T14:10:41.139+0530: 68085.591: Application time: 0.0900708 seconds 2017-01-25T14:10:41.139+0530: 68085.592: Total time for which application threads were stopped: 0.0006986 seconds, Stopping threads took: 0.0000592 seconds 2017-01-25T14:10:41.511+0530: 68085.964: Application time: 0.3716063 seconds 2017-01-25T14:10:41.511+0530: 68085.964: [GC pause (G1 Evacuation Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] desired_size: 3KB slow allocs: 18 refill waste: 48B alloc: 0.00004 155KB refills: 31 waste 3.1% gc: 1928B slow: 1152B fast: 0B TLAB: gc thread: 0x00007f15f8114800 [id: 199] desired_size: 1076KB slow allocs: 0 refill waste: 17216B alloc: 0.01334 53823KB refills: 1 waste 63.6% gc: 701328B slow: 0B fast: 0B TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00000 5KB refills: 2 waste 16.1% gc: 648B slow: 16B fast: 0B TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 47KB slow allocs: 0 refill waste: 752B alloc: 0.00059 2361KB refills: 1 waste 57.1% gc: 27632B slow: 0B fast: 0B TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 30KB slow allocs: 0 refill waste: 480B alloc: 0.00038 1520KB refills: 2 waste 31.7% gc: 19672B slow: 48B fast: 0B TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00002 94KB refills: 2 waste 24.0% gc: 952B slow: 40B fast: 0B TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.99031 3995472KB refills: 2948 waste 0.0% gc: 0B slow: 125952B fast: 0B TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 152KB slow allocs: 0 refill waste: 2440B alloc: 0.00189 7643KB refills: 3 waste 27.9% gc: 130944B slow: 80B fast: 0B TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 247KB slow allocs: 0 refill waste: 3952B alloc: 0.00306 12355KB refills: 1 waste 15.1% gc: 38112B slow: 0B fast: 0B TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 10KB slow allocs: 0 refill waste: 168B alloc: 0.00013 526KB refills: 2 waste 49.9% gc: 10728B slow: 24B fast: 0B TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 12KB slow allocs: 0 refill waste: 200B alloc: 0.00016 635KB refills: 2 waste 4.7% gc: 1200B slow: 32B fast: 0B TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 332KB slow allocs: 0 refill waste: 5312B alloc: 0.00411 16602KB refills: 3 waste 26.0% gc: 264824B slow: 704B fast: 0B TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 2KB slow allocs: 66 refill waste: 40B alloc: 0.00004 145KB refills: 67 waste 2.5% gc: 2048B slow: 3016B fast: 0B TLAB: gc thread: 0x00007f170d35a000 [id: 85] desired_size: 24KB slow allocs: 0 refill waste: 392B alloc: 0.00031 1231KB refills: 3 waste 29.0% gc: 21808B slow: 112B fast: 0B TLAB totals: thrds: 14 refills: 3068 max: 2948 slow allocs: 85 max 66 waste: 0.0% gc: 1221824B max: 701328B slow: 131176B max: 125952B fast: 0B max: 0B Desired survivor size 268435456 bytes, new threshold 15 (max 15) - age 1: 18656 bytes, 18656 total - age 2: 262056 bytes, 280712 total - age 3: 388232 bytes, 668944 total - age 4: 112808 bytes, 781752 total - age 5: 14208 bytes, 795960 total - age 6: 41328 bytes, 837288 total - age 7: 689344 bytes, 1526632 total - age 8: 730928 bytes, 2257560 total - age 9: 142800480 bytes, 145058040 total 68085.964: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 5972, predicted base time: 12.63 ms, remaining time: 137.37 ms, target pause time: 150.00 ms] 68085.964: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 985 regions, survivors: 39 regions, predicted young region time: 108.23 ms] 68085.964: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 985 regions, survivors: 39 regions, old: 0 regions, predicted pause time: 120.86 ms, target pause time: 150.00 ms] 2017-01-25T14:10:41.620+0530: 68086.072: [SoftReference, 0 refs, 0.0012167 secs]2017-01-25T14:10:41.621+0530: 68086.074: [WeakReference, 0 refs, 0.0009153 secs]2017-01-25T14:10:41.622+0530: 68086.075: [FinalReference, 4 refs, 0.0006707 secs]2017-01-25T14:10:41.623+0530: 68086.075: [PhantomReference, 0 refs, 0 refs, 0.0014168 secs]2017-01-25T14:10:41.624+0530: 68086.077: [JNI Weak Reference, 0.0000340 secs], 0.1154024 secs] [Parallel Time: 107.2 ms, GC Workers: 20] [GC Worker Start (ms): Min: 68085964.2, Avg: 68085964.4, Max: 68085964.9, Diff: 0.8] [Ext Root Scanning (ms): Min: 0.0, Avg: 0.4, Max: 1.0, Diff: 1.0, Sum: 8.8] [Update RS (ms): Min: 0.0, Avg: 2.2, Max: 13.5, Diff: 13.5, Sum: 44.1] [Processed Buffers: Min: 0, Avg: 1.8, Max: 6, Diff: 6, Sum: 35] [Scan RS (ms): Min: 0.1, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.9] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 83.6, Avg: 95.1, Max: 100.9, Diff: 17.3, Sum: 1902.7] [Termination (ms): Min: 4.8, Avg: 8.6, Max: 9.4, Diff: 4.7, Sum: 171.2] [Termination Attempts: Min: 2131, Avg: 3342.5, Max: 3658, Diff: 1527, Sum: 66850] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.8] [GC Worker Total (ms): Min: 106.1, Avg: 106.6, Max: 107.0, Diff: 0.9, Sum: 2132.6] [GC Worker End (ms): Min: 68086071.0, Avg: 68086071.1, Max: 68086071.2, Diff: 0.2] [Code Root Fixup: 0.2 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.5 ms] [Other: 7.5 ms] [Choose CSet: 0.0 ms] [Ref Proc: 5.1 ms] [Ref Enq: 0.4 ms] [Redirty Cards: 0.3 ms] [Humongous Register: 0.0 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 1.4 ms] [Eden: 3940.0M(3940.0M)->0.0B(3924.0M) Survivors: 156.0M->172.0M Heap: 5164.0M(8192.0M)->1240.0M(8192.0M)] [Times: user=4.39 sys=0.00, real=0.11 secs] 2017-01-25T14:10:41.627+0530: 68086.079: Total time for which application threads were stopped: 0.1158356 seconds, Stopping threads took: 0.0000512 seconds 2017-01-25T14:10:41.627+0530: 68086.080: Application time: 0.0000789 seconds 2017-01-25T14:10:41.627+0530: 68086.080: Total time for which application threads were stopped: 0.0003353 seconds, Stopping threads took: 0.0000796 seconds 2017-01-25T14:10:42.048+0530: 68086.501: Application time: 0.4211560 seconds 2017-01-25T14:10:42.049+0530: 68086.501: Total time for which application threads were stopped: 0.0003771 seconds, Stopping threads took: 0.0000731 seconds 2017-01-25T14:10:43.046+0530: 68087.499: Application time: 0.9971396 seconds 2017-01-25T14:10:43.046+0530: 68087.499: [GC pause (G1 Evacuation Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] desired_size: 2KB slow allocs: 7 refill waste: 40B alloc: 0.00003 136KB refills: 18 waste 4.1% gc: 1520B slow: 520B fast: 0B TLAB: gc thread: 0x00007f15f8114800 [id: 199] desired_size: 697KB slow allocs: 1 refill waste: 11152B alloc: 0.00868 34866KB refills: 4 waste 23.9% gc: 682392B slow: 120B fast: 0B TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 5KB refills: 1 waste 14.0% gc: 288B slow: 0B fast: 0B TLAB: gc thread: 0x00007f15906a1800 [id: 140] desired_size: 68KB slow allocs: 0 refill waste: 1096B alloc: 0.00085 3426KB refills: 3 waste 32.5% gc: 68328B slow: 96B fast: 0B TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 60KB slow allocs: 0 refill waste: 968B alloc: 0.00075 3030KB refills: 1 waste 8.7% gc: 5400B slow: 0B fast: 0B TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 20KB slow allocs: 0 refill waste: 328B alloc: 0.00026 1031KB refills: 2 waste 17.8% gc: 7472B slow: 40B fast: 0B TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 30KB slow allocs: 0 refill waste: 488B alloc: 0.00038 1529KB refills: 3 waste 24.6% gc: 22952B slow: 104B fast: 0B TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 19KB slow allocs: 0 refill waste: 312B alloc: 0.00025 995KB refills: 5 waste 5.9% gc: 5896B slow: 144B fast: 0B TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 2KB slow allocs: 3 refill waste: 32B alloc: 0.00002 61KB refills: 6 waste 2.5% gc: 128B slow: 176B fast: 0B TLAB: gc thread: 0x00007f159062b000 [id: 115] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 2KB refills: 1 waste 92.2% gc: 1904B slow: 0B fast: 0B TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.99345 3991844KB refills: 2929 waste 0.0% gc: 0B slow: 122976B fast: 0B TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 100KB slow allocs: 0 refill waste: 1600B alloc: 0.00124 5002KB refills: 11 waste 5.6% gc: 62288B slow: 1096B fast: 0B TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 159KB slow allocs: 0 refill waste: 2552B alloc: 0.00199 7998KB refills: 6 waste 7.3% gc: 70968B slow: 1264B fast: 0B TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 6KB slow allocs: 5 refill waste: 104B alloc: 0.00009 344KB refills: 8 waste 5.4% gc: 2712B slow: 360B fast: 0B TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 8KB slow allocs: 1 refill waste: 128B alloc: 0.00010 412KB refills: 7 waste 7.8% gc: 4352B slow: 224B fast: 0B TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 218KB slow allocs: 0 refill waste: 3488B alloc: 0.00271 10904KB refills: 9 waste 6.5% gc: 129144B slow: 2200B fast: 0B TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow allocs: 39 refill waste: 48B alloc: 0.00004 169KB refills: 32 waste 2.2% gc: 1080B slow: 1336B fast: 0B TLAB totals: thrds: 17 refills: 3046 max: 2929 slow allocs: 56 max 39 waste: 0.0% gc: 1066824B max: 682392B slow: 130656B max: 122976B fast: 0B max: 0B Desired survivor size 268435456 bytes, new threshold 15 (max 15) - age 1: 181488 bytes, 181488 total - age 2: 14864 bytes, 196352 total - age 3: 127184 bytes, 323536 total - age 4: 372256 bytes, 695792 total - age 5: 110328 bytes, 806120 total - age 6: 14208 bytes, 820328 total - age 7: 41328 bytes, 861656 total - age 8: 689344 bytes, 1551000 total - age 9: 730928 bytes, 2281928 total - age 10: 142800480 bytes, 145082408 total 68087.499: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 5519, predicted base time: 12.30 ms, remaining time: 137.70 ms, target pause time: 150.00 ms] 68087.499: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 981 regions, survivors: 43 regions, predicted young region time: 105.17 ms] 68087.499: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 981 regions, survivors: 43 regions, old: 0 regions, predicted pause time: 117.47 ms, target pause time: 150.00 ms] 2017-01-25T14:10:43.140+0530: 68087.593: [SoftReference, 0 refs, 0.0011365 secs]2017-01-25T14:10:43.141+0530: 68087.594: [WeakReference, 0 refs, 0.0008092 secs]2017-01-25T14:10:43.142+0530: 68087.595: [FinalReference, 11 refs, 0.0006691 secs]2017-01-25T14:10:43.143+0530: 68087.596: [PhantomReference, 0 refs, 0 refs, 0.0012762 secs]2017-01-25T14:10:43.144+0530: 68087.597: [JNI Weak Reference, 0.0000161 secs], 0.1008873 secs] [Parallel Time: 93.5 ms, GC Workers: 20] [GC Worker Start (ms): Min: 68087499.1, Avg: 68087500.4, Max: 68087509.3, Diff: 10.2] [Ext Root Scanning (ms): Min: 0.0, Avg: 0.3, Max: 1.2, Diff: 1.2, Sum: 6.6] [Update RS (ms): Min: 0.0, Avg: 0.2, Max: 0.4, Diff: 0.4, Sum: 4.2] [Processed Buffers: Min: 0, Avg: 1.9, Max: 6, Diff: 6, Sum: 38] [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 1.8] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 75.8, Avg: 84.6, Max: 88.8, Diff: 12.9, Sum: 1692.3] [Termination (ms): Min: 3.4, Avg: 6.6, Max: 7.9, Diff: 4.5, Sum: 131.6] [Termination Attempts: Min: 1502, Avg: 2767.8, Max: 3171, Diff: 1669, Sum: 55355] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 1.8] [GC Worker Total (ms): Min: 83.0, Avg: 91.9, Max: 93.2, Diff: 10.1, Sum: 1838.3] [GC Worker End (ms): Min: 68087592.2, Avg: 68087592.3, Max: 68087592.4, Diff: 0.2] [Code Root Fixup: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.7 ms] [Other: 6.7 ms] [Choose CSet: 0.0 ms] [Ref Proc: 4.3 ms] [Ref Enq: 0.3 ms] [Redirty Cards: 0.3 ms] [Humongous Register: 0.0 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 1.4 ms] [Eden: 3924.0M(3924.0M)->0.0B(3916.0M) Survivors: 172.0M->180.0M Heap: 5164.0M(8192.0M)->1248.0M(8192.0M)] [Times: user=3.45 sys=0.00, real=0.10 secs] 2017-01-25T14:10:43.147+0530: 68087.600: Total time for which application threads were stopped: 0.1013146 seconds, Stopping threads took: 0.0000407 seconds 2017-01-25T14:10:43.147+0530: 68087.600: Application time: 0.0001099 seconds 2017-01-25T14:10:43.148+0530: 68087.600: Total time for which application threads were stopped: 0.0002593 seconds, Stopping threads took: 0.0000683 seconds 2017-01-25T14:10:43.148+0530: 68087.600: Application time: 0.0000577 seconds 2017-01-25T14:10:43.148+0530: 68087.600: Total time for which application threads were stopped: 0.0000925 seconds, Stopping threads took: 0.0000199 seconds 2017-01-25T14:10:43.148+0530: 68087.600: Application time: 0.0000648 seconds 2017-01-25T14:10:43.148+0530: 68087.601: Total time for which application threads were stopped: 0.0002435 seconds, Stopping threads took: 0.0000844 seconds 2017-01-25T14:10:44.148+0530: 68088.601: Application time: 0.9998199 seconds 2017-01-25T14:10:44.148+0530: 68088.601: Total time for which application threads were stopped: 0.0003774 seconds, Stopping threads took: 0.0000807 seconds 2017-01-25T14:10:44.578+0530: 68089.031: Application time: 0.4299646 seconds 2017-01-25T14:10:44.579+0530: 68089.031: [GC pause (G1 Evacuation Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] desired_size: 2KB slow allocs: 23 refill waste: 32B alloc: 0.00003 106KB refills: 43 waste 2.6% gc: 936B slow: 1480B fast: 0B TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00000 4KB refills: 2 waste 16.1% gc: 648B slow: 16B fast: 0B TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 13KB slow allocs: 1 refill waste: 208B alloc: 0.00017 671KB refills: 6 waste 3.5% gc: 2696B slow: 176B fast: 0B TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 2KB slow allocs: 3 refill waste: 32B alloc: 0.00001 44KB refills: 6 waste 2.5% gc: 128B slow: 176B fast: 0B TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.99518 3990647KB refills: 2928 waste 0.0% gc: 0B slow: 124472B fast: 0B TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 70KB slow allocs: 0 refill waste: 1120B alloc: 0.00087 3500KB refills: 2 waste 13.7% gc: 19624B slow: 16B fast: 0B TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 110KB slow allocs: 0 refill waste: 1768B alloc: 0.00138 5541KB refills: 3 waste 18.9% gc: 62544B slow: 1888B fast: 0B TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 4KB slow allocs: 1 refill waste: 72B alloc: 0.00006 237KB refills: 2 waste 13.1% gc: 1240B slow: 32B fast: 0B TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 5KB slow allocs: 2 refill waste: 88B alloc: 0.00007 289KB refills: 2 waste 49.9% gc: 5840B slow: 48B fast: 0B TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 152KB slow allocs: 4 refill waste: 2440B alloc: 0.00190 7637KB refills: 10 waste 7.7% gc: 119664B slow: 744B fast: 0B TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow allocs: 52 refill waste: 48B alloc: 0.00004 150KB refills: 68 waste 1.6% gc: 544B slow: 2896B fast: 0B TLAB totals: thrds: 11 refills: 3072 max: 2928 slow allocs: 87 max 52 waste: 0.0% gc: 213864B max: 119664B slow: 131944B max: 124472B fast: 0B max: 0B Desired survivor size 268435456 bytes, new threshold 15 (max 15) - age 1: 223936 bytes, 223936 total - age 2: 54448 bytes, 278384 total - age 3: 14720 bytes, 293104 total - age 4: 114344 bytes, 407448 total - age 5: 369856 bytes, 777304 total - age 6: 110328 bytes, 887632 total - age 7: 14208 bytes, 901840 total - age 8: 41328 bytes, 943168 total - age 9: 689344 bytes, 1632512 total - age 10: 730928 bytes, 2363440 total - age 11: 142800480 bytes, 145163920 total 68089.031: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 5691, predicted base time: 11.38 ms, remaining time: 138.62 ms, target pause time: 150.00 ms] 68089.031: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 979 regions, survivors: 45 regions, predicted young region time: 97.24 ms] 68089.031: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 979 regions, survivors: 45 regions, old: 0 regions, predicted pause time: 108.62 ms, target pause time: 150.00 ms] 2017-01-25T14:10:44.672+0530: 68089.125: [SoftReference, 0 refs, 0.0010475 secs]2017-01-25T14:10:44.673+0530: 68089.126: [WeakReference, 0 refs, 0.0007920 secs]2017-01-25T14:10:44.674+0530: 68089.127: [FinalReference, 0 refs, 0.0005465 secs]2017-01-25T14:10:44.675+0530: 68089.127: [PhantomReference, 0 refs, 0 refs, 0.0014752 secs]2017-01-25T14:10:44.676+0530: 68089.129: [JNI Weak Reference, 0.0000168 secs], 0.1004490 secs] [Parallel Time: 93.2 ms, GC Workers: 20] [GC Worker Start (ms): Min: 68089031.4, Avg: 68089033.8, Max: 68089046.3, Diff: 15.0] [Ext Root Scanning (ms): Min: 0.0, Avg: 0.3, Max: 1.1, Diff: 1.0, Sum: 6.0] [Update RS (ms): Min: 0.0, Avg: 0.2, Max: 0.4, Diff: 0.4, Sum: 3.8] [Processed Buffers: Min: 0, Avg: 1.6, Max: 7, Diff: 7, Sum: 33] [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 1.8] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 68.9, Avg: 82.3, Max: 85.7, Diff: 16.8, Sum: 1645.3] [Termination (ms): Min: 3.9, Avg: 7.5, Max: 8.8, Diff: 4.9, Sum: 149.8] [Termination Attempts: Min: 1780, Avg: 2841.3, Max: 3167, Diff: 1387, Sum: 56826] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.5] [GC Worker Total (ms): Min: 77.8, Avg: 90.5, Max: 92.8, Diff: 15.0, Sum: 1809.3] [GC Worker End (ms): Min: 68089124.1, Avg: 68089124.2, Max: 68089124.4, Diff: 0.2] [Code Root Fixup: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.5 ms] [Other: 6.8 ms] [Choose CSet: 0.0 ms] [Ref Proc: 4.3 ms] [Ref Enq: 0.4 ms] [Redirty Cards: 0.3 ms] [Humongous Register: 0.1 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 1.4 ms] [Eden: 3916.0M(3916.0M)->0.0B(3948.0M) Survivors: 180.0M->148.0M Heap: 5164.0M(8192.0M)->1218.0M(8192.0M)] [Times: user=3.48 sys=0.00, real=0.10 secs] 2017-01-25T14:10:44.679+0530: 68089.132: Total time for which application threads were stopped: 0.1008638 seconds, Stopping threads took: 0.0000409 seconds 2017-01-25T14:10:45.148+0530: 68089.601: Application time: 0.4690297 seconds 2017-01-25T14:10:45.149+0530: 68089.601: Total time for which application threads were stopped: 0.0003481 seconds, Stopping threads took: 0.0000565 seconds 2017-01-25T14:10:45.149+0530: 68089.601: Application time: 0.0002611 seconds 2017-01-25T14:10:45.149+0530: 68089.602: Total time for which application threads were stopped: 0.0004220 seconds, Stopping threads took: 0.0000415 seconds 2017-01-25T14:10:45.963+0530: 68090.415: Application time: 0.8134945 seconds 2017-01-25T14:10:45.963+0530: 68090.416: Total time for which application threads were stopped: 0.0002944 seconds, Stopping threads took: 0.0000764 seconds 2017-01-25T14:10:46.149+0530: 68090.601: Application time: 0.1857148 seconds 2017-01-25T14:10:46.149+0530: 68090.602: Total time for which application threads were stopped: 0.0003960 seconds, Stopping threads took: 0.0000857 seconds 2017-01-25T14:10:46.161+0530: 68090.613: Application time: 0.0115789 seconds 2017-01-25T14:10:46.161+0530: 68090.614: [GC pause (G1 Evacuation Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] desired_size: 2KB slow allocs: 25 refill waste: 32B alloc: 0.00003 106KB refills: 43 waste 2.5% gc: 928B slow: 1352B fast: 0B TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00000 4KB refills: 2 waste 16.1% gc: 648B slow: 16B fast: 0B TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 13KB slow allocs: 0 refill waste: 216B alloc: 0.00017 682KB refills: 1 waste 36.1% gc: 5040B slow: 0B fast: 0B TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00001 33KB refills: 1 waste 24.8% gc: 512B slow: 0B fast: 0B TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.99667 4029298KB refills: 2956 waste 0.0% gc: 0B slow: 124400B fast: 0B TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 3KB slow allocs: 2 refill waste: 56B alloc: 0.00005 193KB refills: 3 waste 29.1% gc: 3352B slow: 88B fast: 0B TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 108KB slow allocs: 0 refill waste: 1736B alloc: 0.00134 5434KB refills: 2 waste 41.4% gc: 91848B slow: 328B fast: 0B TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow allocs: 43 refill waste: 56B alloc: 0.00004 177KB refills: 54 waste 3.0% gc: 3568B slow: 2328B fast: 0B TLAB: gc thread: 0x00007f170d35a000 [id: 85] desired_size: 16KB slow allocs: 0 refill waste: 256B alloc: 0.00020 811KB refills: 3 waste 11.3% gc: 5552B slow: 64B fast: 0B TLAB totals: thrds: 9 refills: 3065 max: 2956 slow allocs: 71 max 43 waste: 0.0% gc: 111448B max: 91848B slow: 128576B max: 124400B fast: 0B max: 0B Desired survivor size 268435456 bytes, new threshold 15 (max 15) - age 1: 58040 bytes, 58040 total - age 2: 216064 bytes, 274104 total - age 3: 54304 bytes, 328408 total - age 4: 14720 bytes, 343128 total - age 5: 114344 bytes, 457472 total - age 6: 369856 bytes, 827328 total - age 7: 110328 bytes, 937656 total - age 8: 14208 bytes, 951864 total - age 9: 41328 bytes, 993192 total - age 10: 689344 bytes, 1682536 total - age 11: 730928 bytes, 2413464 total - age 12: 142800480 bytes, 145213944 total 68090.614: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 5688, predicted base time: 10.90 ms, remaining time: 139.10 ms, target pause time: 150.00 ms] 68090.614: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 987 regions, survivors: 37 regions, predicted young region time: 92.86 ms] 68090.614: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 987 regions, survivors: 37 regions, old: 0 regions, predicted pause time: 103.76 ms, target pause time: 150.00 ms] 2017-01-25T14:10:46.255+0530: 68090.707: [SoftReference, 0 refs, 0.0009716 secs]2017-01-25T14:10:46.256+0530: 68090.708: [WeakReference, 0 refs, 0.0005869 secs]2017-01-25T14:10:46.256+0530: 68090.709: [FinalReference, 0 refs, 0.0006482 secs]2017-01-25T14:10:46.257+0530: 68090.710: [PhantomReference, 0 refs, 0 refs, 0.0014339 secs]2017-01-25T14:10:46.258+0530: 68090.711: [JNI Weak Reference, 0.0000166 secs], 0.1000319 secs] [Parallel Time: 93.2 ms, GC Workers: 20] [GC Worker Start (ms): Min: 68090613.7, Avg: 68090615.8, Max: 68090626.3, Diff: 12.6] [Ext Root Scanning (ms): Min: 0.0, Avg: 0.5, Max: 1.1, Diff: 1.1, Sum: 9.1] [Update RS (ms): Min: 0.0, Avg: 0.2, Max: 0.4, Diff: 0.4, Sum: 3.8] [Processed Buffers: Min: 0, Avg: 1.6, Max: 5, Diff: 5, Sum: 32] [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.2, Sum: 2.8] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 72.9, Avg: 83.4, Max: 88.4, Diff: 15.5, Sum: 1667.3] [Termination (ms): Min: 3.2, Avg: 6.6, Max: 7.2, Diff: 4.0, Sum: 131.4] [Termination Attempts: Min: 1661, Avg: 2894.8, Max: 3138, Diff: 1477, Sum: 57895] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.6] [GC Worker Total (ms): Min: 80.2, Avg: 90.9, Max: 93.0, Diff: 12.7, Sum: 1817.0] [GC Worker End (ms): Min: 68090706.5, Avg: 68090706.6, Max: 68090706.7, Diff: 0.2] [Code Root Fixup: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.5 ms] [Other: 6.3 ms] [Choose CSet: 0.0 ms] [Ref Proc: 4.0 ms] [Ref Enq: 0.4 ms] [Redirty Cards: 0.3 ms] [Humongous Register: 0.0 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 1.4 ms] [Eden: 3948.0M(3948.0M)->0.0B(3940.0M) Survivors: 148.0M->156.0M Heap: 5166.0M(8192.0M)->1224.0M(8192.0M)] [Times: user=3.58 sys=0.00, real=0.10 secs] 2017-01-25T14:10:46.261+0530: 68090.714: Total time for which application threads were stopped: 0.1004085 seconds, Stopping threads took: 0.0000355 seconds 2017-01-25T14:10:47.149+0530: 68091.601: Application time: 0.8877422 seconds 2017-01-25T14:10:47.149+0530: 68091.602: Total time for which application threads were stopped: 0.0003494 seconds, Stopping threads took: 0.0000623 seconds 2017-01-25T14:10:47.675+0530: 68092.127: Application time: 0.5255223 seconds 2017-01-25T14:10:47.675+0530: 68092.128: [GC pause (G1 Evacuation Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] desired_size: 2KB slow allocs: 13 refill waste: 32B alloc: 0.00003 104KB refills: 22 waste 4.1% gc: 1000B slow: 912B fast: 0B TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 4KB refills: 1 waste 14.0% gc: 288B slow: 0B fast: 0B TLAB: gc thread: 0x00007f15906a1800 [id: 140] desired_size: 45KB slow allocs: 0 refill waste: 720B alloc: 0.00056 2261KB refills: 1 waste 50.8% gc: 23520B slow: 0B fast: 0B TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 8KB slow allocs: 3 refill waste: 136B alloc: 0.00011 442KB refills: 17 waste 5.7% gc: 8328B slow: 488B fast: 0B TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 20KB slow allocs: 0 refill waste: 320B alloc: 0.00025 1016KB refills: 1 waste 21.3% gc: 4432B slow: 0B fast: 0B TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00001 22KB refills: 4 waste 23.6% gc: 1864B slow: 88B fast: 0B TLAB: gc thread: 0x00007f159005b000 [id: 114] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 1KB refills: 1 waste 98.4% gc: 2032B slow: 0B fast: 0B TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.99777 4025582KB refills: 2945 waste 0.0% gc: 0B slow: 127672B fast: 0B TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 46KB slow allocs: 0 refill waste: 744B alloc: 0.00058 2328KB refills: 7 waste 10.3% gc: 34368B slow: 160B fast: 0B TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 73KB slow allocs: 2 refill waste: 1176B alloc: 0.00092 3695KB refills: 8 waste 8.4% gc: 50640B slow: 224B fast: 0B TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 3KB slow allocs: 4 refill waste: 48B alloc: 0.00004 159KB refills: 7 waste 14.4% gc: 3104B slow: 152B fast: 0B TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 2KB slow allocs: 9 refill waste: 40B alloc: 0.00003 127KB refills: 10 waste 5.4% gc: 984B slow: 416B fast: 0B TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 71KB slow allocs: 9 refill waste: 1144B alloc: 0.00089 3575KB refills: 13 waste 5.5% gc: 51144B slow: 1664B fast: 0B TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow allocs: 16 refill waste: 56B alloc: 0.00005 185KB refills: 28 waste 4.1% gc: 2944B slow: 1448B fast: 0B TLAB totals: thrds: 14 refills: 3065 max: 2945 slow allocs: 56 max 16 waste: 0.0% gc: 184648B max: 51144B slow: 133224B max: 127672B fast: 0B max: 0B Desired survivor size 268435456 bytes, new threshold 15 (max 15) - age 1: 19576 bytes, 19576 total - age 2: 52640 bytes, 72216 total - age 3: 215144 bytes, 287360 total - age 4: 54240 bytes, 341600 total - age 5: 14720 bytes, 356320 total - age 6: 114344 bytes, 470664 total - age 7: 369856 bytes, 840520 total - age 8: 110328 bytes, 950848 total - age 9: 14208 bytes, 965056 total - age 10: 41328 bytes, 1006384 total - age 11: 689344 bytes, 1695728 total - age 12: 730928 bytes, 2426656 total - age 13: 142800480 bytes, 145227136 total 68092.128: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 5988, predicted base time: 10.41 ms, remaining time: 139.59 ms, target pause time: 150.00 ms] 68092.128: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 985 regions, survivors: 39 regions, predicted young region time: 91.70 ms] 68092.128: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 985 regions, survivors: 39 regions, old: 0 regions, predicted pause time: 102.11 ms, target pause time: 150.00 ms] 2017-01-25T14:10:47.753+0530: 68092.205: [SoftReference, 0 refs, 0.0011755 secs]2017-01-25T14:10:47.754+0530: 68092.207: [WeakReference, 0 refs, 0.0008223 secs]2017-01-25T14:10:47.755+0530: 68092.207: [FinalReference, 0 refs, 0.0006542 secs]2017-01-25T14:10:47.756+0530: 68092.208: [PhantomReference, 0 refs, 0 refs, 0.0014601 secs]2017-01-25T14:10:47.757+0530: 68092.210: [JNI Weak Reference, 0.0000167 secs], 0.0847489 secs] [Parallel Time: 77.3 ms, GC Workers: 20] [GC Worker Start (ms): Min: 68092127.8, Avg: 68092130.2, Max: 68092142.4, Diff: 14.5] [Ext Root Scanning (ms): Min: 0.0, Avg: 0.4, Max: 1.4, Diff: 1.4, Sum: 7.6] [Update RS (ms): Min: 0.0, Avg: 0.2, Max: 0.4, Diff: 0.4, Sum: 4.0] [Processed Buffers: Min: 0, Avg: 1.8, Max: 7, Diff: 7, Sum: 36] [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 2.0] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 57.4, Avg: 69.4, Max: 73.9, Diff: 16.6, Sum: 1387.9] [Termination (ms): Min: 1.8, Avg: 4.5, Max: 5.2, Diff: 3.4, Sum: 90.5] [Termination Attempts: Min: 1157, Avg: 2821.8, Max: 3156, Diff: 1999, Sum: 56436] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.2, Sum: 3.0] [GC Worker Total (ms): Min: 62.5, Avg: 74.7, Max: 77.2, Diff: 14.6, Sum: 1494.9] [GC Worker End (ms): Min: 68092204.8, Avg: 68092204.9, Max: 68092205.0, Diff: 0.2] [Code Root Fixup: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.5 ms] [Other: 6.8 ms] [Choose CSet: 0.0 ms] [Ref Proc: 4.5 ms] [Ref Enq: 0.4 ms] [Redirty Cards: 0.3 ms] [Humongous Register: 0.0 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 1.4 ms] [Eden: 3940.0M(3940.0M)->0.0B(3944.0M) Survivors: 156.0M->152.0M Heap: 5164.0M(8192.0M)->1222.0M(8192.0M)] [Times: user=0.00 sys=0.00, real=0.09 secs] 2017-01-25T14:10:47.760+0530: 68092.212: Total time for which application threads were stopped: 0.0851593 seconds, Stopping threads took: 0.0000402 seconds 2017-01-25T14:10:48.149+0530: 68092.602: Application time: 0.3890779 seconds 2017-01-25T14:10:48.149+0530: 68092.602: Total time for which application threads were stopped: 0.0003716 seconds, Stopping threads took: 0.0000733 seconds 2017-01-25T14:10:49.149+0530: 68093.602: Application time: 0.9998228 seconds 2017-01-25T14:10:49.150+0530: 68093.602: Total time for which application threads were stopped: 0.0003580 seconds, Stopping threads took: 0.0000655 seconds 2017-01-25T14:10:49.211+0530: 68093.663: Application time: 0.0611931 seconds 2017-01-25T14:10:49.211+0530: 68093.664: [GC pause (G1 Evacuation Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] desired_size: 2KB slow allocs: 26 refill waste: 32B alloc: 0.00002 86KB refills: 43 waste 2.8% gc: 1064B slow: 1408B fast: 0B TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00000 3KB refills: 2 waste 16.1% gc: 648B slow: 16B fast: 0B TLAB: gc thread: 0x00007f15906a1800 [id: 140] desired_size: 29KB slow allocs: 0 refill waste: 464B alloc: 0.00036 1471KB refills: 1 waste 1.9% gc: 560B slow: 0B fast: 0B TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 39KB slow allocs: 0 refill waste: 632B alloc: 0.00049 1980KB refills: 2 waste 5.0% gc: 3992B slow: 24B fast: 0B TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 6KB slow allocs: 3 refill waste: 104B alloc: 0.00008 342KB refills: 21 waste 2.5% gc: 3080B slow: 624B fast: 0B TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 9KB slow allocs: 6 refill waste: 144B alloc: 0.00012 469KB refills: 60 waste 0.5% gc: 1208B slow: 1952B fast: 0B TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 17KB refills: 5 waste 9.2% gc: 840B slow: 112B fast: 0B TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.99834 4031960KB refills: 2936 waste 0.0% gc: 0B slow: 127024B fast: 0B TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 32KB slow allocs: 0 refill waste: 512B alloc: 0.00040 1612KB refills: 29 waste 3.3% gc: 29896B slow: 1424B fast: 0B TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 50KB slow allocs: 13 refill waste: 808B alloc: 0.00063 2539KB refills: 34 waste 2.8% gc: 47744B slow: 2296B fast: 0B TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 2KB slow allocs: 9 refill waste: 32B alloc: 0.00003 110KB refills: 24 waste 4.9% gc: 1936B slow: 728B fast: 0B TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 2KB slow allocs: 8 refill waste: 32B alloc: 0.00002 92KB refills: 19 waste 6.6% gc: 2000B slow: 584B fast: 0B TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 53KB slow allocs: 33 refill waste: 856B alloc: 0.00067 2686KB refills: 47 waste 0.7% gc: 12280B slow: 5760B fast: 0B TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow allocs: 47 refill waste: 48B alloc: 0.00004 157KB refills: 66 waste 1.3% gc: 120B slow: 2736B fast: 0B TLAB totals: thrds: 14 refills: 3289 max: 2936 slow allocs: 146 max 47 waste: 0.0% gc: 105368B max: 47744B slow: 144688B max: 127024B fast: 0B max: 0B Desired survivor size 268435456 bytes, new threshold 15 (max 15) - age 1: 105064 bytes, 105064 total - age 2: 10968 bytes, 116032 total - age 3: 52184 bytes, 168216 total - age 4: 207256 bytes, 375472 total - age 5: 51760 bytes, 427232 total - age 6: 14720 bytes, 441952 total - age 7: 114320 bytes, 556272 total - age 8: 369856 bytes, 926128 total - age 9: 110328 bytes, 1036456 total - age 10: 14208 bytes, 1050664 total - age 11: 41328 bytes, 1091992 total - age 12: 689344 bytes, 1781336 total - age 13: 730928 bytes, 2512264 total - age 14: 142800480 bytes, 145312744 total 68093.664: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 4729, predicted base time: 10.07 ms, remaining time: 139.93 ms, target pause time: 150.00 ms] 68093.664: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 986 regions, survivors: 38 regions, predicted young region time: 85.68 ms] 68093.664: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 986 regions, survivors: 38 regions, old: 0 regions, predicted pause time: 95.75 ms, target pause time: 150.00 ms] 2017-01-25T14:10:49.311+0530: 68093.764: [SoftReference, 0 refs, 0.0010888 secs]2017-01-25T14:10:49.312+0530: 68093.765: [WeakReference, 0 refs, 0.0005239 secs]2017-01-25T14:10:49.313+0530: 68093.765: [FinalReference, 0 refs, 0.0007986 secs]2017-01-25T14:10:49.313+0530: 68093.766: [PhantomReference, 0 refs, 0 refs, 0.0013813 secs]2017-01-25T14:10:49.315+0530: 68093.767: [JNI Weak Reference, 0.0000169 secs], 0.1066977 secs] [Parallel Time: 99.4 ms, GC Workers: 20] [GC Worker Start (ms): Min: 68093663.9, Avg: 68093665.4, Max: 68093670.5, Diff: 6.7] [Ext Root Scanning (ms): Min: 0.0, Avg: 0.3, Max: 1.1, Diff: 1.1, Sum: 6.0] [Update RS (ms): Min: 0.0, Avg: 1.7, Max: 10.3, Diff: 10.3, Sum: 33.7] [Processed Buffers: Min: 0, Avg: 1.6, Max: 7, Diff: 7, Sum: 31] [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 1.8] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 80.5, Avg: 88.3, Max: 94.1, Diff: 13.6, Sum: 1765.7] [Termination (ms): Min: 3.8, Avg: 7.2, Max: 13.7, Diff: 10.0, Sum: 144.1] [Termination Attempts: Min: 1844, Avg: 2613.2, Max: 2898, Diff: 1054, Sum: 52263] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.2, Sum: 3.0] [GC Worker Total (ms): Min: 92.5, Avg: 97.7, Max: 99.3, Diff: 6.8, Sum: 1954.3] [GC Worker End (ms): Min: 68093762.9, Avg: 68093763.1, Max: 68093763.2, Diff: 0.2] [Code Root Fixup: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.6 ms] [Other: 6.6 ms] [Choose CSet: 0.0 ms] [Ref Proc: 4.2 ms] [Ref Enq: 0.4 ms] [Redirty Cards: 0.3 ms] [Humongous Register: 0.1 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 1.4 ms] [Eden: 3944.0M(3944.0M)->0.0B(3936.0M) Survivors: 152.0M->160.0M Heap: 5166.0M(8192.0M)->1228.0M(8192.0M)] [Times: user=0.00 sys=0.00, real=0.11 secs] 2017-01-25T14:10:49.318+0530: 68093.770: Total time for which application threads were stopped: 0.1071290 seconds, Stopping threads took: 0.0000416 seconds 2017-01-25T14:10:50.149+0530: 68094.602: Application time: 0.8314793 seconds 2017-01-25T14:10:50.150+0530: 68094.602: Total time for which application threads were stopped: 0.0003901 seconds, Stopping threads took: 0.0000748 seconds 2017-01-25T14:10:50.809+0530: 68095.262: Application time: 0.6593091 seconds 2017-01-25T14:10:50.809+0530: 68095.262: [GC pause (G1 Evacuation Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] desired_size: 2KB slow allocs: 15 refill waste: 32B alloc: 0.00002 92KB refills: 22 waste 4.4% gc: 1328B slow: 680B fast: 0B TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 3KB refills: 1 waste 14.0% gc: 288B slow: 0B fast: 0B TLAB: gc thread: 0x00007f1590689800 [id: 126] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00001 58KB refills: 2 waste 25.2% gc: 1008B slow: 32B fast: 0B TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.99817 4023075KB refills: 2949 waste 0.0% gc: 0B slow: 127160B fast: 0B TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow allocs: 23 refill waste: 56B alloc: 0.00004 179KB refills: 31 waste 1.6% gc: 400B slow: 1424B fast: 0B TLAB totals: thrds: 5 refills: 3005 max: 2949 slow allocs: 38 max 23 waste: 0.0% gc: 3024B max: 1328B slow: 129296B max: 127160B fast: 0B max: 0B Desired survivor size 268435456 bytes, new threshold 15 (max 15) - age 1: 359040 bytes, 359040 total - age 2: 96968 bytes, 456008 total - age 3: 10944 bytes, 466952 total - age 4: 50816 bytes, 517768 total - age 5: 205888 bytes, 723656 total - age 6: 51760 bytes, 775416 total - age 7: 14680 bytes, 790096 total - age 8: 114320 bytes, 904416 total - age 9: 369856 bytes, 1274272 total - age 10: 110328 bytes, 1384600 total - age 11: 14208 bytes, 1398808 total - age 12: 41328 bytes, 1440136 total - age 13: 689344 bytes, 2129480 total - age 14: 730904 bytes, 2860384 total - age 15: 142800480 bytes, 145660864 total 68095.262: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 6424, predicted base time: 10.60 ms, remaining time: 139.40 ms, target pause time: 150.00 ms] 68095.262: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 984 regions, survivors: 40 regions, predicted young region time: 86.48 ms] 68095.262: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 984 regions, survivors: 40 regions, old: 0 regions, predicted pause time: 97.07 ms, target pause time: 150.00 ms] 2017-01-25T14:10:50.949+0530: 68095.401: [SoftReference, 0 refs, 0.0012724 secs]2017-01-25T14:10:50.950+0530: 68095.403: [WeakReference, 0 refs, 0.0006785 secs]2017-01-25T14:10:50.951+0530: 68095.403: [FinalReference, 0 refs, 0.0008141 secs]2017-01-25T14:10:50.952+0530: 68095.404: [PhantomReference, 0 refs, 0 refs, 0.0012798 secs]2017-01-25T14:10:50.953+0530: 68095.406: [JNI Weak Reference, 0.0000165 secs], 0.1466546 secs] [Parallel Time: 138.9 ms, GC Workers: 20] [GC Worker Start (ms): Min: 68095262.1, Avg: 68095263.2, Max: 68095275.2, Diff: 13.1] [Ext Root Scanning (ms): Min: 0.0, Avg: 2.1, Max: 9.4, Diff: 9.4, Sum: 42.4] [Update RS (ms): Min: 0.0, Avg: 0.5, Max: 6.5, Diff: 6.5, Sum: 9.9] [Processed Buffers: Min: 0, Avg: 1.6, Max: 6, Diff: 6, Sum: 31] [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 1.9] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 117.9, Avg: 128.4, Max: 134.8, Diff: 16.9, Sum: 2567.8] [Termination (ms): Min: 2.5, Avg: 6.1, Max: 7.6, Diff: 5.2, Sum: 122.3] [Termination Attempts: Min: 1110, Avg: 1929.0, Max: 2192, Diff: 1082, Sum: 38580] [GC Worker Other (ms): Min: 0.0, Avg: 0.2, Max: 0.4, Diff: 0.4, Sum: 4.4] [GC Worker Total (ms): Min: 125.3, Avg: 137.4, Max: 138.6, Diff: 13.4, Sum: 2748.7] [GC Worker End (ms): Min: 68095400.4, Avg: 68095400.6, Max: 68095400.8, Diff: 0.4] [Code Root Fixup: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.8 ms] [Other: 7.0 ms] [Choose CSet: 0.0 ms] [Ref Proc: 4.6 ms] [Ref Enq: 0.3 ms] [Redirty Cards: 0.4 ms] [Humongous Register: 0.1 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 1.4 ms] [Eden: 3936.0M(3936.0M)->0.0B(4060.0M) Survivors: 160.0M->36.0M Heap: 5164.0M(8192.0M)->1245.2M(8192.0M)] [Times: user=5.47 sys=0.00, real=0.15 secs] 2017-01-25T14:10:50.956+0530: 68095.409: Total time for which application threads were stopped: 0.1470861 seconds, Stopping threads took: 0.0000389 seconds 2017-01-25T14:10:50.963+0530: 68095.416: Application time: 0.0071737 seconds 2017-01-25T14:10:50.964+0530: 68095.416: Total time for which application threads were stopped: 0.0002721 seconds, Stopping threads took: 0.0000830 seconds 2017-01-25T14:10:51.149+0530: 68095.602: Application time: 0.1858778 seconds 2017-01-25T14:10:51.150+0530: 68095.602: Total time for which application threads were stopped: 0.0004194 seconds, Stopping threads took: 0.0001064 seconds 2017-01-25T14:10:52.150+0530: 68096.602: Application time: 0.9997142 seconds 2017-01-25T14:10:52.150+0530: 68096.603: Total time for which application threads were stopped: 0.0003868 seconds, Stopping threads took: 0.0000696 seconds 2017-01-25T14:10:52.467+0530: 68096.920: Application time: 0.3171898 seconds 2017-01-25T14:10:52.467+0530: 68096.920: [GC pause (G1 Evacuation Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] desired_size: 2KB slow allocs: 28 refill waste: 32B alloc: 0.00002 80KB refills: 43 waste 3.1% gc: 1328B slow: 1432B fast: 0B TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00000 3KB refills: 2 waste 16.1% gc: 648B slow: 16B fast: 0B TLAB: gc thread: 0x00007f15906a1800 [id: 140] desired_size: 19KB slow allocs: 1 refill waste: 312B alloc: 0.00024 992KB refills: 15 waste 6.2% gc: 18456B slow: 464B fast: 0B TLAB: gc thread: 0x00007f159069b000 [id: 136] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00002 75KB refills: 2 waste 25.0% gc: 1016B slow: 16B fast: 0B TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 26KB slow allocs: 0 refill waste: 424B alloc: 0.00032 1325KB refills: 3 waste 9.9% gc: 8008B slow: 56B fast: 0B TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 5KB slow allocs: 1 refill waste: 88B alloc: 0.00007 278KB refills: 14 waste 4.4% gc: 3032B slow: 488B fast: 0B TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 13KB slow allocs: 1 refill waste: 216B alloc: 0.00016 681KB refills: 9 waste 8.9% gc: 10896B slow: 288B fast: 0B TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 14KB refills: 5 waste 8.4% gc: 760B slow: 112B fast: 0B TLAB: gc thread: 0x00007f159062b000 [id: 115] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 1KB refills: 1 waste 89.9% gc: 1856B slow: 0B fast: 0B TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.99878 4152376KB refills: 3022 waste 0.0% gc: 0B slow: 128312B fast: 0B TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 27KB slow allocs: 1 refill waste: 440B alloc: 0.00033 1387KB refills: 26 waste 3.0% gc: 20128B slow: 1912B fast: 0B TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 49KB slow allocs: 2 refill waste: 784B alloc: 0.00059 2463KB refills: 21 waste 3.0% gc: 30136B slow: 1704B fast: 0B TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 2KB slow allocs: 11 refill waste: 32B alloc: 0.00002 93KB refills: 23 waste 5.4% gc: 1744B slow: 832B fast: 0B TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 2KB slow allocs: 9 refill waste: 32B alloc: 0.00002 77KB refills: 19 waste 6.2% gc: 1888B slow: 552B fast: 0B TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 66KB slow allocs: 18 refill waste: 1064B alloc: 0.00080 3327KB refills: 43 waste 0.5% gc: 6800B slow: 8912B fast: 0B TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow allocs: 49 refill waste: 48B alloc: 0.00004 162KB refills: 62 waste 2.3% gc: 2304B slow: 2432B fast: 0B TLAB: gc thread: 0x00007f170d35a000 [id: 85] desired_size: 10KB slow allocs: 0 refill waste: 168B alloc: 0.00013 549KB refills: 3 waste 12.3% gc: 4064B slow: 72B fast: 0B TLAB totals: thrds: 17 refills: 3313 max: 3022 slow allocs: 122 max 49 waste: 0.0% gc: 113064B max: 30136B slow: 147600B max: 128312B fast: 0B max: 0B Desired survivor size 268435456 bytes, new threshold 15 (max 15) - age 1: 14432 bytes, 14432 total - age 2: 353520 bytes, 367952 total - age 3: 96968 bytes, 464920 total - age 4: 10944 bytes, 475864 total - age 5: 50816 bytes, 526680 total - age 6: 205888 bytes, 732568 total - age 7: 51760 bytes, 784328 total - age 8: 14680 bytes, 799008 total - age 9: 114320 bytes, 913328 total - age 10: 369856 bytes, 1283184 total - age 11: 110328 bytes, 1393512 total - age 12: 14208 bytes, 1407720 total - age 13: 41328 bytes, 1449048 total - age 14: 689344 bytes, 2138392 total - age 15: 730824 bytes, 2869216 total 68096.920: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 35114, predicted base time: 16.14 ms, remaining time: 133.86 ms, target pause time: 150.00 ms] 68096.920: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 1015 regions, survivors: 9 regions, predicted young region time: 37.34 ms] 68096.920: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 1015 regions, survivors: 9 regions, old: 0 regions, predicted pause time: 53.48 ms, target pause time: 150.00 ms] 2017-01-25T14:10:52.481+0530: 68096.933: [SoftReference, 0 refs, 0.0012948 secs]2017-01-25T14:10:52.482+0530: 68096.935: [WeakReference, 0 refs, 0.0006552 secs]2017-01-25T14:10:52.483+0530: 68096.935: [FinalReference, 0 refs, 0.0006372 secs]2017-01-25T14:10:52.483+0530: 68096.936: [PhantomReference, 0 refs, 0 refs, 0.0013651 secs]2017-01-25T14:10:52.485+0530: 68096.937: [JNI Weak Reference, 0.0000170 secs], 0.0211945 secs] [Parallel Time: 12.7 ms, GC Workers: 20] [GC Worker Start (ms): Min: 68096920.3, Avg: 68096920.5, Max: 68096920.7, Diff: 0.4] [Ext Root Scanning (ms): Min: 0.4, Avg: 0.6, Max: 1.0, Diff: 0.7, Sum: 11.5] [Update RS (ms): Min: 3.2, Avg: 3.6, Max: 4.1, Diff: 0.9, Sum: 71.2] [Processed Buffers: Min: 2, Avg: 7.7, Max: 15, Diff: 13, Sum: 153] [Scan RS (ms): Min: 0.0, Avg: 0.3, Max: 0.5, Diff: 0.4, Sum: 6.0] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 2.3, Avg: 3.5, Max: 7.4, Diff: 5.1, Sum: 70.3] [Termination (ms): Min: 0.6, Avg: 4.4, Max: 5.5, Diff: 4.9, Sum: 88.3] [Termination Attempts: Min: 1, Avg: 1.9, Max: 4, Diff: 3, Sum: 37] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 1.4] [GC Worker Total (ms): Min: 12.2, Avg: 12.4, Max: 12.7, Diff: 0.4, Sum: 248.8] [GC Worker End (ms): Min: 68096932.8, Avg: 68096932.9, Max: 68096933.0, Diff: 0.2] [Code Root Fixup: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.5 ms] [Other: 7.9 ms] [Choose CSet: 0.0 ms] [Ref Proc: 4.4 ms] [Ref Enq: 0.4 ms] [Redirty Cards: 0.3 ms] [Humongous Register: 0.1 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 1.5 ms] [Eden: 4060.0M(4060.0M)->0.0B(4056.0M) Survivors: 36.0M->40.0M Heap: 5305.2M(8192.0M)->1285.5M(8192.0M)] [Times: user=0.00 sys=0.00, real=0.02 secs] 2017-01-25T14:10:52.489+0530: 68096.941: Total time for which application threads were stopped: 0.0216177 seconds, Stopping threads took: 0.0000399 seconds 2017-01-25T14:10:53.149+0530: 68097.601: Application time: 0.6599486 seconds 2017-01-25T14:10:53.149+0530: 68097.602: Total time for which application threads were stopped: 0.0003672 seconds, Stopping threads took: 0.0000798 seconds 2017-01-25T14:10:54.004+0530: 68098.457: Application time: 0.8549700 seconds 2017-01-25T14:10:54.004+0530: 68098.457: [GC pause (G1 Evacuation Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] desired_size: 2KB slow allocs: 14 refill waste: 32B alloc: 0.00002 88KB refills: 22 waste 4.3% gc: 1328B slow: 632B fast: 0B TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 3KB refills: 1 waste 14.0% gc: 288B slow: 0B fast: 0B TLAB: gc thread: 0x00007f15906a3800 [id: 141] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00002 78KB refills: 2 waste 26.0% gc: 1056B slow: 16B fast: 0B TLAB: gc thread: 0x00007f15906a1800 [id: 140] desired_size: 14KB slow allocs: 0 refill waste: 232B alloc: 0.00018 747KB refills: 2 waste 13.7% gc: 4152B slow: 24B fast: 0B TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 17KB slow allocs: 0 refill waste: 272B alloc: 0.00021 871KB refills: 5 waste 1.2% gc: 864B slow: 176B fast: 0B TLAB: gc thread: 0x00007f1590692000 [id: 131] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00001 34KB refills: 2 waste 25.2% gc: 1008B slow: 32B fast: 0B TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 4KB slow allocs: 3 refill waste: 64B alloc: 0.00005 205KB refills: 37 waste 2.7% gc: 2888B slow: 1328B fast: 0B TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 9KB slow allocs: 0 refill waste: 152B alloc: 0.00012 481KB refills: 6 waste 14.9% gc: 8584B slow: 216B fast: 0B TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 10KB slow allocs: 0 refill waste: 168B alloc: 0.00013 534KB refills: 5 waste 18.2% gc: 9840B slow: 128B fast: 0B TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 2KB slow allocs: 3 refill waste: 32B alloc: 0.00000 12KB refills: 13 waste 8.3% gc: 1944B slow: 280B fast: 0B TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.99872 4148023KB refills: 3013 waste 0.0% gc: 0B slow: 126696B fast: 0B TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 22KB slow allocs: 1 refill waste: 360B alloc: 0.00027 1135KB refills: 36 waste 1.5% gc: 9552B slow: 2824B fast: 0B TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 38KB slow allocs: 10 refill waste: 608B alloc: 0.00046 1921KB refills: 25 waste 2.9% gc: 25136B slow: 3000B fast: 0B TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 2KB slow allocs: 22 refill waste: 32B alloc: 0.00002 78KB refills: 20 waste 2.8% gc: 424B slow: 720B fast: 0B TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 2KB slow allocs: 15 refill waste: 32B alloc: 0.00002 64KB refills: 28 waste 4.5% gc: 1784B slow: 800B fast: 0B TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 63KB slow allocs: 9 refill waste: 1008B alloc: 0.00076 3174KB refills: 59 waste 1.3% gc: 44896B slow: 5656B fast: 0B TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow allocs: 27 refill waste: 56B alloc: 0.00004 181KB refills: 27 waste 4.2% gc: 2928B slow: 1280B fast: 0B TLAB totals: thrds: 17 refills: 3303 max: 3013 slow allocs: 104 max 27 waste: 0.0% gc: 116672B max: 44896B slow: 143808B max: 126696B fast: 0B max: 0B Desired survivor size 268435456 bytes, new threshold 15 (max 15) - age 1: 300240 bytes, 300240 total - age 2: 9544 bytes, 309784 total - age 3: 352008 bytes, 661792 total - age 4: 93464 bytes, 755256 total - age 5: 10856 bytes, 766112 total - age 6: 50600 bytes, 816712 total - age 7: 205456 bytes, 1022168 total - age 8: 51664 bytes, 1073832 total - age 9: 14656 bytes, 1088488 total - age 10: 114176 bytes, 1202664 total - age 11: 369736 bytes, 1572400 total - age 12: 110328 bytes, 1682728 total - age 13: 14208 bytes, 1696936 total - age 14: 41296 bytes, 1738232 total - age 15: 689344 bytes, 2427576 total 68098.457: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 6677, predicted base time: 10.32 ms, remaining time: 139.68 ms, target pause time: 150.00 ms] 68098.457: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 1014 regions, survivors: 10 regions, predicted young region time: 25.44 ms] 68098.457: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 1014 regions, survivors: 10 regions, old: 0 regions, predicted pause time: 35.75 ms, target pause time: 150.00 ms] 2017-01-25T14:10:54.015+0530: 68098.467: [SoftReference, 0 refs, 0.0011853 secs]2017-01-25T14:10:54.016+0530: 68098.469: [WeakReference, 0 refs, 0.0008073 secs]2017-01-25T14:10:54.017+0530: 68098.469: [FinalReference, 1 refs, 0.0008374 secs]2017-01-25T14:10:54.018+0530: 68098.470: [PhantomReference, 0 refs, 0 refs, 0.0017963 secs]2017-01-25T14:10:54.019+0530: 68098.472: [JNI Weak Reference, 0.0000168 secs], 0.0177960 secs] [Parallel Time: 5.8 ms, GC Workers: 20] [GC Worker Start (ms): Min: 68098457.1, Avg: 68098457.3, Max: 68098457.4, Diff: 0.2] [Ext Root Scanning (ms): Min: 0.4, Avg: 0.6, Max: 1.0, Diff: 0.6, Sum: 11.6] [Update RS (ms): Min: 0.0, Avg: 1.1, Max: 2.7, Diff: 2.7, Sum: 21.9] [Processed Buffers: Min: 0, Avg: 2.0, Max: 9, Diff: 9, Sum: 41] [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 1.7] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 0.0, Avg: 2.0, Max: 3.2, Diff: 3.2, Sum: 39.2] [Termination (ms): Min: 0.0, Avg: 1.7, Max: 2.2, Diff: 2.2, Sum: 34.1] [Termination Attempts: Min: 1, Avg: 2.4, Max: 6, Diff: 5, Sum: 47] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.2] [GC Worker Total (ms): Min: 5.4, Avg: 5.5, Max: 5.7, Diff: 0.4, Sum: 110.8] [GC Worker End (ms): Min: 68098462.7, Avg: 68098462.8, Max: 68098462.9, Diff: 0.2] [Code Root Fixup: 0.1 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.5 ms] [Other: 11.4 ms] [Choose CSet: 0.0 ms] [Ref Proc: 9.2 ms] [Ref Enq: 0.3 ms] [Redirty Cards: 0.3 ms] [Humongous Register: 0.0 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 1.4 ms] [Eden: 4056.0M(4056.0M)->0.0B(4084.0M) Survivors: 40.0M->12.0M Heap: 5341.5M(8192.0M)->1276.8M(8192.0M)] [Times: user=0.00 sys=0.00, real=0.02 secs] 2017-01-25T14:10:54.022+0530: 68098.475: Total time for which application threads were stopped: 0.0181786 seconds, Stopping threads took: 0.0000375 seconds 2017-01-25T14:10:54.149+0530: 68098.601: Application time: 0.1266361 seconds 2017-01-25T14:10:54.149+0530: 68098.602: Total time for which application threads were stopped: 0.0003796 seconds, Stopping threads took: 0.0000827 seconds 2017-01-25T14:10:55.149+0530: 68099.602: Application time: 0.9998625 seconds 2017-01-25T14:10:55.149+0530: 68099.602: Total time for which application threads were stopped: 0.0003472 seconds, Stopping threads took: 0.0000761 seconds 2017-01-25T14:10:55.581+0530: 68100.034: Application time: 0.4318729 seconds 2017-01-25T14:10:55.582+0530: 68100.034: [GC pause (G1 Evacuation Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] desired_size: 2KB slow allocs: 28 refill waste: 32B alloc: 0.00002 76KB refills: 43 waste 3.0% gc: 1328B slow: 1336B fast: 0B TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00000 3KB refills: 2 waste 16.1% gc: 648B slow: 16B fast: 0B TLAB: gc thread: 0x00007f15906a1800 [id: 140] desired_size: 9KB slow allocs: 1 refill waste: 152B alloc: 0.00012 495KB refills: 7 waste 3.5% gc: 2240B slow: 208B fast: 0B TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 11KB slow allocs: 0 refill waste: 184B alloc: 0.00014 595KB refills: 10 waste 8.1% gc: 9608B slow: 272B fast: 0B TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 3KB slow allocs: 4 refill waste: 56B alloc: 0.00005 188KB refills: 19 waste 1.2% gc: 352B slow: 536B fast: 0B TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 2KB slow allocs: 2 refill waste: 32B alloc: 0.00000 17KB refills: 5 waste 12.9% gc: 1168B slow: 160B fast: 0B TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.99867 4176462KB refills: 3040 waste 0.0% gc: 0B slow: 127000B fast: 0B TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 20KB slow allocs: 0 refill waste: 320B alloc: 0.00024 1004KB refills: 24 waste 4.4% gc: 20256B slow: 1256B fast: 0B TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 31KB slow allocs: 4 refill waste: 504B alloc: 0.00038 1595KB refills: 19 waste 3.9% gc: 22504B slow: 1448B fast: 0B TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 2KB slow allocs: 8 refill waste: 32B alloc: 0.00002 67KB refills: 15 waste 7.1% gc: 1696B slow: 512B fast: 0B TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 2KB slow allocs: 11 refill waste: 32B alloc: 0.00002 64KB refills: 19 waste 6.5% gc: 2000B slow: 560B fast: 0B TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 65KB slow allocs: 9 refill waste: 1040B alloc: 0.00078 3255KB refills: 38 waste 2.4% gc: 55624B slow: 5200B fast: 0B TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow allocs: 68 refill waste: 48B alloc: 0.00004 155KB refills: 66 waste 1.5% gc: 336B slow: 2848B fast: 0B TLAB totals: thrds: 13 refills: 3307 max: 3040 slow allocs: 136 max 68 waste: 0.0% gc: 117760B max: 55624B slow: 141352B max: 127000B fast: 0B max: 0B Desired survivor size 268435456 bytes, new threshold 15 (max 15) - age 1: 240808 bytes, 240808 total - age 2: 290624 bytes, 531432 total - age 3: 9544 bytes, 540976 total - age 4: 345432 bytes, 886408 total - age 5: 93392 bytes, 979800 total - age 6: 10856 bytes, 990656 total - age 7: 50576 bytes, 1041232 total - age 8: 205264 bytes, 1246496 total - age 9: 51616 bytes, 1298112 total - age 10: 14656 bytes, 1312768 total - age 11: 114104 bytes, 1426872 total - age 12: 369568 bytes, 1796440 total - age 13: 110304 bytes, 1906744 total - age 14: 14208 bytes, 1920952 total - age 15: 41296 bytes, 1962248 total 68100.034: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 6344, predicted base time: 11.31 ms, remaining time: 138.69 ms, target pause time: 150.00 ms] 68100.034: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 1021 regions, survivors: 3 regions, predicted young region time: 7.85 ms] 68100.034: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 1021 regions, survivors: 3 regions, old: 0 regions, predicted pause time: 19.16 ms, target pause time: 150.00 ms] 2017-01-25T14:10:55.592+0530: 68100.045: [SoftReference, 0 refs, 0.0013160 secs]2017-01-25T14:10:55.593+0530: 68100.046: [WeakReference, 0 refs, 0.0007300 secs]2017-01-25T14:10:55.594+0530: 68100.047: [FinalReference, 1 refs, 0.0006525 secs]2017-01-25T14:10:55.595+0530: 68100.047: [PhantomReference, 0 refs, 0 refs, 0.0012912 secs]2017-01-25T14:10:55.596+0530: 68100.049: [JNI Weak Reference, 0.0000154 secs], 0.0169289 secs] [Parallel Time: 9.7 ms, GC Workers: 20] [GC Worker Start (ms): Min: 68100034.5, Avg: 68100035.5, Max: 68100037.6, Diff: 3.1] [Ext Root Scanning (ms): Min: 0.0, Avg: 0.3, Max: 1.0, Diff: 1.0, Sum: 5.1] [Update RS (ms): Min: 0.0, Avg: 0.7, Max: 8.7, Diff: 8.7, Sum: 14.6] [Processed Buffers: Min: 0, Avg: 1.8, Max: 8, Diff: 8, Sum: 36] [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 1.6] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 0.0, Avg: 4.2, Max: 7.4, Diff: 7.4, Sum: 83.3] [Termination (ms): Min: 0.0, Avg: 3.1, Max: 6.6, Diff: 6.6, Sum: 61.8] [Termination Attempts: Min: 1, Avg: 129.9, Max: 172, Diff: 171, Sum: 2599] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 1.5] [GC Worker Total (ms): Min: 6.3, Avg: 8.4, Max: 9.4, Diff: 3.2, Sum: 167.9] [GC Worker End (ms): Min: 68100043.8, Avg: 68100043.9, Max: 68100044.0, Diff: 0.2] [Code Root Fixup: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.5 ms] [Other: 6.7 ms] [Choose CSet: 0.1 ms] [Ref Proc: 4.5 ms] [Ref Enq: 0.3 ms] [Redirty Cards: 0.2 ms] [Humongous Register: 0.0 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 1.4 ms] [Eden: 4084.0M(4084.0M)->0.0B(4092.0M) Survivors: 12.0M->4096.0K Heap: 5360.8M(8192.0M)->1277.9M(8192.0M)] [Times: user=0.00 sys=0.00, real=0.02 secs] 2017-01-25T14:10:55.599+0530: 68100.051: Total time for which application threads were stopped: 0.0173605 seconds, Stopping threads took: 0.0000391 seconds 2017-01-25T14:10:55.964+0530: 68100.416: Application time: 0.3650777 seconds 2017-01-25T14:10:55.964+0530: 68100.417: Total time for which application threads were stopped: 0.0002977 seconds, Stopping threads took: 0.0000702 seconds 2017-01-25T14:10:56.149+0530: 68100.602: Application time: 0.1851895 seconds 2017-01-25T14:10:56.150+0530: 68100.602: Total time for which application threads were stopped: 0.0003614 seconds, Stopping threads took: 0.0000746 seconds 2017-01-25T14:10:57.133+0530: 68101.586: Application time: 0.9833398 seconds 2017-01-25T14:10:57.133+0530: 68101.586: [GC pause (G1 Evacuation Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] desired_size: 2KB slow allocs: 27 refill waste: 32B alloc: 0.00002 85KB refills: 43 waste 3.0% gc: 1328B slow: 1320B fast: 0B TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00000 3KB refills: 2 waste 16.1% gc: 648B slow: 16B fast: 0B TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 8KB slow allocs: 0 refill waste: 128B alloc: 0.00010 421KB refills: 1 waste 22.8% gc: 1968B slow: 0B fast: 0B TLAB: gc thread: 0x00007f159068b000 [id: 127] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 13KB refills: 2 waste 25.2% gc: 1008B slow: 32B fast: 0B TLAB: gc thread: 0x00007f159067d800 [id: 121] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00001 62KB refills: 2 waste 25.2% gc: 1008B slow: 32B fast: 0B TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.99883 4185307KB refills: 3062 waste 0.0% gc: 0B slow: 132496B fast: 0B TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 54KB slow allocs: 0 refill waste: 872B alloc: 0.00065 2728KB refills: 6 waste 11.0% gc: 36704B slow: 312B fast: 0B TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow allocs: 15 refill waste: 56B alloc: 0.00004 178KB refills: 26 waste 2.6% gc: 1360B slow: 1112B fast: 0B TLAB: gc thread: 0x00007f170d35a000 [id: 85] desired_size: 7KB slow allocs: 0 refill waste: 112B alloc: 0.00009 367KB refills: 4 waste 1.9% gc: 480B slow: 80B fast: 0B TLAB totals: thrds: 9 refills: 3148 max: 3062 slow allocs: 43 max 27 waste: 0.0% gc: 44504B max: 36704B slow: 135400B max: 132496B fast: 0B max: 0B Desired survivor size 268435456 bytes, new threshold 15 (max 15) - age 1: 166472 bytes, 166472 total - age 2: 230296 bytes, 396768 total - age 3: 290624 bytes, 687392 total - age 4: 9544 bytes, 696936 total - age 5: 345432 bytes, 1042368 total - age 6: 93392 bytes, 1135760 total - age 7: 10856 bytes, 1146616 total - age 8: 50576 bytes, 1197192 total - age 9: 205264 bytes, 1402456 total - age 10: 51616 bytes, 1454072 total - age 11: 14656 bytes, 1468728 total - age 12: 114104 bytes, 1582832 total - age 13: 369544 bytes, 1952376 total - age 14: 110304 bytes, 2062680 total - age 15: 14208 bytes, 2076888 total 68101.586: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 5995, predicted base time: 10.55 ms, remaining time: 139.45 ms, target pause time: 150.00 ms] 68101.586: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 1023 regions, survivors: 1 regions, predicted young region time: 5.63 ms] 68101.586: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 1023 regions, survivors: 1 regions, old: 0 regions, predicted pause time: 16.19 ms, target pause time: 150.00 ms] 2017-01-25T14:10:57.139+0530: 68101.591: [SoftReference, 0 refs, 0.0010864 secs]2017-01-25T14:10:57.140+0530: 68101.592: [WeakReference, 0 refs, 0.0007033 secs]2017-01-25T14:10:57.140+0530: 68101.593: [FinalReference, 1 refs, 0.0006381 secs]2017-01-25T14:10:57.141+0530: 68101.594: [PhantomReference, 0 refs, 0 refs, 0.0013022 secs]2017-01-25T14:10:57.142+0530: 68101.595: [JNI Weak Reference, 0.0000164 secs], 0.0119617 secs] [Parallel Time: 4.8 ms, GC Workers: 20] [GC Worker Start (ms): Min: 68101586.0, Avg: 68101586.2, Max: 68101586.4, Diff: 0.4] [Ext Root Scanning (ms): Min: 0.1, Avg: 0.9, Max: 4.2, Diff: 4.1, Sum: 17.7] [Update RS (ms): Min: 0.0, Avg: 0.3, Max: 0.7, Diff: 0.7, Sum: 5.4] [Processed Buffers: Min: 0, Avg: 1.6, Max: 6, Diff: 6, Sum: 33] [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 1.7] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 0.0, Avg: 1.1, Max: 1.5, Diff: 1.5, Sum: 22.4] [Termination (ms): Min: 0.0, Avg: 1.9, Max: 2.2, Diff: 2.2, Sum: 38.7] [Termination Attempts: Min: 1, Avg: 48.0, Max: 74, Diff: 73, Sum: 959] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 1.5] [GC Worker Total (ms): Min: 4.1, Avg: 4.4, Max: 4.6, Diff: 0.4, Sum: 87.5] [GC Worker End (ms): Min: 68101590.5, Avg: 68101590.6, Max: 68101590.7, Diff: 0.2] [Code Root Fixup: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.5 ms] [Other: 6.6 ms] [Choose CSet: 0.0 ms] [Ref Proc: 4.2 ms] [Ref Enq: 0.4 ms] [Redirty Cards: 0.3 ms] [Humongous Register: 0.1 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 1.5 ms] [Eden: 4092.0M(4092.0M)->0.0B(4092.0M) Survivors: 4096.0K->4096.0K Heap: 5369.9M(8192.0M)->1278.0M(8192.0M)] [Times: user=0.00 sys=0.00, real=0.01 secs] 2017-01-25T14:10:57.145+0530: 68101.598: Total time for which application threads were stopped: 0.0123703 seconds, Stopping threads took: 0.0000400 seconds 2017-01-25T14:10:57.149+0530: 68101.602: Application time: 0.0039993 seconds 2017-01-25T14:10:57.150+0530: 68101.602: Total time for which application threads were stopped: 0.0003789 seconds, Stopping threads took: 0.0000725 seconds 2017-01-25T14:10:58.149+0530: 68102.602: Application time: 0.9995898 seconds 2017-01-25T14:10:58.150+0530: 68102.602: Total time for which application threads were stopped: 0.0003840 seconds, Stopping threads took: 0.0000716 seconds 2017-01-25T14:10:58.708+0530: 68103.161: Application time: 0.5585688 seconds 2017-01-25T14:10:58.709+0530: 68103.161: [GC pause (G1 Evacuation Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] desired_size: 2KB slow allocs: 15 refill waste: 32B alloc: 0.00002 91KB refills: 22 waste 4.4% gc: 1328B slow: 648B fast: 0B TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 3KB refills: 1 waste 14.0% gc: 288B slow: 0B fast: 0B TLAB: gc thread: 0x00007f15906a1800 [id: 140] desired_size: 6KB slow allocs: 0 refill waste: 104B alloc: 0.00008 344KB refills: 6 waste 9.1% gc: 3688B slow: 144B fast: 0B TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 5KB slow allocs: 3 refill waste: 80B alloc: 0.00007 274KB refills: 24 waste 2.7% gc: 2856B slow: 824B fast: 0B TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 6KB slow allocs: 0 refill waste: 104B alloc: 0.00008 331KB refills: 4 waste 21.2% gc: 5640B slow: 96B fast: 0B TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 14KB refills: 4 waste 24.4% gc: 1920B slow: 96B fast: 0B TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.99919 4186812KB refills: 3045 waste 0.0% gc: 0B slow: 131144B fast: 0B TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 16KB slow allocs: 2 refill waste: 256B alloc: 0.00020 821KB refills: 24 waste 1.5% gc: 5016B slow: 1136B fast: 0B TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 24KB slow allocs: 3 refill waste: 392B alloc: 0.00030 1249KB refills: 23 waste 4.2% gc: 22928B slow: 1552B fast: 0B TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 2KB slow allocs: 4 refill waste: 32B alloc: 0.00001 55KB refills: 13 waste 8.1% gc: 1792B slow: 376B fast: 0B TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 2KB slow allocs: 6 refill waste: 32B alloc: 0.00001 56KB refills: 13 waste 8.6% gc: 1952B slow: 352B fast: 0B TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 36KB slow allocs: 21 refill waste: 584B alloc: 0.00044 1836KB refills: 45 waste 1.9% gc: 28232B slow: 4360B fast: 0B TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow allocs: 61 refill waste: 48B alloc: 0.00004 150KB refills: 62 waste 1.8% gc: 544B slow: 2912B fast: 0B TLAB totals: thrds: 13 refills: 3286 max: 3045 slow allocs: 115 max 61 waste: 0.0% gc: 76184B max: 28232B slow: 143640B max: 131144B fast: 0B max: 0B Desired survivor size 268435456 bytes, new threshold 15 (max 15) - age 1: 27288 bytes, 27288 total - age 2: 160520 bytes, 187808 total - age 3: 230296 bytes, 418104 total - age 4: 287608 bytes, 705712 total - age 5: 9544 bytes, 715256 total - age 6: 345432 bytes, 1060688 total - age 7: 93392 bytes, 1154080 total - age 8: 10856 bytes, 1164936 total - age 9: 50576 bytes, 1215512 total - age 10: 205264 bytes, 1420776 total - age 11: 51616 bytes, 1472392 total - age 12: 14656 bytes, 1487048 total - age 13: 114104 bytes, 1601152 total - age 14: 369544 bytes, 1970696 total - age 15: 110304 bytes, 2081000 total 68103.161: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 4514, predicted base time: 9.61 ms, remaining time: 140.39 ms, target pause time: 150.00 ms] 68103.161: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 1023 regions, survivors: 1 regions, predicted young region time: 4.47 ms] 68103.161: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 1023 regions, survivors: 1 regions, old: 0 regions, predicted pause time: 14.08 ms, target pause time: 150.00 ms] 2017-01-25T14:10:58.713+0530: 68103.165: [SoftReference, 0 refs, 0.0016775 secs]2017-01-25T14:10:58.715+0530: 68103.167: [WeakReference, 0 refs, 0.0010189 secs]2017-01-25T14:10:58.716+0530: 68103.168: [FinalReference, 1 refs, 0.0005777 secs]2017-01-25T14:10:58.716+0530: 68103.169: [PhantomReference, 0 refs, 0 refs, 0.0013692 secs]2017-01-25T14:10:58.718+0530: 68103.170: [JNI Weak Reference, 0.0000145 secs], 0.0114214 secs] [Parallel Time: 3.4 ms, GC Workers: 20] [GC Worker Start (ms): Min: 68103161.4, Avg: 68103162.0, Max: 68103163.8, Diff: 2.4] [Ext Root Scanning (ms): Min: 0.0, Avg: 0.3, Max: 1.1, Diff: 1.1, Sum: 6.4] [Update RS (ms): Min: 0.0, Avg: 0.4, Max: 2.6, Diff: 2.6, Sum: 8.0] [Processed Buffers: Min: 0, Avg: 1.4, Max: 8, Diff: 8, Sum: 29] [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 1.6] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 0.0, Avg: 0.7, Max: 1.0, Diff: 1.0, Sum: 13.9] [Termination (ms): Min: 0.0, Avg: 1.1, Max: 1.3, Diff: 1.3, Sum: 22.0] [Termination Attempts: Min: 1, Avg: 52.8, Max: 84, Diff: 83, Sum: 1055] [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 1.0] [GC Worker Total (ms): Min: 0.8, Avg: 2.7, Max: 3.3, Diff: 2.5, Sum: 53.0] [GC Worker End (ms): Min: 68103164.6, Avg: 68103164.7, Max: 68103164.7, Diff: 0.1] [Code Root Fixup: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.4 ms] [Other: 7.5 ms] [Choose CSet: 0.0 ms] [Ref Proc: 5.3 ms] [Ref Enq: 0.4 ms] [Redirty Cards: 0.2 ms] [Humongous Register: 0.1 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 1.3 ms] [Eden: 4092.0M(4092.0M)->0.0B(4092.0M) Survivors: 4096.0K->4096.0K Heap: 5370.0M(8192.0M)->1278.6M(8192.0M)] [Times: user=0.00 sys=0.00, real=0.02 secs] 2017-01-25T14:10:58.720+0530: 68103.173: Total time for which application threads were stopped: 0.0117864 seconds, Stopping threads took: 0.0000423 seconds 2017-01-25T14:10:59.149+0530: 68103.602: Application time: 0.4294109 seconds 2017-01-25T14:10:59.150+0530: 68103.602: Total time for which application threads were stopped: 0.0003487 seconds, Stopping threads took: 0.0000772 seconds 2017-01-25T14:11:00.042+0530: 68104.494: Application time: 0.8919065 seconds 2017-01-25T14:11:00.042+0530: 68104.495: Total time for which application threads were stopped: 0.0003228 seconds, Stopping threads took: 0.0001064 seconds 2017-01-25T14:11:00.150+0530: 68104.602: Application time: 0.1075120 seconds 2017-01-25T14:11:00.150+0530: 68104.603: Total time for which application threads were stopped: 0.0003795 seconds, Stopping threads took: 0.0000747 seconds 2017-01-25T14:11:00.317+0530: 68104.770: Application time: 0.1671251 seconds 2017-01-25T14:11:00.317+0530: 68104.770: [GC pause (G1 Evacuation Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] desired_size: 2KB slow allocs: 27 refill waste: 32B alloc: 0.00002 77KB refills: 43 waste 2.7% gc: 1064B slow: 1376B fast: 0B TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00000 3KB refills: 2 waste 16.1% gc: 648B slow: 16B fast: 0B TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 2KB slow allocs: 2 refill waste: 40B alloc: 0.00004 148KB refills: 21 waste 1.9% gc: 528B slow: 672B fast: 0B TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 4KB slow allocs: 0 refill waste: 64B alloc: 0.00005 222KB refills: 34 waste 3.4% gc: 3976B slow: 1200B fast: 0B TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 7KB slow allocs: 0 refill waste: 112B alloc: 0.00009 362KB refills: 4 waste 0.5% gc: 32B slow: 112B fast: 0B TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 11KB refills: 6 waste 11.8% gc: 1320B slow: 144B fast: 0B TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.99921 4186917KB refills: 3045 waste 0.0% gc: 0B slow: 128648B fast: 0B TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 13KB slow allocs: 1 refill waste: 208B alloc: 0.00016 667KB refills: 45 waste 2.2% gc: 11000B slow: 2520B fast: 0B TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 19KB slow allocs: 10 refill waste: 312B alloc: 0.00024 993KB refills: 37 waste 2.6% gc: 17320B slow: 2224B fast: 0B TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 2KB slow allocs: 13 refill waste: 32B alloc: 0.00001 45KB refills: 17 waste 6.7% gc: 1888B slow: 480B fast: 0B TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 2KB slow allocs: 8 refill waste: 32B alloc: 0.00001 46KB refills: 19 waste 5.8% gc: 1736B slow: 528B fast: 0B TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 34KB slow allocs: 22 refill waste: 544B alloc: 0.00041 1709KB refills: 58 waste 0.3% gc: 1280B slow: 5712B fast: 0B TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow allocs: 60 refill waste: 48B alloc: 0.00004 167KB refills: 56 waste 2.8% gc: 2728B slow: 2576B fast: 0B TLAB: gc thread: 0x00007f170d362000 [id: 86] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 0KB refills: 1 waste 98.4% gc: 2032B slow: 0B fast: 0B TLAB totals: thrds: 14 refills: 3388 max: 3045 slow allocs: 144 max 60 waste: 0.0% gc: 45552B max: 17320B slow: 146208B max: 128648B fast: 0B max: 0B Desired survivor size 268435456 bytes, new threshold 15 (max 15) - age 1: 137864 bytes, 137864 total - age 2: 21240 bytes, 159104 total - age 3: 158672 bytes, 317776 total - age 4: 223000 bytes, 540776 total - age 5: 283888 bytes, 824664 total - age 6: 9544 bytes, 834208 total - age 7: 345392 bytes, 1179600 total - age 8: 93392 bytes, 1272992 total - age 9: 10856 bytes, 1283848 total - age 10: 50576 bytes, 1334424 total - age 11: 205264 bytes, 1539688 total - age 12: 51616 bytes, 1591304 total - age 13: 14656 bytes, 1605960 total - age 14: 114104 bytes, 1720064 total - age 15: 369544 bytes, 2089608 total 68104.770: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 5046, predicted base time: 9.36 ms, remaining time: 140.64 ms, target pause time: 150.00 ms] 68104.770: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 1023 regions, survivors: 1 regions, predicted young region time: 3.77 ms] 68104.770: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 1023 regions, survivors: 1 regions, old: 0 regions, predicted pause time: 13.13 ms, target pause time: 150.00 ms] 2017-01-25T14:11:00.321+0530: 68104.773: [SoftReference, 0 refs, 0.0011104 secs]2017-01-25T14:11:00.322+0530: 68104.774: [WeakReference, 0 refs, 0.0056241 secs]2017-01-25T14:11:00.327+0530: 68104.780: [FinalReference, 1 refs, 0.0005825 secs]2017-01-25T14:11:00.328+0530: 68104.781: [PhantomReference, 0 refs, 0 refs, 0.0012515 secs]2017-01-25T14:11:00.329+0530: 68104.782: [JNI Weak Reference, 0.0000189 secs], 0.0143937 secs] [Parallel Time: 2.5 ms, GC Workers: 20] [GC Worker Start (ms): Min: 68104770.2, Avg: 68104770.6, Max: 68104772.1, Diff: 1.8] [Ext Root Scanning (ms): Min: 0.0, Avg: 0.3, Max: 1.0, Diff: 1.0, Sum: 6.1] [Update RS (ms): Min: 0.0, Avg: 0.4, Max: 1.7, Diff: 1.7, Sum: 8.3] [Processed Buffers: Min: 0, Avg: 1.6, Max: 7, Diff: 7, Sum: 32] [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 1.6] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 0.0, Avg: 0.8, Max: 1.1, Diff: 1.1, Sum: 16.9] [Termination (ms): Min: 0.0, Avg: 0.3, Max: 0.6, Diff: 0.6, Sum: 5.7] [Termination Attempts: Min: 1, Avg: 30.1, Max: 55, Diff: 54, Sum: 602] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 1.4] [GC Worker Total (ms): Min: 0.5, Avg: 2.0, Max: 2.4, Diff: 1.9, Sum: 40.0] [GC Worker End (ms): Min: 68104772.5, Avg: 68104772.6, Max: 68104772.7, Diff: 0.2] [Code Root Fixup: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.5 ms] [Other: 11.4 ms] [Choose CSet: 0.0 ms] [Ref Proc: 9.1 ms] [Ref Enq: 0.3 ms] [Redirty Cards: 0.2 ms] [Humongous Register: 0.1 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 1.4 ms] [Eden: 4092.0M(4092.0M)->0.0B(4092.0M) Survivors: 4096.0K->4096.0K Heap: 5370.6M(8192.0M)->1278.8M(8192.0M)] [Times: user=0.00 sys=0.00, real=0.02 secs] 2017-01-25T14:11:00.332+0530: 68104.785: Total time for which application threads were stopped: 0.0147927 seconds, Stopping threads took: 0.0000410 seconds 2017-01-25T14:11:00.964+0530: 68105.417: Application time: 0.6324078 seconds 2017-01-25T14:11:00.965+0530: 68105.417: Total time for which application threads were stopped: 0.0002847 seconds, Stopping threads took: 0.0000800 seconds 2017-01-25T14:11:01.149+0530: 68105.601: Application time: 0.1842524 seconds 2017-01-25T14:11:01.149+0530: 68105.602: Total time for which application threads were stopped: 0.0003608 seconds, Stopping threads took: 0.0000774 seconds 2017-01-25T14:11:01.894+0530: 68106.346: Application time: 0.7445171 seconds 2017-01-25T14:11:01.894+0530: 68106.347: [GC pause (G1 Evacuation Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] desired_size: 2KB slow allocs: 16 refill waste: 32B alloc: 0.00002 86KB refills: 22 waste 4.5% gc: 1328B slow: 696B fast: 0B TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 3KB refills: 1 waste 14.0% gc: 288B slow: 0B fast: 0B TLAB: gc thread: 0x00007f159062b000 [id: 115] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 1KB refills: 1 waste 89.9% gc: 1856B slow: 0B fast: 0B TLAB: gc thread: 0x00007f159005b000 [id: 114] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 0KB refills: 1 waste 98.4% gc: 2032B slow: 0B fast: 0B TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.99916 4186676KB refills: 3064 waste 0.0% gc: 0B slow: 129544B fast: 0B TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 35KB slow allocs: 0 refill waste: 560B alloc: 0.00042 1752KB refills: 3 waste 30.8% gc: 33128B slow: 32B fast: 0B TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow allocs: 15 refill waste: 56B alloc: 0.00004 179KB refills: 26 waste 2.5% gc: 1336B slow: 1072B fast: 0B TLAB: gc thread: 0x00007f170d35a000 [id: 85] desired_size: 4KB slow allocs: 0 refill waste: 72B alloc: 0.00006 246KB refills: 6 waste 2.6% gc: 576B slow: 224B fast: 0B TLAB totals: thrds: 8 refills: 3124 max: 3064 slow allocs: 31 max 16 waste: 0.0% gc: 40544B max: 33128B slow: 131568B max: 129544B fast: 0B max: 0B Desired survivor size 268435456 bytes, new threshold 15 (max 15) - age 1: 173504 bytes, 173504 total - age 2: 133784 bytes, 307288 total - age 3: 21240 bytes, 328528 total - age 4: 150024 bytes, 478552 total - age 5: 221056 bytes, 699608 total - age 6: 283672 bytes, 983280 total - age 7: 9544 bytes, 992824 total - age 8: 345368 bytes, 1338192 total - age 9: 93392 bytes, 1431584 total - age 10: 10856 bytes, 1442440 total - age 11: 50576 bytes, 1493016 total - age 12: 205264 bytes, 1698280 total - age 13: 51616 bytes, 1749896 total - age 14: 14656 bytes, 1764552 total - age 15: 114080 bytes, 1878632 total 68106.347: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 4895, predicted base time: 10.43 ms, remaining time: 139.57 ms, target pause time: 150.00 ms] 68106.347: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 1023 regions, survivors: 1 regions, predicted young region time: 3.38 ms] 68106.347: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 1023 regions, survivors: 1 regions, old: 0 regions, predicted pause time: 13.81 ms, target pause time: 150.00 ms] 2017-01-25T14:11:01.898+0530: 68106.350: [SoftReference, 0 refs, 0.0011375 secs]2017-01-25T14:11:01.899+0530: 68106.351: [WeakReference, 0 refs, 0.0005968 secs]2017-01-25T14:11:01.899+0530: 68106.352: [FinalReference, 0 refs, 0.0005507 secs]2017-01-25T14:11:01.900+0530: 68106.353: [PhantomReference, 0 refs, 0 refs, 0.0013671 secs]2017-01-25T14:11:01.901+0530: 68106.354: [JNI Weak Reference, 0.0000176 secs], 0.0098550 secs] [Parallel Time: 3.0 ms, GC Workers: 20] [GC Worker Start (ms): Min: 68106346.8, Avg: 68106347.1, Max: 68106347.5, Diff: 0.7] [Ext Root Scanning (ms): Min: 0.0, Avg: 0.4, Max: 1.9, Diff: 1.9, Sum: 8.0] [Update RS (ms): Min: 0.0, Avg: 0.6, Max: 2.2, Diff: 2.2, Sum: 12.0] [Processed Buffers: Min: 0, Avg: 1.4, Max: 7, Diff: 7, Sum: 28] [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 1.7] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 0.0, Avg: 0.8, Max: 1.2, Diff: 1.2, Sum: 16.2] [Termination (ms): Min: 0.0, Avg: 0.6, Max: 0.9, Diff: 0.9, Sum: 11.2] [Termination Attempts: Min: 1, Avg: 84.4, Max: 141, Diff: 140, Sum: 1688] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 1.3] [GC Worker Total (ms): Min: 2.1, Avg: 2.5, Max: 2.7, Diff: 0.7, Sum: 50.5] [GC Worker End (ms): Min: 68106349.5, Avg: 68106349.6, Max: 68106349.7, Diff: 0.2] [Code Root Fixup: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.5 ms] [Other: 6.4 ms] [Choose CSet: 0.0 ms] [Ref Proc: 4.2 ms] [Ref Enq: 0.3 ms] [Redirty Cards: 0.2 ms] [Humongous Register: 0.1 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 1.4 ms] [Eden: 4092.0M(4092.0M)->0.0B(4092.0M) Survivors: 4096.0K->4096.0K Heap: 5370.8M(8192.0M)->1279.3M(8192.0M)] [Times: user=0.00 sys=0.00, real=0.01 secs] 2017-01-25T14:11:01.904+0530: 68106.357: Total time for which application threads were stopped: 0.0102433 seconds, Stopping threads took: 0.0000400 seconds 2017-01-25T14:11:02.149+0530: 68106.602: Application time: 0.2449541 seconds 2017-01-25T14:11:02.149+0530: 68106.602: Total time for which application threads were stopped: 0.0003583 seconds, Stopping threads took: 0.0000634 seconds 2017-01-25T14:11:03.149+0530: 68107.602: Application time: 0.9997439 seconds 2017-01-25T14:11:03.149+0530: 68107.602: Total time for which application threads were stopped: 0.0003310 seconds, Stopping threads took: 0.0000656 seconds 2017-01-25T14:11:03.486+0530: 68107.939: Application time: 0.3365735 seconds 2017-01-25T14:11:03.486+0530: 68107.939: [GC pause (G1 Evacuation Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] desired_size: 2KB slow allocs: 25 refill waste: 32B alloc: 0.00002 74KB refills: 43 waste 2.8% gc: 1064B slow: 1416B fast: 0B TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00000 2KB refills: 2 waste 16.1% gc: 648B slow: 16B fast: 0B TLAB: gc thread: 0x00007f15906a8800 [id: 144] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 13KB refills: 2 waste 25.2% gc: 1008B slow: 32B fast: 0B TLAB: gc thread: 0x00007f15906a7000 [id: 143] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 13KB refills: 2 waste 25.2% gc: 1008B slow: 32B fast: 0B TLAB: gc thread: 0x00007f15906a5000 [id: 142] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 13KB refills: 2 waste 26.0% gc: 1056B slow: 16B fast: 0B TLAB: gc thread: 0x00007f15906a0000 [id: 139] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 13KB refills: 2 waste 25.2% gc: 1008B slow: 32B fast: 0B TLAB: gc thread: 0x00007f159069e800 [id: 138] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 13KB refills: 2 waste 25.2% gc: 1008B slow: 32B fast: 0B TLAB: gc thread: 0x00007f159069c800 [id: 137] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 13KB refills: 2 waste 25.2% gc: 1008B slow: 32B fast: 0B TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 4KB slow allocs: 0 refill waste: 72B alloc: 0.00005 225KB refills: 20 waste 3.8% gc: 2856B slow: 600B fast: 0B TLAB: gc thread: 0x00007f1590697800 [id: 134] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 13KB refills: 2 waste 26.0% gc: 1056B slow: 16B fast: 0B TLAB: gc thread: 0x00007f1590695800 [id: 133] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 13KB refills: 2 waste 25.2% gc: 1008B slow: 32B fast: 0B TLAB: gc thread: 0x00007f1590693800 [id: 132] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 13KB refills: 2 waste 25.2% gc: 1008B slow: 32B fast: 0B TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00003 117KB refills: 4 waste 4.5% gc: 336B slow: 96B fast: 0B TLAB: gc thread: 0x00007f159068e800 [id: 129] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 13KB refills: 2 waste 25.0% gc: 1016B slow: 16B fast: 0B TLAB: gc thread: 0x00007f159068d000 [id: 128] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 13KB refills: 2 waste 25.2% gc: 1008B slow: 32B fast: 0B TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 3KB slow allocs: 0 refill waste: 56B alloc: 0.00005 195KB refills: 3 waste 26.9% gc: 3168B slow: 56B fast: 0B TLAB: gc thread: 0x00007f1590674800 [id: 124] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 13KB refills: 2 waste 25.2% gc: 1008B slow: 32B fast: 0B TLAB: gc thread: 0x00007f1590673000 [id: 123] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 13KB refills: 2 waste 25.2% gc: 1008B slow: 32B fast: 0B TLAB: gc thread: 0x00007f1590671800 [id: 122] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 13KB refills: 2 waste 25.0% gc: 1016B slow: 16B fast: 0B TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 4KB slow allocs: 0 refill waste: 72B alloc: 0.00006 243KB refills: 6 waste 3.6% gc: 896B slow: 184B fast: 0B TLAB: gc thread: 0x00007f1590637000 [id: 117] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 13KB refills: 2 waste 25.8% gc: 1048B slow: 16B fast: 0B TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 11KB refills: 4 waste 24.3% gc: 1920B slow: 88B fast: 0B TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.99942 4187777KB refills: 3045 waste 0.0% gc: 0B slow: 127312B fast: 0B TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 12KB slow allocs: 0 refill waste: 200B alloc: 0.00015 635KB refills: 17 waste 4.1% gc: 8512B slow: 640B fast: 0B TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 18KB slow allocs: 5 refill waste: 288B alloc: 0.00022 904KB refills: 14 waste 2.1% gc: 4800B slow: 672B fast: 0B TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 2KB slow allocs: 6 refill waste: 32B alloc: 0.00001 42KB refills: 7 waste 15.2% gc: 2000B slow: 200B fast: 0B TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 2KB slow allocs: 9 refill waste: 32B alloc: 0.00001 44KB refills: 13 waste 9.1% gc: 2000B slow: 448B fast: 0B TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 23KB slow allocs: 8 refill waste: 368B alloc: 0.00028 1163KB refills: 32 waste 2.0% gc: 12824B slow: 2144B fast: 0B TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 2KB slow allocs: 51 refill waste: 40B alloc: 0.00004 150KB refills: 65 waste 2.5% gc: 2736B slow: 2264B fast: 0B TLAB totals: thrds: 29 refills: 3305 max: 3045 slow allocs: 105 max 51 waste: 0.0% gc: 59032B max: 12824B slow: 136536B max: 127312B fast: 0B max: 0B Desired survivor size 268435456 bytes, new threshold 15 (max 15) - age 1: 16648 bytes, 16648 total - age 2: 169208 bytes, 185856 total - age 3: 133784 bytes, 319640 total - age 4: 21176 bytes, 340816 total - age 5: 150024 bytes, 490840 total - age 6: 221056 bytes, 711896 total - age 7: 283672 bytes, 995568 total - age 8: 9544 bytes, 1005112 total - age 9: 345368 bytes, 1350480 total - age 10: 93392 bytes, 1443872 total - age 11: 10856 bytes, 1454728 total - age 12: 50576 bytes, 1505304 total - age 13: 205232 bytes, 1710536 total - age 14: 51616 bytes, 1762152 total - age 15: 14656 bytes, 1776808 total 68107.939: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 5758, predicted base time: 9.80 ms, remaining time: 140.20 ms, target pause time: 150.00 ms] 68107.939: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 1023 regions, survivors: 1 regions, predicted young region time: 3.10 ms] 68107.939: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 1023 regions, survivors: 1 regions, old: 0 regions, predicted pause time: 12.90 ms, target pause time: 150.00 ms] 2017-01-25T14:11:03.491+0530: 68107.944: [SoftReference, 0 refs, 0.0009941 secs]2017-01-25T14:11:03.492+0530: 68107.945: [WeakReference, 0 refs, 0.0007176 secs]2017-01-25T14:11:03.493+0530: 68107.946: [FinalReference, 0 refs, 0.0005977 secs]2017-01-25T14:11:03.494+0530: 68107.946: [PhantomReference, 0 refs, 0 refs, 0.0044912 secs]2017-01-25T14:11:03.498+0530: 68107.951: [JNI Weak Reference, 0.0000168 secs], 0.0142796 secs] [Parallel Time: 4.2 ms, GC Workers: 20] [GC Worker Start (ms): Min: 68107939.2, Avg: 68107939.6, Max: 68107943.0, Diff: 3.8] [Ext Root Scanning (ms): Min: 0.0, Avg: 0.3, Max: 1.0, Diff: 1.0, Sum: 5.9] [Update RS (ms): Min: 0.0, Avg: 0.4, Max: 1.6, Diff: 1.6, Sum: 8.8] [Processed Buffers: Min: 0, Avg: 2.5, Max: 8, Diff: 8, Sum: 49] [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 2.0] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 0.0, Avg: 0.8, Max: 1.1, Diff: 1.1, Sum: 15.5] [Termination (ms): Min: 0.0, Avg: 1.9, Max: 2.1, Diff: 2.1, Sum: 38.5] [Termination Attempts: Min: 1, Avg: 61.9, Max: 87, Diff: 86, Sum: 1237] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 1.3] [GC Worker Total (ms): Min: 0.2, Avg: 3.6, Max: 4.1, Diff: 3.9, Sum: 72.0] [GC Worker End (ms): Min: 68107943.1, Avg: 68107943.2, Max: 68107943.3, Diff: 0.1] [Code Root Fixup: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.4 ms] [Other: 9.6 ms] [Choose CSet: 0.0 ms] [Ref Proc: 7.3 ms] [Ref Enq: 0.3 ms] [Redirty Cards: 0.2 ms] [Humongous Register: 0.1 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 1.4 ms] [Eden: 4092.0M(4092.0M)->0.0B(4092.0M) Survivors: 4096.0K->4096.0K Heap: 5371.3M(8192.0M)->1278.5M(8192.0M)] [Times: user=0.00 sys=0.00, real=0.02 secs] 2017-01-25T14:11:03.501+0530: 68107.953: Total time for which application threads were stopped: 0.0146790 seconds, Stopping threads took: 0.0000401 seconds 2017-01-25T14:11:04.149+0530: 68108.602: Application time: 0.6486007 seconds 2017-01-25T14:11:04.150+0530: 68108.602: Total time for which application threads were stopped: 0.0003741 seconds, Stopping threads took: 0.0000833 seconds 2017-01-25T14:11:05.096+0530: 68109.549: Application time: 0.9466622 seconds 2017-01-25T14:11:05.097+0530: 68109.549: [GC pause (G1 Evacuation Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] desired_size: 2KB slow allocs: 26 refill waste: 32B alloc: 0.00002 84KB refills: 43 waste 2.8% gc: 1064B slow: 1408B fast: 0B TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00000 3KB refills: 2 waste 16.1% gc: 648B slow: 16B fast: 0B TLAB: gc thread: 0x00007f15906a1800 [id: 140] desired_size: 4KB slow allocs: 3 refill waste: 72B alloc: 0.00006 236KB refills: 34 waste 2.4% gc: 2776B slow: 1216B fast: 0B TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 3KB slow allocs: 2 refill waste: 56B alloc: 0.00004 176KB refills: 16 waste 2.4% gc: 840B slow: 536B fast: 0B TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 2KB slow allocs: 1 refill waste: 40B alloc: 0.00003 130KB refills: 15 waste 5.4% gc: 1696B slow: 448B fast: 0B TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00000 9KB refills: 3 waste 28.2% gc: 1640B slow: 104B fast: 0B TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.99946 4187926KB refills: 3059 waste 0.0% gc: 0B slow: 129152B fast: 0B TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 9KB slow allocs: 3 refill waste: 152B alloc: 0.00011 481KB refills: 40 waste 2.6% gc: 8528B slow: 1688B fast: 0B TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 13KB slow allocs: 21 refill waste: 216B alloc: 0.00016 677KB refills: 35 waste 3.0% gc: 12192B slow: 2536B fast: 0B TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 2KB slow allocs: 9 refill waste: 32B alloc: 0.00001 32KB refills: 12 waste 9.6% gc: 1952B slow: 416B fast: 0B TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 2KB slow allocs: 4 refill waste: 32B alloc: 0.00001 38KB refills: 10 waste 10.1% gc: 1840B slow: 248B fast: 0B TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 20KB slow allocs: 31 refill waste: 328B alloc: 0.00025 1037KB refills: 42 waste 0.5% gc: 848B slow: 3408B fast: 0B TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow allocs: 24 refill waste: 48B alloc: 0.00004 171KB refills: 33 waste 2.2% gc: 1224B slow: 1336B fast: 0B TLAB: gc thread: 0x00007f170d2cf000 [id: 84] desired_size: 2KB slow allocs: 1 refill waste: 64B alloc: 0.00000 9KB refills: 1 waste 3.1% gc: 64B slow: 0B fast: 0B TLAB totals: thrds: 14 refills: 3345 max: 3059 slow allocs: 127 max 31 waste: 0.0% gc: 35312B max: 12192B slow: 142512B max: 129152B fast: 0B max: 0B Desired survivor size 268435456 bytes, new threshold 15 (max 15) - age 1: 80480 bytes, 80480 total - age 2: 12664 bytes, 93144 total - age 3: 167000 bytes, 260144 total - age 4: 129112 bytes, 389256 total - age 5: 21176 bytes, 410432 total - age 6: 150024 bytes, 560456 total - age 7: 221056 bytes, 781512 total - age 8: 283672 bytes, 1065184 total - age 9: 9544 bytes, 1074728 total - age 10: 345368 bytes, 1420096 total - age 11: 93392 bytes, 1513488 total - age 12: 10856 bytes, 1524344 total - age 13: 50576 bytes, 1574920 total - age 14: 205232 bytes, 1780152 total - age 15: 51616 bytes, 1831768 total 68109.549: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 4450, predicted base time: 9.88 ms, remaining time: 140.12 ms, target pause time: 150.00 ms] 68109.549: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 1023 regions, survivors: 1 regions, predicted young region time: 2.71 ms] 68109.549: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 1023 regions, survivors: 1 regions, old: 0 regions, predicted pause time: 12.59 ms, target pause time: 150.00 ms] 2017-01-25T14:11:05.101+0530: 68109.554: [SoftReference, 0 refs, 0.0012538 secs]2017-01-25T14:11:05.102+0530: 68109.555: [WeakReference, 0 refs, 0.0008216 secs]2017-01-25T14:11:05.103+0530: 68109.556: [FinalReference, 0 refs, 0.0014361 secs]2017-01-25T14:11:05.105+0530: 68109.557: [PhantomReference, 0 refs, 0 refs, 0.0016842 secs]2017-01-25T14:11:05.106+0530: 68109.559: [JNI Weak Reference, 0.0000150 secs], 0.0122237 secs] [Parallel Time: 3.7 ms, GC Workers: 20] [GC Worker Start (ms): Min: 68109549.4, Avg: 68109550.0, Max: 68109552.8, Diff: 3.4] [Ext Root Scanning (ms): Min: 0.0, Avg: 0.3, Max: 1.1, Diff: 1.0, Sum: 6.5] [Update RS (ms): Min: 0.0, Avg: 0.1, Max: 0.4, Diff: 0.4, Sum: 2.4] [Processed Buffers: Min: 0, Avg: 1.5, Max: 5, Diff: 5, Sum: 30] [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.3] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 0.0, Avg: 0.4, Max: 0.6, Diff: 0.6, Sum: 8.4] [Termination (ms): Min: 0.0, Avg: 1.9, Max: 2.1, Diff: 2.1, Sum: 37.6] [Termination Attempts: Min: 1, Avg: 9.3, Max: 16, Diff: 15, Sum: 186] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 1.3] [GC Worker Total (ms): Min: 0.0, Avg: 2.9, Max: 3.6, Diff: 3.6, Sum: 58.5] [GC Worker End (ms): Min: 68109552.9, Avg: 68109552.9, Max: 68109553.0, Diff: 0.1] [Code Root Fixup: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.4 ms] [Other: 8.1 ms] [Choose CSet: 0.0 ms] [Ref Proc: 5.9 ms] [Ref Enq: 0.3 ms] [Redirty Cards: 0.3 ms] [Humongous Register: 0.1 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 1.3 ms] [Eden: 4092.0M(4092.0M)->0.0B(4092.0M) Survivors: 4096.0K->4096.0K Heap: 5370.5M(8192.0M)->1278.9M(8192.0M)] [Times: user=0.00 sys=0.00, real=0.01 secs] 2017-01-25T14:11:05.109+0530: 68109.562: Total time for which application threads were stopped: 0.0126205 seconds, Stopping threads took: 0.0000411 seconds 2017-01-25T14:11:05.149+0530: 68109.602: Application time: 0.0405377 seconds 2017-01-25T14:11:05.150+0530: 68109.602: Total time for which application threads were stopped: 0.0003956 seconds, Stopping threads took: 0.0000752 seconds 2017-01-25T14:11:05.150+0530: 68109.603: Application time: 0.0002547 seconds 2017-01-25T14:11:05.150+0530: 68109.603: Total time for which application threads were stopped: 0.0003793 seconds, Stopping threads took: 0.0000429 seconds 2017-01-25T14:11:05.965+0530: 68110.417: Application time: 0.8143539 seconds 2017-01-25T14:11:05.965+0530: 68110.418: Total time for which application threads were stopped: 0.0002882 seconds, Stopping threads took: 0.0000777 seconds 2017-01-25T14:11:06.149+0530: 68110.602: Application time: 0.1839350 seconds 2017-01-25T14:11:06.149+0530: 68110.602: Total time for which application threads were stopped: 0.0003335 seconds, Stopping threads took: 0.0000569 seconds 2017-01-25T14:11:06.654+0530: 68111.107: Application time: 0.5045804 seconds 2017-01-25T14:11:06.654+0530: 68111.107: [GC pause (G1 Evacuation Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] desired_size: 2KB slow allocs: 21 refill waste: 32B alloc: 0.00002 90KB refills: 29 waste 4.5% gc: 1800B slow: 912B fast: 0B TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow allocs: 1 refill waste: 64B alloc: 0.00000 3KB refills: 1 waste 0.8% gc: 16B slow: 0B fast: 0B TLAB: gc thread: 0x00007f14d4080000 [id: 197] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 0KB refills: 1 waste 95.0% gc: 1960B slow: 0B fast: 0B TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.99940 4187695KB refills: 3053 waste 0.0% gc: 0B slow: 131312B fast: 0B TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 22KB slow allocs: 1 refill waste: 352B alloc: 0.00027 1114KB refills: 21 waste 1.3% gc: 4336B slow: 2072B fast: 0B TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow allocs: 48 refill waste: 48B alloc: 0.00004 151KB refills: 69 waste 2.4% gc: 2248B slow: 2808B fast: 0B TLAB: gc thread: 0x00007f170d35a000 [id: 85] desired_size: 3KB slow allocs: 230 refill waste: 48B alloc: 0.00004 169KB refills: 553 waste 1.1% gc: 592B slow: 21040B fast: 0B TLAB totals: thrds: 7 refills: 3727 max: 3053 slow allocs: 301 max 230 waste: 0.0% gc: 10952B max: 4336B slow: 158144B max: 131312B fast: 0B max: 0B Desired survivor size 268435456 bytes, new threshold 15 (max 15) - age 1: 139544 bytes, 139544 total - age 2: 71768 bytes, 211312 total - age 3: 12664 bytes, 223976 total - age 4: 161056 bytes, 385032 total - age 5: 128960 bytes, 513992 total - age 6: 21176 bytes, 535168 total - age 7: 150000 bytes, 685168 total - age 8: 221056 bytes, 906224 total - age 9: 283672 bytes, 1189896 total - age 10: 9544 bytes, 1199440 total - age 11: 345368 bytes, 1544808 total - age 12: 93392 bytes, 1638200 total - age 13: 10856 bytes, 1649056 total - age 14: 50576 bytes, 1699632 total - age 15: 205232 bytes, 1904864 total 68111.107: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 3989, predicted base time: 9.52 ms, remaining time: 140.48 ms, target pause time: 150.00 ms] 68111.107: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 1023 regions, survivors: 1 regions, predicted young region time: 2.51 ms] 68111.107: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 1023 regions, survivors: 1 regions, old: 0 regions, predicted pause time: 12.03 ms, target pause time: 150.00 ms] 2017-01-25T14:11:06.659+0530: 68111.112: [SoftReference, 0 refs, 0.0052300 secs]2017-01-25T14:11:06.665+0530: 68111.117: [WeakReference, 0 refs, 0.0008000 secs]2017-01-25T14:11:06.665+0530: 68111.118: [FinalReference, 0 refs, 0.0006699 secs]2017-01-25T14:11:06.666+0530: 68111.119: [PhantomReference, 0 refs, 0 refs, 0.0011966 secs]2017-01-25T14:11:06.667+0530: 68111.120: [JNI Weak Reference, 0.0000165 secs], 0.0154540 secs] [Parallel Time: 1.9 ms, GC Workers: 20] [GC Worker Start (ms): Min: 68111107.1, Avg: 68111107.4, Max: 68111108.7, Diff: 1.6] [Ext Root Scanning (ms): Min: 0.0, Avg: 0.5, Max: 1.6, Diff: 1.5, Sum: 10.9] [Update RS (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.3, Sum: 2.4] [Processed Buffers: Min: 0, Avg: 1.1, Max: 5, Diff: 5, Sum: 23] [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 1.5] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 0.0, Avg: 0.4, Max: 0.6, Diff: 0.6, Sum: 7.4] [Termination (ms): Min: 0.0, Avg: 0.2, Max: 0.3, Diff: 0.3, Sum: 4.9] [Termination Attempts: Min: 1, Avg: 8.5, Max: 17, Diff: 16, Sum: 170] [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 1.0] [GC Worker Total (ms): Min: 0.0, Avg: 1.4, Max: 1.7, Diff: 1.6, Sum: 28.1] [GC Worker End (ms): Min: 68111108.7, Avg: 68111108.8, Max: 68111108.8, Diff: 0.1] [Code Root Fixup: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.4 ms] [Other: 13.1 ms] [Choose CSet: 0.0 ms] [Ref Proc: 10.9 ms] [Ref Enq: 0.3 ms] [Redirty Cards: 0.2 ms] [Humongous Register: 0.1 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 1.3 ms] [Eden: 4092.0M(4092.0M)->0.0B(4092.0M) Survivors: 4096.0K->4096.0K Heap: 5370.9M(8192.0M)->1279.1M(8192.0M)] [Times: user=0.00 sys=0.00, real=0.02 secs] 2017-01-25T14:11:06.670+0530: 68111.122: Total time for which application threads were stopped: 0.0158568 seconds, Stopping threads took: 0.0000405 seconds 2017-01-25T14:11:07.149+0530: 68111.602: Application time: 0.4792650 seconds 2017-01-25T14:11:07.149+0530: 68111.602: Total time for which application threads were stopped: 0.0003629 seconds, Stopping threads took: 0.0000715 seconds 2017-01-25T14:11:08.149+0530: 68112.602: Application time: 0.9997711 seconds 2017-01-25T14:11:08.150+0530: 68112.602: Total time for which application threads were stopped: 0.0003702 seconds, Stopping threads took: 0.0000820 seconds 2017-01-25T14:11:08.209+0530: 68112.662: Application time: 0.0597830 seconds 2017-01-25T14:11:08.210+0530: 68112.662: [GC pause (G1 Evacuation Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] desired_size: 2KB slow allocs: 25 refill waste: 32B alloc: 0.00002 83KB refills: 43 waste 2.8% gc: 1064B slow: 1392B fast: 0B TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00000 3KB refills: 2 waste 16.1% gc: 648B slow: 16B fast: 0B TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00002 98KB refills: 1 waste 98.8% gc: 2040B slow: 0B fast: 0B TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.99938 4187619KB refills: 3060 waste 0.0% gc: 0B slow: 130440B fast: 0B TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 16KB slow allocs: 0 refill waste: 264B alloc: 0.00020 838KB refills: 12 waste 4.3% gc: 7664B slow: 1168B fast: 0B TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow allocs: 50 refill waste: 56B alloc: 0.00004 176KB refills: 56 waste 2.9% gc: 3152B slow: 2680B fast: 0B TLAB totals: thrds: 6 refills: 3174 max: 3060 slow allocs: 76 max 50 waste: 0.0% gc: 14568B max: 7664B slow: 135696B max: 130440B fast: 0B max: 0B Desired survivor size 268435456 bytes, new threshold 15 (max 15) - age 1: 67760 bytes, 67760 total - age 2: 133664 bytes, 201424 total - age 3: 71768 bytes, 273192 total - age 4: 12600 bytes, 285792 total - age 5: 160072 bytes, 445864 total - age 6: 128960 bytes, 574824 total - age 7: 21176 bytes, 596000 total - age 8: 150000 bytes, 746000 total - age 9: 221056 bytes, 967056 total - age 10: 283672 bytes, 1250728 total - age 11: 9544 bytes, 1260272 total - age 12: 345368 bytes, 1605640 total - age 13: 93392 bytes, 1699032 total - age 14: 10856 bytes, 1709888 total - age 15: 50576 bytes, 1760464 total 68112.662: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 3372, predicted base time: 11.25 ms, remaining time: 138.75 ms, target pause time: 150.00 ms] 68112.662: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 1023 regions, survivors: 1 regions, predicted young region time: 2.26 ms] 68112.662: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 1023 regions, survivors: 1 regions, old: 0 regions, predicted pause time: 13.52 ms, target pause time: 150.00 ms] 2017-01-25T14:11:08.219+0530: 68112.672: [SoftReference, 0 refs, 0.0012482 secs]2017-01-25T14:11:08.220+0530: 68112.673: [WeakReference, 0 refs, 0.0006781 secs]2017-01-25T14:11:08.221+0530: 68112.674: [FinalReference, 0 refs, 0.0006530 secs]2017-01-25T14:11:08.222+0530: 68112.674: [PhantomReference, 0 refs, 0 refs, 0.0012487 secs]2017-01-25T14:11:08.223+0530: 68112.676: [JNI Weak Reference, 0.0000147 secs], 0.0157257 secs] [Parallel Time: 8.8 ms, GC Workers: 20] [GC Worker Start (ms): Min: 68112662.5, Avg: 68112663.8, Max: 68112671.0, Diff: 8.5] [Ext Root Scanning (ms): Min: 0.0, Avg: 0.3, Max: 1.1, Diff: 1.1, Sum: 5.4] [Update RS (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.3, Sum: 2.9] [Processed Buffers: Min: 0, Avg: 1.0, Max: 8, Diff: 8, Sum: 20] [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 1.5] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 0.0, Avg: 0.4, Max: 0.5, Diff: 0.5, Sum: 7.2] [Termination (ms): Min: 0.0, Avg: 6.5, Max: 7.3, Diff: 7.3, Sum: 129.5] [Termination Attempts: Min: 1, Avg: 11.1, Max: 19, Diff: 18, Sum: 221] [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.8] [GC Worker Total (ms): Min: 0.0, Avg: 7.4, Max: 8.6, Diff: 8.5, Sum: 147.3] [GC Worker End (ms): Min: 68112671.1, Avg: 68112671.1, Max: 68112671.2, Diff: 0.1] [Code Root Fixup: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.4 ms] [Other: 6.5 ms] [Choose CSet: 0.0 ms] [Ref Proc: 4.3 ms] [Ref Enq: 0.3 ms] [Redirty Cards: 0.3 ms] [Humongous Register: 0.0 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 1.3 ms] [Eden: 4092.0M(4092.0M)->0.0B(4092.0M) Survivors: 4096.0K->4096.0K Heap: 5371.1M(8192.0M)->1279.3M(8192.0M)] [Times: user=0.00 sys=0.00, real=0.01 secs] 2017-01-25T14:11:08.226+0530: 68112.678: Total time for which application threads were stopped: 0.0161537 seconds, Stopping threads took: 0.0000417 seconds 2017-01-25T14:11:09.149+0530: 68113.602: Application time: 0.9238398 seconds 2017-01-25T14:11:09.150+0530: 68113.602: Total time for which application threads were stopped: 0.0003769 seconds, Stopping threads took: 0.0000805 seconds 2017-01-25T14:11:09.778+0530: 68114.230: Application time: 0.6279838 seconds 2017-01-25T14:11:09.778+0530: 68114.231: [GC pause (G1 Evacuation Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] desired_size: 2KB slow allocs: 16 refill waste: 32B alloc: 0.00002 90KB refills: 22 waste 4.5% gc: 1328B slow: 696B fast: 0B TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 3KB refills: 1 waste 14.0% gc: 288B slow: 0B fast: 0B TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 2KB slow allocs: 6 refill waste: 32B alloc: 0.00002 63KB refills: 33 waste 4.2% gc: 1760B slow: 1072B fast: 0B TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 3KB slow allocs: 0 refill waste: 48B alloc: 0.00004 166KB refills: 9 waste 3.3% gc: 712B slow: 280B fast: 0B TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 7KB refills: 7 waste 2.5% gc: 192B slow: 168B fast: 0B TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.99955 4188322KB refills: 3053 waste 0.0% gc: 0B slow: 131056B fast: 0B TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 8KB slow allocs: 2 refill waste: 136B alloc: 0.00011 444KB refills: 23 waste 1.7% gc: 2336B slow: 1280B fast: 0B TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 13KB slow allocs: 3 refill waste: 208B alloc: 0.00016 650KB refills: 19 waste 2.7% gc: 5408B slow: 1312B fast: 0B TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 2KB slow allocs: 3 refill waste: 32B alloc: 0.00001 30KB refills: 6 waste 11.6% gc: 1280B slow: 152B fast: 0B TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 2KB slow allocs: 2 refill waste: 32B alloc: 0.00001 32KB refills: 7 waste 14.4% gc: 1888B slow: 192B fast: 0B TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 11KB slow allocs: 29 refill waste: 184B alloc: 0.00014 590KB refills: 114 waste 0.9% gc: 4144B slow: 8136B fast: 0B TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow allocs: 20 refill waste: 56B alloc: 0.00004 187KB refills: 31 waste 4.3% gc: 3776B slow: 1336B fast: 0B TLAB totals: thrds: 12 refills: 3325 max: 3053 slow allocs: 81 max 29 waste: 0.0% gc: 23112B max: 5408B slow: 145680B max: 131056B fast: 0B max: 0B Desired survivor size 268435456 bytes, new threshold 15 (max 15) - age 1: 22472 bytes, 22472 total - age 2: 64576 bytes, 87048 total - age 3: 133664 bytes, 220712 total - age 4: 71768 bytes, 292480 total - age 5: 12600 bytes, 305080 total - age 6: 160072 bytes, 465152 total - age 7: 128960 bytes, 594112 total - age 8: 21176 bytes, 615288 total - age 9: 150000 bytes, 765288 total - age 10: 221056 bytes, 986344 total - age 11: 283672 bytes, 1270016 total - age 12: 9544 bytes, 1279560 total - age 13: 345368 bytes, 1624928 total - age 14: 93392 bytes, 1718320 total - age 15: 10856 bytes, 1729176 total 68114.231: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 3658, predicted base time: 10.42 ms, remaining time: 139.58 ms, target pause time: 150.00 ms] 68114.231: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 1023 regions, survivors: 1 regions, predicted young region time: 2.10 ms] 68114.231: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 1023 regions, survivors: 1 regions, old: 0 regions, predicted pause time: 12.53 ms, target pause time: 150.00 ms] 2017-01-25T14:11:09.782+0530: 68114.235: [SoftReference, 0 refs, 0.0011872 secs]2017-01-25T14:11:09.783+0530: 68114.236: [WeakReference, 0 refs, 0.0008379 secs]2017-01-25T14:11:09.784+0530: 68114.237: [FinalReference, 0 refs, 0.0037026 secs]2017-01-25T14:11:09.788+0530: 68114.240: [PhantomReference, 0 refs, 0 refs, 0.0015786 secs]2017-01-25T14:11:09.789+0530: 68114.242: [JNI Weak Reference, 0.0000149 secs], 0.0138869 secs] [Parallel Time: 3.1 ms, GC Workers: 20] [GC Worker Start (ms): Min: 68114230.9, Avg: 68114231.7, Max: 68114233.6, Diff: 2.7] [Ext Root Scanning (ms): Min: 0.0, Avg: 0.3, Max: 1.0, Diff: 1.0, Sum: 5.7] [Update RS (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.3, Sum: 2.3] [Processed Buffers: Min: 0, Avg: 1.2, Max: 5, Diff: 5, Sum: 25] [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 1.6] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 0.0, Avg: 0.9, Max: 1.4, Diff: 1.4, Sum: 17.7] [Termination (ms): Min: 0.0, Avg: 0.6, Max: 0.8, Diff: 0.8, Sum: 11.7] [Termination Attempts: Min: 1, Avg: 96.5, Max: 174, Diff: 173, Sum: 1931] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 1.2] [GC Worker Total (ms): Min: 0.0, Avg: 2.0, Max: 2.9, Diff: 2.9, Sum: 40.3] [GC Worker End (ms): Min: 68114233.7, Avg: 68114233.7, Max: 68114233.8, Diff: 0.1] [Code Root Fixup: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.5 ms] [Other: 10.2 ms] [Choose CSet: 0.0 ms] [Ref Proc: 8.0 ms] [Ref Enq: 0.4 ms] [Redirty Cards: 0.3 ms] [Humongous Register: 0.1 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 1.3 ms] [Eden: 4092.0M(4092.0M)->0.0B(4092.0M) Survivors: 4096.0K->4096.0K Heap: 5371.3M(8192.0M)->1279.4M(8192.0M)] [Times: user=0.00 sys=0.00, real=0.01 secs] 2017-01-25T14:11:09.792+0530: 68114.245: Total time for which application threads were stopped: 0.0142584 seconds, Stopping threads took: 0.0000398 seconds 2017-01-25T14:11:10.150+0530: 68114.602: Application time: 0.3575559 seconds 2017-01-25T14:11:10.150+0530: 68114.603: Total time for which application threads were stopped: 0.0003786 seconds, Stopping threads took: 0.0000693 seconds 2017-01-25T14:11:10.965+0530: 68115.418: Application time: 0.8153954 seconds 2017-01-25T14:11:10.966+0530: 68115.418: Total time for which application threads were stopped: 0.0003303 seconds, Stopping threads took: 0.0000827 seconds 2017-01-25T14:11:11.150+0530: 68115.602: Application time: 0.1840399 seconds 2017-01-25T14:11:11.150+0530: 68115.603: Total time for which application threads were stopped: 0.0003683 seconds, Stopping threads took: 0.0000798 seconds 2017-01-25T14:11:11.365+0530: 68115.817: Application time: 0.2147411 seconds 2017-01-25T14:11:11.365+0530: 68115.818: [GC pause (G1 Evacuation Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] desired_size: 2KB slow allocs: 26 refill waste: 32B alloc: 0.00002 76KB refills: 43 waste 2.8% gc: 1064B slow: 1408B fast: 0B TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow allocs: 1 refill waste: 32B alloc: 0.00000 3KB refills: 2 waste 16.1% gc: 648B slow: 16B fast: 0B TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.99952 4188180KB refills: 3064 waste 0.0% gc: 0B slow: 131896B fast: 0B TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 17KB slow allocs: 0 refill waste: 272B alloc: 0.00021 870KB refills: 4 waste 20.9% gc: 14840B slow: 88B fast: 0B TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow allocs: 47 refill waste: 48B alloc: 0.00004 163KB refills: 60 waste 1.8% gc: 1136B slow: 2456B fast: 0B TLAB: gc thread: 0x00007f170d35a000 [id: 85] desired_size: 15KB slow allocs: 0 refill waste: 248B alloc: 0.00019 791KB refills: 2 waste 8.6% gc: 2760B slow: 16B fast: 0B TLAB totals: thrds: 6 refills: 3175 max: 3064 slow allocs: 74 max 47 waste: 0.0% gc: 20448B max: 14840B slow: 135880B max: 131896B fast: 0B max: 0B Desired survivor size 268435456 bytes, new threshold 15 (max 15) - age 1: 71216 bytes, 71216 total - age 2: 17024 bytes, 88240 total - age 3: 64368 bytes, 152608 total - age 4: 128960 bytes, 281568 total - age 5: 69248 bytes, 350816 total - age 6: 12600 bytes, 363416 total - age 7: 160048 bytes, 523464 total - age 8: 128920 bytes, 652384 total - age 9: 21176 bytes, 673560 total - age 10: 150000 bytes, 823560 total - age 11: 221056 bytes, 1044616 total - age 12: 283672 bytes, 1328288 total - age 13: 9544 bytes, 1337832 total - age 14: 345368 bytes, 1683200 total - age 15: 93392 bytes, 1776592 total 68115.818: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 3358, predicted base time: 10.77 ms, remaining time: 139.23 ms, target pause time: 150.00 ms] 68115.818: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 1023 regions, survivors: 1 regions, predicted young region time: 2.18 ms] 68115.818: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 1023 regions, survivors: 1 regions, old: 0 regions, predicted pause time: 12.95 ms, target pause time: 150.00 ms] 2017-01-25T14:11:11.369+0530: 68115.821: [SoftReference, 0 refs, 0.0019318 secs]2017-01-25T14:11:11.371+0530: 68115.823: [WeakReference, 0 refs, 0.0007565 secs]2017-01-25T14:11:11.371+0530: 68115.824: [FinalReference, 0 refs, 0.0064296 secs]2017-01-25T14:11:11.378+0530: 68115.830: [PhantomReference, 0 refs, 0 refs, 0.0012517 secs]2017-01-25T14:11:11.379+0530: 68115.832: [JNI Weak Reference, 0.0000150 secs], 0.0164458 secs] [Parallel Time: 2.9 ms, GC Workers: 20] [GC Worker Start (ms): Min: 68115817.9, Avg: 68115818.6, Max: 68115820.2, Diff: 2.3] [Ext Root Scanning (ms): Min: 0.0, Avg: 0.5, Max: 2.5, Diff: 2.5, Sum: 10.2] [Update RS (ms): Min: 0.0, Avg: 0.1, Max: 0.4, Diff: 0.4, Sum: 2.4] [Processed Buffers: Min: 0, Avg: 1.0, Max: 4, Diff: 4, Sum: 20] [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 1.6] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 0.0, Avg: 0.7, Max: 1.1, Diff: 1.1, Sum: 14.0] [Termination (ms): Min: 0.0, Avg: 0.5, Max: 0.7, Diff: 0.7, Sum: 9.9] [Termination Attempts: Min: 1, Avg: 72.0, Max: 131, Diff: 130, Sum: 1441] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 1.1] [GC Worker Total (ms): Min: 0.3, Avg: 2.0, Max: 2.7, Diff: 2.4, Sum: 39.3] [GC Worker End (ms): Min: 68115820.5, Avg: 68115820.6, Max: 68115820.7, Diff: 0.1] [Code Root Fixup: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.4 ms] [Other: 13.1 ms] [Choose CSet: 0.0 ms] [Ref Proc: 10.8 ms] [Ref Enq: 0.3 ms] [Redirty Cards: 0.3 ms] [Humongous Register: 0.1 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 1.4 ms] [Eden: 4092.0M(4092.0M)->0.0B(4092.0M) Survivors: 4096.0K->4096.0K Heap: 5371.4M(8192.0M)->1279.2M(8192.0M)] [Times: user=0.00 sys=0.00, real=0.02 secs] 2017-01-25T14:11:11.382+0530: 68115.834: Total time for which application threads were stopped: 0.0168706 seconds, Stopping threads took: 0.0000412 seconds 2017-01-25T14:11:12.150+0530: 68116.602: Application time: 0.7681393 seconds 2017-01-25T14:11:12.150+0530: 68116.603: Total time for which application threads were stopped: 0.0003937 seconds, Stopping threads took: 0.0001038 seconds 2017-01-25T14:11:12.956+0530: 68117.408: Application time: 0.8054451 seconds 2017-01-25T14:11:12.956+0530: 68117.409: [GC pause (G1 Evacuation Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] desired_size: 2KB slow allocs: 15 refill waste: 32B alloc: 0.00002 86KB refills: 22 waste 4.4% gc: 1328B slow: 680B fast: 0B TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 3KB refills: 1 waste 14.0% gc: 288B slow: 0B fast: 0B TLAB: gc thread: 0x00007f159062b000 [id: 115] desired_size: 2KB slow allocs: 0 refill waste: 32B alloc: 0.00000 0KB refills: 1 waste 92.2% gc: 1904B slow: 0B fast: 0B TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB slow allocs: 0 refill waste: 32760B alloc: 0.99964 4188703KB refills: 3062 waste 0.0% gc: 0B slow: 129160B fast: 0B TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 11KB slow allocs: 1 refill waste: 184B alloc: 0.00014 581KB refills: 18 waste 1.6% gc: 2080B slow: 1336B fast: 0B TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow allocs: 21 refill waste: 56B alloc: 0.00004 177KB refills: 26 waste 5.2% gc: 3584B slow: 1296B fast: 0B TLAB totals: thrds: 6 refills: 3130 max: 3062 slow allocs: 37 max 21 waste: 0.0% gc: 9184B max: 3584B slow: 132472B max: 129160B fast: 0B max: 0B Desired survivor size 268435456 bytes, new threshold 15 (max 15) - age 1: 17392 bytes, 17392 total - age 2: 67720 bytes, 85112 total - age 3: 17024 bytes, 102136 total - age 4: 64304 bytes, 166440 total - age 5: 128960 bytes, 295400 total - age 6: 69248 bytes, 364648 total - age 7: 12600 bytes, 377248 total - age 8: 160048 bytes, 537296 total - age 9: 128920 bytes, 666216 total - age 10: 21176 bytes, 687392 total - age 11: 150000 bytes, 837392 total - age 12: 221056 bytes, 1058448 total - age 13: 283672 bytes, 1342120 total - age 14: 9544 bytes, 1351664 total - age 15: 345368 bytes, 1697032 total 68117.409: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 3373, predicted base time: 12.09 ms, remaining time: 137.91 ms, target pause time: 150.00 ms] 68117.409: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 1023 regions, survivors: 1 regions, predicted young region time: 2.04 ms] 68117.409: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 1023 regions, survivors: 1 regions, old: 0 regions, predicted pause time: 14.13 ms, target pause time: 150.00 ms] 2017-01-25T14:11:12.959+0530: 68117.412: [SoftReference, 0 refs, 0.0010997 secs]2017-01-25T14:11:12.960+0530: 68117.413: [WeakReference, 0 refs, 0.0057308 secs]2017-01-25T14:11:12.966+0530: 68117.419: [FinalReference, 0 refs, 0.0006110 secs]2017-01-25T14:11:12.967+0530: 68117.419: [PhantomReference, 0 refs, 0 refs, 0.0013420 secs]2017-01-25T14:11:12.968+0530: 68117.421: [JNI Weak Reference, 0.0000150 secs], 0.0145345 secs] [Parallel Time: 2.6 ms, GC Workers: 20] [GC Worker Start (ms): Min: 68117408.8, Avg: 68117409.4, Max: 68117411.1, Diff: 2.3] [Ext Root Scanning (ms): Min: 0.0, Avg: 0.3, Max: 1.1, Diff: 1.1, Sum: 6.3] [Update RS (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.3, Sum: 2.4] [Processed Buffers: Min: 0, Avg: 1.0, Max: 5, Diff: 5, Sum: 20] [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 1.7] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 0.0, Avg: 0.7, Max: 1.1, Diff: 1.1, Sum: 15.0] [Termination (ms): Min: 0.0, Avg: 0.4, Max: 0.6, Diff: 0.6, Sum: 8.6] [Termination Attempts: Min: 1, Avg: 66.7, Max: 103, Diff: 102, Sum: 1333] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 1.5] [GC Worker Total (ms): Min: 0.0, Avg: 1.8, Max: 2.4, Diff: 2.4, Sum: 35.5] [GC Worker End (ms): Min: 68117411.2, Avg: 68117411.2, Max: 68117411.3, Diff: 0.1] [Code Root Fixup: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.5 ms] [Other: 11.4 ms] [Choose CSet: 0.0 ms] [Ref Proc: 9.3 ms] [Ref Enq: 0.4 ms] [Redirty Cards: 0.2 ms] [Humongous Register: 0.1 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 1.3 ms] [Eden: 4092.0M(4092.0M)->0.0B(4092.0M) Survivors: 4096.0K->4096.0K Heap: 5371.2M(8192.0M)->1279.1M(8192.0M)] [Times: user=0.00 sys=0.00, real=0.02 secs] -------------- next part -------------- An HTML attachment was scrubbed... URL: From poonam.bajaj at oracle.com Wed Jan 25 18:00:47 2017 From: poonam.bajaj at oracle.com (Poonam Bajaj Parhar) Date: Wed, 25 Jan 2017 10:00:47 -0800 Subject: Sudden increase in survivor space with G1GC In-Reply-To: References: Message-ID: <985286bf-7fb6-5785-0d28-fc682a05641f@oracle.com> Hello, From the GC log, I think the following is your concern: 68067.568: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 1022 regions, survivors: 2 regions, old: 0 regions, predicted pause time: 21.78 ms, target pause time: 150.00 ms] [Eden: 4088.0M(4088.0M)->0.0B(3584.0M) Survivors: 8192.0K->512.0M Heap: 4176.5M(8192.0M)->599.1M(8192.0M)] 68069.914: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 896 regions, survivors: 128 regions, predicted young region time: 75.53 ms] 68069.914: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 896 regions, survivors: 128 regions, old: 0 regions, predicted pause time: 92.62 ms, target pause time: 150.00 ms] [Eden: 3584.0M(3584.0M)->0.0B(3636.0M) Survivors: 512.0M->460.0M Heap: 4183.1M(8192.0M)->1063.5M(8192.0M)] 68071.923: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 909 regions, survivors: 115 regions, predicted young region time: 302.72 ms] 68071.923: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 909 regions, survivors: 115 regions, old: 0 regions, predicted pause time: 324.24 ms, target pause time: 150.00 ms] [Eden: 3636.0M(3636.0M)->0.0B(3940.0M) Survivors: 460.0M->156.0M Heap: 4699.5M(8192.0M)->1224.0M(8192.0M)] 68073.557: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 985 regions, survivors: 39 regions, predicted young region time: 172.21 ms] After the first line above, the number of Survivors increases from 2 to 128. This happens because the survivor data increased from 8192K to 512M. After this collection the surviving data decreases and the number of regions go down accordingly ( From 128 to 155 and then to 39). It would be worthwhile to check if the load pattern changed around this time that increased the number of surviving objects suddenly. Thanks, Poonam On 1/25/2017 7:15 AM, yugandhar nanda wrote: > Hi, > I'm facing some issue with G1GC, it's suddenly increasing it's > survivor space and then the old gen heap space. > *JVM and command info:* > > Java HotSpot(TM) 64-Bit Server VM (25.121-b13) for linux-amd64 JRE > (1.8.0_121-b13), built on Dec 12 2016 16:36:53 by "java_re" with gcc > 4.3.0 20080428 (Red Hat 4.3.0-8) > > Memory: 4k page, physical 61832044k(18159472k free), swap 0k(0k free) > > CommandLine flags: -XX:ConcGCThreads=8 -XX:G1MaxNewSizePercent=50 > -XX:G1NewSizePercent=50 -XX:+HeapDumpAfterFullGC > -XX:+HeapDumpBeforeFullGC -XX:+HeapDumpOnOutOfMemoryError > -XX:HeapDumpPath=../app-thread_dump.hprof > -XX:InitialHeapSize=8589934592 -XX:InitiatingHeapOccupancyPercent=75 > -XX:MaxGCPauseMillis=150 -XX:MaxHeapSize=8589934592 > -XX:ParallelGCThreads=20 -XX:+ParallelRefProcEnabled > -XX:+PrintAdaptiveSizePolicy -XX:+PrintClassHistogramAfterFullGC > -XX:+PrintClassHistogramBeforeFullGC -XX:+PrintGC > -XX:+PrintGCApplicationConcurrentTime > -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps > -XX:+PrintGCDetails -XX:+PrintGCTaskTimeStamps -XX:+PrintGCTimeStamps > -XX:+PrintJNIGCStalls -XX:+PrintReferenceGC -XX:+PrintTLAB > -XX:+PrintTenuringDistribution -XX:+UnlockExperimentalVMOptions > -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC > > *_In general GC log (without issue):_* > > 2017-01-25T13:02:26.812+0530: 63991.265: Total time for which > application threads were stopped: 0.0130009 seconds, Stopping threads > took: 0.0000569 seconds > > 2017-01-25T13:02:26.812+0530: 63991.265: Application time: 0.0001108 > seconds > > 2017-01-25T13:02:26.813+0530: 63991.265: Total time for which > application threads were stopped: 0.0002348 seconds, Stopping threads > took: 0.0000709 seconds > > 2017-01-25T13:02:26.813+0530: 63991.265: Application time: 0.0001053 > seconds > > 2017-01-25T13:02:26.813+0530: 63991.265: Total time for which > application threads were stopped: 0.0001487 seconds, Stopping threads > took: 0.0000642 seconds > > 2017-01-25T13:02:26.813+0530: 63991.266: Application time: 0.0000308 > seconds > > 2017-01-25T13:02:26.813+0530: 63991.266: Total time for which > application threads were stopped: 0.0000995 seconds, Stopping threads > took: 0.0000221 seconds > > 2017-01-25T13:02:26.813+0530: 63991.266: Application time: 0.0000918 > seconds > > 2017-01-25T13:02:26.813+0530: 63991.266: Total time for which > application threads were stopped: 0.0003873 seconds, Stopping threads > took: 0.0000630 seconds > > 2017-01-25T13:02:30.548+0530: 63995.001: Application time: 3.7348000 > seconds > > 2017-01-25T13:02:30.549+0530: 63995.001: Total time for which > application threads were stopped: 0.0002501 seconds, Stopping threads > took: 0.0000410 seconds > > 2017-01-25T13:02:35.549+0530: 64000.001: Application time: 5.0000879 > seconds > > 2017-01-25T13:02:35.549+0530: 64000.001: Total time for which > application threads were stopped: 0.0002537 seconds, Stopping threads > took: 0.0000520 seconds > > 2017-01-25T13:02:40.549+0530: 64005.002: Application time: 5.0001048 > seconds > > 2017-01-25T13:02:40.549+0530: 64005.002: Total time for which > application threads were stopped: 0.0002789 seconds, Stopping threads > took: 0.0000443 seconds > > 2017-01-25T13:02:45.549+0530: 64010.002: Application time: 5.0002048 > seconds > > 2017-01-25T13:02:45.550+0530: 64010.002: Total time for which > application threads were stopped: 0.0002770 seconds, Stopping threads > took: 0.0000567 seconds > > 2017-01-25T13:02:50.550+0530: 64015.003: Application time: 5.0002496 > seconds > > 2017-01-25T13:02:50.550+0530: 64015.003: Total time for which > application threads were stopped: 0.0003006 seconds, Stopping threads > took: 0.0000513 seconds > > 2017-01-25T13:02:55.551+0530: 64020.003: Application time: 5.0002249 > seconds > > 2017-01-25T13:02:55.551+0530: 64020.003: Total time for which > application threads were stopped: 0.0002606 seconds, Stopping threads > took: 0.0000475 seconds > > 2017-01-25T13:03:00.551+0530: 64025.004: Application time: 5.0001728 > seconds > > 2017-01-25T13:03:00.551+0530: 64025.004: Total time for which > application threads were stopped: 0.0002634 seconds, Stopping threads > took: 0.0000488 seconds > > 2017-01-25T13:03:05.551+0530: 64030.004: Application time: 5.0001301 > seconds > > 2017-01-25T13:03:05.552+0530: 64030.004: Total time for which > application threads were stopped: 0.0002513 seconds, Stopping threads > took: 0.0000492 seconds > > 2017-01-25T13:03:06.552+0530: 64031.004: Application time: 1.0001395 > seconds > > 2017-01-25T13:03:06.552+0530: 64031.005: Total time for which > application threads were stopped: 0.0002258 seconds, Stopping threads > took: 0.0000423 seconds > > 2017-01-25T13:03:10.552+0530: 64035.004: Application time: 3.9998560 > seconds > > 2017-01-25T13:03:10.552+0530: 64035.005: Total time for which > application threads were stopped: 0.0002408 seconds, Stopping threads > took: 0.0000388 seconds > > 2017-01-25T13:03:15.552+0530: 64040.005: Application time: 5.0002342 > seconds > > 2017-01-25T13:03:15.553+0530: 64040.005: Total time for which > application threads were stopped: 0.0002513 seconds, Stopping threads > took: 0.0000489 seconds > > 2017-01-25T13:03:20.553+0530: 64045.005: Application time: 5.0001538 > seconds > > 2017-01-25T13:03:20.553+0530: 64045.006: Total time for which > application threads were stopped: 0.0002752 seconds, Stopping threads > took: 0.0000501 seconds > > 2017-01-25T13:03:25.553+0530: 64050.006: Application time: 5.0001211 > seconds > > 2017-01-25T13:03:25.553+0530: 64050.006: Total time for which > application threads were stopped: 0.0002745 seconds, Stopping threads > took: 0.0000516 seconds > > 2017-01-25T13:03:30.554+0530: 64055.006: Application time: 5.0001992 > seconds > > 2017-01-25T13:03:30.554+0530: 64055.006: Total time for which > application threads were stopped: 0.0002515 seconds, Stopping threads > took: 0.0000456 seconds > > 2017-01-25T13:03:35.554+0530: 64060.007: Application time: 5.0001805 > seconds > > 2017-01-25T13:03:35.554+0530: 64060.007: Total time for which > application threads were stopped: 0.0002972 seconds, Stopping threads > took: 0.0000563 seconds > > 2017-01-25T13:03:40.555+0530: 64065.007: Application time: 5.0002179 > seconds > > 2017-01-25T13:03:40.555+0530: 64065.007: Total time for which > application threads were stopped: 0.0002622 seconds, Stopping threads > took: 0.0000415 seconds > > 2017-01-25T13:03:45.555+0530: 64070.008: Application time: 5.0002244 > seconds > > 2017-01-25T13:03:45.555+0530: 64070.008: Total time for which > application threads were stopped: 0.0002557 seconds, Stopping threads > took: 0.0000399 seconds > > 2017-01-25T13:03:50.556+0530: 64075.008: Application time: 5.0001938 > seconds > > 2017-01-25T13:03:50.556+0530: 64075.008: Total time for which > application threads were stopped: 0.0002542 seconds, Stopping threads > took: 0.0000422 seconds > > 2017-01-25T13:03:55.556+0530: 64080.009: Application time: 5.0001925 > seconds > > 2017-01-25T13:03:55.556+0530: 64080.009: Total time for which > application threads were stopped: 0.0002727 seconds, Stopping threads > took: 0.0000435 seconds > > 2017-01-25T13:04:00.556+0530: 64085.009: Application time: 5.0002240 > seconds > > 2017-01-25T13:04:00.557+0530: 64085.009: Total time for which > application threads were stopped: 0.0002573 seconds, Stopping threads > took: 0.0000387 seconds > > 2017-01-25T13:04:05.557+0530: 64090.010: Application time: 5.0002203 > seconds > > 2017-01-25T13:04:05.557+0530: 64090.010: Total time for which > application threads were stopped: 0.0003009 seconds, Stopping threads > took: 0.0000419 seconds > > 2017-01-25T13:04:10.557+0530: 64095.010: Application time: 5.0002308 > seconds > > 2017-01-25T13:04:10.558+0530: 64095.010: Total time for which > application threads were stopped: 0.0002892 seconds, Stopping threads > took: 0.0000502 seconds > > 2017-01-25T13:04:15.558+0530: 64100.011: Application time: 5.0001824 > seconds > > 2017-01-25T13:04:15.558+0530: 64100.011: Total time for which > application threads were stopped: 0.0002421 seconds, Stopping threads > took: 0.0000407 seconds > > 2017-01-25T13:04:20.558+0530: 64105.011: Application time: 5.0002074 > seconds > > 2017-01-25T13:04:20.559+0530: 64105.011: Total time for which > application threads were stopped: 0.0002598 seconds, Stopping threads > took: 0.0000443 seconds > > 2017-01-25T13:04:25.559+0530: 64110.012: Application time: 5.0002581 > seconds > > 2017-01-25T13:04:25.559+0530: 64110.012: Total time for which > application threads were stopped: 0.0002521 seconds, Stopping threads > took: 0.0000418 seconds > > 2017-01-25T13:04:30.559+0530: 64115.012: Application time: 5.0002090 > seconds > > 2017-01-25T13:04:30.560+0530: 64115.012: Total time for which > application threads were stopped: 0.0002510 seconds, Stopping threads > took: 0.0000409 seconds > > 2017-01-25T13:04:35.560+0530: 64120.012: Application time: 5.0001457 > seconds > > 2017-01-25T13:04:35.560+0530: 64120.013: Total time for which > application threads were stopped: 0.0002465 seconds, Stopping threads > took: 0.0000399 seconds > > 2017-01-25T13:04:37.135+0530: 64121.588: Application time: 1.5748912 > seconds > > 2017-01-25T13:04:37.135+0530: 64121.588: [GC pause (G1 Evacuation > Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] > desired_size: 219KB slow allocs: 0 refill waste: 3504B alloc: 0.00262 > 10964KB refills: 46 waste 0.1% gc: 1536B slow: 7312B fast: 0B > > TLAB: gc thread: 0x00007f15f8114800 [id: 199] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.03023 126678KB > refills: 55 waste 1.3% gc: 1534976B slow: 17184B fast: 0B > > TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 7KB slow > allocs: 0 refill waste: 120B alloc: 0.00009 375KB refills: 45 > waste 2.7% gc: 7512B slow: 1672B fast: 0B > > TLAB: gc thread: 0x00007f15906a8800 [id: 144] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00001 21KB refills: 8 > waste 7.3% gc: 1008B slow: 200B fast: 0B > > TLAB: gc thread: 0x00007f15906a7000 [id: 143] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00001 21KB refills: 8 > waste 6.3% gc: 880B slow: 168B fast: 0B > > TLAB: gc thread: 0x00007f15906a5000 [id: 142] desired_size: 2KB slow > allocs: 2 refill waste: 32B alloc: 0.00001 21KB refills: 8 > waste 7.0% gc: 928B slow: 224B fast: 0B > > TLAB: gc thread: 0x00007f15906a3800 [id: 141] desired_size: 2KB slow > allocs: 7 refill waste: 32B alloc: 0.00001 56KB refills: 41 > waste 1.5% gc: 40B slow: 1240B fast: 0B > > TLAB: gc thread: 0x00007f15906a1800 [id: 140] desired_size: 990KB slow > allocs: 1 refill waste: 15848B alloc: 0.01182 49545KB refills: 42 > waste 2.1% gc: 887440B slow: 5288B fast: 0B > > TLAB: gc thread: 0x00007f15906a0000 [id: 139] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00001 21KB refills: 8 > waste 7.3% gc: 1008B slow: 200B fast: 0B > > TLAB: gc thread: 0x00007f159069e800 [id: 138] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00001 21KB refills: 8 > waste 7.3% gc: 1008B slow: 200B fast: 0B > > TLAB: gc thread: 0x00007f159069c800 [id: 137] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00001 21KB refills: 8 > waste 6.8% gc: 928B slow: 200B fast: 0B > > TLAB: gc thread: 0x00007f159069b000 [id: 136] desired_size: 2KB slow > allocs: 0 refill waste: 40B alloc: 0.00004 147KB refills: 5 > waste 4.8% gc: 568B slow: 152B fast: 0B > > TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 939KB slow > allocs: 0 refill waste: 15032B alloc: 0.01122 46995KB refills: 49 > waste 1.4% gc: 644224B slow: 10456B fast: 0B > > TLAB: gc thread: 0x00007f1590697800 [id: 134] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00001 21KB refills: 8 > waste 6.8% gc: 928B slow: 200B fast: 0B > > TLAB: gc thread: 0x00007f1590695800 [id: 133] desired_size: 2KB slow > allocs: 2 refill waste: 32B alloc: 0.00001 21KB refills: 8 > waste 7.5% gc: 1008B slow: 224B fast: 0B > > TLAB: gc thread: 0x00007f1590693800 [id: 132] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00001 21KB refills: 8 > waste 7.3% gc: 1008B slow: 200B fast: 0B > > TLAB: gc thread: 0x00007f1590692000 [id: 131] desired_size: 2KB slow > allocs: 9 refill waste: 32B alloc: 0.00002 66KB refills: 28 > waste 4.3% gc: 1656B slow: 832B fast: 0B > > TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 966KB slow > allocs: 0 refill waste: 15456B alloc: 0.01153 48313KB refills: 47 > waste 2.1% gc: 972808B slow: 7088B fast: 0B > > TLAB: gc thread: 0x00007f159068e800 [id: 129] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00001 21KB refills: 8 > waste 6.8% gc: 928B slow: 200B fast: 0B > > TLAB: gc thread: 0x00007f159068d000 [id: 128] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00001 21KB refills: 8 > waste 7.3% gc: 1008B slow: 200B fast: 0B > > TLAB: gc thread: 0x00007f159068b000 [id: 127] desired_size: 2KB slow > allocs: 2 refill waste: 32B alloc: 0.00001 22KB refills: 7 > waste 15.5% gc: 2064B slow: 176B fast: 0B > > TLAB: gc thread: 0x00007f1590689800 [id: 126] desired_size: 4KB slow > allocs: 0 refill waste: 64B alloc: 0.00005 207KB refills: 5 > waste 16.6% gc: 3376B slow: 152B fast: 0B > > TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 995KB slow > allocs: 0 refill waste: 15920B alloc: 0.01188 49768KB refills: 46 > waste 1.0% gc: 478608B slow: 11312B fast: 0B > > TLAB: gc thread: 0x00007f1590674800 [id: 124] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00001 21KB refills: 8 > waste 6.8% gc: 928B slow: 200B fast: 0B > > TLAB: gc thread: 0x00007f1590673000 [id: 123] desired_size: 2KB slow > allocs: 2 refill waste: 32B alloc: 0.00001 21KB refills: 8 > waste 7.5% gc: 1008B slow: 224B fast: 0B > > TLAB: gc thread: 0x00007f1590671800 [id: 122] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00001 21KB refills: 8 > waste 7.3% gc: 1008B slow: 200B fast: 0B > > TLAB: gc thread: 0x00007f159067d800 [id: 121] desired_size: 2KB slow > allocs: 12 refill waste: 32B alloc: 0.00002 85KB refills: 73 > waste 1.7% gc: 360B slow: 2264B fast: 0B > > TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 981KB slow > allocs: 0 refill waste: 15696B alloc: 0.01171 49074KB refills: 46 > waste 0.6% gc: 282728B slow: 4480B fast: 0B > > TLAB: gc thread: 0x00007f1590637000 [id: 117] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00001 21KB refills: 8 > waste 9.5% gc: 1360B slow: 216B fast: 0B > > TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 125KB slow > allocs: 0 refill waste: 2008B alloc: 0.00150 6286KB refills: 50 > waste 1.2% gc: 74680B slow: 1784B fast: 0B > > TLAB: gc thread: 0x00007f159062b000 [id: 115] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 4KB refills: 2 > waste 38.8% gc: 1568B slow: 32B fast: 0B > > TLAB: gc thread: 0x00007f159005b000 [id: 114] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 1KB refills: 1 > waste 60.9% gc: 1256B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f158c010800 [id: 113] desired_size: 37KB slow > allocs: 47 refill waste: 592B alloc: 0.00045 1870KB refills: 31 > waste 1.8% gc: 19784B slow: 2160B fast: 0B > > TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.19981 837259KB > refills: 1519 waste 0.0% gc: 0B slow: 91944B fast: 0B > > TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.08702 364643KB > refills: 173 waste 0.2% gc: 650768B slow: 11936B fast: 0B > > TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 2047KB > slow allocs: 40 refill waste: 32760B alloc: 0.14335 600674KB > refills: 261 waste 0.1% gc: 158528B slow: 512864B fast: 0B > > TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 501KB slow > allocs: 0 refill waste: 8024B alloc: 0.00598 25076KB refills: 45 > waste 1.2% gc: 280640B slow: 4848B fast: 0B > > TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 586KB slow > allocs: 0 refill waste: 9384B alloc: 0.00700 29348KB refills: 44 > waste 1.8% gc: 484792B slow: 3672B fast: 0B > > TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 2047KB > slow allocs: 112 refill waste: 32760B alloc: 0.45892 1922957KB > refills: 799 waste 0.1% gc: 747928B slow: 1461464B fast: 0B > > TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 10KB refills: 4 > waste 26.0% gc: 2016B slow: 128B fast: 0B > > TLAB: gc thread: 0x00007f170d35a000 [id: 85] desired_size: 165KB slow > allocs: 0 refill waste: 2640B alloc: 0.00197 8251KB refills: 30 > waste 0.5% gc: 22672B slow: 1704B fast: 0B > > TLAB: gc thread: 0x00007f170d2cf000 [id: 84] desired_size: 2KB slow > allocs: 2 refill waste: 32B alloc: 0.00000 15KB refills: 5 > waste 1.8% gc: 72B slow: 112B fast: 0B > > TLAB: gc thread: 0x00007f170cda3800 [id: 63] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 0KB refills: 1 > waste 86.4% gc: 1784B slow: 0B fast: 0B > > TLAB totals: thrds: 43 refills: 3620 max: 1519 slow allocs: 249 max > 112 waste: 0.2% gc: 7279328B max: 1534976B slow: 2165312B max: > 1461464B fast: 0B max: 0B > > > Desired survivor size 268435456 bytes, new threshold 15 (max 15) > > - age 1: 1800488 bytes, 1800488 total > > - age 2: 14336 bytes, 1814824 total > > - age 3: 464 bytes, 1815288 total > > - age 4: 17296 bytes, 1832584 total > > - age 5: 144 bytes, 1832728 total > > - age 6: 144 bytes, 1832872 total > > - age 7: 5744 bytes, 1838616 total > > - age 8: 72 bytes, 1838688 total > > - age 9: 864 bytes, 1839552 total > > - age 10: 48 bytes, 1839600 total > > - age 11: 48 bytes, 1839648 total > > - age 13: 80 bytes, 1839728 total > > - age 14: 656 bytes, 1840384 total > > - age 15: 48 bytes, 1840432 total > > 64121.588: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 4231, predicted base time: 9.67 ms, remaining time: > 140.33 ms, target pause time: 150.00 ms] > > 64121.588: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 1023 regions, survivors: 1 regions, predicted young region > time: 2.93 ms] > > 64121.588: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 1023 regions, survivors: 1 regions, old: 0 regions, predicted > pause time: 12.60 ms, target pause time: 150.00 ms] > > 2017-01-25T13:04:37.139+0530: 64121.591: [SoftReference, 0 refs, > 0.0052402 secs]2017-01-25T13:04:37.144+0530: 64121.597: > [WeakReference, 3 refs, 0.0008352 secs]2017-01-25T13:04:37.145+0530: > 64121.597: [FinalReference, 35 refs, 0.0012709 > secs]2017-01-25T13:04:37.146+0530: 64121.599: [PhantomReference, 0 > refs, 0 refs, 0.0017444 secs]2017-01-25T13:04:37.148+0530: 64121.601: > [JNI Weak Reference, 0.0000165 secs], 0.0161273 secs] > > [Parallel Time: 2.7 ms, GC Workers: 20] > > [GC Worker Start (ms): Min: 64121588.1, Avg: 64121588.8, Max: > 64121590.5, Diff: 2.4] > > [Ext Root Scanning (ms): Min: 0.0, Avg: 0.7, Max: 2.3, Diff: > 2.3, Sum: 14.9] > > [Update RS (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.3, Sum: 1.7] > > [Processed Buffers: Min: 0, Avg: 2.8, Max: 8, Diff: 8, Sum: 56] > > [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.2, Sum: 2.4] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > > [Object Copy (ms): Min: 0.0, Avg: 0.6, Max: 1.2, Diff: 1.2, Sum: > 13.0] > > [Termination (ms): Min: 0.0, Avg: 0.2, Max: 0.3, Diff: 0.3, Sum: > 3.7] > > [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, > Sum: 20] > > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, > Sum: 1.2] > > [GC Worker Total (ms): Min: 0.1, Avg: 1.8, Max: 2.5, Diff: 2.5, > Sum: 36.8] > > [GC Worker End (ms): Min: 64121590.6, Avg: 64121590.6, Max: > 64121590.7, Diff: 0.1] > > [Code Root Fixup: 0.0 ms] > > [Code Root Purge: 0.0 ms] > > [Clear CT: 0.7 ms] > > [Other: 12.6 ms] > > [Choose CSet: 0.0 ms] > > [Ref Proc: 9.7 ms] > > [Ref Enq: 0.4 ms] > > [Redirty Cards: 0.3 ms] > > [Humongous Register: 0.1 ms] > > [Humongous Reclaim: 0.0 ms] > > [Free CSet: 1.9 ms] > > [Eden: 4092.0M(4092.0M)->0.0B(4088.0M) Survivors: 4096.0K->8192.0K > Heap: 4179.6M(8192.0M)->90.7M(8192.0M)] > > [Times: user=0.16 sys=0.00, real=0.02 secs] > > 2017-01-25T13:04:37.151+0530: 64121.604: Total time for which > application threads were stopped: 0.0165431 seconds, Stopping threads > took: 0.0000626 seconds > > 2017-01-25T13:04:37.152+0530: 64121.604: Application time: 0.0000825 > seconds > > 2017-01-25T13:04:37.152+0530: 64121.604: Total time for which > application threads were stopped: 0.0002369 seconds, Stopping threads > took: 0.0000639 seconds > > 2017-01-25T13:04:37.152+0530: 64121.604: Application time: 0.0000353 > seconds > > 2017-01-25T13:04:37.152+0530: 64121.605: Total time for which > application threads were stopped: 0.0001141 seconds, Stopping threads > took: 0.0000393 seconds > > 2017-01-25T13:04:37.152+0530: 64121.605: Application time: 0.0000278 > seconds > > 2017-01-25T13:04:37.152+0530: 64121.605: Total time for which > application threads were stopped: 0.0001138 seconds, Stopping threads > took: 0.0000388 seconds > > 2017-01-25T13:04:37.152+0530: 64121.605: Application time: 0.0000261 > seconds > > 2017-01-25T13:04:37.152+0530: 64121.605: Total time for which > application threads were stopped: 0.0001220 seconds, Stopping threads > took: 0.0000420 seconds > > 2017-01-25T13:04:37.152+0530: 64121.605: Application time: 0.0000273 > seconds > > 2017-01-25T13:04:37.152+0530: 64121.605: Total time for which > application threads were stopped: 0.0001167 seconds, Stopping threads > took: 0.0000382 seconds > > 2017-01-25T13:04:37.152+0530: 64121.605: Application time: 0.0000297 > seconds > > 2017-01-25T13:04:37.153+0530: 64121.605: Total time for which > application threads were stopped: 0.0001116 seconds, Stopping threads > took: 0.0000387 seconds > > 2017-01-25T13:04:37.153+0530: 64121.605: Application time: 0.0000242 > seconds > > 2017-01-25T13:04:37.153+0530: 64121.605: Total time for which > application threads were stopped: 0.0001297 seconds, Stopping threads > took: 0.0000483 seconds > > 2017-01-25T13:04:37.153+0530: 64121.605: Application time: 0.0000237 > seconds > > 2017-01-25T13:04:37.153+0530: 64121.605: Total time for which > application threads were stopped: 0.0001265 seconds, Stopping threads > took: 0.0000466 seconds > > 2017-01-25T13:04:40.560+0530: 64125.013: Application time: 3.4073931 > seconds > > 2017-01-25T13:04:40.561+0530: 64125.013: Total time for which > application threads were stopped: 0.0002536 seconds, Stopping threads > took: 0.0000435 seconds > > 2017-01-25T13:04:45.561+0530: 64130.013: Application time: 5.0002173 > seconds > > 2017-01-25T13:04:45.561+0530: 64130.014: Total time for which > application threads were stopped: 0.0002343 seconds, Stopping threads > took: 0.0000381 seconds > > 2017-01-25T13:04:50.561+0530: 64135.014: Application time: 5.0002234 > seconds > > 2017-01-25T13:04:50.561+0530: 64135.014: Total time for which > application threads were stopped: 0.0002375 seconds, Stopping threads > took: 0.0000394 seconds > > 2017-01-25T13:04:55.562+0530: 64140.014: Application time: 5.0002266 > seconds > > 2017-01-25T13:04:55.562+0530: 64140.015: Total time for which > application threads were stopped: 0.0002543 seconds, Stopping threads > took: 0.0000466 seconds > > 2017-01-25T13:05:00.562+0530: 64145.015: Application time: 5.0001965 > seconds > > 2017-01-25T13:05:00.562+0530: 64145.015: Total time for which > application threads were stopped: 0.0002787 seconds, Stopping threads > took: 0.0000461 seconds > > 2017-01-25T13:05:05.563+0530: 64150.015: Application time: 5.0002188 > seconds > > 2017-01-25T13:05:05.563+0530: 64150.015: Total time for which > application threads were stopped: 0.0002429 seconds, Stopping threads > took: 0.0000443 seconds > > 2017-01-25T13:05:10.563+0530: 64155.016: Application time: 5.0001550 > seconds > > 2017-01-25T13:05:10.563+0530: 64155.016: Total time for which > application threads were stopped: 0.0002682 seconds, Stopping threads > took: 0.0000528 seconds > > 2017-01-25T13:05:15.563+0530: 64160.016: Application time: 5.0001906 > seconds > > 2017-01-25T13:05:15.564+0530: 64160.016: Total time for which > application threads were stopped: 0.0002624 seconds, Stopping threads > took: 0.0000533 seconds > > 2017-01-25T13:05:20.564+0530: 64165.016: Application time: 5.0001018 > seconds > > 2017-01-25T13:05:20.564+0530: 64165.017: Total time for which > application threads were stopped: 0.0003091 seconds, Stopping threads > took: 0.0000802 seconds > > 2017-01-25T13:05:25.564+0530: 64170.017: Application time: 5.0001915 > seconds > > 2017-01-25T13:05:25.565+0530: 64170.017: Total time for which > application threads were stopped: 0.0002597 seconds, Stopping threads > took: 0.0000489 seconds > > 2017-01-25T13:05:30.565+0530: 64175.017: Application time: 5.0001223 > seconds > > 2017-01-25T13:05:30.565+0530: 64175.018: Total time for which > application threads were stopped: 0.0002646 seconds, Stopping threads > took: 0.0000521 seconds > > 2017-01-25T13:05:35.565+0530: 64180.018: Application time: 5.0001622 > seconds > > 2017-01-25T13:05:35.565+0530: 64180.018: Total time for which > application threads were stopped: 0.0002546 seconds, Stopping threads > took: 0.0000495 seconds > > 2017-01-25T13:05:40.566+0530: 64185.018: Application time: 5.0002130 > seconds > > 2017-01-25T13:05:40.566+0530: 64185.019: Total time for which > application threads were stopped: 0.0002766 seconds, Stopping threads > took: 0.0000417 seconds > > 2017-01-25T13:05:45.566+0530: 64190.019: Application time: 5.0002184 > seconds > > 2017-01-25T13:05:45.566+0530: 64190.019: Total time for which > application threads were stopped: 0.0002403 seconds, Stopping threads > took: 0.0000375 seconds > > 2017-01-25T13:05:50.567+0530: 64195.019: Application time: 5.0002563 > seconds > > 2017-01-25T13:05:50.567+0530: 64195.020: Total time for which > application threads were stopped: 0.0002875 seconds, Stopping threads > took: 0.0000455 seconds > > 2017-01-25T13:05:55.567+0530: 64200.020: Application time: 5.0002347 > seconds > > 2017-01-25T13:05:55.567+0530: 64200.020: Total time for which > application threads were stopped: 0.0002582 seconds, Stopping threads > took: 0.0000465 seconds > > 2017-01-25T13:06:00.035+0530: 64204.488: Application time: 4.4678607 > seconds > > 2017-01-25T13:06:00.036+0530: 64204.488: Total time for which > application threads were stopped: 0.0002706 seconds, Stopping threads > took: 0.0000526 seconds > > 2017-01-25T13:06:00.568+0530: 64205.020: Application time: 0.5320424 > seconds > > 2017-01-25T13:06:00.568+0530: 64205.020: Total time for which > application threads were stopped: 0.0002378 seconds, Stopping threads > took: 0.0000537 seconds > > 2017-01-25T13:06:05.568+0530: 64210.021: Application time: 5.0002275 > seconds > > 2017-01-25T13:06:05.568+0530: 64210.021: Total time for which > application threads were stopped: 0.0002358 seconds, Stopping threads > took: 0.0000394 seconds > > 2017-01-25T13:06:10.569+0530: 64215.021: Application time: 5.0002358 > seconds > > 2017-01-25T13:06:10.569+0530: 64215.021: Total time for which > application threads were stopped: 0.0002551 seconds, Stopping threads > took: 0.0000409 seconds > > 2017-01-25T13:06:15.569+0530: 64220.022: Application time: 5.0002107 > seconds > > 2017-01-25T13:06:15.569+0530: 64220.022: Total time for which > application threads were stopped: 0.0002469 seconds, Stopping threads > took: 0.0000403 seconds > > 2017-01-25T13:06:20.569+0530: 64225.022: Application time: 5.0001874 > seconds > > 2017-01-25T13:06:20.570+0530: 64225.022: Total time for which > application threads were stopped: 0.0002622 seconds, Stopping threads > took: 0.0000424 seconds > > 2017-01-25T13:06:25.570+0530: 64230.023: Application time: 5.0002091 > seconds > > 2017-01-25T13:06:25.570+0530: 64230.023: Total time for which > application threads were stopped: 0.0002537 seconds, Stopping threads > took: 0.0000461 seconds > > 2017-01-25T13:06:30.570+0530: 64235.023: Application time: 5.0002466 > seconds > > 2017-01-25T13:06:30.571+0530: 64235.023: Total time for which > application threads were stopped: 0.0002569 seconds, Stopping threads > took: 0.0000517 seconds > > 2017-01-25T13:06:35.571+0530: 64240.024: Application time: 5.0002156 > seconds > > 2017-01-25T13:06:35.571+0530: 64240.024: Total time for which > application threads were stopped: 0.0002409 seconds, Stopping threads > took: 0.0000435 seconds > > 2017-01-25T13:06:37.364+0530: 64241.817: Application time: 1.7927933 > seconds > > 2017-01-25T13:06:37.364+0530: 64241.817: [GC pause (G1 Evacuation > Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] > desired_size: 188KB slow allocs: 0 refill waste: 3016B alloc: 0.00225 > 9430KB refills: 47 waste 1.0% gc: 79928B slow: 6736B fast: 0B > > TLAB: gc thread: 0x00007f15f8114800 [id: 199] desired_size: 2047KB > slow allocs: 1 refill waste: 32760B alloc: 0.02598 108755KB > refills: 60 waste 0.0% gc: 0B slow: 15240B fast: 0B > > TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 6KB slow > allocs: 0 refill waste: 104B alloc: 0.00008 326KB refills: 43 > waste 0.8% gc: 632B slow: 1752B fast: 0B > > TLAB: gc thread: 0x00007f14d4080000 [id: 197] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 0KB refills: 1 > waste 95.0% gc: 1960B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f15906a8800 [id: 144] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00000 19KB refills: 7 > waste 15.3% gc: 2064B slow: 152B fast: 0B > > TLAB: gc thread: 0x00007f15906a7000 [id: 143] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00000 19KB refills: 7 > waste 15.3% gc: 2064B slow: 152B fast: 0B > > TLAB: gc thread: 0x00007f15906a5000 [id: 142] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00000 19KB refills: 7 > waste 15.3% gc: 2064B slow: 152B fast: 0B > > TLAB: gc thread: 0x00007f15906a3800 [id: 141] desired_size: 2KB slow > allocs: 2 refill waste: 32B alloc: 0.00002 67KB refills: 7 > waste 15.5% gc: 2064B slow: 176B fast: 0B > > TLAB: gc thread: 0x00007f15906a1800 [id: 140] desired_size: 858KB slow > allocs: 0 refill waste: 13736B alloc: 0.01026 42933KB refills: 42 > waste 1.5% gc: 538704B slow: 2328B fast: 0B > > TLAB: gc thread: 0x00007f15906a0000 [id: 139] desired_size: 2KB slow > allocs: 2 refill waste: 32B alloc: 0.00000 19KB refills: 7 > waste 15.5% gc: 2064B slow: 176B fast: 0B > > TLAB: gc thread: 0x00007f159069e800 [id: 138] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00000 19KB refills: 7 > waste 15.3% gc: 2064B slow: 152B fast: 0B > > TLAB: gc thread: 0x00007f159069c800 [id: 137] desired_size: 2KB slow > allocs: 2 refill waste: 32B alloc: 0.00000 19KB refills: 7 > waste 15.5% gc: 2064B slow: 176B fast: 0B > > TLAB: gc thread: 0x00007f159069b000 [id: 136] desired_size: 2KB slow > allocs: 11 refill waste: 32B alloc: 0.00002 99KB refills: 54 > waste 2.1% gc: 704B slow: 1688B fast: 0B > > TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 840KB slow > allocs: 0 refill waste: 13448B alloc: 0.01004 42032KB refills: 45 > waste 2.0% gc: 758712B slow: 3304B fast: 0B > > TLAB: gc thread: 0x00007f1590697800 [id: 134] desired_size: 2KB slow > allocs: 2 refill waste: 32B alloc: 0.00000 19KB refills: 7 > waste 15.5% gc: 2064B slow: 176B fast: 0B > > TLAB: gc thread: 0x00007f1590695800 [id: 133] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00000 19KB refills: 7 > waste 15.3% gc: 2064B slow: 152B fast: 0B > > TLAB: gc thread: 0x00007f1590693800 [id: 132] desired_size: 2KB slow > allocs: 2 refill waste: 32B alloc: 0.00000 19KB refills: 7 > waste 15.5% gc: 2064B slow: 176B fast: 0B > > TLAB: gc thread: 0x00007f1590692000 [id: 131] desired_size: 2KB slow > allocs: 7 refill waste: 32B alloc: 0.00002 63KB refills: 28 > waste 4.4% gc: 1752B slow: 808B fast: 0B > > TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 860KB slow > allocs: 1 refill waste: 13760B alloc: 0.01028 43013KB refills: 49 > waste 1.6% gc: 682184B slow: 18672B fast: 0B > > TLAB: gc thread: 0x00007f159068e800 [id: 129] desired_size: 2KB slow > allocs: 2 refill waste: 32B alloc: 0.00000 19KB refills: 7 > waste 15.5% gc: 2064B slow: 176B fast: 0B > > TLAB: gc thread: 0x00007f159068d000 [id: 128] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00000 19KB refills: 7 > waste 15.3% gc: 2064B slow: 152B fast: 0B > > TLAB: gc thread: 0x00007f159068b000 [id: 127] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00000 19KB refills: 7 > waste 15.3% gc: 2064B slow: 152B fast: 0B > > TLAB: gc thread: 0x00007f1590689800 [id: 126] desired_size: 2KB slow > allocs: 1 refill waste: 40B alloc: 0.00003 140KB refills: 8 > waste 6.8% gc: 1360B slow: 208B fast: 0B > > TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 875KB slow > allocs: 0 refill waste: 14000B alloc: 0.01046 43769KB refills: 49 > waste 1.4% gc: 614224B slow: 1680B fast: 0B > > TLAB: gc thread: 0x00007f1590674800 [id: 124] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00000 19KB refills: 7 > waste 14.3% gc: 1888B slow: 176B fast: 0B > > TLAB: gc thread: 0x00007f1590673000 [id: 123] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00000 19KB refills: 7 > waste 15.3% gc: 2064B slow: 152B fast: 0B > > TLAB: gc thread: 0x00007f1590671800 [id: 122] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00000 19KB refills: 7 > waste 15.3% gc: 2064B slow: 152B fast: 0B > > TLAB: gc thread: 0x00007f159067d800 [id: 121] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00003 111KB refills: 6 > waste 3.7% gc: 376B slow: 128B fast: 0B > > TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 852KB slow > allocs: 1 refill waste: 13640B alloc: 0.01018 42632KB refills: 44 > waste 0.4% gc: 143120B slow: 9160B fast: 0B > > TLAB: gc thread: 0x00007f1590637000 [id: 117] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 19KB refills: 6 > waste 2.3% gc: 136B slow: 144B fast: 0B > > TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 109KB slow > allocs: 0 refill waste: 1744B alloc: 0.00131 5470KB refills: 53 > waste 1.9% gc: 111032B slow: 2056B fast: 0B > > TLAB: gc thread: 0x00007f159062b000 [id: 115] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 4KB refills: 2 > waste 45.7% gc: 1872B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f159005b000 [id: 114] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 1KB refills: 1 > waste 68.6% gc: 1416B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f158c010800 [id: 113] desired_size: 36KB slow > allocs: 26 refill waste: 584B alloc: 0.00044 1829KB refills: 21 > waste 3.0% gc: 21936B slow: 1968B fast: 0B > > TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.30200 1264205KB > refills: 1606 waste 0.0% gc: 143712B slow: 92912B fast: 0B > > TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.07563 316602KB > refills: 169 waste 0.4% gc: 1284112B slow: 13088B fast: 0B > > TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 2047KB > slow allocs: 11 refill waste: 32760B alloc: 0.12518 524007KB > refills: 226 waste 0.2% gc: 799824B slow: 362304B fast: 0B > > TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 435KB slow > allocs: 0 refill waste: 6968B alloc: 0.00520 21777KB refills: 46 > waste 1.6% gc: 321056B slow: 4136B fast: 0B > > TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 509KB slow > allocs: 0 refill waste: 8144B alloc: 0.00609 25475KB refills: 51 > waste 1.9% gc: 487952B slow: 6592B fast: 0B > > TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 2047KB > slow allocs: 83 refill waste: 32760B alloc: 0.40043 1676245KB > refills: 741 waste 0.1% gc: 869336B slow: 1203552B fast: 0B > > TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 9KB refills: 3 > waste 8.3% gc: 432B slow: 80B fast: 0B > > TLAB: gc thread: 0x00007f170d362000 [id: 86] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 0KB refills: 1 > waste 96.9% gc: 2000B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f170d35a000 [id: 85] desired_size: 139KB slow > allocs: 0 refill waste: 2224B alloc: 0.00166 6968KB refills: 38 > waste 1.3% gc: 67248B slow: 2624B fast: 0B > > TLAB: gc thread: 0x00007f170d2cf000 [id: 84] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00000 12KB refills: 4 > waste 11.2% gc: 840B slow: 88B fast: 0B > > TLAB: gc thread: 0x00007f170cda3800 [id: 63] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 0KB refills: 1 > waste 91.5% gc: 1888B slow: 0B fast: 0B > > TLAB totals: thrds: 45 refills: 3557 max: 1606 slow allocs: 165 max > 83 waste: 0.1% gc: 6971296B max: 1284112B slow: 1753864B max: > 1203552B fast: 0B max: 0B > > > Desired survivor size 268435456 bytes, new threshold 15 (max 15) > > - age 1: 4323832 bytes, 4323832 total > > - age 2: 14160 bytes, 4337992 total > > - age 3: 10912 bytes, 4348904 total > > - age 5: 176 bytes, 4349080 total > > - age 6: 144 bytes, 4349224 total > > - age 7: 144 bytes, 4349368 total > > - age 8: 5744 bytes, 4355112 total > > - age 9: 72 bytes, 4355184 total > > - age 10: 832 bytes, 4356016 total > > - age 11: 48 bytes, 4356064 total > > - age 12: 48 bytes, 4356112 total > > - age 14: 80 bytes, 4356192 total > > - age 15: 656 bytes, 4356848 total > > 64241.817: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 3714, predicted base time: 11.21 ms, remaining time: > 138.79 ms, target pause time: 150.00 ms] > > 64241.817: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 1022 regions, survivors: 2 regions, predicted young region > time: 2.25 ms] > > 64241.817: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 1022 regions, survivors: 2 regions, old: 0 regions, predicted > pause time: 13.46 ms, target pause time: 150.00 ms] > > 2017-01-25T13:06:37.367+0530: 64241.820: [SoftReference, 0 refs, > 0.0058087 secs]2017-01-25T13:06:37.373+0530: 64241.826: > [WeakReference, 1 refs, 0.0028315 secs]2017-01-25T13:06:37.376+0530: > 64241.828: [FinalReference, 36 refs, 0.0008956 > secs]2017-01-25T13:06:37.377+0530: 64241.829: [PhantomReference, 0 > refs, 0 refs, 0.0108982 secs]2017-01-25T13:06:37.388+0530: 64241.840: > [JNI Weak Reference, 0.0000261 secs], 0.0263600 secs] > > [Parallel Time: 2.1 ms, GC Workers: 20] > > [GC Worker Start (ms): Min: 64241817.2, Avg: 64241817.3, Max: > 64241817.5, Diff: 0.3] > > [Ext Root Scanning (ms): Min: 0.3, Avg: 1.0, Max: 1.8, Diff: > 1.5, Sum: 19.4] > > [Update RS (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.3, Sum: 1.7] > > [Processed Buffers: Min: 0, Avg: 2.8, Max: 9, Diff: 9, Sum: 56] > > [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 1.7] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > > [Object Copy (ms): Min: 0.0, Avg: 0.3, Max: 0.5, Diff: 0.5, Sum: > 5.6] > > [Termination (ms): Min: 0.0, Avg: 0.3, Max: 0.4, Diff: 0.4, Sum: > 6.0] > > [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, > Sum: 20] > > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, > Sum: 1.0] > > [GC Worker Total (ms): Min: 1.6, Avg: 1.8, Max: 1.9, Diff: 0.3, > Sum: 35.4] > > [GC Worker End (ms): Min: 64241819.1, Avg: 64241819.1, Max: > 64241819.2, Diff: 0.1] > > [Code Root Fixup: 0.1 ms] > > [Code Root Purge: 0.0 ms] > > [Clear CT: 0.6 ms] > > [Other: 23.7 ms] > > [Choose CSet: 0.0 ms] > > [Ref Proc: 21.0 ms] > > [Ref Enq: 0.3 ms] > > [Redirty Cards: 0.2 ms] > > [Humongous Register: 0.1 ms] > > [Humongous Reclaim: 0.0 ms] > > [Free CSet: 1.6 ms] > > [Eden: 4088.0M(4088.0M)->0.0B(4088.0M) Survivors: 8192.0K->8192.0K > Heap: 4178.7M(8192.0M)->90.6M(8192.0M)] > > [Times: user=0.30 sys=0.00, real=0.03 secs] > > 2017-01-25T13:06:37.391+0530: 64241.843: Total time for which > application threads were stopped: 0.0267672 seconds, Stopping threads > took: 0.0000448 seconds > > 2017-01-25T13:06:37.391+0530: 64241.843: Application time: 0.0000947 > seconds > > 2017-01-25T13:06:37.391+0530: 64241.844: Total time for which > application threads were stopped: 0.0002803 seconds, Stopping threads > took: 0.0000729 seconds > > 2017-01-25T13:06:37.391+0530: 64241.844: Application time: 0.0000418 > seconds > > 2017-01-25T13:06:37.391+0530: 64241.844: Total time for which > application threads were stopped: 0.0001173 seconds, Stopping threads > took: 0.0000205 seconds > > 2017-01-25T13:06:37.391+0530: 64241.844: Application time: 0.0000254 > seconds > > 2017-01-25T13:06:37.391+0530: 64241.844: Total time for which > application threads were stopped: 0.0001434 seconds, Stopping threads > took: 0.0000479 seconds > > 2017-01-25T13:06:37.391+0530: 64241.844: Application time: 0.0000188 > seconds > > 2017-01-25T13:06:37.392+0530: 64241.844: Total time for which > application threads were stopped: 0.0001591 seconds, Stopping threads > took: 0.0000433 seconds > > 2017-01-25T13:06:37.392+0530: 64241.844: Application time: 0.0000286 > seconds > > 2017-01-25T13:06:37.392+0530: 64241.844: Total time for which > application threads were stopped: 0.0001666 seconds, Stopping threads > took: 0.0000486 seconds > > 2017-01-25T13:06:37.392+0530: 64241.844: Application time: 0.0000257 > seconds > > 2017-01-25T13:06:37.392+0530: 64241.845: Total time for which > application threads were stopped: 0.0001308 seconds, Stopping threads > took: 0.0000415 seconds > > 2017-01-25T13:06:37.392+0530: 64241.845: Application time: 0.0000210 > seconds > > 2017-01-25T13:06:37.392+0530: 64241.845: Total time for which > application threads were stopped: 0.0001181 seconds, Stopping threads > took: 0.0000279 seconds > > 2017-01-25T13:06:37.392+0530: 64241.845: Application time: 0.0000209 > seconds > > 2017-01-25T13:06:37.392+0530: 64241.845: Total time for which > application threads were stopped: 0.0001475 seconds, Stopping threads > took: 0.0000510 seconds > > 2017-01-25T13:06:40.571+0530: 64245.024: Application time: 3.1790511 > seconds > > 2017-01-25T13:06:40.572+0530: 64245.024: Total time for which > application threads were stopped: 0.0002598 seconds, Stopping threads > took: 0.0000431 seconds > > 2017-01-25T13:06:45.572+0530: 64250.024: Application time: 5.0001376 > seconds > > 2017-01-25T13:06:45.572+0530: 64250.025: Total time for which > application threads were stopped: 0.0002687 seconds, Stopping threads > took: 0.0000431 seconds > > 2017-01-25T13:06:50.572+0530: 64255.025: Application time: 5.0001946 > seconds > > 2017-01-25T13:06:50.573+0530: 64255.025: Total time for which > application threads were stopped: 0.0002942 seconds, Stopping threads > took: 0.0000634 seconds > > 2017-01-25T13:06:55.573+0530: 64260.025: Application time: 5.0001895 > seconds > > 2017-01-25T13:06:55.573+0530: 64260.026: Total time for which > application threads were stopped: 0.0002684 seconds, Stopping threads > took: 0.0000407 seconds > > 2017-01-25T13:07:00.573+0530: 64265.026: Application time: 5.0002077 > seconds > > 2017-01-25T13:07:00.573+0530: 64265.026: Total time for which > application threads were stopped: 0.0002506 seconds, Stopping threads > took: 0.0000418 seconds > > 2017-01-25T13:07:05.574+0530: 64270.026: Application time: 5.0002046 > seconds > > 2017-01-25T13:07:05.574+0530: 64270.026: Total time for which > application threads were stopped: 0.0002517 seconds, Stopping threads > took: 0.0000421 seconds > > 2017-01-25T13:07:10.574+0530: 64275.027: Application time: 5.0001865 > seconds > > 2017-01-25T13:07:10.574+0530: 64275.027: Total time for which > application threads were stopped: 0.0002522 seconds, Stopping threads > took: 0.0000474 seconds > > 2017-01-25T13:07:15.575+0530: 64280.027: Application time: 5.0002286 > seconds > > 2017-01-25T13:07:15.575+0530: 64280.027: Total time for which > application threads were stopped: 0.0002538 seconds, Stopping threads > took: 0.0000479 seconds > > 2017-01-25T13:07:20.575+0530: 64285.028: Application time: 5.0001932 > seconds > > 2017-01-25T13:07:20.575+0530: 64285.028: Total time for which > application threads were stopped: 0.0002462 seconds, Stopping threads > took: 0.0000431 seconds > > 2017-01-25T13:07:25.575+0530: 64290.028: Application time: 5.0001952 > seconds > > 2017-01-25T13:07:25.576+0530: 64290.028: Total time for which > application threads were stopped: 0.0002328 seconds, Stopping threads > took: 0.0000391 seconds > > 2017-01-25T13:07:30.576+0530: 64295.029: Application time: 5.0002049 > seconds > > 2017-01-25T13:07:30.576+0530: 64295.029: Total time for which > application threads were stopped: 0.0002359 seconds, Stopping threads > took: 0.0000408 seconds > > 2017-01-25T13:07:35.576+0530: 64300.029: Application time: 5.0002040 > seconds > > 2017-01-25T13:07:35.577+0530: 64300.029: Total time for which > application threads were stopped: 0.0002753 seconds, Stopping threads > took: 0.0000473 seconds > > 2017-01-25T13:07:40.577+0530: 64305.029: Application time: 5.0002046 > seconds > > 2017-01-25T13:07:40.577+0530: 64305.030: Total time for which > application threads were stopped: 0.0002560 seconds, Stopping threads > took: 0.0000422 seconds > > 2017-01-25T13:07:45.577+0530: 64310.030: Application time: 5.0002068 > seconds > > 2017-01-25T13:07:45.578+0530: 64310.030: Total time for which > application threads were stopped: 0.0002618 seconds, Stopping threads > took: 0.0000518 seconds > > 2017-01-25T13:07:50.578+0530: 64315.030: Application time: 5.0001895 > seconds > > 2017-01-25T13:07:50.578+0530: 64315.031: Total time for which > application threads were stopped: 0.0003045 seconds, Stopping threads > took: 0.0000512 seconds > > 2017-01-25T13:07:55.578+0530: 64320.031: Application time: 5.0002394 > seconds > > 2017-01-25T13:07:55.579+0530: 64320.031: Total time for which > application threads were stopped: 0.0002506 seconds, Stopping threads > took: 0.0000395 seconds > > 2017-01-25T13:08:00.579+0530: 64325.031: Application time: 5.0002288 > seconds > > 2017-01-25T13:08:00.579+0530: 64325.032: Total time for which > application threads were stopped: 0.0002531 seconds, Stopping threads > took: 0.0000406 seconds > > 2017-01-25T13:08:05.579+0530: 64330.032: Application time: 5.0001996 > seconds > > 2017-01-25T13:08:05.580+0530: 64330.032: Total time for which > application threads were stopped: 0.0002486 seconds, Stopping threads > took: 0.0000420 seconds > > 2017-01-25T13:08:10.580+0530: 64335.032: Application time: 5.0001750 > seconds > > 2017-01-25T13:08:10.580+0530: 64335.033: Total time for which > application threads were stopped: 0.0002557 seconds, Stopping threads > took: 0.0000488 seconds > > 2017-01-25T13:08:15.580+0530: 64340.033: Application time: 5.0001817 > seconds > > 2017-01-25T13:08:15.580+0530: 64340.033: Total time for which > application threads were stopped: 0.0002597 seconds, Stopping threads > took: 0.0000493 seconds > > 2017-01-25T13:08:20.581+0530: 64345.033: Application time: 5.0001852 > seconds > > 2017-01-25T13:08:20.581+0530: 64345.033: Total time for which > application threads were stopped: 0.0002825 seconds, Stopping threads > took: 0.0000627 seconds > > 2017-01-25T13:08:25.581+0530: 64350.034: Application time: 5.0002295 > seconds > > 2017-01-25T13:08:25.581+0530: 64350.034: Total time for which > application threads were stopped: 0.0002499 seconds, Stopping threads > took: 0.0000403 seconds > > 2017-01-25T13:08:30.582+0530: 64355.034: Application time: 5.0002247 > seconds > > 2017-01-25T13:08:30.582+0530: 64355.034: Total time for which > application threads were stopped: 0.0002803 seconds, Stopping threads > took: 0.0000593 seconds > > 2017-01-25T13:08:35.582+0530: 64360.035: Application time: 5.0001862 > seconds > > 2017-01-25T13:08:35.582+0530: 64360.035: Total time for which > application threads were stopped: 0.0002972 seconds, Stopping threads > took: 0.0000699 seconds > > 2017-01-25T13:08:40.133+0530: 64364.586: Application time: 4.5507335 > seconds > > 2017-01-25T13:08:40.134+0530: 64364.586: Total time for which > application threads were stopped: 0.0006192 seconds, Stopping threads > took: 0.0000411 seconds > > 2017-01-25T13:08:40.583+0530: 64365.035: Application time: 0.4488614 > seconds > > 2017-01-25T13:08:40.583+0530: 64365.035: Total time for which > application threads were stopped: 0.0002583 seconds, Stopping threads > took: 0.0000548 seconds > > 2017-01-25T13:08:45.583+0530: 64370.036: Application time: 5.0002011 > seconds > > 2017-01-25T13:08:45.583+0530: 64370.036: Total time for which > application threads were stopped: 0.0002594 seconds, Stopping threads > took: 0.0000424 seconds > > 2017-01-25T13:08:50.583+0530: 64375.036: Application time: 5.0002066 > seconds > > 2017-01-25T13:08:50.584+0530: 64375.036: Total time for which > application threads were stopped: 0.0002680 seconds, Stopping threads > took: 0.0000414 seconds > > 2017-01-25T13:08:55.584+0530: 64380.037: Application time: 5.0002392 > seconds > > 2017-01-25T13:08:55.584+0530: 64380.037: Total time for which > application threads were stopped: 0.0002646 seconds, Stopping threads > took: 0.0000370 seconds > > 2017-01-25T13:09:00.584+0530: 64385.037: Application time: 5.0002253 > seconds > > 2017-01-25T13:09:00.585+0530: 64385.037: Total time for which > application threads were stopped: 0.0002727 seconds, Stopping threads > took: 0.0000393 seconds > > 2017-01-25T13:09:05.585+0530: 64390.038: Application time: 5.0002276 > seconds > > 2017-01-25T13:09:05.585+0530: 64390.038: Total time for which > application threads were stopped: 0.0002608 seconds, Stopping threads > took: 0.0000403 seconds > > 2017-01-25T13:09:10.585+0530: 64395.038: Application time: 5.0002179 > seconds > > 2017-01-25T13:09:10.586+0530: 64395.038: Total time for which > application threads were stopped: 0.0002685 seconds, Stopping threads > took: 0.0000433 seconds > > 2017-01-25T13:09:15.586+0530: 64400.039: Application time: 5.0001999 > seconds > > 2017-01-25T13:09:15.586+0530: 64400.039: Total time for which > application threads were stopped: 0.0002479 seconds, Stopping threads > took: 0.0000470 seconds > > 2017-01-25T13:09:20.586+0530: 64405.039: Application time: 5.0000955 > seconds > > 2017-01-25T13:09:20.587+0530: 64405.039: Total time for which > application threads were stopped: 0.0003038 seconds, Stopping threads > took: 0.0000829 seconds > > 2017-01-25T13:09:25.587+0530: 64410.039: Application time: 5.0002022 > seconds > > 2017-01-25T13:09:25.587+0530: 64410.040: Total time for which > application threads were stopped: 0.0002422 seconds, Stopping threads > took: 0.0000428 seconds > > 2017-01-25T13:09:30.587+0530: 64415.040: Application time: 5.0002011 > seconds > > 2017-01-25T13:09:30.587+0530: 64415.040: Total time for which > application threads were stopped: 0.0002440 seconds, Stopping threads > took: 0.0000404 seconds > > 2017-01-25T13:09:35.588+0530: 64420.040: Application time: 5.0001995 > seconds > > 2017-01-25T13:09:35.588+0530: 64420.041: Total time for which > application threads were stopped: 0.0002617 seconds, Stopping threads > took: 0.0000409 seconds > > 2017-01-25T13:09:40.588+0530: 64425.041: Application time: 5.0001848 > seconds > > 2017-01-25T13:09:40.588+0530: 64425.041: Total time for which > application threads were stopped: 0.0002620 seconds, Stopping threads > took: 0.0000436 seconds > > 2017-01-25T13:09:42.834+0530: 64427.286: Application time: 2.2451421 > seconds > > 2017-01-25T13:09:42.834+0530: 64427.286: [GC pause (G1 Evacuation > Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] > desired_size: 166KB slow allocs: 0 refill waste: 2664B alloc: 0.00199 > 8330KB refills: 77 waste 1.4% gc: 168776B slow: 14496B fast: 0B > > TLAB: gc thread: 0x00007f15f8114800 [id: 199] desired_size: 1999KB > slow allocs: 1 refill waste: 31984B alloc: 0.02388 99965KB > refills: 127 waste 0.2% gc: 484968B slow: 15936B fast: 0B > > TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 5KB slow > allocs: 0 refill waste: 88B alloc: 0.00007 286KB refills: 71 > waste 1.8% gc: 3464B slow: 4176B fast: 0B > > TLAB: gc thread: 0x00007f15906a8800 [id: 144] desired_size: 2KB slow > allocs: 2 refill waste: 32B alloc: 0.00000 17KB refills: 10 > waste 11.2% gc: 2064B slow: 248B fast: 0B > > TLAB: gc thread: 0x00007f15906a7000 [id: 143] desired_size: 2KB slow > allocs: 3 refill waste: 32B alloc: 0.00000 17KB refills: 10 > waste 11.3% gc: 2064B slow: 272B fast: 0B > > TLAB: gc thread: 0x00007f15906a5000 [id: 142] desired_size: 2KB slow > allocs: 2 refill waste: 32B alloc: 0.00000 17KB refills: 10 > waste 10.3% gc: 1888B slow: 248B fast: 0B > > TLAB: gc thread: 0x00007f15906a3800 [id: 141] desired_size: 2KB slow > allocs: 42 refill waste: 32B alloc: 0.00001 48KB refills: 132 > waste 2.0% gc: 1104B slow: 4328B fast: 0B > > TLAB: gc thread: 0x00007f15906a1800 [id: 140] desired_size: 750KB slow > allocs: 1 refill waste: 12008B alloc: 0.00897 37540KB refills: 88 > waste 0.9% gc: 625568B slow: 9488B fast: 0B > > TLAB: gc thread: 0x00007f15906a0000 [id: 139] desired_size: 2KB slow > allocs: 3 refill waste: 32B alloc: 0.00000 17KB refills: 10 > waste 11.3% gc: 2064B slow: 272B fast: 0B > > TLAB: gc thread: 0x00007f159069e800 [id: 138] desired_size: 2KB slow > allocs: 2 refill waste: 32B alloc: 0.00000 17KB refills: 10 > waste 11.2% gc: 2064B slow: 248B fast: 0B > > TLAB: gc thread: 0x00007f159069c800 [id: 137] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 17KB refills: 10 > waste 10.3% gc: 1888B slow: 240B fast: 0B > > TLAB: gc thread: 0x00007f159069b000 [id: 136] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00003 105KB refills: 9 > waste 3.8% gc: 488B slow: 240B fast: 0B > > TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 731KB slow > allocs: 0 refill waste: 11696B alloc: 0.00873 36559KB refills: 88 > waste 1.0% gc: 658072B slow: 5032B fast: 0B > > TLAB: gc thread: 0x00007f1590697800 [id: 134] desired_size: 2KB slow > allocs: 2 refill waste: 32B alloc: 0.00000 17KB refills: 10 > waste 11.2% gc: 2064B slow: 248B fast: 0B > > TLAB: gc thread: 0x00007f1590695800 [id: 133] desired_size: 2KB slow > allocs: 3 refill waste: 32B alloc: 0.00000 17KB refills: 10 > waste 11.3% gc: 2064B slow: 272B fast: 0B > > TLAB: gc thread: 0x00007f1590693800 [id: 132] desired_size: 2KB slow > allocs: 2 refill waste: 32B alloc: 0.00000 17KB refills: 10 > waste 10.7% gc: 1952B slow: 264B fast: 0B > > TLAB: gc thread: 0x00007f1590692000 [id: 131] desired_size: 2KB slow > allocs: 63 refill waste: 32B alloc: 0.00001 61KB refills: 258 > waste 1.8% gc: 1832B slow: 7936B fast: 0B > > TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 776KB slow > allocs: 0 refill waste: 12416B alloc: 0.00927 38822KB refills: 84 > waste 0.2% gc: 97352B slow: 8736B fast: 0B > > TLAB: gc thread: 0x00007f159068e800 [id: 129] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 17KB refills: 10 > waste 10.1% gc: 1832B slow: 256B fast: 0B > > TLAB: gc thread: 0x00007f159068d000 [id: 128] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00000 17KB refills: 10 > waste 10.6% gc: 1952B slow: 240B fast: 0B > > TLAB: gc thread: 0x00007f159068b000 [id: 127] desired_size: 2KB slow > allocs: 2 refill waste: 32B alloc: 0.00000 17KB refills: 10 > waste 11.2% gc: 2064B slow: 248B fast: 0B > > TLAB: gc thread: 0x00007f1590689800 [id: 126] desired_size: 2KB slow > allocs: 2 refill waste: 32B alloc: 0.00002 98KB refills: 10 > waste 11.2% gc: 2064B slow: 248B fast: 0B > > TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 775KB slow > allocs: 0 refill waste: 12400B alloc: 0.00926 38755KB refills: 72 > waste 0.1% gc: 36224B slow: 7680B fast: 0B > > TLAB: gc thread: 0x00007f1590674800 [id: 124] desired_size: 2KB slow > allocs: 2 refill waste: 32B alloc: 0.00000 17KB refills: 10 > waste 10.5% gc: 1888B slow: 272B fast: 0B > > TLAB: gc thread: 0x00007f1590673000 [id: 123] desired_size: 2KB slow > allocs: 2 refill waste: 32B alloc: 0.00000 17KB refills: 10 > waste 11.2% gc: 2064B slow: 248B fast: 0B > > TLAB: gc thread: 0x00007f1590671800 [id: 122] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00000 17KB refills: 10 > waste 10.4% gc: 1888B slow: 264B fast: 0B > > TLAB: gc thread: 0x00007f159067d800 [id: 121] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00002 76KB refills: 10 > waste 10.6% gc: 1952B slow: 240B fast: 0B > > TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 750KB slow > allocs: 0 refill waste: 12008B alloc: 0.00897 37549KB refills: 79 > waste 0.7% gc: 403168B slow: 11312B fast: 0B > > TLAB: gc thread: 0x00007f1590637000 [id: 117] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00000 16KB refills: 9 > waste 3.3% gc: 376B slow: 232B fast: 0B > > TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 97KB slow > allocs: 0 refill waste: 1560B alloc: 0.00117 4888KB refills: 82 > waste 0.3% gc: 18624B slow: 2992B fast: 0B > > TLAB: gc thread: 0x00007f159062b000 [id: 115] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 3KB refills: 2 > waste 12.0% gc: 480B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f159005b000 [id: 114] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 1KB refills: 1 > waste 51.6% gc: 1064B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f158c010800 [id: 113] desired_size: 31KB slow > allocs: 44 refill waste: 504B alloc: 0.00038 1575KB refills: 34 > waste 1.9% gc: 18984B slow: 2256B fast: 0B > > TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.38170 1597857KB > refills: 774 waste 0.1% gc: 1492120B slow: 81320B fast: 0B > > TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.06731 281777KB > refills: 249 waste 0.3% gc: 1799984B slow: 18136B fast: 0B > > TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 2047KB > slow allocs: 35 refill waste: 32760B alloc: 0.11055 462756KB > refills: 376 waste 0.3% gc: 1727576B slow: 467192B fast: 0B > > TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 388KB slow > allocs: 0 refill waste: 6216B alloc: 0.00464 19425KB refills: 81 > waste 1.0% gc: 324512B slow: 7440B fast: 0B > > TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 451KB slow > allocs: 0 refill waste: 7216B alloc: 0.00539 22558KB refills: 80 > waste 0.9% gc: 320392B slow: 6880B fast: 0B > > TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 2047KB > slow allocs: 131 refill waste: 32760B alloc: 0.35391 1481518KB > refills: 1175 waste 0.1% gc: 0B slow: 2219392B fast: 0B > > TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 7KB refills: 5 > waste 14.6% gc: 1344B slow: 160B fast: 0B > > TLAB: gc thread: 0x00007f170d35a000 [id: 85] desired_size: 122KB slow > allocs: 0 refill waste: 1960B alloc: 0.00147 6144KB refills: 63 > waste 0.7% gc: 51592B slow: 5096B fast: 0B > > TLAB: gc thread: 0x00007f170d2cf000 [id: 84] desired_size: 2KB slow > allocs: 3 refill waste: 32B alloc: 0.00000 11KB refills: 6 > waste 7.9% gc: 840B slow: 144B fast: 0B > > TLAB: gc thread: 0x00007f170cda3800 [id: 63] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 0KB refills: 1 > waste 91.5% gc: 1888B slow: 0B fast: 0B > > TLAB totals: thrds: 43 refills: 4193 max: 1175 slow allocs: 351 max > 131 waste: 0.2% gc: 8274608B max: 1799984B slow: 2904944B max: > 2219392B fast: 0B max: 0B > > > Desired survivor size 268435456 bytes, new threshold 15 (max 15) > > - age 1: 1828584 bytes, 1828584 total > > - age 2: 32728 bytes, 1861312 total > > - age 3: 3152 bytes, 1864464 total > > - age 4: 10128 bytes, 1874592 total > > - age 6: 176 bytes, 1874768 total > > - age 7: 144 bytes, 1874912 total > > - age 8: 144 bytes, 1875056 total > > - age 9: 5744 bytes, 1880800 total > > - age 10: 72 bytes, 1880872 total > > - age 11: 832 bytes, 1881704 total > > - age 12: 48 bytes, 1881752 total > > - age 13: 48 bytes, 1881800 total > > - age 15: 80 bytes, 1881880 total > > 64427.287: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 3542, predicted base time: 17.08 ms, remaining time: > 132.92 ms, target pause time: 150.00 ms] > > 64427.287: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 1022 regions, survivors: 2 regions, predicted young region > time: 2.13 ms] > > 64427.287: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 1022 regions, survivors: 2 regions, old: 0 regions, predicted > pause time: 19.21 ms, target pause time: 150.00 ms] > > 2017-01-25T13:09:42.839+0530: 64427.292: [SoftReference, 0 refs, > 0.0023408 secs]2017-01-25T13:09:42.841+0530: 64427.294: > [WeakReference, 1 refs, 0.0008582 secs]2017-01-25T13:09:42.842+0530: > 64427.295: [FinalReference, 67 refs, 0.0009552 > secs]2017-01-25T13:09:42.843+0530: 64427.296: [PhantomReference, 0 > refs, 0 refs, 0.0013949 secs]2017-01-25T13:09:42.845+0530: 64427.297: > [JNI Weak Reference, 0.0000180 secs], 0.0139572 secs] > > [Parallel Time: 1.9 ms, GC Workers: 20] > > [GC Worker Start (ms): Min: 64427286.7, Avg: 64427286.9, Max: > 64427288.1, Diff: 1.3] > > [Ext Root Scanning (ms): Min: 0.0, Avg: 0.7, Max: 1.5, Diff: > 1.5, Sum: 14.9] > > [Update RS (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.3, Sum: 1.9] > > [Processed Buffers: Min: 0, Avg: 2.6, Max: 8, Diff: 8, Sum: 53] > > [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.3] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > > [Object Copy (ms): Min: 0.0, Avg: 0.2, Max: 0.3, Diff: 0.3, Sum: > 3.7] > > [Termination (ms): Min: 0.0, Avg: 0.3, Max: 0.4, Diff: 0.4, Sum: > 6.1] > > [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, > Sum: 20] > > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, > Sum: 1.2] > > [GC Worker Total (ms): Min: 0.3, Avg: 1.5, Max: 1.7, Diff: 1.4, > Sum: 30.1] > > [GC Worker End (ms): Min: 64427288.4, Avg: 64427288.4, Max: > 64427288.5, Diff: 0.1] > > [Code Root Fixup: 0.0 ms] > > [Code Root Purge: 0.0 ms] > > [Clear CT: 0.8 ms] > > [Other: 11.3 ms] > > [Choose CSet: 0.0 ms] > > [Ref Proc: 8.6 ms] > > [Ref Enq: 0.3 ms] > > [Redirty Cards: 0.2 ms] > > [Humongous Register: 0.1 ms] > > [Humongous Reclaim: 0.0 ms] > > [Free CSet: 1.6 ms] > > [Eden: 4088.0M(4088.0M)->0.0B(4092.0M) Survivors: 8192.0K->4096.0K > Heap: 4178.6M(8192.0M)->87.1M(8192.0M)] > > [Times: user=0.10 sys=0.01, real=0.01 secs] > > > *_The GC with issue:_* > > 2017-01-25T14:10:05.908+0530: 68050.361: [GC pause (G1 Evacuation > Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] > desired_size: 177KB slow allocs: 0 refill waste: 2840B alloc: 0.00212 > 8881KB refills: 70 waste 0.8% gc: 89264B slow: 10480B fast: 0B > > TLAB: gc thread: 0x00007f15f8114800 [id: 199] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.02722 113928KB > refills: 83 waste 0.3% gc: 587872B slow: 3352B fast: 0B > > TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 6KB slow > allocs: 0 refill waste: 96B alloc: 0.00007 304KB refills: 66 > waste 0.9% gc: 1376B slow: 2384B fast: 0B > > TLAB: gc thread: 0x00007f15906a8800 [id: 144] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00000 18KB refills: 11 > waste 5.0% gc: 880B slow: 264B fast: 0B > > TLAB: gc thread: 0x00007f15906a7000 [id: 143] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00000 18KB refills: 11 > waste 5.0% gc: 880B slow: 264B fast: 0B > > TLAB: gc thread: 0x00007f15906a5000 [id: 142] desired_size: 2KB slow > allocs: 3 refill waste: 32B alloc: 0.00000 18KB refills: 11 > waste 5.5% gc: 928B slow: 320B fast: 0B > > TLAB: gc thread: 0x00007f15906a3800 [id: 141] desired_size: 3KB slow > allocs: 1 refill waste: 48B alloc: 0.00004 172KB refills: 17 > waste 6.6% gc: 3408B slow: 552B fast: 0B > > TLAB: gc thread: 0x00007f15906a1800 [id: 140] desired_size: 865KB slow > allocs: 0 refill waste: 13840B alloc: 0.01033 43261KB refills: 72 > waste 0.8% gc: 523960B slow: 3008B fast: 0B > > TLAB: gc thread: 0x00007f15906a0000 [id: 139] desired_size: 2KB slow > allocs: 2 refill waste: 32B alloc: 0.00000 18KB refills: 11 > waste 5.7% gc: 1008B slow: 296B fast: 0B > > TLAB: gc thread: 0x00007f159069e800 [id: 138] desired_size: 2KB slow > allocs: 2 refill waste: 32B alloc: 0.00000 18KB refills: 11 > waste 5.7% gc: 1008B slow: 296B fast: 0B > > TLAB: gc thread: 0x00007f159069c800 [id: 137] desired_size: 2KB slow > allocs: 3 refill waste: 32B alloc: 0.00000 18KB refills: 11 > waste 5.8% gc: 1008B slow: 320B fast: 0B > > TLAB: gc thread: 0x00007f159069b000 [id: 136] desired_size: 2KB slow > allocs: 14 refill waste: 32B alloc: 0.00001 48KB refills: 110 > waste 2.0% gc: 1200B slow: 3424B fast: 0B > > TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 751KB slow > allocs: 0 refill waste: 12024B alloc: 0.00898 37577KB refills: 75 > waste 0.2% gc: 134904B slow: 6808B fast: 0B > > TLAB: gc thread: 0x00007f1590697800 [id: 134] desired_size: 2KB slow > allocs: 2 refill waste: 32B alloc: 0.00000 18KB refills: 11 > waste 5.7% gc: 1008B slow: 296B fast: 0B > > TLAB: gc thread: 0x00007f1590695800 [id: 133] desired_size: 2KB slow > allocs: 2 refill waste: 32B alloc: 0.00000 18KB refills: 11 > waste 5.1% gc: 880B slow: 288B fast: 0B > > TLAB: gc thread: 0x00007f1590693800 [id: 132] desired_size: 2KB slow > allocs: 2 refill waste: 32B alloc: 0.00000 18KB refills: 11 > waste 4.7% gc: 776B slow: 296B fast: 0B > > TLAB: gc thread: 0x00007f1590692000 [id: 131] desired_size: 2KB slow > allocs: 3 refill waste: 32B alloc: 0.00002 89KB refills: 14 > waste 5.9% gc: 1264B slow: 440B fast: 0B > > TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 873KB slow > allocs: 0 refill waste: 13976B alloc: 0.01044 43692KB refills: 68 > waste 0.3% gc: 192240B slow: 11640B fast: 0B > > TLAB: gc thread: 0x00007f159068e800 [id: 129] desired_size: 2KB slow > allocs: 3 refill waste: 32B alloc: 0.00000 18KB refills: 11 > waste 5.8% gc: 1008B slow: 320B fast: 0B > > TLAB: gc thread: 0x00007f159068d000 [id: 128] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 18KB refills: 11 > waste 4.6% gc: 776B slow: 264B fast: 0B > > TLAB: gc thread: 0x00007f159068b000 [id: 127] desired_size: 2KB slow > allocs: 3 refill waste: 32B alloc: 0.00000 18KB refills: 10 > waste 11.3% gc: 2064B slow: 272B fast: 0B > > TLAB: gc thread: 0x00007f1590689800 [id: 126] desired_size: 2KB slow > allocs: 5 refill waste: 32B alloc: 0.00003 120KB refills: 24 > waste 5.2% gc: 2272B slow: 800B fast: 0B > > TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 890KB slow > allocs: 0 refill waste: 14248B alloc: 0.01064 44534KB refills: 59 > waste 0.8% gc: 409872B slow: 11088B fast: 0B > > TLAB: gc thread: 0x00007f1590674800 [id: 124] desired_size: 2KB slow > allocs: 2 refill waste: 32B alloc: 0.00000 18KB refills: 11 > waste 5.7% gc: 1008B slow: 296B fast: 0B > > TLAB: gc thread: 0x00007f1590673000 [id: 123] desired_size: 2KB slow > allocs: 2 refill waste: 32B alloc: 0.00000 18KB refills: 11 > waste 5.4% gc: 928B slow: 296B fast: 0B > > TLAB: gc thread: 0x00007f1590671800 [id: 122] desired_size: 2KB slow > allocs: 2 refill waste: 32B alloc: 0.00000 18KB refills: 11 > waste 5.7% gc: 1008B slow: 296B fast: 0B > > TLAB: gc thread: 0x00007f159067d800 [id: 121] desired_size: 2KB slow > allocs: 19 refill waste: 32B alloc: 0.00002 77KB refills: 116 > waste 2.0% gc: 1392B slow: 3312B fast: 0B > > TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 807KB slow > allocs: 0 refill waste: 12920B alloc: 0.00965 40392KB refills: 79 > waste 0.8% gc: 509408B slow: 10496B fast: 0B > > TLAB: gc thread: 0x00007f1590637000 [id: 117] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00000 17KB refills: 11 > waste 7.3% gc: 1360B slow: 304B fast: 0B > > TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 109KB slow > allocs: 0 refill waste: 1752B alloc: 0.00131 5491KB refills: 82 > waste 0.9% gc: 78864B slow: 2976B fast: 0B > > TLAB: gc thread: 0x00007f159062b000 [id: 115] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 3KB refills: 2 > waste 12.0% gc: 480B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f159005b000 [id: 114] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 1KB refills: 1 > waste 47.3% gc: 976B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f158c010800 [id: 113] desired_size: 35KB slow > allocs: 42 refill waste: 560B alloc: 0.00042 1764KB refills: 31 > waste 2.2% gc: 23272B slow: 1416B fast: 0B > > TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.30730 1286404KB > refills: 879 waste 0.0% gc: 431048B slow: 78920B fast: 0B > > TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.07518 314707KB > refills: 253 waste 0.2% gc: 1172880B slow: 17344B fast: 0B > > TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 2047KB > slow allocs: 20 refill waste: 32760B alloc: 0.12527 524398KB > refills: 370 waste 0.2% gc: 943176B slow: 444384B fast: 0B > > TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 433KB slow > allocs: 0 refill waste: 6936B alloc: 0.00518 21700KB refills: 78 > waste 0.5% gc: 177608B slow: 7000B fast: 0B > > TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 510KB slow > allocs: 0 refill waste: 8160B alloc: 0.00610 25523KB refills: 75 > waste 0.4% gc: 163696B slow: 6840B fast: 0B > > TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 2047KB > slow allocs: 66 refill waste: 32760B alloc: 0.39765 1664620KB > refills: 1099 waste 0.1% gc: 0B slow: 1894184B fast: 0B > > TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 8KB refills: 5 > waste 15.0% gc: 1376B slow: 176B fast: 0B > > TLAB: gc thread: 0x00007f170d35a000 [id: 85] desired_size: 123KB slow > allocs: 0 refill waste: 1976B alloc: 0.00148 6195KB refills: 80 > waste 0.2% gc: 19776B slow: 4952B fast: 0B > > TLAB: gc thread: 0x00007f170d2cf000 [id: 84] desired_size: 2KB slow > allocs: 3 refill waste: 32B alloc: 0.00000 12KB refills: 7 > waste 1.3% gc: 24B slow: 160B fast: 0B > > TLAB: gc thread: 0x00007f170cda3800 [id: 63] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 0KB refills: 1 > waste 91.5% gc: 1888B slow: 0B fast: 0B > > TLAB totals: thrds: 43 refills: 3991 max: 1099 slow allocs: 204 max > 66 waste: 0.1% gc: 5490024B max: 1172880B slow: 2530840B max: > 1894184B fast: 0B max: 0B > > > Desired survivor size 268435456 bytes, new threshold 15 (max 15) > > - age 1: 1736992 bytes, 1736992 total > > - age 2: 3936 bytes, 1740928 total > > - age 3: 416 bytes, 1741344 total > > - age 4: 168 bytes, 1741512 total > > - age 5: 200 bytes, 1741712 total > > - age 8: 416 bytes, 1742128 total > > - age 11: 1816 bytes, 1743944 total > > - age 12: 24 bytes, 1743968 total > > - age 14: 24 bytes, 1743992 total > > 68050.361: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 4790, predicted base time: 18.67 ms, remaining time: > 131.33 ms, target pause time: 150.00 ms] > > 68050.361: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 1022 regions, survivors: 2 regions, predicted young region > time: 2.16 ms] > > 68050.361: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 1022 regions, survivors: 2 regions, old: 0 regions, predicted > pause time: 20.83 ms, target pause time: 150.00 ms] > > 2017-01-25T14:10:05.914+0530: 68050.366: [SoftReference, 0 refs, > 0.0045583 secs]2017-01-25T14:10:05.918+0530: 68050.371: > [WeakReference, 1 refs, 0.0011573 secs]2017-01-25T14:10:05.920+0530: > 68050.372: [FinalReference, 59 refs, 0.0008955 > secs]2017-01-25T14:10:05.920+0530: 68050.373: [PhantomReference, 0 > refs, 0 refs, 0.0060485 secs]2017-01-25T14:10:05.927+0530: 68050.379: > [JNI Weak Reference, 0.0000168 secs], 0.0213658 secs] > > [Parallel Time: 2.9 ms, GC Workers: 20] > > [GC Worker Start (ms): Min: 68050361.3, Avg: 68050361.5, Max: > 68050361.6, Diff: 0.3] > > [Ext Root Scanning (ms): Min: 0.4, Avg: 0.5, Max: 1.0, Diff: > 0.6, Sum: 10.9] > > [Update RS (ms): Min: 0.0, Avg: 0.3, Max: 1.8, Diff: 1.8, Sum: 7.0] > > [Processed Buffers: Min: 0, Avg: 2.9, Max: 8, Diff: 8, Sum: 58] > > [Scan RS (ms): Min: 0.0, Avg: 0.2, Max: 0.3, Diff: 0.2, Sum: 3.9] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: > 0.1, Sum: 0.1] > > [Object Copy (ms): Min: 0.0, Avg: 0.7, Max: 0.8, Diff: 0.8, Sum: > 13.3] > > [Termination (ms): Min: 0.0, Avg: 0.6, Max: 1.0, Diff: 1.0, Sum: > 12.3] > > [Termination Attempts: Min: 1, Avg: 26.3, Max: 36, Diff: 35, > Sum: 526] > > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, > Sum: 1.4] > > [GC Worker Total (ms): Min: 2.3, Avg: 2.4, Max: 2.7, Diff: 0.4, > Sum: 48.8] > > [GC Worker End (ms): Min: 68050363.9, Avg: 68050363.9, Max: > 68050364.0, Diff: 0.2] > > [Code Root Fixup: 0.0 ms] > > [Code Root Purge: 0.0 ms] > > [Clear CT: 0.6 ms] > > [Other: 17.9 ms] > > [Choose CSet: 0.0 ms] > > [Ref Proc: 15.0 ms] > > [Ref Enq: 0.3 ms] > > [Redirty Cards: 0.2 ms] > > [Humongous Register: 0.0 ms] > > [Humongous Reclaim: 0.0 ms] > > [Free CSet: 1.9 ms] > > [Eden: 4088.0M(4088.0M)->0.0B(4088.0M) Survivors: 8192.0K->8192.0K > Heap: 4177.2M(8192.0M)->88.5M(8192.0M)] > > [Times: user=0.24 sys=0.00, real=0.03 secs] > > 2017-01-25T14:10:05.930+0530: 68050.382: Total time for which > application threads were stopped: 0.0217941 seconds, Stopping threads > took: 0.0000560 seconds > > 2017-01-25T14:10:05.930+0530: 68050.383: Application time: 0.0000388 > seconds > > 2017-01-25T14:10:05.930+0530: 68050.383: Total time for which > application threads were stopped: 0.0002600 seconds, Stopping threads > took: 0.0000840 seconds > > 2017-01-25T14:10:05.930+0530: 68050.383: Application time: 0.0000432 > seconds > > 2017-01-25T14:10:05.930+0530: 68050.383: Total time for which > application threads were stopped: 0.0001275 seconds, Stopping threads > took: 0.0000411 seconds > > 2017-01-25T14:10:05.930+0530: 68050.383: Application time: 0.0000422 > seconds > > 2017-01-25T14:10:05.931+0530: 68050.383: Total time for which > application threads were stopped: 0.0001408 seconds, Stopping threads > took: 0.0000537 seconds > > 2017-01-25T14:10:05.931+0530: 68050.383: Application time: 0.0000273 > seconds > > 2017-01-25T14:10:05.931+0530: 68050.383: Total time for which > application threads were stopped: 0.0001279 seconds, Stopping threads > took: 0.0000303 seconds > > 2017-01-25T14:10:05.931+0530: 68050.383: Application time: 0.0000279 > seconds > > 2017-01-25T14:10:05.931+0530: 68050.383: Total time for which > application threads were stopped: 0.0001316 seconds, Stopping threads > took: 0.0000435 seconds > > 2017-01-25T14:10:05.931+0530: 68050.383: Application time: 0.0000296 > seconds > > 2017-01-25T14:10:05.931+0530: 68050.384: Total time for which > application threads were stopped: 0.0001133 seconds, Stopping threads > took: 0.0000231 seconds > > 2017-01-25T14:10:05.931+0530: 68050.384: Application time: 0.0000331 > seconds > > 2017-01-25T14:10:05.931+0530: 68050.384: Total time for which > application threads were stopped: 0.0001241 seconds, Stopping threads > took: 0.0000469 seconds > > 2017-01-25T14:10:05.931+0530: 68050.384: Application time: 0.0000244 > seconds > > 2017-01-25T14:10:05.931+0530: 68050.384: Total time for which > application threads were stopped: 0.0001041 seconds, Stopping threads > took: 0.0000321 seconds > > 2017-01-25T14:10:05.931+0530: 68050.384: Application time: 0.0000256 > seconds > > 2017-01-25T14:10:05.931+0530: 68050.384: Total time for which > application threads were stopped: 0.0001178 seconds, Stopping threads > took: 0.0000398 seconds > > 2017-01-25T14:10:05.931+0530: 68050.384: Application time: 0.0000258 > seconds > > 2017-01-25T14:10:05.932+0530: 68050.384: Total time for which > application threads were stopped: 0.0001089 seconds, Stopping threads > took: 0.0000301 seconds > > 2017-01-25T14:10:05.932+0530: 68050.384: Application time: 0.0000252 > seconds > > 2017-01-25T14:10:05.932+0530: 68050.384: Total time for which > application threads were stopped: 0.0000885 seconds, Stopping threads > took: 0.0000214 seconds > > 2017-01-25T14:10:05.932+0530: 68050.384: Application time: 0.0000271 > seconds > > 2017-01-25T14:10:05.932+0530: 68050.384: Total time for which > application threads were stopped: 0.0001099 seconds, Stopping threads > took: 0.0000216 seconds > > 2017-01-25T14:10:05.932+0530: 68050.384: Application time: 0.0000299 > seconds > > 2017-01-25T14:10:05.932+0530: 68050.385: Total time for which > application threads were stopped: 0.0001135 seconds, Stopping threads > took: 0.0000388 seconds > > 2017-01-25T14:10:05.932+0530: 68050.385: Application time: 0.0000268 > seconds > > 2017-01-25T14:10:05.932+0530: 68050.385: Total time for which > application threads were stopped: 0.0001216 seconds, Stopping threads > took: 0.0000326 seconds > > 2017-01-25T14:10:05.932+0530: 68050.385: Application time: 0.0001340 > seconds > > 2017-01-25T14:10:05.932+0530: 68050.385: Total time for which > application threads were stopped: 0.0001468 seconds, Stopping threads > took: 0.0000357 seconds > > 2017-01-25T14:10:10.930+0530: 68055.383: Application time: 4.9978906 > seconds > > 2017-01-25T14:10:10.931+0530: 68055.383: Total time for which > application threads were stopped: 0.0002560 seconds, Stopping threads > took: 0.0000478 seconds > > 2017-01-25T14:10:15.931+0530: 68060.383: Application time: 5.0002051 > seconds > > 2017-01-25T14:10:15.931+0530: 68060.384: Total time for which > application threads were stopped: 0.0002450 seconds, Stopping threads > took: 0.0000483 seconds > > 2017-01-25T14:10:20.931+0530: 68065.384: Application time: 5.0002026 > seconds > > 2017-01-25T14:10:20.932+0530: 68065.384: Total time for which > application threads were stopped: 0.0003329 seconds, Stopping threads > took: 0.0000851 seconds > > 2017-01-25T14:10:23.115+0530: 68067.568: Application time: 2.1837577 > seconds > > 2017-01-25T14:10:23.116+0530: 68067.568: [GC pause (G1 Evacuation > Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] > desired_size: 182KB slow allocs: 0 refill waste: 2912B alloc: 0.00218 > 9116KB refills: 8 waste 11.9% gc: 174904B slow: 2912B fast: 0B > > TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 6KB slow > allocs: 0 refill waste: 96B alloc: 0.00007 311KB refills: 8 > waste 2.2% gc: 888B slow: 256B fast: 0B > > TLAB: gc thread: 0x00007f15906a1800 [id: 140] desired_size: 860KB slow > allocs: 0 refill waste: 13760B alloc: 0.01027 43002KB refills: 8 > waste 6.4% gc: 449272B slow: 1056B fast: 0B > > TLAB: gc thread: 0x00007f159069b000 [id: 136] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00003 115KB refills: 2 > waste 28.2% gc: 1288B slow: 32B fast: 0B > > TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 774KB slow > allocs: 0 refill waste: 12392B alloc: 0.00925 38728KB refills: 5 > waste 7.8% gc: 309640B slow: 104B fast: 0B > > TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 859KB slow > allocs: 0 refill waste: 13744B alloc: 0.01026 42960KB refills: 8 > waste 5.6% gc: 395744B slow: 176B fast: 0B > > TLAB: gc thread: 0x00007f1590689800 [id: 126] desired_size: 2KB slow > allocs: 12 refill waste: 32B alloc: 0.00002 97KB refills: 39 > waste 2.3% gc: 632B slow: 1256B fast: 0B > > TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 855KB slow > allocs: 0 refill waste: 13680B alloc: 0.01021 42750KB refills: 6 > waste 10.1% gc: 529496B slow: 360B fast: 0B > > TLAB: gc thread: 0x00007f159067d800 [id: 121] desired_size: 2KB slow > allocs: 0 refill waste: 40B alloc: 0.00003 138KB refills: 1 > waste 69.9% gc: 1968B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 848KB slow > allocs: 0 refill waste: 13576B alloc: 0.01014 42437KB refills: 5 > waste 2.3% gc: 97552B slow: 240B fast: 0B > > TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 113KB slow > allocs: 0 refill waste: 1808B alloc: 0.00135 5668KB refills: 8 > waste 2.9% gc: 26872B slow: 216B fast: 0B > > TLAB: gc thread: 0x00007f159062b000 [id: 115] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 3KB refills: 1 > waste 82.9% gc: 1712B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f159005b000 [id: 114] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 1KB refills: 1 > waste 98.4% gc: 2032B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.30038 1257411KB > refills: 2876 waste 0.0% gc: 0B slow: 131144B fast: 0B > > TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.07714 322937KB > refills: 24 waste 3.5% gc: 1750360B slow: 1128B fast: 0B > > TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 2047KB > slow allocs: 5 refill waste: 32760B alloc: 0.12559 525720KB > refills: 38 waste 1.8% gc: 1429064B slow: 11480B fast: 0B > > TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 444KB slow > allocs: 0 refill waste: 7104B alloc: 0.00530 22206KB refills: 5 > waste 7.5% gc: 170736B slow: 600B fast: 0B > > TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 527KB slow > allocs: 0 refill waste: 8432B alloc: 0.00630 26360KB refills: 8 > waste 2.8% gc: 119208B slow: 384B fast: 0B > > TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 2047KB > slow allocs: 2 refill waste: 32760B alloc: 0.40114 1679210KB > refills: 96 waste 0.6% gc: 1023536B slow: 163696B fast: 0B > > TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00000 9KB refills: 2 > waste 19.4% gc: 728B slow: 72B fast: 0B > > TLAB: gc thread: 0x00007f170d35a000 [id: 85] desired_size: 141KB slow > allocs: 0 refill waste: 2264B alloc: 0.00170 7097KB refills: 1 > waste 38.5% gc: 55928B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f170cda3800 [id: 63] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 0KB refills: 1 > waste 86.4% gc: 1784B slow: 0B fast: 0B > > TLAB totals: thrds: 22 refills: 3151 max: 2876 slow allocs: 20 max 12 > waste: 0.1% gc: 6543344B max: 1750360B slow: 315112B max: 163696B > fast: 0B max: 0B > > > Desired survivor size 268435456 bytes, new threshold 15 (max 15) > > - age 1: 1746112 bytes, 1746112 total > > - age 2: 41776 bytes, 1787888 total > > - age 3: 408 bytes, 1788296 total > > - age 4: 152 bytes, 1788448 total > > - age 5: 168 bytes, 1788616 total > > - age 6: 200 bytes, 1788816 total > > - age 9: 416 bytes, 1789232 total > > - age 12: 1816 bytes, 1791048 total > > - age 13: 24 bytes, 1791072 total > > - age 15: 24 bytes, 1791096 total > > 68067.568: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 4943, predicted base time: 18.96 ms, remaining time: > 131.04 ms, target pause time: 150.00 ms] > > 68067.568: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 1022 regions, survivors: 2 regions, predicted young region > time: 2.82 ms] > > 68067.568: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 1022 regions, survivors: 2 regions, old: 0 regions, predicted > pause time: 21.78 ms, target pause time: 150.00 ms] > > 2017-01-25T14:10:23.397+0530: 68067.850: [SoftReference, 0 refs, > 0.0012874 secs]2017-01-25T14:10:23.398+0530: 68067.851: > [WeakReference, 1 refs, 0.0008602 secs]2017-01-25T14:10:23.399+0530: > 68067.852: [FinalReference, 0 refs, 0.0010149 > secs]2017-01-25T14:10:23.400+0530: 68067.853: [PhantomReference, 0 > refs, 0 refs, 0.0015731 secs]2017-01-25T14:10:23.402+0530: 68067.855: > [JNI Weak Reference, 0.0000199 secs], 0.2898520 secs] > > [Parallel Time: 280.9 ms, GC Workers: 20] > > [GC Worker Start (ms): Min: 68067568.4, Avg: 68067568.5, Max: > 68067568.7, Diff: 0.3] > > [Ext Root Scanning (ms): Min: 0.3, Avg: 1.5, Max: 5.3, Diff: > 5.0, Sum: 30.9] > > [Update RS (ms): Min: 0.0, Avg: 0.8, Max: 13.2, Diff: 13.2, Sum: > 15.0] > > [Processed Buffers: Min: 0, Avg: 1.9, Max: 7, Diff: 7, Sum: 39] > > [Scan RS (ms): Min: 0.0, Avg: 0.2, Max: 0.3, Diff: 0.2, Sum: 3.3] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > > [Object Copy (ms): Min: 266.5, Avg: 277.1, Max: 279.5, Diff: > 13.0, Sum: 5542.9] > > [Termination (ms): Min: 0.0, Avg: 0.9, Max: 1.2, Diff: 1.2, Sum: > 17.0] > > [Termination Attempts: Min: 1, Avg: 2.5, Max: 10, Diff: 9, > Sum: 49] > > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, > Sum: 1.9] > > [GC Worker Total (ms): Min: 280.4, Avg: 280.6, Max: 280.7, Diff: > 0.4, Sum: 5611.1] > > [GC Worker End (ms): Min: 68067849.0, Avg: 68067849.1, Max: > 68067849.2, Diff: 0.2] > > [Code Root Fixup: 0.0 ms] > > [Code Root Purge: 0.0 ms] > > [Clear CT: 0.6 ms] > > [Other: 8.3 ms] > > [Choose CSet: 0.0 ms] > > [Ref Proc: 5.2 ms] > > [Ref Enq: 0.4 ms] > > [Redirty Cards: 0.3 ms] > > [Humongous Register: 0.1 ms] > > [Humongous Reclaim: 0.0 ms] > > [Free CSet: 1.9 ms] > > [Eden: 4088.0M(4088.0M)->0.0B(3584.0M) Survivors: 8192.0K->512.0M > Heap: 4176.5M(8192.0M)->599.1M(8192.0M)] > > [Times: user=1.50 sys=3.93, real=0.29 secs] > > 2017-01-25T14:10:23.406+0530: 68067.858: Total time for which > application threads were stopped: 0.2902731 seconds, Stopping threads > took: 0.0000596 seconds > > 2017-01-25T14:10:25.461+0530: 68069.914: Application time: 2.0554312 > seconds > > 2017-01-25T14:10:25.461+0530: 68069.914: [GC pause (G1 Evacuation > Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] > desired_size: 110KB slow allocs: 0 refill waste: 1760B alloc: 0.00150 > 5504KB refills: 2 waste 4.4% gc: 9864B slow: 96B fast: 0B > > TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 3KB slow > allocs: 0 refill waste: 56B alloc: 0.00005 188KB refills: 1 > waste 7.3% gc: 280B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f15906a3800 [id: 141] desired_size: 2KB slow > allocs: 4 refill waste: 32B alloc: 0.00003 116KB refills: 35 > waste 1.8% gc: 464B slow: 1064B fast: 0B > > TLAB: gc thread: 0x00007f15906a1800 [id: 140] desired_size: 513KB slow > allocs: 0 refill waste: 8208B alloc: 0.00699 25671KB refills: 2 > waste 26.5% gc: 279000B slow: 24B fast: 0B > > TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 459KB slow > allocs: 0 refill waste: 7344B alloc: 0.00626 22973KB refills: 2 > waste 34.2% gc: 321376B slow: 24B fast: 0B > > TLAB: gc thread: 0x00007f1590692000 [id: 131] desired_size: 2KB slow > allocs: 3 refill waste: 32B alloc: 0.00002 59KB refills: 12 > waste 6.7% gc: 1240B slow: 424B fast: 0B > > TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 517KB slow > allocs: 0 refill waste: 8272B alloc: 0.00705 25874KB refills: 1 > waste 16.2% gc: 86056B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 509KB slow > allocs: 0 refill waste: 8152B alloc: 0.00695 25493KB refills: 3 > waste 21.2% gc: 332112B slow: 80B fast: 0B > > TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 501KB slow > allocs: 0 refill waste: 8016B alloc: 0.00683 25060KB refills: 1 > waste 18.4% gc: 94472B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 67KB slow > allocs: 0 refill waste: 1080B alloc: 0.00092 3381KB refills: 1 > waste 0.9% gc: 624B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.52461 1925342KB > refills: 2664 waste 0.0% gc: 0B slow: 122088B fast: 0B > > TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.05268 193343KB > refills: 4 waste 24.6% gc: 2066344B slow: 272B fast: 0B > > TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.08553 313904KB > refills: 4 waste 2.4% gc: 204904B slow: 96B fast: 0B > > TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 264KB slow > allocs: 0 refill waste: 4224B alloc: 0.00360 13203KB refills: 3 > waste 32.1% gc: 260448B slow: 96B fast: 0B > > TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 315KB slow > allocs: 0 refill waste: 5048B alloc: 0.00430 15778KB refills: 1 > waste 2.8% gc: 9056B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.27249 1000030KB > refills: 9 waste 3.1% gc: 589904B slow: 944B fast: 0B > > TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 2KB slow > allocs: 3 refill waste: 64B alloc: 0.00000 6KB refills: 2 > waste 3.1% gc: 72B slow: 56B fast: 0B > > TLAB totals: thrds: 17 refills: 2747 max: 2664 slow allocs: 10 max 4 > waste: 0.1% gc: 4256216B max: 2066344B slow: 125264B max: 122088B > fast: 0B max: 0B > > > Desired survivor size 268435456 bytes, new threshold 1 (max 15) > > - age 1: 530412208 bytes, 530412208 total > > - age 2: 1173416 bytes, 531585624 total > > - age 3: 40248 bytes, 531625872 total > > - age 4: 408 bytes, 531626280 total > > - age 5: 152 bytes, 531626432 total > > - age 6: 168 bytes, 531626600 total > > - age 7: 200 bytes, 531626800 total > > - age 10: 416 bytes, 531627216 total > > - age 13: 1816 bytes, 531629032 total > > - age 14: 24 bytes, 531629056 total > > 68069.914: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 6052, predicted base time: 17.09 ms, remaining time: > 132.91 ms, target pause time: 150.00 ms] > > 68069.914: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 896 regions, survivors: 128 regions, predicted young > region time: 75.53 ms] > > 68069.914: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 896 regions, survivors: 128 regions, old: 0 regions, predicted > pause time: 92.62 ms, target pause time: 150.00 ms] > > 2017-01-25T14:10:25.952+0530: 68070.405: [SoftReference, 0 refs, > 0.0013460 secs]2017-01-25T14:10:25.953+0530: 68070.406: > [WeakReference, 1 refs, 0.0006541 secs]2017-01-25T14:10:25.954+0530: > 68070.407: [FinalReference, 1 refs, 0.0008434 > secs]2017-01-25T14:10:25.955+0530: 68070.408: [PhantomReference, 0 > refs, 0 refs, 0.0014087 secs]2017-01-25T14:10:25.956+0530: 68070.409: > [JNI Weak Reference, 0.0000171 secs], 0.4988131 secs] > > [Parallel Time: 490.1 ms, GC Workers: 20] > > [GC Worker Start (ms): Min: 68069914.2, Avg: 68069914.4, Max: > 68069914.7, Diff: 0.6] > > [Ext Root Scanning (ms): Min: 0.0, Avg: 3.5, Max: 21.2, Diff: > 21.1, Sum: 71.0] > > [Update RS (ms): Min: 0.0, Avg: 1.9, Max: 16.0, Diff: 16.0, Sum: > 37.4] > > [Processed Buffers: Min: 0, Avg: 1.9, Max: 7, Diff: 7, Sum: 39] > > [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.5] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > > [Object Copy (ms): Min: 468.1, Avg: 483.7, Max: 488.6, Diff: > 20.6, Sum: 9673.4] > > [Termination (ms): Min: 0.0, Avg: 0.3, Max: 1.4, Diff: 1.4, Sum: > 6.6] > > [Termination Attempts: Min: 1, Avg: 2.5, Max: 5, Diff: 4, > Sum: 50] > > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, > Sum: 1.6] > > [GC Worker Total (ms): Min: 489.2, Avg: 489.6, Max: 489.9, Diff: > 0.7, Sum: 9792.5] > > [GC Worker End (ms): Min: 68070403.9, Avg: 68070404.0, Max: > 68070404.1, Diff: 0.2] > > [Code Root Fixup: 0.0 ms] > > [Code Root Purge: 0.0 ms] > > [Clear CT: 0.7 ms] > > [Other: 7.9 ms] > > [Choose CSet: 0.1 ms] > > [Ref Proc: 4.7 ms] > > [Ref Enq: 0.4 ms] > > [Redirty Cards: 1.1 ms] > > [Humongous Register: 0.1 ms] > > [Humongous Reclaim: 0.0 ms] > > [Free CSet: 1.4 ms] > > [Eden: 3584.0M(3584.0M)->0.0B(3636.0M) Survivors: 512.0M->460.0M > Heap: 4183.1M(8192.0M)->1063.5M(8192.0M)] > > [Times: user=3.32 sys=2.12, real=0.50 secs] > > 2017-01-25T14:10:25.960+0530: 68070.413: Total time for which > application threads were stopped: 0.4992361 seconds, Stopping threads > took: 0.0000544 seconds > > 2017-01-25T14:10:25.960+0530: 68070.413: Application time: 0.0002027 > seconds > > 2017-01-25T14:10:25.961+0530: 68070.413: Total time for which > application threads were stopped: 0.0002799 seconds, Stopping threads > took: 0.0000997 seconds > > 2017-01-25T14:10:25.977+0530: 68070.429: Application time: 0.0160277 > seconds > > 2017-01-25T14:10:25.977+0530: 68070.430: Total time for which > application threads were stopped: 0.0003467 seconds, Stopping threads > took: 0.0000823 seconds > > 2017-01-25T14:10:26.977+0530: 68071.429: Application time: 0.9997404 > seconds > > 2017-01-25T14:10:26.977+0530: 68071.430: Total time for which > application threads were stopped: 0.0003541 seconds, Stopping threads > took: 0.0000769 seconds > > 2017-01-25T14:10:27.470+0530: 68071.923: Application time: 0.4927146 > seconds > > 2017-01-25T14:10:27.470+0530: 68071.923: [GC pause (G1 Evacuation > Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] > desired_size: 72KB slow allocs: 0 refill waste: 1160B alloc: 0.00098 > 3634KB refills: 4 waste 16.2% gc: 47528B slow: 592B fast: 0B > > TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00003 125KB refills: 2 > waste 30.9% gc: 1544B slow: 32B fast: 0B > > TLAB: gc thread: 0x00007f15906a3800 [id: 141] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00003 106KB refills: 1 > waste 90.8% gc: 1968B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f15906a1800 [id: 140] desired_size: 342KB slow > allocs: 0 refill waste: 5480B alloc: 0.00461 17150KB refills: 1 > waste 84.3% gc: 296088B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 306KB slow > allocs: 0 refill waste: 4896B alloc: 0.00411 15318KB refills: 1 > waste 65.3% gc: 204832B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 344KB slow > allocs: 0 refill waste: 5512B alloc: 0.00463 17230KB refills: 2 > waste 10.6% gc: 75104B slow: 24B fast: 0B > > TLAB: gc thread: 0x00007f159068b000 [id: 127] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 16KB refills: 2 > waste 25.2% gc: 1008B slow: 32B fast: 0B > > TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 342KB slow > allocs: 0 refill waste: 5472B alloc: 0.00460 17113KB refills: 2 > waste 47.1% gc: 330304B slow: 24B fast: 0B > > TLAB: gc thread: 0x00007f159067d800 [id: 121] desired_size: 2KB slow > allocs: 4 refill waste: 32B alloc: 0.00002 80KB refills: 45 > waste 2.6% gc: 1288B slow: 1136B fast: 0B > > TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 335KB slow > allocs: 0 refill waste: 5368B alloc: 0.00451 16793KB refills: 3 > waste 2.7% gc: 27960B slow: 48B fast: 0B > > TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 45KB slow > allocs: 0 refill waste: 720B alloc: 0.00061 2261KB refills: 1 > waste 56.2% gc: 25992B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.68823 2562479KB > refills: 2706 waste 0.0% gc: 0B slow: 121752B fast: 0B > > TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.03467 129069KB > refills: 1 waste 40.0% gc: 839808B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.05604 208660KB > refills: 3 waste 10.5% gc: 658040B slow: 144B fast: 0B > > TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 177KB slow > allocs: 0 refill waste: 2840B alloc: 0.00239 8900KB refills: 1 > waste 42.7% gc: 77840B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 211KB slow > allocs: 0 refill waste: 3376B alloc: 0.00283 10551KB refills: 1 > waste 57.8% gc: 124864B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.17819 663460KB > refills: 5 waste 17.5% gc: 1831760B slow: 3288B fast: 0B > > TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 2KB slow > allocs: 83 refill waste: 32B alloc: 0.00000 5KB refills: 84 > waste 2.2% gc: 608B slow: 3248B fast: 0B > > TLAB: gc thread: 0x00007f170d35a000 [id: 85] desired_size: 82KB slow > allocs: 0 refill waste: 1320B alloc: 0.00111 4136KB refills: 1 > waste 65.0% gc: 55048B slow: 0B fast: 0B > > TLAB totals: thrds: 19 refills: 2866 max: 2706 slow allocs: 87 max 83 > waste: 0.1% gc: 4601584B max: 1831760B slow: 130320B max: 121752B > fast: 0B max: 0B > > > Desired survivor size 268435456 bytes, new threshold 1 (max 15) > > - age 1: 458894880 bytes, 458894880 total > > 68071.923: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 29154, predicted base time: 21.52 ms, remaining time: > 128.48 ms, target pause time: 150.00 ms] > > 68071.923: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 909 regions, survivors: 115 regions, predicted young > region time: 302.72 ms] > > 68071.923: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 909 regions, survivors: 115 regions, old: 0 regions, predicted > pause time: 324.24 ms, target pause time: 150.00 ms] > > 2017-01-25T14:10:27.649+0530: 68072.102: [SoftReference, 0 refs, > 0.0014567 secs]2017-01-25T14:10:27.650+0530: 68072.103: > [WeakReference, 0 refs, 0.0007514 secs]2017-01-25T14:10:27.651+0530: > 68072.104: [FinalReference, 0 refs, 0.0007490 > secs]2017-01-25T14:10:27.652+0530: 68072.105: [PhantomReference, 0 > refs, 0 refs, 0.0015445 secs]2017-01-25T14:10:27.654+0530: 68072.106: > [JNI Weak Reference, 0.0000168 secs], 0.1878181 secs] > > [Parallel Time: 178.2 ms, GC Workers: 20] > > [GC Worker Start (ms): Min: 68071923.1, Avg: 68071923.9, Max: > 68071926.9, Diff: 3.8] > > [Ext Root Scanning (ms): Min: 0.0, Avg: 0.5, Max: 0.9, Diff: > 0.9, Sum: 9.6] > > [Update RS (ms): Min: 0.0, Avg: 2.1, Max: 3.0, Diff: 3.0, Sum: 42.9] > > [Processed Buffers: Min: 0, Avg: 6.6, Max: 15, Diff: 15, Sum: > 132] > > [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.1] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > > [Object Copy (ms): Min: 174.0, Avg: 174.3, Max: 174.4, Diff: > 0.4, Sum: 3485.4] > > [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: > 0.1] > > [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, > Sum: 20] > > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, > Sum: 2.6] > > [GC Worker Total (ms): Min: 174.3, Avg: 177.1, Max: 178.0, Diff: > 3.7, Sum: 3542.6] > > [GC Worker End (ms): Min: 68072101.0, Avg: 68072101.1, Max: > 68072101.2, Diff: 0.2] > > [Code Root Fixup: 0.1 ms] > > [Code Root Purge: 0.0 ms] > > [Clear CT: 0.6 ms] > > [Other: 8.9 ms] > > [Choose CSet: 0.0 ms] > > [Ref Proc: 4.8 ms] > > [Ref Enq: 0.4 ms] > > [Redirty Cards: 0.8 ms] > > [Humongous Register: 0.0 ms] > > [Humongous Reclaim: 0.0 ms] > > [Free CSet: 1.4 ms] > > [Eden: 3636.0M(3636.0M)->0.0B(3940.0M) Survivors: 460.0M->156.0M > Heap: 4699.5M(8192.0M)->1224.0M(8192.0M)] > > [Times: user=2.04 sys=0.04, real=0.19 secs] > > 2017-01-25T14:10:27.658+0530: 68072.111: Total time for which > application threads were stopped: 0.1882558 seconds, Stopping threads > took: 0.0000569 seconds > > 2017-01-25T14:10:27.977+0530: 68072.430: Application time: 0.3188950 > seconds > > 2017-01-25T14:10:27.978+0530: 68072.430: Total time for which > application threads were stopped: 0.0004108 seconds, Stopping threads > took: 0.0001408 seconds > > 2017-01-25T14:10:28.977+0530: 68073.430: Application time: 0.9998111 > seconds > > 2017-01-25T14:10:28.978+0530: 68073.430: Total time for which > application threads were stopped: 0.0004384 seconds, Stopping threads > took: 0.0000808 seconds > > 2017-01-25T14:10:29.104+0530: 68073.556: Application time: 0.1258489 > seconds > > 2017-01-25T14:10:29.104+0530: 68073.556: [GC pause (G1 Evacuation > Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] > desired_size: 52KB slow allocs: 0 refill waste: 840B alloc: 0.00065 > 2627KB refills: 3 waste 2.9% gc: 4568B slow: 72B fast: 0B > > TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00002 90KB refills: 2 > waste 16.1% gc: 648B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f15906a1800 [id: 140] desired_size: 243KB slow > allocs: 0 refill waste: 3888B alloc: 0.00302 12171KB refills: 2 > waste 28.0% gc: 139424B slow: 24B fast: 0B > > TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 216KB slow > allocs: 0 refill waste: 3464B alloc: 0.00269 10844KB refills: 2 > waste 33.0% gc: 146744B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 246KB slow > allocs: 0 refill waste: 3936B alloc: 0.00305 12301KB refills: 4 > waste 9.4% gc: 94712B slow: 80B fast: 0B > > TLAB: gc thread: 0x00007f1590689800 [id: 126] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00002 89KB refills: 1 > waste 95.3% gc: 1968B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 242KB slow > allocs: 0 refill waste: 3872B alloc: 0.00301 12125KB refills: 2 > waste 25.9% gc: 128608B slow: 40B fast: 0B > > TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 242KB slow > allocs: 0 refill waste: 3872B alloc: 0.00300 12111KB refills: 4 > waste 20.0% gc: 198160B slow: 56B fast: 0B > > TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 32KB slow > allocs: 0 refill waste: 512B alloc: 0.00040 1618KB refills: 3 > waste 32.5% gc: 32200B slow: 96B fast: 0B > > TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.79581 3210757KB > refills: 2935 waste 0.0% gc: 0B slow: 125400B fast: 0B > > TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 1818KB > slow allocs: 0 refill waste: 29088B alloc: 0.02254 90920KB > refills: 3 waste 25.2% gc: 1406080B slow: 72B fast: 0B > > TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 2047KB > slow allocs: 1 refill waste: 32760B alloc: 0.03673 148169KB > refills: 4 waste 6.7% gc: 562664B slow: 96B fast: 0B > > TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 125KB slow > allocs: 0 refill waste: 2000B alloc: 0.00155 6272KB refills: 3 > waste 27.1% gc: 104384B slow: 80B fast: 0B > > TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 150KB slow > allocs: 0 refill waste: 2408B alloc: 0.00187 7548KB refills: 2 > waste 47.7% gc: 147464B slow: 32B fast: 0B > > TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 2047KB > slow allocs: 5 refill waste: 32760B alloc: 0.11651 470065KB > refills: 5 waste 0.2% gc: 21408B slow: 264B fast: 0B > > TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 2KB slow > allocs: 94 refill waste: 32B alloc: 0.00002 77KB refills: 92 > waste 2.2% gc: 608B slow: 3480B fast: 0B > > TLAB totals: thrds: 16 refills: 3067 max: 2935 slow allocs: 101 max > 94 waste: 0.1% gc: 2989640B max: 1406080B slow: 129824B max: 125400B > fast: 0B max: 0B > > > Desired survivor size 268435456 bytes, new threshold 15 (max 15) > > - age 1: 142834624 bytes, 142834624 total > > 68073.557: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 32047, predicted base time: 19.88 ms, remaining time: > 130.12 ms, target pause time: 150.00 ms] > > 68073.557: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 985 regions, survivors: 39 regions, predicted young region > time: 172.21 ms] > > 68073.557: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 985 regions, survivors: 39 regions, old: 0 regions, predicted > pause time: 192.09 ms, target pause time: 150.00 ms] > > 2017-01-25T14:10:29.209+0530: 68073.662: [SoftReference, 0 refs, > 0.0009941 secs]2017-01-25T14:10:29.210+0530: 68073.663: > [WeakReference, 1 refs, 0.0005778 secs]2017-01-25T14:10:29.211+0530: > 68073.664: [FinalReference, 0 refs, 0.0007467 > secs]2017-01-25T14:10:29.212+0530: 68073.664: [PhantomReference, 0 > refs, 0 refs, 0.0019718 secs]2017-01-25T14:10:29.214+0530: 68073.666: > [JNI Weak Reference, 0.0000231 secs], 0.1125066 secs] > > [Parallel Time: 104.9 ms, GC Workers: 20] > > [GC Worker Start (ms): Min: 68073556.7, Avg: 68073557.9, Max: > 68073570.3, Diff: 13.6] > > [Ext Root Scanning (ms): Min: 0.0, Avg: 0.5, Max: 1.1, Diff: > 1.1, Sum: 9.9] > > [Update RS (ms): Min: 0.0, Avg: 3.1, Max: 9.1, Diff: 9.1, Sum: 61.3] > > [Processed Buffers: Min: 0, Avg: 7.0, Max: 12, Diff: 12, Sum: > 139] > > [Scan RS (ms): Min: 0.1, Avg: 0.2, Max: 0.3, Diff: 0.2, Sum: 4.2] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > > [Object Copy (ms): Min: 83.1, Avg: 92.1, Max: 96.4, Diff: 13.3, > Sum: 1842.9] > > [Termination (ms): Min: 4.0, Avg: 7.5, Max: 11.2, Diff: 7.3, > Sum: 149.5] > > [Termination Attempts: Min: 1775, Avg: 2783.8, Max: 3057, > Diff: 1282, Sum: 55675] > > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, > Sum: 2.3] > > [GC Worker Total (ms): Min: 91.2, Avg: 103.5, Max: 104.8, Diff: > 13.6, Sum: 2070.1] > > [GC Worker End (ms): Min: 68073661.3, Avg: 68073661.4, Max: > 68073661.5, Diff: 0.2] > > [Code Root Fixup: 0.0 ms] > > [Code Root Purge: 0.0 ms] > > [Clear CT: 0.6 ms] > > [Other: 7.0 ms] > > [Choose CSet: 0.1 ms] > > [Ref Proc: 4.7 ms] > > [Ref Enq: 0.3 ms] > > [Redirty Cards: 0.2 ms] > > [Humongous Register: 0.0 ms] > > [Humongous Reclaim: 0.0 ms] > > [Free CSet: 1.4 ms] > > [Eden: 3940.0M(3940.0M)->0.0B(3936.0M) Survivors: 156.0M->160.0M > Heap: 5164.0M(8192.0M)->1228.0M(8192.0M)] > > [Times: user=2.32 sys=0.00, real=0.11 secs] > > 2017-01-25T14:10:29.217+0530: 68073.669: Total time for which > application threads were stopped: 0.1129090 seconds, Stopping threads > took: 0.0000389 seconds > > 2017-01-25T14:10:29.978+0530: 68074.430: Application time: 0.7611185 > seconds > > 2017-01-25T14:10:29.978+0530: 68074.431: Total time for which > application threads were stopped: 0.0003596 seconds, Stopping threads > took: 0.0000773 seconds > > 2017-01-25T14:10:30.655+0530: 68075.108: Application time: 0.6773986 > seconds > > 2017-01-25T14:10:30.656+0530: 68075.108: [GC pause (G1 Evacuation > Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] > desired_size: 34KB slow allocs: 0 refill waste: 552B alloc: 0.00043 > 1734KB refills: 2 waste 26.2% gc: 18568B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00001 60KB refills: 1 > waste 14.0% gc: 288B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 143KB slow > allocs: 0 refill waste: 2288B alloc: 0.00178 7155KB refills: 1 > waste 41.6% gc: 61008B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f1590692000 [id: 131] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00001 51KB refills: 1 > waste 95.3% gc: 1968B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 165KB slow > allocs: 0 refill waste: 2640B alloc: 0.00205 8255KB refills: 1 > waste 81.8% gc: 138304B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 159KB slow > allocs: 0 refill waste: 2544B alloc: 0.00198 7965KB refills: 8 > waste 9.3% gc: 120768B slow: 264B fast: 0B > > TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 161KB slow > allocs: 0 refill waste: 2576B alloc: 0.00200 8058KB refills: 4 > waste 19.8% gc: 130920B slow: 104B fast: 0B > > TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 21KB slow > allocs: 0 refill waste: 336B alloc: 0.00027 1069KB refills: 3 > waste 29.5% gc: 19336B slow: 40B fast: 0B > > TLAB: gc thread: 0x00007f158c010800 [id: 113] desired_size: 33KB slow > allocs: 7 refill waste: 536B alloc: 0.00042 1678KB refills: 13 > waste 5.9% gc: 24736B slow: 1496B fast: 0B > > TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.86555 3488586KB > refills: 2935 waste 0.0% gc: 0B slow: 129024B fast: 0B > > TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 1202KB > slow allocs: 0 refill waste: 19232B alloc: 0.01491 60103KB > refills: 2 waste 33.4% gc: 823272B slow: 40B fast: 0B > > TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 1948KB > slow allocs: 0 refill waste: 31176B alloc: 0.02418 97446KB > refills: 3 waste 13.0% gc: 777080B slow: 1624B fast: 0B > > TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 83KB slow > allocs: 0 refill waste: 1328B alloc: 0.00103 4153KB refills: 2 > waste 33.2% gc: 56096B slow: 320B fast: 0B > > TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 99KB slow > allocs: 0 refill waste: 1592B alloc: 0.00124 4985KB refills: 4 > waste 23.7% gc: 96856B slow: 80B fast: 0B > > TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.07632 307612KB > refills: 3 waste 6.4% gc: 387864B slow: 12608B fast: 0B > > TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 2KB slow > allocs: 33 refill waste: 40B alloc: 0.00003 125KB refills: 39 > waste 4.0% gc: 2464B slow: 1544B fast: 0B > > TLAB totals: thrds: 16 refills: 3022 max: 2935 slow allocs: 40 max 33 > waste: 0.0% gc: 2659528B max: 823272B slow: 147160B max: 129024B > fast: 0B max: 0B > > > Desired survivor size 268435456 bytes, new threshold 15 (max 15) > > - age 1: 760136 bytes, 760136 total > > - age 2: 142813000 bytes, 143573136 total > > 68075.108: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 4653, predicted base time: 13.35 ms, remaining time: > 136.65 ms, target pause time: 150.00 ms] > > 68075.108: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 984 regions, survivors: 40 regions, predicted young region > time: 183.84 ms] > > 68075.108: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 984 regions, survivors: 40 regions, old: 0 regions, predicted > pause time: 197.19 ms, target pause time: 150.00 ms] > > 2017-01-25T14:10:30.735+0530: 68075.188: [SoftReference, 0 refs, > 0.0011672 secs]2017-01-25T14:10:30.736+0530: 68075.189: > [WeakReference, 0 refs, 0.0007628 secs]2017-01-25T14:10:30.737+0530: > 68075.190: [FinalReference, 0 refs, 0.0009336 > secs]2017-01-25T14:10:30.738+0530: 68075.190: [PhantomReference, 0 > refs, 0 refs, 0.0015168 secs]2017-01-25T14:10:30.739+0530: 68075.192: > [JNI Weak Reference, 0.0000164 secs], 0.0864694 secs] > > [Parallel Time: 78.7 ms, GC Workers: 20] > > [GC Worker Start (ms): Min: 68075108.5, Avg: 68075113.2, Max: > 68075143.8, Diff: 35.3] > > [Ext Root Scanning (ms): Min: 0.0, Avg: 0.3, Max: 1.1, Diff: > 1.1, Sum: 6.1] > > [Update RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 1.3] > > [Processed Buffers: Min: 0, Avg: 1.6, Max: 5, Diff: 5, Sum: 32] > > [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 2.1] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > > [Object Copy (ms): Min: 35.8, Avg: 67.4, Max: 73.2, Diff: 37.4, > Sum: 1348.8] > > [Termination (ms): Min: 2.7, Avg: 5.7, Max: 14.4, Diff: 11.7, > Sum: 113.8] > > [Termination Attempts: Min: 1446, Avg: 2820.5, Max: 3166, > Diff: 1720, Sum: 56410] > > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, > Sum: 2.6] > > [GC Worker Total (ms): Min: 43.2, Avg: 73.7, Max: 78.4, Diff: > 35.2, Sum: 1474.7] > > [GC Worker End (ms): Min: 68075186.8, Avg: 68075186.9, Max: > 68075187.0, Diff: 0.2] > > [Code Root Fixup: 0.0 ms] > > [Code Root Purge: 0.0 ms] > > [Clear CT: 0.5 ms] > > [Other: 7.2 ms] > > [Choose CSet: 0.0 ms] > > [Ref Proc: 4.8 ms] > > [Ref Enq: 0.4 ms] > > [Redirty Cards: 0.3 ms] > > [Humongous Register: 0.0 ms] > > [Humongous Reclaim: 0.0 ms] > > [Free CSet: 1.4 ms] > > [Eden: 3936.0M(3936.0M)->0.0B(3936.0M) Survivors: 160.0M->160.0M > Heap: 5164.0M(8192.0M)->1228.0M(8192.0M)] > > [Times: user=1.55 sys=0.00, real=0.09 secs] > > 2017-01-25T14:10:30.742+0530: 68075.195: Total time for which > application threads were stopped: 0.0868569 seconds, Stopping threads > took: 0.0000422 seconds > > 2017-01-25T14:10:30.961+0530: 68075.414: Application time: 0.2187346 > seconds > > 2017-01-25T14:10:30.961+0530: 68075.414: Total time for which > application threads were stopped: 0.0003312 seconds, Stopping threads > took: 0.0001166 seconds > > 2017-01-25T14:10:30.978+0530: 68075.430: Application time: 0.0164710 > seconds > > 2017-01-25T14:10:30.978+0530: 68075.431: Total time for which > application threads were stopped: 0.0003836 seconds, Stopping threads > took: 0.0000729 seconds > > 2017-01-25T14:10:31.978+0530: 68076.431: Application time: 0.9997933 > seconds > > 2017-01-25T14:10:31.978+0530: 68076.431: Total time for which > application threads were stopped: 0.0003692 seconds, Stopping threads > took: 0.0000542 seconds > > 2017-01-25T14:10:32.213+0530: 68076.666: Application time: 0.2350334 > seconds > > 2017-01-25T14:10:32.214+0530: 68076.666: [GC pause (G1 Evacuation > Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] > desired_size: 23KB slow allocs: 1 refill waste: 368B alloc: 0.00029 > 1156KB refills: 6 waste 11.5% gc: 16024B slow: 256B fast: 0B > > TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00001 39KB refills: 2 > waste 16.1% gc: 648B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f1590695800 [id: 133] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 18KB refills: 2 > waste 26.0% gc: 1056B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 108KB slow > allocs: 0 refill waste: 1736B alloc: 0.00135 5443KB refills: 1 > waste 33.8% gc: 37696B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f1590674800 [id: 124] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 18KB refills: 2 > waste 25.2% gc: 1008B slow: 32B fast: 0B > > TLAB: gc thread: 0x00007f159067d800 [id: 121] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00002 92KB refills: 1 > waste 98.8% gc: 2040B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 14KB slow > allocs: 0 refill waste: 224B alloc: 0.00017 704KB refills: 1 > waste 89.2% gc: 12856B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f159062b000 [id: 115] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 3KB refills: 1 > waste 92.2% gc: 1904B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f159005b000 [id: 114] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 1KB refills: 1 > waste 98.4% gc: 2032B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.91131 3672984KB > refills: 2943 waste 0.0% gc: 0B slow: 126608B fast: 0B > > TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 792KB slow > allocs: 0 refill waste: 12680B alloc: 0.00984 39644KB refills: 1 > waste 84.1% gc: 682600B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 1288KB > slow allocs: 0 refill waste: 20608B alloc: 0.01598 64410KB > refills: 1 waste 87.7% gc: 1157424B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 54KB slow > allocs: 0 refill waste: 872B alloc: 0.00068 2737KB refills: 1 > waste 88.4% gc: 49536B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 65KB slow > allocs: 0 refill waste: 1048B alloc: 0.00081 3279KB refills: 2 > waste 46.4% gc: 62272B slow: 32B fast: 0B > > TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.05015 202117KB > refills: 2 waste 41.0% gc: 1720200B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 2KB slow > allocs: 69 refill waste: 32B alloc: 0.00003 119KB refills: 85 > waste 1.7% gc: 216B slow: 3280B fast: 0B > > TLAB: gc thread: 0x00007f170d35a000 [id: 85] desired_size: 58KB slow > allocs: 0 refill waste: 928B alloc: 0.00072 2910KB refills: 2 > waste 26.8% gc: 31920B slow: 16B fast: 0B > > TLAB totals: thrds: 17 refills: 3054 max: 2943 slow allocs: 71 max 69 > waste: 0.1% gc: 3779432B max: 1720200B slow: 130272B max: 126608B > fast: 0B max: 0B > > > Desired survivor size 268435456 bytes, new threshold 15 (max 15) > > - age 1: 711456 bytes, 711456 total > > - age 2: 751240 bytes, 1462696 total > > - age 3: 142811400 bytes, 144274096 total > > 68076.666: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 3702, predicted base time: 13.05 ms, remaining time: > 136.95 ms, target pause time: 150.00 ms] > > 68076.666: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 984 regions, survivors: 40 regions, predicted young region > time: 156.79 ms] > > 68076.666: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 984 regions, survivors: 40 regions, old: 0 regions, predicted > pause time: 169.84 ms, target pause time: 150.00 ms] > > 2017-01-25T14:10:32.310+0530: 68076.763: [SoftReference, 0 refs, > 0.0012311 secs]2017-01-25T14:10:32.312+0530: 68076.764: > [WeakReference, 1 refs, 0.0007000 secs]2017-01-25T14:10:32.312+0530: > 68076.765: [FinalReference, 0 refs, 0.0006386 > secs]2017-01-25T14:10:32.313+0530: 68076.766: [PhantomReference, 0 > refs, 0 refs, 0.0012583 secs]2017-01-25T14:10:32.314+0530: 68076.767: > [JNI Weak Reference, 0.0000173 secs], 0.1034113 secs] > > [Parallel Time: 96.2 ms, GC Workers: 20] > > [GC Worker Start (ms): Min: 68076666.5, Avg: 68076669.0, Max: > 68076678.6, Diff: 12.1] > > [Ext Root Scanning (ms): Min: 0.0, Avg: 0.3, Max: 1.0, Diff: > 1.0, Sum: 6.1] > > [Update RS (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.3, Sum: 1.4] > > [Processed Buffers: Min: 0, Avg: 1.5, Max: 4, Diff: 4, Sum: 30] > > [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 2.5] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > > [Object Copy (ms): Min: 76.2, Avg: 86.0, Max: 91.6, Diff: 15.4, > Sum: 1720.4] > > [Termination (ms): Min: 3.1, Avg: 6.8, Max: 9.0, Diff: 5.9, Sum: > 136.7] > > [Termination Attempts: Min: 1634, Avg: 2592.2, Max: 2917, > Diff: 1283, Sum: 51845] > > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, > Sum: 2.7] > > [GC Worker Total (ms): Min: 83.9, Avg: 93.5, Max: 96.1, Diff: > 12.2, Sum: 1869.8] > > [GC Worker End (ms): Min: 68076762.4, Avg: 68076762.5, Max: > 68076762.6, Diff: 0.2] > > [Code Root Fixup: 0.0 ms] > > [Code Root Purge: 0.0 ms] > > [Clear CT: 0.7 ms] > > [Other: 6.5 ms] > > [Choose CSet: 0.0 ms] > > [Ref Proc: 4.2 ms] > > [Ref Enq: 0.3 ms] > > [Redirty Cards: 0.3 ms] > > [Humongous Register: 0.1 ms] > > [Humongous Reclaim: 0.0 ms] > > [Free CSet: 1.4 ms] > > [Eden: 3936.0M(3936.0M)->0.0B(3936.0M) Survivors: 160.0M->160.0M > Heap: 5164.0M(8192.0M)->1230.0M(8192.0M)] > > [Times: user=2.15 sys=0.00, real=0.10 secs] > > 2017-01-25T14:10:32.317+0530: 68076.770: Total time for which > application threads were stopped: 0.1038361 seconds, Stopping threads > took: 0.0000385 seconds > > 2017-01-25T14:10:32.978+0530: 68077.431: Application time: 0.6609230 > seconds > > 2017-01-25T14:10:32.978+0530: 68077.431: Total time for which > application threads were stopped: 0.0003272 seconds, Stopping threads > took: 0.0000550 seconds > > 2017-01-25T14:10:33.766+0530: 68078.219: Application time: 0.7878756 > seconds > > 2017-01-25T14:10:33.767+0530: 68078.219: [GC pause (G1 Evacuation > Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] > desired_size: 15KB slow allocs: 1 refill waste: 248B alloc: 0.00020 > 791KB refills: 5 waste 19.1% gc: 15280B slow: 144B fast: 0B > > TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00001 27KB refills: 1 > waste 14.0% gc: 288B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f15906a8800 [id: 144] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 18KB refills: 2 > waste 26.0% gc: 1056B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f15906a7000 [id: 143] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 18KB refills: 2 > waste 26.0% gc: 1056B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f15906a5000 [id: 142] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 18KB refills: 2 > waste 26.0% gc: 1056B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f15906a0000 [id: 139] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 18KB refills: 2 > waste 25.2% gc: 1008B slow: 32B fast: 0B > > TLAB: gc thread: 0x00007f159069e800 [id: 138] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 18KB refills: 2 > waste 25.2% gc: 1008B slow: 32B fast: 0B > > TLAB: gc thread: 0x00007f159069c800 [id: 137] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 18KB refills: 2 > waste 25.2% gc: 1008B slow: 32B fast: 0B > > TLAB: gc thread: 0x00007f1590697800 [id: 134] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 18KB refills: 2 > waste 26.0% gc: 1056B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f1590693800 [id: 132] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 18KB refills: 2 > waste 26.0% gc: 1056B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 108KB slow > allocs: 0 refill waste: 1728B alloc: 0.00134 5419KB refills: 1 > waste 100.0% gc: 110960B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f159068e800 [id: 129] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 18KB refills: 2 > waste 26.0% gc: 1056B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f159068d000 [id: 128] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 18KB refills: 2 > waste 25.0% gc: 1016B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f1590673000 [id: 123] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 18KB refills: 2 > waste 25.2% gc: 1008B slow: 32B fast: 0B > > TLAB: gc thread: 0x00007f1590671800 [id: 122] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 18KB refills: 2 > waste 25.2% gc: 1008B slow: 32B fast: 0B > > TLAB: gc thread: 0x00007f1590637000 [id: 117] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 18KB refills: 2 > waste 25.8% gc: 1048B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 9KB slow > allocs: 0 refill waste: 144B alloc: 0.00011 458KB refills: 2 > waste 41.7% gc: 7816B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.94192 3796378KB > refills: 2942 waste 0.0% gc: 0B slow: 127600B fast: 0B > > TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.03274 131961KB > refills: 1 waste 60.0% gc: 1257536B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow > allocs: 29 refill waste: 48B alloc: 0.00004 155KB refills: 31 > waste 1.9% gc: 608B slow: 1296B fast: 0B > > TLAB totals: thrds: 20 refills: 3009 max: 2942 slow allocs: 30 max 29 > waste: 0.0% gc: 1405928B max: 1257536B slow: 129344B max: 127600B > fast: 0B max: 0B > > > Desired survivor size 268435456 bytes, new threshold 15 (max 15) > > - age 1: 50800 bytes, 50800 total > > - age 2: 708304 bytes, 759104 total > > - age 3: 751208 bytes, 1510312 total > > - age 4: 142803240 bytes, 144313552 total > > 68078.219: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 4372, predicted base time: 12.18 ms, remaining time: > 137.82 ms, target pause time: 150.00 ms] > > 68078.219: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 984 regions, survivors: 40 regions, predicted young region > time: 148.25 ms] > > 68078.219: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 984 regions, survivors: 40 regions, old: 0 regions, predicted > pause time: 160.43 ms, target pause time: 150.00 ms] > > 2017-01-25T14:10:33.847+0530: 68078.299: [SoftReference, 0 refs, > 0.0010865 secs]2017-01-25T14:10:33.848+0530: 68078.300: > [WeakReference, 0 refs, 0.0007960 secs]2017-01-25T14:10:33.848+0530: > 68078.301: [FinalReference, 0 refs, 0.0007028 > secs]2017-01-25T14:10:33.849+0530: 68078.302: [PhantomReference, 0 > refs, 0 refs, 0.0015020 secs]2017-01-25T14:10:33.851+0530: 68078.303: > [JNI Weak Reference, 0.0000198 secs], 0.0873865 secs] > > [Parallel Time: 79.2 ms, GC Workers: 20] > > [GC Worker Start (ms): Min: 68078219.5, Avg: 68078228.0, Max: > 68078264.4, Diff: 44.9] > > [Ext Root Scanning (ms): Min: 0.0, Avg: 0.3, Max: 1.1, Diff: > 1.1, Sum: 5.9] > > [Update RS (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 1.4] > > [Processed Buffers: Min: 0, Avg: 1.9, Max: 6, Diff: 6, Sum: 37] > > [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 2.3] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > > [Object Copy (ms): Min: 29.0, Avg: 64.0, Max: 75.6, Diff: 46.6, > Sum: 1280.6] > > [Termination (ms): Min: 2.1, Avg: 6.0, Max: 19.0, Diff: 16.9, > Sum: 119.3] > > [Termination Attempts: Min: 1180, Avg: 2444.2, Max: 2973, > Diff: 1793, Sum: 48883] > > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, > Sum: 2.3] > > [GC Worker Total (ms): Min: 34.1, Avg: 70.6, Max: 79.1, Diff: > 45.0, Sum: 1411.8] > > [GC Worker End (ms): Min: 68078298.5, Avg: 68078298.6, Max: > 68078298.7, Diff: 0.2] > > [Code Root Fixup: 0.0 ms] > > [Code Root Purge: 0.0 ms] > > [Clear CT: 0.5 ms] > > [Other: 7.6 ms] > > [Choose CSet: 0.0 ms] > > [Ref Proc: 4.5 ms] > > [Ref Enq: 0.4 ms] > > [Redirty Cards: 0.3 ms] > > [Humongous Register: 0.0 ms] > > [Humongous Reclaim: 0.0 ms] > > [Free CSet: 2.0 ms] > > [Eden: 3936.0M(3936.0M)->0.0B(3932.0M) Survivors: 160.0M->164.0M > Heap: 5166.0M(8192.0M)->1232.0M(8192.0M)] > > [Times: user=1.47 sys=0.00, real=0.09 secs] > > 2017-01-25T14:10:33.854+0530: 68078.307: Total time for which > application threads were stopped: 0.0878013 seconds, Stopping threads > took: 0.0000417 seconds > > 2017-01-25T14:10:33.978+0530: 68078.431: Application time: 0.1240628 > seconds > > 2017-01-25T14:10:33.979+0530: 68078.431: Total time for which > application threads were stopped: 0.0003893 seconds, Stopping threads > took: 0.0000755 seconds > > 2017-01-25T14:10:34.978+0530: 68079.431: Application time: 0.9997692 > seconds > > 2017-01-25T14:10:34.979+0530: 68079.431: Total time for which > application threads were stopped: 0.0003924 seconds, Stopping threads > took: 0.0000777 seconds > > 2017-01-25T14:10:35.281+0530: 68079.733: Application time: 0.3019245 > seconds > > 2017-01-25T14:10:35.281+0530: 68079.734: [GC pause (G1 Evacuation > Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] > desired_size: 10KB slow allocs: 3 refill waste: 168B alloc: 0.00013 > 534KB refills: 11 waste 0.4% gc: 120B slow: 416B fast: 0B > > TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00000 18KB refills: 2 > waste 16.1% gc: 648B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 70KB slow > allocs: 0 refill waste: 1120B alloc: 0.00087 3519KB refills: 4 > waste 21.5% gc: 61856B slow: 120B fast: 0B > > TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 107KB slow > allocs: 0 refill waste: 1712B alloc: 0.00133 5363KB refills: 2 > waste 46.5% gc: 102184B slow: 24B fast: 0B > > TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 6KB slow > allocs: 0 refill waste: 96B alloc: 0.00007 301KB refills: 4 > waste 1.4% gc: 272B slow: 80B fast: 0B > > TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.96203 3873484KB > refills: 2938 waste 0.0% gc: 0B slow: 126672B fast: 0B > > TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 517KB slow > allocs: 0 refill waste: 8280B alloc: 0.00643 25881KB refills: 1 > waste 50.0% gc: 265128B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 844KB slow > allocs: 0 refill waste: 13512B alloc: 0.01049 42241KB refills: 1 > waste 47.6% gc: 412024B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 35KB slow > allocs: 0 refill waste: 568B alloc: 0.00044 1787KB refills: 2 > waste 27.5% gc: 20088B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 42KB slow > allocs: 0 refill waste: 680B alloc: 0.00053 2132KB refills: 1 > waste 16.5% gc: 7208B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 1716KB > slow allocs: 0 refill waste: 27456B alloc: 0.02131 85816KB > refills: 1 waste 13.4% gc: 234760B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 2KB slow > allocs: 51 refill waste: 40B alloc: 0.00003 136KB refills: 68 > waste 2.2% gc: 1336B slow: 2792B fast: 0B > > TLAB: gc thread: 0x00007f170d2cf000 [id: 84] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 12KB refills: 2 > waste 49.6% gc: 2032B slow: 16B fast: 0B > > TLAB totals: thrds: 13 refills: 3037 max: 2938 slow allocs: 55 max 51 > waste: 0.0% gc: 1107656B max: 412024B slow: 130152B max: 126672B > fast: 0B max: 0B > > > Desired survivor size 268435456 bytes, new threshold 15 (max 15) > > - age 1: 17912 bytes, 17912 total > > - age 2: 42776 bytes, 60688 total > > - age 3: 708128 bytes, 768816 total > > - age 4: 734632 bytes, 1503448 total > > - age 5: 142801080 bytes, 144304528 total > > 68079.734: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 3423, predicted base time: 13.52 ms, remaining time: > 136.48 ms, target pause time: 150.00 ms] > > 68079.734: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 983 regions, survivors: 41 regions, predicted young region > time: 127.04 ms] > > 68079.734: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 983 regions, survivors: 41 regions, old: 0 regions, predicted > pause time: 140.56 ms, target pause time: 150.00 ms] > > 2017-01-25T14:10:35.378+0530: 68079.831: [SoftReference, 0 refs, > 0.0009839 secs]2017-01-25T14:10:35.379+0530: 68079.832: > [WeakReference, 0 refs, 0.0004729 secs]2017-01-25T14:10:35.380+0530: > 68079.832: [FinalReference, 0 refs, 0.0013721 > secs]2017-01-25T14:10:35.381+0530: 68079.834: [PhantomReference, 0 > refs, 0 refs, 0.0016699 secs]2017-01-25T14:10:35.383+0530: 68079.835: > [JNI Weak Reference, 0.0000267 secs], 0.1043019 secs] > > [Parallel Time: 96.5 ms, GC Workers: 20] > > [GC Worker Start (ms): Min: 68079733.9, Avg: 68079734.3, Max: > 68079737.4, Diff: 3.6] > > [Ext Root Scanning (ms): Min: 0.0, Avg: 0.4, Max: 1.2, Diff: > 1.2, Sum: 7.4] > > [Update RS (ms): Min: 0.0, Avg: 0.5, Max: 9.4, Diff: 9.4, Sum: 10.7] > > [Processed Buffers: Min: 0, Avg: 1.2, Max: 4, Diff: 4, Sum: 25] > > [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.8] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > > [Object Copy (ms): Min: 79.1, Avg: 87.4, Max: 91.7, Diff: 12.7, > Sum: 1748.5] > > [Termination (ms): Min: 3.2, Avg: 7.1, Max: 8.2, Diff: 5.0, Sum: > 142.2] > > [Termination Attempts: Min: 1695, Avg: 2948.3, Max: 3320, > Diff: 1625, Sum: 58966] > > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.2, > Sum: 3.0] > > [GC Worker Total (ms): Min: 92.7, Avg: 95.7, Max: 96.2, Diff: > 3.5, Sum: 1914.6] > > [GC Worker End (ms): Min: 68079829.9, Avg: 68079830.0, Max: > 68079830.1, Diff: 0.2] > > [Code Root Fixup: 0.0 ms] > > [Code Root Purge: 0.0 ms] > > [Clear CT: 0.5 ms] > > [Other: 7.3 ms] > > [Choose CSet: 0.0 ms] > > [Ref Proc: 4.9 ms] > > [Ref Enq: 0.4 ms] > > [Redirty Cards: 0.2 ms] > > [Humongous Register: 0.1 ms] > > [Humongous Reclaim: 0.0 ms] > > [Free CSet: 1.4 ms] > > [Eden: 3932.0M(3932.0M)->0.0B(3944.0M) Survivors: 164.0M->152.0M > Heap: 5164.0M(8192.0M)->1222.0M(8192.0M)] > > [Times: user=2.28 sys=0.00, real=0.10 secs] > > 2017-01-25T14:10:35.385+0530: 68079.838: Total time for which > application threads were stopped: 0.1047001 seconds, Stopping threads > took: 0.0000510 seconds > > 2017-01-25T14:10:35.962+0530: 68080.414: Application time: 0.5761329 > seconds > > 2017-01-25T14:10:35.962+0530: 68080.415: Total time for which > application threads were stopped: 0.0003434 seconds, Stopping threads > took: 0.0000783 seconds > > 2017-01-25T14:10:35.979+0530: 68080.431: Application time: 0.0166615 > seconds > > 2017-01-25T14:10:35.979+0530: 68080.432: Total time for which > application threads were stopped: 0.0004284 seconds, Stopping threads > took: 0.0000705 seconds > > 2017-01-25T14:10:36.852+0530: 68081.305: Application time: 0.8734277 > seconds > > 2017-01-25T14:10:36.853+0530: 68081.305: [GC pause (G1 Evacuation > Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] > desired_size: 7KB slow allocs: 4 refill waste: 120B alloc: 0.00009 > 380KB refills: 8 waste 3.4% gc: 1600B slow: 536B fast: 0B > > TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 13KB refills: 1 > waste 14.0% gc: 288B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f15906a1800 [id: 140] desired_size: 158KB slow > allocs: 0 refill waste: 2536B alloc: 0.00197 7938KB refills: 2 > waste 3.3% gc: 10752B slow: 56B fast: 0B > > TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 93KB slow > allocs: 0 refill waste: 1496B alloc: 0.00116 4686KB refills: 1 > waste 71.5% gc: 68656B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 46KB slow > allocs: 0 refill waste: 744B alloc: 0.00058 2345KB refills: 5 > waste 8.2% gc: 19568B slow: 112B fast: 0B > > TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 71KB slow > allocs: 0 refill waste: 1136B alloc: 0.00088 3560KB refills: 3 > waste 8.9% gc: 19320B slow: 64B fast: 0B > > TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 70KB slow > allocs: 0 refill waste: 1128B alloc: 0.00088 3534KB refills: 3 > waste 19.7% gc: 42520B slow: 152B fast: 0B > > TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 3KB slow > allocs: 1 refill waste: 56B alloc: 0.00005 199KB refills: 11 > waste 5.0% gc: 1952B slow: 272B fast: 0B > > TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.97497 3937562KB > refills: 2941 waste 0.0% gc: 0B slow: 125192B fast: 0B > > TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 337KB slow > allocs: 0 refill waste: 5392B alloc: 0.00418 16874KB refills: 7 > waste 5.5% gc: 131656B slow: 376B fast: 0B > > TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 550KB slow > allocs: 0 refill waste: 8808B alloc: 0.00682 27540KB refills: 5 > waste 7.1% gc: 200160B slow: 768B fast: 0B > > TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 23KB slow > allocs: 0 refill waste: 368B alloc: 0.00029 1166KB refills: 6 > waste 12.3% gc: 17416B slow: 200B fast: 0B > > TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 27KB slow > allocs: 0 refill waste: 440B alloc: 0.00034 1390KB refills: 7 > waste 12.6% gc: 24792B slow: 392B fast: 0B > > TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 1124KB > slow allocs: 0 refill waste: 17992B alloc: 0.01393 56239KB > refills: 5 waste 2.1% gc: 117352B slow: 856B fast: 0B > > TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow > allocs: 37 refill waste: 48B alloc: 0.00004 160KB refills: 30 > waste 3.5% gc: 2016B slow: 1432B fast: 0B > > TLAB: gc thread: 0x00007f170d35a000 [id: 85] desired_size: 37KB slow > allocs: 0 refill waste: 600B alloc: 0.00047 1896KB refills: 1 > waste 23.6% gc: 9144B slow: 0B fast: 0B > > TLAB totals: thrds: 16 refills: 3036 max: 2941 slow allocs: 42 max 37 > waste: 0.0% gc: 667192B max: 200160B slow: 130408B max: 125192B fast: > 0B max: 0B > > > Desired survivor size 268435456 bytes, new threshold 15 (max 15) > > - age 1: 123944 bytes, 123944 total > > - age 2: 14808 bytes, 138752 total > > - age 3: 42672 bytes, 181424 total > > - age 4: 691808 bytes, 873232 total > > - age 5: 731720 bytes, 1604952 total > > - age 6: 142801080 bytes, 144406032 total > > 68081.305: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 4530, predicted base time: 12.52 ms, remaining time: > 137.48 ms, target pause time: 150.00 ms] > > 68081.305: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 986 regions, survivors: 38 regions, predicted young region > time: 121.86 ms] > > 68081.305: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 986 regions, survivors: 38 regions, old: 0 regions, predicted > pause time: 134.38 ms, target pause time: 150.00 ms] > > 2017-01-25T14:10:36.952+0530: 68081.405: [SoftReference, 0 refs, > 0.0011154 secs]2017-01-25T14:10:36.954+0530: 68081.406: > [WeakReference, 0 refs, 0.0007341 secs]2017-01-25T14:10:36.954+0530: > 68081.407: [FinalReference, 0 refs, 0.0006622 > secs]2017-01-25T14:10:36.955+0530: 68081.408: [PhantomReference, 0 > refs, 0 refs, 0.0011375 secs]2017-01-25T14:10:36.956+0530: 68081.409: > [JNI Weak Reference, 0.0000164 secs], 0.1062167 secs] > > [Parallel Time: 99.2 ms, GC Workers: 20] > > [GC Worker Start (ms): Min: 68081305.5, Avg: 68081305.7, Max: > 68081306.1, Diff: 0.6] > > [Ext Root Scanning (ms): Min: 0.0, Avg: 0.6, Max: 5.4, Diff: > 5.4, Sum: 11.8] > > [Update RS (ms): Min: 0.0, Avg: 0.5, Max: 8.3, Diff: 8.3, Sum: 9.8] > > [Processed Buffers: Min: 0, Avg: 1.6, Max: 5, Diff: 5, Sum: 32] > > [Scan RS (ms): Min: 0.0, Avg: 0.2, Max: 0.3, Diff: 0.2, Sum: 3.6] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > > [Object Copy (ms): Min: 82.5, Avg: 90.3, Max: 94.8, Diff: 12.3, > Sum: 1805.9] > > [Termination (ms): Min: 3.2, Avg: 7.0, Max: 8.2, Diff: 5.0, Sum: > 140.6] > > [Termination Attempts: Min: 1726, Avg: 2931.2, Max: 3169, > Diff: 1443, Sum: 58623] > > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, > Sum: 2.5] > > [GC Worker Total (ms): Min: 98.3, Avg: 98.7, Max: 99.0, Diff: > 0.7, Sum: 1974.2] > > [GC Worker End (ms): Min: 68081404.3, Avg: 68081404.4, Max: > 68081404.5, Diff: 0.2] > > [Code Root Fixup: 0.0 ms] > > [Code Root Purge: 0.0 ms] > > [Clear CT: 0.5 ms] > > [Other: 6.5 ms] > > [Choose CSet: 0.0 ms] > > [Ref Proc: 4.1 ms] > > [Ref Enq: 0.4 ms] > > [Redirty Cards: 0.3 ms] > > [Humongous Register: 0.0 ms] > > [Humongous Reclaim: 0.0 ms] > > [Free CSet: 1.4 ms] > > [Eden: 3944.0M(3944.0M)->0.0B(3940.0M) Survivors: 152.0M->156.0M > Heap: 5166.0M(8192.0M)->1224.0M(8192.0M)] > > [Times: user=2.14 sys=0.00, real=0.11 secs] > > 2017-01-25T14:10:36.959+0530: 68081.412: Total time for which > application threads were stopped: 0.1066190 seconds, Stopping threads > took: 0.0000391 seconds > > 2017-01-25T14:10:36.978+0530: 68081.430: Application time: 0.0187454 > seconds > > 2017-01-25T14:10:36.978+0530: 68081.431: Total time for which > application threads were stopped: 0.0003858 seconds, Stopping threads > took: 0.0000614 seconds > > 2017-01-25T14:10:37.978+0530: 68082.431: Application time: 0.9999582 > seconds > > 2017-01-25T14:10:37.979+0530: 68082.431: Total time for which > application threads were stopped: 0.0004068 seconds, Stopping threads > took: 0.0000728 seconds > > 2017-01-25T14:10:38.377+0530: 68082.829: Application time: 0.3980715 > seconds > > 2017-01-25T14:10:38.377+0530: 68082.830: [GC pause (G1 Evacuation > Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] > desired_size: 5KB slow allocs: 8 refill waste: 80B alloc: 0.00007 > 269KB refills: 19 waste 5.3% gc: 4760B slow: 760B fast: 0B > > TLAB: gc thread: 0x00007f15f8114800 [id: 199] desired_size: 2047KB > slow allocs: 1 refill waste: 32760B alloc: 0.02691 108566KB > refills: 27 waste 3.6% gc: 2042560B slow: 840B fast: 0B > > TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00000 9KB refills: 2 > waste 16.1% gc: 648B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f15906a1800 [id: 140] desired_size: 104KB slow > allocs: 0 refill waste: 1664B alloc: 0.00129 5210KB refills: 3 > waste 22.5% gc: 71936B slow: 40B fast: 0B > > TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 31KB slow > allocs: 0 refill waste: 496B alloc: 0.00039 1560KB refills: 3 > waste 28.5% gc: 27208B slow: 88B fast: 0B > > TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 46KB slow > allocs: 0 refill waste: 736B alloc: 0.00058 2323KB refills: 1 > waste 64.9% gc: 30880B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 2KB slow > allocs: 0 refill waste: 40B alloc: 0.00003 141KB refills: 3 > waste 18.9% gc: 1560B slow: 80B fast: 0B > > TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.98287 3965429KB > refills: 2917 waste 0.0% gc: 0B slow: 126104B fast: 0B > > TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 230KB slow > allocs: 0 refill waste: 3680B alloc: 0.00286 11521KB refills: 3 > waste 16.8% gc: 118472B slow: 184B fast: 0B > > TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 372KB slow > allocs: 0 refill waste: 5952B alloc: 0.00461 18616KB refills: 2 > waste 46.2% gc: 352504B slow: 104B fast: 0B > > TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 15KB slow > allocs: 1 refill waste: 248B alloc: 0.00020 796KB refills: 3 > waste 32.5% gc: 15528B slow: 360B fast: 0B > > TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 19KB slow > allocs: 1 refill waste: 304B alloc: 0.00024 964KB refills: 2 > waste 32.3% gc: 12440B slow: 304B fast: 0B > > TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 764KB slow > allocs: 0 refill waste: 12224B alloc: 0.00947 38218KB refills: 1 > waste 17.1% gc: 133552B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 2KB slow > allocs: 58 refill waste: 40B alloc: 0.00003 140KB refills: 73 > waste 2.8% gc: 2416B slow: 3384B fast: 0B > > TLAB totals: thrds: 14 refills: 3059 max: 2917 slow allocs: 70 max 58 > waste: 0.0% gc: 2814464B max: 2042560B slow: 132264B max: 126104B > fast: 0B max: 0B > > > Desired survivor size 268435456 bytes, new threshold 15 (max 15) > > - age 1: 397432 bytes, 397432 total > > - age 2: 112808 bytes, 510240 total > > - age 3: 14208 bytes, 524448 total > > - age 4: 41328 bytes, 565776 total > > - age 5: 689344 bytes, 1255120 total > > - age 6: 730968 bytes, 1986088 total > > - age 7: 142800480 bytes, 144786568 total > > 68082.830: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 6380, predicted base time: 11.57 ms, remaining time: > 138.43 ms, target pause time: 150.00 ms] > > 68082.830: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 985 regions, survivors: 39 regions, predicted young region > time: 119.81 ms] > > 68082.830: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 985 regions, survivors: 39 regions, old: 0 regions, predicted > pause time: 131.38 ms, target pause time: 150.00 ms] > > 2017-01-25T14:10:38.474+0530: 68082.926: [SoftReference, 0 refs, > 0.0013481 secs]2017-01-25T14:10:38.475+0530: 68082.928: > [WeakReference, 0 refs, 0.0007696 secs]2017-01-25T14:10:38.476+0530: > 68082.928: [FinalReference, 3 refs, 0.0007260 > secs]2017-01-25T14:10:38.477+0530: 68082.929: [PhantomReference, 0 > refs, 0 refs, 0.0014520 secs]2017-01-25T14:10:38.478+0530: 68082.931: > [JNI Weak Reference, 0.0000179 secs], 0.1038195 secs] > > [Parallel Time: 95.8 ms, GC Workers: 20] > > [GC Worker Start (ms): Min: 68082829.8, Avg: 68082832.5, Max: > 68082846.4, Diff: 16.6] > > [Ext Root Scanning (ms): Min: 0.0, Avg: 0.3, Max: 1.1, Diff: > 1.1, Sum: 6.2] > > [Update RS (ms): Min: 0.0, Avg: 0.9, Max: 6.8, Diff: 6.8, Sum: 17.8] > > [Processed Buffers: Min: 0, Avg: 1.9, Max: 5, Diff: 5, Sum: 37] > > [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 1.8] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > > [Object Copy (ms): Min: 68.4, Avg: 81.1, Max: 87.1, Diff: 18.7, > Sum: 1622.2] > > [Termination (ms): Min: 2.9, Avg: 6.2, Max: 16.5, Diff: 13.6, > Sum: 124.3] > > [Termination Attempts: Min: 1517, Avg: 2632.0, Max: 3026, > Diff: 1509, Sum: 52640] > > [GC Worker Other (ms): Min: 0.0, Avg: 2.9, Max: 4.5, Diff: 4.4, > Sum: 57.4] > > [GC Worker Total (ms): Min: 74.9, Avg: 91.5, Max: 95.7, Diff: > 20.9, Sum: 1829.7] > > [GC Worker End (ms): Min: 68082921.1, Avg: 68082924.0, Max: > 68082925.6, Diff: 4.4] > > [Code Root Fixup: 0.1 ms] > > [Code Root Purge: 0.0 ms] > > [Clear CT: 0.5 ms] > > [Other: 7.4 ms] > > [Choose CSet: 0.1 ms] > > [Ref Proc: 5.0 ms] > > [Ref Enq: 0.4 ms] > > [Redirty Cards: 0.3 ms] > > [Humongous Register: 0.1 ms] > > [Humongous Reclaim: 0.0 ms] > > [Free CSet: 1.4 ms] > > [Eden: 3940.0M(3940.0M)->0.0B(3940.0M) Survivors: 156.0M->156.0M > Heap: 5164.0M(8192.0M)->1226.0M(8192.0M)] > > [Times: user=3.49 sys=0.00, real=0.11 secs] > > 2017-01-25T14:10:38.481+0530: 68082.933: Total time for which > application threads were stopped: 0.1042423 seconds, Stopping threads > took: 0.0000412 seconds > > 2017-01-25T14:10:38.481+0530: 68082.934: Application time: 0.0001033 > seconds > > 2017-01-25T14:10:38.481+0530: 68082.934: Total time for which > application threads were stopped: 0.0003928 seconds, Stopping threads > took: 0.0000939 seconds > > 2017-01-25T14:10:38.978+0530: 68083.431: Application time: 0.4969123 > seconds > > 2017-01-25T14:10:38.979+0530: 68083.431: Total time for which > application threads were stopped: 0.0004010 seconds, Stopping threads > took: 0.0000855 seconds > > 2017-01-25T14:10:39.933+0530: 68084.386: Application time: 0.9545832 > seconds > > 2017-01-25T14:10:39.934+0530: 68084.386: [GC pause (G1 Evacuation > Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] > desired_size: 4KB slow allocs: 3 refill waste: 64B alloc: 0.00005 > 210KB refills: 13 waste 7.0% gc: 3512B slow: 384B fast: 0B > > TLAB: gc thread: 0x00007f15f8114800 [id: 199] desired_size: 1655KB > slow allocs: 0 refill waste: 26488B alloc: 0.02052 82776KB > refills: 2 waste 48.0% gc: 1626104B slow: 56B fast: 0B > > TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 7KB refills: 1 > waste 14.0% gc: 288B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.98542 3975742KB > refills: 2952 waste 0.0% gc: 0B slow: 125280B fast: 0B > > TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 503KB slow > allocs: 0 refill waste: 8056B alloc: 0.00625 25200KB refills: 1 > waste 77.0% gc: 397600B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow > allocs: 26 refill waste: 48B alloc: 0.00004 169KB refills: 29 > waste 2.2% gc: 1064B slow: 1120B fast: 0B > > TLAB totals: thrds: 6 refills: 2998 max: 2952 slow allocs: 29 max 26 > waste: 0.0% gc: 2028568B max: 1626104B slow: 126840B max: 125280B > fast: 0B max: 0B > > > Desired survivor size 268435456 bytes, new threshold 15 (max 15) > > - age 1: 319896 bytes, 319896 total > > - age 2: 388232 bytes, 708128 total > > - age 3: 112808 bytes, 820936 total > > - age 4: 14208 bytes, 835144 total > > - age 5: 41328 bytes, 876472 total > > - age 6: 689344 bytes, 1565816 total > > - age 7: 730968 bytes, 2296784 total > > - age 8: 142800480 bytes, 145097264 total > > 68084.386: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 5401, predicted base time: 12.96 ms, remaining time: > 137.04 ms, target pause time: 150.00 ms] > > 68084.386: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 985 regions, survivors: 39 regions, predicted young region > time: 110.43 ms] > > 68084.386: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 985 regions, survivors: 39 regions, old: 0 regions, predicted > pause time: 123.39 ms, target pause time: 150.00 ms] > > 2017-01-25T14:10:40.040+0530: 68084.492: [SoftReference, 0 refs, > 0.0018914 secs]2017-01-25T14:10:40.042+0530: 68084.494: > [WeakReference, 0 refs, 0.0009650 secs]2017-01-25T14:10:40.043+0530: > 68084.495: [FinalReference, 1 refs, 0.0010309 > secs]2017-01-25T14:10:40.044+0530: 68084.496: [PhantomReference, 0 > refs, 0 refs, 0.0014095 secs]2017-01-25T14:10:40.045+0530: 68084.498: > [JNI Weak Reference, 0.0000247 secs], 0.1142976 secs] > > [Parallel Time: 105.5 ms, GC Workers: 20] > > [GC Worker Start (ms): Min: 68084386.4, Avg: 68084388.9, Max: > 68084402.4, Diff: 16.0] > > [Ext Root Scanning (ms): Min: 0.0, Avg: 0.3, Max: 1.1, Diff: > 1.1, Sum: 6.4] > > [Update RS (ms): Min: 0.0, Avg: 0.2, Max: 0.4, Diff: 0.4, Sum: 3.7] > > [Processed Buffers: Min: 0, Avg: 1.4, Max: 7, Diff: 7, Sum: 29] > > [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 1.7] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > > [Object Copy (ms): Min: 73.1, Avg: 92.3, Max: 97.6, Diff: 24.5, > Sum: 1845.1] > > [Termination (ms): Min: 3.9, Avg: 9.6, Max: 20.0, Diff: 16.0, > Sum: 192.4] > > [Termination Attempts: Min: 2087, Avg: 3497.8, Max: 3956, > Diff: 1869, Sum: 69955] > > [GC Worker Other (ms): Min: 0.0, Avg: 0.2, Max: 0.3, Diff: 0.3, > Sum: 3.2] > > [GC Worker Total (ms): Min: 89.1, Avg: 102.6, Max: 105.2, Diff: > 16.2, Sum: 2052.5] > > [GC Worker End (ms): Min: 68084491.4, Avg: 68084491.5, Max: > 68084491.7, Diff: 0.3] > > [Code Root Fixup: 0.1 ms] > > [Code Root Purge: 0.0 ms] > > [Clear CT: 0.5 ms] > > [Other: 8.3 ms] > > [Choose CSet: 0.0 ms] > > [Ref Proc: 5.9 ms] > > [Ref Enq: 0.4 ms] > > [Redirty Cards: 0.3 ms] > > [Humongous Register: 0.1 ms] > > [Humongous Reclaim: 0.0 ms] > > [Free CSet: 1.4 ms] > > [Eden: 3940.0M(3940.0M)->0.0B(3940.0M) Survivors: 156.0M->156.0M > Heap: 5166.0M(8192.0M)->1224.0M(8192.0M)] > > [Times: user=3.16 sys=0.00, real=0.11 secs] > > 2017-01-25T14:10:40.048+0530: 68084.501: Total time for which > application threads were stopped: 0.1147353 seconds, Stopping threads > took: 0.0000420 seconds > > 2017-01-25T14:10:40.048+0530: 68084.501: Application time: 0.0001454 > seconds > > 2017-01-25T14:10:40.049+0530: 68084.501: Total time for which > application threads were stopped: 0.0003603 seconds, Stopping threads > took: 0.0000723 seconds > > 2017-01-25T14:10:40.962+0530: 68085.415: Application time: 0.9135759 > seconds > > 2017-01-25T14:10:40.962+0530: 68085.415: Total time for which > application threads were stopped: 0.0002860 seconds, Stopping threads > took: 0.0000639 seconds > > 2017-01-25T14:10:41.048+0530: 68085.501: Application time: 0.0858210 > seconds > > 2017-01-25T14:10:41.049+0530: 68085.501: Total time for which > application threads were stopped: 0.0003932 seconds, Stopping threads > took: 0.0000801 seconds > > 2017-01-25T14:10:41.139+0530: 68085.591: Application time: 0.0900708 > seconds > > 2017-01-25T14:10:41.139+0530: 68085.592: Total time for which > application threads were stopped: 0.0006986 seconds, Stopping threads > took: 0.0000592 seconds > > 2017-01-25T14:10:41.511+0530: 68085.964: Application time: 0.3716063 > seconds > > 2017-01-25T14:10:41.511+0530: 68085.964: [GC pause (G1 Evacuation > Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] > desired_size: 3KB slow allocs: 18 refill waste: 48B alloc: 0.00004 > 155KB refills: 31 waste 3.1% gc: 1928B slow: 1152B fast: 0B > > TLAB: gc thread: 0x00007f15f8114800 [id: 199] desired_size: 1076KB > slow allocs: 0 refill waste: 17216B alloc: 0.01334 53823KB > refills: 1 waste 63.6% gc: 701328B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00000 5KB refills: 2 > waste 16.1% gc: 648B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 47KB slow > allocs: 0 refill waste: 752B alloc: 0.00059 2361KB refills: 1 > waste 57.1% gc: 27632B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 30KB slow > allocs: 0 refill waste: 480B alloc: 0.00038 1520KB refills: 2 > waste 31.7% gc: 19672B slow: 48B fast: 0B > > TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00002 94KB refills: 2 > waste 24.0% gc: 952B slow: 40B fast: 0B > > TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.99031 3995472KB > refills: 2948 waste 0.0% gc: 0B slow: 125952B fast: 0B > > TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 152KB slow > allocs: 0 refill waste: 2440B alloc: 0.00189 7643KB refills: 3 > waste 27.9% gc: 130944B slow: 80B fast: 0B > > TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 247KB slow > allocs: 0 refill waste: 3952B alloc: 0.00306 12355KB refills: 1 > waste 15.1% gc: 38112B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 10KB slow > allocs: 0 refill waste: 168B alloc: 0.00013 526KB refills: 2 > waste 49.9% gc: 10728B slow: 24B fast: 0B > > TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 12KB slow > allocs: 0 refill waste: 200B alloc: 0.00016 635KB refills: 2 > waste 4.7% gc: 1200B slow: 32B fast: 0B > > TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 332KB slow > allocs: 0 refill waste: 5312B alloc: 0.00411 16602KB refills: 3 > waste 26.0% gc: 264824B slow: 704B fast: 0B > > TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 2KB slow > allocs: 66 refill waste: 40B alloc: 0.00004 145KB refills: 67 > waste 2.5% gc: 2048B slow: 3016B fast: 0B > > TLAB: gc thread: 0x00007f170d35a000 [id: 85] desired_size: 24KB slow > allocs: 0 refill waste: 392B alloc: 0.00031 1231KB refills: 3 > waste 29.0% gc: 21808B slow: 112B fast: 0B > > TLAB totals: thrds: 14 refills: 3068 max: 2948 slow allocs: 85 max 66 > waste: 0.0% gc: 1221824B max: 701328B slow: 131176B max: 125952B > fast: 0B max: 0B > > > Desired survivor size 268435456 bytes, new threshold 15 (max 15) > > - age 1: 18656 bytes, 18656 total > > - age 2: 262056 bytes, 280712 total > > - age 3: 388232 bytes, 668944 total > > - age 4: 112808 bytes, 781752 total > > - age 5: 14208 bytes, 795960 total > > - age 6: 41328 bytes, 837288 total > > - age 7: 689344 bytes, 1526632 total > > - age 8: 730928 bytes, 2257560 total > > - age 9: 142800480 bytes, 145058040 total > > 68085.964: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 5972, predicted base time: 12.63 ms, remaining time: > 137.37 ms, target pause time: 150.00 ms] > > 68085.964: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 985 regions, survivors: 39 regions, predicted young region > time: 108.23 ms] > > 68085.964: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 985 regions, survivors: 39 regions, old: 0 regions, predicted > pause time: 120.86 ms, target pause time: 150.00 ms] > > 2017-01-25T14:10:41.620+0530: 68086.072: [SoftReference, 0 refs, > 0.0012167 secs]2017-01-25T14:10:41.621+0530: 68086.074: > [WeakReference, 0 refs, 0.0009153 secs]2017-01-25T14:10:41.622+0530: > 68086.075: [FinalReference, 4 refs, 0.0006707 > secs]2017-01-25T14:10:41.623+0530: 68086.075: [PhantomReference, 0 > refs, 0 refs, 0.0014168 secs]2017-01-25T14:10:41.624+0530: 68086.077: > [JNI Weak Reference, 0.0000340 secs], 0.1154024 secs] > > [Parallel Time: 107.2 ms, GC Workers: 20] > > [GC Worker Start (ms): Min: 68085964.2, Avg: 68085964.4, Max: > 68085964.9, Diff: 0.8] > > [Ext Root Scanning (ms): Min: 0.0, Avg: 0.4, Max: 1.0, Diff: > 1.0, Sum: 8.8] > > [Update RS (ms): Min: 0.0, Avg: 2.2, Max: 13.5, Diff: 13.5, Sum: > 44.1] > > [Processed Buffers: Min: 0, Avg: 1.8, Max: 6, Diff: 6, Sum: 35] > > [Scan RS (ms): Min: 0.1, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.9] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > > [Object Copy (ms): Min: 83.6, Avg: 95.1, Max: 100.9, Diff: 17.3, > Sum: 1902.7] > > [Termination (ms): Min: 4.8, Avg: 8.6, Max: 9.4, Diff: 4.7, Sum: > 171.2] > > [Termination Attempts: Min: 2131, Avg: 3342.5, Max: 3658, > Diff: 1527, Sum: 66850] > > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, > Sum: 2.8] > > [GC Worker Total (ms): Min: 106.1, Avg: 106.6, Max: 107.0, Diff: > 0.9, Sum: 2132.6] > > [GC Worker End (ms): Min: 68086071.0, Avg: 68086071.1, Max: > 68086071.2, Diff: 0.2] > > [Code Root Fixup: 0.2 ms] > > [Code Root Purge: 0.0 ms] > > [Clear CT: 0.5 ms] > > [Other: 7.5 ms] > > [Choose CSet: 0.0 ms] > > [Ref Proc: 5.1 ms] > > [Ref Enq: 0.4 ms] > > [Redirty Cards: 0.3 ms] > > [Humongous Register: 0.0 ms] > > [Humongous Reclaim: 0.0 ms] > > [Free CSet: 1.4 ms] > > [Eden: 3940.0M(3940.0M)->0.0B(3924.0M) Survivors: 156.0M->172.0M > Heap: 5164.0M(8192.0M)->1240.0M(8192.0M)] > > [Times: user=4.39 sys=0.00, real=0.11 secs] > > 2017-01-25T14:10:41.627+0530: 68086.079: Total time for which > application threads were stopped: 0.1158356 seconds, Stopping threads > took: 0.0000512 seconds > > 2017-01-25T14:10:41.627+0530: 68086.080: Application time: 0.0000789 > seconds > > 2017-01-25T14:10:41.627+0530: 68086.080: Total time for which > application threads were stopped: 0.0003353 seconds, Stopping threads > took: 0.0000796 seconds > > 2017-01-25T14:10:42.048+0530: 68086.501: Application time: 0.4211560 > seconds > > 2017-01-25T14:10:42.049+0530: 68086.501: Total time for which > application threads were stopped: 0.0003771 seconds, Stopping threads > took: 0.0000731 seconds > > 2017-01-25T14:10:43.046+0530: 68087.499: Application time: 0.9971396 > seconds > > 2017-01-25T14:10:43.046+0530: 68087.499: [GC pause (G1 Evacuation > Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] > desired_size: 2KB slow allocs: 7 refill waste: 40B alloc: 0.00003 > 136KB refills: 18 waste 4.1% gc: 1520B slow: 520B fast: 0B > > TLAB: gc thread: 0x00007f15f8114800 [id: 199] desired_size: 697KB slow > allocs: 1 refill waste: 11152B alloc: 0.00868 34866KB refills: 4 > waste 23.9% gc: 682392B slow: 120B fast: 0B > > TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 5KB refills: 1 > waste 14.0% gc: 288B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f15906a1800 [id: 140] desired_size: 68KB slow > allocs: 0 refill waste: 1096B alloc: 0.00085 3426KB refills: 3 > waste 32.5% gc: 68328B slow: 96B fast: 0B > > TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 60KB slow > allocs: 0 refill waste: 968B alloc: 0.00075 3030KB refills: 1 > waste 8.7% gc: 5400B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 20KB slow > allocs: 0 refill waste: 328B alloc: 0.00026 1031KB refills: 2 > waste 17.8% gc: 7472B slow: 40B fast: 0B > > TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 30KB slow > allocs: 0 refill waste: 488B alloc: 0.00038 1529KB refills: 3 > waste 24.6% gc: 22952B slow: 104B fast: 0B > > TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 19KB slow > allocs: 0 refill waste: 312B alloc: 0.00025 995KB refills: 5 > waste 5.9% gc: 5896B slow: 144B fast: 0B > > TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 2KB slow > allocs: 3 refill waste: 32B alloc: 0.00002 61KB refills: 6 > waste 2.5% gc: 128B slow: 176B fast: 0B > > TLAB: gc thread: 0x00007f159062b000 [id: 115] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 2KB refills: 1 > waste 92.2% gc: 1904B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.99345 3991844KB > refills: 2929 waste 0.0% gc: 0B slow: 122976B fast: 0B > > TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 100KB slow > allocs: 0 refill waste: 1600B alloc: 0.00124 5002KB refills: 11 > waste 5.6% gc: 62288B slow: 1096B fast: 0B > > TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 159KB slow > allocs: 0 refill waste: 2552B alloc: 0.00199 7998KB refills: 6 > waste 7.3% gc: 70968B slow: 1264B fast: 0B > > TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 6KB slow > allocs: 5 refill waste: 104B alloc: 0.00009 344KB refills: 8 > waste 5.4% gc: 2712B slow: 360B fast: 0B > > TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 8KB slow > allocs: 1 refill waste: 128B alloc: 0.00010 412KB refills: 7 > waste 7.8% gc: 4352B slow: 224B fast: 0B > > TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 218KB slow > allocs: 0 refill waste: 3488B alloc: 0.00271 10904KB refills: 9 > waste 6.5% gc: 129144B slow: 2200B fast: 0B > > TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow > allocs: 39 refill waste: 48B alloc: 0.00004 169KB refills: 32 > waste 2.2% gc: 1080B slow: 1336B fast: 0B > > TLAB totals: thrds: 17 refills: 3046 max: 2929 slow allocs: 56 max 39 > waste: 0.0% gc: 1066824B max: 682392B slow: 130656B max: 122976B > fast: 0B max: 0B > > > Desired survivor size 268435456 bytes, new threshold 15 (max 15) > > - age 1: 181488 bytes, 181488 total > > - age 2: 14864 bytes, 196352 total > > - age 3: 127184 bytes, 323536 total > > - age 4: 372256 bytes, 695792 total > > - age 5: 110328 bytes, 806120 total > > - age 6: 14208 bytes, 820328 total > > - age 7: 41328 bytes, 861656 total > > - age 8: 689344 bytes, 1551000 total > > - age 9: 730928 bytes, 2281928 total > > - age 10: 142800480 bytes, 145082408 total > > 68087.499: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 5519, predicted base time: 12.30 ms, remaining time: > 137.70 ms, target pause time: 150.00 ms] > > 68087.499: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 981 regions, survivors: 43 regions, predicted young region > time: 105.17 ms] > > 68087.499: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 981 regions, survivors: 43 regions, old: 0 regions, predicted > pause time: 117.47 ms, target pause time: 150.00 ms] > > 2017-01-25T14:10:43.140+0530: 68087.593: [SoftReference, 0 refs, > 0.0011365 secs]2017-01-25T14:10:43.141+0530: 68087.594: > [WeakReference, 0 refs, 0.0008092 secs]2017-01-25T14:10:43.142+0530: > 68087.595: [FinalReference, 11 refs, 0.0006691 > secs]2017-01-25T14:10:43.143+0530: 68087.596: [PhantomReference, 0 > refs, 0 refs, 0.0012762 secs]2017-01-25T14:10:43.144+0530: 68087.597: > [JNI Weak Reference, 0.0000161 secs], 0.1008873 secs] > > [Parallel Time: 93.5 ms, GC Workers: 20] > > [GC Worker Start (ms): Min: 68087499.1, Avg: 68087500.4, Max: > 68087509.3, Diff: 10.2] > > [Ext Root Scanning (ms): Min: 0.0, Avg: 0.3, Max: 1.2, Diff: > 1.2, Sum: 6.6] > > [Update RS (ms): Min: 0.0, Avg: 0.2, Max: 0.4, Diff: 0.4, Sum: 4.2] > > [Processed Buffers: Min: 0, Avg: 1.9, Max: 6, Diff: 6, Sum: 38] > > [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 1.8] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > > [Object Copy (ms): Min: 75.8, Avg: 84.6, Max: 88.8, Diff: 12.9, > Sum: 1692.3] > > [Termination (ms): Min: 3.4, Avg: 6.6, Max: 7.9, Diff: 4.5, Sum: > 131.6] > > [Termination Attempts: Min: 1502, Avg: 2767.8, Max: 3171, > Diff: 1669, Sum: 55355] > > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, > Sum: 1.8] > > [GC Worker Total (ms): Min: 83.0, Avg: 91.9, Max: 93.2, Diff: > 10.1, Sum: 1838.3] > > [GC Worker End (ms): Min: 68087592.2, Avg: 68087592.3, Max: > 68087592.4, Diff: 0.2] > > [Code Root Fixup: 0.0 ms] > > [Code Root Purge: 0.0 ms] > > [Clear CT: 0.7 ms] > > [Other: 6.7 ms] > > [Choose CSet: 0.0 ms] > > [Ref Proc: 4.3 ms] > > [Ref Enq: 0.3 ms] > > [Redirty Cards: 0.3 ms] > > [Humongous Register: 0.0 ms] > > [Humongous Reclaim: 0.0 ms] > > [Free CSet: 1.4 ms] > > [Eden: 3924.0M(3924.0M)->0.0B(3916.0M) Survivors: 172.0M->180.0M > Heap: 5164.0M(8192.0M)->1248.0M(8192.0M)] > > [Times: user=3.45 sys=0.00, real=0.10 secs] > > 2017-01-25T14:10:43.147+0530: 68087.600: Total time for which > application threads were stopped: 0.1013146 seconds, Stopping threads > took: 0.0000407 seconds > > 2017-01-25T14:10:43.147+0530: 68087.600: Application time: 0.0001099 > seconds > > 2017-01-25T14:10:43.148+0530: 68087.600: Total time for which > application threads were stopped: 0.0002593 seconds, Stopping threads > took: 0.0000683 seconds > > 2017-01-25T14:10:43.148+0530: 68087.600: Application time: 0.0000577 > seconds > > 2017-01-25T14:10:43.148+0530: 68087.600: Total time for which > application threads were stopped: 0.0000925 seconds, Stopping threads > took: 0.0000199 seconds > > 2017-01-25T14:10:43.148+0530: 68087.600: Application time: 0.0000648 > seconds > > 2017-01-25T14:10:43.148+0530: 68087.601: Total time for which > application threads were stopped: 0.0002435 seconds, Stopping threads > took: 0.0000844 seconds > > 2017-01-25T14:10:44.148+0530: 68088.601: Application time: 0.9998199 > seconds > > 2017-01-25T14:10:44.148+0530: 68088.601: Total time for which > application threads were stopped: 0.0003774 seconds, Stopping threads > took: 0.0000807 seconds > > 2017-01-25T14:10:44.578+0530: 68089.031: Application time: 0.4299646 > seconds > > 2017-01-25T14:10:44.579+0530: 68089.031: [GC pause (G1 Evacuation > Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] > desired_size: 2KB slow allocs: 23 refill waste: 32B alloc: 0.00003 > 106KB refills: 43 waste 2.6% gc: 936B slow: 1480B fast: 0B > > TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00000 4KB refills: 2 > waste 16.1% gc: 648B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 13KB slow > allocs: 1 refill waste: 208B alloc: 0.00017 671KB refills: 6 > waste 3.5% gc: 2696B slow: 176B fast: 0B > > TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 2KB slow > allocs: 3 refill waste: 32B alloc: 0.00001 44KB refills: 6 > waste 2.5% gc: 128B slow: 176B fast: 0B > > TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.99518 3990647KB > refills: 2928 waste 0.0% gc: 0B slow: 124472B fast: 0B > > TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 70KB slow > allocs: 0 refill waste: 1120B alloc: 0.00087 3500KB refills: 2 > waste 13.7% gc: 19624B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 110KB slow > allocs: 0 refill waste: 1768B alloc: 0.00138 5541KB refills: 3 > waste 18.9% gc: 62544B slow: 1888B fast: 0B > > TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 4KB slow > allocs: 1 refill waste: 72B alloc: 0.00006 237KB refills: 2 > waste 13.1% gc: 1240B slow: 32B fast: 0B > > TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 5KB slow > allocs: 2 refill waste: 88B alloc: 0.00007 289KB refills: 2 > waste 49.9% gc: 5840B slow: 48B fast: 0B > > TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 152KB slow > allocs: 4 refill waste: 2440B alloc: 0.00190 7637KB refills: 10 > waste 7.7% gc: 119664B slow: 744B fast: 0B > > TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow > allocs: 52 refill waste: 48B alloc: 0.00004 150KB refills: 68 > waste 1.6% gc: 544B slow: 2896B fast: 0B > > TLAB totals: thrds: 11 refills: 3072 max: 2928 slow allocs: 87 max 52 > waste: 0.0% gc: 213864B max: 119664B slow: 131944B max: 124472B fast: > 0B max: 0B > > > Desired survivor size 268435456 bytes, new threshold 15 (max 15) > > - age 1: 223936 bytes, 223936 total > > - age 2: 54448 bytes, 278384 total > > - age 3: 14720 bytes, 293104 total > > - age 4: 114344 bytes, 407448 total > > - age 5: 369856 bytes, 777304 total > > - age 6: 110328 bytes, 887632 total > > - age 7: 14208 bytes, 901840 total > > - age 8: 41328 bytes, 943168 total > > - age 9: 689344 bytes, 1632512 total > > - age 10: 730928 bytes, 2363440 total > > - age 11: 142800480 bytes, 145163920 total > > 68089.031: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 5691, predicted base time: 11.38 ms, remaining time: > 138.62 ms, target pause time: 150.00 ms] > > 68089.031: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 979 regions, survivors: 45 regions, predicted young region > time: 97.24 ms] > > 68089.031: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 979 regions, survivors: 45 regions, old: 0 regions, predicted > pause time: 108.62 ms, target pause time: 150.00 ms] > > 2017-01-25T14:10:44.672+0530: 68089.125: [SoftReference, 0 refs, > 0.0010475 secs]2017-01-25T14:10:44.673+0530: 68089.126: > [WeakReference, 0 refs, 0.0007920 secs]2017-01-25T14:10:44.674+0530: > 68089.127: [FinalReference, 0 refs, 0.0005465 > secs]2017-01-25T14:10:44.675+0530: 68089.127: [PhantomReference, 0 > refs, 0 refs, 0.0014752 secs]2017-01-25T14:10:44.676+0530: 68089.129: > [JNI Weak Reference, 0.0000168 secs], 0.1004490 secs] > > [Parallel Time: 93.2 ms, GC Workers: 20] > > [GC Worker Start (ms): Min: 68089031.4, Avg: 68089033.8, Max: > 68089046.3, Diff: 15.0] > > [Ext Root Scanning (ms): Min: 0.0, Avg: 0.3, Max: 1.1, Diff: > 1.0, Sum: 6.0] > > [Update RS (ms): Min: 0.0, Avg: 0.2, Max: 0.4, Diff: 0.4, Sum: 3.8] > > [Processed Buffers: Min: 0, Avg: 1.6, Max: 7, Diff: 7, Sum: 33] > > [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 1.8] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > > [Object Copy (ms): Min: 68.9, Avg: 82.3, Max: 85.7, Diff: 16.8, > Sum: 1645.3] > > [Termination (ms): Min: 3.9, Avg: 7.5, Max: 8.8, Diff: 4.9, Sum: > 149.8] > > [Termination Attempts: Min: 1780, Avg: 2841.3, Max: 3167, > Diff: 1387, Sum: 56826] > > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, > Sum: 2.5] > > [GC Worker Total (ms): Min: 77.8, Avg: 90.5, Max: 92.8, Diff: > 15.0, Sum: 1809.3] > > [GC Worker End (ms): Min: 68089124.1, Avg: 68089124.2, Max: > 68089124.4, Diff: 0.2] > > [Code Root Fixup: 0.0 ms] > > [Code Root Purge: 0.0 ms] > > [Clear CT: 0.5 ms] > > [Other: 6.8 ms] > > [Choose CSet: 0.0 ms] > > [Ref Proc: 4.3 ms] > > [Ref Enq: 0.4 ms] > > [Redirty Cards: 0.3 ms] > > [Humongous Register: 0.1 ms] > > [Humongous Reclaim: 0.0 ms] > > [Free CSet: 1.4 ms] > > [Eden: 3916.0M(3916.0M)->0.0B(3948.0M) Survivors: 180.0M->148.0M > Heap: 5164.0M(8192.0M)->1218.0M(8192.0M)] > > [Times: user=3.48 sys=0.00, real=0.10 secs] > > 2017-01-25T14:10:44.679+0530: 68089.132: Total time for which > application threads were stopped: 0.1008638 seconds, Stopping threads > took: 0.0000409 seconds > > 2017-01-25T14:10:45.148+0530: 68089.601: Application time: 0.4690297 > seconds > > 2017-01-25T14:10:45.149+0530: 68089.601: Total time for which > application threads were stopped: 0.0003481 seconds, Stopping threads > took: 0.0000565 seconds > > 2017-01-25T14:10:45.149+0530: 68089.601: Application time: 0.0002611 > seconds > > 2017-01-25T14:10:45.149+0530: 68089.602: Total time for which > application threads were stopped: 0.0004220 seconds, Stopping threads > took: 0.0000415 seconds > > 2017-01-25T14:10:45.963+0530: 68090.415: Application time: 0.8134945 > seconds > > 2017-01-25T14:10:45.963+0530: 68090.416: Total time for which > application threads were stopped: 0.0002944 seconds, Stopping threads > took: 0.0000764 seconds > > 2017-01-25T14:10:46.149+0530: 68090.601: Application time: 0.1857148 > seconds > > 2017-01-25T14:10:46.149+0530: 68090.602: Total time for which > application threads were stopped: 0.0003960 seconds, Stopping threads > took: 0.0000857 seconds > > 2017-01-25T14:10:46.161+0530: 68090.613: Application time: 0.0115789 > seconds > > 2017-01-25T14:10:46.161+0530: 68090.614: [GC pause (G1 Evacuation > Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] > desired_size: 2KB slow allocs: 25 refill waste: 32B alloc: 0.00003 > 106KB refills: 43 waste 2.5% gc: 928B slow: 1352B fast: 0B > > TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00000 4KB refills: 2 > waste 16.1% gc: 648B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 13KB slow > allocs: 0 refill waste: 216B alloc: 0.00017 682KB refills: 1 > waste 36.1% gc: 5040B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00001 33KB refills: 1 > waste 24.8% gc: 512B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.99667 4029298KB > refills: 2956 waste 0.0% gc: 0B slow: 124400B fast: 0B > > TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 3KB slow > allocs: 2 refill waste: 56B alloc: 0.00005 193KB refills: 3 > waste 29.1% gc: 3352B slow: 88B fast: 0B > > TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 108KB slow > allocs: 0 refill waste: 1736B alloc: 0.00134 5434KB refills: 2 > waste 41.4% gc: 91848B slow: 328B fast: 0B > > TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow > allocs: 43 refill waste: 56B alloc: 0.00004 177KB refills: 54 > waste 3.0% gc: 3568B slow: 2328B fast: 0B > > TLAB: gc thread: 0x00007f170d35a000 [id: 85] desired_size: 16KB slow > allocs: 0 refill waste: 256B alloc: 0.00020 811KB refills: 3 > waste 11.3% gc: 5552B slow: 64B fast: 0B > > TLAB totals: thrds: 9 refills: 3065 max: 2956 slow allocs: 71 max 43 > waste: 0.0% gc: 111448B max: 91848B slow: 128576B max: 124400B fast: > 0B max: 0B > > > Desired survivor size 268435456 bytes, new threshold 15 (max 15) > > - age 1: 58040 bytes, 58040 total > > - age 2: 216064 bytes, 274104 total > > - age 3: 54304 bytes, 328408 total > > - age 4: 14720 bytes, 343128 total > > - age 5: 114344 bytes, 457472 total > > - age 6: 369856 bytes, 827328 total > > - age 7: 110328 bytes, 937656 total > > - age 8: 14208 bytes, 951864 total > > - age 9: 41328 bytes, 993192 total > > - age 10: 689344 bytes, 1682536 total > > - age 11: 730928 bytes, 2413464 total > > - age 12: 142800480 bytes, 145213944 total > > 68090.614: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 5688, predicted base time: 10.90 ms, remaining time: > 139.10 ms, target pause time: 150.00 ms] > > 68090.614: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 987 regions, survivors: 37 regions, predicted young region > time: 92.86 ms] > > 68090.614: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 987 regions, survivors: 37 regions, old: 0 regions, predicted > pause time: 103.76 ms, target pause time: 150.00 ms] > > 2017-01-25T14:10:46.255+0530: 68090.707: [SoftReference, 0 refs, > 0.0009716 secs]2017-01-25T14:10:46.256+0530: 68090.708: > [WeakReference, 0 refs, 0.0005869 secs]2017-01-25T14:10:46.256+0530: > 68090.709: [FinalReference, 0 refs, 0.0006482 > secs]2017-01-25T14:10:46.257+0530: 68090.710: [PhantomReference, 0 > refs, 0 refs, 0.0014339 secs]2017-01-25T14:10:46.258+0530: 68090.711: > [JNI Weak Reference, 0.0000166 secs], 0.1000319 secs] > > [Parallel Time: 93.2 ms, GC Workers: 20] > > [GC Worker Start (ms): Min: 68090613.7, Avg: 68090615.8, Max: > 68090626.3, Diff: 12.6] > > [Ext Root Scanning (ms): Min: 0.0, Avg: 0.5, Max: 1.1, Diff: > 1.1, Sum: 9.1] > > [Update RS (ms): Min: 0.0, Avg: 0.2, Max: 0.4, Diff: 0.4, Sum: 3.8] > > [Processed Buffers: Min: 0, Avg: 1.6, Max: 5, Diff: 5, Sum: 32] > > [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.2, Sum: 2.8] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > > [Object Copy (ms): Min: 72.9, Avg: 83.4, Max: 88.4, Diff: 15.5, > Sum: 1667.3] > > [Termination (ms): Min: 3.2, Avg: 6.6, Max: 7.2, Diff: 4.0, Sum: > 131.4] > > [Termination Attempts: Min: 1661, Avg: 2894.8, Max: 3138, > Diff: 1477, Sum: 57895] > > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, > Sum: 2.6] > > [GC Worker Total (ms): Min: 80.2, Avg: 90.9, Max: 93.0, Diff: > 12.7, Sum: 1817.0] > > [GC Worker End (ms): Min: 68090706.5, Avg: 68090706.6, Max: > 68090706.7, Diff: 0.2] > > [Code Root Fixup: 0.0 ms] > > [Code Root Purge: 0.0 ms] > > [Clear CT: 0.5 ms] > > [Other: 6.3 ms] > > [Choose CSet: 0.0 ms] > > [Ref Proc: 4.0 ms] > > [Ref Enq: 0.4 ms] > > [Redirty Cards: 0.3 ms] > > [Humongous Register: 0.0 ms] > > [Humongous Reclaim: 0.0 ms] > > [Free CSet: 1.4 ms] > > [Eden: 3948.0M(3948.0M)->0.0B(3940.0M) Survivors: 148.0M->156.0M > Heap: 5166.0M(8192.0M)->1224.0M(8192.0M)] > > [Times: user=3.58 sys=0.00, real=0.10 secs] > > 2017-01-25T14:10:46.261+0530: 68090.714: Total time for which > application threads were stopped: 0.1004085 seconds, Stopping threads > took: 0.0000355 seconds > > 2017-01-25T14:10:47.149+0530: 68091.601: Application time: 0.8877422 > seconds > > 2017-01-25T14:10:47.149+0530: 68091.602: Total time for which > application threads were stopped: 0.0003494 seconds, Stopping threads > took: 0.0000623 seconds > > 2017-01-25T14:10:47.675+0530: 68092.127: Application time: 0.5255223 > seconds > > 2017-01-25T14:10:47.675+0530: 68092.128: [GC pause (G1 Evacuation > Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] > desired_size: 2KB slow allocs: 13 refill waste: 32B alloc: 0.00003 > 104KB refills: 22 waste 4.1% gc: 1000B slow: 912B fast: 0B > > TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 4KB refills: 1 > waste 14.0% gc: 288B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f15906a1800 [id: 140] desired_size: 45KB slow > allocs: 0 refill waste: 720B alloc: 0.00056 2261KB refills: 1 > waste 50.8% gc: 23520B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 8KB slow > allocs: 3 refill waste: 136B alloc: 0.00011 442KB refills: 17 > waste 5.7% gc: 8328B slow: 488B fast: 0B > > TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 20KB slow > allocs: 0 refill waste: 320B alloc: 0.00025 1016KB refills: 1 > waste 21.3% gc: 4432B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00001 22KB refills: 4 > waste 23.6% gc: 1864B slow: 88B fast: 0B > > TLAB: gc thread: 0x00007f159005b000 [id: 114] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 1KB refills: 1 > waste 98.4% gc: 2032B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.99777 4025582KB > refills: 2945 waste 0.0% gc: 0B slow: 127672B fast: 0B > > TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 46KB slow > allocs: 0 refill waste: 744B alloc: 0.00058 2328KB refills: 7 > waste 10.3% gc: 34368B slow: 160B fast: 0B > > TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 73KB slow > allocs: 2 refill waste: 1176B alloc: 0.00092 3695KB refills: 8 > waste 8.4% gc: 50640B slow: 224B fast: 0B > > TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 3KB slow > allocs: 4 refill waste: 48B alloc: 0.00004 159KB refills: 7 > waste 14.4% gc: 3104B slow: 152B fast: 0B > > TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 2KB slow > allocs: 9 refill waste: 40B alloc: 0.00003 127KB refills: 10 > waste 5.4% gc: 984B slow: 416B fast: 0B > > TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 71KB slow > allocs: 9 refill waste: 1144B alloc: 0.00089 3575KB refills: 13 > waste 5.5% gc: 51144B slow: 1664B fast: 0B > > TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow > allocs: 16 refill waste: 56B alloc: 0.00005 185KB refills: 28 > waste 4.1% gc: 2944B slow: 1448B fast: 0B > > TLAB totals: thrds: 14 refills: 3065 max: 2945 slow allocs: 56 max 16 > waste: 0.0% gc: 184648B max: 51144B slow: 133224B max: 127672B fast: > 0B max: 0B > > > Desired survivor size 268435456 bytes, new threshold 15 (max 15) > > - age 1: 19576 bytes, 19576 total > > - age 2: 52640 bytes, 72216 total > > - age 3: 215144 bytes, 287360 total > > - age 4: 54240 bytes, 341600 total > > - age 5: 14720 bytes, 356320 total > > - age 6: 114344 bytes, 470664 total > > - age 7: 369856 bytes, 840520 total > > - age 8: 110328 bytes, 950848 total > > - age 9: 14208 bytes, 965056 total > > - age 10: 41328 bytes, 1006384 total > > - age 11: 689344 bytes, 1695728 total > > - age 12: 730928 bytes, 2426656 total > > - age 13: 142800480 bytes, 145227136 total > > 68092.128: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 5988, predicted base time: 10.41 ms, remaining time: > 139.59 ms, target pause time: 150.00 ms] > > 68092.128: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 985 regions, survivors: 39 regions, predicted young region > time: 91.70 ms] > > 68092.128: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 985 regions, survivors: 39 regions, old: 0 regions, predicted > pause time: 102.11 ms, target pause time: 150.00 ms] > > 2017-01-25T14:10:47.753+0530: 68092.205: [SoftReference, 0 refs, > 0.0011755 secs]2017-01-25T14:10:47.754+0530: 68092.207: > [WeakReference, 0 refs, 0.0008223 secs]2017-01-25T14:10:47.755+0530: > 68092.207: [FinalReference, 0 refs, 0.0006542 > secs]2017-01-25T14:10:47.756+0530: 68092.208: [PhantomReference, 0 > refs, 0 refs, 0.0014601 secs]2017-01-25T14:10:47.757+0530: 68092.210: > [JNI Weak Reference, 0.0000167 secs], 0.0847489 secs] > > [Parallel Time: 77.3 ms, GC Workers: 20] > > [GC Worker Start (ms): Min: 68092127.8, Avg: 68092130.2, Max: > 68092142.4, Diff: 14.5] > > [Ext Root Scanning (ms): Min: 0.0, Avg: 0.4, Max: 1.4, Diff: > 1.4, Sum: 7.6] > > [Update RS (ms): Min: 0.0, Avg: 0.2, Max: 0.4, Diff: 0.4, Sum: 4.0] > > [Processed Buffers: Min: 0, Avg: 1.8, Max: 7, Diff: 7, Sum: 36] > > [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 2.0] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > > [Object Copy (ms): Min: 57.4, Avg: 69.4, Max: 73.9, Diff: 16.6, > Sum: 1387.9] > > [Termination (ms): Min: 1.8, Avg: 4.5, Max: 5.2, Diff: 3.4, Sum: > 90.5] > > [Termination Attempts: Min: 1157, Avg: 2821.8, Max: 3156, > Diff: 1999, Sum: 56436] > > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.2, > Sum: 3.0] > > [GC Worker Total (ms): Min: 62.5, Avg: 74.7, Max: 77.2, Diff: > 14.6, Sum: 1494.9] > > [GC Worker End (ms): Min: 68092204.8, Avg: 68092204.9, Max: > 68092205.0, Diff: 0.2] > > [Code Root Fixup: 0.0 ms] > > [Code Root Purge: 0.0 ms] > > [Clear CT: 0.5 ms] > > [Other: 6.8 ms] > > [Choose CSet: 0.0 ms] > > [Ref Proc: 4.5 ms] > > [Ref Enq: 0.4 ms] > > [Redirty Cards: 0.3 ms] > > [Humongous Register: 0.0 ms] > > [Humongous Reclaim: 0.0 ms] > > [Free CSet: 1.4 ms] > > [Eden: 3940.0M(3940.0M)->0.0B(3944.0M) Survivors: 156.0M->152.0M > Heap: 5164.0M(8192.0M)->1222.0M(8192.0M)] > > [Times: user=0.00 sys=0.00, real=0.09 secs] > > 2017-01-25T14:10:47.760+0530: 68092.212: Total time for which > application threads were stopped: 0.0851593 seconds, Stopping threads > took: 0.0000402 seconds > > 2017-01-25T14:10:48.149+0530: 68092.602: Application time: 0.3890779 > seconds > > 2017-01-25T14:10:48.149+0530: 68092.602: Total time for which > application threads were stopped: 0.0003716 seconds, Stopping threads > took: 0.0000733 seconds > > 2017-01-25T14:10:49.149+0530: 68093.602: Application time: 0.9998228 > seconds > > 2017-01-25T14:10:49.150+0530: 68093.602: Total time for which > application threads were stopped: 0.0003580 seconds, Stopping threads > took: 0.0000655 seconds > > 2017-01-25T14:10:49.211+0530: 68093.663: Application time: 0.0611931 > seconds > > 2017-01-25T14:10:49.211+0530: 68093.664: [GC pause (G1 Evacuation > Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] > desired_size: 2KB slow allocs: 26 refill waste: 32B alloc: 0.00002 > 86KB refills: 43 waste 2.8% gc: 1064B slow: 1408B fast: 0B > > TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00000 3KB refills: 2 > waste 16.1% gc: 648B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f15906a1800 [id: 140] desired_size: 29KB slow > allocs: 0 refill waste: 464B alloc: 0.00036 1471KB refills: 1 > waste 1.9% gc: 560B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 39KB slow > allocs: 0 refill waste: 632B alloc: 0.00049 1980KB refills: 2 > waste 5.0% gc: 3992B slow: 24B fast: 0B > > TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 6KB slow > allocs: 3 refill waste: 104B alloc: 0.00008 342KB refills: 21 > waste 2.5% gc: 3080B slow: 624B fast: 0B > > TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 9KB slow > allocs: 6 refill waste: 144B alloc: 0.00012 469KB refills: 60 > waste 0.5% gc: 1208B slow: 1952B fast: 0B > > TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 17KB refills: 5 > waste 9.2% gc: 840B slow: 112B fast: 0B > > TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.99834 4031960KB > refills: 2936 waste 0.0% gc: 0B slow: 127024B fast: 0B > > TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 32KB slow > allocs: 0 refill waste: 512B alloc: 0.00040 1612KB refills: 29 > waste 3.3% gc: 29896B slow: 1424B fast: 0B > > TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 50KB slow > allocs: 13 refill waste: 808B alloc: 0.00063 2539KB refills: 34 > waste 2.8% gc: 47744B slow: 2296B fast: 0B > > TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 2KB slow > allocs: 9 refill waste: 32B alloc: 0.00003 110KB refills: 24 > waste 4.9% gc: 1936B slow: 728B fast: 0B > > TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 2KB slow > allocs: 8 refill waste: 32B alloc: 0.00002 92KB refills: 19 > waste 6.6% gc: 2000B slow: 584B fast: 0B > > TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 53KB slow > allocs: 33 refill waste: 856B alloc: 0.00067 2686KB refills: 47 > waste 0.7% gc: 12280B slow: 5760B fast: 0B > > TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow > allocs: 47 refill waste: 48B alloc: 0.00004 157KB refills: 66 > waste 1.3% gc: 120B slow: 2736B fast: 0B > > TLAB totals: thrds: 14 refills: 3289 max: 2936 slow allocs: 146 max > 47 waste: 0.0% gc: 105368B max: 47744B slow: 144688B max: 127024B > fast: 0B max: 0B > > > Desired survivor size 268435456 bytes, new threshold 15 (max 15) > > - age 1: 105064 bytes, 105064 total > > - age 2: 10968 bytes, 116032 total > > - age 3: 52184 bytes, 168216 total > > - age 4: 207256 bytes, 375472 total > > - age 5: 51760 bytes, 427232 total > > - age 6: 14720 bytes, 441952 total > > - age 7: 114320 bytes, 556272 total > > - age 8: 369856 bytes, 926128 total > > - age 9: 110328 bytes, 1036456 total > > - age 10: 14208 bytes, 1050664 total > > - age 11: 41328 bytes, 1091992 total > > - age 12: 689344 bytes, 1781336 total > > - age 13: 730928 bytes, 2512264 total > > - age 14: 142800480 bytes, 145312744 total > > 68093.664: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 4729, predicted base time: 10.07 ms, remaining time: > 139.93 ms, target pause time: 150.00 ms] > > 68093.664: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 986 regions, survivors: 38 regions, predicted young region > time: 85.68 ms] > > 68093.664: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 986 regions, survivors: 38 regions, old: 0 regions, predicted > pause time: 95.75 ms, target pause time: 150.00 ms] > > 2017-01-25T14:10:49.311+0530: 68093.764: [SoftReference, 0 refs, > 0.0010888 secs]2017-01-25T14:10:49.312+0530: 68093.765: > [WeakReference, 0 refs, 0.0005239 secs]2017-01-25T14:10:49.313+0530: > 68093.765: [FinalReference, 0 refs, 0.0007986 > secs]2017-01-25T14:10:49.313+0530: 68093.766: [PhantomReference, 0 > refs, 0 refs, 0.0013813 secs]2017-01-25T14:10:49.315+0530: 68093.767: > [JNI Weak Reference, 0.0000169 secs], 0.1066977 secs] > > [Parallel Time: 99.4 ms, GC Workers: 20] > > [GC Worker Start (ms): Min: 68093663.9, Avg: 68093665.4, Max: > 68093670.5, Diff: 6.7] > > [Ext Root Scanning (ms): Min: 0.0, Avg: 0.3, Max: 1.1, Diff: > 1.1, Sum: 6.0] > > [Update RS (ms): Min: 0.0, Avg: 1.7, Max: 10.3, Diff: 10.3, Sum: > 33.7] > > [Processed Buffers: Min: 0, Avg: 1.6, Max: 7, Diff: 7, Sum: 31] > > [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 1.8] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > > [Object Copy (ms): Min: 80.5, Avg: 88.3, Max: 94.1, Diff: 13.6, > Sum: 1765.7] > > [Termination (ms): Min: 3.8, Avg: 7.2, Max: 13.7, Diff: 10.0, > Sum: 144.1] > > [Termination Attempts: Min: 1844, Avg: 2613.2, Max: 2898, > Diff: 1054, Sum: 52263] > > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.2, > Sum: 3.0] > > [GC Worker Total (ms): Min: 92.5, Avg: 97.7, Max: 99.3, Diff: > 6.8, Sum: 1954.3] > > [GC Worker End (ms): Min: 68093762.9, Avg: 68093763.1, Max: > 68093763.2, Diff: 0.2] > > [Code Root Fixup: 0.0 ms] > > [Code Root Purge: 0.0 ms] > > [Clear CT: 0.6 ms] > > [Other: 6.6 ms] > > [Choose CSet: 0.0 ms] > > [Ref Proc: 4.2 ms] > > [Ref Enq: 0.4 ms] > > [Redirty Cards: 0.3 ms] > > [Humongous Register: 0.1 ms] > > [Humongous Reclaim: 0.0 ms] > > [Free CSet: 1.4 ms] > > [Eden: 3944.0M(3944.0M)->0.0B(3936.0M) Survivors: 152.0M->160.0M > Heap: 5166.0M(8192.0M)->1228.0M(8192.0M)] > > [Times: user=0.00 sys=0.00, real=0.11 secs] > > 2017-01-25T14:10:49.318+0530: 68093.770: Total time for which > application threads were stopped: 0.1071290 seconds, Stopping threads > took: 0.0000416 seconds > > 2017-01-25T14:10:50.149+0530: 68094.602: Application time: 0.8314793 > seconds > > 2017-01-25T14:10:50.150+0530: 68094.602: Total time for which > application threads were stopped: 0.0003901 seconds, Stopping threads > took: 0.0000748 seconds > > 2017-01-25T14:10:50.809+0530: 68095.262: Application time: 0.6593091 > seconds > > 2017-01-25T14:10:50.809+0530: 68095.262: [GC pause (G1 Evacuation > Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] > desired_size: 2KB slow allocs: 15 refill waste: 32B alloc: 0.00002 > 92KB refills: 22 waste 4.4% gc: 1328B slow: 680B fast: 0B > > TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 3KB refills: 1 > waste 14.0% gc: 288B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f1590689800 [id: 126] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00001 58KB refills: 2 > waste 25.2% gc: 1008B slow: 32B fast: 0B > > TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.99817 4023075KB > refills: 2949 waste 0.0% gc: 0B slow: 127160B fast: 0B > > TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow > allocs: 23 refill waste: 56B alloc: 0.00004 179KB refills: 31 > waste 1.6% gc: 400B slow: 1424B fast: 0B > > TLAB totals: thrds: 5 refills: 3005 max: 2949 slow allocs: 38 max 23 > waste: 0.0% gc: 3024B max: 1328B slow: 129296B max: 127160B fast: 0B > max: 0B > > > Desired survivor size 268435456 bytes, new threshold 15 (max 15) > > - age 1: 359040 bytes, 359040 total > > - age 2: 96968 bytes, 456008 total > > - age 3: 10944 bytes, 466952 total > > - age 4: 50816 bytes, 517768 total > > - age 5: 205888 bytes, 723656 total > > - age 6: 51760 bytes, 775416 total > > - age 7: 14680 bytes, 790096 total > > - age 8: 114320 bytes, 904416 total > > - age 9: 369856 bytes, 1274272 total > > - age 10: 110328 bytes, 1384600 total > > - age 11: 14208 bytes, 1398808 total > > - age 12: 41328 bytes, 1440136 total > > - age 13: 689344 bytes, 2129480 total > > - age 14: 730904 bytes, 2860384 total > > - age 15: 142800480 bytes, 145660864 total > > 68095.262: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 6424, predicted base time: 10.60 ms, remaining time: > 139.40 ms, target pause time: 150.00 ms] > > 68095.262: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 984 regions, survivors: 40 regions, predicted young region > time: 86.48 ms] > > 68095.262: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 984 regions, survivors: 40 regions, old: 0 regions, predicted > pause time: 97.07 ms, target pause time: 150.00 ms] > > 2017-01-25T14:10:50.949+0530: 68095.401: [SoftReference, 0 refs, > 0.0012724 secs]2017-01-25T14:10:50.950+0530: 68095.403: > [WeakReference, 0 refs, 0.0006785 secs]2017-01-25T14:10:50.951+0530: > 68095.403: [FinalReference, 0 refs, 0.0008141 > secs]2017-01-25T14:10:50.952+0530: 68095.404: [PhantomReference, 0 > refs, 0 refs, 0.0012798 secs]2017-01-25T14:10:50.953+0530: 68095.406: > [JNI Weak Reference, 0.0000165 secs], 0.1466546 secs] > > [Parallel Time: 138.9 ms, GC Workers: 20] > > [GC Worker Start (ms): Min: 68095262.1, Avg: 68095263.2, Max: > 68095275.2, Diff: 13.1] > > [Ext Root Scanning (ms): Min: 0.0, Avg: 2.1, Max: 9.4, Diff: > 9.4, Sum: 42.4] > > [Update RS (ms): Min: 0.0, Avg: 0.5, Max: 6.5, Diff: 6.5, Sum: 9.9] > > [Processed Buffers: Min: 0, Avg: 1.6, Max: 6, Diff: 6, Sum: 31] > > [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 1.9] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > > [Object Copy (ms): Min: 117.9, Avg: 128.4, Max: 134.8, Diff: > 16.9, Sum: 2567.8] > > [Termination (ms): Min: 2.5, Avg: 6.1, Max: 7.6, Diff: 5.2, Sum: > 122.3] > > [Termination Attempts: Min: 1110, Avg: 1929.0, Max: 2192, > Diff: 1082, Sum: 38580] > > [GC Worker Other (ms): Min: 0.0, Avg: 0.2, Max: 0.4, Diff: 0.4, > Sum: 4.4] > > [GC Worker Total (ms): Min: 125.3, Avg: 137.4, Max: 138.6, Diff: > 13.4, Sum: 2748.7] > > [GC Worker End (ms): Min: 68095400.4, Avg: 68095400.6, Max: > 68095400.8, Diff: 0.4] > > [Code Root Fixup: 0.0 ms] > > [Code Root Purge: 0.0 ms] > > [Clear CT: 0.8 ms] > > [Other: 7.0 ms] > > [Choose CSet: 0.0 ms] > > [Ref Proc: 4.6 ms] > > [Ref Enq: 0.3 ms] > > [Redirty Cards: 0.4 ms] > > [Humongous Register: 0.1 ms] > > [Humongous Reclaim: 0.0 ms] > > [Free CSet: 1.4 ms] > > [Eden: 3936.0M(3936.0M)->0.0B(4060.0M) Survivors: 160.0M->36.0M > Heap: 5164.0M(8192.0M)->1245.2M(8192.0M)] > > [Times: user=5.47 sys=0.00, real=0.15 secs] > > 2017-01-25T14:10:50.956+0530: 68095.409: Total time for which > application threads were stopped: 0.1470861 seconds, Stopping threads > took: 0.0000389 seconds > > 2017-01-25T14:10:50.963+0530: 68095.416: Application time: 0.0071737 > seconds > > 2017-01-25T14:10:50.964+0530: 68095.416: Total time for which > application threads were stopped: 0.0002721 seconds, Stopping threads > took: 0.0000830 seconds > > 2017-01-25T14:10:51.149+0530: 68095.602: Application time: 0.1858778 > seconds > > 2017-01-25T14:10:51.150+0530: 68095.602: Total time for which > application threads were stopped: 0.0004194 seconds, Stopping threads > took: 0.0001064 seconds > > 2017-01-25T14:10:52.150+0530: 68096.602: Application time: 0.9997142 > seconds > > 2017-01-25T14:10:52.150+0530: 68096.603: Total time for which > application threads were stopped: 0.0003868 seconds, Stopping threads > took: 0.0000696 seconds > > 2017-01-25T14:10:52.467+0530: 68096.920: Application time: 0.3171898 > seconds > > 2017-01-25T14:10:52.467+0530: 68096.920: [GC pause (G1 Evacuation > Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] > desired_size: 2KB slow allocs: 28 refill waste: 32B alloc: 0.00002 > 80KB refills: 43 waste 3.1% gc: 1328B slow: 1432B fast: 0B > > TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00000 3KB refills: 2 > waste 16.1% gc: 648B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f15906a1800 [id: 140] desired_size: 19KB slow > allocs: 1 refill waste: 312B alloc: 0.00024 992KB refills: 15 > waste 6.2% gc: 18456B slow: 464B fast: 0B > > TLAB: gc thread: 0x00007f159069b000 [id: 136] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00002 75KB refills: 2 > waste 25.0% gc: 1016B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 26KB slow > allocs: 0 refill waste: 424B alloc: 0.00032 1325KB refills: 3 > waste 9.9% gc: 8008B slow: 56B fast: 0B > > TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 5KB slow > allocs: 1 refill waste: 88B alloc: 0.00007 278KB refills: 14 > waste 4.4% gc: 3032B slow: 488B fast: 0B > > TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 13KB slow > allocs: 1 refill waste: 216B alloc: 0.00016 681KB refills: 9 > waste 8.9% gc: 10896B slow: 288B fast: 0B > > TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 14KB refills: 5 > waste 8.4% gc: 760B slow: 112B fast: 0B > > TLAB: gc thread: 0x00007f159062b000 [id: 115] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 1KB refills: 1 > waste 89.9% gc: 1856B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.99878 4152376KB > refills: 3022 waste 0.0% gc: 0B slow: 128312B fast: 0B > > TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 27KB slow > allocs: 1 refill waste: 440B alloc: 0.00033 1387KB refills: 26 > waste 3.0% gc: 20128B slow: 1912B fast: 0B > > TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 49KB slow > allocs: 2 refill waste: 784B alloc: 0.00059 2463KB refills: 21 > waste 3.0% gc: 30136B slow: 1704B fast: 0B > > TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 2KB slow > allocs: 11 refill waste: 32B alloc: 0.00002 93KB refills: 23 > waste 5.4% gc: 1744B slow: 832B fast: 0B > > TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 2KB slow > allocs: 9 refill waste: 32B alloc: 0.00002 77KB refills: 19 > waste 6.2% gc: 1888B slow: 552B fast: 0B > > TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 66KB slow > allocs: 18 refill waste: 1064B alloc: 0.00080 3327KB refills: 43 > waste 0.5% gc: 6800B slow: 8912B fast: 0B > > TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow > allocs: 49 refill waste: 48B alloc: 0.00004 162KB refills: 62 > waste 2.3% gc: 2304B slow: 2432B fast: 0B > > TLAB: gc thread: 0x00007f170d35a000 [id: 85] desired_size: 10KB slow > allocs: 0 refill waste: 168B alloc: 0.00013 549KB refills: 3 > waste 12.3% gc: 4064B slow: 72B fast: 0B > > TLAB totals: thrds: 17 refills: 3313 max: 3022 slow allocs: 122 max > 49 waste: 0.0% gc: 113064B max: 30136B slow: 147600B max: 128312B > fast: 0B max: 0B > > > Desired survivor size 268435456 bytes, new threshold 15 (max 15) > > - age 1: 14432 bytes, 14432 total > > - age 2: 353520 bytes, 367952 total > > - age 3: 96968 bytes, 464920 total > > - age 4: 10944 bytes, 475864 total > > - age 5: 50816 bytes, 526680 total > > - age 6: 205888 bytes, 732568 total > > - age 7: 51760 bytes, 784328 total > > - age 8: 14680 bytes, 799008 total > > - age 9: 114320 bytes, 913328 total > > - age 10: 369856 bytes, 1283184 total > > - age 11: 110328 bytes, 1393512 total > > - age 12: 14208 bytes, 1407720 total > > - age 13: 41328 bytes, 1449048 total > > - age 14: 689344 bytes, 2138392 total > > - age 15: 730824 bytes, 2869216 total > > 68096.920: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 35114, predicted base time: 16.14 ms, remaining time: > 133.86 ms, target pause time: 150.00 ms] > > 68096.920: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 1015 regions, survivors: 9 regions, predicted young region > time: 37.34 ms] > > 68096.920: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 1015 regions, survivors: 9 regions, old: 0 regions, predicted > pause time: 53.48 ms, target pause time: 150.00 ms] > > 2017-01-25T14:10:52.481+0530: 68096.933: [SoftReference, 0 refs, > 0.0012948 secs]2017-01-25T14:10:52.482+0530: 68096.935: > [WeakReference, 0 refs, 0.0006552 secs]2017-01-25T14:10:52.483+0530: > 68096.935: [FinalReference, 0 refs, 0.0006372 > secs]2017-01-25T14:10:52.483+0530: 68096.936: [PhantomReference, 0 > refs, 0 refs, 0.0013651 secs]2017-01-25T14:10:52.485+0530: 68096.937: > [JNI Weak Reference, 0.0000170 secs], 0.0211945 secs] > > [Parallel Time: 12.7 ms, GC Workers: 20] > > [GC Worker Start (ms): Min: 68096920.3, Avg: 68096920.5, Max: > 68096920.7, Diff: 0.4] > > [Ext Root Scanning (ms): Min: 0.4, Avg: 0.6, Max: 1.0, Diff: > 0.7, Sum: 11.5] > > [Update RS (ms): Min: 3.2, Avg: 3.6, Max: 4.1, Diff: 0.9, Sum: 71.2] > > [Processed Buffers: Min: 2, Avg: 7.7, Max: 15, Diff: 13, Sum: > 153] > > [Scan RS (ms): Min: 0.0, Avg: 0.3, Max: 0.5, Diff: 0.4, Sum: 6.0] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > > [Object Copy (ms): Min: 2.3, Avg: 3.5, Max: 7.4, Diff: 5.1, Sum: > 70.3] > > [Termination (ms): Min: 0.6, Avg: 4.4, Max: 5.5, Diff: 4.9, Sum: > 88.3] > > [Termination Attempts: Min: 1, Avg: 1.9, Max: 4, Diff: 3, > Sum: 37] > > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, > Sum: 1.4] > > [GC Worker Total (ms): Min: 12.2, Avg: 12.4, Max: 12.7, Diff: > 0.4, Sum: 248.8] > > [GC Worker End (ms): Min: 68096932.8, Avg: 68096932.9, Max: > 68096933.0, Diff: 0.2] > > [Code Root Fixup: 0.0 ms] > > [Code Root Purge: 0.0 ms] > > [Clear CT: 0.5 ms] > > [Other: 7.9 ms] > > [Choose CSet: 0.0 ms] > > [Ref Proc: 4.4 ms] > > [Ref Enq: 0.4 ms] > > [Redirty Cards: 0.3 ms] > > [Humongous Register: 0.1 ms] > > [Humongous Reclaim: 0.0 ms] > > [Free CSet: 1.5 ms] > > [Eden: 4060.0M(4060.0M)->0.0B(4056.0M) Survivors: 36.0M->40.0M > Heap: 5305.2M(8192.0M)->1285.5M(8192.0M)] > > [Times: user=0.00 sys=0.00, real=0.02 secs] > > 2017-01-25T14:10:52.489+0530: 68096.941: Total time for which > application threads were stopped: 0.0216177 seconds, Stopping threads > took: 0.0000399 seconds > > 2017-01-25T14:10:53.149+0530: 68097.601: Application time: 0.6599486 > seconds > > 2017-01-25T14:10:53.149+0530: 68097.602: Total time for which > application threads were stopped: 0.0003672 seconds, Stopping threads > took: 0.0000798 seconds > > 2017-01-25T14:10:54.004+0530: 68098.457: Application time: 0.8549700 > seconds > > 2017-01-25T14:10:54.004+0530: 68098.457: [GC pause (G1 Evacuation > Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] > desired_size: 2KB slow allocs: 14 refill waste: 32B alloc: 0.00002 > 88KB refills: 22 waste 4.3% gc: 1328B slow: 632B fast: 0B > > TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 3KB refills: 1 > waste 14.0% gc: 288B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f15906a3800 [id: 141] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00002 78KB refills: 2 > waste 26.0% gc: 1056B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f15906a1800 [id: 140] desired_size: 14KB slow > allocs: 0 refill waste: 232B alloc: 0.00018 747KB refills: 2 > waste 13.7% gc: 4152B slow: 24B fast: 0B > > TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 17KB slow > allocs: 0 refill waste: 272B alloc: 0.00021 871KB refills: 5 > waste 1.2% gc: 864B slow: 176B fast: 0B > > TLAB: gc thread: 0x00007f1590692000 [id: 131] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00001 34KB refills: 2 > waste 25.2% gc: 1008B slow: 32B fast: 0B > > TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 4KB slow > allocs: 3 refill waste: 64B alloc: 0.00005 205KB refills: 37 > waste 2.7% gc: 2888B slow: 1328B fast: 0B > > TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 9KB slow > allocs: 0 refill waste: 152B alloc: 0.00012 481KB refills: 6 > waste 14.9% gc: 8584B slow: 216B fast: 0B > > TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 10KB slow > allocs: 0 refill waste: 168B alloc: 0.00013 534KB refills: 5 > waste 18.2% gc: 9840B slow: 128B fast: 0B > > TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 2KB slow > allocs: 3 refill waste: 32B alloc: 0.00000 12KB refills: 13 > waste 8.3% gc: 1944B slow: 280B fast: 0B > > TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.99872 4148023KB > refills: 3013 waste 0.0% gc: 0B slow: 126696B fast: 0B > > TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 22KB slow > allocs: 1 refill waste: 360B alloc: 0.00027 1135KB refills: 36 > waste 1.5% gc: 9552B slow: 2824B fast: 0B > > TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 38KB slow > allocs: 10 refill waste: 608B alloc: 0.00046 1921KB refills: 25 > waste 2.9% gc: 25136B slow: 3000B fast: 0B > > TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 2KB slow > allocs: 22 refill waste: 32B alloc: 0.00002 78KB refills: 20 > waste 2.8% gc: 424B slow: 720B fast: 0B > > TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 2KB slow > allocs: 15 refill waste: 32B alloc: 0.00002 64KB refills: 28 > waste 4.5% gc: 1784B slow: 800B fast: 0B > > TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 63KB slow > allocs: 9 refill waste: 1008B alloc: 0.00076 3174KB refills: 59 > waste 1.3% gc: 44896B slow: 5656B fast: 0B > > TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow > allocs: 27 refill waste: 56B alloc: 0.00004 181KB refills: 27 > waste 4.2% gc: 2928B slow: 1280B fast: 0B > > TLAB totals: thrds: 17 refills: 3303 max: 3013 slow allocs: 104 max > 27 waste: 0.0% gc: 116672B max: 44896B slow: 143808B max: 126696B > fast: 0B max: 0B > > > Desired survivor size 268435456 bytes, new threshold 15 (max 15) > > - age 1: 300240 bytes, 300240 total > > - age 2: 9544 bytes, 309784 total > > - age 3: 352008 bytes, 661792 total > > - age 4: 93464 bytes, 755256 total > > - age 5: 10856 bytes, 766112 total > > - age 6: 50600 bytes, 816712 total > > - age 7: 205456 bytes, 1022168 total > > - age 8: 51664 bytes, 1073832 total > > - age 9: 14656 bytes, 1088488 total > > - age 10: 114176 bytes, 1202664 total > > - age 11: 369736 bytes, 1572400 total > > - age 12: 110328 bytes, 1682728 total > > - age 13: 14208 bytes, 1696936 total > > - age 14: 41296 bytes, 1738232 total > > - age 15: 689344 bytes, 2427576 total > > 68098.457: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 6677, predicted base time: 10.32 ms, remaining time: > 139.68 ms, target pause time: 150.00 ms] > > 68098.457: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 1014 regions, survivors: 10 regions, predicted young > region time: 25.44 ms] > > 68098.457: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 1014 regions, survivors: 10 regions, old: 0 regions, predicted > pause time: 35.75 ms, target pause time: 150.00 ms] > > 2017-01-25T14:10:54.015+0530: 68098.467: [SoftReference, 0 refs, > 0.0011853 secs]2017-01-25T14:10:54.016+0530: 68098.469: > [WeakReference, 0 refs, 0.0008073 secs]2017-01-25T14:10:54.017+0530: > 68098.469: [FinalReference, 1 refs, 0.0008374 > secs]2017-01-25T14:10:54.018+0530: 68098.470: [PhantomReference, 0 > refs, 0 refs, 0.0017963 secs]2017-01-25T14:10:54.019+0530: 68098.472: > [JNI Weak Reference, 0.0000168 secs], 0.0177960 secs] > > [Parallel Time: 5.8 ms, GC Workers: 20] > > [GC Worker Start (ms): Min: 68098457.1, Avg: 68098457.3, Max: > 68098457.4, Diff: 0.2] > > [Ext Root Scanning (ms): Min: 0.4, Avg: 0.6, Max: 1.0, Diff: > 0.6, Sum: 11.6] > > [Update RS (ms): Min: 0.0, Avg: 1.1, Max: 2.7, Diff: 2.7, Sum: 21.9] > > [Processed Buffers: Min: 0, Avg: 2.0, Max: 9, Diff: 9, Sum: 41] > > [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 1.7] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > > [Object Copy (ms): Min: 0.0, Avg: 2.0, Max: 3.2, Diff: 3.2, Sum: > 39.2] > > [Termination (ms): Min: 0.0, Avg: 1.7, Max: 2.2, Diff: 2.2, Sum: > 34.1] > > [Termination Attempts: Min: 1, Avg: 2.4, Max: 6, Diff: 5, > Sum: 47] > > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, > Sum: 2.2] > > [GC Worker Total (ms): Min: 5.4, Avg: 5.5, Max: 5.7, Diff: 0.4, > Sum: 110.8] > > [GC Worker End (ms): Min: 68098462.7, Avg: 68098462.8, Max: > 68098462.9, Diff: 0.2] > > [Code Root Fixup: 0.1 ms] > > [Code Root Purge: 0.0 ms] > > [Clear CT: 0.5 ms] > > [Other: 11.4 ms] > > [Choose CSet: 0.0 ms] > > [Ref Proc: 9.2 ms] > > [Ref Enq: 0.3 ms] > > [Redirty Cards: 0.3 ms] > > [Humongous Register: 0.0 ms] > > [Humongous Reclaim: 0.0 ms] > > [Free CSet: 1.4 ms] > > [Eden: 4056.0M(4056.0M)->0.0B(4084.0M) Survivors: 40.0M->12.0M > Heap: 5341.5M(8192.0M)->1276.8M(8192.0M)] > > [Times: user=0.00 sys=0.00, real=0.02 secs] > > 2017-01-25T14:10:54.022+0530: 68098.475: Total time for which > application threads were stopped: 0.0181786 seconds, Stopping threads > took: 0.0000375 seconds > > 2017-01-25T14:10:54.149+0530: 68098.601: Application time: 0.1266361 > seconds > > 2017-01-25T14:10:54.149+0530: 68098.602: Total time for which > application threads were stopped: 0.0003796 seconds, Stopping threads > took: 0.0000827 seconds > > 2017-01-25T14:10:55.149+0530: 68099.602: Application time: 0.9998625 > seconds > > 2017-01-25T14:10:55.149+0530: 68099.602: Total time for which > application threads were stopped: 0.0003472 seconds, Stopping threads > took: 0.0000761 seconds > > 2017-01-25T14:10:55.581+0530: 68100.034: Application time: 0.4318729 > seconds > > 2017-01-25T14:10:55.582+0530: 68100.034: [GC pause (G1 Evacuation > Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] > desired_size: 2KB slow allocs: 28 refill waste: 32B alloc: 0.00002 > 76KB refills: 43 waste 3.0% gc: 1328B slow: 1336B fast: 0B > > TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00000 3KB refills: 2 > waste 16.1% gc: 648B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f15906a1800 [id: 140] desired_size: 9KB slow > allocs: 1 refill waste: 152B alloc: 0.00012 495KB refills: 7 > waste 3.5% gc: 2240B slow: 208B fast: 0B > > TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 11KB slow > allocs: 0 refill waste: 184B alloc: 0.00014 595KB refills: 10 > waste 8.1% gc: 9608B slow: 272B fast: 0B > > TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 3KB slow > allocs: 4 refill waste: 56B alloc: 0.00005 188KB refills: 19 > waste 1.2% gc: 352B slow: 536B fast: 0B > > TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 2KB slow > allocs: 2 refill waste: 32B alloc: 0.00000 17KB refills: 5 > waste 12.9% gc: 1168B slow: 160B fast: 0B > > TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.99867 4176462KB > refills: 3040 waste 0.0% gc: 0B slow: 127000B fast: 0B > > TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 20KB slow > allocs: 0 refill waste: 320B alloc: 0.00024 1004KB refills: 24 > waste 4.4% gc: 20256B slow: 1256B fast: 0B > > TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 31KB slow > allocs: 4 refill waste: 504B alloc: 0.00038 1595KB refills: 19 > waste 3.9% gc: 22504B slow: 1448B fast: 0B > > TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 2KB slow > allocs: 8 refill waste: 32B alloc: 0.00002 67KB refills: 15 > waste 7.1% gc: 1696B slow: 512B fast: 0B > > TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 2KB slow > allocs: 11 refill waste: 32B alloc: 0.00002 64KB refills: 19 > waste 6.5% gc: 2000B slow: 560B fast: 0B > > TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 65KB slow > allocs: 9 refill waste: 1040B alloc: 0.00078 3255KB refills: 38 > waste 2.4% gc: 55624B slow: 5200B fast: 0B > > TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow > allocs: 68 refill waste: 48B alloc: 0.00004 155KB refills: 66 > waste 1.5% gc: 336B slow: 2848B fast: 0B > > TLAB totals: thrds: 13 refills: 3307 max: 3040 slow allocs: 136 max > 68 waste: 0.0% gc: 117760B max: 55624B slow: 141352B max: 127000B > fast: 0B max: 0B > > > Desired survivor size 268435456 bytes, new threshold 15 (max 15) > > - age 1: 240808 bytes, 240808 total > > - age 2: 290624 bytes, 531432 total > > - age 3: 9544 bytes, 540976 total > > - age 4: 345432 bytes, 886408 total > > - age 5: 93392 bytes, 979800 total > > - age 6: 10856 bytes, 990656 total > > - age 7: 50576 bytes, 1041232 total > > - age 8: 205264 bytes, 1246496 total > > - age 9: 51616 bytes, 1298112 total > > - age 10: 14656 bytes, 1312768 total > > - age 11: 114104 bytes, 1426872 total > > - age 12: 369568 bytes, 1796440 total > > - age 13: 110304 bytes, 1906744 total > > - age 14: 14208 bytes, 1920952 total > > - age 15: 41296 bytes, 1962248 total > > 68100.034: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 6344, predicted base time: 11.31 ms, remaining time: > 138.69 ms, target pause time: 150.00 ms] > > 68100.034: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 1021 regions, survivors: 3 regions, predicted young region > time: 7.85 ms] > > 68100.034: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 1021 regions, survivors: 3 regions, old: 0 regions, predicted > pause time: 19.16 ms, target pause time: 150.00 ms] > > 2017-01-25T14:10:55.592+0530: 68100.045: [SoftReference, 0 refs, > 0.0013160 secs]2017-01-25T14:10:55.593+0530: 68100.046: > [WeakReference, 0 refs, 0.0007300 secs]2017-01-25T14:10:55.594+0530: > 68100.047: [FinalReference, 1 refs, 0.0006525 > secs]2017-01-25T14:10:55.595+0530: 68100.047: [PhantomReference, 0 > refs, 0 refs, 0.0012912 secs]2017-01-25T14:10:55.596+0530: 68100.049: > [JNI Weak Reference, 0.0000154 secs], 0.0169289 secs] > > [Parallel Time: 9.7 ms, GC Workers: 20] > > [GC Worker Start (ms): Min: 68100034.5, Avg: 68100035.5, Max: > 68100037.6, Diff: 3.1] > > [Ext Root Scanning (ms): Min: 0.0, Avg: 0.3, Max: 1.0, Diff: > 1.0, Sum: 5.1] > > [Update RS (ms): Min: 0.0, Avg: 0.7, Max: 8.7, Diff: 8.7, Sum: 14.6] > > [Processed Buffers: Min: 0, Avg: 1.8, Max: 8, Diff: 8, Sum: 36] > > [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 1.6] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > > [Object Copy (ms): Min: 0.0, Avg: 4.2, Max: 7.4, Diff: 7.4, Sum: > 83.3] > > [Termination (ms): Min: 0.0, Avg: 3.1, Max: 6.6, Diff: 6.6, Sum: > 61.8] > > [Termination Attempts: Min: 1, Avg: 129.9, Max: 172, Diff: > 171, Sum: 2599] > > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, > Sum: 1.5] > > [GC Worker Total (ms): Min: 6.3, Avg: 8.4, Max: 9.4, Diff: 3.2, > Sum: 167.9] > > [GC Worker End (ms): Min: 68100043.8, Avg: 68100043.9, Max: > 68100044.0, Diff: 0.2] > > [Code Root Fixup: 0.0 ms] > > [Code Root Purge: 0.0 ms] > > [Clear CT: 0.5 ms] > > [Other: 6.7 ms] > > [Choose CSet: 0.1 ms] > > [Ref Proc: 4.5 ms] > > [Ref Enq: 0.3 ms] > > [Redirty Cards: 0.2 ms] > > [Humongous Register: 0.0 ms] > > [Humongous Reclaim: 0.0 ms] > > [Free CSet: 1.4 ms] > > [Eden: 4084.0M(4084.0M)->0.0B(4092.0M) Survivors: 12.0M->4096.0K > Heap: 5360.8M(8192.0M)->1277.9M(8192.0M)] > > [Times: user=0.00 sys=0.00, real=0.02 secs] > > 2017-01-25T14:10:55.599+0530: 68100.051: Total time for which > application threads were stopped: 0.0173605 seconds, Stopping threads > took: 0.0000391 seconds > > 2017-01-25T14:10:55.964+0530: 68100.416: Application time: 0.3650777 > seconds > > 2017-01-25T14:10:55.964+0530: 68100.417: Total time for which > application threads were stopped: 0.0002977 seconds, Stopping threads > took: 0.0000702 seconds > > 2017-01-25T14:10:56.149+0530: 68100.602: Application time: 0.1851895 > seconds > > 2017-01-25T14:10:56.150+0530: 68100.602: Total time for which > application threads were stopped: 0.0003614 seconds, Stopping threads > took: 0.0000746 seconds > > 2017-01-25T14:10:57.133+0530: 68101.586: Application time: 0.9833398 > seconds > > 2017-01-25T14:10:57.133+0530: 68101.586: [GC pause (G1 Evacuation > Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] > desired_size: 2KB slow allocs: 27 refill waste: 32B alloc: 0.00002 > 85KB refills: 43 waste 3.0% gc: 1328B slow: 1320B fast: 0B > > TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00000 3KB refills: 2 > waste 16.1% gc: 648B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 8KB slow > allocs: 0 refill waste: 128B alloc: 0.00010 421KB refills: 1 > waste 22.8% gc: 1968B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f159068b000 [id: 127] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 13KB refills: 2 > waste 25.2% gc: 1008B slow: 32B fast: 0B > > TLAB: gc thread: 0x00007f159067d800 [id: 121] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00001 62KB refills: 2 > waste 25.2% gc: 1008B slow: 32B fast: 0B > > TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.99883 4185307KB > refills: 3062 waste 0.0% gc: 0B slow: 132496B fast: 0B > > TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 54KB slow > allocs: 0 refill waste: 872B alloc: 0.00065 2728KB refills: 6 > waste 11.0% gc: 36704B slow: 312B fast: 0B > > TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow > allocs: 15 refill waste: 56B alloc: 0.00004 178KB refills: 26 > waste 2.6% gc: 1360B slow: 1112B fast: 0B > > TLAB: gc thread: 0x00007f170d35a000 [id: 85] desired_size: 7KB slow > allocs: 0 refill waste: 112B alloc: 0.00009 367KB refills: 4 > waste 1.9% gc: 480B slow: 80B fast: 0B > > TLAB totals: thrds: 9 refills: 3148 max: 3062 slow allocs: 43 max 27 > waste: 0.0% gc: 44504B max: 36704B slow: 135400B max: 132496B fast: > 0B max: 0B > > > Desired survivor size 268435456 bytes, new threshold 15 (max 15) > > - age 1: 166472 bytes, 166472 total > > - age 2: 230296 bytes, 396768 total > > - age 3: 290624 bytes, 687392 total > > - age 4: 9544 bytes, 696936 total > > - age 5: 345432 bytes, 1042368 total > > - age 6: 93392 bytes, 1135760 total > > - age 7: 10856 bytes, 1146616 total > > - age 8: 50576 bytes, 1197192 total > > - age 9: 205264 bytes, 1402456 total > > - age 10: 51616 bytes, 1454072 total > > - age 11: 14656 bytes, 1468728 total > > - age 12: 114104 bytes, 1582832 total > > - age 13: 369544 bytes, 1952376 total > > - age 14: 110304 bytes, 2062680 total > > - age 15: 14208 bytes, 2076888 total > > 68101.586: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 5995, predicted base time: 10.55 ms, remaining time: > 139.45 ms, target pause time: 150.00 ms] > > 68101.586: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 1023 regions, survivors: 1 regions, predicted young region > time: 5.63 ms] > > 68101.586: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 1023 regions, survivors: 1 regions, old: 0 regions, predicted > pause time: 16.19 ms, target pause time: 150.00 ms] > > 2017-01-25T14:10:57.139+0530: 68101.591: [SoftReference, 0 refs, > 0.0010864 secs]2017-01-25T14:10:57.140+0530: 68101.592: > [WeakReference, 0 refs, 0.0007033 secs]2017-01-25T14:10:57.140+0530: > 68101.593: [FinalReference, 1 refs, 0.0006381 > secs]2017-01-25T14:10:57.141+0530: 68101.594: [PhantomReference, 0 > refs, 0 refs, 0.0013022 secs]2017-01-25T14:10:57.142+0530: 68101.595: > [JNI Weak Reference, 0.0000164 secs], 0.0119617 secs] > > [Parallel Time: 4.8 ms, GC Workers: 20] > > [GC Worker Start (ms): Min: 68101586.0, Avg: 68101586.2, Max: > 68101586.4, Diff: 0.4] > > [Ext Root Scanning (ms): Min: 0.1, Avg: 0.9, Max: 4.2, Diff: > 4.1, Sum: 17.7] > > [Update RS (ms): Min: 0.0, Avg: 0.3, Max: 0.7, Diff: 0.7, Sum: 5.4] > > [Processed Buffers: Min: 0, Avg: 1.6, Max: 6, Diff: 6, Sum: 33] > > [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 1.7] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > > [Object Copy (ms): Min: 0.0, Avg: 1.1, Max: 1.5, Diff: 1.5, Sum: > 22.4] > > [Termination (ms): Min: 0.0, Avg: 1.9, Max: 2.2, Diff: 2.2, Sum: > 38.7] > > [Termination Attempts: Min: 1, Avg: 48.0, Max: 74, Diff: 73, > Sum: 959] > > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, > Sum: 1.5] > > [GC Worker Total (ms): Min: 4.1, Avg: 4.4, Max: 4.6, Diff: 0.4, > Sum: 87.5] > > [GC Worker End (ms): Min: 68101590.5, Avg: 68101590.6, Max: > 68101590.7, Diff: 0.2] > > [Code Root Fixup: 0.0 ms] > > [Code Root Purge: 0.0 ms] > > [Clear CT: 0.5 ms] > > [Other: 6.6 ms] > > [Choose CSet: 0.0 ms] > > [Ref Proc: 4.2 ms] > > [Ref Enq: 0.4 ms] > > [Redirty Cards: 0.3 ms] > > [Humongous Register: 0.1 ms] > > [Humongous Reclaim: 0.0 ms] > > [Free CSet: 1.5 ms] > > [Eden: 4092.0M(4092.0M)->0.0B(4092.0M) Survivors: 4096.0K->4096.0K > Heap: 5369.9M(8192.0M)->1278.0M(8192.0M)] > > [Times: user=0.00 sys=0.00, real=0.01 secs] > > 2017-01-25T14:10:57.145+0530: 68101.598: Total time for which > application threads were stopped: 0.0123703 seconds, Stopping threads > took: 0.0000400 seconds > > 2017-01-25T14:10:57.149+0530: 68101.602: Application time: 0.0039993 > seconds > > 2017-01-25T14:10:57.150+0530: 68101.602: Total time for which > application threads were stopped: 0.0003789 seconds, Stopping threads > took: 0.0000725 seconds > > 2017-01-25T14:10:58.149+0530: 68102.602: Application time: 0.9995898 > seconds > > 2017-01-25T14:10:58.150+0530: 68102.602: Total time for which > application threads were stopped: 0.0003840 seconds, Stopping threads > took: 0.0000716 seconds > > 2017-01-25T14:10:58.708+0530: 68103.161: Application time: 0.5585688 > seconds > > 2017-01-25T14:10:58.709+0530: 68103.161: [GC pause (G1 Evacuation > Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] > desired_size: 2KB slow allocs: 15 refill waste: 32B alloc: 0.00002 > 91KB refills: 22 waste 4.4% gc: 1328B slow: 648B fast: 0B > > TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 3KB refills: 1 > waste 14.0% gc: 288B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f15906a1800 [id: 140] desired_size: 6KB slow > allocs: 0 refill waste: 104B alloc: 0.00008 344KB refills: 6 > waste 9.1% gc: 3688B slow: 144B fast: 0B > > TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 5KB slow > allocs: 3 refill waste: 80B alloc: 0.00007 274KB refills: 24 > waste 2.7% gc: 2856B slow: 824B fast: 0B > > TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 6KB slow > allocs: 0 refill waste: 104B alloc: 0.00008 331KB refills: 4 > waste 21.2% gc: 5640B slow: 96B fast: 0B > > TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 14KB refills: 4 > waste 24.4% gc: 1920B slow: 96B fast: 0B > > TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.99919 4186812KB > refills: 3045 waste 0.0% gc: 0B slow: 131144B fast: 0B > > TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 16KB slow > allocs: 2 refill waste: 256B alloc: 0.00020 821KB refills: 24 > waste 1.5% gc: 5016B slow: 1136B fast: 0B > > TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 24KB slow > allocs: 3 refill waste: 392B alloc: 0.00030 1249KB refills: 23 > waste 4.2% gc: 22928B slow: 1552B fast: 0B > > TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 2KB slow > allocs: 4 refill waste: 32B alloc: 0.00001 55KB refills: 13 > waste 8.1% gc: 1792B slow: 376B fast: 0B > > TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 2KB slow > allocs: 6 refill waste: 32B alloc: 0.00001 56KB refills: 13 > waste 8.6% gc: 1952B slow: 352B fast: 0B > > TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 36KB slow > allocs: 21 refill waste: 584B alloc: 0.00044 1836KB refills: 45 > waste 1.9% gc: 28232B slow: 4360B fast: 0B > > TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow > allocs: 61 refill waste: 48B alloc: 0.00004 150KB refills: 62 > waste 1.8% gc: 544B slow: 2912B fast: 0B > > TLAB totals: thrds: 13 refills: 3286 max: 3045 slow allocs: 115 max > 61 waste: 0.0% gc: 76184B max: 28232B slow: 143640B max: 131144B > fast: 0B max: 0B > > > Desired survivor size 268435456 bytes, new threshold 15 (max 15) > > - age 1: 27288 bytes, 27288 total > > - age 2: 160520 bytes, 187808 total > > - age 3: 230296 bytes, 418104 total > > - age 4: 287608 bytes, 705712 total > > - age 5: 9544 bytes, 715256 total > > - age 6: 345432 bytes, 1060688 total > > - age 7: 93392 bytes, 1154080 total > > - age 8: 10856 bytes, 1164936 total > > - age 9: 50576 bytes, 1215512 total > > - age 10: 205264 bytes, 1420776 total > > - age 11: 51616 bytes, 1472392 total > > - age 12: 14656 bytes, 1487048 total > > - age 13: 114104 bytes, 1601152 total > > - age 14: 369544 bytes, 1970696 total > > - age 15: 110304 bytes, 2081000 total > > 68103.161: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 4514, predicted base time: 9.61 ms, remaining time: > 140.39 ms, target pause time: 150.00 ms] > > 68103.161: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 1023 regions, survivors: 1 regions, predicted young region > time: 4.47 ms] > > 68103.161: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 1023 regions, survivors: 1 regions, old: 0 regions, predicted > pause time: 14.08 ms, target pause time: 150.00 ms] > > 2017-01-25T14:10:58.713+0530: 68103.165: [SoftReference, 0 refs, > 0.0016775 secs]2017-01-25T14:10:58.715+0530: 68103.167: > [WeakReference, 0 refs, 0.0010189 secs]2017-01-25T14:10:58.716+0530: > 68103.168: [FinalReference, 1 refs, 0.0005777 > secs]2017-01-25T14:10:58.716+0530: 68103.169: [PhantomReference, 0 > refs, 0 refs, 0.0013692 secs]2017-01-25T14:10:58.718+0530: 68103.170: > [JNI Weak Reference, 0.0000145 secs], 0.0114214 secs] > > [Parallel Time: 3.4 ms, GC Workers: 20] > > [GC Worker Start (ms): Min: 68103161.4, Avg: 68103162.0, Max: > 68103163.8, Diff: 2.4] > > [Ext Root Scanning (ms): Min: 0.0, Avg: 0.3, Max: 1.1, Diff: > 1.1, Sum: 6.4] > > [Update RS (ms): Min: 0.0, Avg: 0.4, Max: 2.6, Diff: 2.6, Sum: 8.0] > > [Processed Buffers: Min: 0, Avg: 1.4, Max: 8, Diff: 8, Sum: 29] > > [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 1.6] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > > [Object Copy (ms): Min: 0.0, Avg: 0.7, Max: 1.0, Diff: 1.0, Sum: > 13.9] > > [Termination (ms): Min: 0.0, Avg: 1.1, Max: 1.3, Diff: 1.3, Sum: > 22.0] > > [Termination Attempts: Min: 1, Avg: 52.8, Max: 84, Diff: 83, > Sum: 1055] > > [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, > Sum: 1.0] > > [GC Worker Total (ms): Min: 0.8, Avg: 2.7, Max: 3.3, Diff: 2.5, > Sum: 53.0] > > [GC Worker End (ms): Min: 68103164.6, Avg: 68103164.7, Max: > 68103164.7, Diff: 0.1] > > [Code Root Fixup: 0.0 ms] > > [Code Root Purge: 0.0 ms] > > [Clear CT: 0.4 ms] > > [Other: 7.5 ms] > > [Choose CSet: 0.0 ms] > > [Ref Proc: 5.3 ms] > > [Ref Enq: 0.4 ms] > > [Redirty Cards: 0.2 ms] > > [Humongous Register: 0.1 ms] > > [Humongous Reclaim: 0.0 ms] > > [Free CSet: 1.3 ms] > > [Eden: 4092.0M(4092.0M)->0.0B(4092.0M) Survivors: 4096.0K->4096.0K > Heap: 5370.0M(8192.0M)->1278.6M(8192.0M)] > > [Times: user=0.00 sys=0.00, real=0.02 secs] > > 2017-01-25T14:10:58.720+0530: 68103.173: Total time for which > application threads were stopped: 0.0117864 seconds, Stopping threads > took: 0.0000423 seconds > > 2017-01-25T14:10:59.149+0530: 68103.602: Application time: 0.4294109 > seconds > > 2017-01-25T14:10:59.150+0530: 68103.602: Total time for which > application threads were stopped: 0.0003487 seconds, Stopping threads > took: 0.0000772 seconds > > 2017-01-25T14:11:00.042+0530: 68104.494: Application time: 0.8919065 > seconds > > 2017-01-25T14:11:00.042+0530: 68104.495: Total time for which > application threads were stopped: 0.0003228 seconds, Stopping threads > took: 0.0001064 seconds > > 2017-01-25T14:11:00.150+0530: 68104.602: Application time: 0.1075120 > seconds > > 2017-01-25T14:11:00.150+0530: 68104.603: Total time for which > application threads were stopped: 0.0003795 seconds, Stopping threads > took: 0.0000747 seconds > > 2017-01-25T14:11:00.317+0530: 68104.770: Application time: 0.1671251 > seconds > > 2017-01-25T14:11:00.317+0530: 68104.770: [GC pause (G1 Evacuation > Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] > desired_size: 2KB slow allocs: 27 refill waste: 32B alloc: 0.00002 > 77KB refills: 43 waste 2.7% gc: 1064B slow: 1376B fast: 0B > > TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00000 3KB refills: 2 > waste 16.1% gc: 648B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 2KB slow > allocs: 2 refill waste: 40B alloc: 0.00004 148KB refills: 21 > waste 1.9% gc: 528B slow: 672B fast: 0B > > TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 4KB slow > allocs: 0 refill waste: 64B alloc: 0.00005 222KB refills: 34 > waste 3.4% gc: 3976B slow: 1200B fast: 0B > > TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 7KB slow > allocs: 0 refill waste: 112B alloc: 0.00009 362KB refills: 4 > waste 0.5% gc: 32B slow: 112B fast: 0B > > TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 11KB refills: 6 > waste 11.8% gc: 1320B slow: 144B fast: 0B > > TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.99921 4186917KB > refills: 3045 waste 0.0% gc: 0B slow: 128648B fast: 0B > > TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 13KB slow > allocs: 1 refill waste: 208B alloc: 0.00016 667KB refills: 45 > waste 2.2% gc: 11000B slow: 2520B fast: 0B > > TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 19KB slow > allocs: 10 refill waste: 312B alloc: 0.00024 993KB refills: 37 > waste 2.6% gc: 17320B slow: 2224B fast: 0B > > TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 2KB slow > allocs: 13 refill waste: 32B alloc: 0.00001 45KB refills: 17 > waste 6.7% gc: 1888B slow: 480B fast: 0B > > TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 2KB slow > allocs: 8 refill waste: 32B alloc: 0.00001 46KB refills: 19 > waste 5.8% gc: 1736B slow: 528B fast: 0B > > TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 34KB slow > allocs: 22 refill waste: 544B alloc: 0.00041 1709KB refills: 58 > waste 0.3% gc: 1280B slow: 5712B fast: 0B > > TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow > allocs: 60 refill waste: 48B alloc: 0.00004 167KB refills: 56 > waste 2.8% gc: 2728B slow: 2576B fast: 0B > > TLAB: gc thread: 0x00007f170d362000 [id: 86] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 0KB refills: 1 > waste 98.4% gc: 2032B slow: 0B fast: 0B > > TLAB totals: thrds: 14 refills: 3388 max: 3045 slow allocs: 144 max > 60 waste: 0.0% gc: 45552B max: 17320B slow: 146208B max: 128648B > fast: 0B max: 0B > > > Desired survivor size 268435456 bytes, new threshold 15 (max 15) > > - age 1: 137864 bytes, 137864 total > > - age 2: 21240 bytes, 159104 total > > - age 3: 158672 bytes, 317776 total > > - age 4: 223000 bytes, 540776 total > > - age 5: 283888 bytes, 824664 total > > - age 6: 9544 bytes, 834208 total > > - age 7: 345392 bytes, 1179600 total > > - age 8: 93392 bytes, 1272992 total > > - age 9: 10856 bytes, 1283848 total > > - age 10: 50576 bytes, 1334424 total > > - age 11: 205264 bytes, 1539688 total > > - age 12: 51616 bytes, 1591304 total > > - age 13: 14656 bytes, 1605960 total > > - age 14: 114104 bytes, 1720064 total > > - age 15: 369544 bytes, 2089608 total > > 68104.770: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 5046, predicted base time: 9.36 ms, remaining time: > 140.64 ms, target pause time: 150.00 ms] > > 68104.770: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 1023 regions, survivors: 1 regions, predicted young region > time: 3.77 ms] > > 68104.770: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 1023 regions, survivors: 1 regions, old: 0 regions, predicted > pause time: 13.13 ms, target pause time: 150.00 ms] > > 2017-01-25T14:11:00.321+0530: 68104.773: [SoftReference, 0 refs, > 0.0011104 secs]2017-01-25T14:11:00.322+0530: 68104.774: > [WeakReference, 0 refs, 0.0056241 secs]2017-01-25T14:11:00.327+0530: > 68104.780: [FinalReference, 1 refs, 0.0005825 > secs]2017-01-25T14:11:00.328+0530: 68104.781: [PhantomReference, 0 > refs, 0 refs, 0.0012515 secs]2017-01-25T14:11:00.329+0530: 68104.782: > [JNI Weak Reference, 0.0000189 secs], 0.0143937 secs] > > [Parallel Time: 2.5 ms, GC Workers: 20] > > [GC Worker Start (ms): Min: 68104770.2, Avg: 68104770.6, Max: > 68104772.1, Diff: 1.8] > > [Ext Root Scanning (ms): Min: 0.0, Avg: 0.3, Max: 1.0, Diff: > 1.0, Sum: 6.1] > > [Update RS (ms): Min: 0.0, Avg: 0.4, Max: 1.7, Diff: 1.7, Sum: 8.3] > > [Processed Buffers: Min: 0, Avg: 1.6, Max: 7, Diff: 7, Sum: 32] > > [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 1.6] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > > [Object Copy (ms): Min: 0.0, Avg: 0.8, Max: 1.1, Diff: 1.1, Sum: > 16.9] > > [Termination (ms): Min: 0.0, Avg: 0.3, Max: 0.6, Diff: 0.6, Sum: > 5.7] > > [Termination Attempts: Min: 1, Avg: 30.1, Max: 55, Diff: 54, > Sum: 602] > > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, > Sum: 1.4] > > [GC Worker Total (ms): Min: 0.5, Avg: 2.0, Max: 2.4, Diff: 1.9, > Sum: 40.0] > > [GC Worker End (ms): Min: 68104772.5, Avg: 68104772.6, Max: > 68104772.7, Diff: 0.2] > > [Code Root Fixup: 0.0 ms] > > [Code Root Purge: 0.0 ms] > > [Clear CT: 0.5 ms] > > [Other: 11.4 ms] > > [Choose CSet: 0.0 ms] > > [Ref Proc: 9.1 ms] > > [Ref Enq: 0.3 ms] > > [Redirty Cards: 0.2 ms] > > [Humongous Register: 0.1 ms] > > [Humongous Reclaim: 0.0 ms] > > [Free CSet: 1.4 ms] > > [Eden: 4092.0M(4092.0M)->0.0B(4092.0M) Survivors: 4096.0K->4096.0K > Heap: 5370.6M(8192.0M)->1278.8M(8192.0M)] > > [Times: user=0.00 sys=0.00, real=0.02 secs] > > 2017-01-25T14:11:00.332+0530: 68104.785: Total time for which > application threads were stopped: 0.0147927 seconds, Stopping threads > took: 0.0000410 seconds > > 2017-01-25T14:11:00.964+0530: 68105.417: Application time: 0.6324078 > seconds > > 2017-01-25T14:11:00.965+0530: 68105.417: Total time for which > application threads were stopped: 0.0002847 seconds, Stopping threads > took: 0.0000800 seconds > > 2017-01-25T14:11:01.149+0530: 68105.601: Application time: 0.1842524 > seconds > > 2017-01-25T14:11:01.149+0530: 68105.602: Total time for which > application threads were stopped: 0.0003608 seconds, Stopping threads > took: 0.0000774 seconds > > 2017-01-25T14:11:01.894+0530: 68106.346: Application time: 0.7445171 > seconds > > 2017-01-25T14:11:01.894+0530: 68106.347: [GC pause (G1 Evacuation > Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] > desired_size: 2KB slow allocs: 16 refill waste: 32B alloc: 0.00002 > 86KB refills: 22 waste 4.5% gc: 1328B slow: 696B fast: 0B > > TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 3KB refills: 1 > waste 14.0% gc: 288B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f159062b000 [id: 115] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 1KB refills: 1 > waste 89.9% gc: 1856B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f159005b000 [id: 114] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 0KB refills: 1 > waste 98.4% gc: 2032B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.99916 4186676KB > refills: 3064 waste 0.0% gc: 0B slow: 129544B fast: 0B > > TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 35KB slow > allocs: 0 refill waste: 560B alloc: 0.00042 1752KB refills: 3 > waste 30.8% gc: 33128B slow: 32B fast: 0B > > TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow > allocs: 15 refill waste: 56B alloc: 0.00004 179KB refills: 26 > waste 2.5% gc: 1336B slow: 1072B fast: 0B > > TLAB: gc thread: 0x00007f170d35a000 [id: 85] desired_size: 4KB slow > allocs: 0 refill waste: 72B alloc: 0.00006 246KB refills: 6 > waste 2.6% gc: 576B slow: 224B fast: 0B > > TLAB totals: thrds: 8 refills: 3124 max: 3064 slow allocs: 31 max 16 > waste: 0.0% gc: 40544B max: 33128B slow: 131568B max: 129544B fast: > 0B max: 0B > > > Desired survivor size 268435456 bytes, new threshold 15 (max 15) > > - age 1: 173504 bytes, 173504 total > > - age 2: 133784 bytes, 307288 total > > - age 3: 21240 bytes, 328528 total > > - age 4: 150024 bytes, 478552 total > > - age 5: 221056 bytes, 699608 total > > - age 6: 283672 bytes, 983280 total > > - age 7: 9544 bytes, 992824 total > > - age 8: 345368 bytes, 1338192 total > > - age 9: 93392 bytes, 1431584 total > > - age 10: 10856 bytes, 1442440 total > > - age 11: 50576 bytes, 1493016 total > > - age 12: 205264 bytes, 1698280 total > > - age 13: 51616 bytes, 1749896 total > > - age 14: 14656 bytes, 1764552 total > > - age 15: 114080 bytes, 1878632 total > > 68106.347: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 4895, predicted base time: 10.43 ms, remaining time: > 139.57 ms, target pause time: 150.00 ms] > > 68106.347: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 1023 regions, survivors: 1 regions, predicted young region > time: 3.38 ms] > > 68106.347: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 1023 regions, survivors: 1 regions, old: 0 regions, predicted > pause time: 13.81 ms, target pause time: 150.00 ms] > > 2017-01-25T14:11:01.898+0530: 68106.350: [SoftReference, 0 refs, > 0.0011375 secs]2017-01-25T14:11:01.899+0530: 68106.351: > [WeakReference, 0 refs, 0.0005968 secs]2017-01-25T14:11:01.899+0530: > 68106.352: [FinalReference, 0 refs, 0.0005507 > secs]2017-01-25T14:11:01.900+0530: 68106.353: [PhantomReference, 0 > refs, 0 refs, 0.0013671 secs]2017-01-25T14:11:01.901+0530: 68106.354: > [JNI Weak Reference, 0.0000176 secs], 0.0098550 secs] > > [Parallel Time: 3.0 ms, GC Workers: 20] > > [GC Worker Start (ms): Min: 68106346.8, Avg: 68106347.1, Max: > 68106347.5, Diff: 0.7] > > [Ext Root Scanning (ms): Min: 0.0, Avg: 0.4, Max: 1.9, Diff: > 1.9, Sum: 8.0] > > [Update RS (ms): Min: 0.0, Avg: 0.6, Max: 2.2, Diff: 2.2, Sum: 12.0] > > [Processed Buffers: Min: 0, Avg: 1.4, Max: 7, Diff: 7, Sum: 28] > > [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 1.7] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > > [Object Copy (ms): Min: 0.0, Avg: 0.8, Max: 1.2, Diff: 1.2, Sum: > 16.2] > > [Termination (ms): Min: 0.0, Avg: 0.6, Max: 0.9, Diff: 0.9, Sum: > 11.2] > > [Termination Attempts: Min: 1, Avg: 84.4, Max: 141, Diff: > 140, Sum: 1688] > > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, > Sum: 1.3] > > [GC Worker Total (ms): Min: 2.1, Avg: 2.5, Max: 2.7, Diff: 0.7, > Sum: 50.5] > > [GC Worker End (ms): Min: 68106349.5, Avg: 68106349.6, Max: > 68106349.7, Diff: 0.2] > > [Code Root Fixup: 0.0 ms] > > [Code Root Purge: 0.0 ms] > > [Clear CT: 0.5 ms] > > [Other: 6.4 ms] > > [Choose CSet: 0.0 ms] > > [Ref Proc: 4.2 ms] > > [Ref Enq: 0.3 ms] > > [Redirty Cards: 0.2 ms] > > [Humongous Register: 0.1 ms] > > [Humongous Reclaim: 0.0 ms] > > [Free CSet: 1.4 ms] > > [Eden: 4092.0M(4092.0M)->0.0B(4092.0M) Survivors: 4096.0K->4096.0K > Heap: 5370.8M(8192.0M)->1279.3M(8192.0M)] > > [Times: user=0.00 sys=0.00, real=0.01 secs] > > 2017-01-25T14:11:01.904+0530: 68106.357: Total time for which > application threads were stopped: 0.0102433 seconds, Stopping threads > took: 0.0000400 seconds > > 2017-01-25T14:11:02.149+0530: 68106.602: Application time: 0.2449541 > seconds > > 2017-01-25T14:11:02.149+0530: 68106.602: Total time for which > application threads were stopped: 0.0003583 seconds, Stopping threads > took: 0.0000634 seconds > > 2017-01-25T14:11:03.149+0530: 68107.602: Application time: 0.9997439 > seconds > > 2017-01-25T14:11:03.149+0530: 68107.602: Total time for which > application threads were stopped: 0.0003310 seconds, Stopping threads > took: 0.0000656 seconds > > 2017-01-25T14:11:03.486+0530: 68107.939: Application time: 0.3365735 > seconds > > 2017-01-25T14:11:03.486+0530: 68107.939: [GC pause (G1 Evacuation > Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] > desired_size: 2KB slow allocs: 25 refill waste: 32B alloc: 0.00002 > 74KB refills: 43 waste 2.8% gc: 1064B slow: 1416B fast: 0B > > TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00000 2KB refills: 2 > waste 16.1% gc: 648B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f15906a8800 [id: 144] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 13KB refills: 2 > waste 25.2% gc: 1008B slow: 32B fast: 0B > > TLAB: gc thread: 0x00007f15906a7000 [id: 143] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 13KB refills: 2 > waste 25.2% gc: 1008B slow: 32B fast: 0B > > TLAB: gc thread: 0x00007f15906a5000 [id: 142] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 13KB refills: 2 > waste 26.0% gc: 1056B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f15906a0000 [id: 139] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 13KB refills: 2 > waste 25.2% gc: 1008B slow: 32B fast: 0B > > TLAB: gc thread: 0x00007f159069e800 [id: 138] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 13KB refills: 2 > waste 25.2% gc: 1008B slow: 32B fast: 0B > > TLAB: gc thread: 0x00007f159069c800 [id: 137] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 13KB refills: 2 > waste 25.2% gc: 1008B slow: 32B fast: 0B > > TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 4KB slow > allocs: 0 refill waste: 72B alloc: 0.00005 225KB refills: 20 > waste 3.8% gc: 2856B slow: 600B fast: 0B > > TLAB: gc thread: 0x00007f1590697800 [id: 134] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 13KB refills: 2 > waste 26.0% gc: 1056B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f1590695800 [id: 133] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 13KB refills: 2 > waste 25.2% gc: 1008B slow: 32B fast: 0B > > TLAB: gc thread: 0x00007f1590693800 [id: 132] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 13KB refills: 2 > waste 25.2% gc: 1008B slow: 32B fast: 0B > > TLAB: gc thread: 0x00007f1590690000 [id: 130] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00003 117KB refills: 4 > waste 4.5% gc: 336B slow: 96B fast: 0B > > TLAB: gc thread: 0x00007f159068e800 [id: 129] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 13KB refills: 2 > waste 25.0% gc: 1016B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f159068d000 [id: 128] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 13KB refills: 2 > waste 25.2% gc: 1008B slow: 32B fast: 0B > > TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 3KB slow > allocs: 0 refill waste: 56B alloc: 0.00005 195KB refills: 3 > waste 26.9% gc: 3168B slow: 56B fast: 0B > > TLAB: gc thread: 0x00007f1590674800 [id: 124] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 13KB refills: 2 > waste 25.2% gc: 1008B slow: 32B fast: 0B > > TLAB: gc thread: 0x00007f1590673000 [id: 123] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 13KB refills: 2 > waste 25.2% gc: 1008B slow: 32B fast: 0B > > TLAB: gc thread: 0x00007f1590671800 [id: 122] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 13KB refills: 2 > waste 25.0% gc: 1016B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 4KB slow > allocs: 0 refill waste: 72B alloc: 0.00006 243KB refills: 6 > waste 3.6% gc: 896B slow: 184B fast: 0B > > TLAB: gc thread: 0x00007f1590637000 [id: 117] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 13KB refills: 2 > waste 25.8% gc: 1048B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 11KB refills: 4 > waste 24.3% gc: 1920B slow: 88B fast: 0B > > TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.99942 4187777KB > refills: 3045 waste 0.0% gc: 0B slow: 127312B fast: 0B > > TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 12KB slow > allocs: 0 refill waste: 200B alloc: 0.00015 635KB refills: 17 > waste 4.1% gc: 8512B slow: 640B fast: 0B > > TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 18KB slow > allocs: 5 refill waste: 288B alloc: 0.00022 904KB refills: 14 > waste 2.1% gc: 4800B slow: 672B fast: 0B > > TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 2KB slow > allocs: 6 refill waste: 32B alloc: 0.00001 42KB refills: 7 > waste 15.2% gc: 2000B slow: 200B fast: 0B > > TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 2KB slow > allocs: 9 refill waste: 32B alloc: 0.00001 44KB refills: 13 > waste 9.1% gc: 2000B slow: 448B fast: 0B > > TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 23KB slow > allocs: 8 refill waste: 368B alloc: 0.00028 1163KB refills: 32 > waste 2.0% gc: 12824B slow: 2144B fast: 0B > > TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 2KB slow > allocs: 51 refill waste: 40B alloc: 0.00004 150KB refills: 65 > waste 2.5% gc: 2736B slow: 2264B fast: 0B > > TLAB totals: thrds: 29 refills: 3305 max: 3045 slow allocs: 105 max > 51 waste: 0.0% gc: 59032B max: 12824B slow: 136536B max: 127312B > fast: 0B max: 0B > > > Desired survivor size 268435456 bytes, new threshold 15 (max 15) > > - age 1: 16648 bytes, 16648 total > > - age 2: 169208 bytes, 185856 total > > - age 3: 133784 bytes, 319640 total > > - age 4: 21176 bytes, 340816 total > > - age 5: 150024 bytes, 490840 total > > - age 6: 221056 bytes, 711896 total > > - age 7: 283672 bytes, 995568 total > > - age 8: 9544 bytes, 1005112 total > > - age 9: 345368 bytes, 1350480 total > > - age 10: 93392 bytes, 1443872 total > > - age 11: 10856 bytes, 1454728 total > > - age 12: 50576 bytes, 1505304 total > > - age 13: 205232 bytes, 1710536 total > > - age 14: 51616 bytes, 1762152 total > > - age 15: 14656 bytes, 1776808 total > > 68107.939: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 5758, predicted base time: 9.80 ms, remaining time: > 140.20 ms, target pause time: 150.00 ms] > > 68107.939: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 1023 regions, survivors: 1 regions, predicted young region > time: 3.10 ms] > > 68107.939: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 1023 regions, survivors: 1 regions, old: 0 regions, predicted > pause time: 12.90 ms, target pause time: 150.00 ms] > > 2017-01-25T14:11:03.491+0530: 68107.944: [SoftReference, 0 refs, > 0.0009941 secs]2017-01-25T14:11:03.492+0530: 68107.945: > [WeakReference, 0 refs, 0.0007176 secs]2017-01-25T14:11:03.493+0530: > 68107.946: [FinalReference, 0 refs, 0.0005977 > secs]2017-01-25T14:11:03.494+0530: 68107.946: [PhantomReference, 0 > refs, 0 refs, 0.0044912 secs]2017-01-25T14:11:03.498+0530: 68107.951: > [JNI Weak Reference, 0.0000168 secs], 0.0142796 secs] > > [Parallel Time: 4.2 ms, GC Workers: 20] > > [GC Worker Start (ms): Min: 68107939.2, Avg: 68107939.6, Max: > 68107943.0, Diff: 3.8] > > [Ext Root Scanning (ms): Min: 0.0, Avg: 0.3, Max: 1.0, Diff: > 1.0, Sum: 5.9] > > [Update RS (ms): Min: 0.0, Avg: 0.4, Max: 1.6, Diff: 1.6, Sum: 8.8] > > [Processed Buffers: Min: 0, Avg: 2.5, Max: 8, Diff: 8, Sum: 49] > > [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 2.0] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > > [Object Copy (ms): Min: 0.0, Avg: 0.8, Max: 1.1, Diff: 1.1, Sum: > 15.5] > > [Termination (ms): Min: 0.0, Avg: 1.9, Max: 2.1, Diff: 2.1, Sum: > 38.5] > > [Termination Attempts: Min: 1, Avg: 61.9, Max: 87, Diff: 86, > Sum: 1237] > > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, > Sum: 1.3] > > [GC Worker Total (ms): Min: 0.2, Avg: 3.6, Max: 4.1, Diff: 3.9, > Sum: 72.0] > > [GC Worker End (ms): Min: 68107943.1, Avg: 68107943.2, Max: > 68107943.3, Diff: 0.1] > > [Code Root Fixup: 0.0 ms] > > [Code Root Purge: 0.0 ms] > > [Clear CT: 0.4 ms] > > [Other: 9.6 ms] > > [Choose CSet: 0.0 ms] > > [Ref Proc: 7.3 ms] > > [Ref Enq: 0.3 ms] > > [Redirty Cards: 0.2 ms] > > [Humongous Register: 0.1 ms] > > [Humongous Reclaim: 0.0 ms] > > [Free CSet: 1.4 ms] > > [Eden: 4092.0M(4092.0M)->0.0B(4092.0M) Survivors: 4096.0K->4096.0K > Heap: 5371.3M(8192.0M)->1278.5M(8192.0M)] > > [Times: user=0.00 sys=0.00, real=0.02 secs] > > 2017-01-25T14:11:03.501+0530: 68107.953: Total time for which > application threads were stopped: 0.0146790 seconds, Stopping threads > took: 0.0000401 seconds > > 2017-01-25T14:11:04.149+0530: 68108.602: Application time: 0.6486007 > seconds > > 2017-01-25T14:11:04.150+0530: 68108.602: Total time for which > application threads were stopped: 0.0003741 seconds, Stopping threads > took: 0.0000833 seconds > > 2017-01-25T14:11:05.096+0530: 68109.549: Application time: 0.9466622 > seconds > > 2017-01-25T14:11:05.097+0530: 68109.549: [GC pause (G1 Evacuation > Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] > desired_size: 2KB slow allocs: 26 refill waste: 32B alloc: 0.00002 > 84KB refills: 43 waste 2.8% gc: 1064B slow: 1408B fast: 0B > > TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00000 3KB refills: 2 > waste 16.1% gc: 648B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f15906a1800 [id: 140] desired_size: 4KB slow > allocs: 3 refill waste: 72B alloc: 0.00006 236KB refills: 34 > waste 2.4% gc: 2776B slow: 1216B fast: 0B > > TLAB: gc thread: 0x00007f1590699000 [id: 135] desired_size: 3KB slow > allocs: 2 refill waste: 56B alloc: 0.00004 176KB refills: 16 > waste 2.4% gc: 840B slow: 536B fast: 0B > > TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 2KB slow > allocs: 1 refill waste: 40B alloc: 0.00003 130KB refills: 15 > waste 5.4% gc: 1696B slow: 448B fast: 0B > > TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00000 9KB refills: 3 > waste 28.2% gc: 1640B slow: 104B fast: 0B > > TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.99946 4187926KB > refills: 3059 waste 0.0% gc: 0B slow: 129152B fast: 0B > > TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 9KB slow > allocs: 3 refill waste: 152B alloc: 0.00011 481KB refills: 40 > waste 2.6% gc: 8528B slow: 1688B fast: 0B > > TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 13KB slow > allocs: 21 refill waste: 216B alloc: 0.00016 677KB refills: 35 > waste 3.0% gc: 12192B slow: 2536B fast: 0B > > TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 2KB slow > allocs: 9 refill waste: 32B alloc: 0.00001 32KB refills: 12 > waste 9.6% gc: 1952B slow: 416B fast: 0B > > TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 2KB slow > allocs: 4 refill waste: 32B alloc: 0.00001 38KB refills: 10 > waste 10.1% gc: 1840B slow: 248B fast: 0B > > TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 20KB slow > allocs: 31 refill waste: 328B alloc: 0.00025 1037KB refills: 42 > waste 0.5% gc: 848B slow: 3408B fast: 0B > > TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow > allocs: 24 refill waste: 48B alloc: 0.00004 171KB refills: 33 > waste 2.2% gc: 1224B slow: 1336B fast: 0B > > TLAB: gc thread: 0x00007f170d2cf000 [id: 84] desired_size: 2KB slow > allocs: 1 refill waste: 64B alloc: 0.00000 9KB refills: 1 > waste 3.1% gc: 64B slow: 0B fast: 0B > > TLAB totals: thrds: 14 refills: 3345 max: 3059 slow allocs: 127 max > 31 waste: 0.0% gc: 35312B max: 12192B slow: 142512B max: 129152B > fast: 0B max: 0B > > > Desired survivor size 268435456 bytes, new threshold 15 (max 15) > > - age 1: 80480 bytes, 80480 total > > - age 2: 12664 bytes, 93144 total > > - age 3: 167000 bytes, 260144 total > > - age 4: 129112 bytes, 389256 total > > - age 5: 21176 bytes, 410432 total > > - age 6: 150024 bytes, 560456 total > > - age 7: 221056 bytes, 781512 total > > - age 8: 283672 bytes, 1065184 total > > - age 9: 9544 bytes, 1074728 total > > - age 10: 345368 bytes, 1420096 total > > - age 11: 93392 bytes, 1513488 total > > - age 12: 10856 bytes, 1524344 total > > - age 13: 50576 bytes, 1574920 total > > - age 14: 205232 bytes, 1780152 total > > - age 15: 51616 bytes, 1831768 total > > 68109.549: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 4450, predicted base time: 9.88 ms, remaining time: > 140.12 ms, target pause time: 150.00 ms] > > 68109.549: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 1023 regions, survivors: 1 regions, predicted young region > time: 2.71 ms] > > 68109.549: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 1023 regions, survivors: 1 regions, old: 0 regions, predicted > pause time: 12.59 ms, target pause time: 150.00 ms] > > 2017-01-25T14:11:05.101+0530: 68109.554: [SoftReference, 0 refs, > 0.0012538 secs]2017-01-25T14:11:05.102+0530: 68109.555: > [WeakReference, 0 refs, 0.0008216 secs]2017-01-25T14:11:05.103+0530: > 68109.556: [FinalReference, 0 refs, 0.0014361 > secs]2017-01-25T14:11:05.105+0530: 68109.557: [PhantomReference, 0 > refs, 0 refs, 0.0016842 secs]2017-01-25T14:11:05.106+0530: 68109.559: > [JNI Weak Reference, 0.0000150 secs], 0.0122237 secs] > > [Parallel Time: 3.7 ms, GC Workers: 20] > > [GC Worker Start (ms): Min: 68109549.4, Avg: 68109550.0, Max: > 68109552.8, Diff: 3.4] > > [Ext Root Scanning (ms): Min: 0.0, Avg: 0.3, Max: 1.1, Diff: > 1.0, Sum: 6.5] > > [Update RS (ms): Min: 0.0, Avg: 0.1, Max: 0.4, Diff: 0.4, Sum: 2.4] > > [Processed Buffers: Min: 0, Avg: 1.5, Max: 5, Diff: 5, Sum: 30] > > [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.2, Sum: 2.3] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > > [Object Copy (ms): Min: 0.0, Avg: 0.4, Max: 0.6, Diff: 0.6, Sum: > 8.4] > > [Termination (ms): Min: 0.0, Avg: 1.9, Max: 2.1, Diff: 2.1, Sum: > 37.6] > > [Termination Attempts: Min: 1, Avg: 9.3, Max: 16, Diff: 15, > Sum: 186] > > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, > Sum: 1.3] > > [GC Worker Total (ms): Min: 0.0, Avg: 2.9, Max: 3.6, Diff: 3.6, > Sum: 58.5] > > [GC Worker End (ms): Min: 68109552.9, Avg: 68109552.9, Max: > 68109553.0, Diff: 0.1] > > [Code Root Fixup: 0.0 ms] > > [Code Root Purge: 0.0 ms] > > [Clear CT: 0.4 ms] > > [Other: 8.1 ms] > > [Choose CSet: 0.0 ms] > > [Ref Proc: 5.9 ms] > > [Ref Enq: 0.3 ms] > > [Redirty Cards: 0.3 ms] > > [Humongous Register: 0.1 ms] > > [Humongous Reclaim: 0.0 ms] > > [Free CSet: 1.3 ms] > > [Eden: 4092.0M(4092.0M)->0.0B(4092.0M) Survivors: 4096.0K->4096.0K > Heap: 5370.5M(8192.0M)->1278.9M(8192.0M)] > > [Times: user=0.00 sys=0.00, real=0.01 secs] > > 2017-01-25T14:11:05.109+0530: 68109.562: Total time for which > application threads were stopped: 0.0126205 seconds, Stopping threads > took: 0.0000411 seconds > > 2017-01-25T14:11:05.149+0530: 68109.602: Application time: 0.0405377 > seconds > > 2017-01-25T14:11:05.150+0530: 68109.602: Total time for which > application threads were stopped: 0.0003956 seconds, Stopping threads > took: 0.0000752 seconds > > 2017-01-25T14:11:05.150+0530: 68109.603: Application time: 0.0002547 > seconds > > 2017-01-25T14:11:05.150+0530: 68109.603: Total time for which > application threads were stopped: 0.0003793 seconds, Stopping threads > took: 0.0000429 seconds > > 2017-01-25T14:11:05.965+0530: 68110.417: Application time: 0.8143539 > seconds > > 2017-01-25T14:11:05.965+0530: 68110.418: Total time for which > application threads were stopped: 0.0002882 seconds, Stopping threads > took: 0.0000777 seconds > > 2017-01-25T14:11:06.149+0530: 68110.602: Application time: 0.1839350 > seconds > > 2017-01-25T14:11:06.149+0530: 68110.602: Total time for which > application threads were stopped: 0.0003335 seconds, Stopping threads > took: 0.0000569 seconds > > 2017-01-25T14:11:06.654+0530: 68111.107: Application time: 0.5045804 > seconds > > 2017-01-25T14:11:06.654+0530: 68111.107: [GC pause (G1 Evacuation > Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] > desired_size: 2KB slow allocs: 21 refill waste: 32B alloc: 0.00002 > 90KB refills: 29 waste 4.5% gc: 1800B slow: 912B fast: 0B > > TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow > allocs: 1 refill waste: 64B alloc: 0.00000 3KB refills: 1 > waste 0.8% gc: 16B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f14d4080000 [id: 197] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 0KB refills: 1 > waste 95.0% gc: 1960B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.99940 4187695KB > refills: 3053 waste 0.0% gc: 0B slow: 131312B fast: 0B > > TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 22KB slow > allocs: 1 refill waste: 352B alloc: 0.00027 1114KB refills: 21 > waste 1.3% gc: 4336B slow: 2072B fast: 0B > > TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow > allocs: 48 refill waste: 48B alloc: 0.00004 151KB refills: 69 > waste 2.4% gc: 2248B slow: 2808B fast: 0B > > TLAB: gc thread: 0x00007f170d35a000 [id: 85] desired_size: 3KB slow > allocs: 230 refill waste: 48B alloc: 0.00004 169KB refills: 553 > waste 1.1% gc: 592B slow: 21040B fast: 0B > > TLAB totals: thrds: 7 refills: 3727 max: 3053 slow allocs: 301 max > 230 waste: 0.0% gc: 10952B max: 4336B slow: 158144B max: 131312B > fast: 0B max: 0B > > > Desired survivor size 268435456 bytes, new threshold 15 (max 15) > > - age 1: 139544 bytes, 139544 total > > - age 2: 71768 bytes, 211312 total > > - age 3: 12664 bytes, 223976 total > > - age 4: 161056 bytes, 385032 total > > - age 5: 128960 bytes, 513992 total > > - age 6: 21176 bytes, 535168 total > > - age 7: 150000 bytes, 685168 total > > - age 8: 221056 bytes, 906224 total > > - age 9: 283672 bytes, 1189896 total > > - age 10: 9544 bytes, 1199440 total > > - age 11: 345368 bytes, 1544808 total > > - age 12: 93392 bytes, 1638200 total > > - age 13: 10856 bytes, 1649056 total > > - age 14: 50576 bytes, 1699632 total > > - age 15: 205232 bytes, 1904864 total > > 68111.107: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 3989, predicted base time: 9.52 ms, remaining time: > 140.48 ms, target pause time: 150.00 ms] > > 68111.107: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 1023 regions, survivors: 1 regions, predicted young region > time: 2.51 ms] > > 68111.107: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 1023 regions, survivors: 1 regions, old: 0 regions, predicted > pause time: 12.03 ms, target pause time: 150.00 ms] > > 2017-01-25T14:11:06.659+0530: 68111.112: [SoftReference, 0 refs, > 0.0052300 secs]2017-01-25T14:11:06.665+0530: 68111.117: > [WeakReference, 0 refs, 0.0008000 secs]2017-01-25T14:11:06.665+0530: > 68111.118: [FinalReference, 0 refs, 0.0006699 > secs]2017-01-25T14:11:06.666+0530: 68111.119: [PhantomReference, 0 > refs, 0 refs, 0.0011966 secs]2017-01-25T14:11:06.667+0530: 68111.120: > [JNI Weak Reference, 0.0000165 secs], 0.0154540 secs] > > [Parallel Time: 1.9 ms, GC Workers: 20] > > [GC Worker Start (ms): Min: 68111107.1, Avg: 68111107.4, Max: > 68111108.7, Diff: 1.6] > > [Ext Root Scanning (ms): Min: 0.0, Avg: 0.5, Max: 1.6, Diff: > 1.5, Sum: 10.9] > > [Update RS (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.3, Sum: 2.4] > > [Processed Buffers: Min: 0, Avg: 1.1, Max: 5, Diff: 5, Sum: 23] > > [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 1.5] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > > [Object Copy (ms): Min: 0.0, Avg: 0.4, Max: 0.6, Diff: 0.6, Sum: > 7.4] > > [Termination (ms): Min: 0.0, Avg: 0.2, Max: 0.3, Diff: 0.3, Sum: > 4.9] > > [Termination Attempts: Min: 1, Avg: 8.5, Max: 17, Diff: 16, > Sum: 170] > > [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, > Sum: 1.0] > > [GC Worker Total (ms): Min: 0.0, Avg: 1.4, Max: 1.7, Diff: 1.6, > Sum: 28.1] > > [GC Worker End (ms): Min: 68111108.7, Avg: 68111108.8, Max: > 68111108.8, Diff: 0.1] > > [Code Root Fixup: 0.0 ms] > > [Code Root Purge: 0.0 ms] > > [Clear CT: 0.4 ms] > > [Other: 13.1 ms] > > [Choose CSet: 0.0 ms] > > [Ref Proc: 10.9 ms] > > [Ref Enq: 0.3 ms] > > [Redirty Cards: 0.2 ms] > > [Humongous Register: 0.1 ms] > > [Humongous Reclaim: 0.0 ms] > > [Free CSet: 1.3 ms] > > [Eden: 4092.0M(4092.0M)->0.0B(4092.0M) Survivors: 4096.0K->4096.0K > Heap: 5370.9M(8192.0M)->1279.1M(8192.0M)] > > [Times: user=0.00 sys=0.00, real=0.02 secs] > > 2017-01-25T14:11:06.670+0530: 68111.122: Total time for which > application threads were stopped: 0.0158568 seconds, Stopping threads > took: 0.0000405 seconds > > 2017-01-25T14:11:07.149+0530: 68111.602: Application time: 0.4792650 > seconds > > 2017-01-25T14:11:07.149+0530: 68111.602: Total time for which > application threads were stopped: 0.0003629 seconds, Stopping threads > took: 0.0000715 seconds > > 2017-01-25T14:11:08.149+0530: 68112.602: Application time: 0.9997711 > seconds > > 2017-01-25T14:11:08.150+0530: 68112.602: Total time for which > application threads were stopped: 0.0003702 seconds, Stopping threads > took: 0.0000820 seconds > > 2017-01-25T14:11:08.209+0530: 68112.662: Application time: 0.0597830 > seconds > > 2017-01-25T14:11:08.210+0530: 68112.662: [GC pause (G1 Evacuation > Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] > desired_size: 2KB slow allocs: 25 refill waste: 32B alloc: 0.00002 > 83KB refills: 43 waste 2.8% gc: 1064B slow: 1392B fast: 0B > > TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00000 3KB refills: 2 > waste 16.1% gc: 648B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00002 98KB refills: 1 > waste 98.8% gc: 2040B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.99938 4187619KB > refills: 3060 waste 0.0% gc: 0B slow: 130440B fast: 0B > > TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 16KB slow > allocs: 0 refill waste: 264B alloc: 0.00020 838KB refills: 12 > waste 4.3% gc: 7664B slow: 1168B fast: 0B > > TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow > allocs: 50 refill waste: 56B alloc: 0.00004 176KB refills: 56 > waste 2.9% gc: 3152B slow: 2680B fast: 0B > > TLAB totals: thrds: 6 refills: 3174 max: 3060 slow allocs: 76 max 50 > waste: 0.0% gc: 14568B max: 7664B slow: 135696B max: 130440B fast: 0B > max: 0B > > > Desired survivor size 268435456 bytes, new threshold 15 (max 15) > > - age 1: 67760 bytes, 67760 total > > - age 2: 133664 bytes, 201424 total > > - age 3: 71768 bytes, 273192 total > > - age 4: 12600 bytes, 285792 total > > - age 5: 160072 bytes, 445864 total > > - age 6: 128960 bytes, 574824 total > > - age 7: 21176 bytes, 596000 total > > - age 8: 150000 bytes, 746000 total > > - age 9: 221056 bytes, 967056 total > > - age 10: 283672 bytes, 1250728 total > > - age 11: 9544 bytes, 1260272 total > > - age 12: 345368 bytes, 1605640 total > > - age 13: 93392 bytes, 1699032 total > > - age 14: 10856 bytes, 1709888 total > > - age 15: 50576 bytes, 1760464 total > > 68112.662: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 3372, predicted base time: 11.25 ms, remaining time: > 138.75 ms, target pause time: 150.00 ms] > > 68112.662: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 1023 regions, survivors: 1 regions, predicted young region > time: 2.26 ms] > > 68112.662: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 1023 regions, survivors: 1 regions, old: 0 regions, predicted > pause time: 13.52 ms, target pause time: 150.00 ms] > > 2017-01-25T14:11:08.219+0530: 68112.672: [SoftReference, 0 refs, > 0.0012482 secs]2017-01-25T14:11:08.220+0530: 68112.673: > [WeakReference, 0 refs, 0.0006781 secs]2017-01-25T14:11:08.221+0530: > 68112.674: [FinalReference, 0 refs, 0.0006530 > secs]2017-01-25T14:11:08.222+0530: 68112.674: [PhantomReference, 0 > refs, 0 refs, 0.0012487 secs]2017-01-25T14:11:08.223+0530: 68112.676: > [JNI Weak Reference, 0.0000147 secs], 0.0157257 secs] > > [Parallel Time: 8.8 ms, GC Workers: 20] > > [GC Worker Start (ms): Min: 68112662.5, Avg: 68112663.8, Max: > 68112671.0, Diff: 8.5] > > [Ext Root Scanning (ms): Min: 0.0, Avg: 0.3, Max: 1.1, Diff: > 1.1, Sum: 5.4] > > [Update RS (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.3, Sum: 2.9] > > [Processed Buffers: Min: 0, Avg: 1.0, Max: 8, Diff: 8, Sum: 20] > > [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 1.5] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > > [Object Copy (ms): Min: 0.0, Avg: 0.4, Max: 0.5, Diff: 0.5, Sum: > 7.2] > > [Termination (ms): Min: 0.0, Avg: 6.5, Max: 7.3, Diff: 7.3, Sum: > 129.5] > > [Termination Attempts: Min: 1, Avg: 11.1, Max: 19, Diff: 18, > Sum: 221] > > [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, > Sum: 0.8] > > [GC Worker Total (ms): Min: 0.0, Avg: 7.4, Max: 8.6, Diff: 8.5, > Sum: 147.3] > > [GC Worker End (ms): Min: 68112671.1, Avg: 68112671.1, Max: > 68112671.2, Diff: 0.1] > > [Code Root Fixup: 0.0 ms] > > [Code Root Purge: 0.0 ms] > > [Clear CT: 0.4 ms] > > [Other: 6.5 ms] > > [Choose CSet: 0.0 ms] > > [Ref Proc: 4.3 ms] > > [Ref Enq: 0.3 ms] > > [Redirty Cards: 0.3 ms] > > [Humongous Register: 0.0 ms] > > [Humongous Reclaim: 0.0 ms] > > [Free CSet: 1.3 ms] > > [Eden: 4092.0M(4092.0M)->0.0B(4092.0M) Survivors: 4096.0K->4096.0K > Heap: 5371.1M(8192.0M)->1279.3M(8192.0M)] > > [Times: user=0.00 sys=0.00, real=0.01 secs] > > 2017-01-25T14:11:08.226+0530: 68112.678: Total time for which > application threads were stopped: 0.0161537 seconds, Stopping threads > took: 0.0000417 seconds > > 2017-01-25T14:11:09.149+0530: 68113.602: Application time: 0.9238398 > seconds > > 2017-01-25T14:11:09.150+0530: 68113.602: Total time for which > application threads were stopped: 0.0003769 seconds, Stopping threads > took: 0.0000805 seconds > > 2017-01-25T14:11:09.778+0530: 68114.230: Application time: 0.6279838 > seconds > > 2017-01-25T14:11:09.778+0530: 68114.231: [GC pause (G1 Evacuation > Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] > desired_size: 2KB slow allocs: 16 refill waste: 32B alloc: 0.00002 > 90KB refills: 22 waste 4.5% gc: 1328B slow: 696B fast: 0B > > TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 3KB refills: 1 > waste 14.0% gc: 288B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f1590687800 [id: 125] desired_size: 2KB slow > allocs: 6 refill waste: 32B alloc: 0.00002 63KB refills: 33 > waste 4.2% gc: 1760B slow: 1072B fast: 0B > > TLAB: gc thread: 0x00007f159067c800 [id: 120] desired_size: 3KB slow > allocs: 0 refill waste: 48B alloc: 0.00004 166KB refills: 9 > waste 3.3% gc: 712B slow: 280B fast: 0B > > TLAB: gc thread: 0x00007f158c194800 [id: 116] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 7KB refills: 7 > waste 2.5% gc: 192B slow: 168B fast: 0B > > TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.99955 4188322KB > refills: 3053 waste 0.0% gc: 0B slow: 131056B fast: 0B > > TLAB: gc thread: 0x00007f1594010000 [id: 111] desired_size: 8KB slow > allocs: 2 refill waste: 136B alloc: 0.00011 444KB refills: 23 > waste 1.7% gc: 2336B slow: 1280B fast: 0B > > TLAB: gc thread: 0x00007f159400f000 [id: 110] desired_size: 13KB slow > allocs: 3 refill waste: 208B alloc: 0.00016 650KB refills: 19 > waste 2.7% gc: 5408B slow: 1312B fast: 0B > > TLAB: gc thread: 0x00007f1594008000 [id: 109] desired_size: 2KB slow > allocs: 3 refill waste: 32B alloc: 0.00001 30KB refills: 6 > waste 11.6% gc: 1280B slow: 152B fast: 0B > > TLAB: gc thread: 0x00007f1594007000 [id: 108] desired_size: 2KB slow > allocs: 2 refill waste: 32B alloc: 0.00001 32KB refills: 7 > waste 14.4% gc: 1888B slow: 192B fast: 0B > > TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 11KB slow > allocs: 29 refill waste: 184B alloc: 0.00014 590KB refills: 114 > waste 0.9% gc: 4144B slow: 8136B fast: 0B > > TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow > allocs: 20 refill waste: 56B alloc: 0.00004 187KB refills: 31 > waste 4.3% gc: 3776B slow: 1336B fast: 0B > > TLAB totals: thrds: 12 refills: 3325 max: 3053 slow allocs: 81 max 29 > waste: 0.0% gc: 23112B max: 5408B slow: 145680B max: 131056B fast: 0B > max: 0B > > > Desired survivor size 268435456 bytes, new threshold 15 (max 15) > > - age 1: 22472 bytes, 22472 total > > - age 2: 64576 bytes, 87048 total > > - age 3: 133664 bytes, 220712 total > > - age 4: 71768 bytes, 292480 total > > - age 5: 12600 bytes, 305080 total > > - age 6: 160072 bytes, 465152 total > > - age 7: 128960 bytes, 594112 total > > - age 8: 21176 bytes, 615288 total > > - age 9: 150000 bytes, 765288 total > > - age 10: 221056 bytes, 986344 total > > - age 11: 283672 bytes, 1270016 total > > - age 12: 9544 bytes, 1279560 total > > - age 13: 345368 bytes, 1624928 total > > - age 14: 93392 bytes, 1718320 total > > - age 15: 10856 bytes, 1729176 total > > 68114.231: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 3658, predicted base time: 10.42 ms, remaining time: > 139.58 ms, target pause time: 150.00 ms] > > 68114.231: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 1023 regions, survivors: 1 regions, predicted young region > time: 2.10 ms] > > 68114.231: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 1023 regions, survivors: 1 regions, old: 0 regions, predicted > pause time: 12.53 ms, target pause time: 150.00 ms] > > 2017-01-25T14:11:09.782+0530: 68114.235: [SoftReference, 0 refs, > 0.0011872 secs]2017-01-25T14:11:09.783+0530: 68114.236: > [WeakReference, 0 refs, 0.0008379 secs]2017-01-25T14:11:09.784+0530: > 68114.237: [FinalReference, 0 refs, 0.0037026 > secs]2017-01-25T14:11:09.788+0530: 68114.240: [PhantomReference, 0 > refs, 0 refs, 0.0015786 secs]2017-01-25T14:11:09.789+0530: 68114.242: > [JNI Weak Reference, 0.0000149 secs], 0.0138869 secs] > > [Parallel Time: 3.1 ms, GC Workers: 20] > > [GC Worker Start (ms): Min: 68114230.9, Avg: 68114231.7, Max: > 68114233.6, Diff: 2.7] > > [Ext Root Scanning (ms): Min: 0.0, Avg: 0.3, Max: 1.0, Diff: > 1.0, Sum: 5.7] > > [Update RS (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.3, Sum: 2.3] > > [Processed Buffers: Min: 0, Avg: 1.2, Max: 5, Diff: 5, Sum: 25] > > [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 1.6] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > > [Object Copy (ms): Min: 0.0, Avg: 0.9, Max: 1.4, Diff: 1.4, Sum: > 17.7] > > [Termination (ms): Min: 0.0, Avg: 0.6, Max: 0.8, Diff: 0.8, Sum: > 11.7] > > [Termination Attempts: Min: 1, Avg: 96.5, Max: 174, Diff: > 173, Sum: 1931] > > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, > Sum: 1.2] > > [GC Worker Total (ms): Min: 0.0, Avg: 2.0, Max: 2.9, Diff: 2.9, > Sum: 40.3] > > [GC Worker End (ms): Min: 68114233.7, Avg: 68114233.7, Max: > 68114233.8, Diff: 0.1] > > [Code Root Fixup: 0.0 ms] > > [Code Root Purge: 0.0 ms] > > [Clear CT: 0.5 ms] > > [Other: 10.2 ms] > > [Choose CSet: 0.0 ms] > > [Ref Proc: 8.0 ms] > > [Ref Enq: 0.4 ms] > > [Redirty Cards: 0.3 ms] > > [Humongous Register: 0.1 ms] > > [Humongous Reclaim: 0.0 ms] > > [Free CSet: 1.3 ms] > > [Eden: 4092.0M(4092.0M)->0.0B(4092.0M) Survivors: 4096.0K->4096.0K > Heap: 5371.3M(8192.0M)->1279.4M(8192.0M)] > > [Times: user=0.00 sys=0.00, real=0.01 secs] > > 2017-01-25T14:11:09.792+0530: 68114.245: Total time for which > application threads were stopped: 0.0142584 seconds, Stopping threads > took: 0.0000398 seconds > > 2017-01-25T14:11:10.150+0530: 68114.602: Application time: 0.3575559 > seconds > > 2017-01-25T14:11:10.150+0530: 68114.603: Total time for which > application threads were stopped: 0.0003786 seconds, Stopping threads > took: 0.0000693 seconds > > 2017-01-25T14:11:10.965+0530: 68115.418: Application time: 0.8153954 > seconds > > 2017-01-25T14:11:10.966+0530: 68115.418: Total time for which > application threads were stopped: 0.0003303 seconds, Stopping threads > took: 0.0000827 seconds > > 2017-01-25T14:11:11.150+0530: 68115.602: Application time: 0.1840399 > seconds > > 2017-01-25T14:11:11.150+0530: 68115.603: Total time for which > application threads were stopped: 0.0003683 seconds, Stopping threads > took: 0.0000798 seconds > > 2017-01-25T14:11:11.365+0530: 68115.817: Application time: 0.2147411 > seconds > > 2017-01-25T14:11:11.365+0530: 68115.818: [GC pause (G1 Evacuation > Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] > desired_size: 2KB slow allocs: 26 refill waste: 32B alloc: 0.00002 > 76KB refills: 43 waste 2.8% gc: 1064B slow: 1408B fast: 0B > > TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow > allocs: 1 refill waste: 32B alloc: 0.00000 3KB refills: 2 > waste 16.1% gc: 648B slow: 16B fast: 0B > > TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.99952 4188180KB > refills: 3064 waste 0.0% gc: 0B slow: 131896B fast: 0B > > TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 17KB slow > allocs: 0 refill waste: 272B alloc: 0.00021 870KB refills: 4 > waste 20.9% gc: 14840B slow: 88B fast: 0B > > TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow > allocs: 47 refill waste: 48B alloc: 0.00004 163KB refills: 60 > waste 1.8% gc: 1136B slow: 2456B fast: 0B > > TLAB: gc thread: 0x00007f170d35a000 [id: 85] desired_size: 15KB slow > allocs: 0 refill waste: 248B alloc: 0.00019 791KB refills: 2 > waste 8.6% gc: 2760B slow: 16B fast: 0B > > TLAB totals: thrds: 6 refills: 3175 max: 3064 slow allocs: 74 max 47 > waste: 0.0% gc: 20448B max: 14840B slow: 135880B max: 131896B fast: > 0B max: 0B > > > Desired survivor size 268435456 bytes, new threshold 15 (max 15) > > - age 1: 71216 bytes, 71216 total > > - age 2: 17024 bytes, 88240 total > > - age 3: 64368 bytes, 152608 total > > - age 4: 128960 bytes, 281568 total > > - age 5: 69248 bytes, 350816 total > > - age 6: 12600 bytes, 363416 total > > - age 7: 160048 bytes, 523464 total > > - age 8: 128920 bytes, 652384 total > > - age 9: 21176 bytes, 673560 total > > - age 10: 150000 bytes, 823560 total > > - age 11: 221056 bytes, 1044616 total > > - age 12: 283672 bytes, 1328288 total > > - age 13: 9544 bytes, 1337832 total > > - age 14: 345368 bytes, 1683200 total > > - age 15: 93392 bytes, 1776592 total > > 68115.818: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 3358, predicted base time: 10.77 ms, remaining time: > 139.23 ms, target pause time: 150.00 ms] > > 68115.818: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 1023 regions, survivors: 1 regions, predicted young region > time: 2.18 ms] > > 68115.818: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 1023 regions, survivors: 1 regions, old: 0 regions, predicted > pause time: 12.95 ms, target pause time: 150.00 ms] > > 2017-01-25T14:11:11.369+0530: 68115.821: [SoftReference, 0 refs, > 0.0019318 secs]2017-01-25T14:11:11.371+0530: 68115.823: > [WeakReference, 0 refs, 0.0007565 secs]2017-01-25T14:11:11.371+0530: > 68115.824: [FinalReference, 0 refs, 0.0064296 > secs]2017-01-25T14:11:11.378+0530: 68115.830: [PhantomReference, 0 > refs, 0 refs, 0.0012517 secs]2017-01-25T14:11:11.379+0530: 68115.832: > [JNI Weak Reference, 0.0000150 secs], 0.0164458 secs] > > [Parallel Time: 2.9 ms, GC Workers: 20] > > [GC Worker Start (ms): Min: 68115817.9, Avg: 68115818.6, Max: > 68115820.2, Diff: 2.3] > > [Ext Root Scanning (ms): Min: 0.0, Avg: 0.5, Max: 2.5, Diff: > 2.5, Sum: 10.2] > > [Update RS (ms): Min: 0.0, Avg: 0.1, Max: 0.4, Diff: 0.4, Sum: 2.4] > > [Processed Buffers: Min: 0, Avg: 1.0, Max: 4, Diff: 4, Sum: 20] > > [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 1.6] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > > [Object Copy (ms): Min: 0.0, Avg: 0.7, Max: 1.1, Diff: 1.1, Sum: > 14.0] > > [Termination (ms): Min: 0.0, Avg: 0.5, Max: 0.7, Diff: 0.7, Sum: > 9.9] > > [Termination Attempts: Min: 1, Avg: 72.0, Max: 131, Diff: > 130, Sum: 1441] > > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, > Sum: 1.1] > > [GC Worker Total (ms): Min: 0.3, Avg: 2.0, Max: 2.7, Diff: 2.4, > Sum: 39.3] > > [GC Worker End (ms): Min: 68115820.5, Avg: 68115820.6, Max: > 68115820.7, Diff: 0.1] > > [Code Root Fixup: 0.0 ms] > > [Code Root Purge: 0.0 ms] > > [Clear CT: 0.4 ms] > > [Other: 13.1 ms] > > [Choose CSet: 0.0 ms] > > [Ref Proc: 10.8 ms] > > [Ref Enq: 0.3 ms] > > [Redirty Cards: 0.3 ms] > > [Humongous Register: 0.1 ms] > > [Humongous Reclaim: 0.0 ms] > > [Free CSet: 1.4 ms] > > [Eden: 4092.0M(4092.0M)->0.0B(4092.0M) Survivors: 4096.0K->4096.0K > Heap: 5371.4M(8192.0M)->1279.2M(8192.0M)] > > [Times: user=0.00 sys=0.00, real=0.02 secs] > > 2017-01-25T14:11:11.382+0530: 68115.834: Total time for which > application threads were stopped: 0.0168706 seconds, Stopping threads > took: 0.0000412 seconds > > 2017-01-25T14:11:12.150+0530: 68116.602: Application time: 0.7681393 > seconds > > 2017-01-25T14:11:12.150+0530: 68116.603: Total time for which > application threads were stopped: 0.0003937 seconds, Stopping threads > took: 0.0001038 seconds > > 2017-01-25T14:11:12.956+0530: 68117.408: Application time: 0.8054451 > seconds > > 2017-01-25T14:11:12.956+0530: 68117.409: [GC pause (G1 Evacuation > Pause) (young)TLAB: gc thread: 0x00007f14d0003000 [id: 211] > desired_size: 2KB slow allocs: 15 refill waste: 32B alloc: 0.00002 > 86KB refills: 22 waste 4.4% gc: 1328B slow: 680B fast: 0B > > TLAB: gc thread: 0x00007f14d4084800 [id: 198] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 3KB refills: 1 > waste 14.0% gc: 288B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f159062b000 [id: 115] desired_size: 2KB slow > allocs: 0 refill waste: 32B alloc: 0.00000 0KB refills: 1 > waste 92.2% gc: 1904B slow: 0B fast: 0B > > TLAB: gc thread: 0x00007f1594012800 [id: 112] desired_size: 2047KB > slow allocs: 0 refill waste: 32760B alloc: 0.99964 4188703KB > refills: 3062 waste 0.0% gc: 0B slow: 129160B fast: 0B > > TLAB: gc thread: 0x00007f1594003000 [id: 107] desired_size: 11KB slow > allocs: 1 refill waste: 184B alloc: 0.00014 581KB refills: 18 > waste 1.6% gc: 2080B slow: 1336B fast: 0B > > TLAB: gc thread: 0x00007f170d46d000 [id: 105] desired_size: 3KB slow > allocs: 21 refill waste: 56B alloc: 0.00004 177KB refills: 26 > waste 5.2% gc: 3584B slow: 1296B fast: 0B > > TLAB totals: thrds: 6 refills: 3130 max: 3062 slow allocs: 37 max 21 > waste: 0.0% gc: 9184B max: 3584B slow: 132472B max: 129160B fast: 0B > max: 0B > > > Desired survivor size 268435456 bytes, new threshold 15 (max 15) > > - age 1: 17392 bytes, 17392 total > > - age 2: 67720 bytes, 85112 total > > - age 3: 17024 bytes, 102136 total > > - age 4: 64304 bytes, 166440 total > > - age 5: 128960 bytes, 295400 total > > - age 6: 69248 bytes, 364648 total > > - age 7: 12600 bytes, 377248 total > > - age 8: 160048 bytes, 537296 total > > - age 9: 128920 bytes, 666216 total > > - age 10: 21176 bytes, 687392 total > > - age 11: 150000 bytes, 837392 total > > - age 12: 221056 bytes, 1058448 total > > - age 13: 283672 bytes, 1342120 total > > - age 14: 9544 bytes, 1351664 total > > - age 15: 345368 bytes, 1697032 total > > 68117.409: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 3373, predicted base time: 12.09 ms, remaining time: > 137.91 ms, target pause time: 150.00 ms] > > 68117.409: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 1023 regions, survivors: 1 regions, predicted young region > time: 2.04 ms] > > 68117.409: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 1023 regions, survivors: 1 regions, old: 0 regions, predicted > pause time: 14.13 ms, target pause time: 150.00 ms] > > 2017-01-25T14:11:12.959+0530: 68117.412: [SoftReference, 0 refs, > 0.0010997 secs]2017-01-25T14:11:12.960+0530: 68117.413: > [WeakReference, 0 refs, 0.0057308 secs]2017-01-25T14:11:12.966+0530: > 68117.419: [FinalReference, 0 refs, 0.0006110 > secs]2017-01-25T14:11:12.967+0530: 68117.419: [PhantomReference, 0 > refs, 0 refs, 0.0013420 secs]2017-01-25T14:11:12.968+0530: 68117.421: > [JNI Weak Reference, 0.0000150 secs], 0.0145345 secs] > > [Parallel Time: 2.6 ms, GC Workers: 20] > > [GC Worker Start (ms): Min: 68117408.8, Avg: 68117409.4, Max: > 68117411.1, Diff: 2.3] > > [Ext Root Scanning (ms): Min: 0.0, Avg: 0.3, Max: 1.1, Diff: > 1.1, Sum: 6.3] > > [Update RS (ms): Min: 0.0, Avg: 0.1, Max: 0.3, Diff: 0.3, Sum: 2.4] > > [Processed Buffers: Min: 0, Avg: 1.0, Max: 5, Diff: 5, Sum: 20] > > [Scan RS (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 1.7] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > > [Object Copy (ms): Min: 0.0, Avg: 0.7, Max: 1.1, Diff: 1.1, Sum: > 15.0] > > [Termination (ms): Min: 0.0, Avg: 0.4, Max: 0.6, Diff: 0.6, Sum: > 8.6] > > [Termination Attempts: Min: 1, Avg: 66.7, Max: 103, Diff: > 102, Sum: 1333] > > [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.2, Diff: 0.1, > Sum: 1.5] > > [GC Worker Total (ms): Min: 0.0, Avg: 1.8, Max: 2.4, Diff: 2.4, > Sum: 35.5] > > [GC Worker End (ms): Min: 68117411.2, Avg: 68117411.2, Max: > 68117411.3, Diff: 0.1] > > [Code Root Fixup: 0.0 ms] > > [Code Root Purge: 0.0 ms] > > [Clear CT: 0.5 ms] > > [Other: 11.4 ms] > > [Choose CSet: 0.0 ms] > > [Ref Proc: 9.3 ms] > > [Ref Enq: 0.4 ms] > > [Redirty Cards: 0.2 ms] > > [Humongous Register: 0.1 ms] > > [Humongous Reclaim: 0.0 ms] > > [Free CSet: 1.3 ms] > > [Eden: 4092.0M(4092.0M)->0.0B(4092.0M) Survivors: 4096.0K->4096.0K > Heap: 5371.2M(8192.0M)->1279.1M(8192.0M)] > > [Times: user=0.00 sys=0.00, real=0.02 secs] > > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: From ceeaspb at gmail.com Wed Jan 25 18:24:30 2017 From: ceeaspb at gmail.com (Alex Bagehot) Date: Wed, 25 Jan 2017 18:24:30 +0000 Subject: Need help on G1 GC young gen Update RS and Scan RS pause reduction In-Reply-To: References: <1484852604.6579.27.camel@oracle.com> Message-ID: Hi Amit, On Fri, Jan 20, 2017 at 10:51 AM, Amit Mishra wrote: > Thank you very much Thomas, after making said changes RSUpdate time is in > control but Scan RS is still high. > > I also observed that sys time is very high but my system has sufficient > memory and there has not been a single swapping/paging happened on node, Not > sure how to troubleshoot this high sys time. High system/kernel time, Solaris -> DTrace It won't give you the answer directly but it should at least provide thread/task stacks indicating which user land and kernel functions are involved when high sys time is reported, which may help. > > 244.719: [GC pause (young), 1.3896037 secs] > [Parallel Time: 1356.6 ms, GC Workers: 53] > [GC Worker Start (ms): Min: 244718.9, Avg: 244719.3, Max: 244719.8, > Diff: 0.9] > [Ext Root Scanning (ms): Min: 2.2, Avg: 3.0, Max: 3.8, Diff: 1.6, Sum: > 156.7] > [Update RS (ms): Min: 13.3, Avg: 16.3, Max: 47.0, Diff: 33.6, Sum: > 866.5] > [Processed Buffers: Min: 3, Avg: 6.8, Max: 17, Diff: 14, Sum: 361] > [Scan RS (ms): Min: 972.9, Avg: 1002.8, Max: 1006.0, Diff: 33.1, Sum: > 53149.3] > [Object Copy (ms): Min: 332.1, Avg: 333.1, Max: 334.5, Diff: 2.4, Sum: > 17656.0] > [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 1.0] > [GC Worker Other (ms): Min: 0.0, Avg: 0.2, Max: 0.5, Diff: 0.4, Sum: > 11.5] > [GC Worker Total (ms): Min: 1354.9, Avg: 1355.5, Max: 1356.1, Diff: > 1.2, Sum: 71841.1] > [GC Worker End (ms): Min: 246074.7, Avg: 246074.8, Max: 246075.1, > Diff: 0.4] > [Code Root Fixup: 0.0 ms] > [Clear CT: 2.1 ms] > [Other: 30.8 ms] > [Choose CSet: 0.0 ms] > [Ref Proc: 10.3 ms] > [Ref Enq: 1.6 ms] > [Free CSet: 1.6 ms] > [Eden: 2848.0M(1856.0M)->0.0B(224.0M) Survivors: 192.0M->256.0M Heap: > 10.3G(48.0G)->9324.6M(48.0G)] > [Times: user=16.93 sys=1.07, real=1.39 secs] > > > Thanks, > Amit Mishra > > -----Original Message----- > From: Thomas Schatzl [mailto:thomas.schatzl at oracle.com] > Sent: Friday, January 20, 2017 00:33 > To: Amit Mishra ; hotspot-gc-use at openjdk.java.net > Subject: Re: Need help on G1 GC young gen Update RS and Scan RS pause > reduction > > Hi Amit, > > On Thu, 2017-01-19 at 13:05 +0000, Amit Mishra wrote: >> Hello Friends, >> >> I am using G1 GC on Java 1.7.45 version in my solaris LAB environment >> but am getting very high UpdateRS time and few occasions high Scan RS >> time as well. > > just as reminder, consider at least the latest version of 1.7. There were > a few interesting changes to G1 later than 1.7.45. > > JDK8 would even be better of course :) > >> OS version : Generic_150401-30 i86pc i386 i86pc Number of CPU # 80, A >> day back i have tried to play with below options hoping extra GC >> threads will help me but all of no avail as I had witnessed long >> pauses up to 7 seconds. Right now I have disabled these GC threads. >> >> argv[52]: -XX:ParallelGCThreads=70 >> argv[53]: -XX:ConcGCThreads=60 >> argv[54]: -XX:G1ConcRefinementThreads=70 > > Unless I misunderstand what a virtual cpu in Solaris is, I do not think > using more than there are (64 from an earlier email) gives you any > advantage. > >> I have already put -XX:G1RSetUpdatingPauseTimePercent=5 option and >> relaxed pause time goal by setting it to 500 ms but it doesn?t seems >> to be helping me. > > There is one (compiler) optimization that may generate lots of work for the > Update RS phase called ReduceInitialCardMarks. Basically it tells > g1 to revisit the entire array concurrently. If this allocation happens just > before a GC, the gc needs to do all the work in the pause. > > You can try the option that disables this optimization, i.e. set -XX:- > ReduceInitialCardMarks. Of course it is kind of a shot in the dark, and only > if you know that your application allocates (a lot) of largish objects. > However, this helped in other cases of spurious long Update RS pauses. > > Also note that I am not sure whether disabling this option causes crashes in > 7u - there were some bugfixes in 9 for that. > > Another option (that should help at least a bit) to reduce update rs/scan rs > times may be increasing heap region size to 32M (- XX:G1HeapRegionSize=32M). > >> Your urgent help will be highly appreciated as I need system where >> avg. pause time remain around 200-300 milli-second and Max pause >> remain around 500-600 ms. >> >> Current GC parameters are as below: >> argv[11]: -Xmx48g >> argv[12]: -Xms48g >> argv[13]: -XX:-EliminateLocks >> argv[24]: -Xss1m >> argv[25]: -Xoss1m >> argv[26]: -XX:PermSize=512m >> argv[27]: -XX:MaxPermSize=512m >> argv[28]: -XX:ReservedCodeCacheSize=128m >> argv[29]: -XX:+HeapDumpOnOutOfMemoryError >> argv[30]: -XX:+AggressiveOpts >> argv[35]: -XX:+UseG1GC >> argv[36]: -XX:MaxGCPauseMillis=500 >> argv[37]: -XX:+PrintFlagsFinal >> argv[38]: -XX:G1RSetUpdatingPauseTimePercent=5 >> argv[39]: -XX:+PrintGCTimeStamps >> argv[40]: -XX:+PrintGCDetails >> argv[42]: -verbose:gc >> argv[43]: -XX:+UseLargePages >> argv[44]: -XX:+MaxFDLimit >> argv[48]: -XX:+UnlockExperimentalVMOptions >> argv[49]: -XX:G1NewSizePercent=1 >> argv[50]: -XX:+ParallelRefProcEnabled >> argv[51]: -XX:+DisableExplicitGC >> argv[52]: -XX:G1MaxNewSizePercent=1 > > I am not sure if setting the maximum new size is a good idea (causing long > Object Copy *mostly*). It might be useful to just remove this one. > >> argv[53]: -XX:InitiatingHeapOccupancyPercent=40 >> argv[54]: -XX:+UnlockDiagnosticVMOptions >> argv[55]: -XX:+G1SummarizeRSetStats >> argv[56]: -XX:G1SummarizeRSetStatsPeriod=1 >> argv[57]: -XX:+PerfDisableSharedMem >> argv[58]: -XX:+AlwaysPreTouch >> >> >> >> Pause snapshots: >> >> Very high UpdateRS but low scan RS >> >> 2602.655: [GC pause (young), 7.8981373 secs] >> [Parallel Time: 7883.9 ms, GC Workers: 53] >> [GC Worker Start (ms): Min: 2602654.9, Avg: 2602655.3, Max: >> 2602655.7, Diff: 0.8] >> [Ext Root Scanning (ms): Min: 3.5, Avg: 4.5, Max: 34.2, Diff: >> 30.7, Sum: 240.0] >> [Update RS (ms): Min: 7796.5, Avg: 7826.2, Max: 7827.8, Diff: >> 31.4, Sum: 414789.9] >> [Processed Buffers: Min: 1, Avg: 3.8, Max: 12, Diff: 11, >> Sum: 199] >> [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: >> 0.6] >> [Object Copy (ms): Min: 48.2, Avg: 48.8, Max: 49.4, Diff: 1.3, >> Sum: 2588.2] >> [Termination (ms): Min: 3.1, Avg: 3.5, Max: 3.9, Diff: 0.8, >> Sum: 183.8] >> [GC Worker Other (ms): Min: 0.0, Avg: 0.2, Max: 0.4, Diff: 0.3, >> Sum: 8.4] >> [GC Worker Total (ms): Min: 7882.7, Avg: 7883.2, Max: 7883.8, >> Diff: 1.1, Sum: 417810.9] >> [GC Worker End (ms): Min: 2610538.4, Avg: 2610538.5, Max: >> 2610538.7, Diff: 0.3] >> [Code Root Fixup: 0.0 ms] >> [Clear CT: 0.9 ms] >> [Other: 13.3 ms] >> [Choose CSet: 0.0 ms] >> [Ref Proc: 10.5 ms] >> [Ref Enq: 1.0 ms] >> [Free CSet: 0.5 ms] >> [Eden: 432.0M(432.0M)->0.0B(416.0M) Survivors: 48.0M->64.0M Heap: >> 19.3G(48.0G)->18.9G(48.0G)] >> [Times: user=3.24 sys=0.02, real=7.90 secs] > > Something is wrong here, i.e. a a wall time larger than user time is strange > to say at least. Maybe that machine is overloaded with other applications? > > I recommend trying to get to the bottom of this first. > >> Concurrent RS processed 48786945 cards >> Of 1661300 completed buffers: >> 1661300 (100.0%) by conc RS threads. >> 0 ( 0.0%) by mutator threads. >> Conc RS threads times(s) >> 17.55 15.01 12.67 11.26 10.14 9.56 >> 8.13 7.26 6.61 6.18 5.25 4.64 3.97 >> 3.16 2.92 2.76 2.60 2.54 2.36 2.30 2.16 >> 2.12 2.00 1.92 1.83 1.82 1.75 1.69 >> 1.64 1.64 1.55 1.50 1.45 1.48 1.41 >> 1.36 1.32 1.29 1.26 1.22 1.15 1.18 1.12 >> 1.07 1.05 1.03 1.01 0.97 0.96 0.95 >> 0.89 0.89 0.87 0.33 >> Total heap region rem set sizes = 808374K. Max = 4135K. >> Static structures = 1596K, free_lists = 0K. >> 72965736 occupied cards represented. >> Max size region = >> 35:(O)[0xfffffd73e3000000,0xfffffd73e4000000,0xfffffd73e4000000], >> size = 4136K, occupied = 14130K. >> Did 252 coarsenings. > > This may or may not be what causes long scan rs times; you could try > -XX:G1RSetRegionEntries=2048 to avoid coarsenings (and you should remove the > g1SummarizeRSetStats* options again). > > (the 2048 is just a value higher than the number of regions in the heap, at > least with 32m regions). > > Hth, > Thomas > > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > From yu.zhang at oracle.com Wed Jan 25 23:23:08 2017 From: yu.zhang at oracle.com (yu.zhang at oracle.com) Date: Wed, 25 Jan 2017 15:23:08 -0800 Subject: Unreachable Memory not freed In-Reply-To: <83689bdf-7285-3493-e7ab-c46b597bc02a@freigmbh.de> References: <83689bdf-7285-3493-e7ab-c46b597bc02a@freigmbh.de> Message-ID: Thorsten, Thanks very much for the heap dump and gc log. I assume the gc log and heap dump are from the same run. It is very interesting, in the gc log: 2017-01-24T16:41:17.184-0100: 1301.633: [Full GC (Heap Dump Initiated GC) 3817M->3816M(4096M), 8.9543444 secs] [Eden: 2048.0K(2048.0M)->0.0B(2048.0M) Survivors: 0.0B->0.0B Heap: 3817.7M(4096.0M)->3816.2M(4096.0M)], [Metaspace: 142921K->142921K(1175552K)] [Times: user=16.35 sys=0.11, real=8.95 secs] GC thinks there are 3816.2M of objects alive. But in the heap dump, there is only 1.8G alive. I can see those unreachable objects with yourkit, mostly are org.apache.batik.dom.GenerecText But apparently gc does not think those are dead. It could be a bug in gc, or jvmti dumping the heap, or tool(yourkit, apache MAT). I need to discuss this with dev team. Back to the gc log, it seems the application allocates a lot of humongous objects. I can see some int[] of 1.5m. In general the heap is very full, with a lot of to-space exhausted, and full gc. And the heap usage after full gc is almost the whole heap (4096M). Thanks Jenny On 01/25/2017 01:15 AM, Thorsten Goetzke wrote: > Hello, > > You can find the heap dump and gc logs under > https://filesender.freigmbh.de/filesender/?vid=10f1aeb6-82b4-3d09-c201-00002042c681 > > Link is valid until 28.01.2017 (1/28/2017). Feel free to message me if > you need a reupload. > > Best Regards, > Thorsten > > >> Thorsten, >> >> Can you please put the heap dump and gc logs somewhere for download? >> >> Thanks >> Jenny >> >> On 1/24/2017 8:30 AM, Thorsten Goetzke wrote: >>> Hello, >>> >>> I am working on a closed source software that uses a patched version >>> of apache-batik to display SVG-Documents one by one. If you keep >>> loading and discarding some specific SVG-Documents memory consumption >>> keeps on growing until it reaches more than 95% of the available >>> memory. The application becomes unusable slow, very little memory gets >>> freed, although the gc is constantly running. Sometimes an >>> OutOfMemoryError gets thrown. I took memory snaphsots and analyzed >>> them using jvisualvm and yourkit. These tools report the majority of >>> the memory as unreachable. >>> I have tried different garbage collectors (G1,ConcurrentMarkSweep), >>> and different Versions of Java: 1.8_60;1.8_65;1.8_102, the issue is >>> always reproducible. >>> Is there something i can do to debug this further? I am currently not >>> able to reproduce this issue in a somewhat small self contained >>> example. >>> I could provide gc-logs and the heap dump (about 1gb zipped) or any >>> other debuging info if someone is interested. >>> >>> Best Regards, >>> Thorsten Goetzke >>> > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From yu.zhang at oracle.com Sat Jan 28 00:50:15 2017 From: yu.zhang at oracle.com (Jenny Zhang) Date: Fri, 27 Jan 2017 16:50:15 -0800 Subject: Unreachable Memory not freed In-Reply-To: References: <83689bdf-7285-3493-e7ab-c46b597bc02a@freigmbh.de> Message-ID: <349a5bf5-9c94-4578-5ad5-d13d0c3afc3a@oracle.com> Hi, Thorsten, I can reproduce this with a micro. I have filed https://bugs.openjdk.java.net/browse/JDK-8173594 The micro is mostly based on http://stackoverflow.com/questions/26027313/how-to-load-and-parse-svg-documents with some modification. I will upload the micro and details later. I think this is a issue how jvmti and gc reporting unreachable objects. Those objects should not be reported as unreachable. Thanks Jenny On 1/25/2017 3:23 PM, yu.zhang at oracle.com wrote: > Thorsten, > > Thanks very much for the heap dump and gc log. > > I assume the gc log and heap dump are from the same run. It is very > interesting, in the gc log: > > 2017-01-24T16:41:17.184-0100: 1301.633: [Full GC (Heap Dump Initiated > GC) 3817M->3816M(4096M), 8.9543444 secs] > [Eden: 2048.0K(2048.0M)->0.0B(2048.0M) Survivors: 0.0B->0.0B Heap: > 3817.7M(4096.0M)->3816.2M(4096.0M)], [Metaspace: > 142921K->142921K(1175552K)] > [Times: user=16.35 sys=0.11, real=8.95 secs] > > GC thinks there are 3816.2M of objects alive. But in the heap dump, > there is only 1.8G alive. > > I can see those unreachable objects with yourkit, mostly are > > org.apache.batik.dom.GenerecText > > But apparently gc does not think those are dead. It could be a bug in > gc, or jvmti dumping the heap, or tool(yourkit, apache MAT). I need to > discuss this with dev team. > > Back to the gc log, it seems the application allocates a lot of > humongous objects. I can see some int[] of 1.5m. In general the heap > is very full, with a lot of to-space exhausted, and full gc. And the > heap usage after full gc is almost the whole heap (4096M). > > Thanks > > Jenny > > > On 01/25/2017 01:15 AM, Thorsten Goetzke wrote: >> Hello, >> >> You can find the heap dump and gc logs under >> https://filesender.freigmbh.de/filesender/?vid=10f1aeb6-82b4-3d09-c201-00002042c681 >> >> Link is valid until 28.01.2017 (1/28/2017). Feel free to message me >> if you need a reupload. >> >> Best Regards, >> Thorsten >> >> >>> Thorsten, >>> >>> Can you please put the heap dump and gc logs somewhere for download? >>> >>> Thanks >>> Jenny >>> >>> On 1/24/2017 8:30 AM, Thorsten Goetzke wrote: >>>> Hello, >>>> >>>> I am working on a closed source software that uses a patched version >>>> of apache-batik to display SVG-Documents one by one. If you keep >>>> loading and discarding some specific SVG-Documents memory consumption >>>> keeps on growing until it reaches more than 95% of the available >>>> memory. The application becomes unusable slow, very little memory gets >>>> freed, although the gc is constantly running. Sometimes an >>>> OutOfMemoryError gets thrown. I took memory snaphsots and analyzed >>>> them using jvisualvm and yourkit. These tools report the majority of >>>> the memory as unreachable. >>>> I have tried different garbage collectors (G1,ConcurrentMarkSweep), >>>> and different Versions of Java: 1.8_60;1.8_65;1.8_102, the issue is >>>> always reproducible. >>>> Is there something i can do to debug this further? I am currently not >>>> able to reproduce this issue in a somewhat small self contained >>>> example. >>>> I could provide gc-logs and the heap dump (about 1gb zipped) or any >>>> other debuging info if someone is interested. >>>> >>>> Best Regards, >>>> Thorsten Goetzke >>>> >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > From yu.zhang at oracle.com Sat Jan 28 20:23:55 2017 From: yu.zhang at oracle.com (yu.zhang at oracle.com) Date: Sat, 28 Jan 2017 12:23:55 -0800 Subject: Unreachable Memory not freed In-Reply-To: <349a5bf5-9c94-4578-5ad5-d13d0c3afc3a@oracle.com> References: <83689bdf-7285-3493-e7ab-c46b597bc02a@freigmbh.de> <349a5bf5-9c94-4578-5ad5-d13d0c3afc3a@oracle.com> Message-ID: Thorsten, I have updated the bug with more details. I think there is some inaccuracy when reporting the heap dump. There might be some memory leak in your application as well. Thanks Jenny On 01/27/2017 04:50 PM, Jenny Zhang wrote: > Hi, Thorsten, > > I can reproduce this with a micro. I have filed > https://bugs.openjdk.java.net/browse/JDK-8173594 > > The micro is mostly based on > http://stackoverflow.com/questions/26027313/how-to-load-and-parse-svg-documents > with some modification. I will upload the micro and details later. > > I think this is a issue how jvmti and gc reporting unreachable > objects. Those objects should not be reported as unreachable. > > Thanks > Jenny > > On 1/25/2017 3:23 PM, yu.zhang at oracle.com wrote: >> Thorsten, >> >> Thanks very much for the heap dump and gc log. >> >> I assume the gc log and heap dump are from the same run. It is very >> interesting, in the gc log: >> >> 2017-01-24T16:41:17.184-0100: 1301.633: [Full GC (Heap Dump Initiated >> GC) 3817M->3816M(4096M), 8.9543444 secs] >> [Eden: 2048.0K(2048.0M)->0.0B(2048.0M) Survivors: 0.0B->0.0B Heap: >> 3817.7M(4096.0M)->3816.2M(4096.0M)], [Metaspace: >> 142921K->142921K(1175552K)] >> [Times: user=16.35 sys=0.11, real=8.95 secs] >> >> GC thinks there are 3816.2M of objects alive. But in the heap dump, >> there is only 1.8G alive. >> >> I can see those unreachable objects with yourkit, mostly are >> >> org.apache.batik.dom.GenerecText >> >> But apparently gc does not think those are dead. It could be a bug in >> gc, or jvmti dumping the heap, or tool(yourkit, apache MAT). I need >> to discuss this with dev team. >> >> Back to the gc log, it seems the application allocates a lot of >> humongous objects. I can see some int[] of 1.5m. In general the heap >> is very full, with a lot of to-space exhausted, and full gc. And the >> heap usage after full gc is almost the whole heap (4096M). >> >> Thanks >> >> Jenny >> >> >> On 01/25/2017 01:15 AM, Thorsten Goetzke wrote: >>> Hello, >>> >>> You can find the heap dump and gc logs under >>> https://filesender.freigmbh.de/filesender/?vid=10f1aeb6-82b4-3d09-c201-00002042c681 >>> >>> Link is valid until 28.01.2017 (1/28/2017). Feel free to message me >>> if you need a reupload. >>> >>> Best Regards, >>> Thorsten >>> >>> >>>> Thorsten, >>>> >>>> Can you please put the heap dump and gc logs somewhere for download? >>>> >>>> Thanks >>>> Jenny >>>> >>>> On 1/24/2017 8:30 AM, Thorsten Goetzke wrote: >>>>> Hello, >>>>> >>>>> I am working on a closed source software that uses a patched version >>>>> of apache-batik to display SVG-Documents one by one. If you keep >>>>> loading and discarding some specific SVG-Documents memory consumption >>>>> keeps on growing until it reaches more than 95% of the available >>>>> memory. The application becomes unusable slow, very little memory >>>>> gets >>>>> freed, although the gc is constantly running. Sometimes an >>>>> OutOfMemoryError gets thrown. I took memory snaphsots and analyzed >>>>> them using jvisualvm and yourkit. These tools report the majority of >>>>> the memory as unreachable. >>>>> I have tried different garbage collectors (G1,ConcurrentMarkSweep), >>>>> and different Versions of Java: 1.8_60;1.8_65;1.8_102, the issue is >>>>> always reproducible. >>>>> Is there something i can do to debug this further? I am currently not >>>>> able to reproduce this issue in a somewhat small self contained >>>>> example. >>>>> I could provide gc-logs and the heap dump (about 1gb zipped) or any >>>>> other debuging info if someone is interested. >>>>> >>>>> Best Regards, >>>>> Thorsten Goetzke >>>>> >>> >>> _______________________________________________ >>> hotspot-gc-use mailing list >>> hotspot-gc-use at openjdk.java.net >>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From amit.balode at gmail.com Mon Jan 30 08:24:09 2017 From: amit.balode at gmail.com (Amit Balode) Date: Mon, 30 Jan 2017 13:54:09 +0530 Subject: CMS large objects vs G1 humongous allocations Message-ID: Hello, so the only reason we decided to move to G1 from CMS was due to fragmentation issues of CMS. After moving to G1, we have started humongous allocations resulting in Full GC's. We have some large objects which are allocated in successive order resulting into this issue but the use case for continious allocation from application side seems genuine. Heap is getting full so thats explainable why full GC is happening although this issue did not happen in CMS of full GC. a) Currently running with max 32MB region size, humongous allocations as high as 21MB are happening. So for such large allocation, 32MB seems smaller, might be 64MB would have been appropriate, is that an option available in future G1 releases? b) Given that application behaviour cannot be changed much which can stop continuous large allocation what are some G1 specific settings to tune to make it more resilient? Below is the snippet from application running with 16GB heap, 40ms pause time and 32MB region size. {code} 2017-01-29T14:53:14.770+0000: 189106.959: [GC pause (G1 Evacuation Pause) (young) Desired survivor size 654311424 bytes, new threshold 15 (max 15) - age 1: 240262896 bytes, 240262896 total - age 2: 3476760 bytes, 243739656 total - age 3: 3293240 bytes, 247032896 total - age 4: 3147072 bytes, 250179968 total - age 5: 420832 bytes, 250600800 total - age 6: 614688 bytes, 251215488 total - age 7: 1139960 bytes, 252355448 total - age 8: 632088 bytes, 252987536 total - age 9: 425488 bytes, 253413024 total - age 10: 1592608 bytes, 255005632 total 189106.960: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 29363, predicted base time: 20.87 ms, remaining time: 19.13 ms, target pause time: 40.00 ms] 189106.960: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 276 regions, survivors: 27 regions, predicted young region time: 12.11 ms] 189106.960: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 276 regions, survivors: 27 regions, old: 0 regions, predicted pause time: 32.98 ms, target pause time: 40.00 ms] 189106.961: [G1Ergonomics (Heap Sizing) attempt heap expansion, reason: region allocation request failed, allocation request: 5549208 bytes] 189106.961: [G1Ergonomics (Heap Sizing) expand the heap, requested expansion amount: 5549208 bytes, attempted expansion amount: 33554432 bytes] 189106.961: [G1Ergonomics (Heap Sizing) did not expand the heap, reason: heap already fully expanded] 189114.730: [G1Ergonomics (Concurrent Cycles) do not request concurrent cycle initiation, reason: still doing mixed collections, occupancy: 13119782912 bytes, allocation request : 0 bytes, threshold: 7730941095 bytes (45.00 %), source: end of GC] 189114.730: [G1Ergonomics (Mixed GCs) start mixed GCs, reason: candidate old regions available, candidate old regions: 45 regions, reclaimable: 1026676456 bytes (5.98 %), threshold: 5.00 %] (to-space exhausted), *7.7714626 secs*] [Parallel Time: 7182.7 ms, GC Workers: 18] {code} -- Thanks & Regards, Amit -------------- next part -------------- An HTML attachment was scrubbed... URL: From vitalyd at gmail.com Mon Jan 30 11:44:22 2017 From: vitalyd at gmail.com (Vitaly Davidovich) Date: Mon, 30 Jan 2017 11:44:22 +0000 Subject: CMS large objects vs G1 humongous allocations In-Reply-To: References: Message-ID: I don't believe region sizes >32MB are planned, at least that wasn't the case before - maybe that's changed though. Some tuning options to consider: 1) Increase max heap size (this is always a blunt but frequently effective instrument) 2) Turn InitiatingHeapOccupancyPercent down from its default 45 value. 3) Increase ConcGCThreads What version of the JRE are you running? From the short GC log snippet you pasted, it looks like G1 concurrent cycles are losing the race to your Java threads (i.e. you exhaust the heap before concurrent phase can clean up). How much does a full GC reclaim? You may want to provide more of your GC log - someone is going to ask for that sooner or later :). On Mon, Jan 30, 2017 at 3:24 AM Amit Balode wrote: > Hello, so the only reason we decided to move to G1 from CMS was due to > fragmentation issues of CMS. After moving to G1, we have started humongous > allocations resulting in Full GC's. We have some large objects which are > allocated in successive order resulting into this issue but the use case > for continious allocation from application side seems genuine. Heap is > getting full so thats explainable why full GC is happening although this > issue did not happen in CMS of full GC. > > a) Currently running with max 32MB region size, humongous allocations as > high as 21MB are happening. So for such large allocation, 32MB seems > smaller, might be 64MB would have been appropriate, is that an option > available in future G1 releases? > b) Given that application behaviour cannot be changed much which can > stop continuous large allocation what are some G1 specific settings to tune > to make it more resilient? > > Below is the snippet from application running with 16GB heap, 40ms pause > time and 32MB region size. > {code} > 2017-01-29T14:53:14.770+0000: 189106.959: [GC pause (G1 Evacuation Pause) > (young) > Desired survivor size 654311424 bytes, new threshold 15 (max 15) > - age 1: 240262896 bytes, 240262896 total > - age 2: 3476760 bytes, 243739656 total > - age 3: 3293240 bytes, 247032896 total > - age 4: 3147072 bytes, 250179968 total > - age 5: 420832 bytes, 250600800 total > - age 6: 614688 bytes, 251215488 total > - age 7: 1139960 bytes, 252355448 total > - age 8: 632088 bytes, 252987536 total > - age 9: 425488 bytes, 253413024 total > - age 10: 1592608 bytes, 255005632 total > 189106.960: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 29363, predicted base time: 20.87 ms, remaining time: 19.13 > ms, target pause time: 40.00 ms] > 189106.960: [G1Ergonomics (CSet Construction) add young regions to CSet, > eden: 276 regions, survivors: 27 regions, predicted young region time: > 12.11 ms] > 189106.960: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: > 276 regions, survivors: 27 regions, old: 0 regions, predicted pause time: > 32.98 ms, target pause time: > 40.00 ms] > 189106.961: [G1Ergonomics (Heap Sizing) attempt heap expansion, reason: > region allocation request failed, allocation request: 5549208 bytes] > 189106.961: [G1Ergonomics (Heap Sizing) expand the heap, requested > expansion amount: 5549208 bytes, attempted expansion amount: 33554432 bytes] > 189106.961: [G1Ergonomics (Heap Sizing) did not expand the heap, reason: > heap already fully expanded] > 189114.730: [G1Ergonomics (Concurrent Cycles) do not request concurrent > cycle initiation, reason: still doing mixed collections, occupancy: > 13119782912 bytes, allocation request > : 0 bytes, threshold: 7730941095 bytes (45.00 %), source: end of GC] > 189114.730: [G1Ergonomics (Mixed GCs) start mixed GCs, reason: candidate > old regions available, candidate old regions: 45 regions, reclaimable: > 1026676456 bytes (5.98 %), threshold: 5.00 %] > (to-space exhausted), *7.7714626 secs*] > [Parallel Time: 7182.7 ms, GC Workers: 18] > {code} > > -- > Thanks & Regards, > Amit > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > -- Sent from my phone -------------- next part -------------- An HTML attachment was scrubbed... URL: From amit.balode at gmail.com Mon Jan 30 12:41:46 2017 From: amit.balode at gmail.com (Amit Balode) Date: Mon, 30 Jan 2017 18:11:46 +0530 Subject: CMS large objects vs G1 humongous allocations In-Reply-To: References: Message-ID: Hi Vitaly, JRE: 1.8u40 Settings: -XX:+AlwaysPreTouch -XX:ConcGCThreads=5 -XX:G1HeapRegionSize=33554432 -XX:InitialHeapSize=17179869184 -XX:MaxGCPauseMillis=40 -XX:MaxHeapSize=17179869184 -XX:MaxNewSize=10301210624 full GC reclaim: What i meant is after heap was exhausted, A full pause was done actively to reclaim available pause. Yeah, the next thing I was going try was increasing ConcGCThreads to atleast 16 (given core count is 24). Is it possible that at any given time ConcGCThreads and ParallelGCThreads might be running at the same time? Could you elaborate why you think InitiatingHeapOccupancyPercent reduction might help? InitiatingHeapOccupancyPercent reduction means more frequent concurrent cycles so problem might happen sooner, no? Although combination of increase of ConcGCThreads and reduction of InitiatingHeapOccupancyPercent might help. Pasting more logs as requested. {code} 2017-01-29T14:52:41.570+0000: 189073.759: [GC pause (G1 Humongous Allocation) (young) (initial-mark) Desired survivor size 603979776 bytes, new threshold 15 (max 15) - age 1: 302507344 bytes, 302507344 total - age 2: 4153408 bytes, 306660752 total - age 3: 3325224 bytes, 309985976 total - age 4: 482672 bytes, 310468648 total - age 5: 659992 bytes, 311128640 total - age 6: 1189240 bytes, 312317880 total - age 7: 645256 bytes, 312963136 total - age 8: 438800 bytes, 313401936 total - age 9: 1603768 bytes, 315005704 total 189073.759: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 24579, predicted base time: 18.93 ms, remaining time: 21.07 ms, target pause time: 40.00 ms] 189073.759: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 170 regions, survivors: 26 regions, predicted young region time: 10.94 ms] 189073.759: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 170 regions, survivors: 26 regions, old: 0 regions, predicted pause time: 29.87 ms, target pause time: 40.00 ms] , 0.0690000 secs] [Parallel Time: 60.2 ms, GC Workers: 18] [GC Worker Start (ms): Min: 189073760.5, Avg: 189073760.7, Max: 189073761.0, Diff: 0.5] [Ext Root Scanning (ms): Min: 3.7, Avg: 4.2, Max: 4.5, Diff: 0.8, Sum: 75.1] [Update RS (ms): Min: 6.9, Avg: 7.2, Max: 7.6, Diff: 0.7, Sum: 129.3] [Processed Buffers: Min: 42, Avg: 62.8, Max: 93, Diff: 51, Sum: 1131] [Scan RS (ms): Min: 6.4, Avg: 6.8, Max: 6.9, Diff: 0.5, Sum: 122.6] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 41.4, Avg: 41.5, Max: 41.8, Diff: 0.5, Sum: 746.6] [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 1.3] [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.7] [GC Worker Total (ms): Min: 59.5, Avg: 59.8, Max: 60.0, Diff: 0.5, Sum: 1075.7] [GC Worker End (ms): Min: 189073820.5, Avg: 189073820.5, Max: 189073820.5, Diff: 0.1] [Code Root Fixup: 0.2 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.6 ms] [Other: 8.0 ms] [Choose CSet: 0.0 ms] [Ref Proc: 3.2 ms] [Ref Enq: 0.2 ms] [Redirty Cards: 0.1 ms] [Humongous Reclaim: 1.7 ms] [Free CSet: 0.2 ms] [Eden: 5440.0M(8288.0M)->0.0B(8960.0M) Survivors: 832.0M->864.0M Heap: 10.9G(16.0G)->2359.8M(16.0G)] [Times: user=1.07 sys=0.00, real=0.07 secs] 2017-01-29T14:52:41.639+0000: 189073.828: [GC concurrent-root-region-scan-start] 2017-01-29T14:52:41.640+0000: 189073.828: Total time for which application threads were stopped: 0.0742949 seconds, Stopping threads took: 0.0004049 seconds 2017-01-29T14:52:41.680+0000: 189073.868: [GC concurrent-root-region-scan-end, 0.0402986 secs] 2017-01-29T14:52:41.680+0000: 189073.868: [GC concurrent-mark-start] 2017-01-29T14:52:41.892+0000: 189074.081: [GC concurrent-mark-end, 0.2123457 secs] 2017-01-29T14:52:41.896+0000: 189074.085: [GC remark 189074.085: [Finalize Marking, 0.0012962 secs] 189074.087: [GC ref-proc, 0.0043022 secs] 189074.091: [Unloading, 0.0226266 secs], 0.0300066 secs] [Times: user=0.34 sys=0.01, real=0.03 secs] 2017-01-29T14:52:41.927+0000: 189074.116: Total time for which application threads were stopped: 0.0351083 seconds, Stopping threads took: 0.0004268 seconds 2017-01-29T14:52:41.931+0000: 189074.120: [GC cleanup 2630M->2630M(16G), 0.0020884 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 2017-01-29T14:52:41.933+0000: 189074.122: Total time for which application threads were stopped: 0.0062040 seconds, Stopping threads took: 0.0002670 seconds 2017-01-29T14:52:42.939+0000: 189075.127: Total time for which application threads were stopped: 0.0051245 seconds, Stopping threads took: 0.0004442 seconds 2017-01-29T14:52:43.944+0000: 189076.133: Total time for which application threads were stopped: 0.0052855 seconds, Stopping threads took: 0.0004542 seconds 2017-01-29T14:52:46.950+0000: 189079.139: Total time for which application threads were stopped: 0.0055019 seconds, Stopping threads took: 0.0006084 seconds 2017-01-29T14:52:47.955+0000: 189080.144: Total time for which application threads were stopped: 0.0050989 seconds, Stopping threads took: 0.0004534 seconds 2017-01-29T14:52:49.961+0000: 189082.149: Total time for which application threads were stopped: 0.0053802 seconds, Stopping threads took: 0.0006599 seconds 2017-01-29T14:52:51.966+0000: 189084.154: Total time for which application threads were stopped: 0.0047913 seconds, Stopping threads took: 0.0004446 seconds 2017-01-29T14:52:53.970+0000: 189086.158: Total time for which application threads were stopped: 0.0037714 seconds, Stopping threads took: 0.0002900 seconds 2017-01-29T14:52:55.974+0000: 189088.163: Total time for which application threads were stopped: 0.0044301 seconds, Stopping threads took: 0.0003547 seconds 2017-01-29T14:52:56.979+0000: 189089.167: Total time for which application threads were stopped: 0.0043758 seconds, Stopping threads took: 0.0003496 seconds 2017-01-29T14:52:57.983+0000: 189090.172: Total time for which application threads were stopped: 0.0043504 seconds, Stopping threads took: 0.0003511 seconds 2017-01-29T14:53:06.989+0000: 189099.177: Total time for which application threads were stopped: 0.0045634 seconds, Stopping threads took: 0.0004279 seconds 2017-01-29T14:53:08.993+0000: 189101.182: Total time for which application threads were stopped: 0.0043401 seconds, Stopping threads took: 0.0003958 seconds 2017-01-29T14:53:11.998+0000: 189104.186: Total time for which application threads were stopped: 0.0044074 seconds, Stopping threads took: 0.0003907 seconds 2017-01-29T14:53:14.770+0000: 189106.959: [GC pause (G1 Evacuation Pause) (young) Desired survivor size 654311424 bytes, new threshold 15 (max 15) - age 1: 240262896 bytes, 240262896 total - age 2: 3476760 bytes, 243739656 total - age 3: 3293240 bytes, 247032896 total - age 4: 3147072 bytes, 250179968 total - age 5: 420832 bytes, 250600800 total - age 6: 614688 bytes, 251215488 total - age 7: 1139960 bytes, 252355448 total - age 8: 632088 bytes, 252987536 total - age 9: 425488 bytes, 253413024 total - age 10: 1592608 bytes, 255005632 total 189106.960: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 29363, predicted base time: 20.87 ms, remaining time: 19.13 ms, target pause time: 40.00 ms] 189106.960: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 276 regions, survivors: 27 regions, predicted young region time: 12.11 ms] 189106.960: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 276 regions, survivors: 27 regions, old: 0 regions, predicted pause time: 32.98 ms, target pause time: 40.00 ms] 189106.961: [G1Ergonomics (Heap Sizing) attempt heap expansion, reason: region allocation request failed, allocation request: 5549208 bytes] 189106.961: [G1Ergonomics (Heap Sizing) expand the heap, requested expansion amount: 5549208 bytes, attempted expansion amount: 33554432 bytes] 189106.961: [G1Ergonomics (Heap Sizing) did not expand the heap, reason: heap already fully expanded] 189114.730: [G1Ergonomics (Concurrent Cycles) do not request concurrent cycle initiation, reason: still doing mixed collections, occupancy: 13119782912 bytes, allocation request: 0 bytes, threshold: 7730941095 bytes (45.00 %), source: end of GC] 189114.730: [G1Ergonomics (Mixed GCs) start mixed GCs, reason: candidate old regions available, candidate old regions: 45 regions, reclaimable: 1026676456 bytes (5.98 %), threshold: 5.00 %] (to-space exhausted), 7.7714626 secs] [Parallel Time: 7182.7 ms, GC Workers: 18] [GC Worker Start (ms): Min: 189106960.8, Avg: 189106961.0, Max: 189106961.2, Diff: 0.4] [Ext Root Scanning (ms): Min: 1.0, Avg: 1.7, Max: 2.5, Diff: 1.5, Sum: 30.9] [Update RS (ms): Min: 2.1, Avg: 8.2, Max: 11.5, Diff: 9.4, Sum: 147.0] [Processed Buffers: Min: 15, Avg: 62.9, Max: 130, Diff: 115, Sum: 1132] [Scan RS (ms): Min: 4.0, Avg: 6.1, Max: 6.4, Diff: 2.4, Sum: 109.6] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 7164.5, Avg: 7166.1, Max: 7171.6, Diff: 7.1, Sum: 128989.0] [Termination (ms): Min: 0.0, Avg: 0.2, Max: 0.3, Diff: 0.3, Sum: 3.9] [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.4] [GC Worker Total (ms): Min: 7182.1, Avg: 7182.3, Max: 7182.5, Diff: 0.4, Sum: 129280.9] [GC Worker End (ms): Min: 189114143.2, Avg: 189114143.3, Max: 189114143.3, Diff: 0.1] [Code Root Fixup: 0.3 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.7 ms] [Other: 587.7 ms] [Evacuation Failure: 579.6 ms] [Choose CSet: 0.0 ms] [Ref Proc: 2.8 ms] [Ref Enq: 0.5 ms] [Redirty Cards: 1.4 ms] [Humongous Reclaim: 0.8 ms] [Free CSet: 0.2 ms] [Eden: 8832.0M(8960.0M)->0.0B(800.0M) Survivors: 864.0M->0.0B Heap: 13.9G(16.0G)->11.6G(16.0G)] [Times: user=16.27 sys=0.35, real=7.77 secs] 2017-01-29T14:53:22.542+0000: 189114.731: Total time for which application threads were stopped: 7.7764915 seconds, Stopping threads took: 0.0003511 seconds {code} On Mon, Jan 30, 2017 at 5:14 PM, Vitaly Davidovich wrote: > I don't believe region sizes >32MB are planned, at least that wasn't the > case before - maybe that's changed though. > > Some tuning options to consider: > 1) Increase max heap size (this is always a blunt but frequently effective > instrument) > 2) Turn InitiatingHeapOccupancyPercent down from its default 45 value. > 3) Increase ConcGCThreads > > What version of the JRE are you running? From the short GC log snippet you > pasted, it looks like G1 concurrent cycles are losing the race to your Java > threads (i.e. you exhaust the heap before concurrent phase can clean up). > > How much does a full GC reclaim? You may want to provide more of your GC > log - someone is going to ask for that sooner or later :). > > > On Mon, Jan 30, 2017 at 3:24 AM Amit Balode wrote: > >> Hello, so the only reason we decided to move to G1 from CMS was due to >> fragmentation issues of CMS. After moving to G1, we have started humongous >> allocations resulting in Full GC's. We have some large objects which are >> allocated in successive order resulting into this issue but the use case >> for continious allocation from application side seems genuine. Heap is >> getting full so thats explainable why full GC is happening although this >> issue did not happen in CMS of full GC. >> >> a) Currently running with max 32MB region size, humongous allocations as >> high as 21MB are happening. So for such large allocation, 32MB seems >> smaller, might be 64MB would have been appropriate, is that an option >> available in future G1 releases? >> b) Given that application behaviour cannot be changed much which can >> stop continuous large allocation what are some G1 specific settings to tune >> to make it more resilient? >> >> Below is the snippet from application running with 16GB heap, 40ms pause >> time and 32MB region size. >> {code} >> 2017-01-29T14:53:14.770+0000: 189106.959: [GC pause (G1 Evacuation Pause) >> (young) >> Desired survivor size 654311424 bytes, new threshold 15 (max 15) >> - age 1: 240262896 bytes, 240262896 total >> - age 2: 3476760 bytes, 243739656 total >> - age 3: 3293240 bytes, 247032896 total >> - age 4: 3147072 bytes, 250179968 total >> - age 5: 420832 bytes, 250600800 total >> - age 6: 614688 bytes, 251215488 total >> - age 7: 1139960 bytes, 252355448 total >> - age 8: 632088 bytes, 252987536 total >> - age 9: 425488 bytes, 253413024 total >> - age 10: 1592608 bytes, 255005632 total >> 189106.960: [G1Ergonomics (CSet Construction) start choosing CSet, >> _pending_cards: 29363, predicted base time: 20.87 ms, remaining time: 19.13 >> ms, target pause time: 40.00 ms] >> 189106.960: [G1Ergonomics (CSet Construction) add young regions to CSet, >> eden: 276 regions, survivors: 27 regions, predicted young region time: >> 12.11 ms] >> 189106.960: [G1Ergonomics (CSet Construction) finish choosing CSet, >> eden: 276 regions, survivors: 27 regions, old: 0 regions, predicted pause >> time: 32.98 ms, target pause time: >> 40.00 ms] >> 189106.961: [G1Ergonomics (Heap Sizing) attempt heap expansion, reason: >> region allocation request failed, allocation request: 5549208 bytes] >> 189106.961: [G1Ergonomics (Heap Sizing) expand the heap, requested >> expansion amount: 5549208 bytes, attempted expansion amount: 33554432 bytes] >> 189106.961: [G1Ergonomics (Heap Sizing) did not expand the heap, reason: >> heap already fully expanded] >> 189114.730: [G1Ergonomics (Concurrent Cycles) do not request concurrent >> cycle initiation, reason: still doing mixed collections, occupancy: >> 13119782912 bytes, allocation request >> : 0 bytes, threshold: 7730941095 bytes (45.00 %), source: end of GC] >> 189114.730: [G1Ergonomics (Mixed GCs) start mixed GCs, reason: candidate >> old regions available, candidate old regions: 45 regions, reclaimable: >> 1026676456 bytes (5.98 %), threshold: 5.00 %] >> (to-space exhausted), *7.7714626 secs*] >> [Parallel Time: 7182.7 ms, GC Workers: 18] >> {code} >> >> -- >> Thanks & Regards, >> Amit >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> > -- > Sent from my phone > -- Thanks & Regards, Amit.Balode -------------- next part -------------- An HTML attachment was scrubbed... URL: From vitalyd at gmail.com Mon Jan 30 13:06:34 2017 From: vitalyd at gmail.com (Vitaly Davidovich) Date: Mon, 30 Jan 2017 08:06:34 -0500 Subject: CMS large objects vs G1 humongous allocations In-Reply-To: References: Message-ID: On Mon, Jan 30, 2017 at 7:41 AM, Amit Balode wrote: > Hi Vitaly, > > JRE: 1.8u40 > Settings: -XX:+AlwaysPreTouch -XX:ConcGCThreads=5 > -XX:G1HeapRegionSize=33554432 -XX:InitialHeapSize=17179869184 > <(717)%20986-9184> -XX:MaxGCPauseMillis=40 -XX:MaxHeapSize=17179869184 > <(717)%20986-9184> -XX:MaxNewSize=10301210624 > > full GC reclaim: What i meant is after heap was exhausted, A full pause > was done actively to reclaim available pause. > Right, but how much of the heap was reclaimed? If very little, then your max heap size is too low for your working set. If it reclaimed a bunch, then it's a matter of concurrent GC not keeping up. If you can paste the output of the Full GC phase, it would be helpful. > > > Yeah, the next thing I was going try was increasing ConcGCThreads to > atleast 16 (given core count is 24). Is it possible that at any given time ConcGCThreads > and ParallelGCThreads might be running at the same time? > Yes, they can run concurrently. ConcGCThreads service the concurrent phases of G1's concurrent cycles (e.g. concurrent marking), and the ParallelGCThreads are used during young collections to perform the parallel phases of young collections. > > Could you elaborate why you think InitiatingHeapOccupancyPercent > reduction might help? InitiatingHeapOccupancyPercent reduction means more > frequent concurrent cycles so problem might happen sooner, no? Although > combination of increase of ConcGCThreads and reduction of > InitiatingHeapOccupancyPercent might help. > Reduced IHOP makes concurrent cycles kick in sooner, and if the issue is that concurrent marking cannot keep up, then making it start sooner could help. If you start concurrent marking sooner/earlier, it may be able to complete (and thus clean up) before your Java threads completely fill the heap and necessitate a Full GC. That's the thinking behind that suggestion, but it needs to be processed with respect to your actual situation. > > Pasting more logs as requested. > {code} > 2017-01-29T14:52:41.570+0000: 189073.759: [GC pause (G1 Humongous > Allocation) (young) (initial-mark) > Desired survivor size 603979776 bytes, new threshold 15 (max 15) > - age 1: 302507344 bytes, 302507344 total > - age 2: 4153408 bytes, 306660752 total > - age 3: 3325224 bytes, 309985976 total > - age 4: 482672 bytes, 310468648 total > - age 5: 659992 bytes, 311128640 total > - age 6: 1189240 bytes, 312317880 total > - age 7: 645256 bytes, 312963136 total > - age 8: 438800 bytes, 313401936 total > - age 9: 1603768 bytes, 315005704 total > 189073.759: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 24579, predicted base time: 18.93 ms, remaining time: 21.07 > ms, target pause time: 40.00 ms] > 189073.759: [G1Ergonomics (CSet Construction) add young regions to CSet, > eden: 170 regions, survivors: 26 regions, predicted young region time: > 10.94 ms] > 189073.759: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: > 170 regions, survivors: 26 regions, old: 0 regions, predicted pause time: > 29.87 ms, target pause time: 40.00 ms] > , 0.0690000 secs] > [Parallel Time: 60.2 ms, GC Workers: 18] > [GC Worker Start (ms): Min: 189073760.5, Avg: 189073760.7, Max: > 189073761.0, Diff: 0.5] > [Ext Root Scanning (ms): Min: 3.7, Avg: 4.2, Max: 4.5, Diff: 0.8, > Sum: 75.1] > [Update RS (ms): Min: 6.9, Avg: 7.2, Max: 7.6, Diff: 0.7, Sum: 129.3] > [Processed Buffers: Min: 42, Avg: 62.8, Max: 93, Diff: 51, Sum: > 1131] > [Scan RS (ms): Min: 6.4, Avg: 6.8, Max: 6.9, Diff: 0.5, Sum: 122.6] > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, > Sum: 0.0] > [Object Copy (ms): Min: 41.4, Avg: 41.5, Max: 41.8, Diff: 0.5, Sum: > 746.6] > [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 1.3] > [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: > 0.7] > [GC Worker Total (ms): Min: 59.5, Avg: 59.8, Max: 60.0, Diff: 0.5, > Sum: 1075.7] > [GC Worker End (ms): Min: 189073820.5, Avg: 189073820.5, Max: > 189073820.5, Diff: 0.1] > [Code Root Fixup: 0.2 ms] > [Code Root Purge: 0.0 ms] > [Clear CT: 0.6 ms] > [Other: 8.0 ms] > [Choose CSet: 0.0 ms] > [Ref Proc: 3.2 ms] > [Ref Enq: 0.2 ms] > [Redirty Cards: 0.1 ms] > [Humongous Reclaim: 1.7 ms] > [Free CSet: 0.2 ms] > [Eden: 5440.0M(8288.0M)->0.0B(8960.0M) Survivors: 832.0M->864.0M Heap: > 10.9G(16.0G)->2359.8M(16.0G)] > [Times: user=1.07 sys=0.00, real=0.07 secs] > 2017-01-29T14:52:41.639+0000: 189073.828: [GC concurrent-root-region-scan- > start] > 2017-01-29T14:52:41.640+0000: 189073.828: Total time for which application > threads were stopped: 0.0742949 seconds, Stopping threads took: 0.0004049 > seconds > 2017-01-29T14:52:41.680+0000: 189073.868: [GC concurrent-root-region-scan-end, > 0.0402986 secs] > 2017-01-29T14:52:41.680+0000: 189073.868: [GC concurrent-mark-start] > 2017-01-29T14:52:41.892+0000: 189074.081: [GC concurrent-mark-end, > 0.2123457 secs] > 2017-01-29T14:52:41.896+0000: 189074.085: [GC remark 189074.085: [Finalize > Marking, 0.0012962 secs] 189074.087: [GC ref-proc, 0.0043022 secs] > 189074.091: [Unloading, 0.0226266 secs], 0.0300066 secs] > [Times: user=0.34 sys=0.01, real=0.03 secs] > 2017-01-29T14:52:41.927+0000: 189074.116: Total time for which application > threads were stopped: 0.0351083 seconds, Stopping threads took: 0.0004268 > seconds > 2017-01-29T14:52:41.931+0000: 189074.120: [GC cleanup 2630M->2630M(16G), > 0.0020884 secs] > [Times: user=0.02 sys=0.00, real=0.00 secs] > Let's note a few things here: 1) concurrent marking completes fairly quickly 2) No space is reclaimed. Another thing to consider, in light of this, is to tweak G1MixedGCLiveThresholdPercent (it's an experimental feature, so needs to be unlocked via -XX:+UnlockExperimentalVMOptions). This value (85 by default) controls the liveness of a region when it's considered for mixed collections. Any region that's X% (or more) garbage, where X is the flag's value, is selected for mixed collections. So with the default, a region has to be at least 85% garbage to be considered for mixed GCs. You could experiment with a lower value to see if mixed GC can clean up a bit more. > 2017-01-29T14:52:41.933+0000: 189074.122: Total time for which application > threads were stopped: 0.0062040 seconds, Stopping threads took: 0.0002670 > seconds > 2017-01-29T14:52:42.939+0000: 189075.127: Total time for which application > threads were stopped: 0.0051245 seconds, Stopping threads took: 0.0004442 > seconds > 2017-01-29T14:52:43.944+0000: 189076.133: Total time for which application > threads were stopped: 0.0052855 seconds, Stopping threads took: 0.0004542 > seconds > 2017-01-29T14:52:46.950+0000: 189079.139: Total time for which application > threads were stopped: 0.0055019 seconds, Stopping threads took: 0.0006084 > seconds > 2017-01-29T14:52:47.955+0000: 189080.144: Total time for which application > threads were stopped: 0.0050989 seconds, Stopping threads took: 0.0004534 > seconds > 2017-01-29T14:52:49.961+0000: 189082.149: Total time for which application > threads were stopped: 0.0053802 seconds, Stopping threads took: 0.0006599 > seconds > 2017-01-29T14:52:51.966+0000: 189084.154: Total time for which application > threads were stopped: 0.0047913 seconds, Stopping threads took: 0.0004446 > seconds > 2017-01-29T14:52:53.970+0000: 189086.158: Total time for which application > threads were stopped: 0.0037714 seconds, Stopping threads took: 0.0002900 > seconds > 2017-01-29T14:52:55.974+0000: 189088.163: Total time for which application > threads were stopped: 0.0044301 seconds, Stopping threads took: 0.0003547 > seconds > 2017-01-29T14:52:56.979+0000: 189089.167: Total time for which application > threads were stopped: 0.0043758 seconds, Stopping threads took: 0.0003496 > seconds > 2017-01-29T14:52:57.983+0000: 189090.172: Total time for which application > threads were stopped: 0.0043504 seconds, Stopping threads took: 0.0003511 > seconds > 2017-01-29T14:53:06.989+0000: 189099.177: Total time for which application > threads were stopped: 0.0045634 seconds, Stopping threads took: 0.0004279 > seconds > 2017-01-29T14:53:08.993+0000: 189101.182: Total time for which application > threads were stopped: 0.0043401 seconds, Stopping threads took: 0.0003958 > seconds > 2017-01-29T14:53:11.998+0000: 189104.186: Total time for which application > threads were stopped: 0.0044074 seconds, Stopping threads took: 0.0003907 > seconds > 2017-01-29T14:53:14.770+0000: 189106.959: [GC pause (G1 Evacuation Pause) > (young) > Desired survivor size 654311424 bytes, new threshold 15 (max 15) > - age 1: 240262896 bytes, 240262896 total > - age 2: 3476760 bytes, 243739656 total > - age 3: 3293240 bytes, 247032896 total > - age 4: 3147072 bytes, 250179968 total > - age 5: 420832 bytes, 250600800 total > - age 6: 614688 bytes, 251215488 total > - age 7: 1139960 bytes, 252355448 total > - age 8: 632088 bytes, 252987536 total > - age 9: 425488 bytes, 253413024 total > - age 10: 1592608 bytes, 255005632 total > 189106.960: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 29363, predicted base time: 20.87 ms, remaining time: 19.13 > ms, target pause time: 40.00 ms] > 189106.960: [G1Ergonomics (CSet Construction) add young regions to CSet, > eden: 276 regions, survivors: 27 regions, predicted young region time: > 12.11 ms] > 189106.960: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: > 276 regions, survivors: 27 regions, old: 0 regions, predicted pause time: > 32.98 ms, target pause time: 40.00 ms] > 189106.961: [G1Ergonomics (Heap Sizing) attempt heap expansion, reason: > region allocation request failed, allocation request: 5549208 bytes] > 189106.961: [G1Ergonomics (Heap Sizing) expand the heap, requested > expansion amount: 5549208 bytes, attempted expansion amount: 33554432 bytes] > 189106.961: [G1Ergonomics (Heap Sizing) did not expand the heap, reason: > heap already fully expanded] > 189114.730: [G1Ergonomics (Concurrent Cycles) do not request concurrent > cycle initiation, reason: still doing mixed collections, occupancy: > 13119782912 bytes, allocation request: 0 bytes, threshold: 7730941095 bytes > (45.00 %), source: end of GC] > 189114.730: [G1Ergonomics (Mixed GCs) start mixed GCs, reason: candidate > old regions available, candidate old regions: 45 regions, reclaimable: > 1026676456 bytes (5.98 %), threshold: 5.00 %] > So here it found ~1G reclaimable in old regions, but it's too late - it's hitting to-space exhaustion right below. This is what leads me to believe that G1's concurrent cleanup is "losing the race" against the Java threads (that are concurrently allocating). Or, it could be that your max heap is too small for your allocation rate and current GC params. This is where it's useful to see how much memory a Full GC reclaims. > (to-space exhausted), 7.7714626 secs] > [Parallel Time: 7182.7 ms, GC Workers: 18] > [GC Worker Start (ms): Min: 189106960.8, Avg: 189106961.0, Max: > 189106961.2, Diff: 0.4] > [Ext Root Scanning (ms): Min: 1.0, Avg: 1.7, Max: 2.5, Diff: 1.5, > Sum: 30.9] > [Update RS (ms): Min: 2.1, Avg: 8.2, Max: 11.5, Diff: 9.4, Sum: > 147.0] > [Processed Buffers: Min: 15, Avg: 62.9, Max: 130, Diff: 115, Sum: > 1132] > [Scan RS (ms): Min: 4.0, Avg: 6.1, Max: 6.4, Diff: 2.4, Sum: 109.6] > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, > Sum: 0.0] > [Object Copy (ms): Min: 7164.5, Avg: 7166.1, Max: 7171.6, Diff: 7.1, > Sum: 128989.0] > [Termination (ms): Min: 0.0, Avg: 0.2, Max: 0.3, Diff: 0.3, Sum: 3.9] > [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: > 0.4] > [GC Worker Total (ms): Min: 7182.1, Avg: 7182.3, Max: 7182.5, Diff: > 0.4, Sum: 129280.9] > [GC Worker End (ms): Min: 189114143.2, Avg: 189114143.3, Max: > 189114143.3, Diff: 0.1] > [Code Root Fixup: 0.3 ms] > [Code Root Purge: 0.0 ms] > [Clear CT: 0.7 ms] > [Other: 587.7 ms] > [Evacuation Failure: 579.6 ms] > [Choose CSet: 0.0 ms] > [Ref Proc: 2.8 ms] > [Ref Enq: 0.5 ms] > [Redirty Cards: 1.4 ms] > [Humongous Reclaim: 0.8 ms] > [Free CSet: 0.2 ms] > [Eden: 8832.0M(8960.0M)->0.0B(800.0M) Survivors: 864.0M->0.0B Heap: > 13.9G(16.0G)->11.6G(16.0G)] > [Times: user=16.27 sys=0.35, real=7.77 secs] > 2017-01-29T14:53:22.542+0000: 189114.731: Total time for which application > threads were stopped: 7.7764915 seconds, Stopping threads took: 0.0003511 > seconds > {code} > > > On Mon, Jan 30, 2017 at 5:14 PM, Vitaly Davidovich > wrote: > >> I don't believe region sizes >32MB are planned, at least that wasn't the >> case before - maybe that's changed though. >> >> Some tuning options to consider: >> 1) Increase max heap size (this is always a blunt but frequently >> effective instrument) >> 2) Turn InitiatingHeapOccupancyPercent down from its default 45 value. >> 3) Increase ConcGCThreads >> >> What version of the JRE are you running? From the short GC log snippet >> you pasted, it looks like G1 concurrent cycles are losing the race to your >> Java threads (i.e. you exhaust the heap before concurrent phase can clean >> up). >> >> How much does a full GC reclaim? You may want to provide more of your GC >> log - someone is going to ask for that sooner or later :). >> >> >> On Mon, Jan 30, 2017 at 3:24 AM Amit Balode >> wrote: >> >>> Hello, so the only reason we decided to move to G1 from CMS was due to >>> fragmentation issues of CMS. After moving to G1, we have started humongous >>> allocations resulting in Full GC's. We have some large objects which are >>> allocated in successive order resulting into this issue but the use case >>> for continious allocation from application side seems genuine. Heap is >>> getting full so thats explainable why full GC is happening although this >>> issue did not happen in CMS of full GC. >>> >>> a) Currently running with max 32MB region size, humongous allocations as >>> high as 21MB are happening. So for such large allocation, 32MB seems >>> smaller, might be 64MB would have been appropriate, is that an option >>> available in future G1 releases? >>> b) Given that application behaviour cannot be changed much which can >>> stop continuous large allocation what are some G1 specific settings to tune >>> to make it more resilient? >>> >>> Below is the snippet from application running with 16GB heap, 40ms pause >>> time and 32MB region size. >>> {code} >>> 2017-01-29T14:53:14.770+0000: 189106.959: [GC pause (G1 Evacuation >>> Pause) (young) >>> Desired survivor size 654311424 bytes, new threshold 15 (max 15) >>> - age 1: 240262896 bytes, 240262896 total >>> - age 2: 3476760 bytes, 243739656 total >>> - age 3: 3293240 bytes, 247032896 total >>> - age 4: 3147072 bytes, 250179968 total >>> - age 5: 420832 bytes, 250600800 total >>> - age 6: 614688 bytes, 251215488 total >>> - age 7: 1139960 bytes, 252355448 total >>> - age 8: 632088 bytes, 252987536 total >>> - age 9: 425488 bytes, 253413024 total >>> - age 10: 1592608 bytes, 255005632 total >>> 189106.960: [G1Ergonomics (CSet Construction) start choosing CSet, >>> _pending_cards: 29363, predicted base time: 20.87 ms, remaining time: 19.13 >>> ms, target pause time: 40.00 ms] >>> 189106.960: [G1Ergonomics (CSet Construction) add young regions to >>> CSet, eden: 276 regions, survivors: 27 regions, predicted young region >>> time: 12.11 ms] >>> 189106.960: [G1Ergonomics (CSet Construction) finish choosing CSet, >>> eden: 276 regions, survivors: 27 regions, old: 0 regions, predicted pause >>> time: 32.98 ms, target pause time: >>> 40.00 ms] >>> 189106.961: [G1Ergonomics (Heap Sizing) attempt heap expansion, reason: >>> region allocation request failed, allocation request: 5549208 bytes] >>> 189106.961: [G1Ergonomics (Heap Sizing) expand the heap, requested >>> expansion amount: 5549208 bytes, attempted expansion amount: 33554432 bytes] >>> 189106.961: [G1Ergonomics (Heap Sizing) did not expand the heap, >>> reason: heap already fully expanded] >>> 189114.730: [G1Ergonomics (Concurrent Cycles) do not request concurrent >>> cycle initiation, reason: still doing mixed collections, occupancy: >>> 13119782912 bytes, allocation request >>> : 0 bytes, threshold: 7730941095 bytes (45.00 %), source: end of GC] >>> 189114.730: [G1Ergonomics (Mixed GCs) start mixed GCs, reason: candidate >>> old regions available, candidate old regions: 45 regions, reclaimable: >>> 1026676456 bytes (5.98 %), threshold: 5.00 %] >>> (to-space exhausted), *7.7714626 secs*] >>> [Parallel Time: 7182.7 ms, GC Workers: 18] >>> {code} >>> >>> -- >>> Thanks & Regards, >>> Amit >>> >>> _______________________________________________ >>> hotspot-gc-use mailing list >>> hotspot-gc-use at openjdk.java.net >>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>> >> -- >> Sent from my phone >> > > > > -- > Thanks & Regards, > Amit.Balode > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From amit.balode at gmail.com Mon Jan 30 13:19:27 2017 From: amit.balode at gmail.com (Amit Balode) Date: Mon, 30 Jan 2017 18:49:27 +0530 Subject: CMS large objects vs G1 humongous allocations In-Reply-To: References: Message-ID: Italy, thank you. There is no explicit line which says Full GC. I thought line printed as "(Heap Sizing) did not expand the heap, reason: heap already fully expanded" is an implicit indication that G1 have taken all necessary actions which full GC would take and it lead to 7.7sec pause. Sorry a bit confused about how can 'G1's concurrent cycles' & 'parallel phases of young collections' run concurrently? Does that mean ConcGCThreads + ParallelGCThreads cannot be greater than cpu cores.? Good point about tuning G1MixedGCLiveThresholdPercent. On Mon, Jan 30, 2017 at 6:36 PM, Vitaly Davidovich wrote: > > > On Mon, Jan 30, 2017 at 7:41 AM, Amit Balode > wrote: > >> Hi Vitaly, >> >> JRE: 1.8u40 >> Settings: -XX:+AlwaysPreTouch -XX:ConcGCThreads=5 >> -XX:G1HeapRegionSize=33554432 -XX:InitialHeapSize=17179869184 >> <(717)%20986-9184> -XX:MaxGCPauseMillis=40 -XX:MaxHeapSize=17179869184 >> <(717)%20986-9184> -XX:MaxNewSize=10301210624 >> >> full GC reclaim: What i meant is after heap was exhausted, A full pause >> was done actively to reclaim available pause. >> > Right, but how much of the heap was reclaimed? If very little, then your > max heap size is too low for your working set. If it reclaimed a bunch, > then it's a matter of concurrent GC not keeping up. If you can paste the > output of the Full GC phase, it would be helpful. > >> >> >> Yeah, the next thing I was going try was increasing ConcGCThreads to >> atleast 16 (given core count is 24). Is it possible that at any given time ConcGCThreads >> and ParallelGCThreads might be running at the same time? >> > Yes, they can run concurrently. ConcGCThreads service the concurrent > phases of G1's concurrent cycles (e.g. concurrent marking), and the > ParallelGCThreads are used during young collections to perform the parallel > phases of young collections. > >> >> Could you elaborate why you think InitiatingHeapOccupancyPercent >> reduction might help? InitiatingHeapOccupancyPercent reduction means >> more frequent concurrent cycles so problem might happen sooner, no? >> Although combination of increase of ConcGCThreads and reduction of >> InitiatingHeapOccupancyPercent might help. >> > Reduced IHOP makes concurrent cycles kick in sooner, and if the issue is > that concurrent marking cannot keep up, then making it start sooner could > help. If you start concurrent marking sooner/earlier, it may be able to > complete (and thus clean up) before your Java threads completely fill the > heap and necessitate a Full GC. That's the thinking behind that > suggestion, but it needs to be processed with respect to your actual > situation. > >> >> Pasting more logs as requested. >> {code} >> 2017-01-29T14:52:41.570+0000: 189073.759: [GC pause (G1 Humongous >> Allocation) (young) (initial-mark) >> Desired survivor size 603979776 bytes, new threshold 15 (max 15) >> - age 1: 302507344 bytes, 302507344 total >> - age 2: 4153408 bytes, 306660752 total >> - age 3: 3325224 bytes, 309985976 total >> - age 4: 482672 bytes, 310468648 total >> - age 5: 659992 bytes, 311128640 total >> - age 6: 1189240 bytes, 312317880 total >> - age 7: 645256 bytes, 312963136 total >> - age 8: 438800 bytes, 313401936 total >> - age 9: 1603768 bytes, 315005704 total >> 189073.759: [G1Ergonomics (CSet Construction) start choosing CSet, >> _pending_cards: 24579, predicted base time: 18.93 ms, remaining time: 21.07 >> ms, target pause time: 40.00 ms] >> 189073.759: [G1Ergonomics (CSet Construction) add young regions to CSet, >> eden: 170 regions, survivors: 26 regions, predicted young region time: >> 10.94 ms] >> 189073.759: [G1Ergonomics (CSet Construction) finish choosing CSet, >> eden: 170 regions, survivors: 26 regions, old: 0 regions, predicted pause >> time: 29.87 ms, target pause time: 40.00 ms] >> , 0.0690000 secs] >> [Parallel Time: 60.2 ms, GC Workers: 18] >> [GC Worker Start (ms): Min: 189073760.5, Avg: 189073760.7, Max: >> 189073761.0, Diff: 0.5] >> [Ext Root Scanning (ms): Min: 3.7, Avg: 4.2, Max: 4.5, Diff: 0.8, >> Sum: 75.1] >> [Update RS (ms): Min: 6.9, Avg: 7.2, Max: 7.6, Diff: 0.7, Sum: >> 129.3] >> [Processed Buffers: Min: 42, Avg: 62.8, Max: 93, Diff: 51, Sum: >> 1131] >> [Scan RS (ms): Min: 6.4, Avg: 6.8, Max: 6.9, Diff: 0.5, Sum: 122.6] >> [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, >> Sum: 0.0] >> [Object Copy (ms): Min: 41.4, Avg: 41.5, Max: 41.8, Diff: 0.5, Sum: >> 746.6] >> [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: >> 1.3] >> [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, >> Sum: 0.7] >> [GC Worker Total (ms): Min: 59.5, Avg: 59.8, Max: 60.0, Diff: 0.5, >> Sum: 1075.7] >> [GC Worker End (ms): Min: 189073820.5, Avg: 189073820.5, Max: >> 189073820.5, Diff: 0.1] >> [Code Root Fixup: 0.2 ms] >> [Code Root Purge: 0.0 ms] >> [Clear CT: 0.6 ms] >> [Other: 8.0 ms] >> [Choose CSet: 0.0 ms] >> [Ref Proc: 3.2 ms] >> [Ref Enq: 0.2 ms] >> [Redirty Cards: 0.1 ms] >> [Humongous Reclaim: 1.7 ms] >> [Free CSet: 0.2 ms] >> [Eden: 5440.0M(8288.0M)->0.0B(8960.0M) Survivors: 832.0M->864.0M >> Heap: 10.9G(16.0G)->2359.8M(16.0G)] >> [Times: user=1.07 sys=0.00, real=0.07 secs] >> 2017-01-29T14:52:41.639+0000: 189073.828: [GC >> concurrent-root-region-scan-start] >> 2017-01-29T14:52:41.640+0000: 189073.828: Total time for which >> application threads were stopped: 0.0742949 seconds, Stopping threads took: >> 0.0004049 seconds >> 2017-01-29T14:52:41.680+0000: 189073.868: [GC >> concurrent-root-region-scan-end, 0.0402986 secs] >> 2017-01-29T14:52:41.680+0000: 189073.868: [GC concurrent-mark-start] >> 2017-01-29T14:52:41.892+0000: 189074.081: [GC concurrent-mark-end, >> 0.2123457 secs] >> 2017-01-29T14:52:41.896+0000: 189074.085: [GC remark 189074.085: >> [Finalize Marking, 0.0012962 secs] 189074.087: [GC ref-proc, 0.0043022 >> secs] 189074.091: [Unloading, 0.0226266 secs], 0.0300066 secs] >> [Times: user=0.34 sys=0.01, real=0.03 secs] >> 2017-01-29T14:52:41.927+0000: 189074.116: Total time for which >> application threads were stopped: 0.0351083 seconds, Stopping threads took: >> 0.0004268 seconds >> 2017-01-29T14:52:41.931+0000: 189074.120: [GC cleanup 2630M->2630M(16G), >> 0.0020884 secs] >> [Times: user=0.02 sys=0.00, real=0.00 secs] >> > Let's note a few things here: > 1) concurrent marking completes fairly quickly > 2) No space is reclaimed. > > Another thing to consider, in light of this, is to tweak G1MixedGCLiveThresholdPercent > (it's an experimental feature, so needs to be unlocked via -XX:+UnlockExperimentalVMOptions). > This value (85 by default) controls the liveness of a region when it's > considered for mixed collections. Any region that's X% (or more) garbage, > where X is the flag's value, is selected for mixed collections. So with > the default, a region has to be at least 85% garbage to be considered for > mixed GCs. You could experiment with a lower value to see if mixed GC can > clean up a bit more. > > >> 2017-01-29T14:52:41.933+0000: 189074.122: Total time for which >> application threads were stopped: 0.0062040 seconds, Stopping threads took: >> 0.0002670 seconds >> 2017-01-29T14:52:42.939+0000: 189075.127: Total time for which >> application threads were stopped: 0.0051245 seconds, Stopping threads took: >> 0.0004442 seconds >> 2017-01-29T14:52:43.944+0000: 189076.133: Total time for which >> application threads were stopped: 0.0052855 seconds, Stopping threads took: >> 0.0004542 seconds >> 2017-01-29T14:52:46.950+0000: 189079.139: Total time for which >> application threads were stopped: 0.0055019 seconds, Stopping threads took: >> 0.0006084 seconds >> 2017-01-29T14:52:47.955+0000: 189080.144: Total time for which >> application threads were stopped: 0.0050989 seconds, Stopping threads took: >> 0.0004534 seconds >> 2017-01-29T14:52:49.961+0000: 189082.149: Total time for which >> application threads were stopped: 0.0053802 seconds, Stopping threads took: >> 0.0006599 seconds >> 2017-01-29T14:52:51.966+0000: 189084.154: Total time for which >> application threads were stopped: 0.0047913 seconds, Stopping threads took: >> 0.0004446 seconds >> 2017-01-29T14:52:53.970+0000: 189086.158: Total time for which >> application threads were stopped: 0.0037714 seconds, Stopping threads took: >> 0.0002900 seconds >> 2017-01-29T14:52:55.974+0000: 189088.163: Total time for which >> application threads were stopped: 0.0044301 seconds, Stopping threads took: >> 0.0003547 seconds >> 2017-01-29T14:52:56.979+0000: 189089.167: Total time for which >> application threads were stopped: 0.0043758 seconds, Stopping threads took: >> 0.0003496 seconds >> 2017-01-29T14:52:57.983+0000: 189090.172: Total time for which >> application threads were stopped: 0.0043504 seconds, Stopping threads took: >> 0.0003511 seconds >> 2017-01-29T14:53:06.989+0000: 189099.177: Total time for which >> application threads were stopped: 0.0045634 seconds, Stopping threads took: >> 0.0004279 seconds >> 2017-01-29T14:53:08.993+0000: 189101.182: Total time for which >> application threads were stopped: 0.0043401 seconds, Stopping threads took: >> 0.0003958 seconds >> 2017-01-29T14:53:11.998+0000: 189104.186: Total time for which >> application threads were stopped: 0.0044074 seconds, Stopping threads took: >> 0.0003907 seconds >> 2017-01-29T14:53:14.770+0000: 189106.959: [GC pause (G1 Evacuation Pause) >> (young) >> Desired survivor size 654311424 bytes, new threshold 15 (max 15) >> - age 1: 240262896 bytes, 240262896 total >> - age 2: 3476760 bytes, 243739656 total >> - age 3: 3293240 bytes, 247032896 total >> - age 4: 3147072 bytes, 250179968 total >> - age 5: 420832 bytes, 250600800 total >> - age 6: 614688 bytes, 251215488 total >> - age 7: 1139960 bytes, 252355448 total >> - age 8: 632088 bytes, 252987536 total >> - age 9: 425488 bytes, 253413024 total >> - age 10: 1592608 bytes, 255005632 total >> 189106.960: [G1Ergonomics (CSet Construction) start choosing CSet, >> _pending_cards: 29363, predicted base time: 20.87 ms, remaining time: 19.13 >> ms, target pause time: 40.00 ms] >> 189106.960: [G1Ergonomics (CSet Construction) add young regions to CSet, >> eden: 276 regions, survivors: 27 regions, predicted young region time: >> 12.11 ms] >> 189106.960: [G1Ergonomics (CSet Construction) finish choosing CSet, >> eden: 276 regions, survivors: 27 regions, old: 0 regions, predicted pause >> time: 32.98 ms, target pause time: 40.00 ms] >> 189106.961: [G1Ergonomics (Heap Sizing) attempt heap expansion, reason: >> region allocation request failed, allocation request: 5549208 bytes] >> 189106.961: [G1Ergonomics (Heap Sizing) expand the heap, requested >> expansion amount: 5549208 bytes, attempted expansion amount: 33554432 bytes] >> 189106.961: [G1Ergonomics (Heap Sizing) did not expand the heap, reason: >> heap already fully expanded] >> 189114.730: [G1Ergonomics (Concurrent Cycles) do not request concurrent >> cycle initiation, reason: still doing mixed collections, occupancy: >> 13119782912 bytes, allocation request: 0 bytes, threshold: 7730941095 bytes >> (45.00 %), source: end of GC] >> 189114.730: [G1Ergonomics (Mixed GCs) start mixed GCs, reason: candidate >> old regions available, candidate old regions: 45 regions, reclaimable: >> 1026676456 bytes (5.98 %), threshold: 5.00 %] >> > So here it found ~1G reclaimable in old regions, but it's too late - it's > hitting to-space exhaustion right below. This is what leads me to believe > that G1's concurrent cleanup is "losing the race" against the Java threads > (that are concurrently allocating). Or, it could be that your max heap is > too small for your allocation rate and current GC params. This is where > it's useful to see how much memory a Full GC reclaims. > >> (to-space exhausted), 7.7714626 secs] >> [Parallel Time: 7182.7 ms, GC Workers: 18] >> [GC Worker Start (ms): Min: 189106960.8, Avg: 189106961.0, Max: >> 189106961.2, Diff: 0.4] >> [Ext Root Scanning (ms): Min: 1.0, Avg: 1.7, Max: 2.5, Diff: 1.5, >> Sum: 30.9] >> [Update RS (ms): Min: 2.1, Avg: 8.2, Max: 11.5, Diff: 9.4, Sum: >> 147.0] >> [Processed Buffers: Min: 15, Avg: 62.9, Max: 130, Diff: 115, >> Sum: 1132] >> [Scan RS (ms): Min: 4.0, Avg: 6.1, Max: 6.4, Diff: 2.4, Sum: 109.6] >> [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, >> Sum: 0.0] >> [Object Copy (ms): Min: 7164.5, Avg: 7166.1, Max: 7171.6, Diff: >> 7.1, Sum: 128989.0] >> [Termination (ms): Min: 0.0, Avg: 0.2, Max: 0.3, Diff: 0.3, Sum: >> 3.9] >> [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, >> Sum: 0.4] >> [GC Worker Total (ms): Min: 7182.1, Avg: 7182.3, Max: 7182.5, Diff: >> 0.4, Sum: 129280.9] >> [GC Worker End (ms): Min: 189114143.2, Avg: 189114143.3, Max: >> 189114143.3, Diff: 0.1] >> [Code Root Fixup: 0.3 ms] >> [Code Root Purge: 0.0 ms] >> [Clear CT: 0.7 ms] >> [Other: 587.7 ms] >> [Evacuation Failure: 579.6 ms] >> [Choose CSet: 0.0 ms] >> [Ref Proc: 2.8 ms] >> [Ref Enq: 0.5 ms] >> [Redirty Cards: 1.4 ms] >> [Humongous Reclaim: 0.8 ms] >> [Free CSet: 0.2 ms] >> [Eden: 8832.0M(8960.0M)->0.0B(800.0M) Survivors: 864.0M->0.0B Heap: >> 13.9G(16.0G)->11.6G(16.0G)] >> [Times: user=16.27 sys=0.35, real=7.77 secs] >> 2017-01-29T14:53:22.542+0000: 189114.731: Total time for which >> application threads were stopped: 7.7764915 seconds, Stopping threads took: >> 0.0003511 seconds >> {code} >> >> >> On Mon, Jan 30, 2017 at 5:14 PM, Vitaly Davidovich >> wrote: >> >>> I don't believe region sizes >32MB are planned, at least that wasn't the >>> case before - maybe that's changed though. >>> >>> Some tuning options to consider: >>> 1) Increase max heap size (this is always a blunt but frequently >>> effective instrument) >>> 2) Turn InitiatingHeapOccupancyPercent down from its default 45 value. >>> 3) Increase ConcGCThreads >>> >>> What version of the JRE are you running? From the short GC log snippet >>> you pasted, it looks like G1 concurrent cycles are losing the race to your >>> Java threads (i.e. you exhaust the heap before concurrent phase can clean >>> up). >>> >>> How much does a full GC reclaim? You may want to provide more of your GC >>> log - someone is going to ask for that sooner or later :). >>> >>> >>> On Mon, Jan 30, 2017 at 3:24 AM Amit Balode >>> wrote: >>> >>>> Hello, so the only reason we decided to move to G1 from CMS was due to >>>> fragmentation issues of CMS. After moving to G1, we have started humongous >>>> allocations resulting in Full GC's. We have some large objects which are >>>> allocated in successive order resulting into this issue but the use case >>>> for continious allocation from application side seems genuine. Heap is >>>> getting full so thats explainable why full GC is happening although this >>>> issue did not happen in CMS of full GC. >>>> >>>> a) Currently running with max 32MB region size, humongous allocations >>>> as high as 21MB are happening. So for such large allocation, 32MB seems >>>> smaller, might be 64MB would have been appropriate, is that an option >>>> available in future G1 releases? >>>> b) Given that application behaviour cannot be changed much which can >>>> stop continuous large allocation what are some G1 specific settings to tune >>>> to make it more resilient? >>>> >>>> Below is the snippet from application running with 16GB heap, 40ms >>>> pause time and 32MB region size. >>>> {code} >>>> 2017-01-29T14:53:14.770+0000: 189106.959: [GC pause (G1 Evacuation >>>> Pause) (young) >>>> Desired survivor size 654311424 bytes, new threshold 15 (max 15) >>>> - age 1: 240262896 bytes, 240262896 total >>>> - age 2: 3476760 bytes, 243739656 total >>>> - age 3: 3293240 bytes, 247032896 total >>>> - age 4: 3147072 bytes, 250179968 total >>>> - age 5: 420832 bytes, 250600800 total >>>> - age 6: 614688 bytes, 251215488 total >>>> - age 7: 1139960 bytes, 252355448 total >>>> - age 8: 632088 bytes, 252987536 total >>>> - age 9: 425488 bytes, 253413024 total >>>> - age 10: 1592608 bytes, 255005632 total >>>> 189106.960: [G1Ergonomics (CSet Construction) start choosing CSet, >>>> _pending_cards: 29363, predicted base time: 20.87 ms, remaining time: 19.13 >>>> ms, target pause time: 40.00 ms] >>>> 189106.960: [G1Ergonomics (CSet Construction) add young regions to >>>> CSet, eden: 276 regions, survivors: 27 regions, predicted young region >>>> time: 12.11 ms] >>>> 189106.960: [G1Ergonomics (CSet Construction) finish choosing CSet, >>>> eden: 276 regions, survivors: 27 regions, old: 0 regions, predicted pause >>>> time: 32.98 ms, target pause time: >>>> 40.00 ms] >>>> 189106.961: [G1Ergonomics (Heap Sizing) attempt heap expansion, >>>> reason: region allocation request failed, allocation request: 5549208 bytes] >>>> 189106.961: [G1Ergonomics (Heap Sizing) expand the heap, requested >>>> expansion amount: 5549208 bytes, attempted expansion amount: 33554432 bytes] >>>> 189106.961: [G1Ergonomics (Heap Sizing) did not expand the heap, >>>> reason: heap already fully expanded] >>>> 189114.730: [G1Ergonomics (Concurrent Cycles) do not request >>>> concurrent cycle initiation, reason: still doing mixed collections, >>>> occupancy: 13119782912 bytes, allocation request >>>> : 0 bytes, threshold: 7730941095 bytes (45.00 %), source: end of GC] >>>> 189114.730: [G1Ergonomics (Mixed GCs) start mixed GCs, reason: >>>> candidate old regions available, candidate old regions: 45 regions, >>>> reclaimable: 1026676456 bytes (5.98 %), threshold: 5.00 %] >>>> (to-space exhausted), *7.7714626 secs*] >>>> [Parallel Time: 7182.7 ms, GC Workers: 18] >>>> {code} >>>> >>>> -- >>>> Thanks & Regards, >>>> Amit >>>> >>>> _______________________________________________ >>>> hotspot-gc-use mailing list >>>> hotspot-gc-use at openjdk.java.net >>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>>> >>> -- >>> Sent from my phone >>> >> >> >> >> -- >> Thanks & Regards, >> Amit.Balode >> >> > -- Thanks & Regards, Amit.Balode -------------- next part -------------- An HTML attachment was scrubbed... URL: From vitalyd at gmail.com Mon Jan 30 13:38:41 2017 From: vitalyd at gmail.com (Vitaly Davidovich) Date: Mon, 30 Jan 2017 08:38:41 -0500 Subject: CMS large objects vs G1 humongous allocations In-Reply-To: References: Message-ID: On Mon, Jan 30, 2017 at 8:19 AM, Amit Balode wrote: > Italy, thank you. > > There is no explicit line which says Full GC. I thought line printed as > "(Heap Sizing) did not expand the heap, reason: heap already fully > expanded" is an implicit indication that G1 have taken all necessary > actions which full GC would take and it lead to 7.7sec pause. > Oh really, there's no Full GC that ensues after those lines? The (Heap Sizing) output is G1's ergonomic output. It's saying that it would like to expand the heap because it failed to allocate a new region out of the existing heap. The reason for expansion is G1 may start out with a heap smaller than capacity (i.e. max heap size), and will try to expand the heap (if under capacity/max) as needed. Here, it cannot expand because you're already at max heap size. The 7.7s pause is due to to-space exhaustion, which basically means G1 ran out of space to copy survivors (and you can see that this 7s is all in the Object Copy phase of the pause). In my experience, when you start seeing to-space exhaustion, you pretty soon see a Full GC. > > Sorry a bit confused about how can 'G1's concurrent cycles' & 'parallel > phases of young collections' run concurrently? Does that > mean ConcGCThreads + ParallelGCThreads cannot be greater than cpu cores.? > G1 has concurrent GC cycles, such as concurrent marking - this runs in the background, on the ConcGCThreads, and marks old regions to gather their liveness information (so that these regions can either be cleaned up in the background, if fully empty, or they'll be added to young GCs to make them mixed GCs). At the same time, while that concurrent work is ongoing, your Java threads continue to run and allocate. You can then hit a young pause, where ParallelGCThreads will perform some of the parallel work associated with a young GC. Young GCs look at eden regions (and do some processing of them in parallel), while concurrent GC threads can be processing old regions at the same time. So yes, you can theoretically hit a case where you have ConcGCThreads + ParallelGCThreads threads are runnable, and you may oversubscribe the machine (assuming each of those threads don't stall themselves due to internal issues, such as lock contention, etc). > > Good point about tuning G1MixedGCLiveThresholdPercent. > > On Mon, Jan 30, 2017 at 6:36 PM, Vitaly Davidovich > wrote: > >> >> >> On Mon, Jan 30, 2017 at 7:41 AM, Amit Balode >> wrote: >> >>> Hi Vitaly, >>> >>> JRE: 1.8u40 >>> Settings: -XX:+AlwaysPreTouch -XX:ConcGCThreads=5 >>> -XX:G1HeapRegionSize=33554432 -XX:InitialHeapSize=17179869184 >>> <(717)%20986-9184> -XX:MaxGCPauseMillis=40 -XX:MaxHeapSize=17179869184 >>> <(717)%20986-9184> -XX:MaxNewSize=10301210624 >>> >>> full GC reclaim: What i meant is after heap was exhausted, A full pause >>> was done actively to reclaim available pause. >>> >> Right, but how much of the heap was reclaimed? If very little, then your >> max heap size is too low for your working set. If it reclaimed a bunch, >> then it's a matter of concurrent GC not keeping up. If you can paste the >> output of the Full GC phase, it would be helpful. >> >>> >>> >>> Yeah, the next thing I was going try was increasing ConcGCThreads to >>> atleast 16 (given core count is 24). Is it possible that at any given time ConcGCThreads >>> and ParallelGCThreads might be running at the same time? >>> >> Yes, they can run concurrently. ConcGCThreads service the concurrent >> phases of G1's concurrent cycles (e.g. concurrent marking), and the >> ParallelGCThreads are used during young collections to perform the parallel >> phases of young collections. >> >>> >>> Could you elaborate why you think InitiatingHeapOccupancyPercent >>> reduction might help? InitiatingHeapOccupancyPercent reduction means >>> more frequent concurrent cycles so problem might happen sooner, no? >>> Although combination of increase of ConcGCThreads and reduction of >>> InitiatingHeapOccupancyPercent might help. >>> >> Reduced IHOP makes concurrent cycles kick in sooner, and if the issue is >> that concurrent marking cannot keep up, then making it start sooner could >> help. If you start concurrent marking sooner/earlier, it may be able to >> complete (and thus clean up) before your Java threads completely fill the >> heap and necessitate a Full GC. That's the thinking behind that >> suggestion, but it needs to be processed with respect to your actual >> situation. >> >>> >>> Pasting more logs as requested. >>> {code} >>> 2017-01-29T14:52:41.570+0000: 189073.759: [GC pause (G1 Humongous >>> Allocation) (young) (initial-mark) >>> Desired survivor size 603979776 bytes, new threshold 15 (max 15) >>> - age 1: 302507344 bytes, 302507344 total >>> - age 2: 4153408 bytes, 306660752 total >>> - age 3: 3325224 bytes, 309985976 total >>> - age 4: 482672 bytes, 310468648 total >>> - age 5: 659992 bytes, 311128640 total >>> - age 6: 1189240 bytes, 312317880 total >>> - age 7: 645256 bytes, 312963136 total >>> - age 8: 438800 bytes, 313401936 total >>> - age 9: 1603768 bytes, 315005704 total >>> 189073.759: [G1Ergonomics (CSet Construction) start choosing CSet, >>> _pending_cards: 24579, predicted base time: 18.93 ms, remaining time: 21.07 >>> ms, target pause time: 40.00 ms] >>> 189073.759: [G1Ergonomics (CSet Construction) add young regions to >>> CSet, eden: 170 regions, survivors: 26 regions, predicted young region >>> time: 10.94 ms] >>> 189073.759: [G1Ergonomics (CSet Construction) finish choosing CSet, >>> eden: 170 regions, survivors: 26 regions, old: 0 regions, predicted pause >>> time: 29.87 ms, target pause time: 40.00 ms] >>> , 0.0690000 secs] >>> [Parallel Time: 60.2 ms, GC Workers: 18] >>> [GC Worker Start (ms): Min: 189073760.5, Avg: 189073760.7, Max: >>> 189073761.0, Diff: 0.5] >>> [Ext Root Scanning (ms): Min: 3.7, Avg: 4.2, Max: 4.5, Diff: 0.8, >>> Sum: 75.1] >>> [Update RS (ms): Min: 6.9, Avg: 7.2, Max: 7.6, Diff: 0.7, Sum: >>> 129.3] >>> [Processed Buffers: Min: 42, Avg: 62.8, Max: 93, Diff: 51, Sum: >>> 1131] >>> [Scan RS (ms): Min: 6.4, Avg: 6.8, Max: 6.9, Diff: 0.5, Sum: 122.6] >>> [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, >>> Sum: 0.0] >>> [Object Copy (ms): Min: 41.4, Avg: 41.5, Max: 41.8, Diff: 0.5, >>> Sum: 746.6] >>> [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: >>> 1.3] >>> [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, >>> Sum: 0.7] >>> [GC Worker Total (ms): Min: 59.5, Avg: 59.8, Max: 60.0, Diff: 0.5, >>> Sum: 1075.7] >>> [GC Worker End (ms): Min: 189073820.5, Avg: 189073820.5, Max: >>> 189073820.5, Diff: 0.1] >>> [Code Root Fixup: 0.2 ms] >>> [Code Root Purge: 0.0 ms] >>> [Clear CT: 0.6 ms] >>> [Other: 8.0 ms] >>> [Choose CSet: 0.0 ms] >>> [Ref Proc: 3.2 ms] >>> [Ref Enq: 0.2 ms] >>> [Redirty Cards: 0.1 ms] >>> [Humongous Reclaim: 1.7 ms] >>> [Free CSet: 0.2 ms] >>> [Eden: 5440.0M(8288.0M)->0.0B(8960.0M) Survivors: 832.0M->864.0M >>> Heap: 10.9G(16.0G)->2359.8M(16.0G)] >>> [Times: user=1.07 sys=0.00, real=0.07 secs] >>> 2017-01-29T14:52:41.639+0000: 189073.828: [GC >>> concurrent-root-region-scan-start] >>> 2017-01-29T14:52:41.640+0000: 189073.828: Total time for which >>> application threads were stopped: 0.0742949 seconds, Stopping threads took: >>> 0.0004049 seconds >>> 2017-01-29T14:52:41.680+0000: 189073.868: [GC >>> concurrent-root-region-scan-end, 0.0402986 secs] >>> 2017-01-29T14:52:41.680+0000: 189073.868: [GC concurrent-mark-start] >>> 2017-01-29T14:52:41.892+0000: 189074.081: [GC concurrent-mark-end, >>> 0.2123457 secs] >>> 2017-01-29T14:52:41.896+0000: 189074.085: [GC remark 189074.085: >>> [Finalize Marking, 0.0012962 secs] 189074.087: [GC ref-proc, 0.0043022 >>> secs] 189074.091: [Unloading, 0.0226266 secs], 0.0300066 secs] >>> [Times: user=0.34 sys=0.01, real=0.03 secs] >>> 2017-01-29T14:52:41.927+0000: 189074.116: Total time for which >>> application threads were stopped: 0.0351083 seconds, Stopping threads took: >>> 0.0004268 seconds >>> 2017-01-29T14:52:41.931+0000: 189074.120: [GC cleanup 2630M->2630M(16G), >>> 0.0020884 secs] >>> [Times: user=0.02 sys=0.00, real=0.00 secs] >>> >> Let's note a few things here: >> 1) concurrent marking completes fairly quickly >> 2) No space is reclaimed. >> >> Another thing to consider, in light of this, is to >> tweak G1MixedGCLiveThresholdPercent (it's an experimental feature, so >> needs to be unlocked via -XX:+UnlockExperimentalVMOptions). This value >> (85 by default) controls the liveness of a region when it's considered for >> mixed collections. Any region that's X% (or more) garbage, where X is the >> flag's value, is selected for mixed collections. So with the default, a >> region has to be at least 85% garbage to be considered for mixed GCs. You >> could experiment with a lower value to see if mixed GC can clean up a bit >> more. >> >> >>> 2017-01-29T14:52:41.933+0000: 189074.122: Total time for which >>> application threads were stopped: 0.0062040 seconds, Stopping threads took: >>> 0.0002670 seconds >>> 2017-01-29T14:52:42.939+0000: 189075.127: Total time for which >>> application threads were stopped: 0.0051245 seconds, Stopping threads took: >>> 0.0004442 seconds >>> 2017-01-29T14:52:43.944+0000: 189076.133: Total time for which >>> application threads were stopped: 0.0052855 seconds, Stopping threads took: >>> 0.0004542 seconds >>> 2017-01-29T14:52:46.950+0000: 189079.139: Total time for which >>> application threads were stopped: 0.0055019 seconds, Stopping threads took: >>> 0.0006084 seconds >>> 2017-01-29T14:52:47.955+0000: 189080.144: Total time for which >>> application threads were stopped: 0.0050989 seconds, Stopping threads took: >>> 0.0004534 seconds >>> 2017-01-29T14:52:49.961+0000: 189082.149: Total time for which >>> application threads were stopped: 0.0053802 seconds, Stopping threads took: >>> 0.0006599 seconds >>> 2017-01-29T14:52:51.966+0000: 189084.154: Total time for which >>> application threads were stopped: 0.0047913 seconds, Stopping threads took: >>> 0.0004446 seconds >>> 2017-01-29T14:52:53.970+0000: 189086.158: Total time for which >>> application threads were stopped: 0.0037714 seconds, Stopping threads took: >>> 0.0002900 seconds >>> 2017-01-29T14:52:55.974+0000: 189088.163: Total time for which >>> application threads were stopped: 0.0044301 seconds, Stopping threads took: >>> 0.0003547 seconds >>> 2017-01-29T14:52:56.979+0000: 189089.167: Total time for which >>> application threads were stopped: 0.0043758 seconds, Stopping threads took: >>> 0.0003496 seconds >>> 2017-01-29T14:52:57.983+0000: 189090.172: Total time for which >>> application threads were stopped: 0.0043504 seconds, Stopping threads took: >>> 0.0003511 seconds >>> 2017-01-29T14:53:06.989+0000: 189099.177: Total time for which >>> application threads were stopped: 0.0045634 seconds, Stopping threads took: >>> 0.0004279 seconds >>> 2017-01-29T14:53:08.993+0000: 189101.182: Total time for which >>> application threads were stopped: 0.0043401 seconds, Stopping threads took: >>> 0.0003958 seconds >>> 2017-01-29T14:53:11.998+0000: 189104.186: Total time for which >>> application threads were stopped: 0.0044074 seconds, Stopping threads took: >>> 0.0003907 seconds >>> 2017-01-29T14:53:14.770+0000: 189106.959: [GC pause (G1 Evacuation >>> Pause) (young) >>> Desired survivor size 654311424 bytes, new threshold 15 (max 15) >>> - age 1: 240262896 bytes, 240262896 total >>> - age 2: 3476760 bytes, 243739656 total >>> - age 3: 3293240 bytes, 247032896 total >>> - age 4: 3147072 bytes, 250179968 total >>> - age 5: 420832 bytes, 250600800 total >>> - age 6: 614688 bytes, 251215488 total >>> - age 7: 1139960 bytes, 252355448 total >>> - age 8: 632088 bytes, 252987536 total >>> - age 9: 425488 bytes, 253413024 total >>> - age 10: 1592608 bytes, 255005632 total >>> 189106.960: [G1Ergonomics (CSet Construction) start choosing CSet, >>> _pending_cards: 29363, predicted base time: 20.87 ms, remaining time: 19.13 >>> ms, target pause time: 40.00 ms] >>> 189106.960: [G1Ergonomics (CSet Construction) add young regions to >>> CSet, eden: 276 regions, survivors: 27 regions, predicted young region >>> time: 12.11 ms] >>> 189106.960: [G1Ergonomics (CSet Construction) finish choosing CSet, >>> eden: 276 regions, survivors: 27 regions, old: 0 regions, predicted pause >>> time: 32.98 ms, target pause time: 40.00 ms] >>> 189106.961: [G1Ergonomics (Heap Sizing) attempt heap expansion, reason: >>> region allocation request failed, allocation request: 5549208 bytes] >>> 189106.961: [G1Ergonomics (Heap Sizing) expand the heap, requested >>> expansion amount: 5549208 bytes, attempted expansion amount: 33554432 bytes] >>> 189106.961: [G1Ergonomics (Heap Sizing) did not expand the heap, >>> reason: heap already fully expanded] >>> 189114.730: [G1Ergonomics (Concurrent Cycles) do not request concurrent >>> cycle initiation, reason: still doing mixed collections, occupancy: >>> 13119782912 bytes, allocation request: 0 bytes, threshold: 7730941095 bytes >>> (45.00 %), source: end of GC] >>> 189114.730: [G1Ergonomics (Mixed GCs) start mixed GCs, reason: >>> candidate old regions available, candidate old regions: 45 regions, >>> reclaimable: 1026676456 bytes (5.98 %), threshold: 5.00 %] >>> >> So here it found ~1G reclaimable in old regions, but it's too late - it's >> hitting to-space exhaustion right below. This is what leads me to believe >> that G1's concurrent cleanup is "losing the race" against the Java threads >> (that are concurrently allocating). Or, it could be that your max heap is >> too small for your allocation rate and current GC params. This is where >> it's useful to see how much memory a Full GC reclaims. >> >>> (to-space exhausted), 7.7714626 secs] >>> [Parallel Time: 7182.7 ms, GC Workers: 18] >>> [GC Worker Start (ms): Min: 189106960.8, Avg: 189106961.0, Max: >>> 189106961.2, Diff: 0.4] >>> [Ext Root Scanning (ms): Min: 1.0, Avg: 1.7, Max: 2.5, Diff: 1.5, >>> Sum: 30.9] >>> [Update RS (ms): Min: 2.1, Avg: 8.2, Max: 11.5, Diff: 9.4, Sum: >>> 147.0] >>> [Processed Buffers: Min: 15, Avg: 62.9, Max: 130, Diff: 115, >>> Sum: 1132] >>> [Scan RS (ms): Min: 4.0, Avg: 6.1, Max: 6.4, Diff: 2.4, Sum: 109.6] >>> [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, >>> Sum: 0.0] >>> [Object Copy (ms): Min: 7164.5, Avg: 7166.1, Max: 7171.6, Diff: >>> 7.1, Sum: 128989.0] >>> [Termination (ms): Min: 0.0, Avg: 0.2, Max: 0.3, Diff: 0.3, Sum: >>> 3.9] >>> [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, >>> Sum: 0.4] >>> [GC Worker Total (ms): Min: 7182.1, Avg: 7182.3, Max: 7182.5, >>> Diff: 0.4, Sum: 129280.9] >>> [GC Worker End (ms): Min: 189114143.2, Avg: 189114143.3, Max: >>> 189114143.3, Diff: 0.1] >>> [Code Root Fixup: 0.3 ms] >>> [Code Root Purge: 0.0 ms] >>> [Clear CT: 0.7 ms] >>> [Other: 587.7 ms] >>> [Evacuation Failure: 579.6 ms] >>> [Choose CSet: 0.0 ms] >>> [Ref Proc: 2.8 ms] >>> [Ref Enq: 0.5 ms] >>> [Redirty Cards: 1.4 ms] >>> [Humongous Reclaim: 0.8 ms] >>> [Free CSet: 0.2 ms] >>> [Eden: 8832.0M(8960.0M)->0.0B(800.0M) Survivors: 864.0M->0.0B Heap: >>> 13.9G(16.0G)->11.6G(16.0G)] >>> [Times: user=16.27 sys=0.35, real=7.77 secs] >>> 2017-01-29T14:53:22.542+0000: 189114.731: Total time for which >>> application threads were stopped: 7.7764915 seconds, Stopping threads took: >>> 0.0003511 seconds >>> {code} >>> >>> >>> On Mon, Jan 30, 2017 at 5:14 PM, Vitaly Davidovich >>> wrote: >>> >>>> I don't believe region sizes >32MB are planned, at least that wasn't >>>> the case before - maybe that's changed though. >>>> >>>> Some tuning options to consider: >>>> 1) Increase max heap size (this is always a blunt but frequently >>>> effective instrument) >>>> 2) Turn InitiatingHeapOccupancyPercent down from its default 45 value. >>>> 3) Increase ConcGCThreads >>>> >>>> What version of the JRE are you running? From the short GC log snippet >>>> you pasted, it looks like G1 concurrent cycles are losing the race to your >>>> Java threads (i.e. you exhaust the heap before concurrent phase can clean >>>> up). >>>> >>>> How much does a full GC reclaim? You may want to provide more of your >>>> GC log - someone is going to ask for that sooner or later :). >>>> >>>> >>>> On Mon, Jan 30, 2017 at 3:24 AM Amit Balode >>>> wrote: >>>> >>>>> Hello, so the only reason we decided to move to G1 from CMS was due to >>>>> fragmentation issues of CMS. After moving to G1, we have started humongous >>>>> allocations resulting in Full GC's. We have some large objects which are >>>>> allocated in successive order resulting into this issue but the use case >>>>> for continious allocation from application side seems genuine. Heap >>>>> is getting full so thats explainable why full GC is happening although this >>>>> issue did not happen in CMS of full GC. >>>>> >>>>> a) Currently running with max 32MB region size, humongous allocations >>>>> as high as 21MB are happening. So for such large allocation, 32MB seems >>>>> smaller, might be 64MB would have been appropriate, is that an option >>>>> available in future G1 releases? >>>>> b) Given that application behaviour cannot be changed much which can >>>>> stop continuous large allocation what are some G1 specific settings to tune >>>>> to make it more resilient? >>>>> >>>>> Below is the snippet from application running with 16GB heap, 40ms >>>>> pause time and 32MB region size. >>>>> {code} >>>>> 2017-01-29T14:53:14.770+0000: 189106.959: [GC pause (G1 Evacuation >>>>> Pause) (young) >>>>> Desired survivor size 654311424 bytes, new threshold 15 (max 15) >>>>> - age 1: 240262896 bytes, 240262896 total >>>>> - age 2: 3476760 bytes, 243739656 total >>>>> - age 3: 3293240 bytes, 247032896 total >>>>> - age 4: 3147072 bytes, 250179968 total >>>>> - age 5: 420832 bytes, 250600800 total >>>>> - age 6: 614688 bytes, 251215488 total >>>>> - age 7: 1139960 bytes, 252355448 total >>>>> - age 8: 632088 bytes, 252987536 total >>>>> - age 9: 425488 bytes, 253413024 total >>>>> - age 10: 1592608 bytes, 255005632 total >>>>> 189106.960: [G1Ergonomics (CSet Construction) start choosing CSet, >>>>> _pending_cards: 29363, predicted base time: 20.87 ms, remaining time: 19.13 >>>>> ms, target pause time: 40.00 ms] >>>>> 189106.960: [G1Ergonomics (CSet Construction) add young regions to >>>>> CSet, eden: 276 regions, survivors: 27 regions, predicted young region >>>>> time: 12.11 ms] >>>>> 189106.960: [G1Ergonomics (CSet Construction) finish choosing CSet, >>>>> eden: 276 regions, survivors: 27 regions, old: 0 regions, predicted pause >>>>> time: 32.98 ms, target pause time: >>>>> 40.00 ms] >>>>> 189106.961: [G1Ergonomics (Heap Sizing) attempt heap expansion, >>>>> reason: region allocation request failed, allocation request: 5549208 bytes] >>>>> 189106.961: [G1Ergonomics (Heap Sizing) expand the heap, requested >>>>> expansion amount: 5549208 bytes, attempted expansion amount: 33554432 bytes] >>>>> 189106.961: [G1Ergonomics (Heap Sizing) did not expand the heap, >>>>> reason: heap already fully expanded] >>>>> 189114.730: [G1Ergonomics (Concurrent Cycles) do not request >>>>> concurrent cycle initiation, reason: still doing mixed collections, >>>>> occupancy: 13119782912 bytes, allocation request >>>>> : 0 bytes, threshold: 7730941095 bytes (45.00 %), source: end of GC] >>>>> 189114.730: [G1Ergonomics (Mixed GCs) start mixed GCs, reason: >>>>> candidate old regions available, candidate old regions: 45 regions, >>>>> reclaimable: 1026676456 bytes (5.98 %), threshold: 5.00 %] >>>>> (to-space exhausted), *7.7714626 secs*] >>>>> [Parallel Time: 7182.7 ms, GC Workers: 18] >>>>> {code} >>>>> >>>>> -- >>>>> Thanks & Regards, >>>>> Amit >>>>> >>>>> _______________________________________________ >>>>> hotspot-gc-use mailing list >>>>> hotspot-gc-use at openjdk.java.net >>>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>>>> >>>> -- >>>> Sent from my phone >>>> >>> >>> >>> >>> -- >>> Thanks & Regards, >>> Amit.Balode >>> >>> >> > > > -- > Thanks & Regards, > Amit.Balode > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vitalyd at gmail.com Mon Jan 30 13:49:00 2017 From: vitalyd at gmail.com (Vitaly Davidovich) Date: Mon, 30 Jan 2017 13:49:00 +0000 Subject: CMS large objects vs G1 humongous allocations In-Reply-To: References: Message-ID: Also, you can experiment with relaxing the pause time goal a bit. G1 will use it as a heuristic to determine how many old regions (during mixed GC) to add to a collection. If you add too few, you're not reclaiming fast enough (potentially) for your allocation rate. Is your entire gc log too big? If not, might be good to attach it or put it somewhere (e.g. pastebin) so we can see the bigger picture. On Mon, Jan 30, 2017 at 8:38 AM Vitaly Davidovich wrote: > On Mon, Jan 30, 2017 at 8:19 AM, Amit Balode > wrote: > > Italy, thank you. > > There is no explicit line which says Full GC. I thought line printed as > "(Heap Sizing) did not expand the heap, reason: heap already fully > expanded" is an implicit indication that G1 have taken all necessary > actions which full GC would take and it lead to 7.7sec pause. > > Oh really, there's no Full GC that ensues after those lines? > > The (Heap Sizing) output is G1's ergonomic output. It's saying that it > would like to expand the heap because it failed to allocate a new region > out of the existing heap. The reason for expansion is G1 may start out > with a heap smaller than capacity (i.e. max heap size), and will try to > expand the heap (if under capacity/max) as needed. Here, it cannot expand > because you're already at max heap size. > > The 7.7s pause is due to to-space exhaustion, which basically means G1 ran > out of space to copy survivors (and you can see that this 7s is all in the > Object Copy phase of the pause). In my experience, when you start seeing > to-space exhaustion, you pretty soon see a Full GC. > > > Sorry a bit confused about how can 'G1's concurrent cycles' & 'parallel > phases of young collections' run concurrently? Does that > mean ConcGCThreads + ParallelGCThreads cannot be greater than cpu cores.? > > G1 has concurrent GC cycles, such as concurrent marking - this runs in the > background, on the ConcGCThreads, and marks old regions to gather their > liveness information (so that these regions can either be cleaned up in the > background, if fully empty, or they'll be added to young GCs to make them > mixed GCs). At the same time, while that concurrent work is ongoing, your > Java threads continue to run and allocate. You can then hit a young pause, > where ParallelGCThreads will perform some of the parallel work associated > with a young GC. Young GCs look at eden regions (and do some processing of > them in parallel), while concurrent GC threads can be processing old > regions at the same time. So yes, you can theoretically hit a case where > you have ConcGCThreads + ParallelGCThreads threads are runnable, and you > may oversubscribe the machine (assuming each of those threads don't stall > themselves due to internal issues, such as lock contention, etc). > > > Good point about tuning G1MixedGCLiveThresholdPercent. > > On Mon, Jan 30, 2017 at 6:36 PM, Vitaly Davidovich > wrote: > > > > On Mon, Jan 30, 2017 at 7:41 AM, Amit Balode > wrote: > > Hi Vitaly, > > JRE: 1.8u40 > Settings: -XX:+AlwaysPreTouch -XX:ConcGCThreads=5 > -XX:G1HeapRegionSize=33554432 -XX:InitialHeapSize=17179869184 > <(717)%20986-9184> -XX:MaxGCPauseMillis=40 -XX:MaxHeapSize=17179869184 > <(717)%20986-9184> -XX:MaxNewSize=10301210624 > > full GC reclaim: What i meant is after heap was exhausted, A full pause > was done actively to reclaim available pause. > > Right, but how much of the heap was reclaimed? If very little, then your > max heap size is too low for your working set. If it reclaimed a bunch, > then it's a matter of concurrent GC not keeping up. If you can paste the > output of the Full GC phase, it would be helpful. > > > > Yeah, the next thing I was going try was increasing ConcGCThreads to > atleast 16 (given core count is 24). Is it possible that at any given time ConcGCThreads > and ParallelGCThreads might be running at the same time? > > Yes, they can run concurrently. ConcGCThreads service the concurrent > phases of G1's concurrent cycles (e.g. concurrent marking), and the > ParallelGCThreads are used during young collections to perform the parallel > phases of young collections. > > > Could you elaborate why you think InitiatingHeapOccupancyPercent > reduction might help? InitiatingHeapOccupancyPercent reduction means more > frequent concurrent cycles so problem might happen sooner, no? Although > combination of increase of ConcGCThreads and reduction of InitiatingHeapOccupancyPercent > might help. > > Reduced IHOP makes concurrent cycles kick in sooner, and if the issue is > that concurrent marking cannot keep up, then making it start sooner could > help. If you start concurrent marking sooner/earlier, it may be able to > complete (and thus clean up) before your Java threads completely fill the > heap and necessitate a Full GC. That's the thinking behind that > suggestion, but it needs to be processed with respect to your actual > situation. > > > Pasting more logs as requested. > {code} > 2017-01-29T14:52:41.570+0000: 189073.759: [GC pause (G1 Humongous > Allocation) (young) (initial-mark) > Desired survivor size 603979776 bytes, new threshold 15 (max 15) > - age 1: 302507344 bytes, 302507344 total > - age 2: 4153408 bytes, 306660752 total > - age 3: 3325224 bytes, 309985976 total > - age 4: 482672 bytes, 310468648 total > - age 5: 659992 bytes, 311128640 total > - age 6: 1189240 bytes, 312317880 total > - age 7: 645256 bytes, 312963136 total > - age 8: 438800 bytes, 313401936 total > - age 9: 1603768 bytes, 315005704 total > 189073.759: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 24579, predicted base time: 18.93 ms, remaining time: 21.07 > ms, target pause time: 40.00 ms] > 189073.759: [G1Ergonomics (CSet Construction) add young regions to CSet, > eden: 170 regions, survivors: 26 regions, predicted young region time: > 10.94 ms] > 189073.759: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: > 170 regions, survivors: 26 regions, old: 0 regions, predicted pause time: > 29.87 ms, target pause time: 40.00 ms] > , 0.0690000 secs] > [Parallel Time: 60.2 ms, GC Workers: 18] > [GC Worker Start (ms): Min: 189073760.5, Avg: 189073760.7, Max: > 189073761.0, Diff: 0.5] > [Ext Root Scanning (ms): Min: 3.7, Avg: 4.2, Max: 4.5, Diff: 0.8, > Sum: 75.1] > [Update RS (ms): Min: 6.9, Avg: 7.2, Max: 7.6, Diff: 0.7, Sum: 129.3] > [Processed Buffers: Min: 42, Avg: 62.8, Max: 93, Diff: 51, Sum: > 1131] > [Scan RS (ms): Min: 6.4, Avg: 6.8, Max: 6.9, Diff: 0.5, Sum: 122.6] > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, > Sum: 0.0] > [Object Copy (ms): Min: 41.4, Avg: 41.5, Max: 41.8, Diff: 0.5, Sum: > 746.6] > [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 1.3] > [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: > 0.7] > [GC Worker Total (ms): Min: 59.5, Avg: 59.8, Max: 60.0, Diff: 0.5, > Sum: 1075.7] > [GC Worker End (ms): Min: 189073820.5, Avg: 189073820.5, Max: > 189073820.5, Diff: 0.1] > [Code Root Fixup: 0.2 ms] > [Code Root Purge: 0.0 ms] > [Clear CT: 0.6 ms] > [Other: 8.0 ms] > [Choose CSet: 0.0 ms] > [Ref Proc: 3.2 ms] > [Ref Enq: 0.2 ms] > [Redirty Cards: 0.1 ms] > [Humongous Reclaim: 1.7 ms] > [Free CSet: 0.2 ms] > [Eden: 5440.0M(8288.0M)->0.0B(8960.0M) Survivors: 832.0M->864.0M Heap: > 10.9G(16.0G)->2359.8M(16.0G)] > [Times: user=1.07 sys=0.00, real=0.07 secs] > 2017-01-29T14:52:41.639+0000: 189073.828: [GC > concurrent-root-region-scan-start] > 2017-01-29T14:52:41.640+0000: 189073.828: Total time for which application > threads were stopped: 0.0742949 seconds, Stopping threads took: 0.0004049 > seconds > 2017-01-29T14:52:41.680+0000: 189073.868: [GC > concurrent-root-region-scan-end, 0.0402986 secs] > 2017-01-29T14:52:41.680+0000: 189073.868: [GC concurrent-mark-start] > 2017-01-29T14:52:41.892+0000: 189074.081: [GC concurrent-mark-end, > 0.2123457 secs] > 2017-01-29T14:52:41.896+0000: 189074.085: [GC remark 189074.085: [Finalize > Marking, 0.0012962 secs] 189074.087: [GC ref-proc, 0.0043022 secs] > 189074.091: [Unloading, 0.0226266 secs], 0.0300066 secs] > [Times: user=0.34 sys=0.01, real=0.03 secs] > 2017-01-29T14:52:41.927+0000: 189074.116: Total time for which application > threads were stopped: 0.0351083 seconds, Stopping threads took: 0.0004268 > seconds > 2017-01-29T14:52:41.931+0000: 189074.120: [GC cleanup 2630M->2630M(16G), > 0.0020884 secs] > [Times: user=0.02 sys=0.00, real=0.00 secs] > > Let's note a few things here: > 1) concurrent marking completes fairly quickly > 2) No space is reclaimed. > > Another thing to consider, in light of this, is to > tweak G1MixedGCLiveThresholdPercent (it's an experimental feature, so needs > to be unlocked via -XX:+UnlockExperimentalVMOptions). This value (85 by > default) controls the liveness of a region when it's considered for mixed > collections. Any region that's X% (or more) garbage, where X is the flag's > value, is selected for mixed collections. So with the default, a region > has to be at least 85% garbage to be considered for mixed GCs. You could > experiment with a lower value to see if mixed GC can clean up a bit more. > > > 2017-01-29T14:52:41.933+0000: 189074.122: Total time for which application > threads were stopped: 0.0062040 seconds, Stopping threads took: 0.0002670 > seconds > 2017-01-29T14:52:42.939+0000: 189075.127: Total time for which application > threads were stopped: 0.0051245 seconds, Stopping threads took: 0.0004442 > seconds > 2017-01-29T14:52:43.944+0000: 189076.133: Total time for which application > threads were stopped: 0.0052855 seconds, Stopping threads took: 0.0004542 > seconds > 2017-01-29T14:52:46.950+0000: 189079.139: Total time for which application > threads were stopped: 0.0055019 seconds, Stopping threads took: 0.0006084 > seconds > 2017-01-29T14:52:47.955+0000: 189080.144: Total time for which application > threads were stopped: 0.0050989 seconds, Stopping threads took: 0.0004534 > seconds > 2017-01-29T14:52:49.961+0000: 189082.149: Total time for which application > threads were stopped: 0.0053802 seconds, Stopping threads took: 0.0006599 > seconds > 2017-01-29T14:52:51.966+0000: 189084.154: Total time for which application > threads were stopped: 0.0047913 seconds, Stopping threads took: 0.0004446 > seconds > 2017-01-29T14:52:53.970+0000: 189086.158: Total time for which application > threads were stopped: 0.0037714 seconds, Stopping threads took: 0.0002900 > seconds > 2017-01-29T14:52:55.974+0000: 189088.163: Total time for which application > threads were stopped: 0.0044301 seconds, Stopping threads took: 0.0003547 > seconds > 2017-01-29T14:52:56.979+0000: 189089.167: Total time for which application > threads were stopped: 0.0043758 seconds, Stopping threads took: 0.0003496 > seconds > 2017-01-29T14:52:57.983+0000: 189090.172: Total time for which application > threads were stopped: 0.0043504 seconds, Stopping threads took: 0.0003511 > seconds > 2017-01-29T14:53:06.989+0000: 189099.177: Total time for which application > threads were stopped: 0.0045634 seconds, Stopping threads took: 0.0004279 > seconds > 2017-01-29T14:53:08.993+0000: 189101.182: Total time for which application > threads were stopped: 0.0043401 seconds, Stopping threads took: 0.0003958 > seconds > 2017-01-29T14:53:11.998+0000: 189104.186: Total time for which application > threads were stopped: 0.0044074 seconds, Stopping threads took: 0.0003907 > seconds > 2017-01-29T14:53:14.770+0000: 189106.959: [GC pause (G1 Evacuation Pause) > (young) > Desired survivor size 654311424 bytes, new threshold 15 (max 15) > - age 1: 240262896 bytes, 240262896 total > - age 2: 3476760 bytes, 243739656 total > - age 3: 3293240 bytes, 247032896 total > - age 4: 3147072 bytes, 250179968 total > - age 5: 420832 bytes, 250600800 total > - age 6: 614688 bytes, 251215488 total > - age 7: 1139960 bytes, 252355448 total > - age 8: 632088 bytes, 252987536 total > - age 9: 425488 bytes, 253413024 total > - age 10: 1592608 bytes, 255005632 total > 189106.960: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 29363, predicted base time: 20.87 ms, remaining time: 19.13 > ms, target pause time: 40.00 ms] > 189106.960: [G1Ergonomics (CSet Construction) add young regions to CSet, > eden: 276 regions, survivors: 27 regions, predicted young region time: > 12.11 ms] > 189106.960: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: > 276 regions, survivors: 27 regions, old: 0 regions, predicted pause time: > 32.98 ms, target pause time: 40.00 ms] > 189106.961: [G1Ergonomics (Heap Sizing) attempt heap expansion, reason: > region allocation request failed, allocation request: 5549208 bytes] > 189106.961: [G1Ergonomics (Heap Sizing) expand the heap, requested > expansion amount: 5549208 bytes, attempted expansion amount: 33554432 bytes] > 189106.961: [G1Ergonomics (Heap Sizing) did not expand the heap, reason: > heap already fully expanded] > 189114.730: [G1Ergonomics (Concurrent Cycles) do not request concurrent > cycle initiation, reason: still doing mixed collections, occupancy: > 13119782912 bytes, allocation request: 0 bytes, threshold: 7730941095 bytes > (45.00 %), source: end of GC] > 189114.730: [G1Ergonomics (Mixed GCs) start mixed GCs, reason: candidate > old regions available, candidate old regions: 45 regions, reclaimable: > 1026676456 bytes (5.98 %), threshold: 5.00 %] > > So here it found ~1G reclaimable in old regions, but it's too late - it's > hitting to-space exhaustion right below. This is what leads me to believe > that G1's concurrent cleanup is "losing the race" against the Java threads > (that are concurrently allocating). Or, it could be that your max heap is > too small for your allocation rate and current GC params. This is where > it's useful to see how much memory a Full GC reclaims. > > (to-space exhausted), 7.7714626 secs] > [Parallel Time: 7182.7 ms, GC Workers: 18] > [GC Worker Start (ms): Min: 189106960.8, Avg: 189106961.0, Max: > 189106961.2, Diff: 0.4] > [Ext Root Scanning (ms): Min: 1.0, Avg: 1.7, Max: 2.5, Diff: 1.5, > Sum: 30.9] > [Update RS (ms): Min: 2.1, Avg: 8.2, Max: 11.5, Diff: 9.4, Sum: > 147.0] > [Processed Buffers: Min: 15, Avg: 62.9, Max: 130, Diff: 115, Sum: > 1132] > [Scan RS (ms): Min: 4.0, Avg: 6.1, Max: 6.4, Diff: 2.4, Sum: 109.6] > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, > Sum: 0.0] > [Object Copy (ms): Min: 7164.5, Avg: 7166.1, Max: 7171.6, Diff: 7.1, > Sum: 128989.0] > [Termination (ms): Min: 0.0, Avg: 0.2, Max: 0.3, Diff: 0.3, Sum: 3.9] > [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: > 0.4] > [GC Worker Total (ms): Min: 7182.1, Avg: 7182.3, Max: 7182.5, Diff: > 0.4, Sum: 129280.9] > [GC Worker End (ms): Min: 189114143.2, Avg: 189114143.3, Max: > 189114143.3, Diff: 0.1] > [Code Root Fixup: 0.3 ms] > [Code Root Purge: 0.0 ms] > [Clear CT: 0.7 ms] > [Other: 587.7 ms] > [Evacuation Failure: 579.6 ms] > [Choose CSet: 0.0 ms] > [Ref Proc: 2.8 ms] > [Ref Enq: 0.5 ms] > [Redirty Cards: 1.4 ms] > [Humongous Reclaim: 0.8 ms] > [Free CSet: 0.2 ms] > [Eden: 8832.0M(8960.0M)->0.0B(800.0M) Survivors: 864.0M->0.0B Heap: > 13.9G(16.0G)->11.6G(16.0G)] > [Times: user=16.27 sys=0.35, real=7.77 secs] > 2017-01-29T14:53:22.542+0000: 189114.731: Total time for which application > threads were stopped: 7.7764915 seconds, Stopping threads took: 0.0003511 > seconds > {code} > > > On Mon, Jan 30, 2017 at 5:14 PM, Vitaly Davidovich > wrote: > > I don't believe region sizes >32MB are planned, at least that wasn't the > case before - maybe that's changed though. > > Some tuning options to consider: > 1) Increase max heap size (this is always a blunt but frequently effective > instrument) > 2) Turn InitiatingHeapOccupancyPercent down from its default 45 value. > 3) Increase ConcGCThreads > > What version of the JRE are you running? From the short GC log snippet you > pasted, it looks like G1 concurrent cycles are losing the race to your Java > threads (i.e. you exhaust the heap before concurrent phase can clean up). > > How much does a full GC reclaim? You may want to provide more of your GC > log - someone is going to ask for that sooner or later :). > > > On Mon, Jan 30, 2017 at 3:24 AM Amit Balode wrote: > > Hello, so the only reason we decided to move to G1 from CMS was due to > fragmentation issues of CMS. After moving to G1, we have started humongous > allocations resulting in Full GC's. We have some large objects which are > allocated in successive order resulting into this issue but the use case > for continious allocation from application side seems genuine. Heap is > getting full so thats explainable why full GC is happening although this > issue did not happen in CMS of full GC. > > a) Currently running with max 32MB region size, humongous allocations as > high as 21MB are happening. So for such large allocation, 32MB seems > smaller, might be 64MB would have been appropriate, is that an option > available in future G1 releases? > b) Given that application behaviour cannot be changed much which can > stop continuous large allocation what are some G1 specific settings to tune > to make it more resilient? > > Below is the snippet from application running with 16GB heap, 40ms pause > time and 32MB region size. > {code} > 2017-01-29T14:53:14.770+0000: 189106.959: [GC pause (G1 Evacuation Pause) > (young) > Desired survivor size 654311424 bytes, new threshold 15 (max 15) > - age 1: 240262896 bytes, 240262896 total > - age 2: 3476760 bytes, 243739656 total > - age 3: 3293240 bytes, 247032896 total > - age 4: 3147072 bytes, 250179968 total > - age 5: 420832 bytes, 250600800 total > - age 6: 614688 bytes, 251215488 total > - age 7: 1139960 bytes, 252355448 total > - age 8: 632088 bytes, 252987536 total > - age 9: 425488 bytes, 253413024 total > - age 10: 1592608 bytes, 255005632 total > 189106.960: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 29363, predicted base time: 20.87 ms, remaining time: 19.13 > ms, target pause time: 40.00 ms] > 189106.960: [G1Ergonomics (CSet Construction) add young regions to CSet, > eden: 276 regions, survivors: 27 regions, predicted young region time: > 12.11 ms] > 189106.960: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: > 276 regions, survivors: 27 regions, old: 0 regions, predicted pause time: > 32.98 ms, target pause time: > 40.00 ms] > 189106.961: [G1Ergonomics (Heap Sizing) attempt heap expansion, reason: > region allocation request failed, allocation request: 5549208 bytes] > 189106.961: [G1Ergonomics (Heap Sizing) expand the heap, requested > expansion amount: 5549208 bytes, attempted expansion amount: 33554432 bytes] > 189106.961: [G1Ergonomics (Heap Sizing) did not expand the heap, reason: > heap already fully expanded] > 189114.730: [G1Ergonomics (Concurrent Cycles) do not request concurrent > cycle initiation, reason: still doing mixed collections, occupancy: > 13119782912 bytes, allocation request > : 0 bytes, threshold: 7730941095 bytes (45.00 %), source: end of GC] > 189114.730: [G1Ergonomics (Mixed GCs) start mixed GCs, reason: candidate > old regions available, candidate old regions: 45 regions, reclaimable: > 1026676456 bytes (5.98 %), threshold: 5.00 %] > (to-space exhausted), *7.7714626 secs*] > [Parallel Time: 7182.7 ms, GC Workers: 18] > {code} > > -- > Thanks & Regards, > Amit > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > -- > Sent from my phone > > > > > -- > Thanks & Regards, > Amit.Balode > > > > > > -- > Thanks & Regards, > Amit.Balode > > -- Sent from my phone -------------- next part -------------- An HTML attachment was scrubbed... URL: From tg at freigmbh.de Mon Jan 30 15:00:51 2017 From: tg at freigmbh.de (Thorsten Goetzke) Date: Mon, 30 Jan 2017 16:00:51 +0100 Subject: Unreachable Memory not freed In-Reply-To: References: <83689bdf-7285-3493-e7ab-c46b597bc02a@freigmbh.de> <349a5bf5-9c94-4578-5ad5-d13d0c3afc3a@oracle.com> Message-ID: <0e023b96-b5e6-9642-ee4b-205da144f11a@freigmbh.de> Hello, I have an update. It seems like the unreported/errorous gc Root is somewhere near "securiton.uls.tools.configuration.plugins.graphic.zoneeditor.data.svg.dynamics.impl.LiveItemImpl". I aggressivly nulled out the fields of instances of this Object and now the gc frees sufficient memory. It seems like the LiveItemImpl Objects themself are still leaking without any paths from gc root, but they no longer keep millions of Nodes alive causing our Application to go OutOfmemory within minutes. The Root Cause seems like something Nashorn is doing is confusing the JVM/GC to not free up memory in some specific circumstances. The LiveItemImpl Objects have incoming references from the internals of the nashorn framework. Best Regards, Thorsten Am 28.01.17 um 21:23 schrieb yu.zhang at oracle.com: > Thorsten, > > I have updated the bug with more details. I think there is some > inaccuracy when reporting the heap dump. > > There might be some memory leak in your application as well. > > Thanks > > Jenny > > On 01/27/2017 04:50 PM, Jenny Zhang wrote: >> Hi, Thorsten, >> >> I can reproduce this with a micro. I have filed >> https://bugs.openjdk.java.net/browse/JDK-8173594 >> >> The micro is mostly based on >> http://stackoverflow.com/questions/26027313/how-to-load-and-parse-svg-documents >> with some modification. I will upload the micro and details later. >> >> I think this is a issue how jvmti and gc reporting unreachable >> objects. Those objects should not be reported as unreachable. >> >> Thanks >> Jenny >> >> On 1/25/2017 3:23 PM, yu.zhang at oracle.com wrote: >>> Thorsten, >>> >>> Thanks very much for the heap dump and gc log. >>> >>> I assume the gc log and heap dump are from the same run. It is very >>> interesting, in the gc log: >>> >>> 2017-01-24T16:41:17.184-0100: 1301.633: [Full GC (Heap Dump Initiated >>> GC) 3817M->3816M(4096M), 8.9543444 secs] >>> [Eden: 2048.0K(2048.0M)->0.0B(2048.0M) Survivors: 0.0B->0.0B Heap: >>> 3817.7M(4096.0M)->3816.2M(4096.0M)], [Metaspace: >>> 142921K->142921K(1175552K)] >>> [Times: user=16.35 sys=0.11, real=8.95 secs] >>> >>> GC thinks there are 3816.2M of objects alive. But in the heap dump, >>> there is only 1.8G alive. >>> >>> I can see those unreachable objects with yourkit, mostly are >>> >>> org.apache.batik.dom.GenerecText >>> >>> But apparently gc does not think those are dead. It could be a bug in >>> gc, or jvmti dumping the heap, or tool(yourkit, apache MAT). I need >>> to discuss this with dev team. >>> >>> Back to the gc log, it seems the application allocates a lot of >>> humongous objects. I can see some int[] of 1.5m. In general the heap >>> is very full, with a lot of to-space exhausted, and full gc. And the >>> heap usage after full gc is almost the whole heap (4096M). >>> >>> Thanks >>> >>> Jenny >>> >>> >>> On 01/25/2017 01:15 AM, Thorsten Goetzke wrote: >>>> Hello, >>>> >>>> You can find the heap dump and gc logs under >>>> https://filesender.freigmbh.de/filesender/?vid=10f1aeb6-82b4-3d09-c201-00002042c681 >>>> >>>> >>>> Link is valid until 28.01.2017 (1/28/2017). Feel free to message me >>>> if you need a reupload. >>>> >>>> Best Regards, >>>> Thorsten >>>> >>>> >>>>> Thorsten, >>>>> >>>>> Can you please put the heap dump and gc logs somewhere for download? >>>>> >>>>> Thanks >>>>> Jenny >>>>> >>>>> On 1/24/2017 8:30 AM, Thorsten Goetzke wrote: >>>>>> Hello, >>>>>> >>>>>> I am working on a closed source software that uses a patched version >>>>>> of apache-batik to display SVG-Documents one by one. If you keep >>>>>> loading and discarding some specific SVG-Documents memory consumption >>>>>> keeps on growing until it reaches more than 95% of the available >>>>>> memory. The application becomes unusable slow, very little memory >>>>>> gets >>>>>> freed, although the gc is constantly running. Sometimes an >>>>>> OutOfMemoryError gets thrown. I took memory snaphsots and analyzed >>>>>> them using jvisualvm and yourkit. These tools report the majority of >>>>>> the memory as unreachable. >>>>>> I have tried different garbage collectors (G1,ConcurrentMarkSweep), >>>>>> and different Versions of Java: 1.8_60;1.8_65;1.8_102, the issue is >>>>>> always reproducible. >>>>>> Is there something i can do to debug this further? I am currently not >>>>>> able to reproduce this issue in a somewhat small self contained >>>>>> example. >>>>>> I could provide gc-logs and the heap dump (about 1gb zipped) or any >>>>>> other debuging info if someone is interested. >>>>>> >>>>>> Best Regards, >>>>>> Thorsten Goetzke >>>>>> >>>> >>>> _______________________________________________ >>>> hotspot-gc-use mailing list >>>> hotspot-gc-use at openjdk.java.net >>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>> >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > From thomas.schatzl at oracle.com Tue Jan 31 10:51:06 2017 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Tue, 31 Jan 2017 11:51:06 +0100 Subject: CMS large objects vs G1 humongous allocations In-Reply-To: References: Message-ID: <1485859866.3425.7.camel@oracle.com> Hi, ? just commenting a bit on Vitaly's advice: On Mon, 2017-01-30 at 13:49 +0000, Vitaly Davidovich wrote: > Also, you can experiment with relaxing the pause time goal a bit.? G1 > will use it as a heuristic to determine how many old regions (during > mixed GC) to add to a collection.? If you add too few, you're not > reclaiming fast enough (potentially) for your allocation rate. > > Is your entire gc log too big? If not, might be good to attach it or > put it somewhere (e.g. pastebin) so we can see the bigger picture. Agree. It is definitely helpful to see more than just a single GC. Often the context determines the appropriate response. > > On Mon, Jan 30, 2017 at 8:38 AM Vitaly Davidovich > wrote: > > On Mon, Jan 30, 2017 at 8:19 AM, Amit Balode > > wrote: > > > Italy, thank you. > > > > > > There is no explicit line which says Full GC. I thought line > > > printed as "(Heap Sizing) did not expand the heap, reason: heap > > > already fully expanded" is an implicit indication that G1 have > > > taken all necessary actions which full GC would take and it lead > > > to 7.7sec pause. > > > > > Oh really, there's no Full GC that ensues after those lines? :) Sometimes trying to just continue opposed to start a full gc right away pays off. > > > > The (Heap Sizing) output is G1's ergonomic output.? It's saying > > that it would like to expand the heap because it failed to allocate > > a new region out of the existing heap.? The reason for expansion is > > G1 may start out with a heap smaller than capacity (i.e. max heap > > size), and will try to expand the heap (if under capacity/max) as > > needed.? Here, it cannot expand because you're already at max heap > > size. > > > > The 7.7s pause is due to to-space exhaustion, which basically means > > G1 ran out of space to copy survivors (and you can see that this 7s > > is all in the Object Copy phase of the pause).? In my experience, > > when you start seeing to-space exhaustion, you pretty soon see a > > Full GC. With JDK9 to-space exhaustion management got significantly faster. It should be very close to a regular GC in most cases. In JDK8, the only way to make this problem disappear is as mentioned by Vitaly to avoid them, either by ? - increasing the heap (to let marking have more time) ? - decrease the initial heap occupancy (to start marking earlier) via -XX:InitialHeapOccupancyPercent ? - increase the speed of marking by increasing the number of marking threads via -XX:ConcGCThreads. > > > Sorry a bit confused about how can 'G1's concurrent cycles' & > > > 'parallel phases of young collections' run concurrently? Does > > > that mean?ConcGCThreads + ParallelGCThreads cannot be greater > > > than cpu cores.? > > > > > G1 has concurrent GC cycles, such as concurrent marking - this runs > > in the background, on the ConcGCThreads, and marks old regions to > > gather their liveness information (so that these regions can either > > be cleaned up in the background, if fully empty, or they'll be > > added to young GCs to make them mixed GCs).? At the same time, > > while that concurrent work is ongoing, your Java threads continue > > to run and allocate.? You can then hit a young pause, where > > ParallelGCThreads will perform some of the parallel work associated > > with a young GC.? Young GCs look at eden regions (and do some > > processing of them in parallel), while concurrent GC threads can be > > processing old regions at the same time.? So yes, you can > > theoretically hit a case where you have ConcGCThreads + > > ParallelGCThreads threads are runnable, and you may oversubscribe > > the machine (assuming each of those threads don't stall themselves > > due to internal issues, such as lock contention, etc).? The marking threads will always suspend during GC pauses, so this situation can't occur. Thanks, ? Thomas From vitalyd at gmail.com Tue Jan 31 11:04:00 2017 From: vitalyd at gmail.com (Vitaly Davidovich) Date: Tue, 31 Jan 2017 11:04:00 +0000 Subject: CMS large objects vs G1 humongous allocations In-Reply-To: <1485859866.3425.7.camel@oracle.com> References: <1485859866.3425.7.camel@oracle.com> Message-ID: On Tue, Jan 31, 2017 at 5:51 AM Thomas Schatzl wrote: > Hi, > > just commenting a bit on Vitaly's advice: > > On Mon, 2017-01-30 at 13:49 +0000, Vitaly Davidovich wrote: > > Also, you can experiment with relaxing the pause time goal a bit. G1 > > will use it as a heuristic to determine how many old regions (during > > mixed GC) to add to a collection. If you add too few, you're not > > reclaiming fast enough (potentially) for your allocation rate. > > > > Is your entire gc log too big? If not, might be good to attach it or > > put it somewhere (e.g. pastebin) so we can see the bigger picture. > > Agree. It is definitely helpful to see more than just a single GC. > Often the context determines the appropriate response. > > > > > On Mon, Jan 30, 2017 at 8:38 AM Vitaly Davidovich > > wrote: > > > On Mon, Jan 30, 2017 at 8:19 AM, Amit Balode > > > wrote: > > > > Italy, thank you. > > > > > > > > There is no explicit line which says Full GC. I thought line > > > > printed as "(Heap Sizing) did not expand the heap, reason: heap > > > > already fully expanded" is an implicit indication that G1 have > > > > taken all necessary actions which full GC would take and it lead > > > > to 7.7sec pause. > > > > > > > Oh really, there's no Full GC that ensues after those lines? > > :) Sometimes trying to just continue opposed to start a full gc right > away pays off. > > > > > > > The (Heap Sizing) output is G1's ergonomic output. It's saying > > > that it would like to expand the heap because it failed to allocate > > > a new region out of the existing heap. The reason for expansion is > > > G1 may start out with a heap smaller than capacity (i.e. max heap > > > size), and will try to expand the heap (if under capacity/max) as > > > needed. Here, it cannot expand because you're already at max heap > > > size. > > > > > > The 7.7s pause is due to to-space exhaustion, which basically means > > > G1 ran out of space to copy survivors (and you can see that this 7s > > > is all in the Object Copy phase of the pause). In my experience, > > > when you start seeing to-space exhaustion, you pretty soon see a > > > Full GC. > > With JDK9 to-space exhaustion management got significantly faster. It > should be very close to a regular GC in most cases. > > In JDK8, the only way to make this problem disappear is as mentioned by > Vitaly to avoid them, either by > - increasing the heap (to let marking have more time) > - decrease the initial heap occupancy (to start marking earlier) via > -XX:InitialHeapOccupancyPercent > - increase the speed of marking by increasing the number of marking > threads via -XX:ConcGCThreads. > > > > > Sorry a bit confused about how can 'G1's concurrent cycles' & > > > > 'parallel phases of young collections' run concurrently? Does > > > > that mean ConcGCThreads + ParallelGCThreads cannot be greater > > > > than cpu cores.? > > > > > > > G1 has concurrent GC cycles, such as concurrent marking - this runs > > > in the background, on the ConcGCThreads, and marks old regions to > > > gather their liveness information (so that these regions can either > > > be cleaned up in the background, if fully empty, or they'll be > > > added to young GCs to make them mixed GCs). At the same time, > > > while that concurrent work is ongoing, your Java threads continue > > > to run and allocate. You can then hit a young pause, where > > > ParallelGCThreads will perform some of the parallel work associated > > > with a young GC. Young GCs look at eden regions (and do some > > > processing of them in parallel), while concurrent GC threads can be > > > processing old regions at the same time. So yes, you can > > > theoretically hit a case where you have ConcGCThreads + > > > ParallelGCThreads threads are runnable, and you may oversubscribe > > > the machine (assuming each of those threads don't stall themselves > > > due to internal issues, such as lock contention, etc). > > The marking threads will always suspend during GC pauses, so this > situation can't occur. Ah, thanks for the correction Thomas. I thought they continue as they would be looking at old regions while a young GC is looking at eden/survivor. > > > Thanks, > Thomas > -- Sent from my phone -------------- next part -------------- An HTML attachment was scrubbed... URL: From amit.balode at gmail.com Tue Jan 31 16:34:50 2017 From: amit.balode at gmail.com (Amit Balode) Date: Tue, 31 Jan 2017 22:04:50 +0530 Subject: CMS large objects vs G1 humongous allocations In-Reply-To: References: <1485859866.3425.7.camel@oracle.com> Message-ID: File is a bit huge:) linked it here https://raw.githubusercontent.com/amitbalode/uploads/master/amit.txt On Tue, Jan 31, 2017 at 4:34 PM, Vitaly Davidovich wrote: > > On Tue, Jan 31, 2017 at 5:51 AM Thomas Schatzl > wrote: > >> Hi, >> >> just commenting a bit on Vitaly's advice: >> >> On Mon, 2017-01-30 at 13:49 +0000, Vitaly Davidovich wrote: >> > Also, you can experiment with relaxing the pause time goal a bit. G1 >> > will use it as a heuristic to determine how many old regions (during >> > mixed GC) to add to a collection. If you add too few, you're not >> > reclaiming fast enough (potentially) for your allocation rate. >> > >> > Is your entire gc log too big? If not, might be good to attach it or >> > put it somewhere (e.g. pastebin) so we can see the bigger picture. >> >> Agree. It is definitely helpful to see more than just a single GC. >> Often the context determines the appropriate response. >> >> > >> > On Mon, Jan 30, 2017 at 8:38 AM Vitaly Davidovich >> > wrote: >> > > On Mon, Jan 30, 2017 at 8:19 AM, Amit Balode > > > > wrote: >> > > > Italy, thank you. >> > > > >> > > > There is no explicit line which says Full GC. I thought line >> > > > printed as "(Heap Sizing) did not expand the heap, reason: heap >> > > > already fully expanded" is an implicit indication that G1 have >> > > > taken all necessary actions which full GC would take and it lead >> > > > to 7.7sec pause. >> > > > >> > > Oh really, there's no Full GC that ensues after those lines? >> >> :) Sometimes trying to just continue opposed to start a full gc right >> away pays off. >> >> > > >> > > The (Heap Sizing) output is G1's ergonomic output. It's saying >> > > that it would like to expand the heap because it failed to allocate >> > > a new region out of the existing heap. The reason for expansion is >> > > G1 may start out with a heap smaller than capacity (i.e. max heap >> > > size), and will try to expand the heap (if under capacity/max) as >> > > needed. Here, it cannot expand because you're already at max heap >> > > size. >> > > >> > > The 7.7s pause is due to to-space exhaustion, which basically means >> > > G1 ran out of space to copy survivors (and you can see that this 7s >> > > is all in the Object Copy phase of the pause). In my experience, >> > > when you start seeing to-space exhaustion, you pretty soon see a >> > > Full GC. >> >> With JDK9 to-space exhaustion management got significantly faster. It >> should be very close to a regular GC in most cases. >> >> In JDK8, the only way to make this problem disappear is as mentioned by >> Vitaly to avoid them, either by >> - increasing the heap (to let marking have more time) >> - decrease the initial heap occupancy (to start marking earlier) via >> -XX:InitialHeapOccupancyPercent >> - increase the speed of marking by increasing the number of marking >> threads via -XX:ConcGCThreads. >> >> > > > Sorry a bit confused about how can 'G1's concurrent cycles' & >> > > > 'parallel phases of young collections' run concurrently? Does >> > > > that mean ConcGCThreads + ParallelGCThreads cannot be greater >> > > > than cpu cores.? >> > > > >> > > G1 has concurrent GC cycles, such as concurrent marking - this runs >> > > in the background, on the ConcGCThreads, and marks old regions to >> > > gather their liveness information (so that these regions can either >> > > be cleaned up in the background, if fully empty, or they'll be >> > > added to young GCs to make them mixed GCs). At the same time, >> > > while that concurrent work is ongoing, your Java threads continue >> > > to run and allocate. You can then hit a young pause, where >> > > ParallelGCThreads will perform some of the parallel work associated >> > > with a young GC. Young GCs look at eden regions (and do some >> > > processing of them in parallel), while concurrent GC threads can be >> > > processing old regions at the same time. So yes, you can >> > > theoretically hit a case where you have ConcGCThreads + >> > > ParallelGCThreads threads are runnable, and you may oversubscribe >> > > the machine (assuming each of those threads don't stall themselves >> > > due to internal issues, such as lock contention, etc). >> >> The marking threads will always suspend during GC pauses, so this >> situation can't occur. > > Ah, thanks for the correction Thomas. I thought they continue as they > would be looking at old regions while a young GC is looking at > eden/survivor. > >> >> >> Thanks, >> Thomas >> > -- > Sent from my phone > -- Thanks & Regards, Amit.Balode -------------- next part -------------- An HTML attachment was scrubbed... URL: