From taras.tielkes at gmail.com Wed Feb 1 04:16:37 2012 From: taras.tielkes at gmail.com (Taras Tielkes) Date: Wed, 1 Feb 2012 13:16:37 +0100 Subject: Promotion failures: indication of CMS fragmentation? In-Reply-To: References: <4EF9FCAC.3030208@oracle.com> <4F06A270.3010701@oracle.com> <4F0DBEC4.7040907@oracle.com> <4F1ECE7B.3040502@oracle.com> <4F1F2ED7.6060308@oracle.com> <4F20F78D.9070905@oracle.com> Message-ID: Hi, We've been running the new release in production for almost a week now. The good news is that the frequency of promotion failures has decreased. In addition, the promotion failures with large sizes (exceeding 1M words) are completely gone. This is nice, as it shows that the elimination of large buffer abuse in our application has paid off. There are still remaining, occasional promotion failures though, which I've grouped into two buckets: 1) "Magic 1026" promotion failure On every machine, for every couple of days, we see a promotion failure with 1026 size. The failure size is *always* this number. --------------- 2012-01-27T01:28:34.135+0100: 39574.835: [GC 39574.835: [ParNew (promotion failure size = 1026) (promotion failed): 354844K->368640K(368640K), 0.2103220 secs]39575.046: [CMS: 2729550K->301144K(4833280K), 4.0552810 secs] 3068244K->301144K (5201920K), [CMS Perm : 124003K->123976K(262144K)], 4.2664030 secs] [Times: user=4.54 sys=0.01, real=4.27 secs] --------------- 2) "Parallel" promotion failure In this case all 8 ParNew collection threads fail. Typically the reported failure sizes will be between 8 and 30, with some threads sometimes reporting a larger failure size: --------------- 2012-01-28T00:05:04.418+0100: 120965.117: [GC 120965.118: [ParNew (0: promotion failure size = 4) (1: promotion failure size = 4) (2: promotion failure size = 9) (3: promotion failure size = 1027) (4: promotion failure size = 7) (5: promotion failure size = 4) (6: promotion failure size = 7) (7: promotion failure size = 4) (promotion failed): 368640K->368640K(368640K), 2.2517580 secs]120967.370: [CMS: 3210551K->295337K(4833280K), 4.3763650 secs] 3542272K->295337K( 5201920K), [CMS Perm : 124446K->124411K(262144K)], 6.6289510 secs] [Times: user=7.49 sys=0.52, real=6.63 secs] --------------- Both of these events are occurring near the configured (and fixed) CMS initiating occupancy fraction (68%). The next thing we'll try is to remove the CMSScavengeBeforeRemark flag, which we've had enabled up till now. If there are other things to investigate, or other data to collect, I'd be grateful for any suggestions. Kind regards, Taras On Thu, Jan 26, 2012 at 9:22 PM, Taras Tielkes wrote: > Hi Jon, > > Thanks for clearing up the word size question. > Over the past weeks, I've seen promotion failures exceeding 10M words, > so arrays of over 80 megabytes in size :) > Today, we deployed a production release that eliminates the huge > buffer allocations - instead data is processed in a streaming fashion. > > We'll see how things are performing after collecting operations data > for a few weeks. > > Thanks for your help, > Taras > > On Thu, Jan 26, 2012 at 7:49 AM, Jon Masamitsu wrote: >> >> >> On 1/25/2012 2:41 AM, Taras Tielkes wrote: >>> Hi Jon, >>> >>> At the risk of asking a stupid question, what's the word size on x64 >>> when using CompressedOops? >> >> Word size is the same with and without CompressedOops (8 bytes). ?With >> CompressedOops >> we can just point to words with a 32bit reference (i.e., map the 32bit >> reference to a full >> 64bit address). >> >> Jon >> >>> Thanks, >>> Taras >>> >>> On Tue, Jan 24, 2012 at 11:21 PM, Jon Masamitsu >>> ?wrote: >>>> On 01/24/12 10:15, Taras Tielkes wrote: >>>>> Hi Jon, >>>>> >>>>> Xmx is 5g, PermGen is 256m, new is 400m. >>>>> >>>>> The overall tenured gen usage is at the point when I would expect the >>>>> CMS to kick in though. >>>>> Does this mean I'd have to lower the CMS initiating occupancy setting >>>>> (currently at 68%)? >>>> I don't have any quick answers as to what to try next. >>>> >>>>> In addition, are the promotion failure sizes expressed in bytes? If >>>>> so, I'm surprised to see such odd-sized (7, for example) sizes. >>>> It's in words. >>>> >>>> Jon >>>>> Thanks, >>>>> Taras >>>>> >>>>> On Tue, Jan 24, 2012 at 4:30 PM, Jon Masamitsu ? ?wrote: >>>>>> Taras, >>>>>> >>>>>> The pattern makes sense if the tenured (cms) gen is in fact full. >>>>>> Multiple ?GC workers try to get a chunk of space for >>>>>> an allocation and there is no space. >>>>>> >>>>>> Jon >>>>>> >>>>>> >>>>>> On 01/24/12 04:22, Taras Tielkes wrote: >>>>>>> Hi Jon, >>>>>>> >>>>>>> While inspecting our production logs for promotion failures, I saw the >>>>>>> following one today: >>>>>>> -------- >>>>>>> 2012-01-24T08:37:26.118+0100: 1222467.411: [GC 1222467.411: [ParNew: >>>>>>> 349623K->20008K(368640K), 0.0294350 secs] >>>>>>> 3569266K->3239650K(5201920K), 0.0298770 secs] [Times: user=0.21 >>>>>>> sys=0.00, real=0.03 secs] >>>>>>> 2012-01-24T08:37:27.497+0100: 1222468.790: [GC 1222468.791: [ParNew: >>>>>>> 347688K->40960K(368640K), 0.0536700 secs] >>>>>>> 3567330K->3284097K(5201920K), 0.0541200 secs] [Times: user=0.36 >>>>>>> sys=0.00, real=0.05 secs] >>>>>>> 2012-01-24T08:37:28.716+0100: 1222470.009: [GC 1222470.010: [ParNew >>>>>>> (0: promotion failure size = 6) ?(1: promotion failure size = 6) ?(2: >>>>>>> promotion failure size = 7) ?(3: promotion failure size = 7) ?(4: >>>>>>> promotion failure size = 9) ?(5: p >>>>>>> romotion failure size = 9) ?(6: promotion failure size = 6) ?(7: >>>>>>> promotion failure size = 9) ?(promotion failed): >>>>>>> 368640K->368640K(368640K), 3.1475760 secs]1222473.157: [CMS: >>>>>>> 3315844K->559299K(4833280K), 5.9647110 secs] 3611777K->559299K( >>>>>>> 5201920K), [CMS Perm : 118085K->118072K(262144K)], 9.1128700 secs] >>>>>>> [Times: user=10.17 sys=1.10, real=9.11 secs] >>>>>>> 2012-01-24T08:37:38.601+0100: 1222479.894: [GC 1222479.895: [ParNew: >>>>>>> 327680K->40960K(368640K), 0.0635680 secs] 886979K->624773K(5201920K), >>>>>>> 0.0641090 secs] [Times: user=0.37 sys=0.00, real=0.07 secs] >>>>>>> 2012-01-24T08:37:40.642+0100: 1222481.936: [GC 1222481.936: [ParNew: >>>>>>> 368640K->38479K(368640K), 0.0771480 secs] 952453K->659708K(5201920K), >>>>>>> 0.0776360 secs] [Times: user=0.40 sys=0.01, real=0.08 secs] >>>>>>> -------- >>>>>>> >>>>>>> It's different from the others in two ways: >>>>>>> 1) a "parallel" promotion failure in all 8 ParNew threads? >>>>>>> 2) the very small size of the promoted object >>>>>>> >>>>>>> Do such an promotion failure pattern ring a bell? It does not make sense >>>>>>> to me. >>>>>>> >>>>>>> Thanks, >>>>>>> Taras >>>>>>> >>>>>>> On Wed, Jan 11, 2012 at 5:54 PM, Jon Masamitsu >>>>>>> ? ?wrote: >>>>>>>> Taras, >>>>>>>> >>>>>>>>> I assume that the large sizes for the promotion failures during ParNew >>>>>>>>> are confirming that eliminating large array allocations might help >>>>>>>>> here. Do you agree? >>>>>>>> I agree that eliminating the large array allocation will help but you >>>>>>>> are still having >>>>>>>> promotion failures when the allocation size is small (I think it was >>>>>>>> 1026). ?That >>>>>>>> says that you are filling up the old (cms) generation faster than the GC >>>>>>>> can >>>>>>>> collect it. ?The large arrays are aggrevating the problem but not >>>>>>>> necessarily >>>>>>>> the cause. >>>>>>>> >>>>>>>> If these are still your heap sizes, >>>>>>>> >>>>>>>>> -Xms5g >>>>>>>>> -Xmx5g >>>>>>>>> -Xmn400m >>>>>>>> Start by increasing the young gen size as may already have been >>>>>>>> suggested. ?If you have a test setup where you can experiment, >>>>>>>> try doubling the young gen size to start. >>>>>>>> >>>>>>>> If you have not seen this, it might be helpful. >>>>>>>> >>>>>>>> http://blogs.oracle.com/jonthecollector/entry/what_the_heck_s_a >>>>>>>>> I'm not sure what to make of the concurrent mode >>>>>>>> The concurrent mode failure is a consequence of the promotion failure. >>>>>>>> Once the promotion failure happens the concurrent mode failure is >>>>>>>> inevitable. >>>>>>>> >>>>>>>> Jon >>>>>>>> >>>>>>>> >>>>>>>>> . >>>>>>>> On 1/11/2012 3:00 AM, Taras Tielkes wrote: >>>>>>>>> Hi Jon, >>>>>>>>> >>>>>>>>> We've added the -XX:+PrintPromotionFailure flag to our production >>>>>>>>> application yesterday. >>>>>>>>> The application is running on 4 (homogenous) nodes. >>>>>>>>> >>>>>>>>> In the gc logs of 3 out of 4 nodes, I've found a promotion failure >>>>>>>>> event during ParNew: >>>>>>>>> >>>>>>>>> node-002 >>>>>>>>> ------- >>>>>>>>> 2012-01-11T09:39:14.353+0100: 102975.594: [GC 102975.594: [ParNew: >>>>>>>>> 357592K->23382K(368640K), 0.0298150 secs] >>>>>>>>> 3528237K->3194027K(5201920K), 0.0300860 secs] [Times: user=0.22 >>>>>>>>> sys=0.01, real=0.03 secs] >>>>>>>>> 2012-01-11T09:39:17.489+0100: 102978.730: [GC 102978.730: [ParNew: >>>>>>>>> 351062K->39795K(368640K), 0.0401170 secs] >>>>>>>>> 3521707K->3210439K(5201920K), 0.0403800 secs] [Times: user=0.28 >>>>>>>>> sys=0.00, real=0.04 secs] >>>>>>>>> 2012-01-11T09:39:19.869+0100: 102981.110: [GC 102981.110: [ParNew (4: >>>>>>>>> promotion failure size = 4281460) ?(promotion failed): >>>>>>>>> 350134K->340392K(368640K), 0.1378780 secs]102981.248: [CMS: >>>>>>>>> 3181346K->367952K(4833280K), 4.7036230 secs] 3520778K >>>>>>>>> ->367952K(5201920K), [CMS Perm : 116828K->116809K(262144K)], 4.8418590 >>>>>>>>> secs] [Times: user=5.10 sys=0.00, real=4.84 secs] >>>>>>>>> 2012-01-11T09:39:25.264+0100: 102986.504: [GC 102986.505: [ParNew: >>>>>>>>> 327680K->40960K(368640K), 0.0415470 secs] 695632K->419560K(5201920K), >>>>>>>>> 0.0418770 secs] [Times: user=0.26 sys=0.01, real=0.04 secs] >>>>>>>>> 2012-01-11T09:39:26.035+0100: 102987.276: [GC 102987.276: [ParNew: >>>>>>>>> 368640K->40960K(368640K), 0.0925740 secs] 747240K->481611K(5201920K), >>>>>>>>> 0.0928570 secs] [Times: user=0.54 sys=0.01, real=0.09 secs] >>>>>>>>> >>>>>>>>> node-003 >>>>>>>>> ------- >>>>>>>>> 2012-01-10T17:48:28.369+0100: 45929.686: [GC 45929.686: [ParNew: >>>>>>>>> 346950K->21342K(368640K), 0.0333090 secs] >>>>>>>>> 2712364K->2386756K(5201920K), 0.0335740 secs] [Times: user=0.23 >>>>>>>>> sys=0.00, real=0.03 secs] >>>>>>>>> 2012-01-10T17:48:32.933+0100: 45934.250: [GC 45934.250: [ParNew: >>>>>>>>> 345070K->32211K(368640K), 0.0369260 secs] >>>>>>>>> 2710484K->2397625K(5201920K), 0.0372380 secs] [Times: user=0.25 >>>>>>>>> sys=0.00, real=0.04 secs] >>>>>>>>> 2012-01-10T17:48:34.201+0100: 45935.518: [GC 45935.518: [ParNew (0: >>>>>>>>> promotion failure size = 1266955) ?(promotion failed): >>>>>>>>> 359891K->368640K(368640K), 0.1395570 secs]45935.658: [CMS: >>>>>>>>> 2387690K->348838K(4833280K), 4.5680670 secs] 2725305K->3 >>>>>>>>> 48838K(5201920K), [CMS Perm : 116740K->116715K(262144K)], 4.7079640 >>>>>>>>> secs] [Times: user=5.03 sys=0.00, real=4.71 secs] >>>>>>>>> 2012-01-10T17:48:40.572+0100: 45941.889: [GC 45941.889: [ParNew: >>>>>>>>> 327680K->40960K(368640K), 0.0486510 secs] 676518K->405004K(5201920K), >>>>>>>>> 0.0489930 secs] [Times: user=0.26 sys=0.00, real=0.05 secs] >>>>>>>>> 2012-01-10T17:48:41.959+0100: 45943.276: [GC 45943.277: [ParNew: >>>>>>>>> 360621K->40960K(368640K), 0.0833240 secs] 724666K->479857K(5201920K), >>>>>>>>> 0.0836120 secs] [Times: user=0.48 sys=0.01, real=0.08 secs] >>>>>>>>> >>>>>>>>> node-004 >>>>>>>>> ------- >>>>>>>>> 2012-01-10T18:59:02.338+0100: 50163.649: [GC 50163.649: [ParNew: >>>>>>>>> 358429K->40960K(368640K), 0.0629910 secs] >>>>>>>>> 3569331K->3283304K(5201920K), 0.0632710 secs] [Times: user=0.40 >>>>>>>>> sys=0.02, real=0.06 secs] >>>>>>>>> 2012-01-10T18:59:08.137+0100: 50169.448: [GC 50169.448: [ParNew: >>>>>>>>> 368640K->40960K(368640K), 0.0819780 secs] >>>>>>>>> 3610984K->3323445K(5201920K), 0.0822430 secs] [Times: user=0.40 >>>>>>>>> sys=0.00, real=0.08 secs] >>>>>>>>> 2012-01-10T18:59:13.945+0100: 50175.256: [GC 50175.256: [ParNew (6: >>>>>>>>> promotion failure size = 2788662) ?(promotion failed): >>>>>>>>> 367619K->364864K(368640K), 0.2024350 secs]50175.458: [CMS: >>>>>>>>> 3310044K->330922K(4833280K), 4.5104170 secs] >>>>>>>>> 3650104K->330922K(5201920K), [CMS Perm : 116747K->116728K(262144K)], >>>>>>>>> 4.7132220 secs] [Times: user=4.99 sys=0.01, real=4.72 secs] >>>>>>>>> 2012-01-10T18:59:20.539+0100: 50181.850: [GC 50181.850: [ParNew: >>>>>>>>> 327680K->37328K(368640K), 0.0270660 secs] 658602K->368251K(5201920K), >>>>>>>>> 0.0273800 secs] [Times: user=0.15 sys=0.00, real=0.02 secs] >>>>>>>>> 2012-01-10T18:59:25.183+0100: 50186.494: [GC 50186.494: [ParNew: >>>>>>>>> 363504K->15099K(368640K), 0.0388710 secs] 694427K->362063K(5201920K), >>>>>>>>> 0.0391790 secs] [Times: user=0.18 sys=0.00, real=0.04 secs] >>>>>>>>> >>>>>>>>> On a fourth node, I've found a different event: promotion failure >>>>>>>>> during CMS, with a much smaller size: >>>>>>>>> >>>>>>>>> node-001 >>>>>>>>> ------- >>>>>>>>> 2012-01-10T18:30:07.471+0100: 48428.764: [GC 48428.764: [ParNew: >>>>>>>>> 354039K->40960K(368640K), 0.0667340 secs] >>>>>>>>> 3609061K->3318149K(5201920K), 0.0670150 secs] [Times: user=0.37 >>>>>>>>> sys=0.01, real=0.06 secs] >>>>>>>>> 2012-01-10T18:30:08.706+0100: 48429.999: [GC 48430.000: [ParNew: >>>>>>>>> 368640K->40960K(368640K), 0.2586390 secs] >>>>>>>>> 3645829K->3417273K(5201920K), 0.2589050 secs] [Times: user=0.73 >>>>>>>>> sys=0.13, real=0.26 secs] >>>>>>>>> 2012-01-10T18:30:08.974+0100: 48430.267: [GC [1 CMS-initial-mark: >>>>>>>>> 3376313K(4833280K)] 3427492K(5201920K), 0.0743900 secs] [Times: >>>>>>>>> user=0.07 sys=0.00, real=0.07 secs] >>>>>>>>> 2012-01-10T18:30:09.049+0100: 48430.342: [CMS-concurrent-mark-start] >>>>>>>>> 2012-01-10T18:30:10.009+0100: 48431.302: [CMS-concurrent-mark: >>>>>>>>> 0.933/0.960 secs] [Times: user=4.59 sys=0.13, real=0.96 secs] >>>>>>>>> 2012-01-10T18:30:10.009+0100: 48431.302: [CMS-concurrent-preclean-start] >>>>>>>>> 2012-01-10T18:30:10.089+0100: 48431.382: [CMS-concurrent-preclean: >>>>>>>>> 0.060/0.080 secs] [Times: user=0.34 sys=0.02, real=0.08 secs] >>>>>>>>> 2012-01-10T18:30:10.089+0100: 48431.382: >>>>>>>>> [CMS-concurrent-abortable-preclean-start] >>>>>>>>> 2012-01-10T18:30:10.586+0100: 48431.880: [GC 48431.880: [ParNew: >>>>>>>>> 368640K->40960K(368640K), 0.1214420 secs] >>>>>>>>> 3744953K->3490912K(5201920K), 0.1217480 secs] [Times: user=0.66 >>>>>>>>> sys=0.05, real=0.12 secs] >>>>>>>>> 2012-01-10T18:30:12.785+0100: 48434.078: >>>>>>>>> [CMS-concurrent-abortable-preclean: 2.526/2.696 secs] [Times: >>>>>>>>> user=10.72 sys=0.48, real=2.70 secs] >>>>>>>>> 2012-01-10T18:30:12.787+0100: 48434.081: [GC[YG occupancy: 206521 K >>>>>>>>> (368640 K)]2012-01-10T18:30:12.788+0100: 48434.081: [GC 48434.081: >>>>>>>>> [ParNew (promotion failure size = 1026) ?(promotion failed): >>>>>>>>> 206521K->206521K(368640K), 0.1667280 secs] >>>>>>>>> ? ? 3656474K->3696197K(5201920K), 0.1670260 secs] [Times: user=0.48 >>>>>>>>> sys=0.04, real=0.17 secs] >>>>>>>>> 48434.248: [Rescan (parallel) , 0.1972570 secs]48434.445: [weak refs >>>>>>>>> processing, 0.0011570 secs]48434.446: [class unloading, 0.0277750 >>>>>>>>> secs]48434.474: [scrub symbol& ? ? ? ?string tables, 0.0088370 secs] [1 >>>>>>>>> CMS-remark: 3489675K(4833280K)] 36961 >>>>>>>>> 97K(5201920K), 0.4088040 secs] [Times: user=1.62 sys=0.05, real=0.41 >>>>>>>>> secs] >>>>>>>>> 2012-01-10T18:30:13.197+0100: 48434.490: [CMS-concurrent-sweep-start] >>>>>>>>> 2012-01-10T18:30:17.427+0100: 48438.720: [Full GC 48438.720: >>>>>>>>> [CMS2012-01-10T18:30:21.636+0100: 48442.929: [CMS-concurrent-sweep: >>>>>>>>> 7.949/8.439 secs] [Times: user=15.89 sys=1.57, real=8.44 secs] >>>>>>>>> ? ? (concurrent mode failure): 2505348K->334385K(4833280K), 8.6109050 >>>>>>>>> secs] 2873988K->334385K(5201920K), [CMS Perm : >>>>>>>>> 117788K->117762K(262144K)], 8.6112520 secs] [Times: user=8.61 >>>>>>>>> sys=0.00, real=8.61 secs] >>>>>>>>> 2012-01-10T18:30:26.716+0100: 48448.009: [GC 48448.010: [ParNew: >>>>>>>>> 327680K->40960K(368640K), 0.0407520 secs] 662065K->394656K(5201920K), >>>>>>>>> 0.0411550 secs] [Times: user=0.25 sys=0.00, real=0.04 secs] >>>>>>>>> 2012-01-10T18:30:28.825+0100: 48450.118: [GC 48450.118: [ParNew: >>>>>>>>> 368639K->40960K(368640K), 0.0662780 secs] 722335K->433355K(5201920K), >>>>>>>>> 0.0666190 secs] [Times: user=0.35 sys=0.00, real=0.06 secs] >>>>>>>>> >>>>>>>>> I assume that the large sizes for the promotion failures during ParNew >>>>>>>>> are confirming that eliminating large array allocations might help >>>>>>>>> here. Do you agree? >>>>>>>>> I'm not sure what to make of the concurrent mode failure. >>>>>>>>> >>>>>>>>> Thanks in advance for any suggestions, >>>>>>>>> Taras >>>>>>>>> >>>>>>>>> On Fri, Jan 6, 2012 at 8:27 AM, Jon Masamitsu >>>>>>>>> ? ? ?wrote: >>>>>>>>>> On 1/5/2012 3:32 PM, Taras Tielkes wrote: >>>>>>>>>>> Hi Jon, >>>>>>>>>>> >>>>>>>>>>> We've enabled the PrintPromotionFailure flag in our preprod >>>>>>>>>>> environment, but so far, no failures yet. >>>>>>>>>>> We know that the load we generate there is not representative. But >>>>>>>>>>> perhaps we'll catch something, given enough patience. >>>>>>>>>>> >>>>>>>>>>> The flag will also be enabled in our production environment next week >>>>>>>>>>> - so one way or the other, we'll get more diagnostic data soon. >>>>>>>>>>> I'll also do some allocation profiling of the application in isolation >>>>>>>>>>> - I know that there is abusive large byte[] and char[] allocation in >>>>>>>>>>> there. >>>>>>>>>>> >>>>>>>>>>> I've got two questions for now: >>>>>>>>>>> >>>>>>>>>>> 1) From googling around on the output to expect >>>>>>>>>>> (http://blog.ragozin.info/2011/10/java-cg-hotspots-cms-and-heap.html), >>>>>>>>>>> I see that -XX:+PrintPromotionFailure will generate output like this: >>>>>>>>>>> ------- >>>>>>>>>>> 592.079: [ParNew (0: promotion failure size = 2698) ?(promotion >>>>>>>>>>> failed): 135865K->134943K(138240K), 0.1433555 secs] >>>>>>>>>>> ------- >>>>>>>>>>> In that example line, what does the "0" stand for? >>>>>>>>>> It's the index of the GC worker thread ?that experienced the promotion >>>>>>>>>> failure. >>>>>>>>>> >>>>>>>>>>> 2) Below is a snippet of (real) gc log from our production >>>>>>>>>>> application: >>>>>>>>>>> ------- >>>>>>>>>>> 2011-12-30T22:42:12.684+0100: 2136581.585: [GC 2136581.585: [ParNew: >>>>>>>>>>> 345951K->40960K(368640K), 0.0676780 secs] >>>>>>>>>>> 3608692K->3323692K(5201920K), 0.0680220 secs] [Times: user=0.36 >>>>>>>>>>> sys=0.01, real=0.06 secs] >>>>>>>>>>> 2011-12-30T22:42:22.984+0100: 2136591.886: [GC 2136591.886: [ParNew: >>>>>>>>>>> 368640K->40959K(368640K), 0.0618880 secs] >>>>>>>>>>> 3651372K->3349928K(5201920K), 0.0622330 secs] [Times: user=0.31 >>>>>>>>>>> sys=0.00, real=0.06 secs] >>>>>>>>>>> 2011-12-30T22:42:23.052+0100: 2136591.954: [GC [1 CMS-initial-mark: >>>>>>>>>>> 3308968K(4833280K)] 3350041K(5201920K), 0.0377420 secs] [Times: >>>>>>>>>>> user=0.04 sys=0.00, real=0.04 secs] >>>>>>>>>>> 2011-12-30T22:42:23.090+0100: 2136591.992: [CMS-concurrent-mark-start] >>>>>>>>>>> 2011-12-30T22:42:24.076+0100: 2136592.978: [CMS-concurrent-mark: >>>>>>>>>>> 0.986/0.986 secs] [Times: user=2.05 sys=0.04, real=0.99 secs] >>>>>>>>>>> 2011-12-30T22:42:24.076+0100: 2136592.978: >>>>>>>>>>> [CMS-concurrent-preclean-start] >>>>>>>>>>> 2011-12-30T22:42:24.099+0100: 2136593.000: [CMS-concurrent-preclean: >>>>>>>>>>> 0.021/0.023 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] >>>>>>>>>>> 2011-12-30T22:42:24.099+0100: 2136593.001: >>>>>>>>>>> [CMS-concurrent-abortable-preclean-start] >>>>>>>>>>> ? ? ?CMS: abort preclean due to time 2011-12-30T22:42:29.335+0100: >>>>>>>>>>> 2136598.236: [CMS-concurrent-abortable-preclean: 5.209/5.236 secs] >>>>>>>>>>> [Times: user=5.70 sys=0.23, real=5.23 secs] >>>>>>>>>>> 2011-12-30T22:42:29.340+0100: 2136598.242: [GC[YG occupancy: 123870 K >>>>>>>>>>> (368640 K)]2011-12-30T22:42:29.341+0100: 2136598.242: [GC 2136598.242: >>>>>>>>>>> [ParNew (promotion failed): 123870K->105466K(368640K), 7.4939280 secs] >>>>>>>>>>> 3432839K->3423755K(5201920 >>>>>>>>>>> K), 7.4942670 secs] [Times: user=9.08 sys=2.10, real=7.49 secs] >>>>>>>>>>> 2136605.737: [Rescan (parallel) , 0.0644050 secs]2136605.801: [weak >>>>>>>>>>> refs processing, 0.0034280 secs]2136605.804: [class unloading, >>>>>>>>>>> 0.0289480 secs]2136605.833: [scrub symbol& ? ? ? ? ?string tables, >>>>>>>>>>> 0.0093940 >>>>>>>>>>> secs] [1 CMS-remark: 3318289K(4833280K >>>>>>>>>>> )] 3423755K(5201920K), 7.6077990 secs] [Times: user=9.54 sys=2.10, >>>>>>>>>>> real=7.61 secs] >>>>>>>>>>> 2011-12-30T22:42:36.949+0100: 2136605.850: >>>>>>>>>>> [CMS-concurrent-sweep-start] >>>>>>>>>>> 2011-12-30T22:42:45.006+0100: 2136613.907: [Full GC 2136613.908: >>>>>>>>>>> [CMS2011-12-30T22:42:51.038+0100: 2136619.939: [CMS-concurrent-sweep: >>>>>>>>>>> 12.231/14.089 secs] [Times: user=15.14 sys=5.36, real=14.08 secs] >>>>>>>>>>> ? ? ?(concurrent mode failure): 3141235K->291853K(4833280K), 10.2906040 >>>>>>>>>>> secs] 3491471K->291853K(5201920K), [CMS Perm : >>>>>>>>>>> 121784K->121765K(262144K)], 10.2910040 secs] [Times: user=10.29 >>>>>>>>>>> sys=0.00, real=10.29 secs] >>>>>>>>>>> 2011-12-30T22:42:56.281+0100: 2136625.183: [GC 2136625.183: [ParNew: >>>>>>>>>>> 327680K->25286K(368640K), 0.0287220 secs] 619533K->317140K(5201920K), >>>>>>>>>>> 0.0291610 secs] [Times: user=0.13 sys=0.00, real=0.03 secs] >>>>>>>>>>> 2011-12-30T22:43:10.516+0100: 2136639.418: [GC 2136639.418: [ParNew: >>>>>>>>>>> 352966K->26737K(368640K), 0.0586400 secs] 644820K->338758K(5201920K), >>>>>>>>>>> 0.0589640 secs] [Times: user=0.31 sys=0.00, real=0.06 secs] >>>>>>>>>>> ------- >>>>>>>>>>> >>>>>>>>>>> In this case I don't know how to interpret the output. >>>>>>>>>>> a) There's a promotion failure that took 7.49 secs >>>>>>>>>> This is the time it took to attempt the minor collection (ParNew) and >>>>>>>>>> to >>>>>>>>>> do recovery >>>>>>>>>> from the failure. >>>>>>>>>> >>>>>>>>>>> b) There's a full GC that took 14.08 secs >>>>>>>>>>> c) There's a concurrent mode failure that took 10.29 secs >>>>>>>>>> Not sure about b) and c) because the output is mixed up with the >>>>>>>>>> concurrent-sweep >>>>>>>>>> output but ?I think the "concurrent mode failure" message is part of >>>>>>>>>> the >>>>>>>>>> "Full GC" >>>>>>>>>> message. ?My guess is that the 10.29 is the time for the Full GC and >>>>>>>>>> the >>>>>>>>>> 14.08 >>>>>>>>>> maybe is part of the concurrent-sweep message. ?Really hard to be sure. >>>>>>>>>> >>>>>>>>>> Jon >>>>>>>>>>> How are these events, and their (real) times related to each other? >>>>>>>>>>> >>>>>>>>>>> Thanks in advance, >>>>>>>>>>> Taras >>>>>>>>>>> >>>>>>>>>>> On Tue, Dec 27, 2011 at 6:13 PM, Jon >>>>>>>>>>> Masamitsu ? ? ? ? ?wrote: >>>>>>>>>>>> Taras, >>>>>>>>>>>> >>>>>>>>>>>> PrintPromotionFailure seems like it would go a long >>>>>>>>>>>> way to identify the root of your promotion failures (or >>>>>>>>>>>> at least eliminating some possible causes). ? ?I think it >>>>>>>>>>>> would help focus the discussion if you could send >>>>>>>>>>>> the result of that experiment early. >>>>>>>>>>>> >>>>>>>>>>>> Jon >>>>>>>>>>>> >>>>>>>>>>>> On 12/27/2011 5:07 AM, Taras Tielkes wrote: >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> We're running an application with the CMS/ParNew collectors that is >>>>>>>>>>>>> experiencing occasional promotion failures. >>>>>>>>>>>>> Environment is Linux 2.6.18 (x64), JVM is 1.6.0_29 in server mode. >>>>>>>>>>>>> I've listed the specific JVM options used below (a). >>>>>>>>>>>>> >>>>>>>>>>>>> The application is deployed across a handful of machines, and the >>>>>>>>>>>>> promotion failures are fairly uniform across those. >>>>>>>>>>>>> >>>>>>>>>>>>> The first kind of failure we observe is a promotion failure during >>>>>>>>>>>>> ParNew collection, I've included a snipped from the gc log below >>>>>>>>>>>>> (b). >>>>>>>>>>>>> The second kind of failure is a concurrrent mode failure (perhaps >>>>>>>>>>>>> triggered by the same cause), see (c) below. >>>>>>>>>>>>> The frequency (after running for a some weeks) is approximately once >>>>>>>>>>>>> per day. This is bearable, but obviously we'd like to improve on >>>>>>>>>>>>> this. >>>>>>>>>>>>> >>>>>>>>>>>>> Apart from high-volume request handling (which allocates a lot of >>>>>>>>>>>>> small objects), the application also runs a few dozen background >>>>>>>>>>>>> threads that download and process XML documents, typically in the >>>>>>>>>>>>> 5-30 >>>>>>>>>>>>> MB range. >>>>>>>>>>>>> A known deficiency in the existing code is that the XML content is >>>>>>>>>>>>> copied twice before processing (once to a byte[], and later again to >>>>>>>>>>>>> a >>>>>>>>>>>>> String/char[]). >>>>>>>>>>>>> Given that a 30 MB XML stream will result in a 60 MB >>>>>>>>>>>>> java.lang.String/char[], my suspicion is that these big array >>>>>>>>>>>>> allocations are causing us to run into the CMS fragmentation issue. >>>>>>>>>>>>> >>>>>>>>>>>>> My questions are: >>>>>>>>>>>>> 1) Does the data from the GC logs provide sufficient evidence to >>>>>>>>>>>>> conclude that CMS fragmentation is the cause of the promotion >>>>>>>>>>>>> failure? >>>>>>>>>>>>> 2) If not, what's the next step of investigating the cause? >>>>>>>>>>>>> 3) We're planning to at least add -XX:+PrintPromotionFailure to get >>>>>>>>>>>>> a >>>>>>>>>>>>> feeling for the size of the objects that fail promotion. >>>>>>>>>>>>> Overall, it seem that -XX:PrintFLSStatistics=1 is actually the only >>>>>>>>>>>>> reliable approach to diagnose CMS fragmentation. Is this indeed the >>>>>>>>>>>>> case? >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks in advance, >>>>>>>>>>>>> Taras >>>>>>>>>>>>> >>>>>>>>>>>>> a) Current JVM options: >>>>>>>>>>>>> -------------------------------- >>>>>>>>>>>>> -server >>>>>>>>>>>>> -Xms5g >>>>>>>>>>>>> -Xmx5g >>>>>>>>>>>>> -Xmn400m >>>>>>>>>>>>> -XX:PermSize=256m >>>>>>>>>>>>> -XX:MaxPermSize=256m >>>>>>>>>>>>> -XX:+PrintGCTimeStamps >>>>>>>>>>>>> -verbose:gc >>>>>>>>>>>>> -XX:+PrintGCDateStamps >>>>>>>>>>>>> -XX:+PrintGCDetails >>>>>>>>>>>>> -XX:SurvivorRatio=8 >>>>>>>>>>>>> -XX:+UseConcMarkSweepGC >>>>>>>>>>>>> -XX:+UseParNewGC >>>>>>>>>>>>> -XX:+DisableExplicitGC >>>>>>>>>>>>> -XX:+UseCMSInitiatingOccupancyOnly >>>>>>>>>>>>> -XX:+CMSClassUnloadingEnabled >>>>>>>>>>>>> -XX:+CMSScavengeBeforeRemark >>>>>>>>>>>>> -XX:CMSInitiatingOccupancyFraction=68 >>>>>>>>>>>>> -Xloggc:gc.log >>>>>>>>>>>>> -------------------------------- >>>>>>>>>>>>> >>>>>>>>>>>>> b) Promotion failure during ParNew >>>>>>>>>>>>> -------------------------------- >>>>>>>>>>>>> 2011-12-08T18:14:40.966+0100: 219729.868: [GC 219729.868: [ParNew: >>>>>>>>>>>>> 368640K->40959K(368640K), 0.0693460 secs] >>>>>>>>>>>>> 3504917K->3195098K(5201920K), 0.0696500 secs] [Times: user=0.39 >>>>>>>>>>>>> sys=0.01, real=0.07 secs] >>>>>>>>>>>>> 2011-12-08T18:14:43.778+0100: 219732.679: [GC 219732.679: [ParNew: >>>>>>>>>>>>> 368639K->31321K(368640K), 0.0511400 secs] >>>>>>>>>>>>> 3522778K->3198316K(5201920K), 0.0514420 secs] [Times: user=0.28 >>>>>>>>>>>>> sys=0.00, real=0.05 secs] >>>>>>>>>>>>> 2011-12-08T18:14:46.945+0100: 219735.846: [GC 219735.846: [ParNew: >>>>>>>>>>>>> 359001K->18694K(368640K), 0.0272970 secs] >>>>>>>>>>>>> 3525996K->3185690K(5201920K), 0.0276080 secs] [Times: user=0.19 >>>>>>>>>>>>> sys=0.00, real=0.03 secs] >>>>>>>>>>>>> 2011-12-08T18:14:49.036+0100: 219737.938: [GC 219737.938: [ParNew >>>>>>>>>>>>> (promotion failed): 338813K->361078K(368640K), 0.1321200 >>>>>>>>>>>>> secs]219738.070: [CMS: 3167747K->434291K(4833280K), 4.8881570 secs] >>>>>>>>>>>>> 3505808K->434291K >>>>>>>>>>>>> (5201920K), [CMS Perm : 116893K->116883K(262144K)], 5.0206620 secs] >>>>>>>>>>>>> [Times: user=5.24 sys=0.00, real=5.02 secs] >>>>>>>>>>>>> 2011-12-08T18:14:54.721+0100: 219743.622: [GC 219743.623: [ParNew: >>>>>>>>>>>>> 327680K->40960K(368640K), 0.0949460 secs] >>>>>>>>>>>>> 761971K->514584K(5201920K), >>>>>>>>>>>>> 0.0952820 secs] [Times: user=0.52 sys=0.04, real=0.10 secs] >>>>>>>>>>>>> 2011-12-08T18:14:55.580+0100: 219744.481: [GC 219744.482: [ParNew: >>>>>>>>>>>>> 368640K->40960K(368640K), 0.1299190 secs] >>>>>>>>>>>>> 842264K->625681K(5201920K), >>>>>>>>>>>>> 0.1302190 secs] [Times: user=0.72 sys=0.01, real=0.13 secs] >>>>>>>>>>>>> 2011-12-08T18:14:58.050+0100: 219746.952: [GC 219746.952: [ParNew: >>>>>>>>>>>>> 368640K->40960K(368640K), 0.0870940 secs] >>>>>>>>>>>>> 953361K->684121K(5201920K), >>>>>>>>>>>>> 0.0874110 secs] [Times: user=0.48 sys=0.01, real=0.09 secs] >>>>>>>>>>>>> -------------------------------- >>>>>>>>>>>>> >>>>>>>>>>>>> c) Promotion failure during CMS >>>>>>>>>>>>> -------------------------------- >>>>>>>>>>>>> 2011-12-14T08:29:26.628+0100: 703015.530: [GC 703015.530: [ParNew: >>>>>>>>>>>>> 357228K->40960K(368640K), 0.0525110 secs] >>>>>>>>>>>>> 3603068K->3312743K(5201920K), 0.0528120 secs] [Times: user=0.37 >>>>>>>>>>>>> sys=0.00, real=0.05 secs] >>>>>>>>>>>>> 2011-12-14T08:29:28.864+0100: 703017.766: [GC 703017.766: [ParNew: >>>>>>>>>>>>> 366075K->37119K(368640K), 0.0479780 secs] >>>>>>>>>>>>> 3637859K->3317662K(5201920K), 0.0483090 secs] [Times: user=0.24 >>>>>>>>>>>>> sys=0.01, real=0.05 secs] >>>>>>>>>>>>> 2011-12-14T08:29:29.553+0100: 703018.454: [GC 703018.455: [ParNew: >>>>>>>>>>>>> 364792K->40960K(368640K), 0.0421740 secs] >>>>>>>>>>>>> 3645334K->3334944K(5201920K), 0.0424810 secs] [Times: user=0.30 >>>>>>>>>>>>> sys=0.00, real=0.04 secs] >>>>>>>>>>>>> 2011-12-14T08:29:29.600+0100: 703018.502: [GC [1 CMS-initial-mark: >>>>>>>>>>>>> 3293984K(4833280K)] 3335025K(5201920K), 0.0272490 secs] [Times: >>>>>>>>>>>>> user=0.02 sys=0.00, real=0.03 secs] >>>>>>>>>>>>> 2011-12-14T08:29:29.628+0100: 703018.529: >>>>>>>>>>>>> [CMS-concurrent-mark-start] >>>>>>>>>>>>> 2011-12-14T08:29:30.718+0100: 703019.620: [GC 703019.620: [ParNew: >>>>>>>>>>>>> 368640K->40960K(368640K), 0.0836690 secs] >>>>>>>>>>>>> 3662624K->3386039K(5201920K), 0.0839690 secs] [Times: user=0.50 >>>>>>>>>>>>> sys=0.01, real=0.08 secs] >>>>>>>>>>>>> 2011-12-14T08:29:30.827+0100: 703019.729: [CMS-concurrent-mark: >>>>>>>>>>>>> 1.108/1.200 secs] [Times: user=6.83 sys=0.23, real=1.20 secs] >>>>>>>>>>>>> 2011-12-14T08:29:30.827+0100: 703019.729: >>>>>>>>>>>>> [CMS-concurrent-preclean-start] >>>>>>>>>>>>> 2011-12-14T08:29:30.938+0100: 703019.840: [CMS-concurrent-preclean: >>>>>>>>>>>>> 0.093/0.111 secs] [Times: user=0.48 sys=0.02, real=0.11 secs] >>>>>>>>>>>>> 2011-12-14T08:29:30.938+0100: 703019.840: >>>>>>>>>>>>> [CMS-concurrent-abortable-preclean-start] >>>>>>>>>>>>> 2011-12-14T08:29:32.337+0100: 703021.239: >>>>>>>>>>>>> [CMS-concurrent-abortable-preclean: 1.383/1.399 secs] [Times: >>>>>>>>>>>>> user=6.68 sys=0.27, real=1.40 secs] >>>>>>>>>>>>> 2011-12-14T08:29:32.343+0100: 703021.244: [GC[YG occupancy: 347750 K >>>>>>>>>>>>> (368640 K)]2011-12-14T08:29:32.343+0100: 703021.244: [GC 703021.244: >>>>>>>>>>>>> [ParNew (promotion failed): 347750K->347750K(368640K), 9.8729020 >>>>>>>>>>>>> secs] >>>>>>>>>>>>> ? ? ? 3692829K->3718580K(5201920K), 9.8732380 secs] [Times: user=12.00 >>>>>>>>>>>>> sys=2.58, real=9.88 secs] >>>>>>>>>>>>> 703031.118: [Rescan (parallel) , 0.2826110 secs]703031.400: [weak >>>>>>>>>>>>> refs >>>>>>>>>>>>> processing, 0.0014780 secs]703031.402: [class unloading, 0.0176610 >>>>>>>>>>>>> secs]703031.419: [scrub symbol& ? ? ? ? ? ?string tables, 0.0094960 >>>>>>>>>>>>> secs] [1 CMS >>>>>>>>>>>>> -remark: 3370830K(4833280K)] 3718580K(5201920K), 10.1916910 secs] >>>>>>>>>>>>> [Times: user=13.73 sys=2.59, real=10.19 secs] >>>>>>>>>>>>> 2011-12-14T08:29:42.535+0100: 703031.436: >>>>>>>>>>>>> [CMS-concurrent-sweep-start] >>>>>>>>>>>>> 2011-12-14T08:29:42.591+0100: 703031.493: [Full GC 703031.493: >>>>>>>>>>>>> [CMS2011-12-14T08:29:48.616+0100: 703037.518: [CMS-concurrent-sweep: >>>>>>>>>>>>> 6.046/6.082 secs] [Times: user=6.18 sys=0.01, real=6.09 secs] >>>>>>>>>>>>> ? ? ? (concurrent mode failure): 3370829K->433437K(4833280K), >>>>>>>>>>>>> 10.9594300 >>>>>>>>>>>>> secs] 3739469K->433437K(5201920K), [CMS Perm : >>>>>>>>>>>>> 121702K->121690K(262144K)], 10.9597540 secs] [Times: user=10.95 >>>>>>>>>>>>> sys=0.00, real=10.96 secs] >>>>>>>>>>>>> 2011-12-14T08:29:53.997+0100: 703042.899: [GC 703042.899: [ParNew: >>>>>>>>>>>>> 327680K->40960K(368640K), 0.0799960 secs] >>>>>>>>>>>>> 761117K->517836K(5201920K), >>>>>>>>>>>>> 0.0804100 secs] [Times: user=0.46 sys=0.00, real=0.08 secs] >>>>>>>>>>>>> 2011-12-14T08:29:54.649+0100: 703043.551: [GC 703043.551: [ParNew: >>>>>>>>>>>>> 368640K->40960K(368640K), 0.0784460 secs] >>>>>>>>>>>>> 845516K->557872K(5201920K), >>>>>>>>>>>>> 0.0787920 secs] [Times: user=0.40 sys=0.01, real=0.08 secs] >>>>>>>>>>>>> 2011-12-14T08:29:56.418+0100: 703045.320: [GC 703045.320: [ParNew: >>>>>>>>>>>>> 368640K->40960K(368640K), 0.0784040 secs] >>>>>>>>>>>>> 885552K->603017K(5201920K), >>>>>>>>>>>>> 0.0787630 secs] [Times: user=0.41 sys=0.01, real=0.07 secs] >>>>>>>>>>>>> -------------------------------- >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> hotspot-gc-use mailing list >>>>>>>>>>>>> hotspot-gc-use at openjdk.java.net >>>>>>>>>>>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> hotspot-gc-use mailing list >>>>>>>>>>>> hotspot-gc-use at openjdk.java.net >>>>>>>>>>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> hotspot-gc-use mailing list >>>>>>>>>>> hotspot-gc-use at openjdk.java.net >>>>>>>>>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>>>>>>>>> _______________________________________________ >>>>>>>>>> hotspot-gc-use mailing list >>>>>>>>>> hotspot-gc-use at openjdk.java.net >>>>>>>>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>>>>>>>> _______________________________________________ >>>>>>>>> hotspot-gc-use mailing list >>>>>>>>> hotspot-gc-use at openjdk.java.net >>>>>>>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>>>>>>> _______________________________________________ >>>>>>>> hotspot-gc-use mailing list >>>>>>>> hotspot-gc-use at openjdk.java.net >>>>>>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>>>> _______________________________________________ >>>>> hotspot-gc-use mailing list >>>>> hotspot-gc-use at openjdk.java.net >>>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>>> _______________________________________________ >>>> hotspot-gc-use mailing list >>>> hotspot-gc-use at openjdk.java.net >>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>> _______________________________________________ >>> hotspot-gc-use mailing list >>> hotspot-gc-use at openjdk.java.net >>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From pravah.wordpress at gmail.com Thu Feb 2 09:30:10 2012 From: pravah.wordpress at gmail.com (pravah wordpress) Date: Thu, 2 Feb 2012 23:00:10 +0530 Subject: Problem with Parnew GC Message-ID: Hi Team, For last couple of weeks, while monitoring production logs, we are seeing following 2 scenarios- 2012-01-30T09:45:57.783+0530: 211563.940: [GC 211563.941: [ParNew: 1695683K->20039K(1887488K), 0.1443767 secs] 2704741K->1029575K(5033216K), 0.1456574 secs] [Times: user=0.42 sys=0.02, real=0.15 secs] 2012-01-30T09:46:35.195+0530: 211601.351: [GC 211601.352: [ParNew: 1697863K->21131K(1887488K), 0.1623961 secs] 2707399K->1031401K(5033216K), 0.1636535 secs] [Times: user=0.41 sys=0.09, real=0.16 secs] 2012-01-30T09:47:15.555+0530: 211641.711: [GC 211641.712: [ParNew: 1698955K->19118K(1887488K), 0.1422631 secs] 2709225K->1030329K(5033216K), 0.1435373 secs] [Times: user=0.42 sys=0.01, real=0.14 secs] 2012-01-30T09:47:55.656+0530: 211681.811: [GC 211681.812: [ParNew: 1696942K->21437K(1887488K), 0.1340463 secs] 2708153K->1033449K(5033216K), 0.1352694 secs] [Times: user=0.40 sys=0.01, real=0.14 secs] 2012-01-30T09:48:29.406+0530: 211715.562: [GC 211715.564: [ParNew: 1699261K->18356K(1887488K), 0.1362798 secs] 2711273K->1031807K(5033216K), 0.1387961 secs] [Times: user=0.40 sys=0.03, real=0.14 secs] 2012-01-30T09:48:57.017+0530: 211743.172: [GC 211743.173: [ParNew: 1696035K->22375K(1887488K), 12.3948960 secs] 2709487K->1036955K(5033216K), 12.3962135 secs] [Times: *user=15.73 sys=0.60, real=12.40 secs] *2012-01-30T09:49:41.237+0530: 211787.392: [GC 211787.392: [ParNew: 1698873K->21698K(1887488K), 0.1251361 secs] 2713453K->1037007K(5033216K), 0.1262755 secs] [Times: user=0.39 sys=0.00, real=0.13 secs] 2012-01-30T09:50:24.348+0530: 211830.503: [GC 211830.504: [ParNew: 1699522K->21849K(1887488K), 0.1370179 secs] 2714831K->1037980K(5033216K), 0.1382479 secs] [Times: user=0.40 sys=0.01, real=0.14 secs] 2012-01-30T09:50:58.516+0530: 211864.670: [GC 211864.671: [ParNew: 1699673K->25281K(1887488K), 0.1334443 secs] 2715804K->1042397K(5033216K), 0.1347060 secs] [Times: user=0.43 sys=0.00, real=0.14 secs] 2012-01-30T09:51:33.830+0530: 211899.985: [GC 211899.985: [ParNew: 1695938K->28123K(1887488K), 0.1449732 secs] 2713054K->1046170K(5033216K), 0.1462090 secs] [Times: user=0.44 sys=0.01, real=0.15 secs] This is happening atleast once in a day and the real time is around 10 S. Another observation is as follows 2012-01-30T14:53:11.645+0530: 229997.686: [GC 229997.687: [ParNew: 1686605K->39956K(1887488K), 0.1504394 secs] 3258191K->1612228K(5033216K), 0.1516840 secs] [Times: user=0.47 sys=0.02, real=0.15 secs] 2012-01-30T14:53:45.006+0530: 230031.047: [GC 230031.048: [ParNew: 1717780K->30950K(1887488K), 0.1693496 secs] 3290052K->1604025K(5033216K), 0.1705586 secs] [Times: user=0.52 sys=0.04, real=0.17 secs] 2012-01-30T14:53:45.208+0530: 230031.249: [GC [1 CMS-initial-mark: 1573075K(3145728K)] 1608943K(5033216K), 0.4815794 secs] [Times: user=0.13 sys=0.35, real=0.48 secs] 2012-01-30T14:53:45.692+0530: 230031.733: [CMS-concurrent-mark-start] 2012-01-30T14:53:52.114+0530: 230038.155: [CMS-concurrent-mark: 6.283/6.422 secs] [Times: user=11.70 sys=2.40, *real=6.42 secs*] 2012-01-30T14:53:52.115+0530: 230038.155: [CMS-concurrent-preclean-start] 2012-01-30T14:53:52.470+0530: 230038.510: [CMS-concurrent-preclean: 0.346/0.355 secs] [Times: user=0.45 sys=0.16, real=0.36 secs] 2012-01-30T14:53:52.470+0530: 230038.511: [CMS-concurrent-abortable-preclean-start] CMS: abort preclean due to time 2012-01-30T14:53:57.500+0530: 230043.541: [CMS-concurrent-abortable-preclean: 4.906/5.030 secs] [Times: user=8.73 sys=0.62, *real=5.03 secs*] 2012-01-30T14:53:57.532+0530: 230043.573: [GC[YG occupancy: 839183 K (1887488 K)]230043.573: [Rescan (parallel) , 1.8934193 secs]230045.467: [weak refs processing, 0.0612851 secs] [1 CMS-remark: 1573075K(3145728K)] 2412258K(5033216K), 1.9618014 secs] [Times: user=5.58 sys=0.10, real=1.96 secs] 2012-01-30T14:53:59.497+0530: 230045.538: [CMS-concurrent-sweep-start] 2012-01-30T14:54:08.616+0530: 230054.656: [CMS-concurrent-sweep: 8.635/9.118 secs] [Times: user=17.84 sys=2.36, *real=9.12 secs*] 2012-01-30T14:54:08.617+0530: 230054.657: [CMS-concurrent-reset-start] 2012-01-30T14:54:08.656+0530: 230054.696: [CMS-concurrent-reset: 0.039/0.039 secs] [Times: user=0.05 sys=0.00, real=0.04 secs] 2012-01-30T14:54:12.410+0530: 230058.451: [GC 230058.452: [ParNew: 1708774K->25197K(1887488K), 0.1436152 secs] 2456494K->773832K(5033216K), 0.1448754 secs] [Times: user=0.48 sys=0.01, real=0.15 secs] 2012-01-30T14:54:38.243+0530: 230084.284: [GC 230084.284: [ParNew: 1703021K->62457K(1887488K), 0.2070478 secs] 2451656K->811902K(5033216K), 0.2083137 secs] [Times: user=0.67 sys=0.01, real=0.21 secs] Here CMS concurrent mark and sweep and also remark is taking considerable time many time above 15 seconds. This also happens once in a day. Why is this happening ? Also no full GC is observed. Also Parnew is happening two times per second during peak load. Is this a cause of worry / slow performance ? This is OLTP app and no batch jobs are running. Following is the configuration Java HotSpot(TM) 64-Bit Server VM (16.3-b01) for solaris-sparc JRE (1.6.0_20-b02) -XX:+DisableExplicitGC -XX:-DoEscapeAnalysis -XX:InitialHeapSize=5368709120 -XX:+LogVMOutput -XX:MaxHeapSize=5368709120 -XX:MaxNewSize=2147483648 -XX:MaxPermSize=536870912 -XX:MaxTenuringThreshold=4 -XX:NewRatio=3 -XX:NewSize=2147483648 -XX:ParallelGCThreads=4 -XX:PermSize=536870912 -XX:+PrintCommandLineFlags -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:ThreadStackSize=256 -XX:+TraceClassUnloading -XX:+UnlockDiagnosticVMOptions -XX:+UseConcMarkSweepGC -XX:+UseParNewGC Thanks in advance, Pravah. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120202/58767b2b/attachment.html From gateway at gateways-home.org Thu Feb 2 11:55:15 2012 From: gateway at gateways-home.org (Philipp Stiegler) Date: Thu, 02 Feb 2012 20:55:15 +0100 Subject: Discovering Tenured Threshold programmatically Message-ID: <4F2AEA23.3030406@gateways-home.org> Hello Guys, I tried to write my own monitoring tool to create dynamically views based on command line parameters. If someone needs the code for verification you find the source here: http://www.gateways-home.org/wb/media/jperfmon/jperfmon.zip My problem now is, that I also want to log the tenured threshold. I searched quite a while, but I didn?t find a way for getting this value from the jvm. Basically I connect to the vm via com.sun.tools.attach.VirtualMachine and I read the values out of the MBean structure (e.g. MemoryMXBean) but I don?t find this threshold in there. Maybe someone can help me out? Many thanks in advance! All the best! Philipp Stiegler From rednaxelafx at gmail.com Thu Feb 2 18:58:12 2012 From: rednaxelafx at gmail.com (Krystal Mok) Date: Fri, 3 Feb 2012 10:58:12 +0800 Subject: Discovering Tenured Threshold programmatically In-Reply-To: <4F2AEA23.3030406@gateways-home.org> References: <4F2AEA23.3030406@gateways-home.org> Message-ID: Hi Philipp, I'm not aware of any means of getting the current tenuring threshold via JMX as of now. But it's available through another API, namely the HSPERFDATA (or jvmstat) API. If that doesn't sound familiar to you, the stats you see from jstat or Visual GC actually come from this API. Try this from command line: jstat -J-Djstat.showUnsupported=true -snap | grep 'sun.gc.policy.tenuringThreshold' This will give you the current tenuring threshold, and jstat -J-Djstat.showUnsupported=true -snap | grep 'sun.gc.policy.maxTenuringThreshold' this will give you the max tenuring threshold. I've written an example of getting data from HSPERFDATA programmatically some time ago [1]. The blog is in Chinese, but the code example should give you enough hint. And there's another example of using the -snap parameter with jstat [2]. Hope this helps, - Kris [1]: http://rednaxelafx.iteye.com/blog/790864 [2]: http://rednaxelafx.iteye.com/blog/796343 On Fri, Feb 3, 2012 at 3:55 AM, Philipp Stiegler wrote: > Hello Guys, > > I tried to write my own monitoring tool to create dynamically views > based on command line parameters. > If someone needs the code for verification you find the source here: > http://www.gateways-home.org/wb/media/jperfmon/jperfmon.zip > > My problem now is, that I also want to log the tenured threshold. I > searched quite a while, but I didn?t find a way for getting this value > from the jvm. > Basically I connect to the vm via com.sun.tools.attach.VirtualMachine > and I read the values out of the MBean structure (e.g. MemoryMXBean) but > I don?t find this threshold in there. > Maybe someone can help me out? > > Many thanks in advance! > > All the best! > > Philipp Stiegler > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120203/7d112b21/attachment.html From ysr1729 at gmail.com Thu Feb 2 22:28:43 2012 From: ysr1729 at gmail.com (Srinivas Ramakrishna) Date: Thu, 2 Feb 2012 22:28:43 -0800 Subject: Problem with Parnew GC In-Reply-To: References: Message-ID: Is it possible for you to check if this happens with 1.6.0_29 as well? -- ramki On Thu, Feb 2, 2012 at 9:30 AM, pravah wordpress wrote: > Hi Team, > For last couple of weeks, while monitoring production logs, we are seeing > following 2 scenarios- > 2012-01-30T09:45:57.783+0530: 211563.940: [GC 211563.941: [ParNew: > 1695683K->20039K(1887488K), 0.1443767 secs] 2704741K->1029575K(5033216K), > 0.1456574 secs] [Times: user=0.42 sys=0.02, real=0.15 secs] > 2012-01-30T09:46:35.195+0530: 211601.351: [GC 211601.352: [ParNew: > 1697863K->21131K(1887488K), 0.1623961 secs] 2707399K->1031401K(5033216K), > 0.1636535 secs] [Times: user=0.41 sys=0.09, real=0.16 secs] > 2012-01-30T09:47:15.555+0530: 211641.711: [GC 211641.712: [ParNew: > 1698955K->19118K(1887488K), 0.1422631 secs] 2709225K->1030329K(5033216K), > 0.1435373 secs] [Times: user=0.42 sys=0.01, real=0.14 secs] > 2012-01-30T09:47:55.656+0530: 211681.811: [GC 211681.812: [ParNew: > 1696942K->21437K(1887488K), 0.1340463 secs] 2708153K->1033449K(5033216K), > 0.1352694 secs] [Times: user=0.40 sys=0.01, real=0.14 secs] > 2012-01-30T09:48:29.406+0530: 211715.562: [GC 211715.564: [ParNew: > 1699261K->18356K(1887488K), 0.1362798 secs] 2711273K->1031807K(5033216K), > 0.1387961 secs] [Times: user=0.40 sys=0.03, real=0.14 secs] > 2012-01-30T09:48:57.017+0530: 211743.172: [GC 211743.173: [ParNew: > 1696035K->22375K(1887488K), 12.3948960 secs] 2709487K->1036955K(5033216K), > 12.3962135 secs] [Times: *user=15.73 sys=0.60, real=12.40 secs] > *2012-01-30T09:49:41.237+0530: 211787.392: [GC 211787.392: [ParNew: > 1698873K->21698K(1887488K), 0.1251361 secs] 2713453K->1037007K(5033216K), > 0.1262755 secs] [Times: user=0.39 sys=0.00, real=0.13 secs] > 2012-01-30T09:50:24.348+0530: 211830.503: [GC 211830.504: [ParNew: > 1699522K->21849K(1887488K), 0.1370179 secs] 2714831K->1037980K(5033216K), > 0.1382479 secs] [Times: user=0.40 sys=0.01, real=0.14 secs] > 2012-01-30T09:50:58.516+0530: 211864.670: [GC 211864.671: [ParNew: > 1699673K->25281K(1887488K), 0.1334443 secs] 2715804K->1042397K(5033216K), > 0.1347060 secs] [Times: user=0.43 sys=0.00, real=0.14 secs] > 2012-01-30T09:51:33.830+0530: 211899.985: [GC 211899.985: [ParNew: > 1695938K->28123K(1887488K), 0.1449732 secs] 2713054K->1046170K(5033216K), > 0.1462090 secs] [Times: user=0.44 sys=0.01, real=0.15 secs] > > This is happening atleast once in a day and the real time is around 10 S. > > Another observation is as follows > > 2012-01-30T14:53:11.645+0530: 229997.686: [GC 229997.687: [ParNew: > 1686605K->39956K(1887488K), 0.1504394 secs] 3258191K->1612228K(5033216K), > 0.1516840 secs] [Times: user=0.47 sys=0.02, real=0.15 secs] > 2012-01-30T14:53:45.006+0530: 230031.047: [GC 230031.048: [ParNew: > 1717780K->30950K(1887488K), 0.1693496 secs] 3290052K->1604025K(5033216K), > 0.1705586 secs] [Times: user=0.52 sys=0.04, real=0.17 secs] > 2012-01-30T14:53:45.208+0530: 230031.249: [GC [1 CMS-initial-mark: > 1573075K(3145728K)] 1608943K(5033216K), 0.4815794 secs] [Times: user=0.13 > sys=0.35, real=0.48 secs] > 2012-01-30T14:53:45.692+0530: 230031.733: [CMS-concurrent-mark-start] > 2012-01-30T14:53:52.114+0530: 230038.155: [CMS-concurrent-mark: > 6.283/6.422 secs] [Times: user=11.70 sys=2.40, *real=6.42 secs*] > 2012-01-30T14:53:52.115+0530: 230038.155: [CMS-concurrent-preclean-start] > 2012-01-30T14:53:52.470+0530: 230038.510: [CMS-concurrent-preclean: > 0.346/0.355 secs] [Times: user=0.45 sys=0.16, real=0.36 secs] > 2012-01-30T14:53:52.470+0530: 230038.511: > [CMS-concurrent-abortable-preclean-start] > CMS: abort preclean due to time 2012-01-30T14:53:57.500+0530: 230043.541: > [CMS-concurrent-abortable-preclean: 4.906/5.030 secs] [Times: user=8.73 > sys=0.62, *real=5.03 secs*] > 2012-01-30T14:53:57.532+0530: 230043.573: [GC[YG occupancy: 839183 K > (1887488 K)]230043.573: [Rescan (parallel) , 1.8934193 secs]230045.467: > [weak refs processing, 0.0612851 secs] [1 CMS-remark: 1573075K(3145728K)] > 2412258K(5033216K), 1.9618014 secs] [Times: user=5.58 sys=0.10, real=1.96 > secs] > 2012-01-30T14:53:59.497+0530: 230045.538: [CMS-concurrent-sweep-start] > 2012-01-30T14:54:08.616+0530: 230054.656: [CMS-concurrent-sweep: > 8.635/9.118 secs] [Times: user=17.84 sys=2.36, *real=9.12 secs*] > 2012-01-30T14:54:08.617+0530: 230054.657: [CMS-concurrent-reset-start] > 2012-01-30T14:54:08.656+0530: 230054.696: [CMS-concurrent-reset: > 0.039/0.039 secs] [Times: user=0.05 sys=0.00, real=0.04 secs] > 2012-01-30T14:54:12.410+0530: 230058.451: [GC 230058.452: [ParNew: > 1708774K->25197K(1887488K), 0.1436152 secs] 2456494K->773832K(5033216K), > 0.1448754 secs] [Times: user=0.48 sys=0.01, real=0.15 secs] > 2012-01-30T14:54:38.243+0530: 230084.284: [GC 230084.284: [ParNew: > 1703021K->62457K(1887488K), 0.2070478 secs] 2451656K->811902K(5033216K), > 0.2083137 secs] [Times: user=0.67 sys=0.01, real=0.21 secs] > > > Here CMS concurrent mark and sweep and also remark is taking considerable > time many time above 15 seconds. This also happens once in a day. > Why is this happening ? > > Also no full GC is observed. Also Parnew is happening two times per second > during peak load. Is this a cause of worry / slow performance ? This is > OLTP app and no batch jobs are running. > Following is the configuration > Java HotSpot(TM) 64-Bit Server VM (16.3-b01) for solaris-sparc JRE > (1.6.0_20-b02) > > -XX:+DisableExplicitGC > -XX:-DoEscapeAnalysis > -XX:InitialHeapSize=5368709120 > -XX:+LogVMOutput > -XX:MaxHeapSize=5368709120 > -XX:MaxNewSize=2147483648 > -XX:MaxPermSize=536870912 > -XX:MaxTenuringThreshold=4 > -XX:NewRatio=3 > -XX:NewSize=2147483648 > -XX:ParallelGCThreads=4 > -XX:PermSize=536870912 > -XX:+PrintCommandLineFlags > -XX:+PrintGC > -XX:+PrintGCDateStamps > -XX:+PrintGCDetails > -XX:+PrintGCTimeStamps > -XX:ThreadStackSize=256 > -XX:+TraceClassUnloading > -XX:+UnlockDiagnosticVMOptions > -XX:+UseConcMarkSweepGC > -XX:+UseParNewGC > > Thanks in advance, > Pravah. > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120202/426562ed/attachment.html From pravah.wordpress at gmail.com Fri Feb 3 09:44:23 2012 From: pravah.wordpress at gmail.com (pravah wordpress) Date: Fri, 3 Feb 2012 23:14:23 +0530 Subject: Problem with Parnew GC In-Reply-To: References: Message-ID: Hi, It is not possible to check it with 1.6.0_29 as this is happening in production. Today we checked the iostat and sar output and found that there are no io waits and no swappings. Will allocating more memory to JVM (Total heap+ YG space ) solve this problem of slow response ? Is it normal to have 2 Parnew / second ?(and they are always taking > 0.1 Second) Thanks in advance. Pravah On Fri, Feb 3, 2012 at 11:58 AM, Srinivas Ramakrishna wrote: > Is it possible for you to check if this happens with 1.6.0_29 as well? > > -- ramki > > On Thu, Feb 2, 2012 at 9:30 AM, pravah wordpress < > pravah.wordpress at gmail.com> wrote: > >> Hi Team, >> For last couple of weeks, while monitoring production logs, we are seeing >> following 2 scenarios- >> 2012-01-30T09:45:57.783+0530: 211563.940: [GC 211563.941: [ParNew: >> 1695683K->20039K(1887488K), 0.1443767 secs] 2704741K->1029575K(5033216K), >> 0.1456574 secs] [Times: user=0.42 sys=0.02, real=0.15 secs] >> 2012-01-30T09:46:35.195+0530: 211601.351: [GC 211601.352: [ParNew: >> 1697863K->21131K(1887488K), 0.1623961 secs] 2707399K->1031401K(5033216K), >> 0.1636535 secs] [Times: user=0.41 sys=0.09, real=0.16 secs] >> 2012-01-30T09:47:15.555+0530: 211641.711: [GC 211641.712: [ParNew: >> 1698955K->19118K(1887488K), 0.1422631 secs] 2709225K->1030329K(5033216K), >> 0.1435373 secs] [Times: user=0.42 sys=0.01, real=0.14 secs] >> 2012-01-30T09:47:55.656+0530: 211681.811: [GC 211681.812: [ParNew: >> 1696942K->21437K(1887488K), 0.1340463 secs] 2708153K->1033449K(5033216K), >> 0.1352694 secs] [Times: user=0.40 sys=0.01, real=0.14 secs] >> 2012-01-30T09:48:29.406+0530: 211715.562: [GC 211715.564: [ParNew: >> 1699261K->18356K(1887488K), 0.1362798 secs] 2711273K->1031807K(5033216K), >> 0.1387961 secs] [Times: user=0.40 sys=0.03, real=0.14 secs] >> 2012-01-30T09:48:57.017+0530: 211743.172: [GC 211743.173: [ParNew: >> 1696035K->22375K(1887488K), 12.3948960 secs] 2709487K->1036955K(5033216K), >> 12.3962135 secs] [Times: *user=15.73 sys=0.60, real=12.40 secs] >> *2012-01-30T09:49:41.237+0530: 211787.392: [GC 211787.392: [ParNew: >> 1698873K->21698K(1887488K), 0.1251361 secs] 2713453K->1037007K(5033216K), >> 0.1262755 secs] [Times: user=0.39 sys=0.00, real=0.13 secs] >> 2012-01-30T09:50:24.348+0530: 211830.503: [GC 211830.504: [ParNew: >> 1699522K->21849K(1887488K), 0.1370179 secs] 2714831K->1037980K(5033216K), >> 0.1382479 secs] [Times: user=0.40 sys=0.01, real=0.14 secs] >> 2012-01-30T09:50:58.516+0530: 211864.670: [GC 211864.671: [ParNew: >> 1699673K->25281K(1887488K), 0.1334443 secs] 2715804K->1042397K(5033216K), >> 0.1347060 secs] [Times: user=0.43 sys=0.00, real=0.14 secs] >> 2012-01-30T09:51:33.830+0530: 211899.985: [GC 211899.985: [ParNew: >> 1695938K->28123K(1887488K), 0.1449732 secs] 2713054K->1046170K(5033216K), >> 0.1462090 secs] [Times: user=0.44 sys=0.01, real=0.15 secs] >> >> This is happening atleast once in a day and the real time is around 10 S. >> >> Another observation is as follows >> >> 2012-01-30T14:53:11.645+0530: 229997.686: [GC 229997.687: [ParNew: >> 1686605K->39956K(1887488K), 0.1504394 secs] 3258191K->1612228K(5033216K), >> 0.1516840 secs] [Times: user=0.47 sys=0.02, real=0.15 secs] >> 2012-01-30T14:53:45.006+0530: 230031.047: [GC 230031.048: [ParNew: >> 1717780K->30950K(1887488K), 0.1693496 secs] 3290052K->1604025K(5033216K), >> 0.1705586 secs] [Times: user=0.52 sys=0.04, real=0.17 secs] >> 2012-01-30T14:53:45.208+0530: 230031.249: [GC [1 CMS-initial-mark: >> 1573075K(3145728K)] 1608943K(5033216K), 0.4815794 secs] [Times: user=0.13 >> sys=0.35, real=0.48 secs] >> 2012-01-30T14:53:45.692+0530: 230031.733: [CMS-concurrent-mark-start] >> 2012-01-30T14:53:52.114+0530: 230038.155: [CMS-concurrent-mark: >> 6.283/6.422 secs] [Times: user=11.70 sys=2.40, *real=6.42 secs*] >> 2012-01-30T14:53:52.115+0530: 230038.155: [CMS-concurrent-preclean-start] >> 2012-01-30T14:53:52.470+0530: 230038.510: [CMS-concurrent-preclean: >> 0.346/0.355 secs] [Times: user=0.45 sys=0.16, real=0.36 secs] >> 2012-01-30T14:53:52.470+0530: 230038.511: >> [CMS-concurrent-abortable-preclean-start] >> CMS: abort preclean due to time 2012-01-30T14:53:57.500+0530: >> 230043.541: [CMS-concurrent-abortable-preclean: 4.906/5.030 secs] [Times: >> user=8.73 sys=0.62, *real=5.03 secs*] >> 2012-01-30T14:53:57.532+0530: 230043.573: [GC[YG occupancy: 839183 K >> (1887488 K)]230043.573: [Rescan (parallel) , 1.8934193 secs]230045.467: >> [weak refs processing, 0.0612851 secs] [1 CMS-remark: 1573075K(3145728K)] >> 2412258K(5033216K), 1.9618014 secs] [Times: user=5.58 sys=0.10, real=1.96 >> secs] >> 2012-01-30T14:53:59.497+0530: 230045.538: [CMS-concurrent-sweep-start] >> 2012-01-30T14:54:08.616+0530: 230054.656: [CMS-concurrent-sweep: >> 8.635/9.118 secs] [Times: user=17.84 sys=2.36, *real=9.12 secs*] >> 2012-01-30T14:54:08.617+0530: 230054.657: [CMS-concurrent-reset-start] >> 2012-01-30T14:54:08.656+0530: 230054.696: [CMS-concurrent-reset: >> 0.039/0.039 secs] [Times: user=0.05 sys=0.00, real=0.04 secs] >> 2012-01-30T14:54:12.410+0530: 230058.451: [GC 230058.452: [ParNew: >> 1708774K->25197K(1887488K), 0.1436152 secs] 2456494K->773832K(5033216K), >> 0.1448754 secs] [Times: user=0.48 sys=0.01, real=0.15 secs] >> 2012-01-30T14:54:38.243+0530: 230084.284: [GC 230084.284: [ParNew: >> 1703021K->62457K(1887488K), 0.2070478 secs] 2451656K->811902K(5033216K), >> 0.2083137 secs] [Times: user=0.67 sys=0.01, real=0.21 secs] >> >> >> Here CMS concurrent mark and sweep and also remark is taking considerable >> time many time above 15 seconds. This also happens once in a day. >> Why is this happening ? >> >> Also no full GC is observed. Also Parnew is happening two times per >> second during peak load. Is this a cause of worry / slow performance ? This >> is OLTP app and no batch jobs are running. >> Following is the configuration >> Java HotSpot(TM) 64-Bit Server VM (16.3-b01) for solaris-sparc JRE >> (1.6.0_20-b02) >> >> -XX:+DisableExplicitGC >> -XX:-DoEscapeAnalysis >> -XX:InitialHeapSize=5368709120 >> -XX:+LogVMOutput >> -XX:MaxHeapSize=5368709120 >> -XX:MaxNewSize=2147483648 >> -XX:MaxPermSize=536870912 >> -XX:MaxTenuringThreshold=4 >> -XX:NewRatio=3 >> -XX:NewSize=2147483648 >> -XX:ParallelGCThreads=4 >> -XX:PermSize=536870912 >> -XX:+PrintCommandLineFlags >> -XX:+PrintGC >> -XX:+PrintGCDateStamps >> -XX:+PrintGCDetails >> -XX:+PrintGCTimeStamps >> -XX:ThreadStackSize=256 >> -XX:+TraceClassUnloading >> -XX:+UnlockDiagnosticVMOptions >> -XX:+UseConcMarkSweepGC >> -XX:+UseParNewGC >> >> Thanks in advance, >> Pravah. >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120203/ffa95cb1/attachment-0001.html From ysr1729 at gmail.com Fri Feb 3 10:25:57 2012 From: ysr1729 at gmail.com (Srinivas Ramakrishna) Date: Fri, 3 Feb 2012 10:25:57 -0800 Subject: Problem with Parnew GC In-Reply-To: References: Message-ID: Hi Pravah -- There was a bug fix for taking care of a fragmentation problem a while back.... http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6999988 It may or may not be your problem (after all you do not get promotion failures), but it appears to me that it can result in very poor performance of the "local free list caching" and i can see it resulting in increased promotion times. I can't off the top of my head tell if there's a quick way of making that identification from your 6u20 JVM but perhaps someone may have an idea. As to whether allocating more memory to the JVM will help... If my conjecture as to the root cause is correct (and i will be the first to admit it might not be), then, no: allocating more heap will not help because the bug prevents the efficient use of existing memory rather than that existing memory is quickly used up; so the extra heap will not help. But as you can see I am doing a lot of hand-waving here, so you should probably follow up with Sun/Oracle JVM support. -- ramki On Fri, Feb 3, 2012 at 9:44 AM, pravah wordpress wrote: > Hi, > It is not possible to check it with 1.6.0_29 as this is happening in > production. > Today we checked the iostat and sar output and found that there are no io > waits and no swappings. Will allocating more memory to JVM (Total heap+ YG > space ) solve this problem of slow response ? Is it normal to have 2 Parnew > / second ?(and they are always taking > 0.1 Second) > > Thanks in advance. > Pravah > > > > On Fri, Feb 3, 2012 at 11:58 AM, Srinivas Ramakrishna wrote: > >> Is it possible for you to check if this happens with 1.6.0_29 as well? >> >> -- ramki >> >> On Thu, Feb 2, 2012 at 9:30 AM, pravah wordpress < >> pravah.wordpress at gmail.com> wrote: >> >>> Hi Team, >>> For last couple of weeks, while monitoring production logs, we are >>> seeing following 2 scenarios- >>> 2012-01-30T09:45:57.783+0530: 211563.940: [GC 211563.941: [ParNew: >>> 1695683K->20039K(1887488K), 0.1443767 secs] 2704741K->1029575K(5033216K), >>> 0.1456574 secs] [Times: user=0.42 sys=0.02, real=0.15 secs] >>> 2012-01-30T09:46:35.195+0530: 211601.351: [GC 211601.352: [ParNew: >>> 1697863K->21131K(1887488K), 0.1623961 secs] 2707399K->1031401K(5033216K), >>> 0.1636535 secs] [Times: user=0.41 sys=0.09, real=0.16 secs] >>> 2012-01-30T09:47:15.555+0530: 211641.711: [GC 211641.712: [ParNew: >>> 1698955K->19118K(1887488K), 0.1422631 secs] 2709225K->1030329K(5033216K), >>> 0.1435373 secs] [Times: user=0.42 sys=0.01, real=0.14 secs] >>> 2012-01-30T09:47:55.656+0530: 211681.811: [GC 211681.812: [ParNew: >>> 1696942K->21437K(1887488K), 0.1340463 secs] 2708153K->1033449K(5033216K), >>> 0.1352694 secs] [Times: user=0.40 sys=0.01, real=0.14 secs] >>> 2012-01-30T09:48:29.406+0530: 211715.562: [GC 211715.564: [ParNew: >>> 1699261K->18356K(1887488K), 0.1362798 secs] 2711273K->1031807K(5033216K), >>> 0.1387961 secs] [Times: user=0.40 sys=0.03, real=0.14 secs] >>> 2012-01-30T09:48:57.017+0530: 211743.172: [GC 211743.173: [ParNew: >>> 1696035K->22375K(1887488K), 12.3948960 secs] 2709487K->1036955K(5033216K), >>> 12.3962135 secs] [Times: *user=15.73 sys=0.60, real=12.40 secs] >>> *2012-01-30T09:49:41.237+0530: 211787.392: [GC 211787.392: [ParNew: >>> 1698873K->21698K(1887488K), 0.1251361 secs] 2713453K->1037007K(5033216K), >>> 0.1262755 secs] [Times: user=0.39 sys=0.00, real=0.13 secs] >>> 2012-01-30T09:50:24.348+0530: 211830.503: [GC 211830.504: [ParNew: >>> 1699522K->21849K(1887488K), 0.1370179 secs] 2714831K->1037980K(5033216K), >>> 0.1382479 secs] [Times: user=0.40 sys=0.01, real=0.14 secs] >>> 2012-01-30T09:50:58.516+0530: 211864.670: [GC 211864.671: [ParNew: >>> 1699673K->25281K(1887488K), 0.1334443 secs] 2715804K->1042397K(5033216K), >>> 0.1347060 secs] [Times: user=0.43 sys=0.00, real=0.14 secs] >>> 2012-01-30T09:51:33.830+0530: 211899.985: [GC 211899.985: [ParNew: >>> 1695938K->28123K(1887488K), 0.1449732 secs] 2713054K->1046170K(5033216K), >>> 0.1462090 secs] [Times: user=0.44 sys=0.01, real=0.15 secs] >>> >>> This is happening atleast once in a day and the real time is around 10 S. >>> >>> Another observation is as follows >>> >>> 2012-01-30T14:53:11.645+0530: 229997.686: [GC 229997.687: [ParNew: >>> 1686605K->39956K(1887488K), 0.1504394 secs] 3258191K->1612228K(5033216K), >>> 0.1516840 secs] [Times: user=0.47 sys=0.02, real=0.15 secs] >>> 2012-01-30T14:53:45.006+0530: 230031.047: [GC 230031.048: [ParNew: >>> 1717780K->30950K(1887488K), 0.1693496 secs] 3290052K->1604025K(5033216K), >>> 0.1705586 secs] [Times: user=0.52 sys=0.04, real=0.17 secs] >>> 2012-01-30T14:53:45.208+0530: 230031.249: [GC [1 CMS-initial-mark: >>> 1573075K(3145728K)] 1608943K(5033216K), 0.4815794 secs] [Times: user=0.13 >>> sys=0.35, real=0.48 secs] >>> 2012-01-30T14:53:45.692+0530: 230031.733: [CMS-concurrent-mark-start] >>> 2012-01-30T14:53:52.114+0530: 230038.155: [CMS-concurrent-mark: >>> 6.283/6.422 secs] [Times: user=11.70 sys=2.40, *real=6.42 secs*] >>> 2012-01-30T14:53:52.115+0530: 230038.155: [CMS-concurrent-preclean-start] >>> 2012-01-30T14:53:52.470+0530: 230038.510: [CMS-concurrent-preclean: >>> 0.346/0.355 secs] [Times: user=0.45 sys=0.16, real=0.36 secs] >>> 2012-01-30T14:53:52.470+0530: 230038.511: >>> [CMS-concurrent-abortable-preclean-start] >>> CMS: abort preclean due to time 2012-01-30T14:53:57.500+0530: >>> 230043.541: [CMS-concurrent-abortable-preclean: 4.906/5.030 secs] [Times: >>> user=8.73 sys=0.62, *real=5.03 secs*] >>> 2012-01-30T14:53:57.532+0530: 230043.573: [GC[YG occupancy: 839183 K >>> (1887488 K)]230043.573: [Rescan (parallel) , 1.8934193 secs]230045.467: >>> [weak refs processing, 0.0612851 secs] [1 CMS-remark: 1573075K(3145728K)] >>> 2412258K(5033216K), 1.9618014 secs] [Times: user=5.58 sys=0.10, real=1.96 >>> secs] >>> 2012-01-30T14:53:59.497+0530: 230045.538: [CMS-concurrent-sweep-start] >>> 2012-01-30T14:54:08.616+0530: 230054.656: [CMS-concurrent-sweep: >>> 8.635/9.118 secs] [Times: user=17.84 sys=2.36, *real=9.12 secs*] >>> 2012-01-30T14:54:08.617+0530: 230054.657: [CMS-concurrent-reset-start] >>> 2012-01-30T14:54:08.656+0530: 230054.696: [CMS-concurrent-reset: >>> 0.039/0.039 secs] [Times: user=0.05 sys=0.00, real=0.04 secs] >>> 2012-01-30T14:54:12.410+0530: 230058.451: [GC 230058.452: [ParNew: >>> 1708774K->25197K(1887488K), 0.1436152 secs] 2456494K->773832K(5033216K), >>> 0.1448754 secs] [Times: user=0.48 sys=0.01, real=0.15 secs] >>> 2012-01-30T14:54:38.243+0530: 230084.284: [GC 230084.284: [ParNew: >>> 1703021K->62457K(1887488K), 0.2070478 secs] 2451656K->811902K(5033216K), >>> 0.2083137 secs] [Times: user=0.67 sys=0.01, real=0.21 secs] >>> >>> >>> Here CMS concurrent mark and sweep and also remark is taking >>> considerable time many time above 15 seconds. This also happens once in a >>> day. >>> Why is this happening ? >>> >>> Also no full GC is observed. Also Parnew is happening two times per >>> second during peak load. Is this a cause of worry / slow performance ? This >>> is OLTP app and no batch jobs are running. >>> Following is the configuration >>> Java HotSpot(TM) 64-Bit Server VM (16.3-b01) for solaris-sparc JRE >>> (1.6.0_20-b02) >>> >>> -XX:+DisableExplicitGC >>> -XX:-DoEscapeAnalysis >>> -XX:InitialHeapSize=5368709120 >>> -XX:+LogVMOutput >>> -XX:MaxHeapSize=5368709120 >>> -XX:MaxNewSize=2147483648 >>> -XX:MaxPermSize=536870912 >>> -XX:MaxTenuringThreshold=4 >>> -XX:NewRatio=3 >>> -XX:NewSize=2147483648 >>> -XX:ParallelGCThreads=4 >>> -XX:PermSize=536870912 >>> -XX:+PrintCommandLineFlags >>> -XX:+PrintGC >>> -XX:+PrintGCDateStamps >>> -XX:+PrintGCDetails >>> -XX:+PrintGCTimeStamps >>> -XX:ThreadStackSize=256 >>> -XX:+TraceClassUnloading >>> -XX:+UnlockDiagnosticVMOptions >>> -XX:+UseConcMarkSweepGC >>> -XX:+UseParNewGC >>> >>> Thanks in advance, >>> Pravah. >>> >>> _______________________________________________ >>> hotspot-gc-use mailing list >>> hotspot-gc-use at openjdk.java.net >>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120203/ad852e7f/attachment.html From gateway at gateways-home.org Fri Feb 3 13:40:03 2012 From: gateway at gateways-home.org (Philipp Stiegler) Date: Fri, 03 Feb 2012 22:40:03 +0100 Subject: Discovering Tenured Threshold programmatically In-Reply-To: References: <4F2AEA23.3030406@gateways-home.org> Message-ID: <4F2C5433.2080403@gateways-home.org> Hi Kris, perfectly, thats the API I was searching for. Big load of thanks! All the best! Philipp On 2012-02-03 03:58, Krystal Mok wrote: > Hi Philipp, > > I'm not aware of any means of getting the current tenuring threshold > via JMX as of now. > > But it's available through another API, namely the HSPERFDATA (or > jvmstat) API. If that doesn't sound familiar to you, the stats you see > from jstat or Visual GC actually come from this API. > > Try this from command line: > > jstat -J-Djstat.showUnsupported=true -snap | grep > 'sun.gc.policy.tenuringThreshold' > > This will give you the current tenuring threshold, and > > jstat -J-Djstat.showUnsupported=true -snap | grep > 'sun.gc.policy.maxTenuringThreshold' > > this will give you the max tenuring threshold. > > I've written an example of getting data from HSPERFDATA > programmatically some time ago [1]. The blog is in Chinese, but the > code example should give you enough hint. And there's another example > of using the -snap parameter with jstat [2]. > > Hope this helps, > - Kris > > [1]: http://rednaxelafx.iteye.com/blog/790864 > [2]: http://rednaxelafx.iteye.com/blog/796343 > > On Fri, Feb 3, 2012 at 3:55 AM, Philipp Stiegler > > wrote: > > Hello Guys, > > I tried to write my own monitoring tool to create dynamically views > based on command line parameters. > If someone needs the code for verification you find the source here: > http://www.gateways-home.org/wb/media/jperfmon/jperfmon.zip > > My problem now is, that I also want to log the tenured threshold. I > searched quite a while, but I didn?t find a way for getting this value > from the jvm. > Basically I connect to the vm via com.sun.tools.attach.VirtualMachine > and I read the values out of the MBean structure (e.g. > MemoryMXBean) but > I don?t find this threshold in there. > Maybe someone can help me out? > > Many thanks in advance! > > All the best! > > Philipp Stiegler > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120203/5a1e377b/attachment.html From pravah.wordpress at gmail.com Sun Feb 5 08:57:26 2012 From: pravah.wordpress at gmail.com (pravah wordpress) Date: Sun, 5 Feb 2012 22:27:26 +0530 Subject: Problem with Parnew GC In-Reply-To: References: Message-ID: Hi Ramki, Thank you for your reply. I checked the given bug but it seems that it may not be the root cause as no promotion failures are happening in our case. I think we will try with different values for new memory and see if it improves the performance. I shall post the results of our experiments. Thanks again. Pravah On Fri, Feb 3, 2012 at 11:55 PM, Srinivas Ramakrishna wrote: > Hi Pravah -- > > There was a bug fix for taking care of a fragmentation problem a while > back.... > > http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6999988 > > It may or may not be your problem (after all you do not get promotion > failures), but > it appears to me that it can result in very poor performance of the "local > free list caching" > and i can see it resulting in increased promotion times. I can't off the > top of > my head tell if there's a quick way of making that identification from > your 6u20 JVM > but perhaps someone may have an idea. > > As to whether allocating more memory to the JVM will help... If my > conjecture as to the > root cause is correct (and i will be the first to admit it might not be), > then, no: allocating > more heap will not help because the bug prevents the efficient use of > existing memory > rather than that existing memory is quickly used up; so the extra heap > will not help. > > But as you can see I am doing a lot of hand-waving here, so you should > probably > follow up with Sun/Oracle JVM support. > > -- ramki > > > On Fri, Feb 3, 2012 at 9:44 AM, pravah wordpress < > pravah.wordpress at gmail.com> wrote: > >> Hi, >> It is not possible to check it with 1.6.0_29 as this is happening in >> production. >> Today we checked the iostat and sar output and found that there are no io >> waits and no swappings. Will allocating more memory to JVM (Total heap+ YG >> space ) solve this problem of slow response ? Is it normal to have 2 Parnew >> / second ?(and they are always taking > 0.1 Second) >> >> Thanks in advance. >> Pravah >> >> >> >> On Fri, Feb 3, 2012 at 11:58 AM, Srinivas Ramakrishna wrote: >> >>> Is it possible for you to check if this happens with 1.6.0_29 as well? >>> >>> -- ramki >>> >>> On Thu, Feb 2, 2012 at 9:30 AM, pravah wordpress < >>> pravah.wordpress at gmail.com> wrote: >>> >>>> Hi Team, >>>> For last couple of weeks, while monitoring production logs, we are >>>> seeing following 2 scenarios- >>>> 2012-01-30T09:45:57.783+0530: 211563.940: [GC 211563.941: [ParNew: >>>> 1695683K->20039K(1887488K), 0.1443767 secs] 2704741K->1029575K(5033216K), >>>> 0.1456574 secs] [Times: user=0.42 sys=0.02, real=0.15 secs] >>>> 2012-01-30T09:46:35.195+0530: 211601.351: [GC 211601.352: [ParNew: >>>> 1697863K->21131K(1887488K), 0.1623961 secs] 2707399K->1031401K(5033216K), >>>> 0.1636535 secs] [Times: user=0.41 sys=0.09, real=0.16 secs] >>>> 2012-01-30T09:47:15.555+0530: 211641.711: [GC 211641.712: [ParNew: >>>> 1698955K->19118K(1887488K), 0.1422631 secs] 2709225K->1030329K(5033216K), >>>> 0.1435373 secs] [Times: user=0.42 sys=0.01, real=0.14 secs] >>>> 2012-01-30T09:47:55.656+0530: 211681.811: [GC 211681.812: [ParNew: >>>> 1696942K->21437K(1887488K), 0.1340463 secs] 2708153K->1033449K(5033216K), >>>> 0.1352694 secs] [Times: user=0.40 sys=0.01, real=0.14 secs] >>>> 2012-01-30T09:48:29.406+0530: 211715.562: [GC 211715.564: [ParNew: >>>> 1699261K->18356K(1887488K), 0.1362798 secs] 2711273K->1031807K(5033216K), >>>> 0.1387961 secs] [Times: user=0.40 sys=0.03, real=0.14 secs] >>>> 2012-01-30T09:48:57.017+0530: 211743.172: [GC 211743.173: [ParNew: >>>> 1696035K->22375K(1887488K), 12.3948960 secs] 2709487K->1036955K(5033216K), >>>> 12.3962135 secs] [Times: *user=15.73 sys=0.60, real=12.40 secs] >>>> *2012-01-30T09:49:41.237+0530: 211787.392: [GC 211787.392: [ParNew: >>>> 1698873K->21698K(1887488K), 0.1251361 secs] 2713453K->1037007K(5033216K), >>>> 0.1262755 secs] [Times: user=0.39 sys=0.00, real=0.13 secs] >>>> 2012-01-30T09:50:24.348+0530: 211830.503: [GC 211830.504: [ParNew: >>>> 1699522K->21849K(1887488K), 0.1370179 secs] 2714831K->1037980K(5033216K), >>>> 0.1382479 secs] [Times: user=0.40 sys=0.01, real=0.14 secs] >>>> 2012-01-30T09:50:58.516+0530: 211864.670: [GC 211864.671: [ParNew: >>>> 1699673K->25281K(1887488K), 0.1334443 secs] 2715804K->1042397K(5033216K), >>>> 0.1347060 secs] [Times: user=0.43 sys=0.00, real=0.14 secs] >>>> 2012-01-30T09:51:33.830+0530: 211899.985: [GC 211899.985: [ParNew: >>>> 1695938K->28123K(1887488K), 0.1449732 secs] 2713054K->1046170K(5033216K), >>>> 0.1462090 secs] [Times: user=0.44 sys=0.01, real=0.15 secs] >>>> >>>> This is happening atleast once in a day and the real time is around 10 >>>> S. >>>> >>>> Another observation is as follows >>>> >>>> 2012-01-30T14:53:11.645+0530: 229997.686: [GC 229997.687: [ParNew: >>>> 1686605K->39956K(1887488K), 0.1504394 secs] 3258191K->1612228K(5033216K), >>>> 0.1516840 secs] [Times: user=0.47 sys=0.02, real=0.15 secs] >>>> 2012-01-30T14:53:45.006+0530: 230031.047: [GC 230031.048: [ParNew: >>>> 1717780K->30950K(1887488K), 0.1693496 secs] 3290052K->1604025K(5033216K), >>>> 0.1705586 secs] [Times: user=0.52 sys=0.04, real=0.17 secs] >>>> 2012-01-30T14:53:45.208+0530: 230031.249: [GC [1 CMS-initial-mark: >>>> 1573075K(3145728K)] 1608943K(5033216K), 0.4815794 secs] [Times: user=0.13 >>>> sys=0.35, real=0.48 secs] >>>> 2012-01-30T14:53:45.692+0530: 230031.733: [CMS-concurrent-mark-start] >>>> 2012-01-30T14:53:52.114+0530: 230038.155: [CMS-concurrent-mark: >>>> 6.283/6.422 secs] [Times: user=11.70 sys=2.40, *real=6.42 secs*] >>>> 2012-01-30T14:53:52.115+0530: 230038.155: >>>> [CMS-concurrent-preclean-start] >>>> 2012-01-30T14:53:52.470+0530: 230038.510: [CMS-concurrent-preclean: >>>> 0.346/0.355 secs] [Times: user=0.45 sys=0.16, real=0.36 secs] >>>> 2012-01-30T14:53:52.470+0530: 230038.511: >>>> [CMS-concurrent-abortable-preclean-start] >>>> CMS: abort preclean due to time 2012-01-30T14:53:57.500+0530: >>>> 230043.541: [CMS-concurrent-abortable-preclean: 4.906/5.030 secs] [Times: >>>> user=8.73 sys=0.62, *real=5.03 secs*] >>>> 2012-01-30T14:53:57.532+0530: 230043.573: [GC[YG occupancy: 839183 K >>>> (1887488 K)]230043.573: [Rescan (parallel) , 1.8934193 secs]230045.467: >>>> [weak refs processing, 0.0612851 secs] [1 CMS-remark: 1573075K(3145728K)] >>>> 2412258K(5033216K), 1.9618014 secs] [Times: user=5.58 sys=0.10, real=1.96 >>>> secs] >>>> 2012-01-30T14:53:59.497+0530: 230045.538: [CMS-concurrent-sweep-start] >>>> 2012-01-30T14:54:08.616+0530: 230054.656: [CMS-concurrent-sweep: >>>> 8.635/9.118 secs] [Times: user=17.84 sys=2.36, *real=9.12 secs*] >>>> 2012-01-30T14:54:08.617+0530: 230054.657: [CMS-concurrent-reset-start] >>>> 2012-01-30T14:54:08.656+0530: 230054.696: [CMS-concurrent-reset: >>>> 0.039/0.039 secs] [Times: user=0.05 sys=0.00, real=0.04 secs] >>>> 2012-01-30T14:54:12.410+0530: 230058.451: [GC 230058.452: [ParNew: >>>> 1708774K->25197K(1887488K), 0.1436152 secs] 2456494K->773832K(5033216K), >>>> 0.1448754 secs] [Times: user=0.48 sys=0.01, real=0.15 secs] >>>> 2012-01-30T14:54:38.243+0530: 230084.284: [GC 230084.284: [ParNew: >>>> 1703021K->62457K(1887488K), 0.2070478 secs] 2451656K->811902K(5033216K), >>>> 0.2083137 secs] [Times: user=0.67 sys=0.01, real=0.21 secs] >>>> >>>> >>>> Here CMS concurrent mark and sweep and also remark is taking >>>> considerable time many time above 15 seconds. This also happens once in a >>>> day. >>>> Why is this happening ? >>>> >>>> Also no full GC is observed. Also Parnew is happening two times per >>>> second during peak load. Is this a cause of worry / slow performance ? This >>>> is OLTP app and no batch jobs are running. >>>> Following is the configuration >>>> Java HotSpot(TM) 64-Bit Server VM (16.3-b01) for solaris-sparc JRE >>>> (1.6.0_20-b02) >>>> >>>> -XX:+DisableExplicitGC >>>> -XX:-DoEscapeAnalysis >>>> -XX:InitialHeapSize=5368709120 >>>> -XX:+LogVMOutput >>>> -XX:MaxHeapSize=5368709120 >>>> -XX:MaxNewSize=2147483648 >>>> -XX:MaxPermSize=536870912 >>>> -XX:MaxTenuringThreshold=4 >>>> -XX:NewRatio=3 >>>> -XX:NewSize=2147483648 >>>> -XX:ParallelGCThreads=4 >>>> -XX:PermSize=536870912 >>>> -XX:+PrintCommandLineFlags >>>> -XX:+PrintGC >>>> -XX:+PrintGCDateStamps >>>> -XX:+PrintGCDetails >>>> -XX:+PrintGCTimeStamps >>>> -XX:ThreadStackSize=256 >>>> -XX:+TraceClassUnloading >>>> -XX:+UnlockDiagnosticVMOptions >>>> -XX:+UseConcMarkSweepGC >>>> -XX:+UseParNewGC >>>> >>>> Thanks in advance, >>>> Pravah. >>>> >>>> _______________________________________________ >>>> hotspot-gc-use mailing list >>>> hotspot-gc-use at openjdk.java.net >>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120205/7f0054b7/attachment.html From pravah.wordpress at gmail.com Fri Feb 10 04:51:13 2012 From: pravah.wordpress at gmail.com (pravah wordpress) Date: Fri, 10 Feb 2012 18:21:13 +0530 Subject: Problem with Parnew GC In-Reply-To: References: Message-ID: Hi Ramki, We have done some more analysis using jstat. Following is the output of jstat. S0 S1 E O P YGC YGCT FGC FGCT GCT LGCC GCC 0.00 25.58 52.20 30.75 19.14 227 51.356 0 0.000 51.356 unknown GCCause No GC 0.00 25.58 58.94 30.75 19.14 227 51.356 0 0.000 51.356 unknown GCCause No GC 0.00 25.58 67.09 30.75 19.14 227 51.356 0 0.000 51.356 unknown GCCause No GC 0.00 25.58 80.68 30.75 19.14 227 51.356 0 0.000 51.356 unknown GCCause No GC 0.00 25.58 88.15 30.75 19.14 227 51.356 0 0.000 51.356 unknown GCCause No GC 6.61 25.58 100.00 30.75 19.14 228 51.356 0 0.000 51.356 unknown GCCause Allocation Failure 8.93 0.00 10.75 30.78 19.14 228 51.567 0 0.000 51.567 unknown GCCause No GC 8.93 0.00 18.57 30.78 19.14 228 51.567 0 0.000 51.567 unknown GCCause No GC 8.93 0.00 39.04 30.78 19.14 228 51.567 0 0.000 51.567 unknown GCCause No GC 8.93 0.00 45.56 30.78 19.14 228 51.567 0 0.000 51.567 unknown GCCause No GC As can be seen, lot of YG are happening. I have 2 options- 1-Let some objects go in survior space and YG will collect it 2-Let them flow to Old generation (by setting -XX:MaxTenuringThreshold=0 )and CMS will collect it. Which one is faster ? YG or CMS ? YG is 2G and OLD G is 5G. currently default survivor ration is used. Thanks in advance Pravah. On Sun, Feb 5, 2012 at 10:27 PM, pravah wordpress < pravah.wordpress at gmail.com> wrote: > Hi Ramki, > Thank you for your reply. I checked the given bug but it seems that it may > not be the root > cause as no promotion failures are happening in our case. > I think we will try with different values for new memory and see if it > improves the performance. > I shall post the results of our experiments. > Thanks again. > Pravah > > > > On Fri, Feb 3, 2012 at 11:55 PM, Srinivas Ramakrishna wrote: > >> Hi Pravah -- >> >> There was a bug fix for taking care of a fragmentation problem a while >> back.... >> >> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6999988 >> >> It may or may not be your problem (after all you do not get promotion >> failures), but >> it appears to me that it can result in very poor performance of the >> "local free list caching" >> and i can see it resulting in increased promotion times. I can't off the >> top of >> my head tell if there's a quick way of making that identification from >> your 6u20 JVM >> but perhaps someone may have an idea. >> >> As to whether allocating more memory to the JVM will help... If my >> conjecture as to the >> root cause is correct (and i will be the first to admit it might not be), >> then, no: allocating >> more heap will not help because the bug prevents the efficient use of >> existing memory >> rather than that existing memory is quickly used up; so the extra heap >> will not help. >> >> But as you can see I am doing a lot of hand-waving here, so you should >> probably >> follow up with Sun/Oracle JVM support. >> >> -- ramki >> >> >> On Fri, Feb 3, 2012 at 9:44 AM, pravah wordpress < >> pravah.wordpress at gmail.com> wrote: >> >>> Hi, >>> It is not possible to check it with 1.6.0_29 as this is happening in >>> production. >>> Today we checked the iostat and sar output and found that there are no >>> io waits and no swappings. Will allocating more memory to JVM (Total heap+ >>> YG space ) solve this problem of slow response ? Is it normal to have 2 >>> Parnew / second ?(and they are always taking > 0.1 Second) >>> >>> Thanks in advance. >>> Pravah >>> >>> >>> >>> On Fri, Feb 3, 2012 at 11:58 AM, Srinivas Ramakrishna >> > wrote: >>> >>>> Is it possible for you to check if this happens with 1.6.0_29 as well? >>>> >>>> -- ramki >>>> >>>> On Thu, Feb 2, 2012 at 9:30 AM, pravah wordpress < >>>> pravah.wordpress at gmail.com> wrote: >>>> >>>>> Hi Team, >>>>> For last couple of weeks, while monitoring production logs, we are >>>>> seeing following 2 scenarios- >>>>> 2012-01-30T09:45:57.783+0530: 211563.940: [GC 211563.941: [ParNew: >>>>> 1695683K->20039K(1887488K), 0.1443767 secs] 2704741K->1029575K(5033216K), >>>>> 0.1456574 secs] [Times: user=0.42 sys=0.02, real=0.15 secs] >>>>> 2012-01-30T09:46:35.195+0530: 211601.351: [GC 211601.352: [ParNew: >>>>> 1697863K->21131K(1887488K), 0.1623961 secs] 2707399K->1031401K(5033216K), >>>>> 0.1636535 secs] [Times: user=0.41 sys=0.09, real=0.16 secs] >>>>> 2012-01-30T09:47:15.555+0530: 211641.711: [GC 211641.712: [ParNew: >>>>> 1698955K->19118K(1887488K), 0.1422631 secs] 2709225K->1030329K(5033216K), >>>>> 0.1435373 secs] [Times: user=0.42 sys=0.01, real=0.14 secs] >>>>> 2012-01-30T09:47:55.656+0530: 211681.811: [GC 211681.812: [ParNew: >>>>> 1696942K->21437K(1887488K), 0.1340463 secs] 2708153K->1033449K(5033216K), >>>>> 0.1352694 secs] [Times: user=0.40 sys=0.01, real=0.14 secs] >>>>> 2012-01-30T09:48:29.406+0530: 211715.562: [GC 211715.564: [ParNew: >>>>> 1699261K->18356K(1887488K), 0.1362798 secs] 2711273K->1031807K(5033216K), >>>>> 0.1387961 secs] [Times: user=0.40 sys=0.03, real=0.14 secs] >>>>> 2012-01-30T09:48:57.017+0530: 211743.172: [GC 211743.173: [ParNew: >>>>> 1696035K->22375K(1887488K), 12.3948960 secs] 2709487K->1036955K(5033216K), >>>>> 12.3962135 secs] [Times: *user=15.73 sys=0.60, real=12.40 secs] >>>>> *2012-01-30T09:49:41.237+0530: 211787.392: [GC 211787.392: [ParNew: >>>>> 1698873K->21698K(1887488K), 0.1251361 secs] 2713453K->1037007K(5033216K), >>>>> 0.1262755 secs] [Times: user=0.39 sys=0.00, real=0.13 secs] >>>>> 2012-01-30T09:50:24.348+0530: 211830.503: [GC 211830.504: [ParNew: >>>>> 1699522K->21849K(1887488K), 0.1370179 secs] 2714831K->1037980K(5033216K), >>>>> 0.1382479 secs] [Times: user=0.40 sys=0.01, real=0.14 secs] >>>>> 2012-01-30T09:50:58.516+0530: 211864.670: [GC 211864.671: [ParNew: >>>>> 1699673K->25281K(1887488K), 0.1334443 secs] 2715804K->1042397K(5033216K), >>>>> 0.1347060 secs] [Times: user=0.43 sys=0.00, real=0.14 secs] >>>>> 2012-01-30T09:51:33.830+0530: 211899.985: [GC 211899.985: [ParNew: >>>>> 1695938K->28123K(1887488K), 0.1449732 secs] 2713054K->1046170K(5033216K), >>>>> 0.1462090 secs] [Times: user=0.44 sys=0.01, real=0.15 secs] >>>>> >>>>> This is happening atleast once in a day and the real time is around 10 >>>>> S. >>>>> >>>>> Another observation is as follows >>>>> >>>>> 2012-01-30T14:53:11.645+0530: 229997.686: [GC 229997.687: [ParNew: >>>>> 1686605K->39956K(1887488K), 0.1504394 secs] 3258191K->1612228K(5033216K), >>>>> 0.1516840 secs] [Times: user=0.47 sys=0.02, real=0.15 secs] >>>>> 2012-01-30T14:53:45.006+0530: 230031.047: [GC 230031.048: [ParNew: >>>>> 1717780K->30950K(1887488K), 0.1693496 secs] 3290052K->1604025K(5033216K), >>>>> 0.1705586 secs] [Times: user=0.52 sys=0.04, real=0.17 secs] >>>>> 2012-01-30T14:53:45.208+0530: 230031.249: [GC [1 CMS-initial-mark: >>>>> 1573075K(3145728K)] 1608943K(5033216K), 0.4815794 secs] [Times: user=0.13 >>>>> sys=0.35, real=0.48 secs] >>>>> 2012-01-30T14:53:45.692+0530: 230031.733: [CMS-concurrent-mark-start] >>>>> 2012-01-30T14:53:52.114+0530: 230038.155: [CMS-concurrent-mark: >>>>> 6.283/6.422 secs] [Times: user=11.70 sys=2.40, *real=6.42 secs*] >>>>> 2012-01-30T14:53:52.115+0530: 230038.155: >>>>> [CMS-concurrent-preclean-start] >>>>> 2012-01-30T14:53:52.470+0530: 230038.510: [CMS-concurrent-preclean: >>>>> 0.346/0.355 secs] [Times: user=0.45 sys=0.16, real=0.36 secs] >>>>> 2012-01-30T14:53:52.470+0530: 230038.511: >>>>> [CMS-concurrent-abortable-preclean-start] >>>>> CMS: abort preclean due to time 2012-01-30T14:53:57.500+0530: >>>>> 230043.541: [CMS-concurrent-abortable-preclean: 4.906/5.030 secs] [Times: >>>>> user=8.73 sys=0.62, *real=5.03 secs*] >>>>> 2012-01-30T14:53:57.532+0530: 230043.573: [GC[YG occupancy: 839183 K >>>>> (1887488 K)]230043.573: [Rescan (parallel) , 1.8934193 secs]230045.467: >>>>> [weak refs processing, 0.0612851 secs] [1 CMS-remark: 1573075K(3145728K)] >>>>> 2412258K(5033216K), 1.9618014 secs] [Times: user=5.58 sys=0.10, real=1.96 >>>>> secs] >>>>> 2012-01-30T14:53:59.497+0530: 230045.538: [CMS-concurrent-sweep-start] >>>>> 2012-01-30T14:54:08.616+0530: 230054.656: [CMS-concurrent-sweep: >>>>> 8.635/9.118 secs] [Times: user=17.84 sys=2.36, *real=9.12 secs*] >>>>> 2012-01-30T14:54:08.617+0530: 230054.657: [CMS-concurrent-reset-start] >>>>> 2012-01-30T14:54:08.656+0530: 230054.696: [CMS-concurrent-reset: >>>>> 0.039/0.039 secs] [Times: user=0.05 sys=0.00, real=0.04 secs] >>>>> 2012-01-30T14:54:12.410+0530: 230058.451: [GC 230058.452: [ParNew: >>>>> 1708774K->25197K(1887488K), 0.1436152 secs] 2456494K->773832K(5033216K), >>>>> 0.1448754 secs] [Times: user=0.48 sys=0.01, real=0.15 secs] >>>>> 2012-01-30T14:54:38.243+0530: 230084.284: [GC 230084.284: [ParNew: >>>>> 1703021K->62457K(1887488K), 0.2070478 secs] 2451656K->811902K(5033216K), >>>>> 0.2083137 secs] [Times: user=0.67 sys=0.01, real=0.21 secs] >>>>> >>>>> >>>>> Here CMS concurrent mark and sweep and also remark is taking >>>>> considerable time many time above 15 seconds. This also happens once in a >>>>> day. >>>>> Why is this happening ? >>>>> >>>>> Also no full GC is observed. Also Parnew is happening two times per >>>>> second during peak load. Is this a cause of worry / slow performance ? This >>>>> is OLTP app and no batch jobs are running. >>>>> Following is the configuration >>>>> Java HotSpot(TM) 64-Bit Server VM (16.3-b01) for solaris-sparc JRE >>>>> (1.6.0_20-b02) >>>>> >>>>> -XX:+DisableExplicitGC >>>>> -XX:-DoEscapeAnalysis >>>>> -XX:InitialHeapSize=5368709120 >>>>> -XX:+LogVMOutput >>>>> -XX:MaxHeapSize=5368709120 >>>>> -XX:MaxNewSize=2147483648 >>>>> -XX:MaxPermSize=536870912 >>>>> -XX:MaxTenuringThreshold=4 >>>>> -XX:NewRatio=3 >>>>> -XX:NewSize=2147483648 >>>>> -XX:ParallelGCThreads=4 >>>>> -XX:PermSize=536870912 >>>>> -XX:+PrintCommandLineFlags >>>>> -XX:+PrintGC >>>>> -XX:+PrintGCDateStamps >>>>> -XX:+PrintGCDetails >>>>> -XX:+PrintGCTimeStamps >>>>> -XX:ThreadStackSize=256 >>>>> -XX:+TraceClassUnloading >>>>> -XX:+UnlockDiagnosticVMOptions >>>>> -XX:+UseConcMarkSweepGC >>>>> -XX:+UseParNewGC >>>>> >>>>> Thanks in advance, >>>>> Pravah. >>>>> >>>>> _______________________________________________ >>>>> hotspot-gc-use mailing list >>>>> hotspot-gc-use at openjdk.java.net >>>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120210/7f07ad0b/attachment.html From fancyerii at gmail.com Thu Feb 16 02:55:22 2012 From: fancyerii at gmail.com (Li Li) Date: Thu, 16 Feb 2012 18:55:22 +0800 Subject: why perm generation is continuously increasing? Message-ID: hi all, I found our application's perm generation size is continuously increasing and will reach 100%(we limit maxPermSize to 256MB) and perform a full gc(which will pause for 10s).after gc, perm size usage falls to about 50% and again increasing. what kind of objects will be allocated to perm generation? we don't use reflection in our application. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120216/efbf72e6/attachment.html From fancyerii at gmail.com Thu Feb 16 04:53:59 2012 From: fancyerii at gmail.com (Li Li) Date: Thu, 16 Feb 2012 20:53:59 +0800 Subject: why perm generation is continuously increasing? In-Reply-To: References: Message-ID: I got it. we use String.intern for temporary variables. I used jmap -permstat and found that. btw jmap -permstat will cause jvm hang and should be carefully used. it hangs for a few minutes! On Thu, Feb 16, 2012 at 6:55 PM, Li Li wrote: > hi all, > I found our application's perm generation size is continuously > increasing and will reach 100%(we limit maxPermSize to 256MB) and perform a > full gc(which will pause for 10s).after gc, perm size usage falls to about > 50% and again increasing. what kind of objects will be allocated to perm > generation? we don't use reflection in our application. > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120216/e458df77/attachment.html From jon.masamitsu at oracle.com Thu Feb 16 12:49:43 2012 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Thu, 16 Feb 2012 12:49:43 -0800 Subject: why perm generation is continuously increasing? In-Reply-To: References: Message-ID: <4F3D6BE7.6080609@oracle.com> Which JDK release? On 02/16/12 02:55, Li Li wrote: > hi all, > I found our application's perm generation size is continuously > increasing and will reach 100%(we limit maxPermSize to 256MB) and > perform a full gc(which will pause for 10s).after gc, perm size usage > falls to about 50% and again increasing. what kind of objects will be > allocated to perm generation? we don't use reflection in our application. > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120216/ee941f56/attachment.html From alexey.ragozin at gmail.com Thu Feb 16 22:00:50 2012 From: alexey.ragozin at gmail.com (Alexey Ragozin) Date: Fri, 17 Feb 2012 10:00:50 +0400 Subject: why perm generation is continuously increasing? Message-ID: Hi, FYI Normally permanent generation is collect only during Full GC. -XX:+CMSClassUnloadingEnabled will enable permanent space collection as a part of concurrent old space collection. Regards, Date: Thu, 16 Feb 2012 20:53:59 +0800 > From: Li Li > Subject: Re: why perm generation is continuously increasing? > To: hotspot-gc-use > Message-ID: > > > Content-Type: text/plain; charset="iso-8859-1" > > I got it. we use String.intern for temporary variables. > I used jmap -permstat and found that. > btw jmap -permstat will cause jvm hang and should be carefully used. > it hangs for a few minutes! > > On Thu, Feb 16, 2012 at 6:55 PM, Li Li wrote: > > > hi all, > > I found our application's perm generation size is continuously > > increasing and will reach 100%(we limit maxPermSize to 256MB) and > perform a > > full gc(which will pause for 10s).after gc, perm size usage falls to > about > > 50% and again increasing. what kind of objects will be allocated to perm > > generation? we don't use reflection in our application. > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120216/e458df77/attachment-0001.html > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120217/6ec742d5/attachment.html From fancyerii at gmail.com Thu Feb 16 22:52:13 2012 From: fancyerii at gmail.com (Li Li) Date: Fri, 17 Feb 2012 14:52:13 +0800 Subject: why perm generation is continuously increasing? In-Reply-To: References: Message-ID: I want to use btrace to locate which function call String.intern() but from http://kenai.com/projects/btrace/forums/forum/topics/1109-Tracing-simple-program-on-Windows it seems it's not feasible any other methods? I tried trace method call of String, constructor method and trim method is ok. but native method like intern is not. On Fri, Feb 17, 2012 at 2:00 PM, Alexey Ragozin wrote: > Hi, > > FYI > Normally permanent generation is collect only during Full GC. > -XX:+CMSClassUnloadingEnabled will enable permanent space collection as a > part of concurrent old space collection. > > Regards, > > Date: Thu, 16 Feb 2012 20:53:59 +0800 >> From: Li Li >> Subject: Re: why perm generation is continuously increasing? >> To: hotspot-gc-use >> Message-ID: >> > 0cTCpCB74wD03VgYe_GhgPj78mcU_pk3syaQ at mail.gmail.com> >> Content-Type: text/plain; charset="iso-8859-1" >> >> >> I got it. we use String.intern for temporary variables. >> I used jmap -permstat and found that. >> btw jmap -permstat will cause jvm hang and should be carefully used. >> it hangs for a few minutes! >> >> On Thu, Feb 16, 2012 at 6:55 PM, Li Li wrote: >> >> > hi all, >> > I found our application's perm generation size is continuously >> > increasing and will reach 100%(we limit maxPermSize to 256MB) and >> perform a >> > full gc(which will pause for 10s).after gc, perm size usage falls to >> about >> > 50% and again increasing. what kind of objects will be allocated to perm >> > generation? we don't use reflection in our application. >> > >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: >> http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120216/e458df77/attachment-0001.html >> >> > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120217/56fbc3c4/attachment.html From paul.hohensee at oracle.com Fri Feb 17 08:38:27 2012 From: paul.hohensee at oracle.com (Paul Hohensee) Date: Fri, 17 Feb 2012 11:38:27 -0500 Subject: regarding sudden memory increase In-Reply-To: <1329494066.11411.YahooMailClassic@web120004.mail.ne1.yahoo.com> References: <1329494066.11411.YahooMailClassic@web120004.mail.ne1.yahoo.com> Message-ID: <4F3E8283.2060707@oracle.com> Your question would be better answered by someone on the hotstpot-gc-use list, cc'ed. You don't mention which version of the JDK you're using, but you should be using the latest update of JDK 7, which is update 2. Paul On 2/17/12 10:54 AM, sakthivel wrote: > Dear Sir, > I have the following setting for my system > > -Xms1536M > -Xmx1536M > -verbose:gc > -XX:+PrintGCDateStamps > -XX:NewSize=768M > -XX:MaxNewSize=768M > -XX:PermSize=256m > -XX:MaxPermSize=256m > -XX:+CMSClassUnloadingEnabled > -XX:+UseConcMarkSweepGC > -XX:+CMSIncrementalMode > -XX:+DisableExplicitGC > -XX:+ExplicitGCInvokesConcurrent > -XX:GCTimeRatio=19 > -Xincgc > -XX:SurvivorRatio=2 > -XX:MaxTenuringThreshold=50 > -XX:MaxGCPauseMillis=12000 > > and My logs as below > > 2012-02-16T09:53:47.907-0500: [GC 462167K->82462K(1376256K), 0.0554103 > secs] > 2012-02-16T09:53:52.275-0500: [GC 475678K->68672K(1376256K), 0.0541640 > secs] > 2012-02-16T09:53:53.289-0500: [GC 461888K->68574K(1376256K), 0.0539152 > secs] > 2012-02-16T09:53:56.471-0500: [GC 461790K->68683K(1376256K), 0.0556297 > secs] > 2012-02-16T09:53:58.281-0500: [GC 461899K->95837K(1376256K), 0.0757101 > secs] > > in certain case the tomcat memory increases as follows > > 2012-02-16T12:02:32.182-0500: [GC 454346K->97556K(1376256K), 0.1354231 > secs] > 2012-02-16T12:03:30.542-0500: [GC 490772K->128467K(1376256K), > 0.2537510 secs] > 2012-02-16T12:04:28.792-0500: [GC 521683K->223446K(1376256K), > 0.3725598 secs] > 2012-02-16T12:05:14.235-0500: [GC 616662K->265305K(1376256K), > 0.3163817 secs] > 2012-02-16T12:05:20.459-0500: [GC 452169K(1376256K), 0.2202838 secs] > 2012-02-16T12:05:26.949-0500: [GC 658521K->253437K(1376256K), > 0.2352744 secs] > 2012-02-16T12:05:48.524-0500: [GC 468368K(1376256K), 0.1812084 secs] > > and tomcat crashes, please help in this regard > > Thanks, > Sakthi > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120217/4e0f32a5/attachment.html From hjohn at xs4all.nl Fri Feb 17 08:54:45 2012 From: hjohn at xs4all.nl (John Hendrikx) Date: Fri, 17 Feb 2012 17:54:45 +0100 Subject: Java 7, soft references not being cleaned? Message-ID: <4F3E8655.6050204@xs4all.nl> I've written a fairly small program that runs in 256 MB -- this program will load several large pictures, which are being "cached" using SoftReferences. The more of these pictures get loaded, the more the GC starts panicing (while in reality, more than 200 MB in the VM is only held by SoftReferences). In the output below you can see the GC getting progressively longer after each image is loaded (see Loading Image in log), doing more and more GC action. The performance of the program is severely impacted (the CPU is maxed while this is happening). To be sure that the references are REALLY softly reachable, I've used VisualVM to trigger a Full GC, the result of which is visible on the last line, copied here: 540.704: [Full GC (System) 540.704: [Tenured: 169249K->35169K(174784K), 0.1363699 secs] 175870K->35169K(253440K), [Perm : 17122K->17122K(17152K)], 0.1364401 secs] [Times: user=0.13 sys=0.00, real=0.14 secs] Unfortunately, even AFTER this Full GC, the GC still goes crazy doing much the same thing as when the images got loaded. See the 2nd part of the log. This seems like a pretty serious issue as I was under the impression that SoftReferences would be cleared in time to prevent long GC cycles. I've tested this with 32-bit JDK 1.7u2 and the 1.7u4 developer preview (which was said to have fixed some issues related to references). I'll be happy to provide any more information you might need. --John Hendrikx The output: 0.533: [GC 0.533: [DefNew: 4416K->512K(4928K), 0.0039006 secs] 4416K->517K(15872K), 0.0039602 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 0.799: [GC 0.799: [DefNew: 4928K->512K(4928K), 0.0062266 secs] 4933K->1165K(15872K), 0.0062645 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 1.058: [GC 1.058: [DefNew: 4928K->512K(4928K), 0.0073100 secs] 5581K->1690K(15872K), 0.0073504 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 1.256: [GC 1.256: [DefNew: 4928K->512K(4928K), 0.0069235 secs] 6106K->2515K(15872K), 0.0069609 secs] [Times: user=0.00 sys=0.01, real=0.01 secs] 1.418: [GC 1.418: [DefNew: 4926K->512K(4928K), 0.0051198 secs] 6930K->2793K(15872K), 0.0052875 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 1.553: [GC 1.553: [DefNew: 4926K->512K(4928K), 0.0044132 secs] 7207K->3321K(15872K), 0.0044506 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 1.627: [GC 1.627: [DefNew: 4928K->512K(4928K), 0.0039100 secs] 7737K->3708K(15872K), 0.0039692 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 1.697: [GC 1.697: [DefNew: 4928K->512K(4928K), 0.0041284 secs] 8124K->3844K(15872K), 0.0041961 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 1.833: [GC 1.833: [DefNew: 4928K->277K(4928K), 0.0034055 secs] 8260K->3781K(15872K), 0.0034434 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] [FINE] ProgramController.registerService() - registering new service: hs.mediasystem.screens.SubtitleDownloadService at ae214c [INFO] Navigator.navigateTo() - Destination('Home'; modal=false) 1.978: [GC 1.978: [DefNew: 4693K->512K(4928K), 0.0064901 secs] 8197K->4614K(15872K), 0.0065348 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] [FINE] ProgramController.registerService() - registering new service: hs.mediasystem.screens.MessagePaneExecutorService$GroupWorker at c3f72c 2.306: [GC 2.306: [DefNew: 4928K->512K(4928K), 0.0056779 secs] 9030K->5162K(15872K), 0.0057234 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 2.512: [GC 2.512: [DefNew: 4921K->472K(4928K), 0.0060734 secs] 9571K->5624K(15872K), 0.0061185 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 4.783: [GC 4.783: [DefNew: 4888K->330K(4928K), 0.0056426 secs] 10040K->5879K(15872K), 0.0056834 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] [INFO] Navigator.navigateTo() - Destination('Movies'; modal=false) [INFO] Navigator.navigateTo() - Destination('Movies'; modal=false) 5.476: [GC 5.476: [DefNew: 4746K->255K(4928K), 0.0050245 secs] 10295K->6108K(15872K), 0.0050653 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 5.535: [GC 5.536: [DefNew: 4671K->512K(4928K), 0.0049691 secs] 10524K->6456K(15872K), 0.0050100 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 5.693: [GC 5.693: [DefNew: 4928K->512K(4928K), 0.0085990 secs] 10872K->7455K(15872K), 0.0086437 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 5.873: [GC 5.874: [DefNew: 4928K->512K(4928K), 0.0131007 secs] 11871K->9422K(15872K), 0.0131450 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 6.036: [GC 6.036: [DefNew: 4928K->512K(4928K), 0.0094798 secs] 13838K->10399K(15872K), 0.0095228 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] [FINE] CachedItemEnricher.identifyItem() - Cache Hit [FINE] CachedItemEnricher.identifyItem() - Cache Hit [FINE] CachedItemEnricher.identifyItem() - Cache Hit [FINE] CachedItemEnricher.enrichItem() - Loading from Cache: (MOVIE;TMDB;tt0112281) [FINE] CachedItemEnricher.enrichItem() - Loading from Cache: (MOVIE;TMDB;tt1190080) [FINE] CachedItemEnricher.enrichItem() - Loading from Cache: (MOVIE;TMDB;tt1461312) [FINE] CachedItemEnricher.identifyItem() - Cache Hit [FINE] CachedItemEnricher.enrichItem() - Loading from Cache: (MOVIE;TMDB;tt0080339) [FINE] ItemsDao.getItem() - Selecting Item with type/provider/providerid = MOVIE/TMDB/tt1190080 [FINE] ItemsDao.getItem() - Selecting Item with type/provider/providerid = MOVIE/TMDB/tt0112281 [FINE] ItemsDao.getItem() - Selecting Item with type/provider/providerid = MOVIE/TMDB/tt1461312 6.378: [GC 6.378: [DefNew: 4928K->488K(4928K), 0.0084662 secs]6.386: [Tenured: 11579K->10280K(12252K), 0.0909118 secs] 14815K->10280K(17180K), [Perm : 15406K->15406K(15616K)], 0.0995113 secs] [Times: user=0.06 sys=0.00, real=0.10 secs] [FINE] CachedItemEnricher.identifyItem() - Cache Hit [FINE] CachedItemEnricher.enrichItem() - Loading from Cache: (MOVIE;TMDB;tt0429493) [FINE] ItemsDao.getItem() - Selecting Item with type/provider/providerid = MOVIE/TMDB/tt0429493 [FINE] ItemsDao.getItem() - Selecting Item with type/provider/providerid = MOVIE/TMDB/tt0080339 [FINE] CachedItemEnricher.identifyItem() - Cache Hit [FINE] CachedItemEnricher.enrichItem() - Loading from Cache: (MOVIE;TMDB;tt0462200) [FINE] ItemsDao.getItem() - Selecting Item with type/provider/providerid = MOVIE/TMDB/tt0462200 [FINE] ItemsDao.getImage() - Loading image BACKGROUND(id=1) [FINE] CachedItemEnricher.identifyItem() - Cache Hit [FINE] CachedItemEnricher.enrichItem() - Loading from Cache: (MOVIE;TMDB;tt0090728) [FINE] ItemsDao.getItem() - Selecting Item with type/provider/providerid = MOVIE/TMDB/tt0090728 [FINE] CachedItemEnricher.identifyItem() - Cache Hit [FINE] CachedItemEnricher.enrichItem() - Loading from Cache: (MOVIE;TMDB;tt0096895) [FINE] ItemsDao.getItem() - Selecting Item with type/provider/providerid = MOVIE/TMDB/tt0096895 [FINE] CachedItemEnricher.identifyItem() - Cache Hit [FINE] CachedItemEnricher.enrichItem() - Loading from Cache: (MOVIE;TMDB;tt0286499) [FINE] ItemsDao.getItem() - Selecting Item with type/provider/providerid = MOVIE/TMDB/tt0286499 [FINE] CachedItemEnricher.identifyItem() - Cache Hit [FINE] CachedItemEnricher.enrichItem() - Loading from Cache: (MOVIE;TMDB;tt0088763) [FINE] CachedItemEnricher.identifyItem() - Cache Hit [FINE] CachedItemEnricher.enrichItem() - Loading from Cache: (MOVIE;TMDB;tt0384819) [FINE] ItemsDao.getItem() - Selecting Item with type/provider/providerid = MOVIE/TMDB/tt0384819 [FINE] ItemsDao.getItem() - Selecting Item with type/provider/providerid = MOVIE/TMDB/tt0088763 [FINE] CachedItemEnricher.identifyItem() - Cache Hit [FINE] CachedItemEnricher.enrichItem() - Loading from Cache: (MOVIE;TMDB;tt1043842) [FINE] CachedItemEnricher.identifyItem() - Cache Hit [FINE] CachedItemEnricher.enrichItem() - Loading from Cache: (MOVIE;TMDB;tt0118655) [FINE] ItemsDao.getItem() - Selecting Item with type/provider/providerid = MOVIE/TMDB/tt1043842 [FINE] ItemsDao.getItem() - Selecting Item with type/provider/providerid = MOVIE/TMDB/tt0118655 [FINE] CachedItemEnricher.identifyItem() - Cache Hit [FINE] CachedItemEnricher.enrichItem() - Loading from Cache: (MOVIE;TMDB;tt0066769) [FINE] CachedItemEnricher.identifyItem() - Cache Hit [FINE] CachedItemEnricher.enrichItem() - Loading from Cache: (MOVIE;TMDB;tt0163651) [FINE] ItemsDao.getItem() - Selecting Item with type/provider/providerid = MOVIE/TMDB/tt0066769 [FINE] ItemsDao.getItem() - Selecting Item with type/provider/providerid = MOVIE/TMDB/tt0163651 [FINE] CachedItemEnricher.identifyItem() - Cache Hit [FINE] CachedItemEnricher.enrichItem() - Loading from Cache: (MOVIE;TMDB;tt0078748) [FINE] ItemsDao.getItem() - Selecting Item with type/provider/providerid = MOVIE/TMDB/tt0078748 [FINE] CachedItemEnricher.identifyItem() - Cache Hit [FINE] CachedItemEnricher.enrichItem() - Loading from Cache: (MOVIE;TMDB;tt1014759) [FINE] CachedItemEnricher.identifyItem() - Cache Hit [FINE] CachedItemEnricher.identifyItem() - Cache Hit [FINE] CachedItemEnricher.enrichItem() - Loading from Cache: (MOVIE;TMDB;tt0499549) [FINE] ItemsDao.getItem() - Selecting Item with type/provider/providerid = MOVIE/TMDB/tt1014759 [FINE] ItemsDao.getItem() - Selecting Item with type/provider/providerid = MOVIE/TMDB/tt0499549 [FINE] ItemsDao.getItem() - Selecting Item with type/provider/providerid = MOVIE/TMDB/tt0401233 6.757: [GC 6.757: [DefNew: 4579K->143K(7808K), 0.0028406 secs] 14859K->11675K(24944K), 0.0028785 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] [FINE] ItemsDao.getImage() - Loading image POSTER(id=1) 6.884: [GC 6.884: [DefNew: 6289K->83K(7808K), 0.0079273 secs]6.892: [Tenured: 17606K->17687K(23212K), 0.0705893 secs] 17821K->17687K(31020K), [Perm : 15436K->15436K(15616K)], 0.0786506 secs] [Times: user=0.08 sys=0.00, real=0.08 secs] 7.627: [GC 7.627: [DefNew: 11840K->1472K(13312K), 0.0100919 secs] 29527K->24266K(42792K), 0.0101409 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 13.167: [GC 13.167: [DefNew: 13312K->732K(13312K), 0.0128674 secs] 36106K->24995K(42792K), 0.0129245 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] [FINE] ItemsDao.getImage() - Loading image BACKGROUND(id=11) 17.311: [GC 17.311: [DefNew: 6573K->1472K(13312K), 0.0065557 secs] 30835K->25930K(42792K), 0.0066008 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] [FINE] ItemsDao.getImage() - Loading image POSTER(id=11) 17.419: [GC 17.419: [DefNew: 9050K->488K(13312K), 0.0113187 secs]17.430: [Tenured: 32000K->31257K(35556K), 0.0812767 secs] 33509K->31257K(48868K), [Perm : 17105K->17105K(17152K)], 0.0927683 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] [FINE] ItemsDao.getImage() - Loading image BACKGROUND(id=3) [FINE] ItemsDao.getImage() - Loading image POSTER(id=3) 18.313: [GC 18.313: [DefNew: 18001K->845K(23552K), 0.0109778 secs] 49259K->42572K(75648K), 0.0110272 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] [FINE] ItemsDao.getImage() - Loading image BACKGROUND(id=10) [FINE] ItemsDao.getImage() - Loading image POSTER(id=10) 19.439: [GC 19.439: [DefNew: 21837K->1685K(23552K), 0.0167106 secs]19.456: [Tenured: 56599K->56776K(58172K), 0.1225932 secs] 63564K->56776K(81724K), [Perm : 17112K->17096K(17152K)], 0.1395690 secs] [Times: user=0.13 sys=0.02, real=0.14 secs] [FINE] ItemsDao.getImage() - Loading image BACKGROUND(id=4) [FINE] ItemsDao.getImage() - Loading image POSTER(id=4) [FINE] ItemsDao.getImage() - Loading image BACKGROUND(id=80) [FINE] ItemsDao.getImage() - Loading image POSTER(id=80) [FINE] ItemsDao.getImage() - Loading image BACKGROUND(id=12) 23.176: [GC 23.176: [DefNew: 37362K->4735K(42624K), 0.0218504 secs] 94139K->75888K(137252K), 0.0218968 secs] [Times: user=0.01 sys=0.02, real=0.02 secs] [FINE] ItemsDao.getImage() - Loading image POSTER(id=12) [FINE] ItemsDao.getImage() - Loading image BACKGROUND(id=9) [FINE] ItemsDao.getImage() - Loading image POSTER(id=9) [FINE] ItemsDao.getImage() - Loading image BACKGROUND(id=422) 24.709: [GC 24.709: [DefNew: 41972K->1900K(42624K), 0.0278884 secs]24.737: [Tenured: 96754K->98653K(100700K), 0.0819731 secs] 113125K->98653K(143324K), [Perm : 17108K->17108K(17152K)], 0.1102289 secs] [Times: user=0.09 sys=0.01, real=0.11 secs] [FINE] ItemsDao.getImage() - Loading image POSTER(id=422) [FINE] ItemsDao.getImage() - Loading image BACKGROUND(id=8) [FINE] ItemsDao.getImage() - Loading image POSTER(id=8) 25.721: [GC 25.721: [DefNew: 65899K->7250K(74112K), 0.0257604 secs] 164552K->119050K(238536K), 0.0258098 secs] [Times: user=0.03 sys=0.00, real=0.03 secs] [FINE] ItemsDao.getImage() - Loading image BACKGROUND(id=6) [FINE] ItemsDao.getImage() - Loading image POSTER(id=6) 26.700: [GC 26.700: [DefNew: 73161K->5097K(74112K), 0.0238588 secs] 184961K->130126K(238536K), 0.0239095 secs] [Times: user=0.02 sys=0.02, real=0.02 secs] 26.797: [GC 26.797: [DefNew: 71017K->108K(74112K), 0.0086658 secs] 196046K->130189K(238536K), 0.0087122 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 26.893: [GC 26.893: [DefNew: 66028K->241K(74112K), 0.0055600 secs] 196109K->130321K(238536K), 0.0056000 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] [FINE] ItemsDao.getImage() - Loading image BACKGROUND(id=111) [FINE] ItemsDao.getImage() - Loading image POSTER(id=111) 28.006: [GC 28.006: [DefNew: 66161K->6107K(74112K), 0.0134340 secs] 196241K->142263K(238536K), 0.0134817 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 28.082: [GC 28.082: [DefNew: 72021K->115K(74112K), 0.0097897 secs] 208177K->142318K(238536K), 0.0098378 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 28.154: [GC 28.154: [DefNew: 66002K->175K(74112K), 0.0026793 secs] 208204K->142378K(238536K), 0.0027206 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 28.219: [GC 28.219: [DefNew: 66038K->245K(74112K), 0.0035120 secs] 208240K->142448K(238536K), 0.0035609 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 28.285: [GC 28.285: [DefNew: 66113K->273K(74112K), 0.0029352 secs] 208316K->142476K(238536K), 0.0029803 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 28.345: [GC 28.345: [DefNew: 66193K->299K(74112K), 0.0030522 secs] 208396K->142502K(238536K), 0.0030956 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 28.406: [GC 28.406: [DefNew: 66159K->331K(74112K), 0.0031608 secs] 208362K->142534K(238536K), 0.0032038 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 28.467: [GC 28.467: [DefNew: 66251K->367K(74112K), 0.0030995 secs] 208454K->142570K(238536K), 0.0031450 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 28.530: [GC 28.530: [DefNew: 66248K->394K(74112K), 0.0032489 secs] 208451K->142597K(238536K), 0.0032957 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 28.593: [GC 28.593: [DefNew: 66296K->421K(74112K), 0.0033157 secs] 208499K->142624K(238536K), 0.0033604 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] [FINE] ItemsDao.getImage() - Loading image BACKGROUND(id=7) [FINE] ItemsDao.getImage() - Loading image POSTER(id=7) 29.787: [GC 29.787: [DefNew: 66270K->5694K(74112K), 0.0145336 secs] 208473K->153972K(238536K), 0.0145821 secs] [Times: user=0.01 sys=0.00, real=0.02 secs] 29.857: [GC 29.857: [DefNew: 71614K->115K(74112K), 0.0095751 secs] 219892K->154008K(238536K), 0.0096211 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 29.917: [GC 29.918: [DefNew: 65999K->155K(74112K), 0.0025814 secs] 219893K->154048K(238536K), 0.0026308 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 29.969: [GC 29.969: [DefNew: 66040K->224K(74112K), 0.0025861 secs] 219934K->154118K(238536K), 0.0026274 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 30.016: [GC 30.016: [DefNew: 66135K->252K(74112K), 0.0027219 secs] 220029K->154146K(238536K), 0.0027649 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 30.062: [GC 30.062: [DefNew: 66133K->280K(74112K), 0.0027398 secs] 220027K->154173K(238536K), 0.0027785 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 30.108: [GC 30.108: [DefNew: 66199K->310K(74112K), 0.0028998 secs] 220093K->154203K(238536K), 0.0029543 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 30.157: [GC 30.157: [DefNew: 66230K->333K(74112K), 0.0029032 secs] 220123K->154227K(238536K), 0.0029420 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 30.211: [GC 30.211: [DefNew: 66151K->357K(74112K), 0.0030526 secs] 220044K->154251K(238536K), 0.0030927 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 30.271: [GC 30.271: [DefNew: 66214K->386K(74112K), 0.0034072 secs] 220108K->154280K(238536K), 0.0034494 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 30.318: [GC 30.318: [DefNew: 66217K->420K(74112K), 0.0032110 secs] 220111K->154313K(238536K), 0.0032561 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 30.366: [GC 30.366: [DefNew: 66287K->443K(74112K), 0.0031723 secs] 220180K->154337K(238536K), 0.0032097 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 30.413: [GC 30.413: [DefNew: 66269K->467K(74112K), 0.0036048 secs] 220162K->154361K(238536K), 0.0036512 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 30.461: [GC 30.461: [DefNew: 66304K->491K(74112K), 0.0033319 secs] 220198K->154385K(238536K), 0.0033723 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 30.515: [GC 30.515: [DefNew: 66311K->510K(74112K), 0.0035622 secs] 220205K->154404K(238536K), 0.0036103 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 30.578: [GC 30.578: [DefNew: 66338K->528K(74112K), 0.0033110 secs] 220232K->154422K(238536K), 0.0033472 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 30.645: [GC 30.645: [DefNew: 66353K->517K(74112K), 0.0036388 secs] 220247K->154447K(238536K), 0.0036801 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 30.711: [GC 30.711: [DefNew: 66421K->513K(74112K), 0.0034026 secs] 220350K->154479K(238536K), 0.0034417 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 30.767: [GC 30.767: [DefNew: 66386K->499K(74112K), 0.0035320 secs] 220352K->154497K(238536K), 0.0035797 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 30.825: [GC 30.825: [DefNew: 66360K->493K(74112K), 0.0034634 secs] 220357K->154514K(238536K), 0.0035069 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 30.882: [GC 30.882: [DefNew: 66350K->488K(74112K), 0.0036243 secs] 220371K->154533K(238536K), 0.0036703 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 30.943: [GC 30.943: [DefNew: 66342K->482K(74112K), 0.0034864 secs] 220387K->154550K(238536K), 0.0035328 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 31.002: [GC 31.002: [DefNew: 66320K->476K(74112K), 0.0034315 secs] 220388K->154568K(238536K), 0.0034771 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 31.059: [GC 31.059: [DefNew: 66337K->517K(74112K), 0.0032455 secs] 220430K->154634K(238536K), 0.0032868 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 31.116: [GC 31.116: [DefNew: 66432K->505K(74112K), 0.0035298 secs] 220548K->154646K(238536K), 0.0035779 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 31.174: [GC 31.174: [DefNew: 66282K->493K(74112K), 0.0035460 secs] 220423K->154658K(238536K), 0.0035903 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 31.239: [GC 31.239: [DefNew: 66402K->481K(74112K), 0.0034055 secs] 220567K->154670K(238536K), 0.0034558 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 31.297: [GC 31.297: [DefNew: 66376K->470K(74112K), 0.0033217 secs] 220565K->154683K(238536K), 0.0033681 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 31.354: [GC 31.354: [DefNew: 66377K->458K(74112K), 0.0033306 secs] 220590K->154695K(238536K), 0.0033817 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 31.412: [GC 31.412: [DefNew: 66367K->451K(74112K), 0.0032208 secs] 220603K->154707K(238536K), 0.0032659 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 31.469: [GC 31.470: [DefNew: 66355K->446K(74112K), 0.0032314 secs] 220610K->154719K(238536K), 0.0032757 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 31.528: [GC 31.528: [DefNew: 66223K->450K(74112K), 0.0031684 secs] 220496K->154741K(238536K), 0.0032097 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 31.587: [GC 31.587: [DefNew: 66260K->465K(74112K), 0.0031463 secs] 220551K->154774K(238536K), 0.0031935 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 31.643: [GC 31.643: [DefNew: 66382K->459K(74112K), 0.0031689 secs] 220691K->154786K(238536K), 0.0032136 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 31.702: [GC 31.702: [DefNew: 66366K->453K(74112K), 0.0031310 secs] 220693K->154798K(238536K), 0.0031731 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 31.759: [GC 31.759: [DefNew: 66370K->447K(74112K), 0.0031629 secs] 220715K->154810K(238536K), 0.0032114 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 31.818: [GC 31.819: [DefNew: 66356K->441K(74112K), 0.0030518 secs] 220719K->154822K(238536K), 0.0030965 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 31.877: [GC 31.877: [DefNew: 66357K->436K(74112K), 0.0030531 secs] 220737K->154834K(238536K), 0.0030944 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 31.936: [GC 31.936: [DefNew: 66344K->435K(74112K), 0.0030037 secs] 220743K->154846K(238536K), 0.0030509 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] [FINE] ItemsDao.getImage() - Loading image BACKGROUND(id=15) [FINE] ItemsDao.getImage() - Loading image POSTER(id=15) 32.952: [GC 32.952: [DefNew: 66289K->5639K(74112K), 0.0138950 secs] 220700K->166137K(238536K), 0.0139427 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 33.010: [GC 33.010: [DefNew: 71495K->178K(74112K), 0.0091681 secs]33.019: [Tenured: 165972K->166150K(168820K), 0.0815990 secs] 231993K->166150K(242932K), [Perm : 17117K->17117K(17152K)], 0.0909297 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 33.150: [GC 33.151: [DefNew: 69873K->181K(78656K), 0.0023907 secs] 236024K->166332K(253440K), 0.0024277 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 33.207: [GC 33.207: [DefNew: 70076K->200K(78656K), 0.0025056 secs] 236227K->166351K(253440K), 0.0025490 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 33.267: [GC 33.267: [DefNew: 70061K->222K(78656K), 0.0027496 secs] 236212K->166373K(253440K), 0.0027955 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 33.319: [GC 33.319: [DefNew: 70098K->242K(78656K), 0.0025129 secs] 236249K->166393K(253440K), 0.0025520 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 33.372: [GC 33.372: [DefNew: 70109K->259K(78656K), 0.0026346 secs] 236260K->166409K(253440K), 0.0026776 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 33.422: [GC 33.422: [DefNew: 70132K->278K(78656K), 0.0027372 secs] 236282K->166428K(253440K), 0.0027789 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 33.474: [GC 33.474: [DefNew: 70163K->299K(78656K), 0.0026938 secs] 236313K->166450K(253440K), 0.0027317 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 33.524: [GC 33.524: [DefNew: 70171K->316K(78656K), 0.0027266 secs] 236321K->166466K(253440K), 0.0027670 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 33.579: [GC 33.579: [DefNew: 70183K->333K(78656K), 0.0026929 secs] 236334K->166484K(253440K), 0.0027291 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 33.636: [GC 33.636: [DefNew: 70209K->353K(78656K), 0.0029271 secs] 236360K->166503K(253440K), 0.0029662 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 33.692: [GC 33.693: [DefNew: 70237K->376K(78656K), 0.0028151 secs] 236387K->166527K(253440K), 0.0028496 secs] [Times: user=0.00 sys=0.02, real=0.00 secs] 33.749: [GC 33.749: [DefNew: 70244K->393K(78656K), 0.0029620 secs] 236394K->166544K(253440K), 0.0030011 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 33.799: [GC 33.799: [DefNew: 70260K->410K(78656K), 0.0031863 secs] 236411K->166561K(253440K), 0.0032336 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 33.851: [GC 33.851: [DefNew: 70279K->427K(78656K), 0.0030803 secs] 236430K->166578K(253440K), 0.0031229 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 33.901: [GC 33.901: [DefNew: 70309K->449K(78656K), 0.0034328 secs] 236459K->166599K(253440K), 0.0034805 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 33.952: [GC 33.952: [DefNew: 70346K->458K(78656K), 0.0032216 secs] 236497K->166625K(253440K), 0.0032646 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 34.004: [GC 34.004: [DefNew: 70326K->458K(78656K), 0.0032715 secs] 236493K->166642K(253440K), 0.0033153 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 34.055: [GC 34.055: [DefNew: 70328K->456K(78656K), 0.0032766 secs] 236512K->166660K(253440K), 0.0033166 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 34.108: [GC 34.108: [DefNew: 70324K->456K(78656K), 0.0032863 secs] 236527K->166676K(253440K), 0.0033247 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 34.158: [GC 34.158: [DefNew: 70326K->456K(78656K), 0.0032421 secs] 236546K->166693K(253440K), 0.0032863 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 34.220: [GC 34.220: [DefNew: 70352K->456K(78656K), 0.0030961 secs] 236589K->166710K(253440K), 0.0031340 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 34.278: [GC 34.278: [DefNew: 70308K->457K(78656K), 0.0032050 secs] 236562K->166728K(253440K), 0.0032489 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 34.326: [GC 34.326: [DefNew: 70357K->535K(78656K), 0.0035558 secs] 236628K->166823K(253440K), 0.0036095 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 34.371: [GC 34.371: [DefNew: 70446K->536K(78656K), 0.0032587 secs] 236734K->166841K(253440K), 0.0032991 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 34.415: [GC 34.415: [DefNew: 70470K->545K(78656K), 0.0034426 secs] 236775K->166867K(253440K), 0.0034932 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 34.462: [GC 34.462: [DefNew: 70433K->539K(78656K), 0.0034217 secs] 236755K->166878K(253440K), 0.0034677 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 34.506: [GC 34.506: [DefNew: 70423K->533K(78656K), 0.0034366 secs] 236762K->166889K(253440K), 0.0034830 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 34.553: [GC 34.553: [DefNew: 70426K->528K(78656K), 0.0033200 secs] 236782K->166901K(253440K), 0.0033664 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 34.598: [GC 34.598: [DefNew: 70418K->522K(78656K), 0.0033132 secs] 236791K->166912K(253440K), 0.0033630 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 34.645: [GC 34.645: [DefNew: 70416K->517K(78656K), 0.0031650 secs] 236806K->166923K(253440K), 0.0032050 secs] [Times: user=0.00 sys=0.02, real=0.00 secs] 34.688: [GC 34.688: [DefNew: 70406K->511K(78656K), 0.0033055 secs] 236813K->166935K(253440K), 0.0033494 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 34.736: [GC 34.736: [DefNew: 70401K->505K(78656K), 0.0031114 secs] 236824K->166946K(253440K), 0.0031557 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 34.780: [GC 34.780: [DefNew: 70392K->500K(78656K), 0.0032327 secs] 236833K->166957K(253440K), 0.0032795 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 34.827: [GC 34.827: [DefNew: 70389K->494K(78656K), 0.0033855 secs] 236847K->166969K(253440K), 0.0034302 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 34.871: [GC 34.871: [DefNew: 70382K->488K(78656K), 0.0033574 secs] 236857K->166980K(253440K), 0.0034026 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 34.919: [GC 34.919: [DefNew: 70377K->483K(78656K), 0.0031037 secs] 236869K->166991K(253440K), 0.0031459 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 34.963: [GC 34.964: [DefNew: 70374K->476K(78656K), 0.0031007 secs] 236883K->167003K(253440K), 0.0031433 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 35.010: [GC 35.010: [DefNew: 70363K->473K(78656K), 0.0030769 secs] 236889K->167014K(253440K), 0.0031148 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 35.054: [GC 35.054: [DefNew: 70361K->452K(78656K), 0.0031348 secs] 236902K->167025K(253440K), 0.0031812 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 35.099: [GC 35.099: [DefNew: 70343K->410K(78656K), 0.0029781 secs] 236917K->167037K(253440K), 0.0030173 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 35.147: [GC 35.147: [DefNew: 70298K->410K(78656K), 0.0031965 secs] 236925K->167048K(253440K), 0.0032417 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 35.199: [GC 35.199: [DefNew: 70322K->410K(78656K), 0.0030420 secs] 236960K->167059K(253440K), 0.0030901 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 35.255: [GC 35.255: [DefNew: 70221K->489K(78656K), 0.0033864 secs] 236871K->167150K(253440K), 0.0034375 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 35.301: [GC 35.301: [DefNew: 70220K->505K(78656K), 0.0032063 secs] 236881K->167177K(253440K), 0.0032574 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 35.346: [GC 35.346: [DefNew: 70399K->505K(78656K), 0.0030194 secs] 237071K->167189K(253440K), 0.0030633 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 35.389: [GC 35.389: [DefNew: 70393K->505K(78656K), 0.0035098 secs] 237076K->167200K(253440K), 0.0035609 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 35.435: [GC 35.435: [DefNew: 70397K->505K(78656K), 0.0030735 secs] 237092K->167211K(253440K), 0.0031152 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 35.480: [GC 35.480: [DefNew: 70394K->505K(78656K), 0.0033745 secs] 237101K->167223K(253440K), 0.0034204 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 35.527: [GC 35.527: [DefNew: 70391K->505K(78656K), 0.0030684 secs] 237109K->167234K(253440K), 0.0031076 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 35.571: [GC 35.571: [DefNew: 70398K->505K(78656K), 0.0033221 secs] 237127K->167245K(253440K), 0.0033664 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 35.618: [GC 35.618: [DefNew: 70385K->505K(78656K), 0.0030339 secs] 237126K->167256K(253440K), 0.0030761 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 35.671: [GC 35.671: [DefNew: 70325K->502K(78656K), 0.0033476 secs] 237077K->167265K(253440K), 0.0033979 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 35.728: [GC 35.728: [DefNew: 70324K->499K(78656K), 0.0030629 secs] 237087K->167273K(253440K), 0.0031033 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 35.782: [GC 35.782: [DefNew: 70320K->496K(78656K), 0.0029871 secs] 237094K->167282K(253440K), 0.0030245 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 35.837: [GC 35.837: [DefNew: 70319K->493K(78656K), 0.0028555 secs] 237105K->167290K(253440K), 0.0028985 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 35.891: [GC 35.891: [DefNew: 70318K->491K(78656K), 0.0031208 secs] 237115K->167299K(253440K), 0.0031701 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 35.949: [GC 35.949: [DefNew: 70317K->488K(78656K), 0.0029122 secs] 237125K->167307K(253440K), 0.0029547 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 36.003: [GC 36.003: [DefNew: 70311K->453K(78656K), 0.0030467 secs] 237131K->167316K(253440K), 0.0030901 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 36.061: [GC 36.061: [DefNew: 70277K->387K(78656K), 0.0029134 secs] 237140K->167324K(253440K), 0.0029577 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 36.115: [GC 36.115: [DefNew: 70209K->384K(78656K), 0.0029747 secs] 237147K->167333K(253440K), 0.0030186 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 36.173: [GC 36.173: [DefNew: 70233K->381K(78656K), 0.0029381 secs] 237182K->167341K(253440K), 0.0029790 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 36.236: [GC 36.236: [DefNew: 70198K->378K(78656K), 0.0028811 secs] 237158K->167350K(253440K), 0.0029249 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 36.294: [GC 36.294: [DefNew: 70104K->376K(78656K), 0.0029858 secs] 237076K->167359K(253440K), 0.0030314 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 36.349: [GC 36.349: [DefNew: 70194K->373K(78656K), 0.0030356 secs] 237177K->167368K(253440K), 0.0030829 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 36.405: [GC 36.405: [DefNew: 70192K->371K(78656K), 0.0027870 secs] 237186K->167376K(253440K), 0.0028279 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 36.461: [GC 36.461: [DefNew: 70191K->368K(78656K), 0.0030156 secs] 237197K->167385K(253440K), 0.0030633 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 36.516: [GC 36.516: [DefNew: 70187K->368K(78656K), 0.0028956 secs] 237203K->167393K(253440K), 0.0029420 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 36.571: [GC 36.571: [DefNew: 70190K->368K(78656K), 0.0029207 secs] 237215K->167402K(253440K), 0.0029654 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 36.636: [GC 36.636: [DefNew: 70187K->368K(78656K), 0.0026687 secs] 237221K->167410K(253440K), 0.0027053 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 36.695: [GC 36.695: [DefNew: 70190K->368K(78656K), 0.0026768 secs] 237232K->167419K(253440K), 0.0027112 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 36.755: [GC 36.755: [DefNew: 70190K->368K(78656K), 0.0028096 secs] 237241K->167427K(253440K), 0.0028513 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 36.810: [GC 36.810: [DefNew: 70188K->368K(78656K), 0.0031182 secs] 237247K->167436K(253440K), 0.0031676 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 36.867: [GC 36.868: [DefNew: 70190K->368K(78656K), 0.0029543 secs] 237258K->167444K(253440K), 0.0030007 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 36.925: [GC 36.925: [DefNew: 70184K->368K(78656K), 0.0027900 secs] 237260K->167453K(253440K), 0.0028296 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 36.981: [GC 36.981: [DefNew: 70188K->368K(78656K), 0.0031995 secs] 237273K->167461K(253440K), 0.0032527 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 37.038: [GC 37.038: [DefNew: 70187K->368K(78656K), 0.0028692 secs] 237280K->167470K(253440K), 0.0029109 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 37.095: [GC 37.095: [DefNew: 70236K->415K(78656K), 0.0030207 secs] 237338K->167526K(253440K), 0.0030684 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 37.152: [GC 37.152: [DefNew: 70331K->509K(78656K), 0.0031599 secs] 237441K->167629K(253440K), 0.0032076 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 37.217: [GC 37.217: [DefNew: 70354K->509K(78656K), 0.0031642 secs] 237474K->167637K(253440K), 0.0032136 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 37.278: [GC 37.278: [DefNew: 70235K->510K(78656K), 0.0031276 secs] 237363K->167647K(253440K), 0.0031804 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 37.334: [GC 37.334: [DefNew: 70330K->510K(78656K), 0.0030790 secs] 237466K->167655K(253440K), 0.0031263 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 37.391: [GC 37.391: [DefNew: 70332K->510K(78656K), 0.0032067 secs] 237477K->167664K(253440K), 0.0032561 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 37.449: [GC 37.449: [DefNew: 70332K->510K(78656K), 0.0033489 secs] 237486K->167672K(253440K), 0.0033949 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 37.506: [GC 37.506: [DefNew: 70330K->510K(78656K), 0.0042714 secs] 237492K->167681K(253440K), 0.0043216 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 37.564: [GC 37.564: [DefNew: 70332K->510K(78656K), 0.0030901 secs] 237503K->167689K(253440K), 0.0032468 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 37.622: [GC 37.622: [DefNew: 70330K->510K(78656K), 0.0030284 secs] 237509K->167698K(253440K), 0.0030714 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 37.679: [GC 37.679: [DefNew: 70332K->510K(78656K), 0.0030778 secs] 237520K->167706K(253440K), 0.0031220 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 37.739: [GC 37.739: [DefNew: 70330K->510K(78656K), 0.0030326 secs] 237526K->167715K(253440K), 0.0030735 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 37.795: [GC 37.795: [DefNew: 70330K->510K(78656K), 0.0031352 secs] 237534K->167723K(253440K), 0.0031812 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 37.854: [GC 37.854: [DefNew: 70331K->510K(78656K), 0.0030373 secs] 237544K->167732K(253440K), 0.0030803 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 37.913: [GC 37.913: [DefNew: 70330K->510K(78656K), 0.0031152 secs] 237551K->167740K(253440K), 0.0031574 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 37.972: [GC 37.972: [DefNew: 70332K->463K(78656K), 0.0030633 secs] 237562K->167749K(253440K), 0.0031088 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 38.032: [GC 38.032: [DefNew: 70282K->368K(78656K), 0.0031876 secs] 237568K->167757K(253440K), 0.0032293 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 38.090: [GC 38.090: [DefNew: 70190K->368K(78656K), 0.0030109 secs] 237579K->167766K(253440K), 0.0030565 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 38.149: [GC 38.149: [DefNew: 70190K->367K(78656K), 0.0029977 secs] 237588K->167774K(253440K), 0.0030407 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 38.215: [GC 38.215: [DefNew: 70213K->367K(78656K), 0.0049559 secs] 237620K->167783K(253440K), 0.0050010 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 38.278: [GC 38.278: [DefNew: 70089K->368K(78656K), 0.0030186 secs] 237505K->167792K(253440K), 0.0030612 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 38.337: [GC 38.337: [DefNew: 70187K->368K(78656K), 0.0029505 secs] 237611K->167800K(253440K), 0.0029947 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 38.396: [GC 38.396: [DefNew: 70189K->368K(78656K), 0.0029696 secs] 237621K->167809K(253440K), 0.0030109 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 38.455: [GC 38.455: [DefNew: 70189K->368K(78656K), 0.0029437 secs] 237630K->167817K(253440K), 0.0029858 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 38.513: [GC 38.513: [DefNew: 70187K->368K(78656K), 0.0029760 secs] 237637K->167826K(253440K), 0.0030182 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 38.573: [GC 38.573: [DefNew: 70189K->368K(78656K), 0.0029688 secs] 237647K->167834K(253440K), 0.0030126 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 38.634: [GC 38.634: [DefNew: 70187K->368K(78656K), 0.0028177 secs] 237654K->167843K(253440K), 0.0028581 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 38.692: [GC 38.692: [DefNew: 70190K->368K(78656K), 0.0029501 secs] 237665K->167851K(253440K), 0.0029913 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 38.756: [GC 38.756: [DefNew: 70190K->368K(78656K), 0.0029701 secs] 237673K->167860K(253440K), 0.0030105 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 38.817: [GC 38.817: [DefNew: 70188K->368K(78656K), 0.0035192 secs] 237680K->167868K(253440K), 0.0035648 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 38.878: [GC 38.878: [DefNew: 70186K->368K(78656K), 0.0029266 secs] 237686K->167877K(253440K), 0.0029718 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 38.938: [GC 38.938: [DefNew: 70186K->368K(78656K), 0.0030463 secs] 237695K->167885K(253440K), 0.0030867 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 38.997: [GC 38.997: [DefNew: 70189K->368K(78656K), 0.0029194 secs] 237706K->167894K(253440K), 0.0029607 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 39.058: [GC 39.058: [DefNew: 70189K->368K(78656K), 0.0029645 secs] 237715K->167902K(253440K), 0.0030084 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 39.118: [GC 39.118: [DefNew: 70243K->369K(78656K), 0.0032374 secs] 237777K->167912K(253440K), 0.0032808 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 39.182: [GC 39.182: [DefNew: 70216K->368K(78656K), 0.0027334 secs] 237759K->167920K(253440K), 0.0027704 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 39.250: [GC 39.250: [DefNew: 70190K->368K(78656K), 0.0029092 secs] 237742K->167929K(253440K), 0.0029509 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 39.311: [GC 39.311: [DefNew: 70093K->369K(78656K), 0.0029875 secs] 237654K->167938K(253440K), 0.0030280 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 39.371: [GC 39.371: [DefNew: 70191K->369K(78656K), 0.0029701 secs] 237760K->167947K(253440K), 0.0030177 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 39.431: [GC 39.431: [DefNew: 70189K->369K(78656K), 0.0030578 secs] 237767K->167955K(253440K), 0.0030990 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 39.491: [GC 39.491: [DefNew: 70191K->369K(78656K), 0.0030288 secs] 237777K->167964K(253440K), 0.0030752 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 39.552: [GC 39.552: [DefNew: 70191K->369K(78656K), 0.0029730 secs] 237786K->167972K(253440K), 0.0030156 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 39.616: [GC 39.616: [DefNew: 70189K->369K(78656K), 0.0027559 secs] 237792K->167981K(253440K), 0.0027938 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 39.686: [GC 39.686: [DefNew: 70191K->369K(78656K), 0.0029564 secs] 237803K->167989K(253440K), 0.0029960 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 39.757: [GC 39.757: [DefNew: 70189K->369K(78656K), 0.0027108 secs] 237809K->167998K(253440K), 0.0027538 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 39.818: [GC 39.818: [DefNew: 70188K->369K(78656K), 0.0037074 secs] 237817K->168006K(253440K), 0.0037576 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 39.880: [GC 39.880: [DefNew: 70191K->369K(78656K), 0.0030092 secs] 237828K->168015K(253440K), 0.0030578 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 39.943: [GC 39.943: [DefNew: 70189K->369K(78656K), 0.0030518 secs] 237834K->168023K(253440K), 0.0030990 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 40.005: [GC 40.005: [DefNew: 70191K->369K(78656K), 0.0030995 secs] 237845K->168032K(253440K), 0.0031433 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 40.068: [GC 40.068: [DefNew: 70191K->368K(78656K), 0.0030595 secs] 237854K->168040K(253440K), 0.0031037 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 40.129: [GC 40.130: [DefNew: 70188K->368K(78656K), 0.0030633 secs] 237860K->168049K(253440K), 0.0031097 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 40.196: [GC 40.196: [DefNew: 70216K->368K(78656K), 0.0029560 secs] 237896K->168057K(253440K), 0.0030071 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 40.266: [GC 40.266: [DefNew: 70096K->368K(78656K), 0.0027768 secs] 237785K->168066K(253440K), 0.0028223 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 40.329: [GC 40.329: [DefNew: 70198K->368K(78656K), 0.0029632 secs] 237896K->168075K(253440K), 0.0030054 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 40.391: [GC 40.391: [DefNew: 70189K->368K(78656K), 0.0027960 secs] 237896K->168083K(253440K), 0.0028419 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 40.453: [GC 40.453: [DefNew: 70189K->368K(78656K), 0.0030837 secs] 237904K->168092K(253440K), 0.0031416 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 40.517: [GC 40.517: [DefNew: 70258K->439K(78656K), 0.0030203 secs] 237982K->168172K(253440K), 0.0030637 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 40.596: [GC 40.596: [DefNew: 70162K->581K(78656K), 0.0030880 secs] 237895K->168322K(253440K), 0.0031284 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 40.666: [GC 40.666: [DefNew: 70403K->581K(78656K), 0.0030961 secs] 238144K->168331K(253440K), 0.0031403 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] [FINE] ItemsDao.getImage() - Loading image BACKGROUND(id=13) [FINE] ItemsDao.getImage() - Loading image POSTER(id=13) 42.161: [GC 42.161: [DefNew: 70294K->6140K(78656K), 0.0157502 secs] 238044K->179973K(253440K), 0.0158060 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 42.230: [GC 42.230: [DefNew: 75850K->75850K(78656K), 0.0000175 secs]42.230: [Tenured: 173833K->174734K(174784K), 0.1042122 secs] 249684K->179972K(253440K), [Perm : 17126K->17126K(17152K)], 0.1043190 secs] [Times: user=0.09 sys=0.00, real=0.11 secs] 42.379: [Full GC 42.379: [Tenured: 174734K->173476K(174784K), 0.1745277 secs] 253047K->178289K(253440K), [Perm : 17126K->17126K(17152K)], 0.1745924 secs] [Times: user=0.17 sys=0.00, real=0.18 secs] 42.597: [Full GC 42.597: [Tenured: 174556K->174556K(174784K), 0.0899911 secs] 253024K->179018K(253440K), [Perm : 17126K->17126K(17152K)], 0.0900558 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 42.737: [Full GC 42.737: [Tenured: 174556K->174556K(174784K), 0.0914405 secs] 253032K->179387K(253440K), [Perm : 17126K->17126K(17152K)], 0.0915065 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 42.870: [Full GC 42.870: [Tenured: 174556K->174556K(174784K), 0.0859870 secs] 253043K->179396K(253440K), [Perm : 17126K->17126K(17152K)], 0.0860500 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 42.998: [Full GC 42.998: [Tenured: 174556K->173476K(174784K), 0.0873824 secs] 253050K->178325K(253440K), [Perm : 17126K->17126K(17152K)], 0.0874467 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 43.128: [Full GC 43.128: [Tenured: 174556K->174556K(174784K), 0.0861173 secs] 253062K->179055K(253440K), [Perm : 17126K->17126K(17152K)], 0.0861837 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 43.267: [Full GC 43.267: [Tenured: 174556K->174556K(174784K), 0.0887080 secs] 253087K->179423K(253440K), [Perm : 17126K->17126K(17152K)], 0.0887732 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 43.401: [Full GC 43.401: [Tenured: 174556K->174556K(174784K), 0.0862837 secs] 252860K->179431K(253440K), [Perm : 17126K->17126K(17152K)], 0.0863480 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 43.530: [Full GC 43.530: [Tenured: 174556K->173476K(174784K), 0.0863131 secs] 253082K->178360K(253440K), [Perm : 17126K->17126K(17152K)], 0.0863782 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 43.661: [Full GC 43.661: [Tenured: 174556K->174556K(174784K), 0.0861245 secs] 253101K->179091K(253440K), [Perm : 17126K->17126K(17152K)], 0.0861849 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 43.788: [Full GC 43.788: [Tenured: 174556K->174556K(174784K), 0.0881231 secs] 253102K->179459K(253440K), [Perm : 17126K->17126K(17152K)], 0.0881831 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 43.923: [Full GC 43.923: [Tenured: 174556K->174556K(174784K), 0.0863314 secs] 253107K->179467K(253440K), [Perm : 17126K->17126K(17152K)], 0.0863940 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 44.050: [Full GC 44.050: [Tenured: 174556K->173476K(174784K), 0.0884058 secs] 253115K->178395K(253440K), [Perm : 17126K->17126K(17152K)], 0.0884718 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 44.188: [Full GC 44.188: [Tenured: 174556K->174556K(174784K), 0.0919118 secs] 253158K->179126K(253440K), [Perm : 17126K->17126K(17152K)], 0.0919735 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 44.328: [Full GC 44.328: [Tenured: 174556K->174556K(174784K), 0.0864280 secs] 252934K->179498K(253440K), [Perm : 17126K->17126K(17152K)], 0.0864919 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 44.455: [Full GC 44.455: [Tenured: 174556K->174556K(174784K), 0.0876340 secs] 253149K->179506K(253440K), [Perm : 17126K->17126K(17152K)], 0.0876957 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 44.585: [Full GC 44.585: [Tenured: 174556K->173476K(174784K), 0.0861752 secs] 253152K->178433K(253440K), [Perm : 17126K->17126K(17152K)], 0.0862343 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 44.716: [Full GC 44.716: [Tenured: 174556K->174556K(174784K), 0.0866919 secs] 253163K->179162K(253440K), [Perm : 17126K->17126K(17152K)], 0.0867511 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 44.844: [Full GC 44.844: [Tenured: 174556K->174556K(174784K), 0.0885424 secs] 252816K->179530K(253440K), [Perm : 17126K->17126K(17152K)], 0.0886097 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 44.975: [Full GC 44.975: [Tenured: 174556K->174556K(174784K), 0.0862352 secs] 252830K->179537K(253440K), [Perm : 17126K->17126K(17152K)], 0.0862995 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 45.103: [Full GC 45.103: [Tenured: 174556K->173476K(174784K), 0.0920025 secs] 252831K->178465K(253440K), [Perm : 17126K->17126K(17152K)], 0.0920612 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 45.251: [Full GC 45.251: [Tenured: 174556K->174556K(174784K), 0.0989766 secs] 252873K->179197K(253440K), [Perm : 17126K->17126K(17152K)], 0.0990379 secs] [Times: user=0.11 sys=0.00, real=0.10 secs] 45.397: [Full GC 45.397: [Tenured: 174556K->174556K(174784K), 0.0865685 secs] 253012K->179571K(253440K), [Perm : 17126K->17126K(17152K)], 0.0866298 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 45.525: [Full GC 45.525: [Tenured: 174556K->174556K(174784K), 0.0886220 secs] 252861K->179579K(253440K), [Perm : 17126K->17126K(17152K)], 0.0886842 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 45.655: [Full GC 45.655: [Tenured: 174556K->173476K(174784K), 0.0998046 secs] 252871K->178507K(253440K), [Perm : 17126K->17126K(17152K)], 0.0998608 secs] [Times: user=0.11 sys=0.00, real=0.10 secs] 45.805: [Full GC 45.805: [Tenured: 174556K->174556K(174784K), 0.0881938 secs] 252876K->179235K(253440K), [Perm : 17126K->17126K(17152K)], 0.0882572 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 45.940: [Full GC 45.940: [Tenured: 174556K->174556K(174784K), 0.0867792 secs] 252884K->179603K(253440K), [Perm : 17126K->17126K(17152K)], 0.0868460 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 46.068: [Full GC 46.068: [Tenured: 174556K->174556K(174784K), 0.0881363 secs] 252895K->179611K(253440K), [Perm : 17126K->17126K(17152K)], 0.0881997 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 46.202: [Full GC 46.202: [Tenured: 174556K->173476K(174784K), 0.0980648 secs] 252923K->178538K(253440K), [Perm : 17126K->17126K(17152K)], 0.0981278 secs] [Times: user=0.09 sys=0.00, real=0.10 secs] 46.345: [Full GC 46.345: [Tenured: 174556K->174556K(174784K), 0.0867924 secs] 253055K->179267K(253440K), [Perm : 17126K->17126K(17152K)], 0.0868533 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 46.473: [Full GC 46.473: [Tenured: 174556K->174556K(174784K), 0.0881159 secs] 252915K->179635K(253440K), [Perm : 17126K->17126K(17152K)], 0.0881734 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 46.606: [Full GC 46.606: [Tenured: 174556K->174556K(174784K), 0.0865966 secs] 252921K->179642K(253440K), [Perm : 17126K->17126K(17152K)], 0.0866562 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 46.734: [Full GC 46.734: [Tenured: 174556K->173476K(174784K), 0.0875314 secs] 252931K->178570K(253440K), [Perm : 17126K->17126K(17152K)], 0.0875936 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 46.866: [Full GC 46.866: [Tenured: 174556K->174556K(174784K), 0.0869499 secs] 252955K->179303K(253440K), [Perm : 17126K->17126K(17152K)], 0.0870104 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 46.994: [Full GC 46.994: [Tenured: 174556K->174556K(174784K), 0.0891009 secs] 252978K->179681K(253440K), [Perm : 17126K->17126K(17152K)], 0.0891639 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 47.128: [Full GC 47.128: [Tenured: 174556K->174556K(174784K), 0.0872398 secs] 252970K->179688K(253440K), [Perm : 17126K->17126K(17152K)], 0.0872994 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 47.268: [Full GC 47.268: [Tenured: 174556K->173476K(174784K), 0.0885412 secs] 253004K->178616K(253440K), [Perm : 17126K->17126K(17152K)], 0.0886054 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 47.402: [Full GC 47.402: [Tenured: 174556K->174556K(174784K), 0.0872143 secs] 253132K->179344K(253440K), [Perm : 17126K->17126K(17152K)], 0.0872768 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 47.531: [Full GC 47.531: [Tenured: 174556K->174556K(174784K), 0.0869022 secs] 252994K->179712K(253440K), [Perm : 17126K->17126K(17152K)], 0.0869631 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 47.660: [Full GC 47.660: [Tenured: 174556K->174556K(174784K), 0.0888821 secs] 253002K->179720K(253440K), [Perm : 17126K->17126K(17152K)], 0.0889422 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 47.789: [Full GC 47.789: [Tenured: 174556K->173476K(174784K), 0.0886084 secs] 253010K->178648K(253440K), [Perm : 17126K->17126K(17152K)], 0.0886684 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 47.922: [Full GC 47.922: [Tenured: 174556K->174556K(174784K), 0.0868848 secs] 253018K->179376K(253440K), [Perm : 17126K->17126K(17152K)], 0.0869469 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 48.051: [Full GC 48.051: [Tenured: 174556K->174556K(174784K), 0.0885267 secs] 253026K->179744K(253440K), [Perm : 17126K->17126K(17152K)], 0.0885880 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 48.183: [Full GC 48.183: [Tenured: 174556K->174556K(174784K), 0.0879575 secs] 253031K->179752K(253440K), [Perm : 17126K->17126K(17152K)], 0.0880197 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 48.324: [Full GC 48.324: [Tenured: 174556K->173476K(174784K), 0.0901779 secs] 253067K->178680K(253440K), [Perm : 17126K->17126K(17152K)], 0.0902371 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 48.461: [Full GC 48.461: [Tenured: 174556K->174556K(174784K), 0.0871206 secs] 252835K->179408K(253440K), [Perm : 17126K->17126K(17152K)], 0.0871789 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 48.591: [Full GC 48.591: [Tenured: 174556K->174556K(174784K), 0.0887463 secs] 253057K->179776K(253440K), [Perm : 17126K->17126K(17152K)], 0.0888081 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 48.732: [Full GC 48.732: [Tenured: 174556K->174556K(174784K), 0.0948912 secs] 253065K->179784K(253440K), [Perm : 17126K->17126K(17152K)], 0.0949449 secs] [Times: user=0.09 sys=0.00, real=0.10 secs] 48.879: [Full GC 48.879: [Tenured: 174556K->173476K(174784K), 0.0880231 secs] 253069K->178712K(253440K), [Perm : 17126K->17126K(17152K)], 0.0880835 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 49.008: [Full GC 49.008: [Tenured: 174556K->174556K(174784K), 0.0896118 secs] 253084K->179440K(253440K), [Perm : 17126K->17126K(17152K)], 0.0896705 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 49.141: [Full GC 49.141: [Tenured: 174556K->174556K(174784K), 0.0882479 secs] 253111K->179815K(253440K), [Perm : 17126K->17126K(17152K)], 0.0883087 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 49.282: [Full GC 49.282: [Tenured: 174556K->174556K(174784K), 0.0904159 secs] 253170K->179837K(253440K), [Perm : 17126K->17126K(17152K)], 0.0904729 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 49.416: [Full GC 49.416: [Tenured: 174556K->173476K(174784K), 0.0869669 secs] 252913K->178765K(253440K), [Perm : 17126K->17126K(17152K)], 0.0870295 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 49.544: [Full GC 49.545: [Tenured: 174556K->174556K(174784K), 0.0880363 secs] 253138K->179493K(253440K), [Perm : 17126K->17126K(17152K)], 0.0880963 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 49.676: [Full GC 49.676: [Tenured: 174556K->174556K(174784K), 0.0874433 secs] 253144K->179861K(253440K), [Perm : 17126K->17126K(17152K)], 0.0875037 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 49.804: [Full GC 49.804: [Tenured: 174556K->174556K(174784K), 0.0893210 secs] 253149K->179869K(253440K), [Perm : 17126K->17126K(17152K)], 0.0893806 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 49.939: [Full GC 49.939: [Tenured: 174556K->173476K(174784K), 0.0886472 secs] 253155K->178797K(253440K), [Perm : 17126K->17126K(17152K)], 0.0887072 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 50.070: [Full GC 50.070: [Tenured: 174556K->174556K(174784K), 0.0883530 secs] 253166K->179525K(253440K), [Perm : 17126K->17126K(17152K)], 0.0884122 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 50.206: [Full GC 50.206: [Tenured: 174556K->174556K(174784K), 0.0958435 secs] 253193K->179893K(253440K), [Perm : 17126K->17126K(17152K)], 0.0959005 secs] [Times: user=0.09 sys=0.00, real=0.10 secs] 50.345: [Full GC 50.345: [Tenured: 174556K->174556K(174784K), 0.0881287 secs] 253050K->179901K(253440K), [Perm : 17126K->17126K(17152K)], 0.0881917 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 50.476: [Full GC 50.476: [Tenured: 174556K->173476K(174784K), 0.0882666 secs] 252898K->178829K(253440K), [Perm : 17126K->17126K(17152K)], 0.0883351 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 50.606: [Full GC 50.606: [Tenured: 174556K->174556K(174784K), 0.0900715 secs] 252878K->179557K(253440K), [Perm : 17126K->17126K(17152K)], 0.0901320 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 50.740: [Full GC 50.740: [Tenured: 174556K->174556K(174784K), 0.0873462 secs] 252850K->179925K(253440K), [Perm : 17126K->17126K(17152K)], 0.0874109 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 50.868: [Full GC 50.868: [Tenured: 174556K->174556K(174784K), 0.0892129 secs] 252877K->179932K(253440K), [Perm : 17126K->17126K(17152K)], 0.0892738 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 51.000: [Full GC 51.001: [Tenured: 174556K->173476K(174784K), 0.0878375 secs] 252860K->178860K(253440K), [Perm : 17126K->17126K(17152K)], 0.0878988 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 51.129: [Full GC 51.129: [Tenured: 174556K->174556K(174784K), 0.0909204 secs] 252870K->179588K(253440K), [Perm : 17126K->17126K(17152K)], 0.0909799 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 51.274: [Full GC 51.274: [Tenured: 174556K->174556K(174784K), 0.0900792 secs] 252907K->179956K(253440K), [Perm : 17126K->17126K(17152K)], 0.0901367 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 51.409: [Full GC 51.409: [Tenured: 174556K->174556K(174784K), 0.0883053 secs] 253032K->179964K(253440K), [Perm : 17126K->17126K(17152K)], 0.0883709 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 51.538: [Full GC 51.538: [Tenured: 174556K->173476K(174784K), 0.0891163 secs] 252898K->178892K(253440K), [Perm : 17126K->17126K(17152K)], 0.0891793 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 51.670: [Full GC 51.670: [Tenured: 174556K->174556K(174784K), 0.0935005 secs] 252907K->179620K(253440K), [Perm : 17126K->17126K(17152K)], 0.0935592 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 51.813: [Full GC 51.813: [Tenured: 174556K->174556K(174784K), 0.0936682 secs] 252911K->179988K(253440K), [Perm : 17126K->17126K(17152K)], 0.0937299 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 51.947: [Full GC 51.948: [Tenured: 174556K->174556K(174784K), 0.0895292 secs] 252916K->179996K(253440K), [Perm : 17126K->17126K(17152K)], 0.0895896 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 52.080: [Full GC 52.080: [Tenured: 174556K->173476K(174784K), 0.0900149 secs] 252921K->178924K(253440K), [Perm : 17126K->17126K(17152K)], 0.0900762 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 52.217: [Full GC 52.217: [Tenured: 174556K->174556K(174784K), 0.0997658 secs] 252961K->179652K(253440K), [Perm : 17126K->17126K(17152K)], 0.0998229 secs] [Times: user=0.06 sys=0.00, real=0.10 secs] 52.361: [Full GC 52.361: [Tenured: 174556K->174556K(174784K), 0.0880989 secs] 253086K->180020K(253440K), [Perm : 17126K->17126K(17152K)], 0.0881644 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 52.489: [Full GC 52.489: [Tenured: 174556K->174556K(174784K), 0.0898940 secs] 252947K->180028K(253440K), [Perm : 17126K->17126K(17152K)], 0.0899583 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 52.628: [Full GC 52.628: [Tenured: 174556K->173476K(174784K), 0.0898459 secs] 252952K->178955K(253440K), [Perm : 17126K->17126K(17152K)], 0.0899051 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 52.759: [Full GC 52.759: [Tenured: 174556K->174556K(174784K), 0.0896897 secs] 252995K->179694K(253440K), [Perm : 17126K->17126K(17152K)], 0.0897497 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 52.892: [Full GC 52.892: [Tenured: 174556K->174556K(174784K), 0.0895705 secs] 253047K->180083K(253440K), [Perm : 17126K->17126K(17152K)], 0.0896297 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 53.026: [Full GC 53.026: [Tenured: 174556K->174556K(174784K), 0.0895854 secs] 253008K->180091K(253440K), [Perm : 17126K->17126K(17152K)], 0.0896458 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 53.157: [Full GC 53.157: [Tenured: 174556K->173476K(174784K), 0.0899327 secs] 253019K->179019K(253440K), [Perm : 17126K->17126K(17152K)], 0.0899915 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 53.301: [Full GC 53.301: [Tenured: 174556K->174556K(174784K), 0.0918271 secs] 253053K->179747K(253440K), [Perm : 17126K->17126K(17152K)], 0.0918888 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 53.438: [Full GC 53.438: [Tenured: 174556K->174556K(174784K), 0.0884794 secs] 252817K->180115K(253440K), [Perm : 17126K->17126K(17152K)], 0.0885395 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 53.566: [Full GC 53.566: [Tenured: 174556K->174556K(174784K), 0.0901962 secs] 253042K->180123K(253440K), [Perm : 17126K->17126K(17152K)], 0.0902580 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 53.700: [Full GC 53.700: [Tenured: 174556K->173476K(174784K), 0.0884675 secs] 253050K->179050K(253440K), [Perm : 17126K->17126K(17152K)], 0.0885254 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 53.829: [Full GC 53.829: [Tenured: 174556K->174556K(174784K), 0.0898476 secs] 253058K->179779K(253440K), [Perm : 17126K->17126K(17152K)], 0.0899191 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 53.962: [Full GC 53.962: [Tenured: 174556K->174556K(174784K), 0.0883287 secs] 253066K->180146K(253440K), [Perm : 17126K->17126K(17152K)], 0.0883879 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 54.091: [Full GC 54.091: [Tenured: 174556K->174556K(174784K), 0.0900958 secs] 253074K->180154K(253440K), [Perm : 17126K->17126K(17152K)], 0.0901558 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 54.232: [Full GC 54.233: [Tenured: 174556K->173476K(174784K), 0.0932698 secs] 253108K->179082K(253440K), [Perm : 17126K->17126K(17152K)], 0.0933319 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 54.372: [Full GC 54.372: [Tenured: 174556K->174556K(174784K), 0.0883687 secs] 252876K->179810K(253440K), [Perm : 17126K->17126K(17152K)], 0.0884309 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 54.501: [Full GC 54.501: [Tenured: 174556K->174556K(174784K), 0.0901145 secs] 253098K->180178K(253440K), [Perm : 17126K->17126K(17152K)], 0.0901784 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 54.634: [Full GC 54.634: [Tenured: 174556K->174556K(174784K), 0.0899417 secs] 253106K->180186K(253440K), [Perm : 17126K->17126K(17152K)], 0.0900451 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 54.770: [Full GC 54.770: [Tenured: 174556K->173476K(174784K), 0.0931237 secs] 253114K->179114K(253440K), [Perm : 17126K->17126K(17152K)], 0.0931833 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 54.917: [Full GC 54.917: [Tenured: 174556K->174556K(174784K), 0.0901571 secs] 253122K->179842K(253440K), [Perm : 17126K->17126K(17152K)], 0.0902205 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 55.047: [Full GC 55.047: [Tenured: 174556K->174556K(174784K), 0.0883432 secs] 253130K->180210K(253440K), [Perm : 17126K->17126K(17152K)], 0.0884032 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 55.178: [Full GC 55.178: [Tenured: 174556K->174556K(174784K), 0.0895381 secs] 253134K->180218K(253440K), [Perm : 17126K->17126K(17152K)], 0.0895969 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 55.320: [Full GC 55.321: [Tenured: 174556K->173476K(174784K), 0.0933855 secs] 253169K->179145K(253440K), [Perm : 17126K->17126K(17152K)], 0.0934434 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 55.461: [Full GC 55.461: [Tenured: 174556K->174556K(174784K), 0.0890277 secs] 252951K->179874K(253440K), [Perm : 17126K->17126K(17152K)], 0.0890886 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 55.590: [Full GC 55.590: [Tenured: 174556K->174556K(174784K), 0.0905760 secs] 253158K->180242K(253440K), [Perm : 17126K->17126K(17152K)], 0.0906432 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 55.724: [Full GC 55.724: [Tenured: 174556K->174556K(174784K), 0.0889413 secs] 252827K->180249K(253440K), [Perm : 17126K->17126K(17152K)], 0.0890039 secs] [Times: user=0.06 sys=0.00, real=0.09 secs] 55.853: [Full GC 55.853: [Tenured: 174556K->173476K(174784K), 0.0908757 secs] 252825K->179177K(253440K), [Perm : 17126K->17126K(17152K)], 0.0909340 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 55.987: [Full GC 55.988: [Tenured: 174556K->174556K(174784K), 0.0883777 secs] 252828K->179905K(253440K), [Perm : 17126K->17126K(17152K)], 0.0884373 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 56.117: [Full GC 56.118: [Tenured: 174556K->174556K(174784K), 0.0906547 secs] 252837K->180273K(253440K), [Perm : 17126K->17126K(17152K)], 0.0907156 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 56.261: [Full GC 56.261: [Tenured: 174556K->174556K(174784K), 0.0951849 secs] 252868K->180281K(253440K), [Perm : 17126K->17126K(17152K)], 0.0952411 secs] [Times: user=0.09 sys=0.00, real=0.10 secs] 56.401: [Full GC 56.401: [Tenured: 174556K->173476K(174784K), 0.0889937 secs] 252994K->179209K(253440K), [Perm : 17126K->17126K(17152K)], 0.0890545 secs] [Times: user=0.08 sys=0.02, real=0.09 secs] 56.531: [Full GC 56.531: [Tenured: 174556K->174556K(174784K), 0.0896207 secs] 252859K->179937K(253440K), [Perm : 17126K->17126K(17152K)], 0.0896820 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 56.663: [Full GC 56.663: [Tenured: 174556K->174556K(174784K), 0.0889907 secs] 252862K->180305K(253440K), [Perm : 17126K->17126K(17152K)], 0.0890528 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 56.793: [Full GC 56.793: [Tenured: 174556K->174556K(174784K), 0.0901771 secs] 252868K->180312K(253440K), [Perm : 17126K->17126K(17152K)], 0.0902371 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 56.926: [Full GC 56.926: [Tenured: 174556K->173476K(174784K), 0.0889822 secs] 252878K->179240K(253440K), [Perm : 17126K->17126K(17152K)], 0.0890422 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 57.057: [Full GC 57.057: [Tenured: 174556K->174556K(174784K), 0.0898834 secs] 252889K->179968K(253440K), [Perm : 17126K->17126K(17152K)], 0.0899540 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 57.189: [Full GC 57.189: [Tenured: 174556K->174556K(174784K), 0.0891997 secs] 252894K->180336K(253440K), [Perm : 17126K->17126K(17152K)], 0.0892606 secs] [Times: user=0.06 sys=0.00, real=0.09 secs] 57.331: [Full GC 57.331: [Tenured: 174556K->174556K(174784K), 0.0932842 secs] 252928K->180344K(253440K), [Perm : 17126K->17126K(17152K)], 0.0933506 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 57.471: [Full GC 57.471: [Tenured: 174556K->173476K(174784K), 0.0884620 secs] 253054K->179272K(253440K), [Perm : 17126K->17126K(17152K)], 0.0885237 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 57.602: [Full GC 57.602: [Tenured: 174556K->174556K(174784K), 0.0907343 secs] 252920K->180000K(253440K), [Perm : 17126K->17126K(17152K)], 0.0907931 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 57.734: [Full GC 57.734: [Tenured: 174556K->174556K(174784K), 0.0952288 secs] 252928K->180368K(253440K), [Perm : 17126K->17126K(17152K)], 0.0952854 secs] [Times: user=0.09 sys=0.00, real=0.10 secs] 57.878: [Full GC 57.878: [Tenured: 174556K->174556K(174784K), 0.0964033 secs] 252932K->180376K(253440K), [Perm : 17126K->17126K(17152K)], 0.0964629 secs] [Times: user=0.09 sys=0.00, real=0.10 secs] 58.017: [Full GC 58.017: [Tenured: 174556K->173476K(174784K), 0.0907164 secs] 252943K->179303K(253440K), [Perm : 17126K->17126K(17152K)], 0.0907756 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 58.150: [Full GC 58.150: [Tenured: 174556K->174556K(174784K), 0.0905870 secs] 252998K->180047K(253440K), [Perm : 17126K->17126K(17152K)], 0.0906479 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 58.293: [Full GC 58.293: [Tenured: 174556K->174556K(174784K), 0.0943902 secs] 253093K->180447K(253440K), [Perm : 17126K->17126K(17152K)], 0.0944510 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 58.432: [Full GC 58.432: [Tenured: 174556K->174556K(174784K), 0.0904287 secs] 253157K->180455K(253440K), [Perm : 17126K->17126K(17152K)], 0.0904887 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 58.563: [Full GC 58.563: [Tenured: 174556K->173476K(174784K), 0.0893687 secs] 253027K->179382K(253440K), [Perm : 17126K->17126K(17152K)], 0.0894296 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 58.695: [Full GC 58.695: [Tenured: 174556K->174556K(174784K), 0.0896399 secs] 253035K->180110K(253440K), [Perm : 17126K->17126K(17152K)], 0.0896952 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 58.825: [Full GC 58.825: [Tenured: 174556K->174556K(174784K), 0.0909876 secs] 253036K->180478K(253440K), [Perm : 17126K->17126K(17152K)], 0.0910468 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 58.959: [Full GC 58.959: [Tenured: 174556K->174556K(174784K), 0.0898004 secs] 253045K->180486K(253440K), [Perm : 17126K->17126K(17152K)], 0.0898587 secs] [Times: user=0.08 sys=0.02, real=0.09 secs] 59.089: [Full GC 59.089: [Tenured: 174556K->173476K(174784K), 0.0913388 secs] 253051K->179414K(253440K), [Perm : 17126K->17126K(17152K)], 0.0913997 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 59.232: [Full GC 59.232: [Tenured: 174556K->174556K(174784K), 0.0946962 secs] 253091K->180142K(253440K), [Perm : 17126K->17126K(17152K)], 0.0947593 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 59.372: [Full GC 59.372: [Tenured: 174556K->174556K(174784K), 0.0896335 secs] 252852K->180510K(253440K), [Perm : 17126K->17126K(17152K)], 0.0896944 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 59.502: [Full GC 59.502: [Tenured: 174556K->174556K(174784K), 0.0929705 secs] 253074K->180518K(253440K), [Perm : 17126K->17126K(17152K)], 0.0930309 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 59.641: [Full GC 59.641: [Tenured: 174556K->173476K(174784K), 0.0902052 secs] 253081K->179445K(253440K), [Perm : 17126K->17126K(17152K)], 0.0902669 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 59.776: [Full GC 59.776: [Tenured: 174556K->174556K(174784K), 0.0892014 secs] 253095K->180174K(253440K), [Perm : 17126K->17126K(17152K)], 0.0892610 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 59.909: [Full GC 59.909: [Tenured: 174556K->174556K(174784K), 0.0904351 secs] 253103K->180541K(253440K), [Perm : 17126K->17126K(17152K)], 0.0904955 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 60.044: [Full GC 60.044: [Tenured: 174556K->174556K(174784K), 0.0897327 secs] 253106K->180549K(253440K), [Perm : 17126K->17126K(17152K)], 0.0897940 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 60.176: [Full GC 60.176: [Tenured: 174556K->173476K(174784K), 0.0901315 secs] 253113K->179477K(253440K), [Perm : 17126K->17126K(17152K)], 0.0901924 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 60.319: [Full GC 60.319: [Tenured: 174556K->174556K(174784K), 0.0943753 secs] 253149K->180205K(253440K), [Perm : 17126K->17126K(17152K)], 0.0944327 secs] [Times: user=0.09 sys=0.00, real=0.10 secs] 60.459: [Full GC 60.460: [Tenured: 174556K->174556K(174784K), 0.0916364 secs] 252930K->180573K(253440K), [Perm : 17126K->17126K(17152K)], 0.0916972 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 60.591: [Full GC 60.591: [Tenured: 174556K->174556K(174784K), 0.0895424 secs] 253140K->180581K(253440K), [Perm : 17126K->17126K(17152K)], 0.0896037 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 60.724: [Full GC 60.724: [Tenured: 174556K->173476K(174784K), 0.0918407 secs] 253148K->179508K(253440K), [Perm : 17126K->17126K(17152K)], 0.0918994 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 60.867: [Full GC 60.867: [Tenured: 174556K->174556K(174784K), 0.0970972 secs] 253155K->180237K(253440K), [Perm : 17126K->17126K(17152K)], 0.0971568 secs] [Times: user=0.11 sys=0.00, real=0.10 secs] 61.005: [Full GC 61.005: [Tenured: 174556K->174556K(174784K), 0.0921391 secs] 253161K->180604K(253440K), [Perm : 17126K->17126K(17152K)], 0.0922000 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 61.140: [Full GC 61.140: [Tenured: 174556K->174556K(174784K), 0.0965182 secs] 252814K->180612K(253440K), [Perm : 17126K->17126K(17152K)], 0.0965774 secs] [Times: user=0.09 sys=0.00, real=0.10 secs] 61.287: [Full GC 61.287: [Tenured: 174556K->173476K(174784K), 0.0963092 secs] 252843K->179540K(253440K), [Perm : 17126K->17126K(17152K)], 0.0963739 secs] [Times: user=0.11 sys=0.00, real=0.10 secs] 61.431: [Full GC 61.431: [Tenured: 174556K->174556K(174784K), 0.0904525 secs] 252975K->180268K(253440K), [Perm : 17126K->17126K(17152K)], 0.0905189 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 61.564: [Full GC 61.564: [Tenured: 174556K->174556K(174784K), 0.0903793 secs] 252835K->180636K(253440K), [Perm : 17126K->17126K(17152K)], 0.0904380 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 61.694: [Full GC 61.694: [Tenured: 174556K->174556K(174784K), 0.0897335 secs] 252841K->180644K(253440K), [Perm : 17126K->17126K(17152K)], 0.0897901 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 61.823: [Full GC 61.823: [Tenured: 174556K->173476K(174784K), 0.0920310 secs] 252848K->179571K(253440K), [Perm : 17126K->17126K(17152K)], 0.0920914 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 61.958: [Full GC 61.958: [Tenured: 174556K->174556K(174784K), 0.0909685 secs] 252861K->180299K(253440K), [Perm : 17126K->17126K(17152K)], 0.0910276 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 62.090: [Full GC 62.090: [Tenured: 174556K->174556K(174784K), 0.0916538 secs] 252864K->180667K(253440K), [Perm : 17126K->17126K(17152K)], 0.0917138 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 62.235: [Full GC 62.235: [Tenured: 174556K->174556K(174784K), 0.1001822 secs] 252899K->180675K(253440K), [Perm : 17126K->17126K(17152K)], 0.1002447 secs] [Times: user=0.08 sys=0.00, real=0.10 secs] 62.381: [Full GC 62.381: [Tenured: 174556K->173476K(174784K), 0.0906675 secs] 253023K->179603K(253440K), [Perm : 17126K->17126K(17152K)], 0.0907365 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 62.513: [Full GC 62.513: [Tenured: 174556K->174556K(174784K), 0.0908786 secs] 252890K->180331K(253440K), [Perm : 17126K->17126K(17152K)], 0.0909374 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 62.648: [Full GC 62.648: [Tenured: 174556K->174556K(174784K), 0.0905368 secs] 253034K->180877K(253440K), [Perm : 17126K->17126K(17152K)], 0.0905981 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 62.778: [Full GC 62.778: [Tenured: 174556K->174556K(174784K), 0.0918905 secs] 252837K->180882K(253440K), [Perm : 17126K->17126K(17152K)], 0.0919510 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 62.913: [Full GC 62.913: [Tenured: 174556K->173476K(174784K), 0.0920999 secs] 252845K->179807K(253440K), [Perm : 17126K->17126K(17152K)], 0.0921595 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 63.050: [Full GC 63.050: [Tenured: 174556K->174556K(174784K), 0.0902512 secs] 252848K->180353K(253440K), [Perm : 17126K->17126K(17152K)], 0.0903116 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 63.182: [Full GC 63.182: [Tenured: 174556K->174556K(174784K), 0.0911677 secs] 252854K->180898K(253440K), [Perm : 17126K->17126K(17152K)], 0.0912277 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 63.326: [Full GC 63.326: [Tenured: 174556K->174556K(174784K), 0.0941403 secs] 252886K->180903K(253440K), [Perm : 17126K->17126K(17152K)], 0.0942016 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 63.467: [Full GC 63.467: [Tenured: 174556K->173476K(174784K), 0.0917696 secs] 253009K->179828K(253440K), [Perm : 17126K->17126K(17152K)], 0.0918296 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 63.599: [Full GC 63.599: [Tenured: 174556K->174556K(174784K), 0.0914197 secs] 252872K->180373K(253440K), [Perm : 17126K->17126K(17152K)], 0.0914780 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 63.734: [Full GC 63.734: [Tenured: 174556K->174556K(174784K), 0.0911055 secs] 252876K->180919K(253440K), [Perm : 17126K->17126K(17152K)], 0.0911630 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 63.876: [Full GC 63.876: [Tenured: 174556K->174556K(174784K), 0.0954736 secs] 252879K->180924K(253440K), [Perm : 17126K->17126K(17152K)], 0.0955336 secs] [Times: user=0.08 sys=0.00, real=0.10 secs] 64.019: [Full GC 64.019: [Tenured: 174556K->173476K(174784K), 0.0922123 secs] 252884K->179849K(253440K), [Perm : 17126K->17126K(17152K)], 0.0922698 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 64.152: [Full GC 64.152: [Tenured: 174556K->174556K(174784K), 0.0924269 secs] 252892K->180394K(253440K), [Perm : 17126K->17126K(17152K)], 0.0924861 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 64.296: [Full GC 64.296: [Tenured: 174556K->174556K(174784K), 0.0950108 secs] 252923K->180940K(253440K), [Perm : 17126K->17126K(17152K)], 0.0951735 secs] [Times: user=0.09 sys=0.00, real=0.10 secs] 64.436: [Full GC 64.436: [Tenured: 174556K->174556K(174784K), 0.0911260 secs] 253047K->180945K(253440K), [Perm : 17126K->17126K(17152K)], 0.0911868 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 64.567: [Full GC 64.567: [Tenured: 174556K->173476K(174784K), 0.0917590 secs] 252908K->179870K(253440K), [Perm : 17126K->17126K(17152K)], 0.0918173 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 64.702: [Full GC 64.702: [Tenured: 174556K->174556K(174784K), 0.0903950 secs] 252912K->180415K(253440K), [Perm : 17126K->17126K(17152K)], 0.0904538 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 64.832: [Full GC 64.832: [Tenured: 174556K->174556K(174784K), 0.0925261 secs] 252915K->180961K(253440K), [Perm : 17126K->17126K(17152K)], 0.0925878 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 64.967: [Full GC 64.967: [Tenured: 174556K->174556K(174784K), 0.0908791 secs] 252923K->180966K(253440K), [Perm : 17126K->17126K(17152K)], 0.0909416 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 65.097: [Full GC 65.097: [Tenured: 174556K->173476K(174784K), 0.0925631 secs] 252926K->179891K(253440K), [Perm : 17126K->17126K(17152K)], 0.0926253 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 65.242: [Full GC 65.243: [Tenured: 174556K->174556K(174784K), 0.0933340 secs] 252960K->180436K(253440K), [Perm : 17126K->17126K(17152K)], 0.0934051 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 65.382: [Full GC 65.382: [Tenured: 174556K->174556K(174784K), 0.0920063 secs] 253086K->180982K(253440K), [Perm : 17126K->17126K(17152K)], 0.0920684 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 65.514: [Full GC 65.514: [Tenured: 174556K->174556K(174784K), 0.0909693 secs] 252942K->180987K(253440K), [Perm : 17126K->17126K(17152K)], 0.0910281 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 65.645: [Full GC 65.645: [Tenured: 174556K->173476K(174784K), 0.0917990 secs] 252947K->179912K(253440K), [Perm : 17126K->17126K(17152K)], 0.0918594 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 65.779: [Full GC 65.779: [Tenured: 174556K->174556K(174784K), 0.0908501 secs] 252955K->180457K(253440K), [Perm : 17126K->17126K(17152K)], 0.0909093 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 65.912: [Full GC 65.912: [Tenured: 174556K->174556K(174784K), 0.0923213 secs] 252960K->181002K(253440K), [Perm : 17126K->17126K(17152K)], 0.0923818 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 66.046: [Full GC 66.046: [Tenured: 174556K->174556K(174784K), 0.0919237 secs] 252963K->181008K(253440K), [Perm : 17126K->17126K(17152K)], 0.0919833 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 66.178: [Full GC 66.178: [Tenured: 174556K->173476K(174784K), 0.0936652 secs] 252969K->179933K(253440K), [Perm : 17126K->17126K(17152K)], 0.0937231 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 66.324: [Full GC 66.324: [Tenured: 174556K->174556K(174784K), 0.0968179 secs] 253002K->180478K(253440K), [Perm : 17126K->17126K(17152K)], 0.0968796 secs] [Times: user=0.08 sys=0.00, real=0.10 secs] 66.465: [Full GC 66.465: [Tenured: 174556K->174556K(174784K), 0.0921761 secs] 253124K->181023K(253440K), [Perm : 17126K->17126K(17152K)], 0.0922430 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 66.597: [Full GC 66.597: [Tenured: 174556K->174556K(174784K), 0.0907799 secs] 252986K->181029K(253440K), [Perm : 17126K->17126K(17152K)], 0.0908497 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 66.728: [Full GC 66.728: [Tenured: 174556K->173476K(174784K), 0.0921974 secs] 252989K->179954K(253440K), [Perm : 17126K->17126K(17152K)], 0.0922553 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 66.869: [Full GC 66.869: [Tenured: 174556K->174556K(174784K), 0.0964505 secs] 252997K->180499K(253440K), [Perm : 17126K->17126K(17152K)], 0.0965135 secs] [Times: user=0.08 sys=0.00, real=0.10 secs] 67.015: [Full GC 67.015: [Tenured: 174556K->174556K(174784K), 0.0918948 secs] 253001K->181044K(253440K), [Perm : 17126K->17126K(17152K)], 0.0919514 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 67.150: [Full GC 67.150: [Tenured: 174556K->174556K(174784K), 0.0919441 secs] 253004K->181049K(253440K), [Perm : 17126K->17126K(17152K)], 0.0920033 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 67.292: [Full GC 67.292: [Tenured: 174556K->173476K(174784K), 0.0948550 secs] 253035K->179975K(253440K), [Perm : 17126K->17126K(17152K)], 0.0949159 secs] [Times: user=0.09 sys=0.00, real=0.10 secs] 67.443: [Full GC 67.443: [Tenured: 174556K->174556K(174784K), 0.0916453 secs] 253162K->180520K(253440K), [Perm : 17126K->17126K(17152K)], 0.0917058 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 67.577: [Full GC 67.577: [Tenured: 174556K->174556K(174784K), 0.0921421 secs] 253025K->181065K(253440K), [Perm : 17126K->17126K(17152K)], 0.0922017 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 67.709: [Full GC 67.709: [Tenured: 174556K->174556K(174784K), 0.0907790 secs] 253024K->181070K(253440K), [Perm : 17126K->17126K(17152K)], 0.0908382 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 67.839: [Full GC 67.839: [Tenured: 174556K->173476K(174784K), 0.0936580 secs] 253032K->179995K(253440K), [Perm : 17126K->17126K(17152K)], 0.0937189 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 67.976: [Full GC 67.976: [Tenured: 174556K->174556K(174784K), 0.0917334 secs] 253037K->180541K(253440K), [Perm : 17126K->17126K(17152K)], 0.0917917 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 68.110: [Full GC 68.110: [Tenured: 174556K->174556K(174784K), 0.0934758 secs] 253041K->181086K(253440K), [Perm : 17126K->17126K(17152K)], 0.0935379 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 68.258: [Full GC 68.258: [Tenured: 174556K->174556K(174784K), 0.0952586 secs] 253145K->181115K(253440K), [Perm : 17126K->17126K(17152K)], 0.0953199 secs] [Times: user=0.09 sys=0.00, real=0.10 secs] 68.400: [Full GC 68.400: [Tenured: 174556K->173476K(174784K), 0.0922919 secs] 252823K->180087K(253440K), [Perm : 17126K->17126K(17152K)], 0.0923545 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 68.534: [Full GC 68.534: [Tenured: 174556K->174556K(174784K), 0.0922076 secs] 253131K->180633K(253440K), [Perm : 17126K->17126K(17152K)], 0.0922668 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 68.669: [Full GC 68.669: [Tenured: 174556K->174556K(174784K), 0.0913465 secs] 253132K->181178K(253440K), [Perm : 17126K->17126K(17152K)], 0.0914090 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 68.803: [Full GC 68.803: [Tenured: 174556K->174556K(174784K), 0.0929628 secs] 253140K->181183K(253440K), [Perm : 17126K->17126K(17152K)], 0.0930250 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 68.944: [Full GC 68.944: [Tenured: 174556K->173476K(174784K), 0.0923362 secs] 253121K->180107K(253440K), [Perm : 17126K->17126K(17152K)], 0.0923992 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 69.084: [Full GC 69.084: [Tenured: 174556K->174556K(174784K), 0.0924605 secs] 253118K->180651K(253440K), [Perm : 17126K->17126K(17152K)], 0.0925205 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 69.234: [Full GC 69.234: [Tenured: 174556K->174556K(174784K), 0.0952437 secs] 253150K->181195K(253440K), [Perm : 17126K->17126K(17152K)], 0.0953046 secs] [Times: user=0.09 sys=0.00, real=0.10 secs] 69.382: [Full GC 69.382: [Tenured: 174556K->174556K(174784K), 0.0903325 secs] 252729K->181199K(253440K), [Perm : 17126K->17126K(17152K)], 0.0903950 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 69.521: [Full GC 69.521: [Tenured: 174556K->173476K(174784K), 0.0931242 secs] 253130K->180123K(253440K), [Perm : 17126K->17126K(17152K)], 0.0931838 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 69.662: [Full GC 69.662: [Tenured: 174556K->174556K(174784K), 0.0922443 secs] 253132K->180667K(253440K), [Perm : 17126K->17126K(17152K)], 0.0923102 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 69.803: [Full GC 69.804: [Tenured: 174556K->174556K(174784K), 0.0963803 secs] 253140K->181211K(253440K), [Perm : 17126K->17126K(17152K)], 0.0964378 secs] [Times: user=0.09 sys=0.00, real=0.10 secs] 69.955: [Full GC 69.955: [Tenured: 174556K->174556K(174784K), 0.0968898 secs] 253140K->181215K(253440K), [Perm : 17126K->17126K(17152K)], 0.0969499 secs] [Times: user=0.09 sys=0.00, real=0.10 secs] 70.099: [Full GC 70.099: [Tenured: 174556K->173476K(174784K), 0.0926946 secs] 253144K->180139K(253440K), [Perm : 17126K->17126K(17152K)], 0.0927551 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 70.249: [Full GC 70.249: [Tenured: 174556K->174556K(174784K), 0.1000149 secs] 252636K->180683K(253440K), [Perm : 17126K->17126K(17152K)], 0.1000770 secs] [Times: user=0.09 sys=0.00, real=0.10 secs] 70.405: [Full GC 70.405: [Tenured: 174556K->174556K(174784K), 0.0915440 secs] 252769K->181227K(253440K), [Perm : 17126K->17126K(17152K)], 0.0916057 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 70.544: [Full GC 70.544: [Tenured: 174556K->174556K(174784K), 0.0926087 secs] 253154K->181231K(253440K), [Perm : 17126K->17126K(17152K)], 0.0926695 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 70.686: [Full GC 70.686: [Tenured: 174556K->173476K(174784K), 0.0917896 secs] 253159K->180154K(253440K), [Perm : 17126K->17126K(17152K)], 0.0918479 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 70.826: [Full GC 70.826: [Tenured: 174556K->174556K(174784K), 0.0938351 secs] 252630K->180698K(253440K), [Perm : 17126K->17126K(17152K)], 0.0938955 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 70.969: [Full GC 70.969: [Tenured: 174556K->174556K(174784K), 0.0918679 secs] 252626K->181242K(253440K), [Perm : 17126K->17126K(17152K)], 0.0919297 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 71.108: [Full GC 71.109: [Tenured: 174556K->174556K(174784K), 0.0935941 secs] 252629K->181246K(253440K), [Perm : 17126K->17126K(17152K)], 0.0936537 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 71.260: [Full GC 71.260: [Tenured: 174556K->173476K(174784K), 0.0955132 secs] 252660K->180170K(253440K), [Perm : 17126K->17126K(17152K)], 0.0955762 secs] [Times: user=0.08 sys=0.00, real=0.10 secs] 71.408: [Full GC 71.408: [Tenured: 174556K->174556K(174784K), 0.0927517 secs] 252787K->180714K(253440K), [Perm : 17126K->17126K(17152K)], 0.0928130 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 71.549: [Full GC 71.549: [Tenured: 174556K->174556K(174784K), 0.0918535 secs] 252642K->181258K(253440K), [Perm : 17126K->17126K(17152K)], 0.0919118 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 71.688: [Full GC 71.688: [Tenured: 174556K->174556K(174784K), 0.0933085 secs] 252646K->181262K(253440K), [Perm : 17126K->17126K(17152K)], 0.0933685 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 71.830: [Full GC 71.830: [Tenured: 174556K->173476K(174784K), 0.0922472 secs] 252650K->180185K(253440K), [Perm : 17126K->17126K(17152K)], 0.0923085 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 71.970: [Full GC 71.970: [Tenured: 174556K->174556K(174784K), 0.0948767 secs] 252658K->180729K(253440K), [Perm : 17126K->17126K(17152K)], 0.0949346 secs] [Times: user=0.09 sys=0.00, real=0.10 secs] 72.113: [Full GC 72.113: [Tenured: 174556K->174556K(174784K), 0.0926559 secs] 252658K->181273K(253440K), [Perm : 17126K->17126K(17152K)], 0.0927159 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 72.265: [Full GC 72.265: [Tenured: 174556K->174556K(174784K), 0.0958690 secs] 252690K->181278K(253440K), [Perm : 17126K->17126K(17152K)], 0.0959312 secs] [Times: user=0.08 sys=0.00, real=0.10 secs] 72.412: [Full GC 72.412: [Tenured: 174556K->173476K(174784K), 0.0924984 secs] 252814K->180202K(253440K), [Perm : 17126K->17126K(17152K)], 0.0925593 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 72.552: [Full GC 72.552: [Tenured: 174556K->174556K(174784K), 0.0925150 secs] 252671K->180746K(253440K), [Perm : 17126K->17126K(17152K)], 0.0925754 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 72.694: [Full GC 72.694: [Tenured: 174556K->174556K(174784K), 0.0913941 secs] 252677K->181290K(253440K), [Perm : 17126K->17126K(17152K)], 0.0914499 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 72.835: [Full GC 72.835: [Tenured: 174556K->174556K(174784K), 0.0961866 secs] 252679K->181294K(253440K), [Perm : 17126K->17126K(17152K)], 0.0962432 secs] [Times: user=0.09 sys=0.00, real=0.10 secs] 72.987: [Full GC 72.987: [Tenured: 174556K->173476K(174784K), 0.0982052 secs] 252683K->180217K(253440K), [Perm : 17126K->17126K(17152K)], 0.0982648 secs] [Times: user=0.11 sys=0.00, real=0.10 secs] 73.132: [Full GC 73.132: [Tenured: 174556K->174556K(174784K), 0.0937146 secs] 252687K->180761K(253440K), [Perm : 17126K->17126K(17152K)], 0.0937763 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 73.283: [Full GC 73.283: [Tenured: 174556K->174556K(174784K), 0.0959597 secs] 252714K->181305K(253440K), [Perm : 17126K->17126K(17152K)], 0.0960227 secs] [Times: user=0.09 sys=0.00, real=0.10 secs] 73.428: [Full GC 73.428: [Tenured: 174556K->174556K(174784K), 0.0923967 secs] 252841K->181309K(253440K), [Perm : 17126K->17126K(17152K)], 0.0924588 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 73.567: [Full GC 73.567: [Tenured: 174556K->173476K(174784K), 0.0935933 secs] 252698K->180233K(253440K), [Perm : 17126K->17126K(17152K)], 0.0936559 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 73.711: [Full GC 73.711: [Tenured: 174556K->174556K(174784K), 0.0915253 secs] 252704K->180777K(253440K), [Perm : 17126K->17126K(17152K)], 0.0915849 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 73.850: [Full GC 73.850: [Tenured: 174556K->174556K(174784K), 0.0946916 secs] 252706K->181321K(253440K), [Perm : 17126K->17126K(17152K)], 0.0947537 secs] [Times: user=0.09 sys=0.00, real=0.10 secs] 73.994: [Full GC 73.994: [Tenured: 174556K->174556K(174784K), 0.0918884 secs] 252710K->181325K(253440K), [Perm : 17126K->17126K(17152K)], 0.0919488 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 74.134: [Full GC 74.134: [Tenured: 174556K->173476K(174784K), 0.0938151 secs] 252714K->180248K(253440K), [Perm : 17126K->17126K(17152K)], 0.0938755 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 74.287: [Full GC 74.287: [Tenured: 174556K->174556K(174784K), 0.0960342 secs] 252746K->180792K(253440K), [Perm : 17126K->17126K(17152K)], 0.0960895 secs] [Times: user=0.09 sys=0.00, real=0.10 secs] 74.432: [Full GC 74.432: [Tenured: 174556K->174556K(174784K), 0.0928947 secs] 252868K->181336K(253440K), [Perm : 17126K->17126K(17152K)], 0.0929560 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 74.575: [Full GC 74.575: [Tenured: 174556K->174556K(174784K), 0.0966191 secs] 252726K->181340K(253440K), [Perm : 17126K->17126K(17152K)], 0.0966744 secs] [Times: user=0.09 sys=0.00, real=0.10 secs] 74.725: [Full GC 74.725: [Tenured: 174556K->173476K(174784K), 0.0941667 secs] 252729K->180264K(253440K), [Perm : 17126K->17126K(17152K)], 0.0942280 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 74.867: [Full GC 74.867: [Tenured: 174556K->174556K(174784K), 0.0940854 secs] 252733K->180808K(253440K), [Perm : 17126K->17126K(17152K)], 0.0941492 secs] [Times: user=0.09 sys=0.00, real=0.10 secs] 75.011: [Full GC 75.011: [Tenured: 174556K->174556K(174784K), 0.0922404 secs] 252739K->181352K(253440K), [Perm : 17126K->17126K(17152K)], 0.0922996 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 75.150: [Full GC 75.150: [Tenured: 174556K->174556K(174784K), 0.0937529 secs] 252741K->181356K(253440K), [Perm : 17126K->17126K(17152K)], 0.0938159 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 75.301: [Full GC 75.301: [Tenured: 174556K->173475K(174784K), 0.1771257 secs] 252771K->180278K(253440K), [Perm : 17126K->17126K(17152K)], 0.1771938 secs] [Times: user=0.17 sys=0.00, real=0.18 secs] 75.530: [Full GC 75.531: [Tenured: 174555K->174555K(174784K), 0.0921957 secs] 252898K->180822K(253440K), [Perm : 17126K->17126K(17152K)], 0.0922583 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 75.670: [Full GC 75.670: [Tenured: 174555K->174555K(174784K), 0.0932953 secs] 252752K->181366K(253440K), [Perm : 17126K->17126K(17152K)], 0.0933557 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 75.813: [Full GC 75.813: [Tenured: 174555K->174555K(174784K), 0.0967613 secs] 252756K->181370K(253440K), [Perm : 17126K->17126K(17152K)], 0.0968234 secs] [Times: user=0.09 sys=0.00, real=0.10 secs] 75.964: [Full GC 75.964: [Tenured: 174555K->173241K(174784K), 0.1877438 secs] 252759K->180060K(253440K), [Perm : 17126K->17124K(17152K)], 0.1878076 secs] [Times: user=0.19 sys=0.00, real=0.19 secs] 76.203: [Full GC 76.203: [Tenured: 174321K->174321K(174784K), 0.0906815 secs] 252550K->180604K(253440K), [Perm : 17124K->17124K(17152K)], 0.0907416 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 76.355: [Full GC 76.355: [Tenured: 174321K->174321K(174784K), 0.0943489 secs] 252545K->181148K(253440K), [Perm : 17124K->17124K(17152K)], 0.0944085 secs] [Times: user=0.09 sys=0.00, real=0.10 secs] 76.501: [Full GC 76.501: [Tenured: 174321K->174321K(174784K), 0.0958222 secs] 252683K->181152K(253440K), [Perm : 17124K->17124K(17152K)], 0.0958839 secs] [Times: user=0.08 sys=0.00, real=0.10 secs] 76.645: [Full GC 76.645: [Tenured: 174321K->173194K(174784K), 0.1750156 secs] 252541K->180029K(253440K), [Perm : 17124K->17121K(17152K)], 0.1750760 secs] [Times: user=0.17 sys=0.00, real=0.17 secs] 76.869: [Full GC 76.869: [Tenured: 174274K->174274K(174784K), 0.0912524 secs] 252500K->180573K(253440K), [Perm : 17121K->17121K(17152K)], 0.0913175 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 77.008: [Full GC 77.008: [Tenured: 174274K->174274K(174784K), 0.0930352 secs] 252503K->181117K(253440K), [Perm : 17121K->17121K(17152K)], 0.0930973 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 77.153: [Full GC 77.153: [Tenured: 174274K->174530K(174784K), 0.1806351 secs] 252506K->175070K(253440K), [Perm : 17121K->17121K(17152K)], 0.1806969 secs] [Times: user=0.19 sys=0.00, real=0.18 secs] 77.398: [Full GC 77.398: [Tenured: 174530K->167748K(174784K), 0.1809714 secs] 252974K->167748K(253440K), [Perm : 17121K->17121K(17152K)], 0.1810366 secs] [Times: user=0.17 sys=0.00, real=0.18 secs] 77.628: [GC 77.628: [DefNew: 69911K->545K(78656K), 0.0025767 secs] 237659K->168293K(253440K), 0.0026206 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 77.679: [GC 77.679: [DefNew: 70310K->549K(78656K), 0.0027793 secs] 238058K->168297K(253440K), 0.0028211 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 77.729: [GC 77.729: [DefNew: 70314K->553K(78656K), 0.0029654 secs] 238062K->168301K(253440K), 0.0030177 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 77.781: [GC 77.781: [DefNew: 70318K->556K(78656K), 0.0025699 secs] 238066K->168305K(253440K), 0.0026108 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 77.831: [GC 77.831: [DefNew: 70319K->560K(78656K), 0.0028679 secs] 238068K->168308K(253440K), 0.0029113 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 77.883: [GC 77.883: [DefNew: 70325K->564K(78656K), 0.0027900 secs] 238073K->168312K(253440K), 0.0028330 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 77.933: [GC 77.933: [DefNew: 70329K->568K(78656K), 0.0027368 secs] 238077K->168316K(253440K), 0.0027849 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 77.986: [GC 77.986: [DefNew: 70333K->572K(78656K), 0.0026252 secs] 238081K->168320K(253440K), 0.0026661 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 78.036: [GC 78.036: [DefNew: 70334K->575K(78656K), 0.0028892 secs] 238083K->168324K(253440K), 0.0029369 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 78.089: [GC 78.089: [DefNew: 70340K->579K(78656K), 0.0028343 secs] 238089K->168327K(253440K), 0.0028794 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 78.139: [GC 78.139: [DefNew: 70344K->583K(78656K), 0.0029130 secs] 238092K->168331K(253440K), 0.0029628 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 78.190: [GC 78.190: [DefNew: 70348K->587K(78656K), 0.0028338 secs] 238096K->168335K(253440K), 0.0028794 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 78.257: [GC 78.257: [DefNew: 70375K->590K(78656K), 0.0026065 secs] 238124K->168339K(253440K), 0.0026436 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 78.311: [GC 78.311: [DefNew: 70355K->594K(78656K), 0.0031706 secs] 238104K->168343K(253440K), 0.0032216 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 78.364: [GC 78.364: [DefNew: 70359K->598K(78656K), 0.0026942 secs] 238108K->168346K(253440K), 0.0027364 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 78.417: [GC 78.417: [DefNew: 70508K->597K(78656K), 0.0032732 secs] 238257K->168351K(253440K), 0.0033251 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 78.469: [GC 78.469: [DefNew: 70360K->597K(78656K), 0.0028394 secs] 238114K->168355K(253440K), 0.0028807 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 78.519: [GC 78.519: [DefNew: 70362K->597K(78656K), 0.0030194 secs] 238120K->168359K(253440K), 0.0030658 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 78.572: [GC 78.572: [DefNew: 70362K->597K(78656K), 0.0027955 secs] 238123K->168362K(253440K), 0.0028343 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 78.622: [GC 78.622: [DefNew: 70362K->597K(78656K), 0.0028947 secs] 238127K->168366K(253440K), 0.0029369 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 78.674: [GC 78.674: [DefNew: 70360K->597K(78656K), 0.0028398 secs] 238129K->168370K(253440K), 0.0028802 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 78.728: [GC 78.728: [DefNew: 70362K->597K(78656K), 0.0029577 secs] 238135K->168374K(253440K), 0.0030020 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 78.778: [GC 78.778: [DefNew: 70362K->597K(78656K), 0.0030394 secs] 238139K->168377K(253440K), 0.0030850 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 78.831: [GC 78.831: [DefNew: 70362K->597K(78656K), 0.0027236 secs] 238143K->168381K(253440K), 0.0027632 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 78.881: [GC 78.881: [DefNew: 70360K->597K(78656K), 0.0028466 secs] 238144K->168385K(253440K), 0.0028883 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 78.932: [GC 78.932: [DefNew: 70362K->597K(78656K), 0.0029458 secs] 238150K->168389K(253440K), 0.0029867 secs] [Times: user=0.00 sys=0.02, real=0.00 secs] 78.991: [GC 78.991: [DefNew: 70364K->597K(78656K), 0.0027423 secs] 238156K->168392K(253440K), 0.0027823 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 79.049: [GC 79.049: [DefNew: 70360K->597K(78656K), 0.0027393 secs] 238156K->168396K(253440K), 0.0027751 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 79.105: [GC 79.105: [DefNew: 70362K->597K(78656K), 0.0027636 secs] 238161K->168400K(253440K), 0.0028104 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 79.164: [GC 79.164: [DefNew: 70360K->597K(78656K), 0.0030390 secs] 238163K->168404K(253440K), 0.0030893 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 79.221: [GC 79.221: [DefNew: 70390K->596K(78656K), 0.0029156 secs] 238197K->168408K(253440K), 0.0029603 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 79.284: [GC 79.284: [DefNew: 70360K->596K(78656K), 0.0027776 secs] 238171K->168411K(253440K), 0.0028202 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 79.334: [GC 79.334: [DefNew: 70361K->596K(78656K), 0.0030714 secs] 238176K->168415K(253440K), 0.0031182 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 79.406: [GC 79.406: [DefNew: 70361K->596K(78656K), 0.0030067 secs] 238180K->168419K(253440K), 0.0030582 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 79.468: [GC 79.468: [DefNew: 70361K->596K(78656K), 0.0029168 secs] 238184K->168423K(253440K), 0.0029641 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 79.525: [GC 79.525: [DefNew: 70507K->597K(78656K), 0.0028049 secs] 238333K->168427K(253440K), 0.0028466 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 79.577: [GC 79.578: [DefNew: 70360K->597K(78656K), 0.0029049 secs] 238190K->168431K(253440K), 0.0029501 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 79.628: [GC 79.628: [DefNew: 70362K->597K(78656K), 0.0031152 secs] 238196K->168435K(253440K), 0.0031595 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 79.681: [GC 79.681: [DefNew: 70362K->597K(78656K), 0.0027274 secs] 238200K->168439K(253440K), 0.0027649 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 79.732: [GC 79.732: [DefNew: 70362K->597K(78656K), 0.0029151 secs] 238204K->168442K(253440K), 0.0029611 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 79.783: [GC 79.783: [DefNew: 70360K->597K(78656K), 0.0028223 secs] 238205K->168446K(253440K), 0.0028615 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 79.837: [GC 79.837: [DefNew: 70362K->597K(78656K), 0.0028790 secs] 238211K->168450K(253440K), 0.0029220 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 79.888: [GC 79.888: [DefNew: 70362K->597K(78656K), 0.0030301 secs] 238215K->168454K(253440K), 0.0030761 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 79.943: [GC 79.943: [DefNew: 70362K->597K(78656K), 0.0027559 secs] 238219K->168458K(253440K), 0.0028015 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 79.996: [GC 79.996: [DefNew: 70362K->597K(78656K), 0.0029249 secs] 238223K->168461K(253440K), 0.0029696 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 80.047: [GC 80.047: [DefNew: 70360K->597K(78656K), 0.0032361 secs] 238224K->168465K(253440K), 0.0032868 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 80.100: [GC 80.100: [DefNew: 70364K->597K(78656K), 0.0027602 secs] 238232K->168469K(253440K), 0.0028019 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 80.154: [GC 80.154: [DefNew: 70360K->597K(78656K), 0.0028824 secs] 238232K->168473K(253440K), 0.0029275 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 80.208: [GC 80.208: [DefNew: 70388K->597K(78656K), 0.0030211 secs] 238264K->168477K(253440K), 0.0030709 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 80.270: [GC 80.270: [DefNew: 70358K->597K(78656K), 0.0029569 secs] 238238K->168480K(253440K), 0.0029994 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 80.324: [GC 80.324: [DefNew: 70364K->596K(78656K), 0.0029011 secs] 238247K->168484K(253440K), 0.0029479 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 80.376: [GC 80.376: [DefNew: 70360K->596K(78656K), 0.0027547 secs] 238247K->168488K(253440K), 0.0027972 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 80.432: [GC 80.432: [DefNew: 70518K->597K(78656K), 0.0028040 secs] 238409K->168492K(253440K), 0.0028483 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 80.483: [GC 80.483: [DefNew: 70360K->597K(78656K), 0.0030271 secs] 238255K->168496K(253440K), 0.0030769 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 80.536: [GC 80.536: [DefNew: 70364K->597K(78656K), 0.0028853 secs] 238263K->168500K(253440K), 0.0029271 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 80.589: [GC 80.589: [DefNew: 70360K->597K(78656K), 0.0027347 secs] 238263K->168504K(253440K), 0.0028726 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 80.640: [GC 80.640: [DefNew: 70362K->597K(78656K), 0.0031514 secs] 238269K->168507K(253440K), 0.0032012 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 80.693: [GC 80.693: [DefNew: 70362K->597K(78656K), 0.0026580 secs] 238272K->168511K(253440K), 0.0026951 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 80.744: [GC 80.744: [DefNew: 70362K->597K(78656K), 0.0029305 secs] 238276K->168515K(253440K), 0.0029756 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 80.795: [GC 80.795: [DefNew: 70362K->597K(78656K), 0.0027657 secs] 238280K->168519K(253440K), 0.0028062 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 80.849: [GC 80.849: [DefNew: 70360K->597K(78656K), 0.0028500 secs] 238282K->168523K(253440K), 0.0028900 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 80.901: [GC 80.901: [DefNew: 70362K->597K(78656K), 0.0029764 secs] 238288K->168526K(253440K), 0.0030194 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 80.952: [GC 80.952: [DefNew: 70362K->597K(78656K), 0.0029811 secs] 238291K->168530K(253440K), 0.0030271 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 81.005: [GC 81.005: [DefNew: 70362K->597K(78656K), 0.0028479 secs] 238295K->168534K(253440K), 0.0028888 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 81.056: [GC 81.056: [DefNew: 70360K->597K(78656K), 0.0031331 secs] 238297K->168538K(253440K), 0.0031769 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 81.109: [GC 81.109: [DefNew: 70362K->597K(78656K), 0.0028091 secs] 238303K->168542K(253440K), 0.0028492 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 81.160: [GC 81.160: [DefNew: 70362K->597K(78656K), 0.0029926 secs] 238307K->168545K(253440K), 0.0030348 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 81.215: [GC 81.215: [DefNew: 70388K->596K(78656K), 0.0029445 secs] 238336K->168549K(253440K), 0.0029837 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 81.280: [GC 81.280: [DefNew: 70359K->596K(78656K), 0.0031276 secs] 238312K->168553K(253440K), 0.0031761 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 81.335: [GC 81.335: [DefNew: 70361K->596K(78656K), 0.0028764 secs] 238318K->168557K(253440K), 0.0029203 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 81.387: [GC 81.387: [DefNew: 70361K->596K(78656K), 0.0027453 secs] 238322K->168560K(253440K), 0.0027832 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 81.440: [GC 81.441: [DefNew: 70507K->597K(78656K), 0.0031288 secs] 238471K->168565K(253440K), 0.0031855 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 81.494: [GC 81.494: [DefNew: 70362K->597K(78656K), 0.0028177 secs] 238330K->168569K(253440K), 0.0028590 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 81.548: [GC 81.548: [DefNew: 70360K->597K(78656K), 0.0029620 secs] 238332K->168572K(253440K), 0.0030045 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 81.599: [GC 81.599: [DefNew: 70364K->597K(78656K), 0.0030480 secs] 238339K->168576K(253440K), 0.0030905 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 81.651: [GC 81.651: [DefNew: 70360K->597K(78656K), 0.0030744 secs] 238339K->168580K(253440K), 0.0031156 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 81.710: [GC 81.710: [DefNew: 70362K->597K(78656K), 0.0028492 secs] 238345K->168584K(253440K), 0.0028900 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 81.763: [GC 81.763: [DefNew: 70360K->597K(78656K), 0.0028032 secs] 238347K->168588K(253440K), 0.0028428 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 81.817: [GC 81.817: [DefNew: 70364K->597K(78656K), 0.0027470 secs] 238355K->168591K(253440K), 0.0027862 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 81.868: [GC 81.869: [DefNew: 70360K->597K(78656K), 0.0028849 secs] 238354K->168595K(253440K), 0.0029300 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 81.921: [GC 81.921: [DefNew: 70362K->597K(78656K), 0.0027738 secs] 238360K->168599K(253440K), 0.0028130 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 81.975: [GC 81.975: [DefNew: 70360K->597K(78656K), 0.0027883 secs] 238362K->168603K(253440K), 0.0028287 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 82.033: [GC 82.033: [DefNew: 70362K->597K(78656K), 0.0028296 secs] 238368K->168606K(253440K), 0.0028658 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 82.095: [GC 82.095: [DefNew: 70362K->597K(78656K), 0.0026257 secs] 238372K->168610K(253440K), 0.0026648 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 82.155: [GC 82.155: [DefNew: 70362K->597K(78656K), 0.0029611 secs] 238375K->168614K(253440K), 0.0030037 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 82.210: [GC 82.210: [DefNew: 70386K->597K(78656K), 0.0029381 secs] 238403K->168618K(253440K), 0.0029799 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 82.272: [GC 82.272: [DefNew: 70364K->596K(78656K), 0.0029598 secs] 238385K->168621K(253440K), 0.0030084 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 82.325: [GC 82.325: [DefNew: 70360K->596K(78656K), 0.0030522 secs] 238385K->168625K(253440K), 0.0030965 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 82.379: [GC 82.379: [DefNew: 70361K->596K(78656K), 0.0030812 secs] 238390K->168629K(253440K), 0.0031318 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 82.434: [GC 82.434: [DefNew: 70505K->597K(78656K), 0.0030824 secs] 238537K->168634K(253440K), 0.0031305 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 82.487: [GC 82.487: [DefNew: 70364K->597K(78656K), 0.0027725 secs] 238401K->168637K(253440K), 0.0028134 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 82.542: [GC 82.542: [DefNew: 70360K->597K(78656K), 0.0028394 secs] 238400K->168641K(253440K), 0.0028781 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 82.594: [GC 82.594: [DefNew: 70362K->597K(78656K), 0.0033085 secs] 238406K->168645K(253440K), 0.0033613 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 82.648: [GC 82.648: [DefNew: 70360K->597K(78656K), 0.0027713 secs] 238408K->168649K(253440K), 0.0028147 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 82.703: [GC 82.703: [DefNew: 70364K->597K(78656K), 0.0027623 secs] 238416K->168652K(253440K), 0.0028019 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 82.755: [GC 82.756: [DefNew: 70360K->597K(78656K), 0.0029015 secs] 238416K->168656K(253440K), 0.0029488 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 82.807: [GC 82.807: [DefNew: 70362K->597K(78656K), 0.0029667 secs] 238421K->168660K(253440K), 0.0030071 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 82.861: [GC 82.861: [DefNew: 70360K->597K(78656K), 0.0029339 secs] 238423K->168664K(253440K), 0.0029730 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 82.914: [GC 82.914: [DefNew: 70364K->597K(78656K), 0.0031608 secs] 238431K->168667K(253440K), 0.0032093 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 82.968: [GC 82.968: [DefNew: 70362K->597K(78656K), 0.0028696 secs] 238433K->168671K(253440K), 0.0029088 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 83.022: [GC 83.022: [DefNew: 70360K->597K(78656K), 0.0028360 secs] 238434K->168675K(253440K), 0.0028768 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 83.074: [GC 83.074: [DefNew: 70362K->597K(78656K), 0.0031612 secs] 238440K->168679K(253440K), 0.0032170 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 83.130: [GC 83.130: [DefNew: 70362K->597K(78656K), 0.0028002 secs] 238444K->168683K(253440K), 0.0028432 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 83.184: [GC 83.184: [DefNew: 70362K->597K(78656K), 0.0027653 secs] 238448K->168686K(253440K), 0.0028049 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 83.245: [GC 83.245: [DefNew: 70386K->596K(78656K), 0.0030020 secs] 238475K->168690K(253440K), 0.0030446 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 83.303: [GC 83.303: [DefNew: 70361K->596K(78656K), 0.0030314 secs] 238455K->168694K(253440K), 0.0030799 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 83.358: [GC 83.358: [DefNew: 70361K->596K(78656K), 0.0028947 secs] 238459K->168698K(253440K), 0.0029373 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 83.411: [GC 83.411: [DefNew: 70362K->596K(78656K), 0.0029615 secs] 238463K->168702K(253440K), 0.0030045 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 83.466: [GC 83.466: [DefNew: 70504K->597K(78656K), 0.0029594 secs] 238610K->168706K(253440K), 0.0030024 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 83.519: [GC 83.519: [DefNew: 70362K->597K(78656K), 0.0032680 secs] 238471K->168710K(253440K), 0.0033149 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 83.573: [GC 83.573: [DefNew: 70362K->597K(78656K), 0.0027891 secs] 238475K->168713K(253440K), 0.0028283 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 83.627: [GC 83.627: [DefNew: 70362K->597K(78656K), 0.0027696 secs] 238479K->168717K(253440K), 0.0028091 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 83.680: [GC 83.680: [DefNew: 70362K->597K(78656K), 0.0030518 secs] 238482K->168721K(253440K), 0.0030982 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 83.735: [GC 83.735: [DefNew: 70360K->597K(78656K), 0.0029113 secs] 238484K->168725K(253440K), 0.0029539 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 83.788: [GC 83.788: [DefNew: 70364K->597K(78656K), 0.0029428 secs] 238492K->168729K(253440K), 0.0029837 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 83.842: [GC 83.842: [DefNew: 70360K->597K(78656K), 0.0028739 secs] 238492K->168732K(253440K), 0.0029151 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 83.894: [GC 83.894: [DefNew: 70362K->597K(78656K), 0.0028309 secs] 238497K->168736K(253440K), 0.0028739 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 83.948: [GC 83.948: [DefNew: 70360K->597K(78656K), 0.0030641 secs] 238499K->168740K(253440K), 0.0031105 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 84.003: [GC 84.003: [DefNew: 70364K->597K(78656K), 0.0029292 secs] 238507K->168744K(253440K), 0.0029735 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 84.055: [GC 84.055: [DefNew: 70360K->597K(78656K), 0.0030267 secs] 238507K->168748K(253440K), 0.0030761 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 84.109: [GC 84.109: [DefNew: 70362K->597K(78656K), 0.0027862 secs] 238512K->168751K(253440K), 0.0028266 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 84.165: [GC 84.165: [DefNew: 70360K->597K(78656K), 0.0028564 secs] 238514K->168755K(253440K), 0.0029007 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 84.223: [GC 84.223: [DefNew: 70390K->597K(78656K), 0.0027470 secs] 238548K->168759K(253440K), 0.0027968 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 84.285: [GC 84.285: [DefNew: 70469K->703K(78656K), 0.0031348 secs] 238631K->168869K(253440K), 0.0031850 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 84.341: [GC 84.341: [DefNew: 70139K->916K(78656K), 0.0030122 secs] 238306K->169086K(253440K), 0.0030543 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 84.393: [GC 84.393: [DefNew: 70681K->916K(78656K), 0.0036273 secs] 238851K->169090K(253440K), 0.0036797 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 84.451: [GC 84.451: [DefNew: 70825K->917K(78656K), 0.0030356 secs] 238999K->169095K(253440K), 0.0030820 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 84.505: [GC 84.505: [DefNew: 70684K->917K(78656K), 0.0030224 secs] 238862K->169098K(253440K), 0.0030675 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 84.560: [GC 84.560: [DefNew: 70680K->917K(78656K), 0.0028926 secs] 238862K->169102K(253440K), 0.0029369 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 84.613: [GC 84.613: [DefNew: 70682K->917K(78656K), 0.0033408 secs] 238867K->169106K(253440K), 0.0033902 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 84.669: [GC 84.669: [DefNew: 70680K->917K(78656K), 0.0031365 secs] 238869K->169110K(253440K), 0.0031799 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 84.724: [GC 84.725: [DefNew: 70684K->917K(78656K), 0.0031386 secs] 238877K->169114K(253440K), 0.0031821 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 84.780: [GC 84.780: [DefNew: 70682K->917K(78656K), 0.0032012 secs] 238879K->169117K(253440K), 0.0032446 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 84.833: [GC 84.833: [DefNew: 70680K->917K(78656K), 0.0036112 secs] 238880K->169121K(253440K), 0.0036588 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 84.889: [GC 84.889: [DefNew: 70682K->917K(78656K), 0.0033459 secs] 238886K->169125K(253440K), 0.0033940 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 84.945: [GC 84.945: [DefNew: 70682K->917K(78656K), 0.0033289 secs] 238890K->169129K(253440K), 0.0033740 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 84.999: [GC 84.999: [DefNew: 70682K->917K(78656K), 0.0031433 secs] 238894K->169132K(253440K), 0.0031859 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 85.061: [GC 85.061: [DefNew: 70680K->917K(78656K), 0.0031901 secs] 238895K->169136K(253440K), 0.0032310 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 85.119: [GC 85.119: [DefNew: 70682K->810K(78656K), 0.0031625 secs] 238901K->169140K(253440K), 0.0032016 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 85.181: [GC 85.181: [DefNew: 70575K->597K(78656K), 0.0029403 secs] 238905K->169144K(253440K), 0.0029837 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 85.241: [GC 85.241: [DefNew: 70388K->597K(78656K), 0.0029173 secs] 238935K->169148K(253440K), 0.0029654 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 85.303: [GC 85.303: [DefNew: 70360K->596K(78656K), 0.0030412 secs] 238911K->169151K(253440K), 0.0030876 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 85.358: [GC 85.358: [DefNew: 70361K->596K(78656K), 0.0028062 secs] 238916K->169155K(253440K), 0.0028466 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 85.411: [GC 85.411: [DefNew: 70363K->596K(78656K), 0.0031297 secs] 238922K->169159K(253440K), 0.0031752 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 85.465: [GC 85.465: [DefNew: 70505K->597K(78656K), 0.0029837 secs] 239067K->169163K(253440K), 0.0030322 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 85.521: [GC 85.521: [DefNew: 70362K->597K(78656K), 0.0028845 secs] 238928K->169167K(253440K), 0.0029241 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 85.575: [GC 85.575: [DefNew: 70360K->597K(78656K), 0.0028406 secs] 238930K->169171K(253440K), 0.0028811 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 85.630: [GC 85.630: [DefNew: 70364K->597K(78656K), 0.0026431 secs] 238938K->169175K(253440K), 0.0026840 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 85.683: [GC 85.683: [DefNew: 70360K->597K(78656K), 0.0029054 secs] 238938K->169178K(253440K), 0.0029501 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 85.738: [GC 85.738: [DefNew: 70362K->597K(78656K), 0.0028773 secs] 238943K->169182K(253440K), 0.0029211 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 85.793: [GC 85.793: [DefNew: 70360K->597K(78656K), 0.0028432 secs] 238945K->169186K(253440K), 0.0028870 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 85.846: [GC 85.846: [DefNew: 70364K->597K(78656K), 0.0029415 secs] 238953K->169190K(253440K), 0.0029875 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 85.901: [GC 85.901: [DefNew: 70360K->597K(78656K), 0.0028853 secs] 238953K->169194K(253440K), 0.0029254 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 85.956: [GC 85.956: [DefNew: 70362K->597K(78656K), 0.0029398 secs] 238959K->169197K(253440K), 0.0029803 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 86.012: [GC 86.012: [DefNew: 70360K->597K(78656K), 0.0026980 secs] 238960K->169201K(253440K), 0.0027385 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 86.065: [GC 86.065: [DefNew: 70364K->597K(78656K), 0.0029726 secs] 238968K->169205K(253440K), 0.0030160 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 86.121: [GC 86.121: [DefNew: 70362K->597K(78656K), 0.0028419 secs] 238970K->169209K(253440K), 0.0028841 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 86.175: [GC 86.175: [DefNew: 70360K->597K(78656K), 0.0027811 secs] 238972K->169212K(253440K), 0.0028202 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 86.235: [GC 86.235: [DefNew: 70388K->597K(78656K), 0.0029803 secs] 239003K->169216K(253440K), 0.0030250 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 86.295: [GC 86.295: [DefNew: 70364K->596K(78656K), 0.0030990 secs] 238983K->169220K(253440K), 0.0031459 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 86.350: [GC 86.350: [DefNew: 70362K->596K(78656K), 0.0028377 secs] 238985K->169224K(253440K), 0.0028790 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 86.403: [GC 86.403: [DefNew: 70359K->596K(78656K), 0.0031225 secs] 238987K->169228K(253440K), 0.0031680 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 86.461: [GC 86.461: [DefNew: 70507K->597K(78656K), 0.0029386 secs] 239138K->169232K(253440K), 0.0029867 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 86.516: [GC 86.516: [DefNew: 70362K->597K(78656K), 0.0029139 secs] 238997K->169236K(253440K), 0.0029581 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 86.570: [GC 86.570: [DefNew: 70362K->597K(78656K), 0.0030356 secs] 239001K->169240K(253440K), 0.0030795 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 86.626: [GC 86.626: [DefNew: 70360K->597K(78656K), 0.0029611 secs] 239003K->169243K(253440K), 0.0030045 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 86.681: [GC 86.681: [DefNew: 70362K->597K(78656K), 0.0029475 secs] 239009K->169247K(253440K), 0.0029901 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 86.736: [GC 86.736: [DefNew: 70362K->597K(78656K), 0.0027440 secs] 239012K->169251K(253440K), 0.0027870 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 86.791: [GC 86.791: [DefNew: 70362K->597K(78656K), 0.0029543 secs] 239016K->169255K(253440K), 0.0029977 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 86.845: [GC 86.845: [DefNew: 70360K->597K(78656K), 0.0030169 secs] 239018K->169259K(253440K), 0.0030607 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 86.901: [GC 86.901: [DefNew: 70362K->597K(78656K), 0.0030501 secs] 239024K->169263K(253440K), 0.0030939 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 86.955: [GC 86.955: [DefNew: 70364K->597K(78656K), 0.0032944 secs] 239029K->169266K(253440K), 0.0033447 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 87.010: [GC 87.010: [DefNew: 70360K->597K(78656K), 0.0027683 secs] 239029K->169270K(253440K), 0.0028079 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 87.066: [GC 87.066: [DefNew: 70362K->597K(78656K), 0.0028057 secs] 239035K->169274K(253440K), 0.0028453 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 87.119: [GC 87.119: [DefNew: 70360K->597K(78656K), 0.0029054 secs] 239037K->169278K(253440K), 0.0029501 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 87.174: [GC 87.174: [DefNew: 70364K->597K(78656K), 0.0030267 secs] 239045K->169281K(253440K), 0.0030675 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 87.235: [GC 87.235: [DefNew: 70386K->597K(78656K), 0.0138806 secs] 239070K->169285K(253440K), 0.0139270 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 87.305: [GC 87.305: [DefNew: 70362K->596K(78656K), 0.0027994 secs] 239050K->169289K(253440K), 0.0028389 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 87.360: [GC 87.360: [DefNew: 70361K->596K(78656K), 0.0028015 secs] 239054K->169293K(253440K), 0.0028419 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 87.413: [GC 87.413: [DefNew: 70361K->596K(78656K), 0.0030956 secs] 239058K->169296K(253440K), 0.0031433 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 87.470: [GC 87.470: [DefNew: 70507K->597K(78656K), 0.0030160 secs] 239207K->169301K(253440K), 0.0030616 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 87.524: [GC 87.524: [DefNew: 70360K->597K(78656K), 0.0041646 secs] 239064K->169305K(253440K), 0.0042118 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 87.580: [GC 87.580: [DefNew: 70362K->597K(78656K), 0.0028666 secs] 239070K->169309K(253440K), 0.0029088 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 87.637: [GC 87.637: [DefNew: 70362K->597K(78656K), 0.0026823 secs] 239073K->169312K(253440K), 0.0027261 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 87.692: [GC 87.692: [DefNew: 70362K->597K(78656K), 0.0039202 secs] 239077K->169316K(253440K), 0.0039645 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 87.746: [GC 87.746: [DefNew: 70360K->597K(78656K), 0.0030735 secs] 239079K->169320K(253440K), 0.0031220 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 87.802: [GC 87.802: [DefNew: 70362K->597K(78656K), 0.0028449 secs] 239085K->169324K(253440K), 0.0028943 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 87.857: [GC 87.857: [DefNew: 70364K->597K(78656K), 0.0032216 secs] 239091K->169327K(253440K), 0.0032680 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 87.912: [GC 87.912: [DefNew: 70360K->597K(78656K), 0.0029535 secs] 239090K->169331K(253440K), 0.0029952 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 87.966: [GC 87.966: [DefNew: 70362K->597K(78656K), 0.0029743 secs] 239096K->169335K(253440K), 0.0030190 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 88.021: [GC 88.021: [DefNew: 70360K->597K(78656K), 0.0028658 secs] 239098K->169339K(253440K), 0.0029066 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 88.083: [GC 88.083: [DefNew: 70362K->597K(78656K), 0.0027921 secs] 239104K->169342K(253440K), 0.0028313 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 88.144: [GC 88.144: [DefNew: 70362K->597K(78656K), 0.0028121 secs] 239108K->169346K(253440K), 0.0028551 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 88.209: [GC 88.209: [DefNew: 70388K->597K(78656K), 0.0050402 secs] 239137K->169350K(253440K), 0.0050917 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 88.278: [GC 88.278: [DefNew: 70360K->597K(78656K), 0.0033536 secs] 239113K->169354K(253440K), 0.0033962 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 88.335: [GC 88.335: [DefNew: 70364K->596K(78656K), 0.0027687 secs] 239121K->169357K(253440K), 0.0028074 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 88.389: [GC 88.389: [DefNew: 70362K->596K(78656K), 0.0028262 secs] 239123K->169361K(253440K), 0.0028683 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 88.450: [GC 88.450: [DefNew: 70504K->597K(78656K), 0.0029739 secs] 239269K->169366K(253440K), 0.0030148 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 88.504: [GC 88.504: [DefNew: 70362K->597K(78656K), 0.0031161 secs] 239131K->169370K(253440K), 0.0031650 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 88.560: [GC 88.560: [DefNew: 70362K->597K(78656K), 0.0029015 secs] 239135K->169373K(253440K), 0.0029420 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 88.615: [GC 88.615: [DefNew: 70362K->597K(78656K), 0.0026776 secs] 239139K->169377K(253440K), 0.0027163 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 88.669: [GC 88.669: [DefNew: 70360K->597K(78656K), 0.0030092 secs] 239140K->169381K(253440K), 0.0030586 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 88.725: [GC 88.725: [DefNew: 70362K->597K(78656K), 0.0029850 secs] 239146K->169385K(253440K), 0.0030280 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 88.780: [GC 88.780: [DefNew: 70362K->597K(78656K), 0.0027917 secs] 239150K->169388K(253440K), 0.0028300 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 88.835: [GC 88.835: [DefNew: 70362K->597K(78656K), 0.0029122 secs] 239154K->169392K(253440K), 0.0029560 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 88.890: [GC 88.890: [DefNew: 70362K->597K(78656K), 0.0030267 secs] 239157K->169396K(253440K), 0.0031088 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 88.948: [GC 88.948: [DefNew: 70360K->597K(78656K), 0.0029752 secs] 239159K->169400K(253440K), 0.0030190 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 89.004: [GC 89.004: [DefNew: 70364K->597K(78656K), 0.0027849 secs] 239167K->169404K(253440K), 0.0028266 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 89.059: [GC 89.059: [DefNew: 70360K->597K(78656K), 0.0029058 secs] 239167K->169407K(253440K), 0.0029445 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 89.115: [GC 89.115: [DefNew: 70362K->597K(78656K), 0.0026968 secs] 239172K->169411K(253440K), 0.0027393 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 89.169: [GC 89.169: [DefNew: 70360K->597K(78656K), 0.0029620 secs] 239174K->169415K(253440K), 0.0030084 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 89.230: [GC 89.230: [DefNew: 70390K->597K(78656K), 0.0027393 secs] 239208K->169419K(253440K), 0.0027806 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 89.292: [GC 89.293: [DefNew: 70362K->596K(78656K), 0.0029862 secs] 239184K->169422K(253440K), 0.0030288 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 89.348: [GC 89.348: [DefNew: 70359K->596K(78656K), 0.0027666 secs] 239185K->169426K(253440K), 0.0028045 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 89.402: [GC 89.402: [DefNew: 70361K->596K(78656K), 0.0032314 secs] 239191K->169430K(253440K), 0.0032808 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 89.460: [GC 89.460: [DefNew: 70507K->597K(78656K), 0.0040517 secs] 239340K->169434K(253440K), 0.0040994 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 89.515: [GC 89.515: [DefNew: 70362K->597K(78656K), 0.0028819 secs] 239200K->169438K(253440K), 0.0029249 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 89.571: [GC 89.571: [DefNew: 70360K->597K(78656K), 0.0027832 secs] 239201K->169442K(253440K), 0.0028219 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 89.626: [GC 89.626: [DefNew: 70362K->597K(78656K), 0.0028951 secs] 239207K->169446K(253440K), 0.0029364 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 89.682: [GC 89.682: [DefNew: 70362K->597K(78656K), 0.0029045 secs] 239211K->169450K(253440K), 0.0029492 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 89.737: [GC 89.737: [DefNew: 70362K->597K(78656K), 0.0029411 secs] 239215K->169453K(253440K), 0.0029879 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 89.793: [GC 89.793: [DefNew: 70362K->597K(78656K), 0.0027734 secs] 239218K->169457K(253440K), 0.0028151 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 89.848: [GC 89.848: [DefNew: 70360K->597K(78656K), 0.0029258 secs] 239220K->169461K(253440K), 0.0029650 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 89.904: [GC 89.904: [DefNew: 70364K->597K(78656K), 0.0029679 secs] 239228K->169465K(253440K), 0.0030122 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 89.959: [GC 89.959: [DefNew: 70360K->597K(78656K), 0.0029398 secs] 239228K->169469K(253440K), 0.0029854 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 90.016: [GC 90.016: [DefNew: 70362K->597K(78656K), 0.0028066 secs] 239233K->169472K(253440K), 0.0028462 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 90.071: [GC 90.071: [DefNew: 70360K->597K(78656K), 0.0028466 secs] 239235K->169476K(253440K), 0.0029015 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 90.127: [GC 90.127: [DefNew: 70364K->597K(78656K), 0.0039155 secs] 239243K->169480K(253440K), 0.0039624 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 90.182: [GC 90.183: [DefNew: 70360K->597K(78656K), 0.0029181 secs] 239243K->169484K(253440K), 0.0029628 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 90.247: [GC 90.247: [DefNew: 70388K->597K(78656K), 0.0030731 secs] 239274K->169487K(253440K), 0.0031246 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 90.306: [GC 90.306: [DefNew: 70362K->596K(78656K), 0.0029552 secs] 239252K->169491K(253440K), 0.0029999 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 90.363: [GC 90.363: [DefNew: 70361K->596K(78656K), 0.0031395 secs] 239256K->169495K(253440K), 0.0031859 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 90.417: [GC 90.417: [DefNew: 70362K->596K(78656K), 0.0031084 secs] 239260K->169499K(253440K), 0.0031620 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 90.475: [GC 90.475: [DefNew: 70515K->597K(78656K), 0.0029943 secs] 239417K->169503K(253440K), 0.0030424 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 90.530: [GC 90.530: [DefNew: 70362K->597K(78656K), 0.0031608 secs] 239268K->169507K(253440K), 0.0032076 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 90.587: [GC 90.587: [DefNew: 70362K->597K(78656K), 0.0031280 secs] 239272K->169511K(253440K), 0.0031710 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 90.644: [GC 90.644: [DefNew: 70362K->597K(78656K), 0.0039990 secs] 239276K->169515K(253440K), 0.0040471 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 90.701: [GC 90.701: [DefNew: 70362K->597K(78656K), 0.0030761 secs] 239279K->169518K(253440K), 0.0031237 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 90.757: [GC 90.757: [DefNew: 70360K->597K(78656K), 0.0028015 secs] 239281K->169522K(253440K), 0.0028389 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 90.813: [GC 90.813: [DefNew: 70364K->597K(78656K), 0.0030978 secs] 239289K->169526K(253440K), 0.0031454 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 90.869: [GC 90.869: [DefNew: 70360K->597K(78656K), 0.0029352 secs] 239289K->169530K(253440K), 0.0029828 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 90.924: [GC 90.924: [DefNew: 70362K->597K(78656K), 0.0029892 secs] 239295K->169533K(253440K), 0.0030386 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 90.980: [GC 90.980: [DefNew: 70360K->597K(78656K), 0.0029254 secs] 239296K->169537K(253440K), 0.0029658 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 91.036: [GC 91.036: [DefNew: 70364K->597K(78656K), 0.0028487 secs] 239304K->169541K(253440K), 0.0028913 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 91.096: [GC 91.096: [DefNew: 70362K->597K(78656K), 0.0028194 secs] 239306K->169545K(253440K), 0.0028573 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 91.158: [GC 91.158: [DefNew: 70360K->597K(78656K), 0.0026814 secs] 239308K->169549K(253440K), 0.0027198 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 91.225: [GC 91.225: [DefNew: 70362K->597K(78656K), 0.0027291 secs] 239314K->169552K(253440K), 0.0027666 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 91.290: [GC 91.290: [DefNew: 70388K->597K(78656K), 0.0029709 secs] 239343K->169556K(253440K), 0.0030148 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 91.350: [GC 91.350: [DefNew: 70362K->596K(78656K), 0.0027708 secs] 239321K->169560K(253440K), 0.0028109 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 91.405: [GC 91.405: [DefNew: 70359K->596K(78656K), 0.0031267 secs] 239323K->169564K(253440K), 0.0031727 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 91.463: [GC 91.463: [DefNew: 70507K->597K(78656K), 0.0032468 secs] 239474K->169568K(253440K), 0.0032910 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 91.519: [GC 91.519: [DefNew: 70364K->597K(78656K), 0.0029015 secs] 239335K->169572K(253440K), 0.0029441 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 91.576: [GC 91.576: [DefNew: 70360K->597K(78656K), 0.0029130 secs] 239335K->169576K(253440K), 0.0029581 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 91.631: [GC 91.631: [DefNew: 70362K->597K(78656K), 0.0033072 secs] 239341K->169579K(253440K), 0.0033587 secs] [Times: user=0.00 sys=0.02, real=0.00 secs] 91.688: [GC 91.688: [DefNew: 70360K->597K(78656K), 0.0029960 secs] 239342K->169583K(253440K), 0.0030412 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 91.748: [GC 91.748: [DefNew: 70364K->597K(78656K), 0.0029564 secs] 239350K->169587K(253440K), 0.0030071 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 91.803: [GC 91.803: [DefNew: 70360K->597K(78656K), 0.0031169 secs] 239350K->169591K(253440K), 0.0031940 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 91.860: [GC 91.861: [DefNew: 70362K->597K(78656K), 0.0030297 secs] 239356K->169595K(253440K), 0.0030735 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 91.918: [GC 91.918: [DefNew: 70360K->597K(78656K), 0.0029058 secs] 239358K->169598K(253440K), 0.0029479 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 91.975: [GC 91.975: [DefNew: 70364K->597K(78656K), 0.0030684 secs] 239365K->169602K(253440K), 0.0031135 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 92.030: [GC 92.030: [DefNew: 70362K->597K(78656K), 0.0039985 secs] 239367K->169606K(253440K), 0.0040449 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 92.086: [GC 92.086: [DefNew: 70360K->597K(78656K), 0.0031156 secs] 239369K->169610K(253440K), 0.0031608 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 92.144: [GC 92.144: [DefNew: 70362K->597K(78656K), 0.0030475 secs] 239375K->169614K(253440K), 0.0030918 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 92.199: [GC 92.199: [DefNew: 70362K->597K(78656K), 0.0029854 secs] 239379K->169617K(253440K), 0.0030326 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 92.260: [GC 92.260: [DefNew: 70388K->597K(78656K), 0.0031033 secs] 239408K->169621K(253440K), 0.0031501 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 92.337: [GC 92.338: [DefNew: 70362K->596K(78656K), 0.0029956 secs] 239386K->169625K(253440K), 0.0030373 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 92.394: [GC 92.394: [DefNew: 70359K->596K(78656K), 0.0028121 secs] 239388K->169629K(253440K), 0.0028521 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 92.453: [GC 92.453: [DefNew: 70509K->597K(78656K), 0.0032574 secs] 239541K->169633K(253440K), 0.0033076 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 92.510: [GC 92.510: [DefNew: 70360K->597K(78656K), 0.0029530 secs] 239396K->169637K(253440K), 0.0029990 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 92.567: [GC 92.567: [DefNew: 70362K->597K(78656K), 0.0027951 secs] 239402K->169641K(253440K), 0.0028347 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 92.623: [GC 92.623: [DefNew: 70362K->597K(78656K), 0.0032566 secs] 239406K->169644K(253440K), 0.0033030 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 92.680: [GC 92.680: [DefNew: 70362K->597K(78656K), 0.0029288 secs] 239409K->169648K(253440K), 0.0029756 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 92.737: [GC 92.737: [DefNew: 70362K->597K(78656K), 0.0031199 secs] 239413K->169652K(253440K), 0.0031655 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 92.793: [GC 92.794: [DefNew: 70360K->597K(78656K), 0.0029794 secs] 239415K->169656K(253440K), 0.0030224 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 92.851: [GC 92.851: [DefNew: 70362K->597K(78656K), 0.0029275 secs] 239421K->169660K(253440K), 0.0029692 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 92.909: [GC 92.909: [DefNew: 70362K->597K(78656K), 0.0028998 secs] 239425K->169663K(253440K), 0.0029424 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 92.971: [GC 92.971: [DefNew: 70362K->597K(78656K), 0.0030663 secs] 239428K->169667K(253440K), 0.0031114 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 93.037: [GC 93.037: [DefNew: 70362K->597K(78656K), 0.0027419 secs] 239432K->169671K(253440K), 0.0027793 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 93.117: [GC 93.117: [DefNew: 70360K->597K(78656K), 0.0035869 secs] 239434K->169675K(253440K), 0.0036269 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 93.184: [GC 93.184: [DefNew: 70364K->597K(78656K), 0.0028641 secs] 239442K->169678K(253440K), 0.0029054 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 93.247: [GC 93.247: [DefNew: 70384K->597K(78656K), 0.0028645 secs] 239465K->169682K(253440K), 0.0029079 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 93.316: [GC 93.316: [DefNew: 70364K->597K(78656K), 0.0052143 secs] 239449K->169686K(253440K), 0.0052628 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 93.385: [GC 93.385: [DefNew: 70362K->596K(78656K), 0.0027977 secs] 239451K->169690K(253440K), 0.0028389 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 93.451: [GC 93.451: [DefNew: 70507K->597K(78656K), 0.0029194 secs] 239600K->169694K(253440K), 0.0029654 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 93.509: [GC 93.509: [DefNew: 70362K->597K(78656K), 0.0029313 secs] 239459K->169698K(253440K), 0.0029735 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 93.565: [GC 93.565: [DefNew: 70360K->597K(78656K), 0.0031263 secs] 239461K->169702K(253440K), 0.0031701 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 93.622: [GC 93.622: [DefNew: 70362K->597K(78656K), 0.0029275 secs] 239467K->169706K(253440K), 0.0029688 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 93.679: [GC 93.679: [DefNew: 70364K->597K(78656K), 0.0030109 secs] 239473K->169709K(253440K), 0.0030573 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 93.737: [GC 93.737: [DefNew: 70360K->597K(78656K), 0.0029403 secs] 239472K->169713K(253440K), 0.0029807 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 93.793: [GC 93.793: [DefNew: 70362K->597K(78656K), 0.0030412 secs] 239478K->169717K(253440K), 0.0030871 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 93.850: [GC 93.850: [DefNew: 70360K->597K(78656K), 0.0030131 secs] 239480K->169721K(253440K), 0.0030578 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 93.909: [GC 93.909: [DefNew: 70364K->597K(78656K), 0.0029471 secs] 239488K->169724K(253440K), 0.0029875 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 93.965: [GC 93.965: [DefNew: 70360K->597K(78656K), 0.0030458 secs] 239488K->169728K(253440K), 0.0030901 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 94.022: [GC 94.022: [DefNew: 70362K->597K(78656K), 0.0030054 secs] 239493K->169732K(253440K), 0.0030505 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 94.080: [GC 94.080: [DefNew: 70362K->597K(78656K), 0.0030105 secs] 239497K->169736K(253440K), 0.0030543 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 94.145: [GC 94.145: [DefNew: 70362K->597K(78656K), 0.0029569 secs] 239501K->169739K(253440K), 0.0029999 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 94.207: [GC 94.207: [DefNew: 70362K->597K(78656K), 0.0027683 secs] 239505K->169743K(253440K), 0.0028053 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 94.272: [GC 94.272: [DefNew: 70386K->597K(78656K), 0.0026716 secs] 239532K->169747K(253440K), 0.0027121 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 94.337: [GC 94.337: [DefNew: 70362K->596K(78656K), 0.0036320 secs] 239512K->169751K(253440K), 0.0036874 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 94.401: [GC 94.401: [DefNew: 70363K->596K(78656K), 0.0027721 secs] 239518K->169755K(253440K), 0.0028117 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 94.457: [GC 94.457: [DefNew: 70505K->597K(78656K), 0.0029790 secs] 239663K->169759K(253440K), 0.0030250 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 94.515: [GC 94.515: [DefNew: 70362K->597K(78656K), 0.0029692 secs] 239524K->169763K(253440K), 0.0030148 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 94.572: [GC 94.572: [DefNew: 70360K->597K(78656K), 0.0030458 secs] 239526K->169767K(253440K), 0.0030901 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 94.630: [GC 94.630: [DefNew: 70364K->597K(78656K), 0.0029160 secs] 239534K->169770K(253440K), 0.0029598 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 94.686: [GC 94.686: [DefNew: 70362K->597K(78656K), 0.0030228 secs] 239536K->169774K(253440K), 0.0030663 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 94.744: [GC 94.744: [DefNew: 70360K->597K(78656K), 0.0030037 secs] 239537K->169778K(253440K), 0.0030450 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 94.801: [GC 94.801: [DefNew: 70362K->597K(78656K), 0.0039147 secs] 239543K->169782K(253440K), 0.0039645 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 94.860: [GC 94.860: [DefNew: 70362K->597K(78656K), 0.0029305 secs] 239547K->169785K(253440K), 0.0029743 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 94.918: [GC 94.918: [DefNew: 70362K->597K(78656K), 0.0030535 secs] 239551K->169789K(253440K), 0.0030956 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 94.977: [GC 94.977: [DefNew: 70362K->597K(78656K), 0.0029164 secs] 239554K->169793K(253440K), 0.0029569 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 95.034: [GC 95.034: [DefNew: 70360K->597K(78656K), 0.0041199 secs] 239556K->169797K(253440K), 0.0041658 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 95.092: [GC 95.092: [DefNew: 70364K->597K(78656K), 0.0029807 secs] 239564K->169801K(253440K), 0.0030263 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 95.151: [GC 95.151: [DefNew: 70360K->597K(78656K), 0.0029445 secs] 239564K->169804K(253440K), 0.0029901 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 95.209: [GC 95.209: [DefNew: 70362K->597K(78656K), 0.0029428 secs] 239569K->169808K(253440K), 0.0029850 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 95.267: [GC 95.267: [DefNew: 70386K->597K(78656K), 0.0033809 secs] 239597K->169812K(253440K), 0.0034238 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 95.334: [GC 95.334: [DefNew: 70364K->596K(78656K), 0.0025857 secs] 239579K->169816K(253440K), 0.0026218 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 95.392: [GC 95.392: [DefNew: 70360K->596K(78656K), 0.0030850 secs] 239579K->169819K(253440K), 0.0031318 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 95.450: [GC 95.450: [DefNew: 70506K->597K(78656K), 0.0032519 secs] 239730K->169824K(253440K), 0.0032961 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 95.509: [GC 95.509: [DefNew: 70360K->597K(78656K), 0.0030607 secs] 239587K->169828K(253440K), 0.0031071 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 95.566: [GC 95.566: [DefNew: 70364K->597K(78656K), 0.0030067 secs] 239595K->169831K(253440K), 0.0030526 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 95.624: [GC 95.624: [DefNew: 70360K->597K(78656K), 0.0027606 secs] 239595K->169835K(253440K), 0.0027998 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 95.681: [GC 95.681: [DefNew: 70362K->597K(78656K), 0.0029466 secs] 239600K->169839K(253440K), 0.0029982 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 208.497: [GC 208.497: [DefNew: 70549K->926K(78656K), 0.0067911 secs] 239791K->170171K(253440K), 0.0068400 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 518.900: [GC 518.900: [DefNew: 70878K->1267K(78656K), 0.0090123 secs] 240123K->170517K(253440K), 0.0090715 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 540.704: [Full GC (System) 540.704: [Tenured: 169249K->35169K(174784K), 0.1363699 secs] 175870K->35169K(253440K), [Perm : 17122K->17122K(17152K)], 0.1364401 secs] [Times: user=0.13 sys=0.00, real=0.14 secs] ==== after full GC, loading one more image... ==== 780.866: [GC 780.867: [DefNew: 69952K->221K(78656K), 0.0032540 secs] 105121K->35390K(253440K), 0.0033047 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] [FINE] ImageCache.cleanReferenceQueue() - Removed 28/30 images. [FINE] ItemsDao.getImage() - Loading image BACKGROUND(id=15) [FINE] ItemsDao.getImage() - Loading image POSTER(id=15) 958.646: [GC 958.646: [DefNew: 69987K->6444K(78656K), 0.0133042 secs] 105156K->47688K(253440K), 0.0133587 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 958.706: [GC 958.706: [DefNew: 75927K->546K(78656K), 0.0095079 secs] 117171K->47689K(253440K), 0.0095594 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 958.757: [GC 958.757: [DefNew: 70465K->574K(78656K), 0.0020795 secs] 117608K->47717K(253440K), 0.0021204 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 958.798: [GC 958.798: [DefNew: 70232K->556K(78656K), 0.0020284 secs] 117375K->47699K(253440K), 0.0020778 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 958.845: [GC 958.845: [DefNew: 70418K->562K(78656K), 0.0020327 secs] 117561K->47704K(253440K), 0.0020787 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 958.887: [GC 958.887: [DefNew: 70360K->567K(78656K), 0.0022545 secs] 117503K->47710K(253440K), 0.0023004 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 958.931: [GC 958.931: [DefNew: 70361K->572K(78656K), 0.0018790 secs] 117504K->47715K(253440K), 0.0019207 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 958.972: [GC 958.972: [DefNew: 70382K->577K(78656K), 0.0020676 secs] 117525K->47720K(253440K), 0.0021110 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 959.017: [GC 959.017: [DefNew: 70379K->582K(78656K), 0.0021859 secs] 117522K->47725K(253440K), 0.0022319 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 959.058: [GC 959.058: [DefNew: 70384K->587K(78656K), 0.0021366 secs] 117527K->47730K(253440K), 0.0021821 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 959.104: [GC 959.104: [DefNew: 70383K->592K(78656K), 0.0020016 secs] 117526K->47735K(253440K), 0.0020433 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 959.146: [GC 959.146: [DefNew: 70389K->597K(78656K), 0.0023230 secs] 117532K->47740K(253440K), 0.0023673 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 959.190: [GC 959.190: [DefNew: 70396K->602K(78656K), 0.0021102 secs] 117539K->47745K(253440K), 0.0021536 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 959.232: [GC 959.232: [DefNew: 70397K->607K(78656K), 0.0020884 secs] 117540K->47750K(253440K), 0.0021344 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 959.276: [GC 959.277: [DefNew: 70405K->612K(78656K), 0.0020638 secs] 117548K->47755K(253440K), 0.0021059 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 959.317: [GC 959.318: [DefNew: 70406K->617K(78656K), 0.0022251 secs] 117549K->47760K(253440K), 0.0022719 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 959.361: [GC 959.361: [DefNew: 70411K->616K(78656K), 0.0020557 secs] 117554K->47765K(253440K), 0.0020991 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 959.410: [GC 959.410: [DefNew: 70114K->615K(78656K), 0.0022157 secs] 117263K->47770K(253440K), 0.0022604 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 959.463: [GC 959.463: [DefNew: 70410K->615K(78656K), 0.0020314 secs] 117564K->47775K(253440K), 0.0020744 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 959.503: [GC 959.503: [DefNew: 70410K->615K(78656K), 0.0023771 secs] 117570K->47780K(253440K), 0.0024209 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 959.549: [GC 959.549: [DefNew: 70411K->615K(78656K), 0.0019697 secs] 117576K->47785K(253440K), 0.0020084 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 959.589: [GC 959.589: [DefNew: 70413K->615K(78656K), 0.0023779 secs] 117583K->47790K(253440K), 0.0024243 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 959.634: [GC 959.634: [DefNew: 70410K->615K(78656K), 0.0020186 secs] 117585K->47795K(253440K), 0.0020587 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 959.675: [GC 959.675: [DefNew: 70413K->615K(78656K), 0.0024660 secs] 117593K->47800K(253440K), 0.0025167 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 959.718: [GC 959.718: [DefNew: 70411K->615K(78656K), 0.0022434 secs] 117596K->47805K(253440K), 0.0022907 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 959.764: [GC 959.764: [DefNew: 70031K->616K(78656K), 0.0023511 secs] 117221K->47811K(253440K), 0.0023958 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 959.804: [GC 959.804: [DefNew: 70414K->616K(78656K), 0.0024694 secs] 117609K->47816K(253440K), 0.0025167 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 959.848: [GC 959.848: [DefNew: 70410K->616K(78656K), 0.0022902 secs] 117610K->47821K(253440K), 0.0023349 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 959.889: [GC 959.889: [DefNew: 70414K->616K(78656K), 0.0023039 secs] 117619K->47826K(253440K), 0.0023507 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 959.933: [GC 959.933: [DefNew: 70410K->616K(78656K), 0.0020842 secs] 117620K->47831K(253440K), 0.0021268 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 959.974: [GC 959.974: [DefNew: 70414K->616K(78656K), 0.0023056 secs] 117629K->47836K(253440K), 0.0023511 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 960.019: [GC 960.019: [DefNew: 70410K->616K(78656K), 0.0020518 secs] 117630K->47841K(253440K), 0.0020944 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] ..... many more lines ending with.... 1086.953: [GC 1086.953: [DefNew: 70546K->1240K(78656K), 0.0025771 secs] 125014K->55711K(253440K), 0.0026201 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 1087.010: [GC 1087.010: [DefNew: 70550K->1240K(78656K), 0.0029309 secs] 125020K->55712K(253440K), 0.0029760 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 1087.065: [GC 1087.065: [DefNew: 70545K->1241K(78656K), 0.0024401 secs] 125017K->55714K(253440K), 0.0024797 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 1087.122: [GC 1087.122: [DefNew: 70552K->1240K(78656K), 0.0023869 secs] 125025K->55716K(253440K), 0.0024218 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 1087.181: [GC 1087.181: [DefNew: 70546K->1240K(78656K), 0.0024801 secs] 125021K->55717K(253440K), 0.0025210 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 1087.248: [GC 1087.248: [DefNew: 70550K->1240K(78656K), 0.0023677 secs] 125027K->55719K(253440K), 0.0024073 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 1087.309: [GC 1087.309: [DefNew: 70547K->1241K(78656K), 0.0026623 secs] 125026K->55721K(253440K), 0.0027083 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 1087.383: [GC 1087.383: [DefNew: 70550K->1240K(78656K), 0.0024384 secs] 125030K->55722K(253440K), 0.0024771 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 1087.461: [GC 1087.461: [DefNew: 70933K->1600K(78656K), 0.0028441 secs] 125416K->56084K(253440K), 0.0028849 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] From ysr1729 at gmail.com Fri Feb 17 09:36:49 2012 From: ysr1729 at gmail.com (Srinivas Ramakrishna) Date: Fri, 17 Feb 2012 09:36:49 -0800 Subject: regarding sudden memory increase In-Reply-To: <4F3E8283.2060707@oracle.com> References: <1329494066.11411.YahooMailClassic@web120004.mail.ne1.yahoo.com> <4F3E8283.2060707@oracle.com> Message-ID: Also, please use -XX:+PrintGCDetails for more detail on what's happening. If there's a crash there would usually be an hs_error log file in your $CWD which might tell you what may have caused the crash. -- ramki On Fri, Feb 17, 2012 at 8:38 AM, Paul Hohensee wrote: > Your question would be better answered by someone on the hotstpot-gc-use > list, cc'ed. > > You don't mention which version of the JDK you're using, but you should be > using the latest update of JDK 7, which is update 2. > > Paul > > On 2/17/12 10:54 AM, sakthivel wrote: > > Dear Sir, > I have the following setting for my system > > -Xms1536M > -Xmx1536M > -verbose:gc > -XX:+PrintGCDateStamps > -XX:NewSize=768M > -XX:MaxNewSize=768M > -XX:PermSize=256m > -XX:MaxPermSize=256m > -XX:+CMSClassUnloadingEnabled > -XX:+UseConcMarkSweepGC > -XX:+CMSIncrementalMode > -XX:+DisableExplicitGC > -XX:+ExplicitGCInvokesConcurrent > -XX:GCTimeRatio=19 > -Xincgc > -XX:SurvivorRatio=2 > -XX:MaxTenuringThreshold=50 > -XX:MaxGCPauseMillis=12000 > > and My logs as below > > 2012-02-16T09:53:47.907-0500: [GC 462167K->82462K(1376256K), 0.0554103 > secs] > 2012-02-16T09:53:52.275-0500: [GC 475678K->68672K(1376256K), 0.0541640 > secs] > 2012-02-16T09:53:53.289-0500: [GC 461888K->68574K(1376256K), 0.0539152 > secs] > 2012-02-16T09:53:56.471-0500: [GC 461790K->68683K(1376256K), 0.0556297 > secs] > 2012-02-16T09:53:58.281-0500: [GC 461899K->95837K(1376256K), 0.0757101 > secs] > > in certain case the tomcat memory increases as follows > > 2012-02-16T12:02:32.182-0500: [GC 454346K->97556K(1376256K), 0.1354231 > secs] > 2012-02-16T12:03:30.542-0500: [GC 490772K->128467K(1376256K), 0.2537510 > secs] > 2012-02-16T12:04:28.792-0500: [GC 521683K->223446K(1376256K), 0.3725598 > secs] > 2012-02-16T12:05:14.235-0500: [GC 616662K->265305K(1376256K), 0.3163817 > secs] > 2012-02-16T12:05:20.459-0500: [GC 452169K(1376256K), 0.2202838 secs] > 2012-02-16T12:05:26.949-0500: [GC 658521K->253437K(1376256K), 0.2352744 > secs] > 2012-02-16T12:05:48.524-0500: [GC 468368K(1376256K), 0.1812084 secs] > > and tomcat crashes, please help in this regard > > Thanks, > Sakthi > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120217/2e30dee3/attachment.html From ysr1729 at gmail.com Fri Feb 17 09:46:39 2012 From: ysr1729 at gmail.com (Srinivas Ramakrishna) Date: Fri, 17 Feb 2012 09:46:39 -0800 Subject: value of dense prefix address used? Message-ID: Hi Jo{h}n, all -- Is there some way i can get at the dense prefix value used for ParOld in each (major) collection? I couldn't find an obvious product flag for eliciting that info, but wondered if you knew/remembered. JMX would be fine too -- as long as the info can be obtained in a product build. I am seeing a curious looking log where one specific major gc seems to have greater user and real time, lower "parallelism" [=(user+sys)/real] and takes much longer than the rest of the ParOld's. It also lowers the footprint a tad more (but definitely not proportionately more vis-a-vis the user time ratios) than the gc's preceding (but not succeeding) that one, so one conjecture was that perhaps something happens with the dense prefix computation at that time and we somehow end up copying more. We'll see if we can get some data with printing the ParOld phase times, but i wondered if we might also be able to elicit the dense prefix address/size. I'll continue to dig around meanwhile. thanks for any info! -- ramki -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120217/755a4944/attachment.html From jon.masamitsu at oracle.com Fri Feb 17 12:10:33 2012 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Fri, 17 Feb 2012 12:10:33 -0800 Subject: value of dense prefix address used? In-Reply-To: References: Message-ID: <4F3EB439.1090709@oracle.com> Ramki, I didn't find a product flag that would print the end of the dense prefix. Don't know about jmx. The phase accounting (PrintParallelOldGCPhaseTimes) as you say is a good place to start. The summary phase is serial so look for an increase in that phase. Does this pattern repeat? You could also try changing HeapMaximumCompactionInterval and see if it affects the pattern. Jon On 2/17/2012 9:46 AM, Srinivas Ramakrishna wrote: > Hi Jo{h}n, all -- > > Is there some way i can get at the dense prefix value used for ParOld in > each (major) collection? I couldn't find an obvious product flag for > eliciting that info, but wondered if you knew/remembered. > JMX would be fine too -- as long as the info can be obtained in a product > build. > > I am seeing a curious looking log where one specific major gc seems to have > greater user and real time, lower "parallelism" [=(user+sys)/real] and > takes much longer than the rest of the ParOld's. It > also lowers the footprint a tad more (but definitely not proportionately > more vis-a-vis the user time ratios) than the gc's preceding (but not > succeeding) that one, so one conjecture was that perhaps > something happens with the dense prefix computation at that time and we > somehow end up copying more. We'll see if we can get some data with > printing the ParOld phase times, but i wondered if > we might also be able to elicit the dense prefix address/size. I'll > continue to dig around meanwhile. > > thanks for any info! > -- ramki > > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120217/50435576/attachment.html From Andreas.Loew at oracle.com Fri Feb 17 12:43:24 2012 From: Andreas.Loew at oracle.com (Andreas Loew) Date: Fri, 17 Feb 2012 21:43:24 +0100 Subject: why perm generation is continuously increasing? In-Reply-To: References: Message-ID: <4F3EBBEC.5070408@oracle.com> Hi Li, citing from here: http://kenai.com/projects/btrace/forums/forum/topics/1109-Tracing-simple-program-on-Windows "BTrace currently does not support tracing native methods. But, it may be feasible to support tracing native methods using native method prefixing support of java.lang.instrument. (http://java.sun.com/javase/6/docs/api/java/lang/instrument/Instrumentation.html#isNativeMethodPrefixSupported%28%29)." I have never tried this myself, but see http://docs.oracle.com/javase/6/docs/api/java/lang/instrument/Instrumentation.html#setNativeMethodPrefix%28java.lang.instrument.ClassFileTransformer,%20java.lang.String%29 on how you should probably be able to work around this by using byte code instrumentation and then have BTrace apply your probe into the intern() method of your instrumentation wrapper class... HTH & best regards, Andreas Am 17.02.2012 07:52, schrieb Li Li: > I want to use btrace to locate which function call String.intern() > but from > http://kenai.com/projects/btrace/forums/forum/topics/1109-Tracing-simple-program-on-Windows > it seems it's not feasible > any other methods? > I tried trace method call of String, constructor method and trim > method is ok. but native method like intern is not. > > On Fri, Feb 17, 2012 at 2:00 PM, Alexey Ragozin > > wrote: > > Hi, > > FYI > Normally permanent generation is collect only during Full GC. > -XX:+CMSClassUnloadingEnabled will enable permanent space > collection as a part of concurrent old space collection. > > Regards, > > Date: Thu, 16 Feb 2012 20:53:59 +0800 > From: Li Li > > Subject: Re: why perm generation is continuously increasing? > To: hotspot-gc-use > > Message-ID: > > > Content-Type: text/plain; charset="iso-8859-1" > > > I got it. we use String.intern for temporary variables. > I used jmap -permstat and found that. > btw jmap -permstat will cause jvm hang and should be carefully > used. > it hangs for a few minutes! > > On Thu, Feb 16, 2012 at 6:55 PM, Li Li > wrote: > > > hi all, > > I found our application's perm generation size is > continuously > > increasing and will reach 100%(we limit maxPermSize to > 256MB) and perform a > > full gc(which will pause for 10s).after gc, perm size usage > falls to about > > 50% and again increasing. what kind of objects will be > allocated to perm > > generation? we don't use reflection in our application. > > > -- Andreas Loew | Senior Java Architect ACS Principal Service Delivery Engineer ORACLE Deutschland B.V. & Co. KG From ysr1729 at gmail.com Fri Feb 17 12:48:29 2012 From: ysr1729 at gmail.com (Srinivas Ramakrishna) Date: Fri, 17 Feb 2012 12:48:29 -0800 Subject: value of dense prefix address used? In-Reply-To: <4F3EB439.1090709@oracle.com> References: <4F3EB439.1090709@oracle.com> Message-ID: Hi John, thanks for those suggestions... So far the pattern has not repeated, but occurred on two different servers (in each case it was the same full gc ordinal too, albeit at different time). There didn't seem anything external that would explain the difference observed. Yes, we'll play around a bit with the compaction related parameters and look at the phase times as well. I am also looking at how the dense prefix address is computed to see if it sheds a bit of light may be, but it could also be something happening early in the life of the process that doesn't happen later that causes this... it's all a bit of a mystery at the moment. Thanks! -- ramki On Fri, Feb 17, 2012 at 12:10 PM, Jon Masamitsu wrote: > ** > Ramki, > > I didn't find a product flag that would print the end of the dense prefix. > Don't know about jmx. > > The phase accounting (PrintParallelOldGCPhaseTimes) > as you say is a good place to start. The summary phase is > serial so look for an increase in that phase. Does this pattern > repeat? > > You could also try changing HeapMaximumCompactionInterval > and see if it affects the pattern. > > Jon > > > On 2/17/2012 9:46 AM, Srinivas Ramakrishna wrote: > > Hi Jo{h}n, all -- > > Is there some way i can get at the dense prefix value used for ParOld in > each (major) collection? I couldn't find an obvious product flag for > eliciting that info, but wondered if you knew/remembered. > JMX would be fine too -- as long as the info can be obtained in a product > build. > > I am seeing a curious looking log where one specific major gc seems to have > greater user and real time, lower "parallelism" [=(user+sys)/real] and > takes much longer than the rest of the ParOld's. It > also lowers the footprint a tad more (but definitely not proportionately > more vis-a-vis the user time ratios) than the gc's preceding (but not > succeeding) that one, so one conjecture was that perhaps > something happens with the dense prefix computation at that time and we > somehow end up copying more. We'll see if we can get some data with > printing the ParOld phase times, but i wondered if > we might also be able to elicit the dense prefix address/size. I'll > continue to dig around meanwhile. > > thanks for any info! > -- ramki > > > > _______________________________________________ > hotspot-gc-use mailing listhotspot-gc-use at openjdk.java.nethttp://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120217/237250a0/attachment.html From Peter.B.Kessler at Oracle.COM Fri Feb 17 18:08:39 2012 From: Peter.B.Kessler at Oracle.COM (Peter B. Kessler) Date: Fri, 17 Feb 2012 18:08:39 -0800 Subject: Java 7, soft references not being cleaned? In-Reply-To: <4F3E8655.6050204@xs4all.nl> References: <4F3E8655.6050204@xs4all.nl> Message-ID: <4F3F0827.5070900@Oracle.COM> I think you are on the edge of running out of memory. Why did you limit the heap to 256MB? Except for the frequency of the young generation collections, things go pretty well for the first 40 seconds. Young generations are cleaning up pretty well, and not promoting that much to the old generation. E.g., 39.943: [GC 39.943: [DefNew: 70189K->369K(78656K), 0.0030518 secs] 237834K->168023K(253440K), 0.0030990 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 40.005: [GC 40.005: [DefNew: 70191K->369K(78656K), 0.0030995 secs] 237845K->168032K(253440K), 0.0031433 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] where the young generation collection at 40.005 turns 70191KB of young generation into 369KB of survivor objects in the young generation and 168032KB-168023KB=9KB of promoted objects in the old generation. That pattern continues, piling up about 9KB in the old generation until the old generation fills up. Actually there's a change in the pattern just before the old generation fills up 40.329: [GC 40.329: [DefNew: 70198K->368K(78656K), 0.0029632 secs] 237896K->168075K(253440K), 0.0030054 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 40.391: [GC 40.391: [DefNew: 70189K->368K(78656K), 0.0027960 secs] 237896K->168083K(253440K), 0.0028419 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 40.453: [GC 40.453: [DefNew: 70189K->368K(78656K), 0.0030837 secs] 237904K->168092K(253440K), 0.0031416 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 40.517: [GC 40.517: [DefNew: 70258K->439K(78656K), 0.0030203 secs] 237982K->168172K(253440K), 0.0030637 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 40.596: [GC 40.596: [DefNew: 70162K->581K(78656K), 0.0030880 secs] 237895K->168322K(253440K), 0.0031284 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 40.666: [GC 40.666: [DefNew: 70403K->581K(78656K), 0.0030961 secs] 238144K->168331K(253440K), 0.0031403 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] [FINE] ItemsDao.getImage() - Loading image BACKGROUND(id=13) [FINE] ItemsDao.getImage() - Loading image POSTER(id=13) 42.161: [GC 42.161: [DefNew: 70294K->6140K(78656K), 0.0157502 secs] 238044K->179973K(253440K), 0.0158060 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 42.230: [GC 42.230: [DefNew: 75850K->75850K(78656K), 0.0000175 secs]42.230: [Tenured: 173833K->174734K(174784K), 0.1042122 secs] 249684K->179972K(253440K), [Perm : 17126K->17126K(17152K)], 0.1043190 secs] [Times: user=0.09 sys=0.00, real=0.11 secs] The collections at 40.329, 40.391, 40.453 promote about 9KB each. Then 40.517 promotes 80KB, 40.596 promotes 150KB, and then 40.666 settles back to 9KB. I can't explain that hiccup. At 42.161 things are completely different, in a bad way. We leave 6140KB in the young generation survivor space, and promote 11642KB to the old generation. (Okay, maybe that's the new image coming in.) But then the old generation is full, so the young generation collection at 42.230 bails out (in 0.0000175 seconds!) and starts an old generation collection, which manages to make negative progress by promoting some objects to the old generation, and maybe collecting a few objects from the old generation, but ending up with more in the old generation than it started with (173833K->174734K(174784K)). Things are going to go south from here. We launch into a long series of full collections (because there isn't enough free space in the old generation for a young generation collection to think about starting). Those collections, too, have a pattern 42.379: [Full GC 42.379: [Tenured: 174734K->173476K(174784K), 0.1745277 secs] 253047K->178289K(253440K), [Perm : 17126K->17126K(17152K)], 0.1745924 secs] [Times: user=0.17 sys=0.00, real=0.18 secs] 42.597: [Full GC 42.597: [Tenured: 174556K->174556K(174784K), 0.0899911 secs] 253024K->179018K(253440K), [Perm : 17126K->17126K(17152K)], 0.0900558 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 42.737: [Full GC 42.737: [Tenured: 174556K->174556K(174784K), 0.0914405 secs] 253032K->179387K(253440K), [Perm : 17126K->17126K(17152K)], 0.0915065 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 42.870: [Full GC 42.870: [Tenured: 174556K->174556K(174784K), 0.0859870 secs] 253043K->179396K(253440K), [Perm : 17126K->17126K(17152K)], 0.0860500 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 42.998: [Full GC 42.998: [Tenured: 174556K->173476K(174784K), 0.0873824 secs] 253050K->178325K(253440K), [Perm : 17126K->17126K(17152K)], 0.0874467 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 43.128: [Full GC 43.128: [Tenured: 174556K->174556K(174784K), 0.0861173 secs] 253062K->179055K(253440K), [Perm : 17126K->17126K(17152K)], 0.0861837 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 43.267: [Full GC 43.267: [Tenured: 174556K->174556K(174784K), 0.0887080 secs] 253087K->179423K(253440K), [Perm : 17126K->17126K(17152K)], 0.0887732 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 43.401: [Full GC 43.401: [Tenured: 174556K->174556K(174784K), 0.0862837 secs] 252860K->179431K(253440K), [Perm : 17126K->17126K(17152K)], 0.0863480 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 43.530: [Full GC 43.530: [Tenured: 174556K->173476K(174784K), 0.0863131 secs] 253082K->178360K(253440K), [Perm : 17126K->17126K(17152K)], 0.0863782 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 43.661: [Full GC 43.661: [Tenured: 174556K->174556K(174784K), 0.0861245 secs] 253101K->179091K(253440K), [Perm : 17126K->17126K(17152K)], 0.0861849 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 43.788: [Full GC 43.788: [Tenured: 174556K->174556K(174784K), 0.0881231 secs] 253102K->179459K(253440K), [Perm : 17126K->17126K(17152K)], 0.0881831 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 43.923: [Full GC 43.923: [Tenured: 174556K->174556K(174784K), 0.0863314 secs] 253107K->179467K(253440K), [Perm : 17126K->17126K(17152K)], 0.0863940 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 44.050: [Full GC 44.050: [Tenured: 174556K->173476K(174784K), 0.0884058 secs] 253115K->178395K(253440K), [Perm : 17126K->17126K(17152K)], 0.0884718 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] The first collection of the series manages to knock the old generation occupancy down to 173476KB, leaving 1208KB free. The next three collections only knock the old generation occupancy down to 174556KB, leaving an even scantier 228KB free. Then the cycle repeats: a little down, then 3 down hardly at all. All those collections (except the first) start with an occupancy of 174556KB, and the groups of three end with an occupancy of 174556KB, so we aren't exactly out of memory, but we aren't making much progress. Note that your code is running in there. E.g., the collection at 42.998 takes 0.0874467 seconds, meaning it is finished at 43.085, so your code runs until the next collection at 43.128. In that 0.042 seconds, you allocate your way through the remaining free space, and cause the next collection. No one is happy about this situation. Things continue badly for entirely too long, until 75.670: [Full GC 75.670: [Tenured: 174555K->174555K(174784K), 0.0932953 secs] 252752K->181366K(253440K), [Perm : 17126K->17126K(17152K)], 0.0933557 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 75.813: [Full GC 75.813: [Tenured: 174555K->174555K(174784K), 0.0967613 secs] 252756K->181370K(253440K), [Perm : 17126K->17126K(17152K)], 0.0968234 secs] [Times: user=0.09 sys=0.00, real=0.10 secs] 75.964: [Full GC 75.964: [Tenured: 174555K->173241K(174784K), 0.1877438 secs] 252759K->180060K(253440K), [Perm : 17126K->17124K(17152K)], 0.1878076 secs] [Times: user=0.19 sys=0.00, real=0.19 secs] 76.203: [Full GC 76.203: [Tenured: 174321K->174321K(174784K), 0.0906815 secs] 252550K->180604K(253440K), [Perm : 17124K->17124K(17152K)], 0.0907416 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 76.355: [Full GC 76.355: [Tenured: 174321K->174321K(174784K), 0.0943489 secs] 252545K->181148K(253440K), [Perm : 17124K->17124K(17152K)], 0.0944085 secs] [Times: user=0.09 sys=0.00, real=0.10 secs] 76.501: [Full GC 76.501: [Tenured: 174321K->174321K(174784K), 0.0958222 secs] 252683K->181152K(253440K), [Perm : 17124K->17124K(17152K)], 0.0958839 secs] [Times: user=0.08 sys=0.00, real=0.10 secs] 76.645: [Full GC 76.645: [Tenured: 174321K->173194K(174784K), 0.1750156 secs] 252541K->180029K(253440K), [Perm : 17124K->17121K(17152K)], 0.1750760 secs] [Times: user=0.17 sys=0.00, real=0.17 secs] 76.869: [Full GC 76.869: [Tenured: 174274K->174274K(174784K), 0.0912524 secs] 252500K->180573K(253440K), [Perm : 17121K->17121K(17152K)], 0.0913175 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 77.008: [Full GC 77.008: [Tenured: 174274K->174274K(174784K), 0.0930352 secs] 252503K->181117K(253440K), [Perm : 17121K->17121K(17152K)], 0.0930973 secs] [Times: user=0.08 sys=0.00, real=0.09 secs] 77.153: [Full GC 77.153: [Tenured: 174274K->174530K(174784K), 0.1806351 secs] 252506K->175070K(253440K), [Perm : 17121K->17121K(17152K)], 0.1806969 secs] [Times: user=0.19 sys=0.00, real=0.18 secs] 77.398: [Full GC 77.398: [Tenured: 174530K->167748K(174784K), 0.1809714 secs] 252974K->167748K(253440K), [Perm : 17121K->17121K(17152K)], 0.1810366 secs] [Times: user=0.17 sys=0.00, real=0.18 secs] 77.628: [GC 77.628: [DefNew: 69911K->545K(78656K), 0.0025767 secs] 237659K->168293K(253440K), 0.0026206 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 77.679: [GC 77.679: [DefNew: 70310K->549K(78656K), 0.0027793 secs] 238058K->168297K(253440K), 0.0028211 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 77.729: [GC 77.729: [DefNew: 70314K->553K(78656K), 0.0029654 secs] 238062K->168301K(253440K), 0.0030177 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] we fight our way down to the collection at 77.398 when the jam breaks, we make a significant dent in the old generation occupancy, and young generation collections can run again. Those young generation collections are leaving about 550KB in the young generation survivors, and promoting 4KB per collection. The old generation is substantially full, but that's what memory is for. After that, young generation collections continue at that frequency until 95.566: [GC 95.566: [DefNew: 70364K->597K(78656K), 0.0030067 secs] 239595K->169831K(253440K), 0.0030526 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 95.624: [GC 95.624: [DefNew: 70360K->597K(78656K), 0.0027606 secs] 239595K->169835K(253440K), 0.0027998 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 95.681: [GC 95.681: [DefNew: 70362K->597K(78656K), 0.0029466 secs] 239600K->169839K(253440K), 0.0029982 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 208.497: [GC 208.497: [DefNew: 70549K->926K(78656K), 0.0067911 secs] 239791K->170171K(253440K), 0.0068400 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 518.900: [GC 518.900: [DefNew: 70878K->1267K(78656K), 0.0090123 secs] 240123K->170517K(253440K), 0.0090715 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 540.704: [Full GC (System) 540.704: [Tenured: 169249K->35169K(174784K), 0.1363699 secs] 175870K->35169K(253440K), [Perm : 17122K->17122K(17152K)], 0.1364401 secs] [Times: user=0.13 sys=0.00, real=0.14 secs] I assume your application went nearly idle between 95.681 and 540.704, because it was only allocating enough to cause the occasional young generation collections shown. Then you used VisualVM to cause a collection, and that really made progress, reducing the old generation from 169249KB to 35169KB. You never actually ran into an out of memory situation, because we were able to run collections and recover enough space to keep you limping along. We were never compelled (by the JVM spec) to try clearing all your SoftReferences. Absent running out of memory, SoftReferences are cleared if they haven't been used in a while. I have no idea what the usage pattern is for your SoftReferences, but that might be why they stuck around. If that's what happened. Maybe the little progress we made during the series of full collections was cleaning up SoftReferences as they became eligible. If you didn't use them in the apparently idle time leading up to the collection at 540.704, that might explain why we decided they could be cleared by the VisualVM collection. Or it may be that VisualVM asks for a full collection that clears all SoftReferences. (It can do that, but I forget the details.) You can tune the urgency with which to collector tries to clear SoftReferences, but it's not for the faint of heart. I suggest you try making your heap enough larger to allow your program to run without that series of old generation collections, watch how frequently you use your SoftReferences, and report back if things go better (or not). If you made the young generation larger, the young generation collections would happen less often, though you don't seem to be complaining about the frequency or duration of the young generation pauses. ... peter P.S. Please try not to have your mailer wrap the GC log lines. It makes them more difficult to analyze. John Hendrikx wrote: > I've written a fairly small program that runs in 256 MB -- this program > will load several large pictures, which are being "cached" using > SoftReferences. > > The more of these pictures get loaded, the more the GC starts panicing > (while in reality, more than 200 MB in the VM is only held by > SoftReferences). > > In the output below you can see the GC getting progressively longer > after each image is loaded (see Loading Image in log), doing more and > more GC action. The performance of the program is severely impacted > (the CPU is maxed while this is happening). > > To be sure that the references are REALLY softly reachable, I've used > VisualVM to trigger a Full GC, the result of which is visible on the > last line, copied here: > > 540.704: [Full GC (System) 540.704: [Tenured: 169249K->35169K(174784K), > 0.1363699 secs] 175870K->35169K(253440K), [Perm : > 17122K->17122K(17152K)], 0.1364401 secs] [Times: user=0.13 sys=0.00, > real=0.14 secs] > > Unfortunately, even AFTER this Full GC, the GC still goes crazy doing > much the same thing as when the images got loaded. See the 2nd part of > the log. > > This seems like a pretty serious issue as I was under the impression > that SoftReferences would be cleared in time to prevent long GC cycles. > I've tested this with 32-bit JDK 1.7u2 and the 1.7u4 developer preview > (which was said to have fixed some issues related to references). > > I'll be happy to provide any more information you might need. > --John Hendrikx > > The output: > > .... > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From hjohn at xs4all.nl Sat Feb 18 15:21:00 2012 From: hjohn at xs4all.nl (John Hendrikx) Date: Sun, 19 Feb 2012 00:21:00 +0100 Subject: Java 7, soft references not being cleaned? In-Reply-To: <4F3E8655.6050204@xs4all.nl> References: <4F3E8655.6050204@xs4all.nl> Message-ID: <4F40325C.1090303@xs4all.nl> Please disregard this post, it turns out the issue had nothing to do with SoftReferences or the garbage collector at all. It only looked as if the GC was having trouble cleaning up things despite the heap not being full yet. Instead it was a piece of code which was doing exponentionally more work every time it was triggered (related to JavaFX's binding mechanism), so it was this code that was using all the CPU (doing basically nothing but triggering a lot of GC activity) and not the GC. My apologies, --John Hendrikx On 17/02/2012 17:54, John Hendrikx wrote: > I've written a fairly small program that runs in 256 MB -- this program > will load several large pictures, which are being "cached" using > SoftReferences. > > The more of these pictures get loaded, the more the GC starts panicing > (while in reality, more than 200 MB in the VM is only held by > SoftReferences). > > From ysr1729 at gmail.com Sun Feb 19 02:13:30 2012 From: ysr1729 at gmail.com (Srinivas Ramakrishna) Date: Sun, 19 Feb 2012 02:13:30 -0800 Subject: value of dense prefix address used? In-Reply-To: References: <4F3EB439.1090709@oracle.com> Message-ID: Hi Jon -- After looking at the code and the pattern we observed, we are pretty confident now that the maximal compaction was the root of the problem. We are going to effectively turn off the maximal compaction and see if it does any harm (don't think it will), and use that to work around the problem of extreme degradation when doing parallel compaction. It;s interesting why maximal compaction would degrade parallel compaction by so much... some experiments would be useful and perhaps help correct a specific issue the lack of initial parallelism may be causing to make the whole collection so much more inefficient. Hopefully we'll be able to collect some numbers that might help you folks address the issue. later. -- ramki On Fri, Feb 17, 2012 at 12:48 PM, Srinivas Ramakrishna wrote: > Hi John, thanks for those suggestions... > > So far the pattern has not repeated, but occurred on two different servers > (in each case it was the > same full gc ordinal too, albeit at different time). There didn't seem > anything external that would > explain the difference observed. Yes, we'll play around a bit with the > compaction related parameters and look at the phase times > as well. I am also looking at how the dense prefix address is computed to > see if it sheds a bit of > light may be, but it could also be something happening early in the life > of the process that doesn't happen > later that causes this... it's all a bit of a mystery at the moment. > Thanks! > > -- ramki > > > On Fri, Feb 17, 2012 at 12:10 PM, Jon Masamitsu wrote: > >> ** >> Ramki, >> >> I didn't find a product flag that would print the end of the dense prefix. >> Don't know about jmx. >> >> The phase accounting (PrintParallelOldGCPhaseTimes) >> as you say is a good place to start. The summary phase is >> serial so look for an increase in that phase. Does this pattern >> repeat? >> >> You could also try changing HeapMaximumCompactionInterval >> and see if it affects the pattern. >> >> Jon >> >> >> On 2/17/2012 9:46 AM, Srinivas Ramakrishna wrote: >> >> Hi Jo{h}n, all -- >> >> Is there some way i can get at the dense prefix value used for ParOld in >> each (major) collection? I couldn't find an obvious product flag for >> eliciting that info, but wondered if you knew/remembered. >> JMX would be fine too -- as long as the info can be obtained in a product >> build. >> >> I am seeing a curious looking log where one specific major gc seems to have >> greater user and real time, lower "parallelism" [=(user+sys)/real] and >> takes much longer than the rest of the ParOld's. It >> also lowers the footprint a tad more (but definitely not proportionately >> more vis-a-vis the user time ratios) than the gc's preceding (but not >> succeeding) that one, so one conjecture was that perhaps >> something happens with the dense prefix computation at that time and we >> somehow end up copying more. We'll see if we can get some data with >> printing the ParOld phase times, but i wondered if >> we might also be able to elicit the dense prefix address/size. I'll >> continue to dig around meanwhile. >> >> thanks for any info! >> -- ramki >> >> >> >> _______________________________________________ >> hotspot-gc-use mailing listhotspot-gc-use at openjdk.java.nethttp://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120219/836bf9ae/attachment.html From fancyerii at gmail.com Sun Feb 19 18:44:12 2012 From: fancyerii at gmail.com (Li Li) Date: Mon, 20 Feb 2012 10:44:12 +0800 Subject: why perm generation is continuously increasing? In-Reply-To: <4F3EBBEC.5070408@oracle.com> References: <4F3EBBEC.5070408@oracle.com> Message-ID: I am not familiar with Instrumentation. is there any tutorial for me to get started? On Sat, Feb 18, 2012 at 4:43 AM, Andreas Loew wrote: > Hi Li, > > citing from here: http://kenai.com/projects/**btrace/forums/forum/topics/* > *1109-Tracing-simple-program-**on-Windows > > "BTrace currently does not support tracing native methods. But, it may be > feasible to support tracing native methods using native method prefixing > support of java.lang.instrument. > (http://java.sun.com/javase/6/**docs/api/java/lang/instrument/** > Instrumentation.html#**isNativeMethodPrefixSupported%**28%29) > ." > > I have never tried this myself, but see http://docs.oracle.com/javase/** > 6/docs/api/java/lang/**instrument/Instrumentation.** > html#setNativeMethodPrefix%**28java.lang.instrument.** > ClassFileTransformer,%20java.**lang.String%29on how you should probably be able to work around this by using byte code > instrumentation and then have BTrace apply your probe into the intern() > method of your instrumentation wrapper class... > > HTH & best regards, > > Andreas > > > Am 17.02.2012 07:52, schrieb Li Li: > >> I want to use btrace to locate which function call String.intern() >> but from http://kenai.com/projects/**btrace/forums/forum/topics/** >> 1109-Tracing-simple-program-**on-Windows >> it seems it's not feasible >> any other methods? >> I tried trace method call of String, constructor method and trim method >> is ok. but native method like intern is not. >> >> On Fri, Feb 17, 2012 at 2:00 PM, Alexey Ragozin > alexey.ragozin at gmail.**com >> wrote: >> >> Hi, >> >> FYI >> Normally permanent generation is collect only during Full GC. >> -XX:+CMSClassUnloadingEnabled will enable permanent space >> collection as a part of concurrent old space collection. >> >> Regards, >> >> Date: Thu, 16 Feb 2012 20:53:59 +0800 >> From: Li Li > >> >> Subject: Re: why perm generation is continuously increasing? >> To: hotspot-gc-use >> >> >> >> Message-ID: >> > gmail.com <0cTCpCB74wD03VgYe_GhgPj78mcU_pk3syaQ at mail.gmail.com> >> >> >> >> >> Content-Type: text/plain; charset="iso-8859-1" >> >> >> I got it. we use String.intern for temporary variables. >> I used jmap -permstat and found that. >> btw jmap -permstat will cause jvm hang and should be carefully >> used. >> it hangs for a few minutes! >> >> On Thu, Feb 16, 2012 at 6:55 PM, Li Li > > wrote: >> >> > hi all, >> > I found our application's perm generation size is >> continuously >> > increasing and will reach 100%(we limit maxPermSize to >> 256MB) and perform a >> > full gc(which will pause for 10s).after gc, perm size usage >> falls to about >> > 50% and again increasing. what kind of objects will be >> allocated to perm >> > generation? we don't use reflection in our application. >> > >> >> -- > Andreas Loew | Senior Java Architect > ACS Principal Service Delivery Engineer > ORACLE Deutschland B.V. & Co. KG > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120220/1d2f7755/attachment.html From Andreas.Loew at oracle.com Mon Feb 20 06:37:56 2012 From: Andreas.Loew at oracle.com (Andreas Loew) Date: Mon, 20 Feb 2012 15:37:56 +0100 Subject: why perm generation is continuously increasing? In-Reply-To: References: <4F3EBBEC.5070408@oracle.com> Message-ID: <4F425AC4.1010109@oracle.com> Hi Li, to the best of my knowledge, there is no tutorial describing this specific use case (setNativeMethodPrefix()), but there are some tutorials which show how to write a ClassFileTransformer and configure instrumentation with a "-javaagent" from the command line: http://www.javalobby.org/java/forums/t19309.html http://schuchert.wikispaces.com/WritingYourOwnJavaAgent http://dhruba.name/2010/02/07/creation-dynamic-loading-and-instrumentation-with-javaagents HTH & best regards, Andreas Am 20.02.2012 03:44, schrieb Li Li: > I am not familiar with Instrumentation. is there any tutorial for me > to get started? > > On Sat, Feb 18, 2012 at 4:43 AM, Andreas Loew > wrote: > > Hi Li, > > citing from here: > http://kenai.com/projects/btrace/forums/forum/topics/1109-Tracing-simple-program-on-Windows > > "BTrace currently does not support tracing native methods. But, it > may be feasible to support tracing native methods using native > method prefixing support of java.lang.instrument. > (http://java.sun.com/javase/6/docs/api/java/lang/instrument/Instrumentation.html#isNativeMethodPrefixSupported%28%29) > ." > > I have never tried this myself, but see > http://docs.oracle.com/javase/6/docs/api/java/lang/instrument/Instrumentation.html#setNativeMethodPrefix%28java.lang.instrument.ClassFileTransformer,%20java.lang.String%29 > on how you should probably be able to work around this by using > byte code instrumentation and then have BTrace apply your probe > into the intern() method of your instrumentation wrapper class... > > HTH & best regards, > > Andreas > > Am 17.02.2012 07:52, schrieb Li Li: > > I want to use btrace to locate which function call String.intern() > but from > http://kenai.com/projects/btrace/forums/forum/topics/1109-Tracing-simple-program-on-Windows > it seems it's not feasible > any other methods? > I tried trace method call of String, constructor method and > trim method is ok. but native method like intern is not. > -- Andreas Loew | Senior Java Architect ACS Principal Service Delivery Engineer ORACLE Deutschland B.V. & Co. KG From jon.masamitsu at oracle.com Mon Feb 20 09:34:09 2012 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Mon, 20 Feb 2012 09:34:09 -0800 Subject: value of dense prefix address used? In-Reply-To: References: <4F3EB439.1090709@oracle.com> Message-ID: <4F428411.80504@oracle.com> Ramki, This is speculation on my part but we don't do anything at the start of the compaction phase to explicitly create more parallelism. What I mean is that we compact into a chunk when it becomes available and it becomes available when it is empty or when we can start compacting it into itself. This conceivably could result in near serial compaction of the former dense prefix. The extreme example is 1 small dead object at the beginning of the heap could allow only 1 chunk to be compacted at a time in the dense prefix. If not allowing maximal compactions does not work, you might try more frequent maximal compactions. If there is some adverse situation building up in the dense prefix, more frequent maximal compactions might avoid the more extreme degradation. Jon On 02/19/12 02:13, Srinivas Ramakrishna wrote: > Hi Jon -- > > After looking at the code and the pattern we observed, we are pretty > confident now that the maximal > compaction was the root of the problem. We are going to effectively > turn off the maximal compaction > and see if it does any harm (don't think it will), and use that to > work around the problem of extreme degradation > when doing parallel compaction. It;s interesting why maximal > compaction would degrade parallel compaction > by so much... some experiments would be useful and perhaps help > correct a specific issue the lack of initial parallelism > may be causing to make the whole collection so much more inefficient. > Hopefully we'll be able to collect some > numbers that might help you folks address the issue. > > later. > -- ramki > > On Fri, Feb 17, 2012 at 12:48 PM, Srinivas Ramakrishna > > wrote: > > Hi John, thanks for those suggestions... > > So far the pattern has not repeated, but occurred on two different > servers (in each case it was the > same full gc ordinal too, albeit at different time). There didn't > seem anything external that would > explain the difference observed. Yes, we'll play around a bit with > the compaction related parameters and look at the phase times > as well. I am also looking at how the dense prefix address is > computed to see if it sheds a bit of > light may be, but it could also be something happening early in > the life of the process that doesn't happen > later that causes this... it's all a bit of a mystery at the > moment. Thanks! > > -- ramki > > > On Fri, Feb 17, 2012 at 12:10 PM, Jon Masamitsu > > wrote: > > Ramki, > > I didn't find a product flag that would print the end of the > dense prefix. > Don't know about jmx. > > The phase accounting (PrintParallelOldGCPhaseTimes) > as you say is a good place to start. The summary phase is > serial so look for an increase in that phase. Does this pattern > repeat? > > You could also try changing HeapMaximumCompactionInterval > and see if it affects the pattern. > > Jon > > > On 2/17/2012 9:46 AM, Srinivas Ramakrishna wrote: >> Hi Jo{h}n, all -- >> >> Is there some way i can get at the dense prefix value used for ParOld in >> each (major) collection? I couldn't find an obvious product flag for >> eliciting that info, but wondered if you knew/remembered. >> JMX would be fine too -- as long as the info can be obtained in a product >> build. >> >> I am seeing a curious looking log where one specific major gc seems to have >> greater user and real time, lower "parallelism" [=(user+sys)/real] and >> takes much longer than the rest of the ParOld's. It >> also lowers the footprint a tad more (but definitely not proportionately >> more vis-a-vis the user time ratios) than the gc's preceding (but not >> succeeding) that one, so one conjecture was that perhaps >> something happens with the dense prefix computation at that time and we >> somehow end up copying more. We'll see if we can get some data with >> printing the ParOld phase times, but i wondered if >> we might also be able to elicit the dense prefix address/size. I'll >> continue to dig around meanwhile. >> >> thanks for any info! >> -- ramki >> >> >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120220/60a9becf/attachment-0001.html From taras.tielkes at gmail.com Tue Feb 21 08:55:15 2012 From: taras.tielkes at gmail.com (Taras Tielkes) Date: Tue, 21 Feb 2012 17:55:15 +0100 Subject: Promotion failures: indication of CMS fragmentation? In-Reply-To: References: <4EF9FCAC.3030208@oracle.com> <4F06A270.3010701@oracle.com> <4F0DBEC4.7040907@oracle.com> <4F1ECE7B.3040502@oracle.com> <4F1F2ED7.6060308@oracle.com> <4F20F78D.9070905@oracle.com> Message-ID: Hi, We've removed the "-XX:+CMSScavengeBeforeRemark" setting from 50% of our production nodes. After running for a few weeks, it seems that there's no impact from removing this option. Which is good, since it seems we can remove it from the other nodes as well, simplifying our overall JVM configuration ;-) However, we're still seeing promotion failures on all nodes, once every day or so. There's still the "Magic 1026": this accounts for ~60% of the promotion failures that we're seeing (single ParNew thread thread, 1026 failure size): -------------------- 2012-02-06T09:13:51.806+0100: 328095.085: [GC 328095.086: [ParNew: 359895K->29357K(368640K), 0.0429070 secs] 3471021K->3143476K(5201920K), 0.0434950 secs] [Times: user=0.32 sys=0.00, real=0.04 secs] 2012-02-06T09:13:55.922+0100: 328099.201: [GC 328099.201: [ParNew: 357037K->31817K(368640K), 0.0429130 secs] 3471156K->3148946K(5201920K), 0.0434930 secs] [Times: user=0.31 sys=0.00, real=0.04 secs] 2012-02-06T09:13:59.044+0100: 328102.324: [GC 328102.324: [ParNew (promotion failure size = 1026) (promotion failed): 359497K->368640K(368640K), 0.2226790 secs]328102.547: [CMS: 3125609K->451515K(4833280K), 5.6225880 secs] 3476626K->4515 15K(5201920K), [CMS Perm : 124373K->124353K(262144K)], 5.8459380 secs] [Times: user=6.20 sys=0.01, real=5.85 secs] 2012-02-06T09:14:05.243+0100: 328108.522: [GC 328108.523: [ParNew: 327680K->40960K(368640K), 0.0319160 secs] 779195K->497658K(5201920K), 0.0325360 secs] [Times: user=0.21 sys=0.01, real=0.03 secs] 2012-02-06T09:14:07.836+0100: 328111.116: [GC 328111.116: [ParNew: 368640K->32785K(368640K), 0.0744670 secs] 825338K->520234K(5201920K), 0.0750390 secs] [Times: user=0.40 sys=0.02, real=0.08 secs] -------------------- Given the 1026 word size, I'm wondering if I should be hunting for an overuse of BufferedInputStream/BufferedOutoutStream, since both have 8192 as a default buffer size. The second group of promotion failures look like this (multiple ParNew threads, small failure sizes): -------------------- 2012-02-06T09:50:15.773+0100: 328756.964: [GC 328756.964: [ParNew: 356116K->29934K(368640K), 0.0461100 secs] 3203863K->2880162K(5201920K), 0.0468870 secs] [Times: user=0.34 sys=0.01, real=0.05 secs] 2012-02-06T09:50:19.153+0100: 328760.344: [GC 328760.344: [ParNew: 357614K->30359K(368640K), 0.0454680 secs] 3207842K->2882892K(5201920K), 0.0462280 secs] [Times: user=0.33 sys=0.01, real=0.05 secs] 2012-02-06T09:50:22.658+0100: 328763.849: [GC 328763.849: [ParNew (1: promotion failure size = 25) (4: promotion failure size = 25) (6: promotion failure size = 25) (7: promotion failure size = 144) (promotion failed): 358039K->358358 K(368640K), 0.2148680 secs]328764.064: [CMS: 2854709K->446750K(4833280K), 5.8368270 secs] 3210572K->446750K(5201920K), [CMS Perm : 124670K->124644K(262144K)], 6.0525230 secs] [Times: user=6.32 sys=0.00, real=6.05 secs] 2012-02-06T09:50:29.896+0100: 328771.086: [GC 328771.087: [ParNew: 327680K->22569K(368640K), 0.0227080 secs] 774430K->469319K(5201920K), 0.0235020 secs] [Times: user=0.16 sys=0.00, real=0.02 secs] 2012-02-06T09:50:31.076+0100: 328772.266: [GC 328772.267: [ParNew: 350249K->22264K(368640K), 0.0235480 secs] 796999K->469014K(5201920K), 0.0243000 secs] [Times: user=0.18 sys=0.01, real=0.02 secs] -------------------- We're going to try to double the new size on a single node, to see the effects of that. Beyond this experiment, is there any additional data I can collect to better understand the nature of the promotion failures? Am I facing collecting free list statistics at this point? Thanks, Taras On Wed, Feb 1, 2012 at 1:16 PM, Taras Tielkes wrote: > Hi, > > We've been running the new release in production for almost a week > now. The good news is that the frequency of promotion failures has > decreased. > In addition, the promotion failures with large sizes (exceeding 1M > words) are completely gone. This is nice, as it shows that the > elimination of large buffer abuse in our application has paid off. > > There are still remaining, occasional promotion failures though, which > I've grouped into two buckets: > > 1) "Magic 1026" promotion failure > > On every machine, for every couple of days, we see a promotion failure > with 1026 size. The failure size is *always* this number. > --------------- > 2012-01-27T01:28:34.135+0100: 39574.835: [GC 39574.835: [ParNew > (promotion failure size = 1026) ?(promotion failed): > 354844K->368640K(368640K), 0.2103220 secs]39575.046: [CMS: > 2729550K->301144K(4833280K), 4.0552810 secs] 3068244K->301144K > (5201920K), [CMS Perm : 124003K->123976K(262144K)], 4.2664030 secs] > [Times: user=4.54 sys=0.01, real=4.27 secs] > --------------- > > 2) "Parallel" promotion failure > > In this case all 8 ParNew collection threads fail. Typically the > reported failure sizes will be between 8 and 30, with some threads > sometimes reporting a larger failure size: > --------------- > 2012-01-28T00:05:04.418+0100: 120965.117: [GC 120965.118: [ParNew (0: > promotion failure size = 4) ?(1: promotion failure size = 4) ?(2: > promotion failure size = 9) ?(3: promotion failure size = 1027) ?(4: > promotion failure size = 7) ?(5: > promotion failure size = 4) ?(6: promotion failure size = 7) ?(7: > promotion failure size = 4) ?(promotion failed): > 368640K->368640K(368640K), 2.2517580 secs]120967.370: [CMS: > 3210551K->295337K(4833280K), 4.3763650 secs] 3542272K->295337K( > 5201920K), [CMS Perm : 124446K->124411K(262144K)], 6.6289510 secs] > [Times: user=7.49 sys=0.52, real=6.63 secs] > --------------- > > Both of these events are occurring near the configured (and fixed) CMS > initiating occupancy fraction (68%). > The next thing we'll try is to remove the CMSScavengeBeforeRemark > flag, which we've had enabled up till now. > > If there are other things to investigate, or other data to collect, > I'd be grateful for any suggestions. > > Kind regards, > Taras > > On Thu, Jan 26, 2012 at 9:22 PM, Taras Tielkes wrote: >> Hi Jon, >> >> Thanks for clearing up the word size question. >> Over the past weeks, I've seen promotion failures exceeding 10M words, >> so arrays of over 80 megabytes in size :) >> Today, we deployed a production release that eliminates the huge >> buffer allocations - instead data is processed in a streaming fashion. >> >> We'll see how things are performing after collecting operations data >> for a few weeks. >> >> Thanks for your help, >> Taras >> >> On Thu, Jan 26, 2012 at 7:49 AM, Jon Masamitsu wrote: >>> >>> >>> On 1/25/2012 2:41 AM, Taras Tielkes wrote: >>>> Hi Jon, >>>> >>>> At the risk of asking a stupid question, what's the word size on x64 >>>> when using CompressedOops? >>> >>> Word size is the same with and without CompressedOops (8 bytes). ?With >>> CompressedOops >>> we can just point to words with a 32bit reference (i.e., map the 32bit >>> reference to a full >>> 64bit address). >>> >>> Jon >>> >>>> Thanks, >>>> Taras >>>> >>>> On Tue, Jan 24, 2012 at 11:21 PM, Jon Masamitsu >>>> ?wrote: >>>>> On 01/24/12 10:15, Taras Tielkes wrote: >>>>>> Hi Jon, >>>>>> >>>>>> Xmx is 5g, PermGen is 256m, new is 400m. >>>>>> >>>>>> The overall tenured gen usage is at the point when I would expect the >>>>>> CMS to kick in though. >>>>>> Does this mean I'd have to lower the CMS initiating occupancy setting >>>>>> (currently at 68%)? >>>>> I don't have any quick answers as to what to try next. >>>>> >>>>>> In addition, are the promotion failure sizes expressed in bytes? If >>>>>> so, I'm surprised to see such odd-sized (7, for example) sizes. >>>>> It's in words. >>>>> >>>>> Jon >>>>>> Thanks, >>>>>> Taras >>>>>> >>>>>> On Tue, Jan 24, 2012 at 4:30 PM, Jon Masamitsu ? ?wrote: >>>>>>> Taras, >>>>>>> >>>>>>> The pattern makes sense if the tenured (cms) gen is in fact full. >>>>>>> Multiple ?GC workers try to get a chunk of space for >>>>>>> an allocation and there is no space. >>>>>>> >>>>>>> Jon >>>>>>> >>>>>>> >>>>>>> On 01/24/12 04:22, Taras Tielkes wrote: >>>>>>>> Hi Jon, >>>>>>>> >>>>>>>> While inspecting our production logs for promotion failures, I saw the >>>>>>>> following one today: >>>>>>>> -------- >>>>>>>> 2012-01-24T08:37:26.118+0100: 1222467.411: [GC 1222467.411: [ParNew: >>>>>>>> 349623K->20008K(368640K), 0.0294350 secs] >>>>>>>> 3569266K->3239650K(5201920K), 0.0298770 secs] [Times: user=0.21 >>>>>>>> sys=0.00, real=0.03 secs] >>>>>>>> 2012-01-24T08:37:27.497+0100: 1222468.790: [GC 1222468.791: [ParNew: >>>>>>>> 347688K->40960K(368640K), 0.0536700 secs] >>>>>>>> 3567330K->3284097K(5201920K), 0.0541200 secs] [Times: user=0.36 >>>>>>>> sys=0.00, real=0.05 secs] >>>>>>>> 2012-01-24T08:37:28.716+0100: 1222470.009: [GC 1222470.010: [ParNew >>>>>>>> (0: promotion failure size = 6) ?(1: promotion failure size = 6) ?(2: >>>>>>>> promotion failure size = 7) ?(3: promotion failure size = 7) ?(4: >>>>>>>> promotion failure size = 9) ?(5: p >>>>>>>> romotion failure size = 9) ?(6: promotion failure size = 6) ?(7: >>>>>>>> promotion failure size = 9) ?(promotion failed): >>>>>>>> 368640K->368640K(368640K), 3.1475760 secs]1222473.157: [CMS: >>>>>>>> 3315844K->559299K(4833280K), 5.9647110 secs] 3611777K->559299K( >>>>>>>> 5201920K), [CMS Perm : 118085K->118072K(262144K)], 9.1128700 secs] >>>>>>>> [Times: user=10.17 sys=1.10, real=9.11 secs] >>>>>>>> 2012-01-24T08:37:38.601+0100: 1222479.894: [GC 1222479.895: [ParNew: >>>>>>>> 327680K->40960K(368640K), 0.0635680 secs] 886979K->624773K(5201920K), >>>>>>>> 0.0641090 secs] [Times: user=0.37 sys=0.00, real=0.07 secs] >>>>>>>> 2012-01-24T08:37:40.642+0100: 1222481.936: [GC 1222481.936: [ParNew: >>>>>>>> 368640K->38479K(368640K), 0.0771480 secs] 952453K->659708K(5201920K), >>>>>>>> 0.0776360 secs] [Times: user=0.40 sys=0.01, real=0.08 secs] >>>>>>>> -------- >>>>>>>> >>>>>>>> It's different from the others in two ways: >>>>>>>> 1) a "parallel" promotion failure in all 8 ParNew threads? >>>>>>>> 2) the very small size of the promoted object >>>>>>>> >>>>>>>> Do such an promotion failure pattern ring a bell? It does not make sense >>>>>>>> to me. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Taras >>>>>>>> >>>>>>>> On Wed, Jan 11, 2012 at 5:54 PM, Jon Masamitsu >>>>>>>> ? ?wrote: >>>>>>>>> Taras, >>>>>>>>> >>>>>>>>>> I assume that the large sizes for the promotion failures during ParNew >>>>>>>>>> are confirming that eliminating large array allocations might help >>>>>>>>>> here. Do you agree? >>>>>>>>> I agree that eliminating the large array allocation will help but you >>>>>>>>> are still having >>>>>>>>> promotion failures when the allocation size is small (I think it was >>>>>>>>> 1026). ?That >>>>>>>>> says that you are filling up the old (cms) generation faster than the GC >>>>>>>>> can >>>>>>>>> collect it. ?The large arrays are aggrevating the problem but not >>>>>>>>> necessarily >>>>>>>>> the cause. >>>>>>>>> >>>>>>>>> If these are still your heap sizes, >>>>>>>>> >>>>>>>>>> -Xms5g >>>>>>>>>> -Xmx5g >>>>>>>>>> -Xmn400m >>>>>>>>> Start by increasing the young gen size as may already have been >>>>>>>>> suggested. ?If you have a test setup where you can experiment, >>>>>>>>> try doubling the young gen size to start. >>>>>>>>> >>>>>>>>> If you have not seen this, it might be helpful. >>>>>>>>> >>>>>>>>> http://blogs.oracle.com/jonthecollector/entry/what_the_heck_s_a >>>>>>>>>> I'm not sure what to make of the concurrent mode >>>>>>>>> The concurrent mode failure is a consequence of the promotion failure. >>>>>>>>> Once the promotion failure happens the concurrent mode failure is >>>>>>>>> inevitable. >>>>>>>>> >>>>>>>>> Jon >>>>>>>>> >>>>>>>>> >>>>>>>>>> . >>>>>>>>> On 1/11/2012 3:00 AM, Taras Tielkes wrote: >>>>>>>>>> Hi Jon, >>>>>>>>>> >>>>>>>>>> We've added the -XX:+PrintPromotionFailure flag to our production >>>>>>>>>> application yesterday. >>>>>>>>>> The application is running on 4 (homogenous) nodes. >>>>>>>>>> >>>>>>>>>> In the gc logs of 3 out of 4 nodes, I've found a promotion failure >>>>>>>>>> event during ParNew: >>>>>>>>>> >>>>>>>>>> node-002 >>>>>>>>>> ------- >>>>>>>>>> 2012-01-11T09:39:14.353+0100: 102975.594: [GC 102975.594: [ParNew: >>>>>>>>>> 357592K->23382K(368640K), 0.0298150 secs] >>>>>>>>>> 3528237K->3194027K(5201920K), 0.0300860 secs] [Times: user=0.22 >>>>>>>>>> sys=0.01, real=0.03 secs] >>>>>>>>>> 2012-01-11T09:39:17.489+0100: 102978.730: [GC 102978.730: [ParNew: >>>>>>>>>> 351062K->39795K(368640K), 0.0401170 secs] >>>>>>>>>> 3521707K->3210439K(5201920K), 0.0403800 secs] [Times: user=0.28 >>>>>>>>>> sys=0.00, real=0.04 secs] >>>>>>>>>> 2012-01-11T09:39:19.869+0100: 102981.110: [GC 102981.110: [ParNew (4: >>>>>>>>>> promotion failure size = 4281460) ?(promotion failed): >>>>>>>>>> 350134K->340392K(368640K), 0.1378780 secs]102981.248: [CMS: >>>>>>>>>> 3181346K->367952K(4833280K), 4.7036230 secs] 3520778K >>>>>>>>>> ->367952K(5201920K), [CMS Perm : 116828K->116809K(262144K)], 4.8418590 >>>>>>>>>> secs] [Times: user=5.10 sys=0.00, real=4.84 secs] >>>>>>>>>> 2012-01-11T09:39:25.264+0100: 102986.504: [GC 102986.505: [ParNew: >>>>>>>>>> 327680K->40960K(368640K), 0.0415470 secs] 695632K->419560K(5201920K), >>>>>>>>>> 0.0418770 secs] [Times: user=0.26 sys=0.01, real=0.04 secs] >>>>>>>>>> 2012-01-11T09:39:26.035+0100: 102987.276: [GC 102987.276: [ParNew: >>>>>>>>>> 368640K->40960K(368640K), 0.0925740 secs] 747240K->481611K(5201920K), >>>>>>>>>> 0.0928570 secs] [Times: user=0.54 sys=0.01, real=0.09 secs] >>>>>>>>>> >>>>>>>>>> node-003 >>>>>>>>>> ------- >>>>>>>>>> 2012-01-10T17:48:28.369+0100: 45929.686: [GC 45929.686: [ParNew: >>>>>>>>>> 346950K->21342K(368640K), 0.0333090 secs] >>>>>>>>>> 2712364K->2386756K(5201920K), 0.0335740 secs] [Times: user=0.23 >>>>>>>>>> sys=0.00, real=0.03 secs] >>>>>>>>>> 2012-01-10T17:48:32.933+0100: 45934.250: [GC 45934.250: [ParNew: >>>>>>>>>> 345070K->32211K(368640K), 0.0369260 secs] >>>>>>>>>> 2710484K->2397625K(5201920K), 0.0372380 secs] [Times: user=0.25 >>>>>>>>>> sys=0.00, real=0.04 secs] >>>>>>>>>> 2012-01-10T17:48:34.201+0100: 45935.518: [GC 45935.518: [ParNew (0: >>>>>>>>>> promotion failure size = 1266955) ?(promotion failed): >>>>>>>>>> 359891K->368640K(368640K), 0.1395570 secs]45935.658: [CMS: >>>>>>>>>> 2387690K->348838K(4833280K), 4.5680670 secs] 2725305K->3 >>>>>>>>>> 48838K(5201920K), [CMS Perm : 116740K->116715K(262144K)], 4.7079640 >>>>>>>>>> secs] [Times: user=5.03 sys=0.00, real=4.71 secs] >>>>>>>>>> 2012-01-10T17:48:40.572+0100: 45941.889: [GC 45941.889: [ParNew: >>>>>>>>>> 327680K->40960K(368640K), 0.0486510 secs] 676518K->405004K(5201920K), >>>>>>>>>> 0.0489930 secs] [Times: user=0.26 sys=0.00, real=0.05 secs] >>>>>>>>>> 2012-01-10T17:48:41.959+0100: 45943.276: [GC 45943.277: [ParNew: >>>>>>>>>> 360621K->40960K(368640K), 0.0833240 secs] 724666K->479857K(5201920K), >>>>>>>>>> 0.0836120 secs] [Times: user=0.48 sys=0.01, real=0.08 secs] >>>>>>>>>> >>>>>>>>>> node-004 >>>>>>>>>> ------- >>>>>>>>>> 2012-01-10T18:59:02.338+0100: 50163.649: [GC 50163.649: [ParNew: >>>>>>>>>> 358429K->40960K(368640K), 0.0629910 secs] >>>>>>>>>> 3569331K->3283304K(5201920K), 0.0632710 secs] [Times: user=0.40 >>>>>>>>>> sys=0.02, real=0.06 secs] >>>>>>>>>> 2012-01-10T18:59:08.137+0100: 50169.448: [GC 50169.448: [ParNew: >>>>>>>>>> 368640K->40960K(368640K), 0.0819780 secs] >>>>>>>>>> 3610984K->3323445K(5201920K), 0.0822430 secs] [Times: user=0.40 >>>>>>>>>> sys=0.00, real=0.08 secs] >>>>>>>>>> 2012-01-10T18:59:13.945+0100: 50175.256: [GC 50175.256: [ParNew (6: >>>>>>>>>> promotion failure size = 2788662) ?(promotion failed): >>>>>>>>>> 367619K->364864K(368640K), 0.2024350 secs]50175.458: [CMS: >>>>>>>>>> 3310044K->330922K(4833280K), 4.5104170 secs] >>>>>>>>>> 3650104K->330922K(5201920K), [CMS Perm : 116747K->116728K(262144K)], >>>>>>>>>> 4.7132220 secs] [Times: user=4.99 sys=0.01, real=4.72 secs] >>>>>>>>>> 2012-01-10T18:59:20.539+0100: 50181.850: [GC 50181.850: [ParNew: >>>>>>>>>> 327680K->37328K(368640K), 0.0270660 secs] 658602K->368251K(5201920K), >>>>>>>>>> 0.0273800 secs] [Times: user=0.15 sys=0.00, real=0.02 secs] >>>>>>>>>> 2012-01-10T18:59:25.183+0100: 50186.494: [GC 50186.494: [ParNew: >>>>>>>>>> 363504K->15099K(368640K), 0.0388710 secs] 694427K->362063K(5201920K), >>>>>>>>>> 0.0391790 secs] [Times: user=0.18 sys=0.00, real=0.04 secs] >>>>>>>>>> >>>>>>>>>> On a fourth node, I've found a different event: promotion failure >>>>>>>>>> during CMS, with a much smaller size: >>>>>>>>>> >>>>>>>>>> node-001 >>>>>>>>>> ------- >>>>>>>>>> 2012-01-10T18:30:07.471+0100: 48428.764: [GC 48428.764: [ParNew: >>>>>>>>>> 354039K->40960K(368640K), 0.0667340 secs] >>>>>>>>>> 3609061K->3318149K(5201920K), 0.0670150 secs] [Times: user=0.37 >>>>>>>>>> sys=0.01, real=0.06 secs] >>>>>>>>>> 2012-01-10T18:30:08.706+0100: 48429.999: [GC 48430.000: [ParNew: >>>>>>>>>> 368640K->40960K(368640K), 0.2586390 secs] >>>>>>>>>> 3645829K->3417273K(5201920K), 0.2589050 secs] [Times: user=0.73 >>>>>>>>>> sys=0.13, real=0.26 secs] >>>>>>>>>> 2012-01-10T18:30:08.974+0100: 48430.267: [GC [1 CMS-initial-mark: >>>>>>>>>> 3376313K(4833280K)] 3427492K(5201920K), 0.0743900 secs] [Times: >>>>>>>>>> user=0.07 sys=0.00, real=0.07 secs] >>>>>>>>>> 2012-01-10T18:30:09.049+0100: 48430.342: [CMS-concurrent-mark-start] >>>>>>>>>> 2012-01-10T18:30:10.009+0100: 48431.302: [CMS-concurrent-mark: >>>>>>>>>> 0.933/0.960 secs] [Times: user=4.59 sys=0.13, real=0.96 secs] >>>>>>>>>> 2012-01-10T18:30:10.009+0100: 48431.302: [CMS-concurrent-preclean-start] >>>>>>>>>> 2012-01-10T18:30:10.089+0100: 48431.382: [CMS-concurrent-preclean: >>>>>>>>>> 0.060/0.080 secs] [Times: user=0.34 sys=0.02, real=0.08 secs] >>>>>>>>>> 2012-01-10T18:30:10.089+0100: 48431.382: >>>>>>>>>> [CMS-concurrent-abortable-preclean-start] >>>>>>>>>> 2012-01-10T18:30:10.586+0100: 48431.880: [GC 48431.880: [ParNew: >>>>>>>>>> 368640K->40960K(368640K), 0.1214420 secs] >>>>>>>>>> 3744953K->3490912K(5201920K), 0.1217480 secs] [Times: user=0.66 >>>>>>>>>> sys=0.05, real=0.12 secs] >>>>>>>>>> 2012-01-10T18:30:12.785+0100: 48434.078: >>>>>>>>>> [CMS-concurrent-abortable-preclean: 2.526/2.696 secs] [Times: >>>>>>>>>> user=10.72 sys=0.48, real=2.70 secs] >>>>>>>>>> 2012-01-10T18:30:12.787+0100: 48434.081: [GC[YG occupancy: 206521 K >>>>>>>>>> (368640 K)]2012-01-10T18:30:12.788+0100: 48434.081: [GC 48434.081: >>>>>>>>>> [ParNew (promotion failure size = 1026) ?(promotion failed): >>>>>>>>>> 206521K->206521K(368640K), 0.1667280 secs] >>>>>>>>>> ? ? 3656474K->3696197K(5201920K), 0.1670260 secs] [Times: user=0.48 >>>>>>>>>> sys=0.04, real=0.17 secs] >>>>>>>>>> 48434.248: [Rescan (parallel) , 0.1972570 secs]48434.445: [weak refs >>>>>>>>>> processing, 0.0011570 secs]48434.446: [class unloading, 0.0277750 >>>>>>>>>> secs]48434.474: [scrub symbol& ? ? ? ?string tables, 0.0088370 secs] [1 >>>>>>>>>> CMS-remark: 3489675K(4833280K)] 36961 >>>>>>>>>> 97K(5201920K), 0.4088040 secs] [Times: user=1.62 sys=0.05, real=0.41 >>>>>>>>>> secs] >>>>>>>>>> 2012-01-10T18:30:13.197+0100: 48434.490: [CMS-concurrent-sweep-start] >>>>>>>>>> 2012-01-10T18:30:17.427+0100: 48438.720: [Full GC 48438.720: >>>>>>>>>> [CMS2012-01-10T18:30:21.636+0100: 48442.929: [CMS-concurrent-sweep: >>>>>>>>>> 7.949/8.439 secs] [Times: user=15.89 sys=1.57, real=8.44 secs] >>>>>>>>>> ? ? (concurrent mode failure): 2505348K->334385K(4833280K), 8.6109050 >>>>>>>>>> secs] 2873988K->334385K(5201920K), [CMS Perm : >>>>>>>>>> 117788K->117762K(262144K)], 8.6112520 secs] [Times: user=8.61 >>>>>>>>>> sys=0.00, real=8.61 secs] >>>>>>>>>> 2012-01-10T18:30:26.716+0100: 48448.009: [GC 48448.010: [ParNew: >>>>>>>>>> 327680K->40960K(368640K), 0.0407520 secs] 662065K->394656K(5201920K), >>>>>>>>>> 0.0411550 secs] [Times: user=0.25 sys=0.00, real=0.04 secs] >>>>>>>>>> 2012-01-10T18:30:28.825+0100: 48450.118: [GC 48450.118: [ParNew: >>>>>>>>>> 368639K->40960K(368640K), 0.0662780 secs] 722335K->433355K(5201920K), >>>>>>>>>> 0.0666190 secs] [Times: user=0.35 sys=0.00, real=0.06 secs] >>>>>>>>>> >>>>>>>>>> I assume that the large sizes for the promotion failures during ParNew >>>>>>>>>> are confirming that eliminating large array allocations might help >>>>>>>>>> here. Do you agree? >>>>>>>>>> I'm not sure what to make of the concurrent mode failure. >>>>>>>>>> >>>>>>>>>> Thanks in advance for any suggestions, >>>>>>>>>> Taras >>>>>>>>>> >>>>>>>>>> On Fri, Jan 6, 2012 at 8:27 AM, Jon Masamitsu >>>>>>>>>> ? ? ?wrote: >>>>>>>>>>> On 1/5/2012 3:32 PM, Taras Tielkes wrote: >>>>>>>>>>>> Hi Jon, >>>>>>>>>>>> >>>>>>>>>>>> We've enabled the PrintPromotionFailure flag in our preprod >>>>>>>>>>>> environment, but so far, no failures yet. >>>>>>>>>>>> We know that the load we generate there is not representative. But >>>>>>>>>>>> perhaps we'll catch something, given enough patience. >>>>>>>>>>>> >>>>>>>>>>>> The flag will also be enabled in our production environment next week >>>>>>>>>>>> - so one way or the other, we'll get more diagnostic data soon. >>>>>>>>>>>> I'll also do some allocation profiling of the application in isolation >>>>>>>>>>>> - I know that there is abusive large byte[] and char[] allocation in >>>>>>>>>>>> there. >>>>>>>>>>>> >>>>>>>>>>>> I've got two questions for now: >>>>>>>>>>>> >>>>>>>>>>>> 1) From googling around on the output to expect >>>>>>>>>>>> (http://blog.ragozin.info/2011/10/java-cg-hotspots-cms-and-heap.html), >>>>>>>>>>>> I see that -XX:+PrintPromotionFailure will generate output like this: >>>>>>>>>>>> ------- >>>>>>>>>>>> 592.079: [ParNew (0: promotion failure size = 2698) ?(promotion >>>>>>>>>>>> failed): 135865K->134943K(138240K), 0.1433555 secs] >>>>>>>>>>>> ------- >>>>>>>>>>>> In that example line, what does the "0" stand for? >>>>>>>>>>> It's the index of the GC worker thread ?that experienced the promotion >>>>>>>>>>> failure. >>>>>>>>>>> >>>>>>>>>>>> 2) Below is a snippet of (real) gc log from our production >>>>>>>>>>>> application: >>>>>>>>>>>> ------- >>>>>>>>>>>> 2011-12-30T22:42:12.684+0100: 2136581.585: [GC 2136581.585: [ParNew: >>>>>>>>>>>> 345951K->40960K(368640K), 0.0676780 secs] >>>>>>>>>>>> 3608692K->3323692K(5201920K), 0.0680220 secs] [Times: user=0.36 >>>>>>>>>>>> sys=0.01, real=0.06 secs] >>>>>>>>>>>> 2011-12-30T22:42:22.984+0100: 2136591.886: [GC 2136591.886: [ParNew: >>>>>>>>>>>> 368640K->40959K(368640K), 0.0618880 secs] >>>>>>>>>>>> 3651372K->3349928K(5201920K), 0.0622330 secs] [Times: user=0.31 >>>>>>>>>>>> sys=0.00, real=0.06 secs] >>>>>>>>>>>> 2011-12-30T22:42:23.052+0100: 2136591.954: [GC [1 CMS-initial-mark: >>>>>>>>>>>> 3308968K(4833280K)] 3350041K(5201920K), 0.0377420 secs] [Times: >>>>>>>>>>>> user=0.04 sys=0.00, real=0.04 secs] >>>>>>>>>>>> 2011-12-30T22:42:23.090+0100: 2136591.992: [CMS-concurrent-mark-start] >>>>>>>>>>>> 2011-12-30T22:42:24.076+0100: 2136592.978: [CMS-concurrent-mark: >>>>>>>>>>>> 0.986/0.986 secs] [Times: user=2.05 sys=0.04, real=0.99 secs] >>>>>>>>>>>> 2011-12-30T22:42:24.076+0100: 2136592.978: >>>>>>>>>>>> [CMS-concurrent-preclean-start] >>>>>>>>>>>> 2011-12-30T22:42:24.099+0100: 2136593.000: [CMS-concurrent-preclean: >>>>>>>>>>>> 0.021/0.023 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] >>>>>>>>>>>> 2011-12-30T22:42:24.099+0100: 2136593.001: >>>>>>>>>>>> [CMS-concurrent-abortable-preclean-start] >>>>>>>>>>>> ? ? ?CMS: abort preclean due to time 2011-12-30T22:42:29.335+0100: >>>>>>>>>>>> 2136598.236: [CMS-concurrent-abortable-preclean: 5.209/5.236 secs] >>>>>>>>>>>> [Times: user=5.70 sys=0.23, real=5.23 secs] >>>>>>>>>>>> 2011-12-30T22:42:29.340+0100: 2136598.242: [GC[YG occupancy: 123870 K >>>>>>>>>>>> (368640 K)]2011-12-30T22:42:29.341+0100: 2136598.242: [GC 2136598.242: >>>>>>>>>>>> [ParNew (promotion failed): 123870K->105466K(368640K), 7.4939280 secs] >>>>>>>>>>>> 3432839K->3423755K(5201920 >>>>>>>>>>>> K), 7.4942670 secs] [Times: user=9.08 sys=2.10, real=7.49 secs] >>>>>>>>>>>> 2136605.737: [Rescan (parallel) , 0.0644050 secs]2136605.801: [weak >>>>>>>>>>>> refs processing, 0.0034280 secs]2136605.804: [class unloading, >>>>>>>>>>>> 0.0289480 secs]2136605.833: [scrub symbol& ? ? ? ? ?string tables, >>>>>>>>>>>> 0.0093940 >>>>>>>>>>>> secs] [1 CMS-remark: 3318289K(4833280K >>>>>>>>>>>> )] 3423755K(5201920K), 7.6077990 secs] [Times: user=9.54 sys=2.10, >>>>>>>>>>>> real=7.61 secs] >>>>>>>>>>>> 2011-12-30T22:42:36.949+0100: 2136605.850: >>>>>>>>>>>> [CMS-concurrent-sweep-start] >>>>>>>>>>>> 2011-12-30T22:42:45.006+0100: 2136613.907: [Full GC 2136613.908: >>>>>>>>>>>> [CMS2011-12-30T22:42:51.038+0100: 2136619.939: [CMS-concurrent-sweep: >>>>>>>>>>>> 12.231/14.089 secs] [Times: user=15.14 sys=5.36, real=14.08 secs] >>>>>>>>>>>> ? ? ?(concurrent mode failure): 3141235K->291853K(4833280K), 10.2906040 >>>>>>>>>>>> secs] 3491471K->291853K(5201920K), [CMS Perm : >>>>>>>>>>>> 121784K->121765K(262144K)], 10.2910040 secs] [Times: user=10.29 >>>>>>>>>>>> sys=0.00, real=10.29 secs] >>>>>>>>>>>> 2011-12-30T22:42:56.281+0100: 2136625.183: [GC 2136625.183: [ParNew: >>>>>>>>>>>> 327680K->25286K(368640K), 0.0287220 secs] 619533K->317140K(5201920K), >>>>>>>>>>>> 0.0291610 secs] [Times: user=0.13 sys=0.00, real=0.03 secs] >>>>>>>>>>>> 2011-12-30T22:43:10.516+0100: 2136639.418: [GC 2136639.418: [ParNew: >>>>>>>>>>>> 352966K->26737K(368640K), 0.0586400 secs] 644820K->338758K(5201920K), >>>>>>>>>>>> 0.0589640 secs] [Times: user=0.31 sys=0.00, real=0.06 secs] >>>>>>>>>>>> ------- >>>>>>>>>>>> >>>>>>>>>>>> In this case I don't know how to interpret the output. >>>>>>>>>>>> a) There's a promotion failure that took 7.49 secs >>>>>>>>>>> This is the time it took to attempt the minor collection (ParNew) and >>>>>>>>>>> to >>>>>>>>>>> do recovery >>>>>>>>>>> from the failure. >>>>>>>>>>> >>>>>>>>>>>> b) There's a full GC that took 14.08 secs >>>>>>>>>>>> c) There's a concurrent mode failure that took 10.29 secs >>>>>>>>>>> Not sure about b) and c) because the output is mixed up with the >>>>>>>>>>> concurrent-sweep >>>>>>>>>>> output but ?I think the "concurrent mode failure" message is part of >>>>>>>>>>> the >>>>>>>>>>> "Full GC" >>>>>>>>>>> message. ?My guess is that the 10.29 is the time for the Full GC and >>>>>>>>>>> the >>>>>>>>>>> 14.08 >>>>>>>>>>> maybe is part of the concurrent-sweep message. ?Really hard to be sure. >>>>>>>>>>> >>>>>>>>>>> Jon >>>>>>>>>>>> How are these events, and their (real) times related to each other? >>>>>>>>>>>> >>>>>>>>>>>> Thanks in advance, >>>>>>>>>>>> Taras >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Dec 27, 2011 at 6:13 PM, Jon >>>>>>>>>>>> Masamitsu ? ? ? ? ?wrote: >>>>>>>>>>>>> Taras, >>>>>>>>>>>>> >>>>>>>>>>>>> PrintPromotionFailure seems like it would go a long >>>>>>>>>>>>> way to identify the root of your promotion failures (or >>>>>>>>>>>>> at least eliminating some possible causes). ? ?I think it >>>>>>>>>>>>> would help focus the discussion if you could send >>>>>>>>>>>>> the result of that experiment early. >>>>>>>>>>>>> >>>>>>>>>>>>> Jon >>>>>>>>>>>>> >>>>>>>>>>>>> On 12/27/2011 5:07 AM, Taras Tielkes wrote: >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> We're running an application with the CMS/ParNew collectors that is >>>>>>>>>>>>>> experiencing occasional promotion failures. >>>>>>>>>>>>>> Environment is Linux 2.6.18 (x64), JVM is 1.6.0_29 in server mode. >>>>>>>>>>>>>> I've listed the specific JVM options used below (a). >>>>>>>>>>>>>> >>>>>>>>>>>>>> The application is deployed across a handful of machines, and the >>>>>>>>>>>>>> promotion failures are fairly uniform across those. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The first kind of failure we observe is a promotion failure during >>>>>>>>>>>>>> ParNew collection, I've included a snipped from the gc log below >>>>>>>>>>>>>> (b). >>>>>>>>>>>>>> The second kind of failure is a concurrrent mode failure (perhaps >>>>>>>>>>>>>> triggered by the same cause), see (c) below. >>>>>>>>>>>>>> The frequency (after running for a some weeks) is approximately once >>>>>>>>>>>>>> per day. This is bearable, but obviously we'd like to improve on >>>>>>>>>>>>>> this. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Apart from high-volume request handling (which allocates a lot of >>>>>>>>>>>>>> small objects), the application also runs a few dozen background >>>>>>>>>>>>>> threads that download and process XML documents, typically in the >>>>>>>>>>>>>> 5-30 >>>>>>>>>>>>>> MB range. >>>>>>>>>>>>>> A known deficiency in the existing code is that the XML content is >>>>>>>>>>>>>> copied twice before processing (once to a byte[], and later again to >>>>>>>>>>>>>> a >>>>>>>>>>>>>> String/char[]). >>>>>>>>>>>>>> Given that a 30 MB XML stream will result in a 60 MB >>>>>>>>>>>>>> java.lang.String/char[], my suspicion is that these big array >>>>>>>>>>>>>> allocations are causing us to run into the CMS fragmentation issue. >>>>>>>>>>>>>> >>>>>>>>>>>>>> My questions are: >>>>>>>>>>>>>> 1) Does the data from the GC logs provide sufficient evidence to >>>>>>>>>>>>>> conclude that CMS fragmentation is the cause of the promotion >>>>>>>>>>>>>> failure? >>>>>>>>>>>>>> 2) If not, what's the next step of investigating the cause? >>>>>>>>>>>>>> 3) We're planning to at least add -XX:+PrintPromotionFailure to get >>>>>>>>>>>>>> a >>>>>>>>>>>>>> feeling for the size of the objects that fail promotion. >>>>>>>>>>>>>> Overall, it seem that -XX:PrintFLSStatistics=1 is actually the only >>>>>>>>>>>>>> reliable approach to diagnose CMS fragmentation. Is this indeed the >>>>>>>>>>>>>> case? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks in advance, >>>>>>>>>>>>>> Taras >>>>>>>>>>>>>> >>>>>>>>>>>>>> a) Current JVM options: >>>>>>>>>>>>>> -------------------------------- >>>>>>>>>>>>>> -server >>>>>>>>>>>>>> -Xms5g >>>>>>>>>>>>>> -Xmx5g >>>>>>>>>>>>>> -Xmn400m >>>>>>>>>>>>>> -XX:PermSize=256m >>>>>>>>>>>>>> -XX:MaxPermSize=256m >>>>>>>>>>>>>> -XX:+PrintGCTimeStamps >>>>>>>>>>>>>> -verbose:gc >>>>>>>>>>>>>> -XX:+PrintGCDateStamps >>>>>>>>>>>>>> -XX:+PrintGCDetails >>>>>>>>>>>>>> -XX:SurvivorRatio=8 >>>>>>>>>>>>>> -XX:+UseConcMarkSweepGC >>>>>>>>>>>>>> -XX:+UseParNewGC >>>>>>>>>>>>>> -XX:+DisableExplicitGC >>>>>>>>>>>>>> -XX:+UseCMSInitiatingOccupancyOnly >>>>>>>>>>>>>> -XX:+CMSClassUnloadingEnabled >>>>>>>>>>>>>> -XX:+CMSScavengeBeforeRemark >>>>>>>>>>>>>> -XX:CMSInitiatingOccupancyFraction=68 >>>>>>>>>>>>>> -Xloggc:gc.log >>>>>>>>>>>>>> -------------------------------- >>>>>>>>>>>>>> >>>>>>>>>>>>>> b) Promotion failure during ParNew >>>>>>>>>>>>>> -------------------------------- >>>>>>>>>>>>>> 2011-12-08T18:14:40.966+0100: 219729.868: [GC 219729.868: [ParNew: >>>>>>>>>>>>>> 368640K->40959K(368640K), 0.0693460 secs] >>>>>>>>>>>>>> 3504917K->3195098K(5201920K), 0.0696500 secs] [Times: user=0.39 >>>>>>>>>>>>>> sys=0.01, real=0.07 secs] >>>>>>>>>>>>>> 2011-12-08T18:14:43.778+0100: 219732.679: [GC 219732.679: [ParNew: >>>>>>>>>>>>>> 368639K->31321K(368640K), 0.0511400 secs] >>>>>>>>>>>>>> 3522778K->3198316K(5201920K), 0.0514420 secs] [Times: user=0.28 >>>>>>>>>>>>>> sys=0.00, real=0.05 secs] >>>>>>>>>>>>>> 2011-12-08T18:14:46.945+0100: 219735.846: [GC 219735.846: [ParNew: >>>>>>>>>>>>>> 359001K->18694K(368640K), 0.0272970 secs] >>>>>>>>>>>>>> 3525996K->3185690K(5201920K), 0.0276080 secs] [Times: user=0.19 >>>>>>>>>>>>>> sys=0.00, real=0.03 secs] >>>>>>>>>>>>>> 2011-12-08T18:14:49.036+0100: 219737.938: [GC 219737.938: [ParNew >>>>>>>>>>>>>> (promotion failed): 338813K->361078K(368640K), 0.1321200 >>>>>>>>>>>>>> secs]219738.070: [CMS: 3167747K->434291K(4833280K), 4.8881570 secs] >>>>>>>>>>>>>> 3505808K->434291K >>>>>>>>>>>>>> (5201920K), [CMS Perm : 116893K->116883K(262144K)], 5.0206620 secs] >>>>>>>>>>>>>> [Times: user=5.24 sys=0.00, real=5.02 secs] >>>>>>>>>>>>>> 2011-12-08T18:14:54.721+0100: 219743.622: [GC 219743.623: [ParNew: >>>>>>>>>>>>>> 327680K->40960K(368640K), 0.0949460 secs] >>>>>>>>>>>>>> 761971K->514584K(5201920K), >>>>>>>>>>>>>> 0.0952820 secs] [Times: user=0.52 sys=0.04, real=0.10 secs] >>>>>>>>>>>>>> 2011-12-08T18:14:55.580+0100: 219744.481: [GC 219744.482: [ParNew: >>>>>>>>>>>>>> 368640K->40960K(368640K), 0.1299190 secs] >>>>>>>>>>>>>> 842264K->625681K(5201920K), >>>>>>>>>>>>>> 0.1302190 secs] [Times: user=0.72 sys=0.01, real=0.13 secs] >>>>>>>>>>>>>> 2011-12-08T18:14:58.050+0100: 219746.952: [GC 219746.952: [ParNew: >>>>>>>>>>>>>> 368640K->40960K(368640K), 0.0870940 secs] >>>>>>>>>>>>>> 953361K->684121K(5201920K), >>>>>>>>>>>>>> 0.0874110 secs] [Times: user=0.48 sys=0.01, real=0.09 secs] >>>>>>>>>>>>>> -------------------------------- >>>>>>>>>>>>>> >>>>>>>>>>>>>> c) Promotion failure during CMS >>>>>>>>>>>>>> -------------------------------- >>>>>>>>>>>>>> 2011-12-14T08:29:26.628+0100: 703015.530: [GC 703015.530: [ParNew: >>>>>>>>>>>>>> 357228K->40960K(368640K), 0.0525110 secs] >>>>>>>>>>>>>> 3603068K->3312743K(5201920K), 0.0528120 secs] [Times: user=0.37 >>>>>>>>>>>>>> sys=0.00, real=0.05 secs] >>>>>>>>>>>>>> 2011-12-14T08:29:28.864+0100: 703017.766: [GC 703017.766: [ParNew: >>>>>>>>>>>>>> 366075K->37119K(368640K), 0.0479780 secs] >>>>>>>>>>>>>> 3637859K->3317662K(5201920K), 0.0483090 secs] [Times: user=0.24 >>>>>>>>>>>>>> sys=0.01, real=0.05 secs] >>>>>>>>>>>>>> 2011-12-14T08:29:29.553+0100: 703018.454: [GC 703018.455: [ParNew: >>>>>>>>>>>>>> 364792K->40960K(368640K), 0.0421740 secs] >>>>>>>>>>>>>> 3645334K->3334944K(5201920K), 0.0424810 secs] [Times: user=0.30 >>>>>>>>>>>>>> sys=0.00, real=0.04 secs] >>>>>>>>>>>>>> 2011-12-14T08:29:29.600+0100: 703018.502: [GC [1 CMS-initial-mark: >>>>>>>>>>>>>> 3293984K(4833280K)] 3335025K(5201920K), 0.0272490 secs] [Times: >>>>>>>>>>>>>> user=0.02 sys=0.00, real=0.03 secs] >>>>>>>>>>>>>> 2011-12-14T08:29:29.628+0100: 703018.529: >>>>>>>>>>>>>> [CMS-concurrent-mark-start] >>>>>>>>>>>>>> 2011-12-14T08:29:30.718+0100: 703019.620: [GC 703019.620: [ParNew: >>>>>>>>>>>>>> 368640K->40960K(368640K), 0.0836690 secs] >>>>>>>>>>>>>> 3662624K->3386039K(5201920K), 0.0839690 secs] [Times: user=0.50 >>>>>>>>>>>>>> sys=0.01, real=0.08 secs] >>>>>>>>>>>>>> 2011-12-14T08:29:30.827+0100: 703019.729: [CMS-concurrent-mark: >>>>>>>>>>>>>> 1.108/1.200 secs] [Times: user=6.83 sys=0.23, real=1.20 secs] >>>>>>>>>>>>>> 2011-12-14T08:29:30.827+0100: 703019.729: >>>>>>>>>>>>>> [CMS-concurrent-preclean-start] >>>>>>>>>>>>>> 2011-12-14T08:29:30.938+0100: 703019.840: [CMS-concurrent-preclean: >>>>>>>>>>>>>> 0.093/0.111 secs] [Times: user=0.48 sys=0.02, real=0.11 secs] >>>>>>>>>>>>>> 2011-12-14T08:29:30.938+0100: 703019.840: >>>>>>>>>>>>>> [CMS-concurrent-abortable-preclean-start] >>>>>>>>>>>>>> 2011-12-14T08:29:32.337+0100: 703021.239: >>>>>>>>>>>>>> [CMS-concurrent-abortable-preclean: 1.383/1.399 secs] [Times: >>>>>>>>>>>>>> user=6.68 sys=0.27, real=1.40 secs] >>>>>>>>>>>>>> 2011-12-14T08:29:32.343+0100: 703021.244: [GC[YG occupancy: 347750 K >>>>>>>>>>>>>> (368640 K)]2011-12-14T08:29:32.343+0100: 703021.244: [GC 703021.244: >>>>>>>>>>>>>> [ParNew (promotion failed): 347750K->347750K(368640K), 9.8729020 >>>>>>>>>>>>>> secs] >>>>>>>>>>>>>> ? ? ? 3692829K->3718580K(5201920K), 9.8732380 secs] [Times: user=12.00 >>>>>>>>>>>>>> sys=2.58, real=9.88 secs] >>>>>>>>>>>>>> 703031.118: [Rescan (parallel) , 0.2826110 secs]703031.400: [weak >>>>>>>>>>>>>> refs >>>>>>>>>>>>>> processing, 0.0014780 secs]703031.402: [class unloading, 0.0176610 >>>>>>>>>>>>>> secs]703031.419: [scrub symbol& ? ? ? ? ? ?string tables, 0.0094960 >>>>>>>>>>>>>> secs] [1 CMS >>>>>>>>>>>>>> -remark: 3370830K(4833280K)] 3718580K(5201920K), 10.1916910 secs] >>>>>>>>>>>>>> [Times: user=13.73 sys=2.59, real=10.19 secs] >>>>>>>>>>>>>> 2011-12-14T08:29:42.535+0100: 703031.436: >>>>>>>>>>>>>> [CMS-concurrent-sweep-start] >>>>>>>>>>>>>> 2011-12-14T08:29:42.591+0100: 703031.493: [Full GC 703031.493: >>>>>>>>>>>>>> [CMS2011-12-14T08:29:48.616+0100: 703037.518: [CMS-concurrent-sweep: >>>>>>>>>>>>>> 6.046/6.082 secs] [Times: user=6.18 sys=0.01, real=6.09 secs] >>>>>>>>>>>>>> ? ? ? (concurrent mode failure): 3370829K->433437K(4833280K), >>>>>>>>>>>>>> 10.9594300 >>>>>>>>>>>>>> secs] 3739469K->433437K(5201920K), [CMS Perm : >>>>>>>>>>>>>> 121702K->121690K(262144K)], 10.9597540 secs] [Times: user=10.95 >>>>>>>>>>>>>> sys=0.00, real=10.96 secs] >>>>>>>>>>>>>> 2011-12-14T08:29:53.997+0100: 703042.899: [GC 703042.899: [ParNew: >>>>>>>>>>>>>> 327680K->40960K(368640K), 0.0799960 secs] >>>>>>>>>>>>>> 761117K->517836K(5201920K), >>>>>>>>>>>>>> 0.0804100 secs] [Times: user=0.46 sys=0.00, real=0.08 secs] >>>>>>>>>>>>>> 2011-12-14T08:29:54.649+0100: 703043.551: [GC 703043.551: [ParNew: >>>>>>>>>>>>>> 368640K->40960K(368640K), 0.0784460 secs] >>>>>>>>>>>>>> 845516K->557872K(5201920K), >>>>>>>>>>>>>> 0.0787920 secs] [Times: user=0.40 sys=0.01, real=0.08 secs] >>>>>>>>>>>>>> 2011-12-14T08:29:56.418+0100: 703045.320: [GC 703045.320: [ParNew: >>>>>>>>>>>>>> 368640K->40960K(368640K), 0.0784040 secs] >>>>>>>>>>>>>> 885552K->603017K(5201920K), >>>>>>>>>>>>>> 0.0787630 secs] [Times: user=0.41 sys=0.01, real=0.07 secs] >>>>>>>>>>>>>> -------------------------------- >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> hotspot-gc-use mailing list >>>>>>>>>>>>>> hotspot-gc-use at openjdk.java.net >>>>>>>>>>>>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> hotspot-gc-use mailing list >>>>>>>>>>>>> hotspot-gc-use at openjdk.java.net >>>>>>>>>>>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> hotspot-gc-use mailing list >>>>>>>>>>>> hotspot-gc-use at openjdk.java.net >>>>>>>>>>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> hotspot-gc-use mailing list >>>>>>>>>>> hotspot-gc-use at openjdk.java.net >>>>>>>>>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>>>>>>>>> _______________________________________________ >>>>>>>>>> hotspot-gc-use mailing list >>>>>>>>>> hotspot-gc-use at openjdk.java.net >>>>>>>>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>>>>>>>> _______________________________________________ >>>>>>>>> hotspot-gc-use mailing list >>>>>>>>> hotspot-gc-use at openjdk.java.net >>>>>>>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>>>>> _______________________________________________ >>>>>> hotspot-gc-use mailing list >>>>>> hotspot-gc-use at openjdk.java.net >>>>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>>>> _______________________________________________ >>>>> hotspot-gc-use mailing list >>>>> hotspot-gc-use at openjdk.java.net >>>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>>> _______________________________________________ >>>> hotspot-gc-use mailing list >>>> hotspot-gc-use at openjdk.java.net >>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>> _______________________________________________ >>> hotspot-gc-use mailing list >>> hotspot-gc-use at openjdk.java.net >>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From chkwok at digibites.nl Tue Feb 21 10:16:47 2012 From: chkwok at digibites.nl (Chi Ho Kwok) Date: Tue, 21 Feb 2012 19:16:47 +0100 Subject: Promotion failures: indication of CMS fragmentation? In-Reply-To: References: <4EF9FCAC.3030208@oracle.com> <4F06A270.3010701@oracle.com> <4F0DBEC4.7040907@oracle.com> <4F1ECE7B.3040502@oracle.com> <4F1F2ED7.6060308@oracle.com> <4F20F78D.9070905@oracle.com> Message-ID: Hi Teras, I think you may want to look into sizing the new and especially the survivor spaces differently. We run something similar to what you described, high volume request processing with large dataset loading, and what we've seen at the start is that the survivor spaces are completely overloaded, causing premature promotions. We've configured our vm with the following goals/guideline: - old space is for semi-permanent data, living for at least 30s, average ~10 minutes - new space contains only temporary and just loaded data - surviving objects from new should never reach old in 1 gc, so the survivor space may never be 100% full With jstat -gcutil `pidof java` 2000, we see things like: S0 S1 E O P YGC YGCT FGC FGCT GCT 70.20 0.00 19.65 57.60 59.90 124808 29474.299 2498 191.110 29665.409 70.20 0.00 92.89 57.60 59.90 124808 29474.299 2498 191.110 29665.409 70.20 0.00 93.47 57.60 59.90 124808 29474.299 2498 191.110 29665.409 0.00 65.69 78.07 58.09 59.90 124809 29474.526 2498 191.110 29665.636 84.97 0.00 48.19 58.57 59.90 124810 29474.774 2498 191.110 29665.884 84.97 0.00 81.30 58.57 59.90 124810 29474.774 2498 191.110 29665.884 0.00 62.64 27.22 59.12 59.90 124811 29474.992 2498 191.110 29666.102 0.00 62.64 54.47 59.12 59.90 124811 29474.992 2498 191.110 29666.102 75.68 0.00 6.80 59.53 59.90 124812 29475.228 2498 191.110 29666.338 75.68 0.00 23.38 59.53 59.90 124812 29475.228 2498 191.110 29666.338 75.68 0.00 27.72 59.53 59.90 124812 29475.228 2498 191.110 29666.338 If you follow the lines, you can see Eden fill up to 100% on line 4, surviving objects are copied into S1, S0 is collected and added 0.49% to Old. On line 5, another GC happened, with Eden->S0, S1->Old, etc. No objects is ever transferred from Eden to Old, unless there's a huge peak of requests. This is with a: 32GB heap, Mxn1200M, SurvivorRatio 2 (600MB Eden, 300MB S0, 300MB S1), MaxTenuringThreshold 1 (whatever is still alive in S0/1 on the second GC is copied to old, don't wait, web requests are quite bursty). With about 1 collection every 2-5 seconds, objects promoted to Old must live for at 4-10 seconds; as that's longer than an average request (50ms-1s), none of the temporary data ever makes it into Old, which is much more expensive to collect. It works even with a higher than default CMSInitiatingOccupancyFraction=76 to optimize for space available for the large data cache we have. With your config of 400MB Total new, with 350MB Eden, 25MB S0, 25MB S1 (SurvivorRatio 8), no tenuring threshold, I think loads of new objects get copied from Eden to Old directly, causing trouble for the CMS. You can use jstat to get live stats and tweak until it doesn't happen. If you can't make changes on live that easil, try doubling the new size indeed, with a 400 Eden, 200 S0, 200 S1 and MaxTenuringThreshold 1 setting. It's probably overkill, but if should solve the problem if it is caused by premature promotion. Chi Ho Kwok On Tue, Feb 21, 2012 at 5:55 PM, Taras Tielkes wrote: > Hi, > > We've removed the "-XX:+CMSScavengeBeforeRemark" setting from 50% of > our production nodes. > After running for a few weeks, it seems that there's no impact from > removing this option. > Which is good, since it seems we can remove it from the other nodes as > well, simplifying our overall JVM configuration ;-) > > However, we're still seeing promotion failures on all nodes, once > every day or so. > > There's still the "Magic 1026": this accounts for ~60% of the > promotion failures that we're seeing (single ParNew thread thread, > 1026 failure size): > -------------------- > 2012-02-06T09:13:51.806+0100: 328095.085: [GC 328095.086: [ParNew: > 359895K->29357K(368640K), 0.0429070 secs] > 3471021K->3143476K(5201920K), 0.0434950 secs] [Times: user=0.32 > sys=0.00, real=0.04 secs] > 2012-02-06T09:13:55.922+0100: 328099.201: [GC 328099.201: [ParNew: > 357037K->31817K(368640K), 0.0429130 secs] > 3471156K->3148946K(5201920K), 0.0434930 secs] [Times: user=0.31 > sys=0.00, real=0.04 secs] > 2012-02-06T09:13:59.044+0100: 328102.324: [GC 328102.324: [ParNew > (promotion failure size = 1026) (promotion failed): > 359497K->368640K(368640K), 0.2226790 secs]328102.547: [CMS: > 3125609K->451515K(4833280K), 5.6225880 secs] 3476626K->4515 > 15K(5201920K), [CMS Perm : 124373K->124353K(262144K)], 5.8459380 secs] > [Times: user=6.20 sys=0.01, real=5.85 secs] > 2012-02-06T09:14:05.243+0100: 328108.522: [GC 328108.523: [ParNew: > 327680K->40960K(368640K), 0.0319160 secs] 779195K->497658K(5201920K), > 0.0325360 secs] [Times: user=0.21 sys=0.01, real=0.03 secs] > 2012-02-06T09:14:07.836+0100: 328111.116: [GC 328111.116: [ParNew: > 368640K->32785K(368640K), 0.0744670 secs] 825338K->520234K(5201920K), > 0.0750390 secs] [Times: user=0.40 sys=0.02, real=0.08 secs] > -------------------- > Given the 1026 word size, I'm wondering if I should be hunting for an > overuse of BufferedInputStream/BufferedOutoutStream, since both have > 8192 as a default buffer size. > > The second group of promotion failures look like this (multiple ParNew > threads, small failure sizes): > -------------------- > 2012-02-06T09:50:15.773+0100: 328756.964: [GC 328756.964: [ParNew: > 356116K->29934K(368640K), 0.0461100 secs] > 3203863K->2880162K(5201920K), 0.0468870 secs] [Times: user=0.34 > sys=0.01, real=0.05 secs] > 2012-02-06T09:50:19.153+0100: 328760.344: [GC 328760.344: [ParNew: > 357614K->30359K(368640K), 0.0454680 secs] > 3207842K->2882892K(5201920K), 0.0462280 secs] [Times: user=0.33 > sys=0.01, real=0.05 secs] > 2012-02-06T09:50:22.658+0100: 328763.849: [GC 328763.849: [ParNew (1: > promotion failure size = 25) (4: promotion failure size = 25) (6: > promotion failure size = 25) (7: promotion failure size = 144) > (promotion failed): 358039K->358358 > K(368640K), 0.2148680 secs]328764.064: [CMS: > 2854709K->446750K(4833280K), 5.8368270 secs] > 3210572K->446750K(5201920K), [CMS Perm : 124670K->124644K(262144K)], > 6.0525230 secs] [Times: user=6.32 sys=0.00, real=6.05 secs] > 2012-02-06T09:50:29.896+0100: 328771.086: [GC 328771.087: [ParNew: > 327680K->22569K(368640K), 0.0227080 secs] 774430K->469319K(5201920K), > 0.0235020 secs] [Times: user=0.16 sys=0.00, real=0.02 secs] > 2012-02-06T09:50:31.076+0100: 328772.266: [GC 328772.267: [ParNew: > 350249K->22264K(368640K), 0.0235480 secs] 796999K->469014K(5201920K), > 0.0243000 secs] [Times: user=0.18 sys=0.01, real=0.02 secs] > -------------------- > > We're going to try to double the new size on a single node, to see the > effects of that. > > Beyond this experiment, is there any additional data I can collect to > better understand the nature of the promotion failures? > Am I facing collecting free list statistics at this point? > > Thanks, > Taras -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120221/0c84660f/attachment.html From ysr1729 at gmail.com Tue Feb 21 15:40:52 2012 From: ysr1729 at gmail.com (Srinivas Ramakrishna) Date: Tue, 21 Feb 2012 15:40:52 -0800 Subject: Promotion failures: indication of CMS fragmentation? In-Reply-To: References: <4EF9FCAC.3030208@oracle.com> <4F06A270.3010701@oracle.com> <4F0DBEC4.7040907@oracle.com> <4F1ECE7B.3040502@oracle.com> <4F1F2ED7.6060308@oracle.com> <4F20F78D.9070905@oracle.com> Message-ID: I agree that premature promotions are almost always the first and most important thing to fix when running into fragmentation or overload issues with CMS. However, I can also imagine long-lived objects with a highly non-stationary size distribution which can also cause problems for CMS despite best efforts to tune against premature promotion. I didn't think Treas was running with MTT=0, although MTT > 0 is no recipe for avoiding premature promotion with bursty loads that case overflow the survivor spaces -- as you say large survivor spaces with a low TargetSurvivorRatio -- so as to leave plenty of space to absorb/accommodate spiking/bursty loads is definitely a "best practice" for CMS (and possibly for other concurrent collectors as well). One thing Taras can do to see if premature promotion might be an issue is to look at the tenuring threshold in his case. A rough proxy (if PrintTenuringDistribution is not enabled) is to look at the promotion volume per scavenge. It may be possible, if premature promotion is a cause, to see some kind of medium-term correlation between high promotion volume and eventual promotion failure despite frequent CMS collections. One other point which may or may not be relevant. I see that Taras is not using CompressedOops... Using that alone would greatly decrease memory pressure and provide more breathing room to CMS, which is also almost always a good idea. -- ramki On Tue, Feb 21, 2012 at 10:16 AM, Chi Ho Kwok wrote: > Hi Teras, > > I think you may want to look into sizing the new and especially the > survivor spaces differently. We run something similar to what you > described, high volume request processing with large dataset loading, and > what we've seen at the start is that the survivor spaces are completely > overloaded, causing premature promotions. > > We've configured our vm with the following goals/guideline: > > - old space is for semi-permanent data, living for at least 30s, > average ~10 minutes > - new space contains only temporary and just loaded data > - surviving objects from new should never reach old in 1 gc, so the > survivor space may never be 100% full > > With jstat -gcutil `pidof java` 2000, we see things like: > > S0 S1 E O P YGC YGCT FGC FGCT GCT > 70.20 0.00 19.65 57.60 59.90 124808 29474.299 2498 191.110 > 29665.409 > 70.20 0.00 92.89 57.60 59.90 124808 29474.299 2498 191.110 > 29665.409 > 70.20 0.00 93.47 57.60 59.90 124808 29474.299 2498 191.110 > 29665.409 > 0.00 65.69 78.07 58.09 59.90 124809 29474.526 2498 191.110 > 29665.636 > 84.97 0.00 48.19 58.57 59.90 124810 29474.774 2498 191.110 > 29665.884 > 84.97 0.00 81.30 58.57 59.90 124810 29474.774 2498 191.110 > 29665.884 > 0.00 62.64 27.22 59.12 59.90 124811 29474.992 2498 191.110 > 29666.102 > 0.00 62.64 54.47 59.12 59.90 124811 29474.992 2498 191.110 > 29666.102 > 75.68 0.00 6.80 59.53 59.90 124812 29475.228 2498 191.110 > 29666.338 > 75.68 0.00 23.38 59.53 59.90 124812 29475.228 2498 191.110 > 29666.338 > 75.68 0.00 27.72 59.53 59.90 124812 29475.228 2498 191.110 > 29666.338 > > If you follow the lines, you can see Eden fill up to 100% on line 4, > surviving objects are copied into S1, S0 is collected and added 0.49% to > Old. On line 5, another GC happened, with Eden->S0, S1->Old, etc. No > objects is ever transferred from Eden to Old, unless there's a huge peak of > requests. > > This is with a: 32GB heap, Mxn1200M, SurvivorRatio 2 (600MB Eden, 300MB > S0, 300MB S1), MaxTenuringThreshold 1 (whatever is still alive in S0/1 on > the second GC is copied to old, don't wait, web requests are quite bursty). > With about 1 collection every 2-5 seconds, objects promoted to Old must > live for at 4-10 seconds; as that's longer than an average request > (50ms-1s), none of the temporary data ever makes it into Old, which is much > more expensive to collect. It works even with a higher than default > CMSInitiatingOccupancyFraction=76 to optimize for space available for the > large data cache we have. > > > With your config of 400MB Total new, with 350MB Eden, 25MB S0, 25MB S1 > (SurvivorRatio 8), no tenuring threshold, I think loads of new objects get > copied from Eden to Old directly, causing trouble for the CMS. You can use > jstat to get live stats and tweak until it doesn't happen. If you can't > make changes on live that easil, try doubling the new size indeed, with a > 400 Eden, 200 S0, 200 S1 and MaxTenuringThreshold 1 setting. It's probably > overkill, but if should solve the problem if it is caused by premature > promotion. > > > Chi Ho Kwok > > > On Tue, Feb 21, 2012 at 5:55 PM, Taras Tielkes wrote: > >> Hi, >> >> We've removed the "-XX:+CMSScavengeBeforeRemark" setting from 50% of >> our production nodes. >> After running for a few weeks, it seems that there's no impact from >> removing this option. >> Which is good, since it seems we can remove it from the other nodes as >> well, simplifying our overall JVM configuration ;-) >> >> However, we're still seeing promotion failures on all nodes, once >> every day or so. >> >> There's still the "Magic 1026": this accounts for ~60% of the >> promotion failures that we're seeing (single ParNew thread thread, >> 1026 failure size): >> -------------------- >> 2012-02-06T09:13:51.806+0100: 328095.085: [GC 328095.086: [ParNew: >> 359895K->29357K(368640K), 0.0429070 secs] >> 3471021K->3143476K(5201920K), 0.0434950 secs] [Times: user=0.32 >> sys=0.00, real=0.04 secs] >> 2012-02-06T09:13:55.922+0100: 328099.201: [GC 328099.201: [ParNew: >> 357037K->31817K(368640K), 0.0429130 secs] >> 3471156K->3148946K(5201920K), 0.0434930 secs] [Times: user=0.31 >> sys=0.00, real=0.04 secs] >> 2012-02-06T09:13:59.044+0100: 328102.324: [GC 328102.324: [ParNew >> (promotion failure size = 1026) (promotion failed): >> 359497K->368640K(368640K), 0.2226790 secs]328102.547: [CMS: >> 3125609K->451515K(4833280K), 5.6225880 secs] 3476626K->4515 >> 15K(5201920K), [CMS Perm : 124373K->124353K(262144K)], 5.8459380 secs] >> [Times: user=6.20 sys=0.01, real=5.85 secs] >> 2012-02-06T09:14:05.243+0100: 328108.522: [GC 328108.523: [ParNew: >> 327680K->40960K(368640K), 0.0319160 secs] 779195K->497658K(5201920K), >> 0.0325360 secs] [Times: user=0.21 sys=0.01, real=0.03 secs] >> 2012-02-06T09:14:07.836+0100: 328111.116: [GC 328111.116: [ParNew: >> 368640K->32785K(368640K), 0.0744670 secs] 825338K->520234K(5201920K), >> 0.0750390 secs] [Times: user=0.40 sys=0.02, real=0.08 secs] >> -------------------- >> Given the 1026 word size, I'm wondering if I should be hunting for an >> overuse of BufferedInputStream/BufferedOutoutStream, since both have >> 8192 as a default buffer size. >> >> The second group of promotion failures look like this (multiple ParNew >> threads, small failure sizes): >> -------------------- >> 2012-02-06T09:50:15.773+0100: 328756.964: [GC 328756.964: [ParNew: >> 356116K->29934K(368640K), 0.0461100 secs] >> 3203863K->2880162K(5201920K), 0.0468870 secs] [Times: user=0.34 >> sys=0.01, real=0.05 secs] >> 2012-02-06T09:50:19.153+0100: 328760.344: [GC 328760.344: [ParNew: >> 357614K->30359K(368640K), 0.0454680 secs] >> 3207842K->2882892K(5201920K), 0.0462280 secs] [Times: user=0.33 >> sys=0.01, real=0.05 secs] >> 2012-02-06T09:50:22.658+0100: 328763.849: [GC 328763.849: [ParNew (1: >> promotion failure size = 25) (4: promotion failure size = 25) (6: >> promotion failure size = 25) (7: promotion failure size = 144) >> (promotion failed): 358039K->358358 >> K(368640K), 0.2148680 secs]328764.064: [CMS: >> 2854709K->446750K(4833280K), 5.8368270 secs] >> 3210572K->446750K(5201920K), [CMS Perm : 124670K->124644K(262144K)], >> 6.0525230 secs] [Times: user=6.32 sys=0.00, real=6.05 secs] >> 2012-02-06T09:50:29.896+0100: 328771.086: [GC 328771.087: [ParNew: >> 327680K->22569K(368640K), 0.0227080 secs] 774430K->469319K(5201920K), >> 0.0235020 secs] [Times: user=0.16 sys=0.00, real=0.02 secs] >> 2012-02-06T09:50:31.076+0100: 328772.266: [GC 328772.267: [ParNew: >> 350249K->22264K(368640K), 0.0235480 secs] 796999K->469014K(5201920K), >> 0.0243000 secs] [Times: user=0.18 sys=0.01, real=0.02 secs] >> -------------------- >> >> We're going to try to double the new size on a single node, to see the >> effects of that. >> >> Beyond this experiment, is there any additional data I can collect to >> better understand the nature of the promotion failures? >> Am I facing collecting free list statistics at this point? >> >> Thanks, >> Taras >> > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120221/bd819e6e/attachment.html From taras.tielkes at gmail.com Wed Feb 22 01:51:19 2012 From: taras.tielkes at gmail.com (Taras Tielkes) Date: Wed, 22 Feb 2012 10:51:19 +0100 Subject: Promotion failures: indication of CMS fragmentation? In-Reply-To: References: <4EF9FCAC.3030208@oracle.com> <4F06A270.3010701@oracle.com> <4F0DBEC4.7040907@oracle.com> <4F1ECE7B.3040502@oracle.com> <4F1F2ED7.6060308@oracle.com> <4F20F78D.9070905@oracle.com> Message-ID: (this time properly responding to the list alias) Hi Srinivas, We're running 1.6.0 u29 on Linux x64. My understanding is that CompressedOops is enabled by default since u23. At least this page seems to support that: http://docs.oracle.com/javase/7/docs/technotes/guides/vm/performance-enhancements-7.html Regarding the other remarks (also from Todd and Chi), I'll comment later. The first thing on my list is to collect PrintTenuringDistribution data now. Kind regards, Taras On Wed, Feb 22, 2012 at 10:50 AM, Taras Tielkes wrote: > Hi Srinivas, > > We're running 1.6.0 u29 on Linux x64. My understanding is that > CompressedOops is enabled by default since u23. > > At least this page seems to support that: > http://docs.oracle.com/javase/7/docs/technotes/guides/vm/performance-enhancements-7.html > > Regarding the other remarks (also from Todd and Chi), I'll comment > later. The first thing on my list is to collect > PrintTenuringDistribution data now. > > Kind regards, > Taras > > On Wed, Feb 22, 2012 at 12:40 AM, Srinivas Ramakrishna > wrote: >> I agree that premature promotions are almost always the first and most >> important thing to fix when running >> into fragmentation or overload issues with CMS. However, I can also imagine >> long-lived objects with a highly >> non-stationary size distribution which can also cause problems for CMS >> despite best efforts to tune against >> premature promotion. >> >> I didn't think Treas was running with MTT=0, although MTT > 0 is no recipe >> for avoiding premature promotion >> with bursty loads that case overflow the survivor spaces -- as you say large >> survivor spaces with a low >> TargetSurvivorRatio -- so as to leave plenty of space to absorb/accommodate >> spiking/bursty loads? is >> definitely a "best practice" for CMS (and possibly for other concurrent >> collectors as well). >> >> One thing Taras can do to see if premature promotion might be an issue is to >> look at the tenuring >> threshold in his case. A rough proxy (if PrintTenuringDistribution is not >> enabled) is to look at the >> promotion volume per scavenge. It may be possible, if premature promotion is >> a cause, to see >> some kind of medium-term correlation between high promotion volume and >> eventual promotion >> failure despite frequent CMS collections. >> >> One other point which may or may not be relevant. I see that Taras is not >> using CompressedOops... >> Using that alone would greatly decrease memory pressure and provide more >> breathing room to CMS, >> which is also almost always a good idea. >> >> -- ramki >> >> On Tue, Feb 21, 2012 at 10:16 AM, Chi Ho Kwok wrote: >>> >>> Hi Teras, >>> >>> I think you may want to look into sizing the new and especially the >>> survivor spaces differently. We run something similar to what you described, >>> high volume request processing with large dataset loading, and what we've >>> seen at the start is that the survivor spaces are completely overloaded, >>> causing premature promotions. >>> >>> We've configured our vm with the following goals/guideline: >>> >>> old space is for semi-permanent data, living for at least 30s, average ~10 >>> minutes >>> new space contains only temporary and just loaded data >>> surviving objects from new should never reach old in 1 gc, so the survivor >>> space may never be 100% full >>> >>> With jstat -gcutil `pidof java` 2000, we see things like: >>> >>> ? S0 ? ? S1 ? ? E ? ? ?O ? ? ?P ? ? YGC ? ? YGCT ? ?FGC ? ?FGCT ? ? GCT >>> ?70.20 ? 0.00 ?19.65 ?57.60 ?59.90 124808 29474.299 ?2498 ?191.110 >>> 29665.409 >>> ?70.20 ? 0.00 ?92.89 ?57.60 ?59.90 124808 29474.299 ?2498 ?191.110 >>> 29665.409 >>> ?70.20 ? 0.00 ?93.47 ?57.60 ?59.90 124808 29474.299 ?2498 ?191.110 >>> 29665.409 >>> ? 0.00 ?65.69 ?78.07 ?58.09 ?59.90 124809 29474.526 ?2498 ?191.110 >>> 29665.636 >>> ?84.97 ? 0.00 ?48.19 ?58.57 ?59.90 124810 29474.774 ?2498 ?191.110 >>> 29665.884 >>> ?84.97 ? 0.00 ?81.30 ?58.57 ?59.90 124810 29474.774 ?2498 ?191.110 >>> 29665.884 >>> ? 0.00 ?62.64 ?27.22 ?59.12 ?59.90 124811 29474.992 ?2498 ?191.110 >>> 29666.102 >>> ? 0.00 ?62.64 ?54.47 ?59.12 ?59.90 124811 29474.992 ?2498 ?191.110 >>> 29666.102 >>> ?75.68 ? 0.00 ? 6.80 ?59.53 ?59.90 124812 29475.228 ?2498 ?191.110 >>> 29666.338 >>> ?75.68 ? 0.00 ?23.38 ?59.53 ?59.90 124812 29475.228 ?2498 ?191.110 >>> 29666.338 >>> ?75.68 ? 0.00 ?27.72 ?59.53 ?59.90 124812 29475.228 ?2498 ?191.110 >>> 29666.338 >>> >>> If you follow the lines, you can see Eden fill up to 100% on line 4, >>> surviving objects are copied into S1, S0 is collected and added 0.49% to >>> Old. On line 5, another GC happened, with Eden->S0, S1->Old, etc. No objects >>> is ever transferred from Eden to Old, unless there's a huge peak of >>> requests. >>> >>> This is with a: 32GB heap, Mxn1200M, SurvivorRatio 2 (600MB Eden, 300MB >>> S0, 300MB S1), MaxTenuringThreshold 1 (whatever is still alive in S0/1 on >>> the second GC is copied to old, don't wait, web requests are quite bursty). >>> With about 1 collection every 2-5 seconds, objects promoted to Old must live >>> for at 4-10 seconds; as that's longer than an average request (50ms-1s), >>> none of the temporary data ever makes it into Old, which is much more >>> expensive to collect. It works even with a higher than default >>> CMSInitiatingOccupancyFraction=76 to optimize for space available for the >>> large data cache we have. >>> >>> >>> With your config of 400MB Total new, with 350MB Eden, 25MB S0, 25MB S1 >>> (SurvivorRatio 8), no tenuring threshold, I think loads of new objects get >>> copied from Eden to Old directly, causing trouble for the CMS. You can use >>> jstat to get live stats and tweak until it doesn't happen. If you can't make >>> changes on live that easil, try doubling the new size indeed, with a 400 >>> Eden, 200 S0, 200 S1 and?MaxTenuringThreshold?1 setting. It's probably >>> overkill, but if should solve the problem if it is caused by premature >>> promotion. >>> >>> >>> Chi Ho Kwok >>> >>> >>> On Tue, Feb 21, 2012 at 5:55 PM, Taras Tielkes >>> wrote: >>>> >>>> Hi, >>>> >>>> We've removed the "-XX:+CMSScavengeBeforeRemark" setting from 50% of >>>> our production nodes. >>>> After running for a few weeks, it seems that there's no impact from >>>> removing this option. >>>> Which is good, since it seems we can remove it from the other nodes as >>>> well, simplifying our overall JVM configuration ;-) >>>> >>>> However, we're still seeing promotion failures on all nodes, once >>>> every day or so. >>>> >>>> There's still the "Magic 1026": this accounts for ~60% of the >>>> promotion failures that we're seeing (single ParNew thread thread, >>>> 1026 failure size): >>>> -------------------- >>>> 2012-02-06T09:13:51.806+0100: 328095.085: [GC 328095.086: [ParNew: >>>> 359895K->29357K(368640K), 0.0429070 secs] >>>> 3471021K->3143476K(5201920K), 0.0434950 secs] [Times: user=0.32 >>>> sys=0.00, real=0.04 secs] >>>> 2012-02-06T09:13:55.922+0100: 328099.201: [GC 328099.201: [ParNew: >>>> 357037K->31817K(368640K), 0.0429130 secs] >>>> 3471156K->3148946K(5201920K), 0.0434930 secs] [Times: user=0.31 >>>> sys=0.00, real=0.04 secs] >>>> 2012-02-06T09:13:59.044+0100: 328102.324: [GC 328102.324: [ParNew >>>> (promotion failure size = 1026) ?(promotion failed): >>>> 359497K->368640K(368640K), 0.2226790 secs]328102.547: [CMS: >>>> 3125609K->451515K(4833280K), 5.6225880 secs] 3476626K->4515 >>>> 15K(5201920K), [CMS Perm : 124373K->124353K(262144K)], 5.8459380 secs] >>>> [Times: user=6.20 sys=0.01, real=5.85 secs] >>>> 2012-02-06T09:14:05.243+0100: 328108.522: [GC 328108.523: [ParNew: >>>> 327680K->40960K(368640K), 0.0319160 secs] 779195K->497658K(5201920K), >>>> 0.0325360 secs] [Times: user=0.21 sys=0.01, real=0.03 secs] >>>> 2012-02-06T09:14:07.836+0100: 328111.116: [GC 328111.116: [ParNew: >>>> 368640K->32785K(368640K), 0.0744670 secs] 825338K->520234K(5201920K), >>>> 0.0750390 secs] [Times: user=0.40 sys=0.02, real=0.08 secs] >>>> -------------------- >>>> Given the 1026 word size, I'm wondering if I should be hunting for an >>>> overuse of BufferedInputStream/BufferedOutoutStream, since both have >>>> 8192 as a default buffer size. >>>> >>>> The second group of promotion failures look like this (multiple ParNew >>>> threads, small failure sizes): >>>> -------------------- >>>> 2012-02-06T09:50:15.773+0100: 328756.964: [GC 328756.964: [ParNew: >>>> 356116K->29934K(368640K), 0.0461100 secs] >>>> 3203863K->2880162K(5201920K), 0.0468870 secs] [Times: user=0.34 >>>> sys=0.01, real=0.05 secs] >>>> 2012-02-06T09:50:19.153+0100: 328760.344: [GC 328760.344: [ParNew: >>>> 357614K->30359K(368640K), 0.0454680 secs] >>>> 3207842K->2882892K(5201920K), 0.0462280 secs] [Times: user=0.33 >>>> sys=0.01, real=0.05 secs] >>>> 2012-02-06T09:50:22.658+0100: 328763.849: [GC 328763.849: [ParNew (1: >>>> promotion failure size = 25) ?(4: promotion failure size = 25) ?(6: >>>> promotion failure size = 25) ?(7: promotion failure size = 144) >>>> (promotion failed): 358039K->358358 >>>> K(368640K), 0.2148680 secs]328764.064: [CMS: >>>> 2854709K->446750K(4833280K), 5.8368270 secs] >>>> 3210572K->446750K(5201920K), [CMS Perm : 124670K->124644K(262144K)], >>>> 6.0525230 secs] [Times: user=6.32 sys=0.00, real=6.05 secs] >>>> 2012-02-06T09:50:29.896+0100: 328771.086: [GC 328771.087: [ParNew: >>>> 327680K->22569K(368640K), 0.0227080 secs] 774430K->469319K(5201920K), >>>> 0.0235020 secs] [Times: user=0.16 sys=0.00, real=0.02 secs] >>>> 2012-02-06T09:50:31.076+0100: 328772.266: [GC 328772.267: [ParNew: >>>> 350249K->22264K(368640K), 0.0235480 secs] 796999K->469014K(5201920K), >>>> 0.0243000 secs] [Times: user=0.18 sys=0.01, real=0.02 secs] >>>> -------------------- >>>> >>>> We're going to try to double the new size on a single node, to see the >>>> effects of that. >>>> >>>> Beyond this experiment, is there any additional data I can collect to >>>> better understand the nature of the promotion failures? >>>> Am I facing collecting free list statistics at this point? >>>> >>>> Thanks, >>>> Taras >>> >>> >>> _______________________________________________ >>> hotspot-gc-use mailing list >>> hotspot-gc-use at openjdk.java.net >>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>> >> From liquan.liu at alipay.com Sun Feb 26 20:50:38 2012 From: liquan.liu at alipay.com (=?gb2312?B?usbD9w==?=) Date: Mon, 27 Feb 2012 12:50:38 +0800 Subject: Class.forName cause ClassLoader objects moved to tenured generation Message-ID: <201202271250381099594@alipay.com> Hi all, I have seen Class.forName() causes the tenured generation got filled with custom ClassLoader objects. I suspect something JVM done internally will move the ClassLoader objects to tenured generation. For example, the following code: ======================== public class Test { public static void main(String[] args) throws Exception { for (int i=0 ;i<30000;i++) { test(); } } private static void test() throws Exception { MyClassLoader mcl = new MyClassLoader(); Class.forName("java.lang.String", false, mcl); } } public class MyClassLoader extends ClassLoader {} ========================= will output gc log: [GC [DefNew: 512K->64K(576K), 0.0041095 secs] 512K->344K(1984K), 0.0042064 secs] [GC [DefNew: 576K->64K(576K), 0.0032096 secs] 856K->682K(1984K), 0.0032937 secs] [GC [DefNew: 575K->63K(576K), 0.0032085 secs] 1194K->1021K(1984K), 0.0033686 secs] [GC [DefNew: 575K->64K(576K), 0.0025146 secs] 1533K->1359K(1984K), 0.0026305 secs] [GC [DefNew: 576K->64K(576K), 0.0025942 secs][Tenured: 1634K->166K(1664K), 0.0169541 secs] 1871K->166K(2240K), 0.0197106 secs] [GC [DefNew: 512K->64K(576K), 0.0019209 secs] 678K->505K(1984K), 0.0020053 secs] [GC [DefNew: 576K->63K(576K), 0.0022846 secs] 1017K->844K(1984K), 0.0024271 secs] [GC [DefNew: 575K->63K(576K), 0.0023358 secs] 1356K->1182K(1984K), 0.0024235 secs] [GC [DefNew: 575K->64K(576K), 0.0025660 secs][Tenured: 1457K->166K(1536K), 0.0136841 secs] 1694K->166K(2112K), 0.0164004 secs] If change the Class.forName to loadClass: ======================== private static void test() throws Exception { MyClassLoader mcl = new MyClassLoader(); mcl.loadClass("java.lang.String"); } ======================== then gc log will be: [GC [DefNew: 512K->63K(576K), 0.0028769 secs] 512K->138K(1984K), 0.0029627 secs] [GC [DefNew: 575K->0K(576K), 0.0009856 secs] 650K->138K(1984K), 0.0010711 secs] [GC [DefNew: 512K->0K(576K), 0.0006255 secs] 650K->138K(1984K), 0.0007062 secs] [GC [DefNew: 512K->0K(576K), 0.0002065 secs] 650K->138K(1984K), 0.0002861 secs] [GC [DefNew: 512K->0K(576K), 0.0001936 secs] 650K->138K(1984K), 0.0002674 secs] [GC [DefNew: 512K->0K(576K), 0.0002045 secs] 650K->138K(1984K), 0.0002796 secs] [GC [DefNew: 512K->0K(576K), 0.0001704 secs] 650K->138K(1984K), 0.0002481 secs] [GC [DefNew: 512K->0K(576K), 0.0002229 secs] 650K->138K(1984K), 0.0003118 secs] Reproduced in both jdk1.5 and 1.6. Can someone enlight me what's happening inside jvm (regarding class load and gc) ? Thanks. ________________________________ This email (including any attachments) is confidential and may be legally privileged. If you received this email in error, please delete it immediately and do not copy it or use it for any purpose or disclose its contents to any other person. Thank you. ??????(????????????)???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? From rednaxelafx at gmail.com Sun Feb 26 21:29:42 2012 From: rednaxelafx at gmail.com (Krystal Mok) Date: Mon, 27 Feb 2012 13:29:42 +0800 Subject: Class.forName cause ClassLoader objects moved to tenured generation In-Reply-To: <201202271250381099594@alipay.com> References: <201202271250381099594@alipay.com> Message-ID: Hi Liquan, That's because Class.forName() will keep a reference to your class loader in the SystemDictionary (for being an initiating loader of the class, in your example "java.lang.String"), whereas ClassLoader.loadClass() won't. SystemDictionary is the place where HotSpot VM keeps track of all currently loaded classes, with the mapping of: (class name, class loader) => class The "class loader" part includes both defining class loaders and initiating class loaders. During minor GC: The whole SystemDictionary is a part of the root set in a minor GC. Class unloading doesn't happen. The class loaders referenced by the SystemDictionary aren't collected, either. So in you example, the loader are all kept alive with Class.forName(). [GC [DefNew: 576K->64K(576K), 0.0025942 secs][Tenured: 1634K->166K(1664K), 0.0169541 secs] 1871K->166K(2240K), 0.0197106 secs] This kind of GC log shows that a collection started out as a minor GC, but failed to allocate enough space in the old generation to promotion surviving objects, so it bailed out and fell back to a full GC which collects the whole GC heap. This kind of fallback doesn't print "[Full GC" in the GC logs, so it's kind of tricky to tell if you don't know the gory details. You could turn on -XX:+PrintPromotionFailure to see promotion failure in action. A full GC will collect dead class loaders and unload unused classes. So your example won't get an OOME just for filling up the old generation with class loaders. Just to be clear: this behavior wouldn't be considered a bug in the VM. You'll have to take care of class loading more carefully in your application. HTH, - Kris 2012/2/27 ???? > Hi all, > > I have seen Class.forName() causes the tenured generation got filled with > custom ClassLoader objects. I suspect something JVM done internally will > move the ClassLoader objects to tenured generation. For example, the > following code: > ======================== > public class Test { > public static void main(String[] args) throws Exception { > for (int i=0 ;i<30000;i++) { > test(); > } > } > private static void test() throws Exception { > MyClassLoader mcl = new MyClassLoader(); > Class.forName("java.lang.String", false, mcl); > } > } > > public class MyClassLoader extends ClassLoader {} > ========================= > will output gc log: > > [GC [DefNew: 512K->64K(576K), 0.0041095 secs] 512K->344K(1984K), 0.0042064 > secs] > [GC [DefNew: 576K->64K(576K), 0.0032096 secs] 856K->682K(1984K), 0.0032937 > secs] > [GC [DefNew: 575K->63K(576K), 0.0032085 secs] 1194K->1021K(1984K), > 0.0033686 secs] > [GC [DefNew: 575K->64K(576K), 0.0025146 secs] 1533K->1359K(1984K), > 0.0026305 secs] > [GC [DefNew: 576K->64K(576K), 0.0025942 secs][Tenured: 1634K->166K(1664K), > 0.0169541 secs] 1871K->166K(2240K), 0.0197106 secs] > [GC [DefNew: 512K->64K(576K), 0.0019209 secs] 678K->505K(1984K), 0.0020053 > secs] > [GC [DefNew: 576K->63K(576K), 0.0022846 secs] 1017K->844K(1984K), > 0.0024271 secs] > [GC [DefNew: 575K->63K(576K), 0.0023358 secs] 1356K->1182K(1984K), > 0.0024235 secs] > [GC [DefNew: 575K->64K(576K), 0.0025660 secs][Tenured: 1457K->166K(1536K), > 0.0136841 secs] 1694K->166K(2112K), 0.0164004 secs] > > > If change the Class.forName to loadClass: > ======================== > private static void test() throws Exception { > MyClassLoader mcl = new MyClassLoader(); > mcl.loadClass("java.lang.String"); > } > ======================== > then gc log will be: > > [GC [DefNew: 512K->63K(576K), 0.0028769 secs] 512K->138K(1984K), 0.0029627 > secs] > [GC [DefNew: 575K->0K(576K), 0.0009856 secs] 650K->138K(1984K), 0.0010711 > secs] > [GC [DefNew: 512K->0K(576K), 0.0006255 secs] 650K->138K(1984K), 0.0007062 > secs] > [GC [DefNew: 512K->0K(576K), 0.0002065 secs] 650K->138K(1984K), 0.0002861 > secs] > [GC [DefNew: 512K->0K(576K), 0.0001936 secs] 650K->138K(1984K), 0.0002674 > secs] > [GC [DefNew: 512K->0K(576K), 0.0002045 secs] 650K->138K(1984K), 0.0002796 > secs] > [GC [DefNew: 512K->0K(576K), 0.0001704 secs] 650K->138K(1984K), 0.0002481 > secs] > [GC [DefNew: 512K->0K(576K), 0.0002229 secs] 650K->138K(1984K), 0.0003118 > secs] > > Reproduced in both jdk1.5 and 1.6. > > Can someone enlight me what's happening inside jvm (regarding class load > and gc) ? > > Thanks. > > ________________________________ > > This email (including any attachments) is confidential and may be legally > privileged. If you received this email in error, please delete it > immediately and do not copy it or use it for any purpose or disclose its > contents to any other person. Thank you. > > > ??????(????????????)???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120227/c3ab8844/attachment.html