October 2015 - hotspot-gc-dev

[JDK 6u21]: does CMS report concurrent mode failure on perm gen expansion ?
by Charlie Hunt 05 Dec '18

05 Dec '18

I'm looking at a GC log and surprised by the "concurrent mode failure" messages. I'm seeing the following GC log entries as the first 8 GCs: 3.398: [Full GC 3.398: [CMS (concurrent mode failure)[YG occupancy: 594957 K (1887488 K)]3.489: [weak refs processing, 0.0000140 secs]: 0K->0K(6291456K), 0.1308660 secs] 594957K->594957K(8178944K), [CMS Perm : 21247K->21247K(21248K)], 0.1310260 secs] [Times: user=0.13 sys=0.03, real=0.14 secs] 3.529: [Full GC 3.529: [CMS (concurrent mode failure)[YG occupancy: 594957 K (1887488 K)]3.617: [weak refs processing, 0.0000110 secs]: 0K->0K(6291456K), 0.1010290 secs] 594957K->594957K(8178944K), [CMS Perm : 21248K->21248K(42496K)], 0.1011530 secs] [Times: user=0.12 sys=0.00, real=0.10 secs] 3.638: [Full GC 3.638: [CMS (concurrent mode failure)[YG occupancy: 662071 K (1887488 K)]3.730: [weak refs processing, 0.0000140 secs]: 0K->0K(6291456K), 0.1068710 secs] 662071K->662071K(8178944K), [CMS Perm : 21319K->21319K(42688K)], 0.1069610 secs] [Times: user=0.15 sys=0.00, real=0.11 secs] 3.772: [Full GC 3.772: [CMS (concurrent mode failure)[YG occupancy: 729184 K (1887488 K)]3.915: [weak refs processing, 0.0000140 secs]: 0K->0K(6291456K), 0.1582170 secs] 729184K->729184K(8178944K), [CMS Perm : 21499K->21499K(42880K)], 0.1583500 secs] [Times: user=0.20 sys=0.00, real=0.16 secs] 3.962: [Full GC 3.962: [CMS (concurrent mode failure)[YG occupancy: 804686 K (1887488 K)]4.107: [weak refs processing, 0.0000130 secs]: 0K->0K(6291456K), 0.1598050 secs] 804686K->804686K(8178944K), [CMS Perm : 21702K->21702K(43072K)], 0.1599420 secs] [Times: user=0.21 sys=0.00, real=0.16 secs] 4.497: [Full GC 4.497: [CMS (concurrent mode failure)[YG occupancy: 939150 K (1887488 K)]4.672: [weak refs processing, 0.0000130 secs]: 0K->0K(6291456K), 0.1895350 secs] 939150K->939150K(8178944K), [CMS Perm : 22202K->22202K(43408K)], 0.1896680 secs] [Times: user=0.23 sys=0.00, real=0.19 secs] 5.054: [Full GC 5.054: [CMS (concurrent mode failure)[YG occupancy: 992483 K (1887488 K)]5.274: [weak refs processing, 0.0000130 secs]: 0K->0K(6291456K), 0.2347170 secs] 992483K->992483K(8178944K), [CMS Perm : 24058K->24058K(44408K)], 0.2348700 secs] [Times: user=0.28 sys=0.00, real=0.23 secs] 6.662: [Full GC 6.662: [CMS (concurrent mode failure)[YG occupancy: 1292895 K (1887488 K)]6.953: [weak refs processing, 0.0000140 secs]: 0K->0K(6291456K), 0.3063690 secs] 1292895K->1292895K(8178944K), [CMS Perm : 30692K->30692K(48120K)], 0.3065860 secs] [Times: user=0.66 sys=0.01, real=0.30 secs] After these 8 GCs, the GC log is much inline with what I'd expect, i.e. ParNew GCs, initial-marks, remarks, etc. In each of the above GCs: - young gen is increasing in occupancy - old gen remains empty - perm gen space size is expanding I can understand the Full GC being reported since perm gen is expanding, but I don't understand the reporting of "concurrent mode failure". Am I missing something? Is it possible HotSpot is reporting concurrent mode failure when perm gen is expanding? Here's the set of command line options in use: -Xmx8G -Xms8G -Xmn1500m -XX:+UseCompressedOops -XX:-UseGCOverheadLimit -XX:+UseConcMarkSweepGC -XX:MaxDirectMemorySize=5000m thanks, charlie ... PS: I didn't go look at / investigate OpenJDK HotSpot source code since this is JDK 6u21 and thought it might be possible something may have changed between 6u21 and what's in OpenJDK.

4 4

Re: Low-Overhead Heap Profiling
by Tony Printezis 04 Apr '17

04 Apr '17

Jeremy (and all), I’m not on the serviceability list so I won’t include the messages so far. :-) Also CCing the hotspot GC list, in case they have some feedback on this. Could I suggest a (much) simpler but at least as powerful and flexible way to do this? (This is something we’ve been meaning to do for a while now for TwitterJDK, the JDK we develop and deploy here at Twitter.) You can force allocations to go into the slow path periodically by artificially setting the TLAB top to a lower value. So, imagine a TLAB is 4M. You can set top to (bottom+1M). When an allocation thinks the TLAB is full (in this case, the first 1MB is full) it will call the allocation slow path. There, you can intercept it, sample the allocation (and, like in your case, you’ll also have the correct stack trace), notice that the TLAB is not actually full, extend its to top to, say, (bottom+2M), and you’re done. Advantages of this approach: * This is a much smaller, simpler, and self-contained change (no compiler changes necessary to maintain...). * When it’s off, the overhead is only one extra test at the slow path TLAB allocation (i.e., negligible; we do some sampling on TLABs in TwitterJDK using a similar mechanism and, when it’s off, I’ve observed no performance overhead). * (most importantly) You can turn this on and off, and adjust the sampling rate, dynamically. If you do the sampling based on JITed code, you’ll have to recompile all methods with allocation sites to turn the sampling on or off. (You can of course have it always on and just discard the output; it’d be nice not to have to do that though. IMHO, at least.) * You can also very cheaply turn this on and off (or adjust the sampling frequncy) per thread, if that’s be helpful in some way (just add the appropriate info on the thread’s TLAB). A few extra comments on the previous discussion: * "JFR samples per new TLAB allocation. It provides really very good picture and I haven't seen overhead more than 2” : When TLABs get very large, I don’t think sampling one object per TLAB is enough to get a good sample (IMHO, at least). It’s probably OK for something like jbb which mostly allocates instances of a handful of classes and has very few allocation sites. But, a lot of the code we run at Twitter is a lot more elaborate than that and, in our experience, sampling one object per TLAB is not enough. You can, of course, decrease the TLAB size to increase the sampling size. But it’d be good not to have to do that given a smaller TLAB size could increase contention across threads. * "Should it *just* take a stack trace, or should the behavior be configurable?” : I think we’d have to separate the allocation sampling mechanism from the consumption of the allocation samples. Once the sampling mechanism is in, different JVMs can take advantage of it in different ways. I assume that the Oracle folks would like at least a JFR event for every such sample. But in your build you can add extra code to collect the information in the way you have now. * Talking of JFR, it’s a bit unfortunate that the AllocObjectInNewTLAB event has both the new TLAB information and the allocation information. It would have been nice if that event was split into two, say NewTLAB and AllocObjectInTLAB, and we’d be able to fire the latter for each sample. * "Should the interval between samples be configurable?” : Totally. In fact, it’d be helpful if it was configurable dynamically. Imagine if a JVM starts misbehaving after 2-3 weeks of running. You can dynamically increase the sampling rate to get a better profile if the default is not giving fine-grain enough information. * "As long of these features don’t contribute to sampling bias” : If the sampling interval is fixed, sampling bias would be a very real concern. In the above example, I’d increment top by 1M (the sampling frequency) + p% (a fudge factor). * "Yes, a perhaps optional callbacks would be nice too.” : Oh, no. :-) But, as I said, we should definitely separate the sampling mechanism from the mechanism that consumes the samples. * "Another problem with our submitting things is that we can't really test on anything other than Linux.” : Another reason to go with a as platform independent solution as possible. :-) Regards, Tony ----- Tony Printezis | JVM/GC Engineer / VM Team | Twitter @TonyPrintezis tprintezis(a)twitter.com

7 31

PerfData counter: sun.gc.policy.generations in JDK 8
by Srinivas Ramakrishna 29 Apr '16

29 Apr '16

With perm gen going away (and being replaced by metaspace) in JDK 8, it makes sense that the counter sun.gc.policy.generations should be "2", rather than "3". However, in JDK 8 that counter still says 3. As I understand, the intention was that this counter would allow you to (for example) know the range of the sun.gc.generation.$num.* counters describing each of $num < sun.gc.policy.generations in the heap. Recall that the erstwhile perm gen in JDK 7 used to be synonymous with sun.gc.generation.2, but the JDK 8 avatars are now sun.gc.metaspace and sun.gc.compressedclassspace. The fix is simple, and I can submit a patch. Is there an existing bug for this? thanks! -- ramki

4 7

RFR (S): 8135200: Add White Box method that enumerates G1 old regions with less than specified liveness and collects statistics.
by Kirill Zhaldybin 13 Apr '16

13 Apr '16

Hi! Could you please review a patch for JDK-8135200? I added White Box method that enumerates G1 old regions with less than specified liveness and collects statistics. CR: https://bugs.openjdk.java.net/browse/JDK-8135200 Webrev: http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8135200/webrev.00/ hotspot: http://cr.openjdk.java.net/~kzhaldyb/webrevs/JDK-8135200/hotspot/webrev.00/ Thank you. Regards, Kirill

5 8

RFR(xs): 8134749: SoftReferences declared dead too early
by Per Liden 10 Mar '16

10 Mar '16

Hi, While preparing a different patch I looked through the ReferenceProcessor and noticed what looks like a silent mistreatment of SoftReferences. Summary: RefProcPhase1Task uses WorkerThread::id() to select the list to process. After the recent changes to how WorkGang works[1], this no longer guarantees that all reference lists are processed. This in turn means we might kill SoftReferences even when the policy says they should be kept alive. The fix here would be to use the worker_id (named "i" in this code) passed into the Task's work() function instead of using WorkerThread::id(). [1] There's no longer any guarantee that all GangWorker threads will execute the Task's work() function, since a quick-to-wakeup worker thread might steal the work intended for a slow-to-wakeup worker thread. Webrev: http://cr.openjdk.java.net/~pliden/8134749/webrev.0/ Bug: https://bugs.openjdk.java.net/browse/JDK-8134749 Testing: jprt Btw, I'll to send out a follow-up patch (JDK-8134744) which makes sure WorkerThread::id() and the worker_id always match to avoid this type of bugs in the future. /Per

3 7

RFR (XS): 8078673: Update TEST.groups for recent GC tests
by Derek White 01 Mar '16

01 Mar '16

Please review this simple for GC tests that require certain collectors (e.g. shouldn't run in embedded). Certain GC regression tests require specific collectors to be enabled in the JVM (by using -XX:+UseParallelGC, etc). But these tests were not listed in TEST.groups. This webrev adds the following tests to the appropriate needs_g1gc, needs_serialgc, needs_parallelgc, and needs_cmsgc lists. * gc/TestSmallHeap.java * gc/TestCardTablePageCommits.java * gc/arguments/TestParallelHeapSizeFlags.java gc/ergonomics/TestDynamicNumberOfGCThreads.java * gc/logging/TestGCId.java CR: https://bugs.openjdk.java.net/browse/JDK-8078673 Webrev: http://cr.openjdk.java.net/~drwhite/8078673/webrev.00/ Testing: JPRT Thanks, - Derek

7 21

RFR (M): JDK-8068394: Trace event for concurrent GC phases
by sangheon.kim 18 Dec '15

18 Dec '15

Hi all, Can I get some reviews for this change of adding a trace event for concurrent phases on CMS and G1? Currently we only measure pause times for major phases. But I want to add 'concurrent mark' and 'concurrent sweep' phases for CMS and 'concurrent mark' phase for G1. To achieve this, I had to change ConcurrentGCTimer and related classes. This patch includes: 1) A concurrent phase can be measured only from ConcurrentGCTimer and measuring an overlap between concurrent and pause is not allowed as currently we don't have that use case. And TimePartitions class(or related classes) will be simpler than an overlap allowed case. 2) I removed PausePhase and ConcurrentPhase which are derived from GCPhase because I wanted to avoid heap allocation when adding to GrowableArray. Instead introduced 'type' member variable at GCPhase. CR: https://bugs.openjdk.java.net/browse/JDK-8068394 Webrev: http://cr.openjdk.java.net/~sangheki/8068394/webrev.00/ Testing: JPRT Thanks, Sangheon

4 19

RFR (XS): Forward port of 8065331: Add trace events for failed allocations
by Marcus Larsson 18 Dec '15

18 Dec '15

Hi, Please review the following patch to forward port the JEP-242 event to JDK9. The patch was originally reviewed in April/March [1], but was only integrated in 8u60. The original patch no longer applies cleanly to 9, due to directory restructuring and changes to includes, so I need a quick re-review for this. The new patch has updated #includes for the new directory structure but should otherwise be identical to the previous patch. Webrev: http://cr.openjdk.java.net/~mlarsson/8065331_to_9/webrev.00/ Original webrev: http://cr.openjdk.java.net/~mlarsson/8065331/webrev.02/ Issue: https://bugs.openjdk.java.net/browse/JDK-8065331 Thanks, Marcus [1]: http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2015-March/012408.html

3 9

Request for code review - 8139664: Delete ConcurrentMarkSweepThread::is_ConcurrentGC_thread()
by Alexander Harlap 03 Dec '15

03 Dec '15

This is for technical debt , trivial change - remove virtual method with same implementation as on base class. JDK-8139664 <https://bugs.openjdk.java.net/browse/JDK-8139664> Proposed change: http://cr.openjdk.java.net/~aharlap/8139664/webrev.00/ Thank you, Alex

7 37

RFR (S) 8137099: OoME with G1 GC before doing a full GC
by Siebenborn, Axel 27 Nov '15

27 Nov '15

Hi, we regularly see OoM-Errors with G1 in our stress tests. We run the tests with the same heap size with ParallelGC and CMS without that problem. The stress tests are based on real world application code with a lot of threads. Scenario: We have an application with a lot of threads and spend time in critical native sections. 1. An evacuation failure happens during a GC. 2. After clean-up work, the safepoint is left. 3. An other thread can't allocate and triggers a new incremental gc. 4. A thread, that can't allocate after an incremental GC, triggers a full GC. However, the GC doesn't start because an other thread started an incremental GC, the GC-locker is active or the GCLocker initiated GC has not yet been performed. If an incremental GC doesn't succeed due to the GC-locker, and if this happens more often than GCLockerRetryAllocationCount (=2) an OOME is thrown. Without critical native code, we would try to trigger a full gc until we succeed. In this case there is just a performance issue, but not an OOME. Despite to other GCs, the safepoint is left after an evacuation failure. The proposed fix is to start a full GC before leaving the safepoint. Bug: https://bugs.openjdk.java.net/browse/JDK-8137099 Webrev: http://cr.openjdk.java.net/~asiebenborn/8137099/webrev/ Thanks, Axel

4 9