From thomas.schatzl at oracle.com Mon Aug 5 09:48:34 2019 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Mon, 5 Aug 2019 02:48:34 -0700 (PDT) Subject: JVMLS 2019 workshop: G1 XP. Pain Paints. slides Message-ID: Hi all, at JVMLS 2019 there has been a workshop discussion from me about G1 experience and pain points. Part of that included a short retrospective and update on future plans with G1. Slides are available at http://cr.openjdk.java.net/~tschatzl/jvmls2019/jvmls2019.pdf . Hth. Thanks, Thomas From yangang at ec.com.cn Mon Aug 12 18:26:50 2019 From: yangang at ec.com.cn (=?gbk?B?WWFuIEdhbmc=?=) Date: Mon, 12 Aug 2019 18:26:50 +0800(CST) Subject: =?gbk?B?cXVlc3Rpb24gYWJvdXQgTGFzdCBkaXRjaCBjb2xsZWN0aW9u?= Message-ID: <466X8C225Hz5vNR@mx3.ec.com.cn> An HTML attachment was scrubbed... URL: From thomas.schatzl at oracle.com Tue Aug 13 15:50:29 2019 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Tue, 13 Aug 2019 17:50:29 +0200 Subject: question about Last ditch collection In-Reply-To: <466X8C225Hz5vNR@mx3.ec.com.cn> References: <466X8C225Hz5vNR@mx3.ec.com.cn> Message-ID: <5ee3e20d-9ff2-298d-952f-0fc6736cdc34@oracle.com> Hi, On 12.08.19 12:26, Yan Gang wrote: > Hi all: > > Question about Last ditch collection. > I used JDK 8 , find out info about Last ditch collection from gc log. > > Cause? ? ? ? ? ? ? ?CountAvg Time? ? ? ? ?Max Time? ? ? ? ? Total Time > ? ? ? Time % > Last ditch collection 8420 min 46 sec 593 ms1 hr 6 min 17 sec29 hrs 5 > min 13 sec49.27% > Metadata GC Threshold 10216 min 58 sec 215 ms59 min 70 ms? ? ? ? 28 hrs > 50 min 57 sec48.86% > > Have question: > > How to get?root cause? Metaspace?fragment ? The log should contain information about whether the last ditch collections collections are caused by a full java heap (i.e. heap memory usage) or not. I think the "Metadata GC threshold" collections also contain metadata usage information. Thanks, Thomas From thomas.schatzl at oracle.com Mon Aug 19 09:43:55 2019 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Mon, 19 Aug 2019 11:43:55 +0200 Subject: G1GC is not able reclaim bytes In-Reply-To: References: Message-ID: <994b71be-de12-df4b-6ff2-1243a372a4f6@oracle.com> Hi, (cc'ed hotspot-gc-use as it is the appropriate mailing list for such quesstions). On 13.08.19 17:01, shang xinli wrote: > There are two connected full GC but they are not able to reclaim bytes. Here are gc logs. > > > [Times: user=0.35 sys=0.00, real=0.03 secs] > > 2019-08-01T15:12:14.594+0000: 755123.074: [Full GC (Allocation Failure) 132G->132G(180G), 37.4525886 secs] > > [Eden: 0.0B(9216.0M)->0.0B(9216.0M) Survivors: 0.0B->0.0B Heap: 132.1G(180.0G)->132.1G(180.0G)], [Metaspace: 204550K->204459K(229376K)] > > [Times: user=68.14 sys=0.07, real=37.45 secs] > > 2019-08-01T15:12:52.048+0000: 755160.527: [Full GC (Allocation Failure) 132G->132G(180G), 37.4584894 secs] > > [Eden: 0.0B(9216.0M)->0.0B(9216.0M) Survivors: 0.0B->0.0B Heap: 132.1G(180.0G)->132.1G(180.0G)], [Metaspace: 204459K->204451K(229376K)] > > [Times: user=68.38 sys=0.05, real=37.46 secs] > > Here is the chart. We can see before and after GC, the heap is almost the same. > [cid:65dca3e8-3bc4-4b87-93dc-997ba0e601f2] > > > The service OOMs eventually, but we don't configure to generate heap dump. Is there a way to investigate why? > > The reason of the full gc might be humongous objects that use up lots of additional space given that you get full gcs at 132 GB out of 180. So does your application happen to use many - and I mean many - large objects (~16M in size)? You can add -XX:+UnlockExperimentalVMOptions -XX:G1LogLevel=finest to your command line? (I assume you use some JDK8 given the log output, but not sure) Of particular interest are the following lines. [Humongous Register: x.x ms] [Humongous Total: y] [Humongous Candidate: z] Where y shows you the number of live humongous objects at the time of the GC(s). If that value is rather high, we found the issue. The reason is that objects that have a total size that is just above the humongous object threshold always use up full regions. This can waste a lot of space. That threshold is >= half a region size, i.e. 16M in your case, since regions are 32M at that heap size by default. Such a humongous object wastes the other half of the region. Remember that an object adds a small header, so an object that has exactly 16M of usable data is actually 16M + a little bit in size. If possible, try to lower your object size a bit. Then the waste will be much smaller in any case. The only other option I can see is increasing the heap, or use a different collector. Thanks, Thomas