From xpromache at gmail.com Fri Jul 7 11:58:11 2017 From: xpromache at gmail.com (Nicolae Mihalache) Date: Fri, 7 Jul 2017 13:58:11 +0200 Subject: SoftReferences not cleaned up Message-ID: Hello, I have problems understanding why the SoftReferences are not cleaned up. This is my version of java: java -version java version "1.8.0_92" Java(TM) SE Runtime Environment (build 1.8.0_92-b14) Java HotSpot(TM) 64-Bit Server VM (build 25.92-b14, mixed mode) and I run it with the options -Xmx512m -XX:SoftRefLRUPolicyMSPerMB=1 Still the number of SoftReferences keep increasing and I see that some of them are 30+ hours old. I have a heap dump: https://drive.google.com/open?id=0B8_hkIRtZSShNUtTZUx3UTFuMEk I don't know how to debug further, any hint would be appreciated. thanks, nicolae -------------- next part -------------- An HTML attachment was scrubbed... URL: From ecki at zusammenkunft.net Fri Jul 7 21:16:52 2017 From: ecki at zusammenkunft.net (Bernd Eckenfels) Date: Fri, 7 Jul 2017 21:16:52 +0000 Subject: SoftReferences not cleaned up In-Reply-To: References: Message-ID: Can you upload the verbose GC log somewhere? I suspect you don't see old/mixed GCs since 30h. Gruss Bernd -- http://bernd.eckenfels.net ________________________________ From: hotspot-gc-use on behalf of Nicolae Mihalache Sent: Friday, July 7, 2017 1:58:11 PM To: hotspot-gc-use at openjdk.java.net Subject: SoftReferences not cleaned up Hello, I have problems understanding why the SoftReferences are not cleaned up. This is my version of java: java -version java version "1.8.0_92" Java(TM) SE Runtime Environment (build 1.8.0_92-b14) Java HotSpot(TM) 64-Bit Server VM (build 25.92-b14, mixed mode) and I run it with the options -Xmx512m -XX:SoftRefLRUPolicyMSPerMB=1 Still the number of SoftReferences keep increasing and I see that some of them are 30+ hours old. I have a heap dump: https://drive.google.com/open?id=0B8_hkIRtZSShNUtTZUx3UTFuMEk I don't know how to debug further, any hint would be appreciated. thanks, nicolae -------------- next part -------------- An HTML attachment was scrubbed... URL: From xpromache at gmail.com Mon Jul 10 12:52:17 2017 From: xpromache at gmail.com (Nicolae Mihalache) Date: Mon, 10 Jul 2017 14:52:17 +0200 Subject: SoftReferences not cleaned up In-Reply-To: References: Message-ID: I didn't have verbose gc enabled but I did enable it, and here is the log in the last 3 days: https://drive.google.com/drive/u/0/folders/0B8_hkIRtZSShdkdUTGhRMjNPVVk The "Full GC" is related to a jmap -histo:live I was running. On Fri, Jul 7, 2017 at 11:16 PM, Bernd Eckenfels wrote: > Can you upload the verbose GC log somewhere? I suspect you don't see > old/mixed GCs since 30h. > > Gruss > Bernd > -- > http://bernd.eckenfels.net > ------------------------------ > *From:* hotspot-gc-use on > behalf of Nicolae Mihalache > *Sent:* Friday, July 7, 2017 1:58:11 PM > *To:* hotspot-gc-use at openjdk.java.net > *Subject:* SoftReferences not cleaned up > > Hello, > > I have problems understanding why the SoftReferences are not cleaned up. > This is my version of java: > java -version > java version "1.8.0_92" > Java(TM) SE Runtime Environment (build 1.8.0_92-b14) > Java HotSpot(TM) 64-Bit Server VM (build 25.92-b14, mixed mode) > > and I run it with the options -Xmx512m -XX:SoftRefLRUPolicyMSPerMB=1 > > Still the number of SoftReferences keep increasing and I see that some of > them are 30+ hours old. I have a heap dump: > https://drive.google.com/open?id=0B8_hkIRtZSShNUtTZUx3UTFuMEk > > I don't know how to debug further, any hint would be appreciated. > > thanks, > nicolae > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yu.zhang at oracle.com Mon Jul 10 16:30:21 2017 From: yu.zhang at oracle.com (yu.zhang at oracle.com) Date: Mon, 10 Jul 2017 09:30:21 -0700 Subject: SoftReferences not cleaned up In-Reply-To: References:

Message-ID: Can you add PrintReferenceGC to print out number of references for each gc? Thanks Jenny On 07/10/2017 05:52 AM, Nicolae Mihalache wrote: > I didn't have verbose gc enabled but I did enable it, and here is the > log in the last 3 days: > https://drive.google.com/drive/u/0/folders/0B8_hkIRtZSShdkdUTGhRMjNPVVk > > > The "Full GC" is related to a jmap -histo:live I was running. > > On Fri, Jul 7, 2017 at 11:16 PM, Bernd Eckenfels > > wrote: > > Can you upload the verbose GC log somewhere? I suspect you don't > see old/mixed GCs since 30h. > > Gruss > Bernd > -- > http://bernd.eckenfels.net > ------------------------------------------------------------------------ > *From:* hotspot-gc-use

> on behalf of > Nicolae Mihalache > > *Sent:* Friday, July 7, 2017 1:58:11 PM > *To:* hotspot-gc-use at openjdk.java.net > > *Subject:* SoftReferences not cleaned up > Hello, > > I have problems understanding why the SoftReferences are not > cleaned up. This is my version of java: > java -version > java version "1.8.0_92" > Java(TM) SE Runtime Environment (build 1.8.0_92-b14) > Java HotSpot(TM) 64-Bit Server VM (build 25.92-b14, mixed mode) > > and I run it with the options -Xmx512m -XX:SoftRefLRUPolicyMSPerMB=1 > > Still the number of SoftReferences keep increasing and I see that > some of them are 30+ hours old. I have a heap dump: > https://drive.google.com/open?id=0B8_hkIRtZSShNUtTZUx3UTFuMEk > > > I don't know how to debug further, any hint would be appreciated. > > thanks, > nicolae > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > > > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: From xpromache at gmail.com Tue Jul 11 08:37:28 2017 From: xpromache at gmail.com (Nicolae Mihalache) Date: Tue, 11 Jul 2017 10:37:28 +0200 Subject: SoftReferences not cleaned up In-Reply-To: References:

Message-ID: Here is a new output with PrintReferenceGC on: https://drive.google.com/open?id=0B8_hkIRtZSShWTVpeVFONGlqVjQ I see that each time when I run jmap -histo:live a few hundreds of them are collected but still the number reported by jmap keeps increasing (5665 last time it was run). On Mon, Jul 10, 2017 at 6:30 PM, yu.zhang at oracle.com wrote: > Can you add PrintReferenceGC to print out number of references for each gc? > > Thanks > > Jenny > > On 07/10/2017 05:52 AM, Nicolae Mihalache wrote: > > I didn't have verbose gc enabled but I did enable it, and here is the log > in the last 3 days: > https://drive.google.com/drive/u/0/folders/0B8_hkIRtZSShdkdUTGhRMjNPVVk > > > The "Full GC" is related to a jmap -histo:live I was running. > > On Fri, Jul 7, 2017 at 11:16 PM, Bernd Eckenfels > wrote: > >> Can you upload the verbose GC log somewhere? I suspect you don't see >> old/mixed GCs since 30h. >> >> Gruss >> Bernd >> -- >> http://bernd.eckenfels.net >> ------------------------------ >> *From:* hotspot-gc-use on >> behalf of Nicolae Mihalache >> *Sent:* Friday, July 7, 2017 1:58:11 PM >> *To:* hotspot-gc-use at openjdk.java.net >> *Subject:* SoftReferences not cleaned up >> >> Hello, >> >> I have problems understanding why the SoftReferences are not cleaned up. >> This is my version of java: >> java -version >> java version "1.8.0_92" >> Java(TM) SE Runtime Environment (build 1.8.0_92-b14) >> Java HotSpot(TM) 64-Bit Server VM (build 25.92-b14, mixed mode) >> >> and I run it with the options -Xmx512m -XX:SoftRefLRUPolicyMSPerMB=1 >> >> Still the number of SoftReferences keep increasing and I see that some of >> them are 30+ hours old. I have a heap dump: >> https://drive.google.com/open?id=0B8_hkIRtZSShNUtTZUx3UTFuMEk >> >> I don't know how to debug further, any hint would be appreciated. >> >> thanks, >> nicolae >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> > > > _______________________________________________ > hotspot-gc-use mailing listhotspot-gc-use at openjdk.java.nethttp://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yu.zhang at oracle.com Wed Jul 12 19:18:27 2017 From: yu.zhang at oracle.com (yu.zhang at oracle.com) Date: Wed, 12 Jul 2017 12:18:27 -0700 Subject: SoftReferences not cleaned up In-Reply-To: References:

Message-ID: <36d76c03-8dee-b268-19ed-cdd31c29b06a@oracle.com> Hi, From the gc logs the SoftReferences are not that many. Thanks Jenny On 07/11/2017 01:37 AM, Nicolae Mihalache wrote: > Here is a new output with PrintReferenceGC on: > https://drive.google.com/open?id=0B8_hkIRtZSShWTVpeVFONGlqVjQ > > I see that each time when I run jmap -histo:live a few hundreds of > them are collected but still the number reported by jmap keeps > increasing (5665 last time it was run). > > On Mon, Jul 10, 2017 at 6:30 PM, yu.zhang at oracle.com > > wrote: > > Can you add PrintReferenceGC to print out number of references for > each gc? > > Thanks > > Jenny > > > On 07/10/2017 05:52 AM, Nicolae Mihalache wrote: >> I didn't have verbose gc enabled but I did enable it, and here >> is the log in the last 3 days: >> https://drive.google.com/drive/u/0/folders/0B8_hkIRtZSShdkdUTGhRMjNPVVk >> >> >> >> The "Full GC" is related to a jmap -histo:live I was running. >> >> On Fri, Jul 7, 2017 at 11:16 PM, Bernd Eckenfels >> > wrote: >> >> Can you upload the verbose GC log somewhere? I suspect you >> don't see old/mixed GCs since 30h. >> >> Gruss >> Bernd >> -- >> http://bernd.eckenfels.net >> ------------------------------------------------------------------------ >> *From:* hotspot-gc-use >> > > on behalf >> of Nicolae Mihalache > > >> *Sent:* Friday, July 7, 2017 1:58:11 PM >> *To:* hotspot-gc-use at openjdk.java.net >> >> *Subject:* SoftReferences not cleaned up >> Hello, >> >> I have problems understanding why the SoftReferences are not >> cleaned up. This is my version of java: >> java -version >> java version "1.8.0_92" >> Java(TM) SE Runtime Environment (build 1.8.0_92-b14) >> Java HotSpot(TM) 64-Bit Server VM (build 25.92-b14, mixed mode) >> >> and I run it with the options >> -Xmx512m -XX:SoftRefLRUPolicyMSPerMB=1 >> >> Still the number of SoftReferences keep increasing and I see >> that some of them are 30+ hours old. I have a heap dump: >> https://drive.google.com/open?id=0B8_hkIRtZSShNUtTZUx3UTFuMEk >> >> >> I don't know how to debug further, any hint would be appreciated. >> >> thanks, >> nicolae >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> >> >> >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From email.sundarms at gmail.com Thu Jul 13 06:52:30 2017 From: email.sundarms at gmail.com (Sundara Mohan M) Date: Wed, 12 Jul 2017 23:52:30 -0700 Subject: High Reference Processing/Object Copy time Message-ID: Sending to correct ilist, -- ________________________________ From: hotspot-gc-dev on behalf of Sundara Mohan M Sent: Thursday, July 13, 2017 4:11:41 AM To: hotspot-gc-dev at openjdk.java.net Subject: High Reference Processing/Object Copy time Hi, I am observing a odd behaviour (very high ref proc time once) with G1GC gc log snippet, flags used Java HotSpot(TM) 64-Bit Server VM (25.112-b15) for linux-amd64 JRE (1.8.0_112-b15), built on Sep 22 2016 21:10:53 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8) Memory: 4k page, physical 132290100k(1065596k free), swap 132120572k(131992000k free) CommandLine flags: -XX:G1MaxNewSizePercent=30 -XX: G1OldCSetRegionThresholdPercent=20 -XX:GCLogFileSize=20971520 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=out-of-memory-heap-dump -XX:InitialHeapSize=33285996544 -XX:MaxGCPauseMillis=500 -XX:MaxHeapSize=33285996544 -XX:MetaspaceSize=536870912 -XX:NumberOfGCLogFiles=20 -XX:+ParallelRefProcEnabled -XX:+PrintAdaptiveSizePolicy -XX:+PrintGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+UnlockExperimentalVMOptions -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC -XX:+UseGCLogFileRotation -XX:+UseStringDeduplication .... 2017-07-12T17:02:40.227+0000: 77743.943: [GC pause (G1 Evacuation Pause) (young) Desired survivor size 104857600 bytes, new threshold 2 (max 15) - age 1: 38456192 bytes, 38456192 total - age 2: 86746408 bytes, 125202600 total 77743.943: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 149039, predicted base time: 374.57 ms, remaining time: 125.43 ms, target pause time: 50 0.00 ms] 77743.943: [G1Ergonomics (CSet Construction) add young regions to CSet, eden: 174 regions, survivors: 24 regions, predicted young region time: 1277.98 ms] 77743.943: [G1Ergonomics (CSet Construction) finish choosing CSet, eden: 174 regions, survivors: 24 regions, old: 0 regions, predicted pause time: 1652.55 ms, target paus e time: 500.00 ms] 77751.132: [G1Ergonomics (Concurrent Cycles) request concurrent cycle initiation, reason: occupancy higher than threshold, occupancy: 21147680768 bytes, allocation reques t: 0 bytes, threshold: 14978698425 bytes (45.00 %), source: end of GC] , 7.1891696 secs] [Parallel Time: 2253.1 ms, GC Workers: 13] [GC Worker Start (ms): Min: 77743943.2, Avg: 77743943.3, Max: 77743943.4, Diff: 0.2] [Ext Root Scanning (ms): Min: 1.7, Avg: 3.5, Max: 6.5, Diff: 4.8, Sum: 44.9] [Update RS (ms): Min: 39.2, Avg: 42.4, Max: 45.1, Diff: 5.9, Sum: 551.8] [Processed Buffers: Min: 26, Avg: 57.4, Max: 78, Diff: 52, Sum: 746] [Scan RS (ms): Min: 1.8, Avg: 3.7, Max: 4.5, Diff: 2.7, Sum: 47.5] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0] [Object Copy (ms): Min: 2198.1, Avg: 2198.7, Max: 2202.7, Diff: 4.6, Sum: 28583.3] [Termination (ms): Min: 0.0, Avg: 4.5, Max: 4.9, Diff: 4.9, Sum: 58.4] [Termination Attempts: Min: 1, Avg: 16.7, Max: 28, Diff: 27, Sum: 217] [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.4] [GC Worker Total (ms): Min: 2252.7, Avg: 2252.8, Max: 2252.9, Diff: 0.2, Sum: 29286.3] [GC Worker End (ms): Min: 77746196.1, Avg: 77746196.1, Max: 77746196.1, Diff: 0.0] [Code Root Fixup: 0.1 ms] [Code Root Purge: 0.0 ms] [String Dedup Fixup: 167.7 ms, GC Workers: 13] [Queue Fixup (ms): Min: 0.0, Avg: 0.4, Max: 1.2, Diff: 1.2, Sum: 5.1] [Table Fixup (ms): Min: 165.5, Avg: 165.9, Max: 166.3, Diff: 0.9, Sum: 2156.9] [Clear CT: 1.5 ms] [Other: 4766.8 ms] [Choose CSet: 0.0 ms] [Ref Proc: 4763.9 ms] [Ref Enq: 0.8 ms] [Redirty Cards: 0.7 ms] [Humongous Register: 0.2 ms] [Humongous Reclaim: 0.1 ms] [Free CSet: 0.4 ms] [Eden: 1392.0M(1392.0M)->0.0B(1440.0M) Survivors: 192.0M->144.0M Heap: 20.8G(31.0G)->19.6G(31.0G)] [Times: user=22.82 sys=13.83, real=7.19 secs] Question 1. Is there a way to find out why Ref Proc took 4.7 s at this instance only? all other instances it was less than a second. 2. Why object copy took 2.1s even though young gen region size is 1.3G in this case and there was not much garbage collected in this case. 3. Why is this happening occasionally and is there a way to enable more logs when this happens. Thanks, Sundar Date: Thu, 13 Jul 2017 04:58:51 +0000 From: Bernd Eckenfels To: "hotspot-gc-dev at openjdk.java.net" Subject: Re: High Reference Processing/Object Copy time Message-ID: < HE1PR08MB2795E9C7342192F3E5524B9AFFAC0 at HE1PR08MB2795.eurprd08.prod.outlook.com > Content-Type: text/plain; charset="us-ascii" The sys time is very high in this snippet, how to the other snippets compare? Did you turn off transparent huge pages (THP) in your OS and is there no swapping happening? BTW: this is more a discussion for the user mailing list. Gruss Bernd -- http://bernd.eckenfels.net ________________________________ -- I do see THP is used in the process a lot of entries are there like this /proc/619/smaps:AnonHugePages: 120832 kB Also there are some swap activity in the system. Machine has 124G RAM. Thanks Sundar -------------- next part -------------- An HTML attachment was scrubbed... URL: From xpromache at gmail.com Thu Jul 13 07:48:04 2017 From: xpromache at gmail.com (Nicolae Mihalache) Date: Thu, 13 Jul 2017 09:48:04 +0200 Subject: SoftReferences not cleaned up In-Reply-To: <36d76c03-8dee-b268-19ed-cdd31c29b06a@oracle.com> References:

<36d76c03-8dee-b268-19ed-cdd31c29b06a@oracle.com> Message-ID: After running for two days: - jmap -histo:live |grep SoftReference -> reports 191255 SoftReference objects in he heap - grep GC |sed -r 's/^.*SoftReference, ([0-9]+).*$/\1/g' | awk '{s+=$1} END {print s}' -> reports 301025 soft references cleared I had a look through a heap dump and I see most of the 191K SoftReferences objects with the referent null. In fact there is one ReferenceQueue with 179K queue length where most SoftReference in that queue are like this (i.e. refer to null). So the question is why those SoftReference objects keep accumulating in the ReferenceQueue and they are not cleaned up after the object they refer to is cleaned. nicolae On Wed, Jul 12, 2017 at 9:18 PM, yu.zhang at oracle.com wrote: > Hi, > > From the gc logs the SoftReferences are not that many. > > Thanks > > Jenny > > On 07/11/2017 01:37 AM, Nicolae Mihalache wrote: > > Here is a new output with PrintReferenceGC on: > https://drive.google.com/open?id=0B8_hkIRtZSShWTVpeVFONGlqVjQ > > I see that each time when I run jmap -histo:live a few hundreds of them > are collected but still the number reported by jmap keeps increasing (5665 > last time it was run). > > On Mon, Jul 10, 2017 at 6:30 PM, yu.zhang at oracle.com > wrote: > >> Can you add PrintReferenceGC to print out number of references for each >> gc? >> >> Thanks >> >> Jenny >> >> On 07/10/2017 05:52 AM, Nicolae Mihalache wrote: >> >> I didn't have verbose gc enabled but I did enable it, and here is the >> log in the last 3 days: >> https://drive.google.com/drive/u/0/folders/0B8_hkIRtZSShdkdUTGhRMjNPVVk >> >> >> The "Full GC" is related to a jmap -histo:live I was running. >> >> On Fri, Jul 7, 2017 at 11:16 PM, Bernd Eckenfels >> wrote: >> >>> Can you upload the verbose GC log somewhere? I suspect you don't see >>> old/mixed GCs since 30h. >>> >>> Gruss >>> Bernd >>> -- >>> http://bernd.eckenfels.net >>> ------------------------------ >>> *From:* hotspot-gc-use on >>> behalf of Nicolae Mihalache >>> *Sent:* Friday, July 7, 2017 1:58:11 PM >>> *To:* hotspot-gc-use at openjdk.java.net >>> *Subject:* SoftReferences not cleaned up >>> >>> Hello, >>> >>> I have problems understanding why the SoftReferences are not cleaned >>> up. This is my version of java: >>> java -version >>> java version "1.8.0_92" >>> Java(TM) SE Runtime Environment (build 1.8.0_92-b14) >>> Java HotSpot(TM) 64-Bit Server VM (build 25.92-b14, mixed mode) >>> >>> and I run it with the options -Xmx512m -XX:SoftRefLRUPolicyMSPerMB=1 >>> >>> Still the number of SoftReferences keep increasing and I see that some >>> of them are 30+ hours old. I have a heap dump: >>> https://drive.google.com/open?id=0B8_hkIRtZSShNUtTZUx3UTFuMEk >>> >>> I don't know how to debug further, any hint would be appreciated. >>> >>> thanks, >>> nicolae >>> >>> _______________________________________________ >>> hotspot-gc-use mailing list >>> hotspot-gc-use at openjdk.java.net >>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>> >>> >> >> >> _______________________________________________ >> hotspot-gc-use mailing listhotspot-gc-use at openjdk.java.nethttp://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.schatzl at oracle.com Thu Jul 13 08:08:28 2017 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Thu, 13 Jul 2017 10:08:28 +0200 Subject: High Reference Processing/Object Copy time In-Reply-To: References: Message-ID: <1499933308.2815.12.camel@oracle.com> Hi, ? copying the answer from the hotspot-gc-dev list here (thanks!): The documentation offers some more potential issues when there is high system time: https://docs.oracle.com/javase/9/gctuning/garbage-first-ga rbage-collector-tuning.htm#GUID-8D9B2530-E370-4B8B-8ADD-A43674FC6658 (T he section is applicable to both JDK8 and 9). The VM/garbage collector is a user level program. High system time (at least as high as in your snippet) strongly indicates a problem in the environment (or in the interaction with your environment, i.e. memory or I/O related). [...] Thanks, ? Thomas On Wed, 2017-07-12 at 23:52 -0700, Sundara Mohan M wrote: > Sending to correct ilist, > -- > ________________________________ > From: hotspot-gc-dev on > behalf of Sundara Mohan M > Sent: Thursday, July 13, 2017 4:11:41 AM > To:?hotspot-gc-dev at openjdk.java.net > Subject: High Reference Processing/Object Copy time > > Hi, > ? ?I am observing a odd behaviour (very high ref proc time once) with > G1GC > > gc log snippet, > > flags used > Java HotSpot(TM) 64-Bit Server VM (25.112-b15) for linux-amd64 JRE > (1.8.0_112-b15), built on Sep 22 2016 21:10:53 by "java_re" with gcc > 4.3.0 20080428 (Red Hat 4.3.0-8) > Memory: 4k page, physical 132290100k(1065596k free), swap > 132120572k(131992000k free) > CommandLine flags: -XX:G1MaxNewSizePercent=30 > -XX:G1OldCSetRegionThresholdPercent=20 -XX:GCLogFileSize=20971520 > -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=out-of-memory-heap- > dump -XX:InitialHeapSize=33285996544 -XX:MaxGCPauseMillis=500 > -XX:MaxHeapSize=33285996544 -XX:MetaspaceSize=536870912 > -XX:NumberOfGCLogFiles=20 -XX:+ParallelRefProcEnabled > -XX:+PrintAdaptiveSizePolicy -XX:+PrintGC > -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps > -XX:+PrintGCDetails -XX:+PrintGCTimeStamps > -XX:+PrintTenuringDistribution -XX:+UnlockExperimentalVMOptions > -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC > -XX:+UseGCLogFileRotation -XX:+UseStringDeduplication > > > .... > 2017-07-12T17:02:40.227+0000: 77743.943: [GC pause (G1 Evacuation > Pause) (young) > Desired survivor size 104857600 bytes, new threshold 2 (max 15) > - age? ?1:? ?38456192 bytes,? ?38456192 total > - age? ?2:? ?86746408 bytes,? 125202600 total > ?77743.943: [G1Ergonomics (CSet Construction) start choosing CSet, > _pending_cards: 149039, predicted base time: 374.57 ms, remaining > time: 125.43 ms, target pause time: 50 > 0.00 ms] > ?77743.943: [G1Ergonomics (CSet Construction) add young regions to > CSet, eden: 174 regions, survivors: 24 regions, predicted young > region time: 1277.98 ms] > ?77743.943: [G1Ergonomics (CSet Construction) finish choosing CSet, > eden: 174 regions, survivors: 24 regions, old: 0 regions, predicted > pause time: 1652.55 ms, target paus > e time: 500.00 ms] > ?77751.132: [G1Ergonomics (Concurrent Cycles) request concurrent > cycle initiation, reason: occupancy higher than threshold, occupancy: > 21147680768 bytes, allocation reques > t: 0 bytes, threshold: 14978698425 bytes (45.00 %), source: end of > GC] > , 7.1891696 secs] > ? ?[Parallel Time: 2253.1 ms, GC Workers: 13] > ? ? ? [GC Worker Start (ms): Min: 77743943.2, Avg: 77743943.3, Max: > 77743943.4, Diff: 0.2] > ? ? ? [Ext Root Scanning (ms): Min: 1.7, Avg: 3.5, Max: 6.5, Diff: > 4.8, Sum: 44.9] > ? ? ? [Update RS (ms): Min: 39.2, Avg: 42.4, Max: 45.1, Diff: 5.9, > Sum: 551.8] > ? ? ? ? ?[Processed Buffers: Min: 26, Avg: 57.4, Max: 78, Diff: 52, > Sum: 746] > ? ? ? [Scan RS (ms): Min: 1.8, Avg: 3.7, Max: 4.5, Diff: 2.7, Sum: > 47.5] > ? ? ? [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > 0.0, Sum: 0.0] > ? ? ? [Object Copy (ms): Min: 2198.1, Avg: 2198.7, Max: 2202.7, Diff: > 4.6, Sum: 28583.3] > ? ? ? [Termination (ms): Min: 0.0, Avg: 4.5, Max: 4.9, Diff: 4.9, > Sum: 58.4] > ? ? ? ? ?[Termination Attempts: Min: 1, Avg: 16.7, Max: 28, Diff: 27, > Sum: 217] > ? ? ? [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, > Sum: 0.4] > ? ? ? [GC Worker Total (ms): Min: 2252.7, Avg: 2252.8, Max: 2252.9, > Diff: 0.2, Sum: 29286.3] > ? ? ? [GC Worker End (ms): Min: 77746196.1, Avg: 77746196.1, Max: > 77746196.1, Diff: 0.0] > ? ?[Code Root Fixup: 0.1 ms] > ? ?[Code Root Purge: 0.0 ms] > ? ?[String Dedup Fixup: 167.7 ms, GC Workers: 13] > ? ? ? [Queue Fixup (ms): Min: 0.0, Avg: 0.4, Max: 1.2, Diff: 1.2, > Sum: 5.1] > ? ? ? [Table Fixup (ms): Min: 165.5, Avg: 165.9, Max: 166.3, Diff: > 0.9, Sum: 2156.9] > ? ?[Clear CT: 1.5 ms] > ? ?[Other: 4766.8 ms] > ? ? ? [Choose CSet: 0.0 ms] > ? ? ? [Ref Proc: 4763.9 ms] > ? ? ? [Ref Enq: 0.8 ms] > ? ? ? [Redirty Cards: 0.7 ms] > ? ? ? [Humongous Register: 0.2 ms] > ? ? ? [Humongous Reclaim: 0.1 ms] > ? ? ? [Free CSet: 0.4 ms] > ? ?[Eden: 1392.0M(1392.0M)->0.0B(1440.0M) Survivors: 192.0M->144.0M > Heap: 20.8G(31.0G)->19.6G(31.0G)] > ?[Times: user=22.82 sys=13.83, real=7.19 secs] > > Question > 1. Is there a way to find out why Ref Proc took 4.7 s at this > instance only? all other instances it was less than a second. > 2. Why object copy took 2.1s even though young gen region size is > 1.3G in this case and there was not much garbage collected in this > case. > 3. Why is this happening occasionally and is there a way to enable > more logs when this happens. > > > Thanks, > Sundar > Date: Thu, 13 Jul 2017 04:58:51 +0000 > From: Bernd Eckenfels > To: "hotspot-gc-dev at openjdk.java.net" > ? ? ? ? > Subject: Re: High Reference Processing/Object Copy time > Message-ID: > ? ? ? ? 8.prod.outlook.com> > > Content-Type: text/plain; charset="us-ascii" > > The sys time is very high in this snippet, how to the other snippets > compare? Did you turn off transparent huge pages (THP) in your OS and > is there no swapping happening? > > BTW: this is more a discussion for the user mailing list. > > Gruss > Bernd > -- > http://bernd.eckenfels.net > ________________________________ > > -- > > > I do see THP is used in the process > a lot of entries are there like this > /proc/619/smaps:AnonHugePages: ? ?120832 kB > > Also there are some swap activity in the system. > Machine has 124G RAM. > > Thanks > Sundar > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From xpromache at gmail.com Thu Jul 13 14:30:20 2017 From: xpromache at gmail.com (Nicolae Mihalache) Date: Thu, 13 Jul 2017 16:30:20 +0200 Subject: SoftReferences not cleaned up In-Reply-To: References:

<36d76c03-8dee-b268-19ed-cdd31c29b06a@oracle.com> Message-ID: Finally I see the light... I was confused between objects referred by soft references not being cleared by GC (they were) and the SoftReference object themselves which were supposed to be removed from the ReferenceQueue by another thread which was never started. My problem was all due to a bug in Apache VFS which was actually fixed 2 years ago (but I never updated to the new version): https://issues.apache.org/jira/browse/VFS-480 nicolae On Thu, Jul 13, 2017 at 9:48 AM, Nicolae Mihalache wrote: > After running for two days: > > - jmap -histo:live |grep SoftReference > -> reports 191255 SoftReference objects in he heap > > > > - grep GC |sed -r 's/^.*SoftReference, ([0-9]+).*$/\1/g' | > awk '{s+=$1} END {print s}' > -> reports 301025 soft references cleared > > I had a look through a heap dump and I see most of the 191K SoftReferences > objects with the referent null. In fact there is one ReferenceQueue with > 179K queue length where most SoftReference in that queue are like this > (i.e. refer to null). > So the question is why those SoftReference objects keep accumulating in > the ReferenceQueue and they are not cleaned up after the object they refer > to is cleaned. > > > nicolae > > > > > On Wed, Jul 12, 2017 at 9:18 PM, yu.zhang at oracle.com > wrote: > >> Hi, >> >> From the gc logs the SoftReferences are not that many. >> >> Thanks >> >> Jenny >> >> On 07/11/2017 01:37 AM, Nicolae Mihalache wrote: >> >> Here is a new output with PrintReferenceGC on: >> https://drive.google.com/open?id=0B8_hkIRtZSShWTVpeVFONGlqVjQ >> >> I see that each time when I run jmap -histo:live a few hundreds of them >> are collected but still the number reported by jmap keeps increasing (5665 >> last time it was run). >> >> On Mon, Jul 10, 2017 at 6:30 PM, yu.zhang at oracle.com > > wrote: >> >>> Can you add PrintReferenceGC to print out number of references for each >>> gc? >>> >>> Thanks >>> >>> Jenny >>> >>> On 07/10/2017 05:52 AM, Nicolae Mihalache wrote: >>> >>> I didn't have verbose gc enabled but I did enable it, and here is the >>> log in the last 3 days: >>> https://drive.google.com/drive/u/0/folders/0B8_hkIRtZSShdkdUTGhRMjNPVVk >>> >>> >>> The "Full GC" is related to a jmap -histo:live I was running. >>> >>> On Fri, Jul 7, 2017 at 11:16 PM, Bernd Eckenfels >> > wrote: >>> >>>> Can you upload the verbose GC log somewhere? I suspect you don't see >>>> old/mixed GCs since 30h. >>>> >>>> Gruss >>>> Bernd >>>> -- >>>> http://bernd.eckenfels.net >>>> ------------------------------ >>>> *From:* hotspot-gc-use on >>>> behalf of Nicolae Mihalache >>>> *Sent:* Friday, July 7, 2017 1:58:11 PM >>>> *To:* hotspot-gc-use at openjdk.java.net >>>> *Subject:* SoftReferences not cleaned up >>>> >>>> Hello, >>>> >>>> I have problems understanding why the SoftReferences are not cleaned >>>> up. This is my version of java: >>>> java -version >>>> java version "1.8.0_92" >>>> Java(TM) SE Runtime Environment (build 1.8.0_92-b14) >>>> Java HotSpot(TM) 64-Bit Server VM (build 25.92-b14, mixed mode) >>>> >>>> and I run it with the options -Xmx512m -XX:SoftRefLRUPolicyMSPerMB=1 >>>> >>>> Still the number of SoftReferences keep increasing and I see that some >>>> of them are 30+ hours old. I have a heap dump: >>>> https://drive.google.com/open?id=0B8_hkIRtZSShNUtTZUx3UTFuMEk >>>> >>>> I don't know how to debug further, any hint would be appreciated. >>>> >>>> thanks, >>>> nicolae >>>> >>>> _______________________________________________ >>>> hotspot-gc-use mailing list >>>> hotspot-gc-use at openjdk.java.net >>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>>> >>>> >>> >>> >>> _______________________________________________ >>> hotspot-gc-use mailing listhotspot-gc-use at openjdk.java.nethttp://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>> >>> >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From email.sundarms at gmail.com Thu Jul 13 15:51:57 2017 From: email.sundarms at gmail.com (Sundara Mohan M) Date: Thu, 13 Jul 2017 08:51:57 -0700 Subject: High Reference Processing/Object Copy time In-Reply-To: <1499933308.2815.12.camel@oracle.com> References: <1499933308.2815.12.camel@oracle.com> Message-ID: Thanks for tuning link. I have 2 more JVM (31G heap with CMS GC) running on same machine but gc logs never showed sys time > 1.4 only the JVM running with G1GC shows very high sys time. Another thing noticed is resident size is high compared to another process which has CMS GC. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1438 test 20 0 42.0g 27g 34m S 283.8 21.7 5232:24 java . >>> CMSGC 619 test 20 0 42.0g 26g 34m S 105.0 21.3 5085:47 java . >>> CMSGC 600 test 20 0 43.4g *33g* 33m S 4.3 26.9 361:54.17 java . >>> G1GC Is this something to do with G1GC region based allocation? Thanks Sundar On Thu, Jul 13, 2017 at 1:08 AM, Thomas Schatzl wrote: > Hi, > > copying the answer from the hotspot-gc-dev list here (thanks!): > > The documentation offers some more potential issues when there is high > system time: https://docs.oracle.com/javase/9/gctuning/garbage-first-ga > rbage-collector-tuning.htm#GUID-8D9B2530-E370-4B8B-8ADD-A43674FC6658 (T > he section is applicable to both JDK8 and 9). > > The VM/garbage collector is a user level program. High system time (at > least as high as in your snippet) strongly indicates a problem in the > environment (or in the interaction with your environment, i.e. memory > or I/O related). > > [...] > > Thanks, > Thomas > > On Wed, 2017-07-12 at 23:52 -0700, Sundara Mohan M wrote: > > Sending to correct ilist, > > -- > > ________________________________ > > From: hotspot-gc-dev on > > behalf of Sundara Mohan M > > Sent: Thursday, July 13, 2017 4:11:41 AM > > To: hotspot-gc-dev at openjdk.java.net > > Subject: High Reference Processing/Object Copy time > > > > Hi, > > I am observing a odd behaviour (very high ref proc time once) with > > G1GC > > > > gc log snippet, > > > > flags used > > Java HotSpot(TM) 64-Bit Server VM (25.112-b15) for linux-amd64 JRE > > (1.8.0_112-b15), built on Sep 22 2016 21:10:53 by "java_re" with gcc > > 4.3.0 20080428 (Red Hat 4.3.0-8) > > Memory: 4k page, physical 132290100k(1065596k free), swap > > 132120572k(131992000k free) > > CommandLine flags: -XX:G1MaxNewSizePercent=30 > > -XX:G1OldCSetRegionThresholdPercent=20 -XX:GCLogFileSize=20971520 > > -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=out-of-memory-heap- > > dump -XX:InitialHeapSize=33285996544 -XX:MaxGCPauseMillis=500 > > -XX:MaxHeapSize=33285996544 -XX:MetaspaceSize=536870912 > > -XX:NumberOfGCLogFiles=20 -XX:+ParallelRefProcEnabled > > -XX:+PrintAdaptiveSizePolicy -XX:+PrintGC > > -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps > > -XX:+PrintGCDetails -XX:+PrintGCTimeStamps > > -XX:+PrintTenuringDistribution -XX:+UnlockExperimentalVMOptions > > -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC > > -XX:+UseGCLogFileRotation -XX:+UseStringDeduplication > > > > > > .... > > 2017-07-12T17:02:40.227+0000: 77743.943: [GC pause (G1 Evacuation > > Pause) (young) > > Desired survivor size 104857600 bytes, new threshold 2 (max 15) > > - age 1: 38456192 bytes, 38456192 total > > - age 2: 86746408 bytes, 125202600 total > > 77743.943: [G1Ergonomics (CSet Construction) start choosing CSet, > > _pending_cards: 149039, predicted base time: 374.57 ms, remaining > > time: 125.43 ms, target pause time: 50 > > 0.00 ms] > > 77743.943: [G1Ergonomics (CSet Construction) add young regions to > > CSet, eden: 174 regions, survivors: 24 regions, predicted young > > region time: 1277.98 ms] > > 77743.943: [G1Ergonomics (CSet Construction) finish choosing CSet, > > eden: 174 regions, survivors: 24 regions, old: 0 regions, predicted > > pause time: 1652.55 ms, target paus > > e time: 500.00 ms] > > 77751.132: [G1Ergonomics (Concurrent Cycles) request concurrent > > cycle initiation, reason: occupancy higher than threshold, occupancy: > > 21147680768 bytes, allocation reques > > t: 0 bytes, threshold: 14978698425 bytes (45.00 %), source: end of > > GC] > > , 7.1891696 secs] > > [Parallel Time: 2253.1 ms, GC Workers: 13] > > [GC Worker Start (ms): Min: 77743943.2, Avg: 77743943.3, Max: > > 77743943.4, Diff: 0.2] > > [Ext Root Scanning (ms): Min: 1.7, Avg: 3.5, Max: 6.5, Diff: > > 4.8, Sum: 44.9] > > [Update RS (ms): Min: 39.2, Avg: 42.4, Max: 45.1, Diff: 5.9, > > Sum: 551.8] > > [Processed Buffers: Min: 26, Avg: 57.4, Max: 78, Diff: 52, > > Sum: 746] > > [Scan RS (ms): Min: 1.8, Avg: 3.7, Max: 4.5, Diff: 2.7, Sum: > > 47.5] > > [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: > > 0.0, Sum: 0.0] > > [Object Copy (ms): Min: 2198.1, Avg: 2198.7, Max: 2202.7, Diff: > > 4.6, Sum: 28583.3] > > [Termination (ms): Min: 0.0, Avg: 4.5, Max: 4.9, Diff: 4.9, > > Sum: 58.4] > > [Termination Attempts: Min: 1, Avg: 16.7, Max: 28, Diff: 27, > > Sum: 217] > > [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, > > Sum: 0.4] > > [GC Worker Total (ms): Min: 2252.7, Avg: 2252.8, Max: 2252.9, > > Diff: 0.2, Sum: 29286.3] > > [GC Worker End (ms): Min: 77746196.1, Avg: 77746196.1, Max: > > 77746196.1, Diff: 0.0] > > [Code Root Fixup: 0.1 ms] > > [Code Root Purge: 0.0 ms] > > [String Dedup Fixup: 167.7 ms, GC Workers: 13] > > [Queue Fixup (ms): Min: 0.0, Avg: 0.4, Max: 1.2, Diff: 1.2, > > Sum: 5.1] > > [Table Fixup (ms): Min: 165.5, Avg: 165.9, Max: 166.3, Diff: > > 0.9, Sum: 2156.9] > > [Clear CT: 1.5 ms] > > [Other: 4766.8 ms] > > [Choose CSet: 0.0 ms] > > [Ref Proc: 4763.9 ms] > > [Ref Enq: 0.8 ms] > > [Redirty Cards: 0.7 ms] > > [Humongous Register: 0.2 ms] > > [Humongous Reclaim: 0.1 ms] > > [Free CSet: 0.4 ms] > > [Eden: 1392.0M(1392.0M)->0.0B(1440.0M) Survivors: 192.0M->144.0M > > Heap: 20.8G(31.0G)->19.6G(31.0G)] > > [Times: user=22.82 sys=13.83, real=7.19 secs] > > > > Question > > 1. Is there a way to find out why Ref Proc took 4.7 s at this > > instance only? all other instances it was less than a second. > > 2. Why object copy took 2.1s even though young gen region size is > > 1.3G in this case and there was not much garbage collected in this > > case. > > 3. Why is this happening occasionally and is there a way to enable > > more logs when this happens. > > > > > > Thanks, > > Sundar > > Date: Thu, 13 Jul 2017 04:58:51 +0000 > > From: Bernd Eckenfels > > To: "hotspot-gc-dev at openjdk.java.net" > > > > Subject: Re: High Reference Processing/Object Copy time > > Message-ID: > > > 8.prod.outlook.com> > > > > Content-Type: text/plain; charset="us-ascii" > > > > The sys time is very high in this snippet, how to the other snippets > > compare? Did you turn off transparent huge pages (THP) in your OS and > > is there no swapping happening? > > > > BTW: this is more a discussion for the user mailing list. > > > > Gruss > > Bernd > > -- > > http://bernd.eckenfels.net > > ________________________________ > > > > -- > > > > > > I do see THP is used in the process > > a lot of entries are there like this > > /proc/619/smaps:AnonHugePages: 120832 kB > > > > Also there are some swap activity in the system. > > Machine has 124G RAM. > > > > Thanks > > Sundar > > > > > > _______________________________________________ > > hotspot-gc-use mailing list > > hotspot-gc-use at openjdk.java.net > > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.schatzl at oracle.com Mon Jul 17 14:12:00 2017 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Mon, 17 Jul 2017 16:12:00 +0200 Subject: High Reference Processing/Object Copy time In-Reply-To: References: <1499933308.2815.12.camel@oracle.com> Message-ID: <1500300720.2845.21.camel@oracle.com> Hi, On Thu, 2017-07-13 at 08:51 -0700, Sundara Mohan M wrote: > Thanks for tuning link. > > I have 2 more JVM (31G heap with CMS GC) running on same machine but > gc?logs never showed sys time > 1.4 only the JVM running with G1GC > shows very high sys time. > Another thing noticed is resident size is high compared to another > process which has CMS GC. > ? PID USER ? ? ?PR ?NI ?VIRT ?RES ?SHR S %CPU %MEM ? ?TIME+ ?COMMAND > ?1438 test ? ? ? ?20 ? 0 42.0g ?27g ?34m S 283.8 21.7 ? 5232:24 java > . >>> CMSGC > ? 619 test ? ? ? ? 20 ? 0 42.0g ?26g ?34m S 105.0 21.3 ? 5085:47 java > . >>> CMSGC > ? 600 test ? ? ? ?20 ? 0 43.4g ?33g ?33m S ?4.3 26.9 361:54.17 java . > ?>>> G1GC > > > Is this something to do with G1GC region based allocation? Yes, regions play into that, most likely remembered sets (used to do per-region evacuation, but can't tell without an NMT snapshots [0], and even then it is somewhat guessing). You can decrease remembered set memory usage by increasing heap region size (I think it is only 16m in your case, i.e. you are free to try 32m), or try to rebalance remembered set memory usage. The threads [1] and [2] give some insights on what to look for and what knobs to turn. JDK9 contains some improvements in the remembered set area. Also make sure to compare apples-to-apples. G1 has different heap sizing mechanisms, so it might just use a larger part of your provided Java heap memory. NMT can also show you that in addition to your logs. Thanks, ? Thomas [0]?https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot /tooldescr007.html [1]?http://mail.openjdk.java.net/pipermail/hotspot-gc-use/2017-February /002609.html [2]?http://mail.openjdk.java.net/pipermail/hotspot-gc-use/2 017-March/002640.html From email.sundarms at gmail.com Tue Jul 18 17:41:23 2017 From: email.sundarms at gmail.com (Sundara Mohan M) Date: Tue, 18 Jul 2017 10:41:23 -0700 Subject: G1 Old Generation GC collection event not sent after Mixed evacuation pause Message-ID: Hi, I am trying to capture all GC event happened on G1 Old Generation. I was expecting when G1 Evacuation pause (mixed) happens it should trigger event to my app but instead, i am getting notification only when Full GC happens. Here is what i am doing gcMxBeanListener = myFunction(...) oldGen = ManagementFactiory.getGarbageCollectorMXBeans find { "G1 Old Generation" } oldGen.addNotificationListener(gcMxBeanListener, null, null) But in the case of CMS GC, it is triggered when the concurrent cycle is completed. Is there something i am missing? Thanks, Sundar -------------- next part -------------- An HTML attachment was scrubbed... URL: From yu.zhang at oracle.com Wed Jul 19 15:24:42 2017 From: yu.zhang at oracle.com (yu.zhang at oracle.com) Date: Wed, 19 Jul 2017 08:24:42 -0700 Subject: G1 Old Generation GC collection event not sent after Mixed evacuation pause In-Reply-To: References: Message-ID: Sundara, I have filed this bug https://bugs.openjdk.java.net/browse/JDK-8145923 But has not been fixed. Jenny On 07/18/2017 10:41 AM, Sundara Mohan M wrote: > Hi, > I am trying to capture all GC event happened on G1 Old Generation. > I was expecting when G1 Evacuation pause (mixed) happens it should > trigger event to my app but instead, i am getting notification only > when Full GC happens. > > Here is what i am doing > gcMxBeanListener = myFunction(...) > oldGen = ManagementFactiory.getGarbageCollectorMXBeans find { "G1 Old > Generation" } > oldGen.addNotificationListener(gcMxBeanListener, null, null) > > But in the case of CMS GC, it is triggered when the concurrent cycle > is completed. > > > Is there something i am missing? > > Thanks, > Sundar > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: From email.sundarms at gmail.com Wed Jul 19 18:03:00 2017 From: email.sundarms at gmail.com (Sundara Mohan M) Date: Wed, 19 Jul 2017 11:03:00 -0700 Subject: G1 Old Generation GC collection event not sent after Mixed evacuation pause In-Reply-To: References:

Message-ID: Jenny, Thanks for the update. Another question i was looking at the source and saw this, enum G1YCType { Normal, InitialMark, DuringMark, Mixed, G1YCTypeEndSentinel }; Any idea why InitialMark are categorized as Young Collection, though it is processing in Old Generation? Also mixed can include old regions in which case it should be categorized as Old Generation. Just trying to understand if this is designed like this or this is kind of bug? Thanks, Sundar On Wed, Jul 19, 2017 at 8:24 AM, yu.zhang at oracle.com wrote: > Sundara, > > I have filed this bug https://bugs.openjdk.java.net/browse/JDK-8145923 > > But has not been fixed. > > Jenny > > On 07/18/2017 10:41 AM, Sundara Mohan M wrote: > > Hi, > I am trying to capture all GC event happened on G1 Old Generation. > I was expecting when G1 Evacuation pause (mixed) happens it should trigger > event to my app but instead, i am getting notification only when Full GC > happens. > > Here is what i am doing > > gcMxBeanListener = myFunction(...) > oldGen = ManagementFactiory.getGarbageCollectorMXBeans find { "G1 Old > Generation" } > oldGen.addNotificationListener(gcMxBeanListener, null, null) > > But in the case of CMS GC, it is triggered when the concurrent cycle is > completed. > > > Is there something i am missing? > > Thanks, > Sundar > > > _______________________________________________ > hotspot-gc-use mailing listhotspot-gc-use at openjdk.java.nethttp://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yu.zhang at oracle.com Wed Jul 19 19:09:01 2017 From: yu.zhang at oracle.com (Jenny Zhang) Date: Wed, 19 Jul 2017 12:09:01 -0700 Subject: G1 Old Generation GC collection event not sent after Mixed evacuation pause In-Reply-To: References:

Message-ID: <622bf13a-10f8-03c4-776d-abb1d1d68a95@oracle.com> Sundar, Initial mark is a young gc piggy backed with some initial marking activities. "Initial Mark : This type of collection starts the marking process in addition to performing a regular young-only collection. Concurrent marking determines all currently reachable (live) objects in the old generation regions to be kept for the following space-reclamation phase. While marking hasn?t completely finished, regular young collections may occur. Marking finishes with two special stop-the-world pauses: Remark and Cleanup." From http://docs.oracle.com/javase/9/gctuning/garbage-first-garbage-collector.htm#JSGCT-GUID-1CDEB6B6-9463-4998-815D-05E095BFBD0F Thanks Jenny On 7/19/2017 11:03 AM, Sundara Mohan M wrote: > Jenny, > Thanks for the update. > > Another question i was looking at the source and saw this, > > enum G1YCType { > Normal, > InitialMark, > DuringMark, > Mixed, > G1YCTypeEndSentinel > }; > > > Any idea why InitialMark are categorized as Young Collection, though > it is processing in Old Generation? > Also mixed can include old regions in which case it should be > categorized as Old Generation. > > Just trying to understand if this is designed like this or this is > kind of bug? > > > Thanks, > Sundar > > On Wed, Jul 19, 2017 at 8:24 AM, yu.zhang at oracle.com > > wrote: > > Sundara, > > I have filed this bug > https://bugs.openjdk.java.net/browse/JDK-8145923 > > > But has not been fixed. > > Jenny > > > On 07/18/2017 10:41 AM, Sundara Mohan M wrote: >> Hi, >> I am trying to capture all GC event happened on G1 Old Generation. >> I was expecting when G1 Evacuation pause (mixed) happens it >> should trigger event to my app but instead, i am getting >> notification only when Full GC happens. >> >> Here is what i am doing >> gcMxBeanListener = myFunction(...) >> oldGen = ManagementFactiory.getGarbageCollectorMXBeans find { >> "G1 Old Generation" } >> oldGen.addNotificationListener(gcMxBeanListener, null, null) >> >> But in the case of CMS GC, it is triggered when the concurrent >> cycle is completed. >> >> >> Is there something i am missing? >> >> Thanks, >> Sundar >> >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From email.sundarms at gmail.com Wed Jul 19 19:15:35 2017 From: email.sundarms at gmail.com (Sundara Mohan M) Date: Wed, 19 Jul 2017 12:15:35 -0700 Subject: G1 Old Generation GC collection event not sent after Mixed evacuation pause In-Reply-To: <622bf13a-10f8-03c4-776d-abb1d1d68a95@oracle.com> References:

<622bf13a-10f8-03c4-776d-abb1d1d68a95@oracle.com> Message-ID: >From the document, i see Space reclamation phase (multiple mixed collections) is when the object from old gen is collected. So shouldn't we get notified via OldGen GarbageCollectorMXBean instead of YoungGen MXBean? I am still trying to understand if this was designed to be like this because of logical space/generation in G1GC or it is a bug? Thanks Sundar On Wed, Jul 19, 2017 at 12:09 PM, Jenny Zhang wrote: > Sundar, > > Initial mark is a young gc piggy backed with some initial marking > activities. > > "Initial Mark : This type of collection starts the marking process in > addition to performing a regular young-only collection. Concurrent marking > determines all currently reachable (live) objects in the old generation > regions to be kept for the following space-reclamation phase. While marking > hasn?t completely finished, regular young collections may occur. Marking > finishes with two special stop-the-world pauses: Remark and Cleanup." > > From http://docs.oracle.com/javase/9/gctuning/garbage-first- > garbage-collector.htm#JSGCT-GUID-1CDEB6B6-9463-4998-815D-05E095BFBD0F > > Thanks > > Jenny > > On 7/19/2017 11:03 AM, Sundara Mohan M wrote: > > Jenny, > Thanks for the update. > > Another question i was looking at the source and saw this, > > enum G1YCType { > Normal, > InitialMark, > DuringMark, > Mixed, > G1YCTypeEndSentinel > }; > > > Any idea why InitialMark are categorized as Young Collection, though it is > processing in Old Generation? > Also mixed can include old regions in which case it should be categorized > as Old Generation. > > Just trying to understand if this is designed like this or this is kind of > bug? > > > Thanks, > Sundar > > On Wed, Jul 19, 2017 at 8:24 AM, yu.zhang at oracle.com > wrote: > >> Sundara, >> >> I have filed this bug https://bugs.openjdk.java.net/browse/JDK-8145923 >> >> But has not been fixed. >> >> Jenny >> >> On 07/18/2017 10:41 AM, Sundara Mohan M wrote: >> >> Hi, >> I am trying to capture all GC event happened on G1 Old Generation. >> I was expecting when G1 Evacuation pause (mixed) happens it should >> trigger event to my app but instead, i am getting notification only when >> Full GC happens. >> >> Here is what i am doing >> >> gcMxBeanListener = myFunction(...) >> oldGen = ManagementFactiory.getGarbageCollectorMXBeans find { "G1 Old >> Generation" } >> oldGen.addNotificationListener(gcMxBeanListener, null, null) >> >> But in the case of CMS GC, it is triggered when the concurrent cycle is >> completed. >> >> >> Is there something i am missing? >> >> Thanks, >> Sundar >> >> >> _______________________________________________ >> hotspot-gc-use mailing listhotspot-gc-use at openjdk.java.nethttp://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yu.zhang at oracle.com Wed Jul 19 20:35:58 2017 From: yu.zhang at oracle.com (Jenny Zhang) Date: Wed, 19 Jul 2017 13:35:58 -0700 Subject: G1 Old Generation GC collection event not sent after Mixed evacuation pause In-Reply-To: References:

<622bf13a-10f8-03c4-776d-abb1d1d68a95@oracle.com> Message-ID: <2e4a76e5-9c72-968a-ef5b-508e0adae83b@oracle.com> You are right about the mixed gc reclaiming old gen. If I understand you correctly, your question is about why initial marking is young gc? It only collects young gen. thanks Jenny On 7/19/2017 12:15 PM, Sundara Mohan M wrote: > From the document, i see Space reclamation phase (multiple mixed > collections) is when the object from old gen is collected. So > shouldn't we get notified via OldGen GarbageCollectorMXBean instead of > YoungGen MXBean? > > I am still trying to understand if this was designed to be like this > because of logical space/generation in G1GC or it is a bug? > > Thanks > Sundar > > On Wed, Jul 19, 2017 at 12:09 PM, Jenny Zhang > wrote: > > Sundar, > > Initial mark is a young gc piggy backed with some initial marking > activities. > > "Initial Mark : This type of collection starts the marking process > in addition to performing a regular young-only collection. > Concurrent marking determines all currently reachable (live) > objects in the old generation regions to be kept for the following > space-reclamation phase. While marking hasn?t completely finished, > regular young collections may occur. Marking finishes with two > special stop-the-world pauses: Remark and Cleanup." > > From > http://docs.oracle.com/javase/9/gctuning/garbage-first-garbage-collector.htm#JSGCT-GUID-1CDEB6B6-9463-4998-815D-05E095BFBD0F > > > Thanks > > Jenny > > > On 7/19/2017 11:03 AM, Sundara Mohan M wrote: >> Jenny, >> Thanks for the update. >> >> Another question i was looking at the source and saw this, >> >> enum G1YCType { >> Normal, >> InitialMark, >> DuringMark, >> Mixed, >> G1YCTypeEndSentinel >> }; >> >> >> Any idea why InitialMark are categorized as Young Collection, >> though it is processing in Old Generation? >> Also mixed can include old regions in which case it should be >> categorized as Old Generation. >> >> Just trying to understand if this is designed like this or this >> is kind of bug? >> >> >> Thanks, >> Sundar >> >> On Wed, Jul 19, 2017 at 8:24 AM, yu.zhang at oracle.com >> > > wrote: >> >> Sundara, >> >> I have filed this bug >> https://bugs.openjdk.java.net/browse/JDK-8145923 >> >> >> But has not been fixed. >> >> Jenny >> >> >> On 07/18/2017 10:41 AM, Sundara Mohan M wrote: >>> Hi, >>> I am trying to capture all GC event happened on G1 Old >>> Generation. >>> I was expecting when G1 Evacuation pause (mixed) happens it >>> should trigger event to my app but instead, i am getting >>> notification only when Full GC happens. >>> >>> Here is what i am doing >>> gcMxBeanListener = myFunction(...) >>> oldGen = ManagementFactiory.getGarbageCollectorMXBeans find >>> { "G1 Old Generation" } >>> oldGen.addNotificationListener(gcMxBeanListener, null, null) >>> >>> But in the case of CMS GC, it is triggered when the >>> concurrent cycle is completed. >>> >>> >>> Is there something i am missing? >>> >>> Thanks, >>> Sundar >>> >>> >>> _______________________________________________ >>> hotspot-gc-use mailing list >>> hotspot-gc-use at openjdk.java.net >>> >>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From email.sundarms at gmail.com Wed Jul 19 21:35:33 2017 From: email.sundarms at gmail.com (Sundara Mohan M) Date: Wed, 19 Jul 2017 14:35:33 -0700 Subject: G1 Old Generation GC collection event not sent after Mixed evacuation pause In-Reply-To: <2e4a76e5-9c72-968a-ef5b-508e0adae83b@oracle.com> References:

<622bf13a-10f8-03c4-776d-abb1d1d68a95@oracle.com> <2e4a76e5-9c72-968a-ef5b-508e0adae83b@oracle.com> Message-ID: Thanks Jenny. Got why initial marking is young (it just process old gen and collects only young gen). So mixed GC reclaiming is still a bug and not designed to be like that correct? Thanks, Sundar On Wed, Jul 19, 2017 at 1:35 PM, Jenny Zhang wrote: > You are right about the mixed gc reclaiming old gen. If I understand you > correctly, your question is about why initial marking is young gc? It only > collects young gen. > > thanks > > Jenny > > On 7/19/2017 12:15 PM, Sundara Mohan M wrote: > > From the document, i see Space reclamation phase (multiple mixed > collections) is when the object from old gen is collected. So shouldn't we > get notified via OldGen GarbageCollectorMXBean instead of YoungGen MXBean? > > I am still trying to understand if this was designed to be like this > because of logical space/generation in G1GC or it is a bug? > > Thanks > Sundar > > On Wed, Jul 19, 2017 at 12:09 PM, Jenny Zhang wrote: > >> Sundar, >> >> Initial mark is a young gc piggy backed with some initial marking >> activities. >> >> "Initial Mark : This type of collection starts the marking process in >> addition to performing a regular young-only collection. Concurrent marking >> determines all currently reachable (live) objects in the old generation >> regions to be kept for the following space-reclamation phase. While marking >> hasn?t completely finished, regular young collections may occur. Marking >> finishes with two special stop-the-world pauses: Remark and Cleanup." >> >> From http://docs.oracle.com/javase/9/gctuning/garbage-first-garba >> ge-collector.htm#JSGCT-GUID-1CDEB6B6-9463-4998-815D-05E095BFBD0F >> >> Thanks >> >> Jenny >> >> On 7/19/2017 11:03 AM, Sundara Mohan M wrote: >> >> Jenny, >> Thanks for the update. >> >> Another question i was looking at the source and saw this, >> >> enum G1YCType { >> Normal, >> InitialMark, >> DuringMark, >> Mixed, >> G1YCTypeEndSentinel >> }; >> >> >> Any idea why InitialMark are categorized as Young Collection, though it >> is processing in Old Generation? >> Also mixed can include old regions in which case it should be categorized >> as Old Generation. >> >> Just trying to understand if this is designed like this or this is kind >> of bug? >> >> >> Thanks, >> Sundar >> >> On Wed, Jul 19, 2017 at 8:24 AM, yu.zhang at oracle.com > > wrote: >> >>> Sundara, >>> >>> I have filed this bug https://bugs.openjdk.java.net/browse/JDK-8145923 >>> >>> But has not been fixed. >>> >>> Jenny >>> >>> On 07/18/2017 10:41 AM, Sundara Mohan M wrote: >>> >>> Hi, >>> I am trying to capture all GC event happened on G1 Old Generation. >>> I was expecting when G1 Evacuation pause (mixed) happens it should >>> trigger event to my app but instead, i am getting notification only when >>> Full GC happens. >>> >>> Here is what i am doing >>> >>> gcMxBeanListener = myFunction(...) >>> oldGen = ManagementFactiory.getGarbageCollectorMXBeans find { "G1 Old >>> Generation" } >>> oldGen.addNotificationListener(gcMxBeanListener, null, null) >>> >>> But in the case of CMS GC, it is triggered when the concurrent cycle is >>> completed. >>> >>> >>> Is there something i am missing? >>> >>> Thanks, >>> Sundar >>> >>> >>> _______________________________________________ >>> hotspot-gc-use mailing listhotspot-gc-use at openjdk.java.nethttp://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>> >>> >>> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From monishkumarit at gmail.com Thu Jul 6 16:57:46 2017 From: monishkumarit at gmail.com (monish kumar) Date: Thu, 06 Jul 2017 16:57:46 -0000 Subject: Fwd: Reg : Questions for tuning in G1GC Message-ID: Hi Team, We are hosting a application with 56 Gig of ram among that we have allocated 36 Gig for Jvm. I have attached my recent GC Logs please reivew all of that, I have a question do we really need to have space for permenant generation when we are using G1GC. Available processors 8 Here is the Jvm Args we used. please let me know if anything i need to add or remove args which also i need to provide valid explanation to my higher officials. -XX:PermSize=12288m -XX:MaxPermSize=12288m -Xms24576m -Xmx24576m -Duser.timezone=UTC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc: log\gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10m -XX:+HeapDumpOnOutOfMemoryError -XX:ReservedCodeCacheSize=512m -XX:HeapDumpPath=C:\temp\ -XX:SoftRefLRUPolicyMSPerMB=100 -XX:+UseG1GC -Dsun.rmi.dgc.server.gcInterval=31536000000 -Dsun.rmi.dgc.client.gcInterval=31536000000 -XX:CMSMaxAbortablePrecleanTime=15000 -- Thanks & Regards, A.Monish Kumar. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: gc.log1.zip Type: application/zip Size: 1343295 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: gc.log2.zip Type: application/zip Size: 1368927 bytes Desc: not available URL: From monishkumarit at gmail.com Thu Jul 6 17:00:15 2017 From: monishkumarit at gmail.com (monish kumar) Date: Thu, 06 Jul 2017 17:00:15 -0000 Subject: Reg : Questions for tuning in G1GC In-Reply-To: References: Message-ID: ++latest logs On Thu, Jul 6, 2017 at 10:20 PM, monish kumar wrote: > Hi Team, > > We are hosting a application with 56 Gig of ram among that we have > allocated 36 Gig for Jvm. > > I have attached my recent GC Logs please reivew all of that, I have a > question do we really need to have space for permenant generation when we > are using G1GC. > > Available processors 8 > > Here is the Jvm Args we used. > > please let me know if anything i need to add or remove args which also i > need to provide valid explanation to my higher officials. > > -XX:PermSize=12288m > -XX:MaxPermSize=12288m > -Xms24576m > -Xmx24576m > -Duser.timezone=UTC > -XX:+PrintGCDetails > -XX:+PrintGCDateStamps > -Xloggc: log\gc.log > -XX:+UseGCLogFileRotation > -XX:NumberOfGCLogFiles=10 > -XX:GCLogFileSize=10m > -XX:+HeapDumpOnOutOfMemoryError > -XX:ReservedCodeCacheSize=512m > -XX:HeapDumpPath=C:\temp\ > > -XX:SoftRefLRUPolicyMSPerMB=100 > -XX:+UseG1GC > -Dsun.rmi.dgc.server.gcInterval=31536000000 > -Dsun.rmi.dgc.client.gcInterval=31536000000 > -XX:CMSMaxAbortablePrecleanTime=15000 > > > -- > Thanks & Regards, > A.Monish Kumar. > > > > -- Thanks & Regards, A.Monish Kumar. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: gc.log3.zip Type: application/zip Size: 1320271 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: gc.log03.zip Type: application/zip Size: 1320487 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: gc.log01.zip Type: application/zip Size: 1402800 bytes Desc: not available URL: