From java at elyograg.org Thu Jan 1 19:00:24 2015 From: java at elyograg.org (Shawn Heisey) Date: Thu, 01 Jan 2015 12:00:24 -0700 Subject: G1 with Solr - thread from dev@lucene.apache.org In-Reply-To: <1420035545.3277.2.camel@oracle.com> References: <54906C34.8080408@elyograg.org> <1418831456.3255.22.camel@oracle.com> <5491D659.1090703@elyograg.org> <1418849513.3293.3.camel@oracle.com> <5494D0D7.2010606@elyograg.org> <1419170477.6868.1.camel@oracle.com> <5499AA73.9090003@elyograg.org> <1419357347.3128.1.camel@oracle.com> <5499D8F9.9020607@elyograg.org> <1419934339.3250.1.camel@oracle.com> <54A2ECFC.1010401@elyograg.org> <54A321F5.7060209@oracle.com> <54A34373.9000509@elyograg.org> <1420035545.3277.2.camel@oracle.com> Message-ID: <54A59948.8050602@elyograg.org> On 12/31/2014 7:19 AM, Thomas Schatzl wrote: > -XX:G1HeapRegionSize=4M should be sufficient: all the objects I have > seen are slightly smaller than 2M, which corresponds to Shawn's > statement about having around 16.3M bits in length. > > With -Xms4G -Xmx6G the default region size is 2M, not 4M. Using > -XX:G1HeapRegionSize=8M seems overkill. Sounds reasonable. Smaller regions probably means better performance, but I think I'll leave my region size at 8MB. My indexes are always growing, and will soon be large enough that every filter will be larger than 50 percent of a 4MB region. I would have to increase it to 8MB anyway in the near future. > There are two other options that could be tried to improve the situation > (although I think increasing the heap region size should be sufficient), > that is > > -XX:-ResizePLAB > > which decreases the amount of space G1 will waste during GC (it does so > for performance reasons, but the logic is somewhat flawed - I am > currently working on that). > > The other is to cap the young gen size so that the amount of potential > survivors is smaller in the first place, e.g. > > -XX:G1MaxNewSize=1536M // 1.5G seems reasonable without decreasing > throughput too much; a lot of these full gcs seem to appear after G1 > using extremely large eden sizes. > > This is most likely due to the spiky allocation behavior of the > application: i.e. long stretches of almost every object dying, and then > short bursts. Since G1 tunes itself to the former, it will simply try to > use too much eden size for these spikes. > > But I recommend first seeing the impact of the increase in region size. With the 8M region size, the situation looks MUCH better -- the multi-second pauses are gone. There are a handful of longer pauses, but the average is well below the 200ms target. The gcviewer program doesn't calculate median and percentile (75th, 95th, 99th, etc) values ... I would be very interested in knowing what those numbers are. Perhaps I need to grab the source code and fiddle with it. https://www.dropbox.com/s/wzn6dhckl9sr05n/gc-idxa1-g1-7u72-8m-region-size.zip?dl=0 I can continue to tweak settings as you have suggested, but the graph of this log looks far better than I dared hope ... any further improvement is just icing on the cake. I have updated my personal Solr wiki page with my current GC settings and some notes about the G1 region size: https://wiki.apache.org/solr/ShawnHeisey#GC_Tuning Thanks, Shawn From yu.zhang at oracle.com Fri Jan 2 21:41:09 2015 From: yu.zhang at oracle.com (Yu Zhang) Date: Fri, 02 Jan 2015 13:41:09 -0800 Subject: G1 with Solr - thread from dev@lucene.apache.org In-Reply-To: <54A59948.8050602@elyograg.org> References: <54906C34.8080408@elyograg.org> <1418831456.3255.22.camel@oracle.com> <5491D659.1090703@elyograg.org> <1418849513.3293.3.camel@oracle.com> <5494D0D7.2010606@elyograg.org> <1419170477.6868.1.camel@oracle.com> <5499AA73.9090003@elyograg.org> <1419357347.3128.1.camel@oracle.com> <5499D8F9.9020607@elyograg.org> <1419934339.3250.1.camel@oracle.com> <54A2ECFC.1010401@elyograg.org> <54A321F5.7060209@oracle.com> <54A34373.9000509@elyograg.org> <1420035545.3277.2.camel@oracle.com> <54A59948.8050602@elyograg.org> Message-ID: <54A71075.9030106@oracle.com> Shawn, Glad it works out for you. There are a lot of weak references in the gc log. Maybe you can investigate this in the application. I dumped the gc pauses from your gc log. You can import those to Excel and get the statistics you are looking for. I will send you those csv files to you separately. Thanks, Jenny On 1/1/2015 11:00 AM, Shawn Heisey wrote: > On 12/31/2014 7:19 AM, Thomas Schatzl wrote: >> -XX:G1HeapRegionSize=4M should be sufficient: all the objects I have >> seen are slightly smaller than 2M, which corresponds to Shawn's >> statement about having around 16.3M bits in length. >> >> With -Xms4G -Xmx6G the default region size is 2M, not 4M. Using >> -XX:G1HeapRegionSize=8M seems overkill. > Sounds reasonable. Smaller regions probably means better performance, > but I think I'll leave my region size at 8MB. My indexes are always > growing, and will soon be large enough that every filter will be larger > than 50 percent of a 4MB region. I would have to increase it to 8MB > anyway in the near future. > >> There are two other options that could be tried to improve the situation >> (although I think increasing the heap region size should be sufficient), >> that is >> >> -XX:-ResizePLAB >> >> which decreases the amount of space G1 will waste during GC (it does so >> for performance reasons, but the logic is somewhat flawed - I am >> currently working on that). >> >> The other is to cap the young gen size so that the amount of potential >> survivors is smaller in the first place, e.g. >> >> -XX:G1MaxNewSize=1536M // 1.5G seems reasonable without decreasing >> throughput too much; a lot of these full gcs seem to appear after G1 >> using extremely large eden sizes. >> >> This is most likely due to the spiky allocation behavior of the >> application: i.e. long stretches of almost every object dying, and then >> short bursts. Since G1 tunes itself to the former, it will simply try to >> use too much eden size for these spikes. >> >> But I recommend first seeing the impact of the increase in region size. > With the 8M region size, the situation looks MUCH better -- the > multi-second pauses are gone. There are a handful of longer pauses, but > the average is well below the 200ms target. The gcviewer program > doesn't calculate median and percentile (75th, 95th, 99th, etc) values > ... I would be very interested in knowing what those numbers are. > Perhaps I need to grab the source code and fiddle with it. > > https://www.dropbox.com/s/wzn6dhckl9sr05n/gc-idxa1-g1-7u72-8m-region-size.zip?dl=0 > > I can continue to tweak settings as you have suggested, but the graph of > this log looks far better than I dared hope ... any further improvement > is just icing on the cake. > > I have updated my personal Solr wiki page with my current GC settings > and some notes about the G1 region size: > > https://wiki.apache.org/solr/ShawnHeisey#GC_Tuning > > Thanks, > Shawn > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From pdugan at azulsystems.com Fri Jan 2 23:15:10 2015 From: pdugan at azulsystems.com (Patricia Dugan) Date: Fri, 2 Jan 2015 23:15:10 +0000 Subject: what about Azul's Zing JVM? Message-ID: <82950260-E814-4C02-A17F-C9353E1844ED@azulsystems.com> Hey there. We would love to answer all of your questions about Zulu, Zing or Java. Please let me know if you?re still looking around for info. Gary is right, QCON has a ton of great talks by Gil Tene, our CTO. Thanks, Gary! Here?s one from React last month: http://bit.ly/java-latency-jvm I?d love to chat with you about Zulu, our Free, Trusted, Certified, Open Source implementation of Java 8 & OpenJDK 8. You can download it here: http://bit.ly/azul-zulu-java Please follow us on Twitter for conversation and let?s keep it going here! Thanks for your patience. =] @patricia_dugan @azulsystems -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.schatzl at oracle.com Wed Jan 7 12:52:05 2015 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Wed, 07 Jan 2015 13:52:05 +0100 Subject: G1 with Solr - thread from dev@lucene.apache.org In-Reply-To: <54A59948.8050602@elyograg.org> References: <54906C34.8080408@elyograg.org> <1418831456.3255.22.camel@oracle.com> <5491D659.1090703@elyograg.org> <1418849513.3293.3.camel@oracle.com> <5494D0D7.2010606@elyograg.org> <1419170477.6868.1.camel@oracle.com> <5499AA73.9090003@elyograg.org> <1419357347.3128.1.camel@oracle.com> <5499D8F9.9020607@elyograg.org> <1419934339.3250.1.camel@oracle.com> <54A2ECFC.1010401@elyograg.org> <54A321F5.7060209@oracle.com> <54A34373.9000509@elyograg.org> <1420035545.3277.2.camel@oracle.com> <54A59948.8050602@elyograg.org> Message-ID: <1420635125.3312.14.camel@oracle.com> Hi, On Thu, 2015-01-01 at 12:00 -0700, Shawn Heisey wrote: > On 12/31/2014 7:19 AM, Thomas Schatzl wrote: > > -XX:G1HeapRegionSize=4M should be sufficient: all the objects I have > > seen are slightly smaller than 2M, which corresponds to Shawn's > > statement about having around 16.3M bits in length. > > > > With -Xms4G -Xmx6G the default region size is 2M, not 4M. Using > > -XX:G1HeapRegionSize=8M seems overkill. > > Sounds reasonable. Smaller regions probably means better performance > but I think I'll leave my region size at 8MB. My indexes are always > growing, and will soon be large enough that every filter will be larger > than 50 percent of a 4MB region. I would have to increase it to 8MB > anyway in the near future. Okay. > > > > This is most likely due to the spiky allocation behavior of the > > application: i.e. long stretches of almost every object dying, and then > > short bursts. Since G1 tunes itself to the former, it will simply try to > > use too much eden size for these spikes. > > > > But I recommend first seeing the impact of the increase in region size. > > With the 8M region size, the situation looks MUCH better -- the > multi-second pauses are gone. There are a handful of longer pauses, but > the average is well below the 200ms target. The gcviewer program > doesn't calculate median and percentile (75th, 95th, 99th, etc) values > ... I would be very interested in knowing what those numbers are. > Perhaps I need to grab the source code and fiddle with it. > > https://www.dropbox.com/s/wzn6dhckl9sr05n/gc-idxa1-g1-7u72-8m-region-size.zip?dl=0 > > I can continue to tweak settings as you have suggested, but the graph of > this log looks far better than I dared hope ... any further improvement > is just icing on the cake. > > I have updated my personal Solr wiki page with my current GC settings > and some notes about the G1 region size: > > https://wiki.apache.org/solr/ShawnHeisey#GC_Tuning Good to see that it works for you. One final comment on how to achieve higher throughput (if needed): from the logs I can see that the background marking is running all the time, which may be wasted CPU cycles. I.e. heap occupancy seems to hover roughly around 2.75G (=67% of the usual 4G that are used - G1 never wants to expand the heap due to pause time problems - only once slightly because it cannot get a large enough contiguous amount of space for a large object). By default (and currently) marking starts at 45% of heap occupancy. You may want to try to set this to something higher like 70-75% to not have it eat up CPU all the time. As mentioned, in the log file G1 keeps the specified minimum heap size of 4G (-Xms setting). If you have the memory you may want to give it more, or lower the pause time goal. Lowering the pause time goal to something below 200ms (with the intent of achieving 200ms) may also help achieving better medians and percentiles. Thanks, Thomas From java at elyograg.org Wed Jan 7 18:49:54 2015 From: java at elyograg.org (Shawn Heisey) Date: Wed, 07 Jan 2015 11:49:54 -0700 Subject: G1 with Solr - thread from dev@lucene.apache.org In-Reply-To: <1420635125.3312.14.camel@oracle.com> References: <54906C34.8080408@elyograg.org> <1418831456.3255.22.camel@oracle.com> <5491D659.1090703@elyograg.org> <1418849513.3293.3.camel@oracle.com> <5494D0D7.2010606@elyograg.org> <1419170477.6868.1.camel@oracle.com> <5499AA73.9090003@elyograg.org> <1419357347.3128.1.camel@oracle.com> <5499D8F9.9020607@elyograg.org> <1419934339.3250.1.camel@oracle.com> <54A2ECFC.1010401@elyograg.org> <54A321F5.7060209@oracle.com> <54A34373.9000509@elyograg.org> <1420035545.3277.2.camel@oracle.com> <54A59948.8050602@elyograg.org> <1420635125.3312.14.camel@oracle.com> Message-ID: <54AD7FD2.8000805@elyograg.org> On 1/7/2015 5:52 AM, Thomas Schatzl wrote: > Good to see that it works for you. > > One final comment on how to achieve higher throughput (if needed): from > the logs I can see that the background marking is running all the time, > which may be wasted CPU cycles. > > I.e. heap occupancy seems to hover roughly around 2.75G (=67% of the > usual 4G that are used - G1 never wants to expand the heap due to pause > time problems - only once slightly because it cannot get a large enough > contiguous amount of space for a large object). > > By default (and currently) marking starts at 45% of heap occupancy. You > may want to try to set this to something higher like 70-75% to not have > it eat up CPU all the time. Looks like I need to change -XX:InitiatingHeapOccupancyPercent=nn for that. Thanks for the info. CPU utilization is not a problem for these machines, but anything that drops resource usage will make performance better, even if limits are not being pushed. > As mentioned, in the log file G1 keeps the specified minimum heap size > of 4G (-Xms setting). If you have the memory you may want to give it > more, or lower the pause time goal. The machine has 64GB of RAM, so there's plenty of memory ... but I want to keep as much memory as possible available for OS disk caching. I might increase Xms so it's the same as Xmx (6GB). > Lowering the pause time goal to something below 200ms (with the intent > of achieving 200ms) may also help achieving better medians and > percentiles. I can certainly do this. I do not want to be greedy, which is why I did not try for a pause target of 100ms. :) I chose 200ms because the setting I see in many places around the Internet is 100, so 200 seemed more realistic. I didn't know when I chose that number that the default was 200ms. Thanks, Shawn From yiyeguhu at gmail.com Thu Jan 15 18:13:44 2015 From: yiyeguhu at gmail.com (Tao Mao) Date: Thu, 15 Jan 2015 10:13:44 -0800 Subject: G1GC Compatibility with JDK tools Message-ID: Hi all, Does someone know about G1GC compatibility with JDK tools (listed here http://docs.oracle.com/javase/7/docs/technotes/tools/)? Or can provide an informational pointer. I am currently weighing pros and cons of migrating to G1GC for my team? Thanks. Tao -------------- next part -------------- An HTML attachment was scrubbed... URL: From ecki at zusammenkunft.net Thu Jan 15 18:53:45 2015 From: ecki at zusammenkunft.net (Bernd) Date: Thu, 15 Jan 2015 19:53:45 +0100 Subject: G1GC Compatibility with JDK tools In-Reply-To: References: Message-ID: Hello, They are all pretty independent of the GC used. The only tools/interfaces you might want to worry about are jstat counters and JMX. But for both the G1 support is pretty ok (but different metrics). So looking at the tools is not the main cincern when planning for G1. It is more about performance testing gruss Bernd Am 15.01.2015 19:43 schrieb "Tao Mao" : > Hi all, > > Does someone know about G1GC compatibility with JDK tools (listed here > http://docs.oracle.com/javase/7/docs/technotes/tools/)? Or can provide an > informational pointer. > > I am currently weighing pros and cons of migrating to G1GC for my team? > > Thanks. > Tao > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Andreas.Mueller at mgm-tp.com Thu Jan 22 13:58:46 2015 From: Andreas.Mueller at mgm-tp.com (=?iso-8859-1?Q?Andreas_M=FCller?=) Date: Thu, 22 Jan 2015 13:58:46 +0000 Subject: Minor GCs grow longer after Full GC Message-ID: <46FF8393B58AD84D95E444264805D98F01A047FCA6@edata01.mgm-edv.de> Hi all, for some time I have been looking at an application (Tomcat 7, Java 7 on Linux, ParallelGC collector, 4GB of heap, 3 GB of perm heap, 20 virtual CPUs on single KVM guest) which shows strange GC behavior: - immediately after every Full GC run minor GCs are fast (below 100ms) - with growing heap occupation minor GC pauses grow to around 800ms immediately before every Full GC Find a plot of heap occupation (blue line, left scale) and GC pause duration (grey columns, right scale) attached. We use the following GC settings: -Xms4g -Xmx4g -XX:NewSize=800m -XX:MaxNewSize=800m -XX:PermSize=3g -XX:MaxPermSize=3g -XX:TargetSurvivorRatio=50 -XX:SurvivorRatio=8 -XX:ParallelGCThreads=20 I am not sure whether minor GC pauses grow with old gen or perm gen occupation but assume that one of these is the cause. Perm gen occupation also grows over time because there are (re)generated classes and this would eventually trigger a Full GC if old gen were not filled first. That's the reason why we configured a rather large perm space. I can imagine that old gen occupation might cause such a problem if there are many references from old gen to new gen objects. In this case, management of "remembered sets" during new gen collections could require growing effort when the number of old gen objects grows. Is this correct? I have tested G1 for this application and results were worse. As I understand it, management of "remembered sets" is more of a limiting factor with G1 than with ParallelGC, and therefore this seems to corroborate my assumption. Is this correct? Are there any other ideas about the cause of growing minor GC pauses? Are there any ideas how to investigate further? How to achieve shorter minor GCs without more frequent major GCs? Best regards / Mit freundlichen Gr??en Andreas M?ller mgm technology partners GmbH Frankfurter Ring 105a 80807 M?nchen Tel. +49 (89) 35 86 80-633 Fax +49 (89) 35 86 80-288 E-Mail Andreas.Mueller at mgm-tp.com Innovation Implemented. Sitz der Gesellschaft: M?nchen Gesch?ftsf?hrer: Hamarz Mehmanesh Handelsregister: AG M?nchen HRB 105068 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: GCPlot.jpg Type: image/jpeg Size: 36306 bytes Desc: GCPlot.jpg URL: From thomas.schatzl at oracle.com Thu Jan 22 14:08:10 2015 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Thu, 22 Jan 2015 15:08:10 +0100 Subject: Minor GCs grow longer after Full GC In-Reply-To: <46FF8393B58AD84D95E444264805D98F01A047FCA6@edata01.mgm-edv.de> References: <46FF8393B58AD84D95E444264805D98F01A047FCA6@edata01.mgm-edv.de> Message-ID: <1421935690.3426.19.camel@oracle.com> Hi Andreas, On Thu, 2015-01-22 at 13:58 +0000, Andreas M?ller wrote: > Hi all, > > for some time I have been looking at an application (Tomcat 7, Java 7 > on Linux, ParallelGC collector, 4GB of heap, 3 GB of perm heap, 20 > virtual CPUs on single KVM guest) which shows strange GC behavior: > - immediately after every Full GC run minor GCs are fast (below > 100ms) > - with growing heap occupation minor GC pauses grow to around > 800ms immediately before every Full GC > Find a plot of heap occupation (blue line, left scale) and GC pause > duration (grey columns, right scale) attached. > We use the following GC settings: > -Xms4g ?Xmx4g ?XX:NewSize=800m ?XX:MaxNewSize=800m ?XX:PermSize=3g ? > XX:MaxPermSize=3g ?XX:TargetSurvivorRatio=50 ?XX:SurvivorRatio=8 ? > XX:ParallelGCThreads=20 > > I am not sure whether minor GC pauses grow with old gen or perm gen > occupation but assume that one of these is the cause. > > Perm gen occupation also grows over time because there are > (re)generated classes and this would eventually trigger a Full GC if > old gen were not filled first. Which exact version of Java were you using? > That?s the reason why we configured a rather large perm space. > > I can imagine that old gen occupation might cause such a problem if > there are many references from old gen to new gen objects. Yes. Or other sources of roots like code cache (or string table) or others. > In this case, management of ?remembered sets? during new gen > collections could require growing effort when the number of old gen > objects grows. Is this correct? Yes, but it seems that in your case the perm gen (or one of those additional roots) seems to be the problem. > I have tested G1 for this application and results were worse. As I > understand it, management of ?remembered sets? is more of a limiting > factor with G1 than with ParallelGC, and > therefore this seems to corroborate my assumption. Is this correct? In 7u60 there have been some improvements to remembered set management from code cache. This often decreases the impact of this behavior, but does not remove it. Young GC times will still grow over time. Can you provide a log with -XX:+PrintGCDetails (with G1)? If you notice ever increasing "Ext Root Scan times", this is the exact problem. With 8u20, G1 will not need full gc for clearing the "perm gen" any more. It is piggy-backed on concurrent marking. > Are there any other ideas about the cause of growing minor GC pauses? > Are there any ideas how to investigate further? > How to achieve shorter minor GCs without more frequent major GCs? Use G1 with 8u20+. As mentioned, it can do the necessary cleanup at the end of marking (which impacts that pause time, but it will be significantly faster than a Full GC). Thanks, Thomas From thomas.schatzl at oracle.com Thu Jan 22 14:32:47 2015 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Thu, 22 Jan 2015 15:32:47 +0100 Subject: Minor GCs grow longer after Full GC In-Reply-To: <1421935690.3426.19.camel@oracle.com> References: <46FF8393B58AD84D95E444264805D98F01A047FCA6@edata01.mgm-edv.de> <1421935690.3426.19.camel@oracle.com> Message-ID: <1421937167.3426.21.camel@oracle.com> Hi again, On Thu, 2015-01-22 at 15:08 +0100, Thomas Schatzl wrote: > Hi Andreas, > > On Thu, 2015-01-22 at 13:58 +0000, Andreas M?ller wrote: > > Hi all, > > > > I have tested G1 for this application and results were worse. As I > > understand it, management of ?remembered sets? is more of a limiting > > factor with G1 than with ParallelGC, and > > therefore this seems to corroborate my assumption. Is this correct? > > In 7u60 there have been some improvements to remembered set management > from code cache. This often decreases the impact of this behavior, but > does not remove it. Young GC times will still grow over time. > > Can you provide a log with -XX:+PrintGCDetails (with G1)? If you notice > ever increasing "Ext Root Scan times", this is the exact problem. > > With 8u20, G1 will not need full gc for clearing the "perm gen" any > more. It is piggy-backed on concurrent marking. some correction: G1 class unloading during concurrent marking has only been introduced with 8u40, not 8u20. Sorry for the misinformation. Thomas From jon.masamitsu at oracle.com Thu Jan 22 18:44:28 2015 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Thu, 22 Jan 2015 10:44:28 -0800 Subject: Minor GCs grow longer after Full GC In-Reply-To: <1421935690.3426.19.camel@oracle.com> References: <46FF8393B58AD84D95E444264805D98F01A047FCA6@edata01.mgm-edv.de> <1421935690.3426.19.camel@oracle.com> Message-ID: <54C1450C.8020703@oracle.com> Andreas, What Thomas said. And try turning on PrintReferenceGC to see if you are spending much time doing Reference processing. Jon On 01/22/2015 06:08 AM, Thomas Schatzl wrote: > Hi Andreas, > > On Thu, 2015-01-22 at 13:58 +0000, Andreas M?ller wrote: >> Hi all, >> >> for some time I have been looking at an application (Tomcat 7, Java 7 >> on Linux, ParallelGC collector, 4GB of heap, 3 GB of perm heap, 20 >> virtual CPUs on single KVM guest) which shows strange GC behavior: >> - immediately after every Full GC run minor GCs are fast (below >> 100ms) >> - with growing heap occupation minor GC pauses grow to around >> 800ms immediately before every Full GC >> Find a plot of heap occupation (blue line, left scale) and GC pause >> duration (grey columns, right scale) attached. >> We use the following GC settings: >> -Xms4g ?Xmx4g ?XX:NewSize=800m ?XX:MaxNewSize=800m ?XX:PermSize=3g ? >> XX:MaxPermSize=3g ?XX:TargetSurvivorRatio=50 ?XX:SurvivorRatio=8 ? >> XX:ParallelGCThreads=20 >> >> I am not sure whether minor GC pauses grow with old gen or perm gen >> occupation but assume that one of these is the cause. >> >> Perm gen occupation also grows over time because there are >> (re)generated classes and this would eventually trigger a Full GC if >> old gen were not filled first. > Which exact version of Java were you using? > >> That?s the reason why we configured a rather large perm space. >> >> I can imagine that old gen occupation might cause such a problem if >> there are many references from old gen to new gen objects. > Yes. Or other sources of roots like code cache (or string table) or > others. > >> In this case, management of ?remembered sets? during new gen >> collections could require growing effort when the number of old gen >> objects grows. Is this correct? > Yes, but it seems that in your case the perm gen (or one of those > additional roots) seems to be the problem. > >> I have tested G1 for this application and results were worse. As I >> understand it, management of ?remembered sets? is more of a limiting >> factor with G1 than with ParallelGC, and >> therefore this seems to corroborate my assumption. Is this correct? > In 7u60 there have been some improvements to remembered set management > from code cache. This often decreases the impact of this behavior, but > does not remove it. Young GC times will still grow over time. > > Can you provide a log with -XX:+PrintGCDetails (with G1)? If you notice > ever increasing "Ext Root Scan times", this is the exact problem. > > With 8u20, G1 will not need full gc for clearing the "perm gen" any > more. It is piggy-backed on concurrent marking. > >> Are there any other ideas about the cause of growing minor GC pauses? >> Are there any ideas how to investigate further? >> How to achieve shorter minor GCs without more frequent major GCs? > Use G1 with 8u20+. As mentioned, it can do the necessary cleanup at the > end of marking (which impacts that pause time, but it will be > significantly faster than a Full GC). > > Thanks, > Thomas > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From Andreas.Mueller at mgm-tp.com Thu Jan 22 19:52:39 2015 From: Andreas.Mueller at mgm-tp.com (=?utf-8?B?QW5kcmVhcyBNw7xsbGVy?=) Date: Thu, 22 Jan 2015 19:52:39 +0000 Subject: AW: Minor GCs grow longer after Full GC In-Reply-To: <1421935690.3426.19.camel@oracle.com> References: <46FF8393B58AD84D95E444264805D98F01A047FCA6@edata01.mgm-edv.de> <1421935690.3426.19.camel@oracle.com> Message-ID: <46FF8393B58AD84D95E444264805D98F01A047FCF4@edata01.mgm-edv.de> Hi Thomas, thanks for your helpful comments. >Which exact version of Java were you using? So far, we have been using Java 7_65 but we are going to switch to Java 8 these days. I will press to use 8_40+ following you recommendation. > Yes, but it seems that in your case the perm gen (or one of those additional roots) seems to be the problem. Good hint, but from what do you conclude that? It's a good idea because we have recently decreased old gen and increased perm gen. We did that because a full perm gen triggered major GCs very frequently and with CMS this sometimes resulted in very long running (single-threaded) Full GCs. In the end we ditched CMS altogether to avoid that risk. As you can see from the plot, ParallelGC delivers reasonable Full GC runs of 3-5 seconds about once an hour. That's fine and more robust than CMS, which seems to be at odds with perm gen triggered major GCs: 95% of them were concurrent and below 1s but some took as much as 50s! That's why I also tested G1 as an alternative. >In 7u60 there have been some improvements to remembered set management from code cache. This often decreases the impact of this behavior, but does not remove it. Young GC times will still grow over time. Code cache could be another source of trouble. Ours is huge (512 MB) for the same reason perm gen is huge. Thanks for pointing at that, I did not take into account that code cache contains references to new gen objects, too. Best regards Andreas -----Urspr?ngliche Nachricht----- Von: Thomas Schatzl [mailto:thomas.schatzl at oracle.com] Gesendet: Donnerstag, 22. Januar 2015 15:08 An: Andreas M?ller Cc: 'hotspot-gc-use at openjdk.java.net' (hotspot-gc-use at openjdk.java.net) Betreff: Re: Minor GCs grow longer after Full GC Hi Andreas, On Thu, 2015-01-22 at 13:58 +0000, Andreas M?ller wrote: > Hi all, > > for some time I have been looking at an application (Tomcat 7, Java 7 > on Linux, ParallelGC collector, 4GB of heap, 3 GB of perm heap, 20 > virtual CPUs on single KVM guest) which shows strange GC behavior: > - immediately after every Full GC run minor GCs are fast (below > 100ms) > - with growing heap occupation minor GC pauses grow to around > 800ms immediately before every Full GC Find a plot of heap occupation > (blue line, left scale) and GC pause duration (grey columns, right > scale) attached. > We use the following GC settings: > -Xms4g ?Xmx4g ?XX:NewSize=800m ?XX:MaxNewSize=800m ?XX:PermSize=3g ? > XX:MaxPermSize=3g ?XX:TargetSurvivorRatio=50 ?XX:SurvivorRatio=8 ? > XX:ParallelGCThreads=20 > > I am not sure whether minor GC pauses grow with old gen or perm gen > occupation but assume that one of these is the cause. > > Perm gen occupation also grows over time because there are > (re)generated classes and this would eventually trigger a Full GC if > old gen were not filled first. Which exact version of Java were you using? > That?s the reason why we configured a rather large perm space. > > I can imagine that old gen occupation might cause such a problem if > there are many references from old gen to new gen objects. Yes. Or other sources of roots like code cache (or string table) or others. > In this case, management of ?remembered sets? during new gen > collections could require growing effort when the number of old gen > objects grows. Is this correct? Yes, but it seems that in your case the perm gen (or one of those additional roots) seems to be the problem. > I have tested G1 for this application and results were worse. As I > understand it, management of ?remembered sets? is more of a limiting > factor with G1 than with ParallelGC, and therefore this seems to > corroborate my assumption. Is this correct? In 7u60 there have been some improvements to remembered set management from code cache. This often decreases the impact of this behavior, but does not remove it. Young GC times will still grow over time. Can you provide a log with -XX:+PrintGCDetails (with G1)? If you notice ever increasing "Ext Root Scan times", this is the exact problem. With 8u20, G1 will not need full gc for clearing the "perm gen" any more. It is piggy-backed on concurrent marking. > Are there any other ideas about the cause of growing minor GC pauses? > Are there any ideas how to investigate further? > How to achieve shorter minor GCs without more frequent major GCs? Use G1 with 8u20+. As mentioned, it can do the necessary cleanup at the end of marking (which impacts that pause time, but it will be significantly faster than a Full GC). Thanks, Thomas From thomas.schatzl at oracle.com Wed Jan 28 17:36:06 2015 From: thomas.schatzl at oracle.com (Thomas Schatzl) Date: Wed, 28 Jan 2015 18:36:06 +0100 Subject: AW: Minor GCs grow longer after Full GC In-Reply-To: <46FF8393B58AD84D95E444264805D98F01A047FCF4@edata01.mgm-edv.de> References: <46FF8393B58AD84D95E444264805D98F01A047FCA6@edata01.mgm-edv.de> <1421935690.3426.19.camel@oracle.com> <46FF8393B58AD84D95E444264805D98F01A047FCF4@edata01.mgm-edv.de> Message-ID: <1422466566.3211.4.camel@oracle.com> Hi Andreas, sorry for the somewhat late reply... On Thu, 2015-01-22 at 19:52 +0000, Andreas M?ller wrote: > Hi Thomas, > > thanks for your helpful comments. > > >Which exact version of Java were you using? > So far, we have been using Java 7_65 but we are going to switch to Java 8 these days. I will press to use > 8_40+ following you recommendation. Please report back any experience. This helps us a lot to further improve G1. Preferably, if there are issues, with a log :) > > Yes, but it seems that in your case the perm gen (or one of those additional roots) seems to be the problem. > Good hint, but from what do you conclude that? From that your application is a tomcat application server, the huge perm gen in conjunction with your comments about perm gen, and some guessing :) We have worked together with people on another application server where the same behavior has been seen, and we found that there has been a similar problem with code roots and class unloading. Thanks, Thomas From joyxiong at yahoo.com Thu Jan 29 18:27:19 2015 From: joyxiong at yahoo.com (Joy Xiong) Date: Thu, 29 Jan 2015 18:27:19 +0000 (UTC) Subject: longer ParNew collection in JDK8? Message-ID: <1905701844.2549870.1422556039324.JavaMail.yahoo@mail.yahoo.com> Hi, We recently move our services from JDK6 to JDK8, but observe longer ParNew GC pause in JDK8. For example, the pause time increase from 27ms to 43ms in one service. The service has the JVM parameters as below:-Xms32684m -Xmx32684m -XX:NewSize=2048m -XX:MaxNewSize=2048m -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=70 -XX:SurvivorRatio=2 -XX:+AlwaysPreTouch -XX:+UseCompressedOops -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:logs/gc.log -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime Is this longer ParNew pause expected? Any suggestions to mitigate the longer pause? thanks,-Joy -------------- next part -------------- An HTML attachment was scrubbed... URL: From yu.zhang at oracle.com Thu Jan 29 18:55:30 2015 From: yu.zhang at oracle.com (Yu Zhang) Date: Thu, 29 Jan 2015 10:55:30 -0800 Subject: longer ParNew collection in JDK8? In-Reply-To: <1905701844.2549870.1422556039324.JavaMail.yahoo@mail.yahoo.com> References: <1905701844.2549870.1422556039324.JavaMail.yahoo@mail.yahoo.com> Message-ID: <54CA8222.3020503@oracle.com> Joy, Have you tried g1 in jdk8? Thanks, Jenny On 1/29/2015 10:27 AM, Joy Xiong wrote: > > Hi, > > We recently move our services from JDK6 to JDK8, but observe longer > ParNew GC pause in JDK8. For example, the pause time increase from > 27ms to 43ms in one service. The service has the JVM parameters as below: > -Xms32684m -Xmx32684m -XX:NewSize=2048m -XX:MaxNewSize=2048m > -XX:+UseConcMarkSweepGC -XX:+UseParNewGC > -XX:CMSInitiatingOccupancyFraction=70 -XX:SurvivorRatio=2 > -XX:+AlwaysPreTouch -XX:+UseCompressedOops > -XX:+PrintTenuringDistribution -XX:+PrintGCDetails > -XX:+PrintGCDateStamps -Xloggc:logs/gc.log > -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime > > Is this longer ParNew pause expected? Any suggestions to mitigate the > longer pause? > > thanks, > -Joy > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: