From anthony.warden at nomura.com Wed Feb 3 14:43:52 2010 From: anthony.warden at nomura.com (anthony.warden at nomura.com) Date: Wed, 3 Feb 2010 22:43:52 -0000 Subject: Fully disable GC - no pauses - run with massive heap until outofmemory Message-ID: <2E97E78D7F99D64DA5108FE2E9F0E828070EC6F1@LONEV3201.EUROPE.NOM> How best to do this please? Command line options / hack the gc thread etc. --------------------------------- This e-mail (including any attachments) is confidential, may contain proprietary or privileged information and is intended for the named recipient(s) only. Unintended recipients are prohibited from taking action on the basis of information in this e-mail and must delete all copies. Nomura will not accept responsibility or liability for the accuracy or completeness of, or the presence of any virus or disabling code in, this e-mail. If verification is sought please request a hard copy. Any reference to the terms of executed transactions should be treated as preliminary only and subject to formal written confirmation by Nomura. Nomura reserves the right to monitor e-mail communications through its networks (in accordance with applicable laws). No confidentiality or privilege is waived or lost by Nomura by any mistransmission of this e-mail. Any reference to "Nomura" is a reference to any entity in the Nomura Holdings, Inc. group. Please read our Electronic Communications Legal Notice which forms part of this e-mail: http://www.Nomura.com/email_disclaimer.htm -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20100203/bebc1daa/attachment.html From Y.S.Ramakrishna at Sun.COM Wed Feb 3 15:58:26 2010 From: Y.S.Ramakrishna at Sun.COM (Y. Srinivas Ramakrishna) Date: Wed, 03 Feb 2010 15:58:26 -0800 Subject: Fully disable GC - no pauses - run with massive heap until outofmemory In-Reply-To: <2E97E78D7F99D64DA5108FE2E9F0E828070EC6F1@LONEV3201.EUROPE.NOM> References: <2E97E78D7F99D64DA5108FE2E9F0E828070EC6F1@LONEV3201.EUROPE.NOM> Message-ID: <4B6A0DA2.7090202@Sun.COM> Hello Anthony -- On 02/03/10 14:43, anthony.warden at nomura.com wrote: > How best to do this please? > > Command line options / hack the gc thread etc. Shape the heap so as to give all (or most) of it to the young gen, use no survivor spaces. For example, say you want a 32 G heap. Let's give 4 M to the old gen may be. So you could do:- -Xmn32g -Xmx(32*1024+4)m -Xms(32*1204+4)m -XX:SurvivorRatio=1g (of course you use the computed values of the expressions in parentheses above.) Unfortunately, this will not give you an OOM when the heap gets full. It'll trigger a scavenge. If you want to give up the ghost at the first scavenge you can (if you are running on Solaris for example) write a Dtrace script to detect the invocation of a GC and kill the JVM process. Don't know if that's the behaviour you seek here. -- ramki > > > > --------------------------------- > > > > This e-mail (including any attachments) is confidential, may contain > proprietary or privileged information and is intended for the named > recipient(s) only. Unintended recipients are prohibited from taking action > on the basis of information in this e-mail and must delete all copies. > Nomura will not accept responsibility or liability for the accuracy or > completeness of, or the presence of any virus or disabling code in, this > e-mail. If verification is sought please request a hard copy. Any reference > to the terms of executed transactions should be treated as preliminary only > and subject to formal written confirmation by Nomura. Nomura reserves the > right to monitor e-mail communications through its networks (in accordance > with applicable laws). No confidentiality or privilege is waived or lost by > Nomura by any mistransmission of this e-mail. Any reference to "Nomura" is > a reference to any entity in the Nomura Holdings, Inc. group. Please read > our Electronic Communications Legal Notice which forms part of this e-mail: > http://www.Nomura.com/email_disclaimer.htm > > > ------------------------------------------------------------------------ > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From Y.S.Ramakrishna at Sun.COM Wed Feb 3 16:03:52 2010 From: Y.S.Ramakrishna at Sun.COM (Y. Srinivas Ramakrishna) Date: Wed, 03 Feb 2010 16:03:52 -0800 Subject: Fully disable GC - no pauses - run with massive heap until outofmemory In-Reply-To: <4B6A0DA2.7090202@Sun.COM> References: <2E97E78D7F99D64DA5108FE2E9F0E828070EC6F1@LONEV3201.EUROPE.NOM> <4B6A0DA2.7090202@Sun.COM> Message-ID: <4B6A0EE8.2080006@Sun.COM> On 02/03/10 15:58, Y. Srinivas Ramakrishna wrote: > Hello Anthony -- > > On 02/03/10 14:43, anthony.warden at nomura.com wrote: >> How best to do this please? >> >> Command line options / hack the gc thread etc. > > Shape the heap so as to give all (or most) of it to the young gen, > use no survivor spaces. For example, say you want a 32 G heap. > Let's give 4 M to the old gen may be. So you could do:- > > -Xmn32g -Xmx(32*1024+4)m -Xms(32*1204+4)m -XX:SurvivorRatio=1g i meant 1024 instead of 1204 above. Sorry for the typo. -- ramki > > (of course you use the computed values of the expressions in parentheses above.) > > Unfortunately, this will not give you an OOM when the heap gets > full. It'll trigger a scavenge. If you want to give up the ghost at > the first scavenge you can (if you are running on Solaris for example) > write a Dtrace script to detect the invocation of a GC and kill > the JVM process. Don't know if that's the behaviour you seek here. > > -- ramki > >> >> >> --------------------------------- >> >> >> >> This e-mail (including any attachments) is confidential, may contain >> proprietary or privileged information and is intended for the named >> recipient(s) only. Unintended recipients are prohibited from taking action >> on the basis of information in this e-mail and must delete all copies. >> Nomura will not accept responsibility or liability for the accuracy or >> completeness of, or the presence of any virus or disabling code in, this >> e-mail. If verification is sought please request a hard copy. Any reference >> to the terms of executed transactions should be treated as preliminary only >> and subject to formal written confirmation by Nomura. Nomura reserves the >> right to monitor e-mail communications through its networks (in accordance >> with applicable laws). No confidentiality or privilege is waived or lost by >> Nomura by any mistransmission of this e-mail. Any reference to "Nomura" is >> a reference to any entity in the Nomura Holdings, Inc. group. Please read >> our Electronic Communications Legal Notice which forms part of this e-mail: >> http://www.Nomura.com/email_disclaimer.htm >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From tony.printezis at sun.com Wed Feb 3 16:07:11 2010 From: tony.printezis at sun.com (Tony Printezis) Date: Wed, 03 Feb 2010 19:07:11 -0500 Subject: Fully disable GC - no pauses - run with massive heap until outofmemory In-Reply-To: <4B6A0DA2.7090202@Sun.COM> References: <2E97E78D7F99D64DA5108FE2E9F0E828070EC6F1@LONEV3201.EUROPE.NOM> <4B6A0DA2.7090202@Sun.COM> Message-ID: <4B6A0FAF.50500@sun.com> Hi Anthony, I hope you are well! An alternative is to use an MBean and monitor the occupancy of the eden. When that goes over a certain threshold, you can somehow kill the JVM. In fact, you can be a bit more graceful this way, since you can notice early-ish that the JVM will run out of memory a bit into the future and shutdown all processing gracefully (i.e., let the current Tx complete), while redirecting new requests to a new JVM. Are you going to have any long-lived data in the JVM? Maybe, size the old generation large enough to accommodate it and after initialization do a System.gc() to move all that into the old generation. That way, you'll start processing with a totally empty eden. GC group: should we really be encouraging this? :-) Tony, HS GC Group Y. Srinivas Ramakrishna wrote: > Hello Anthony -- > > On 02/03/10 14:43, anthony.warden at nomura.com wrote: > >> How best to do this please? >> >> Command line options / hack the gc thread etc. >> > > Shape the heap so as to give all (or most) of it to the young gen, > use no survivor spaces. For example, say you want a 32 G heap. > Let's give 4 M to the old gen may be. So you could do:- > > -Xmn32g -Xmx(32*1024+4)m -Xms(32*1204+4)m -XX:SurvivorRatio=1g > > (of course you use the computed values of the expressions in parentheses above.) > > Unfortunately, this will not give you an OOM when the heap gets > full. It'll trigger a scavenge. If you want to give up the ghost at > the first scavenge you can (if you are running on Solaris for example) > write a Dtrace script to detect the invocation of a GC and kill > the JVM process. Don't know if that's the behaviour you seek here. > > -- ramki > > >> >> --------------------------------- >> >> >> >> This e-mail (including any attachments) is confidential, may contain >> proprietary or privileged information and is intended for the named >> recipient(s) only. Unintended recipients are prohibited from taking action >> on the basis of information in this e-mail and must delete all copies. >> Nomura will not accept responsibility or liability for the accuracy or >> completeness of, or the presence of any virus or disabling code in, this >> e-mail. If verification is sought please request a hard copy. Any reference >> to the terms of executed transactions should be treated as preliminary only >> and subject to formal written confirmation by Nomura. Nomura reserves the >> right to monitor e-mail communications through its networks (in accordance >> with applicable laws). No confidentiality or privilege is waived or lost by >> Nomura by any mistransmission of this e-mail. Any reference to "Nomura" is >> a reference to any entity in the Nomura Holdings, Inc. group. Please read >> our Electronic Communications Legal Notice which forms part of this e-mail: >> http://www.Nomura.com/email_disclaimer.htm >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > From Y.S.Ramakrishna at Sun.COM Wed Feb 3 16:27:52 2010 From: Y.S.Ramakrishna at Sun.COM (Y. Srinivas Ramakrishna) Date: Wed, 03 Feb 2010 16:27:52 -0800 Subject: Fully disable GC - no pauses - run with massive heap until outofmemory In-Reply-To: <4B6A0FAF.50500@sun.com> References: <2E97E78D7F99D64DA5108FE2E9F0E828070EC6F1@LONEV3201.EUROPE.NOM> <4B6A0DA2.7090202@Sun.COM> <4B6A0FAF.50500@sun.com> Message-ID: <4B6A1488.7070801@Sun.COM> On 02/03/10 16:07, Tony Printezis wrote: > Hi Anthony, > > I hope you are well! > > An alternative is to use an MBean and monitor the occupancy of the eden. > When that goes over a certain threshold, you can somehow kill the JVM. Is there a threshold monitor that will look at generation occupancy _before_ a GC which is what you'd want here, right? I was always under the impression these detected occupancy after a GC and only then would sound the alarms; but perhaps there are options that specify precise behaviour in this regard. Mandy, cc'd, would know for sure. (Don't know where the spec is.) -- ramki > In fact, you can be a bit more graceful this way, since you can notice > early-ish that the JVM will run out of memory a bit into the future and > shutdown all processing gracefully (i.e., let the current Tx complete), > while redirecting new requests to a new JVM. > > Are you going to have any long-lived data in the JVM? Maybe, size the > old generation large enough to accommodate it and after initialization > do a System.gc() to move all that into the old generation. That way, > you'll start processing with a totally empty eden. > > GC group: should we really be encouraging this? :-) > > Tony, HS GC Group > > Y. Srinivas Ramakrishna wrote: >> Hello Anthony -- >> >> On 02/03/10 14:43, anthony.warden at nomura.com wrote: >> >>> How best to do this please? >>> >>> Command line options / hack the gc thread etc. >>> >> Shape the heap so as to give all (or most) of it to the young gen, >> use no survivor spaces. For example, say you want a 32 G heap. >> Let's give 4 M to the old gen may be. So you could do:- >> >> -Xmn32g -Xmx(32*1024+4)m -Xms(32*1204+4)m -XX:SurvivorRatio=1g >> >> (of course you use the computed values of the expressions in parentheses above.) >> >> Unfortunately, this will not give you an OOM when the heap gets >> full. It'll trigger a scavenge. If you want to give up the ghost at >> the first scavenge you can (if you are running on Solaris for example) >> write a Dtrace script to detect the invocation of a GC and kill >> the JVM process. Don't know if that's the behaviour you seek here. >> >> -- ramki >> >> >>> --------------------------------- >>> >>> >>> >>> This e-mail (including any attachments) is confidential, may contain >>> proprietary or privileged information and is intended for the named >>> recipient(s) only. Unintended recipients are prohibited from taking action >>> on the basis of information in this e-mail and must delete all copies. >>> Nomura will not accept responsibility or liability for the accuracy or >>> completeness of, or the presence of any virus or disabling code in, this >>> e-mail. If verification is sought please request a hard copy. Any reference >>> to the terms of executed transactions should be treated as preliminary only >>> and subject to formal written confirmation by Nomura. Nomura reserves the >>> right to monitor e-mail communications through its networks (in accordance >>> with applicable laws). No confidentiality or privilege is waived or lost by >>> Nomura by any mistransmission of this e-mail. Any reference to "Nomura" is >>> a reference to any entity in the Nomura Holdings, Inc. group. Please read >>> our Electronic Communications Legal Notice which forms part of this e-mail: >>> http://www.Nomura.com/email_disclaimer.htm >>> >>> >>> ------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> hotspot-gc-use mailing list >>> hotspot-gc-use at openjdk.java.net >>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From anthony.warden at nomura.com Wed Feb 3 16:53:55 2010 From: anthony.warden at nomura.com (anthony.warden at nomura.com) Date: Thu, 4 Feb 2010 00:53:55 -0000 Subject: Fully disable GC - no pauses - run with massive heap untiloutofmemory Message-ID: <2E97E78D7F99D64DA5108FE2E9F0E828070EC6F5@LONEV3201.EUROPE.NOM> Many thanks Tony and Ramki. That's exactly what I need and Tony also guessed the next step :) --------------------------------- ----- Original Message ----- From: Antonios.Printezis at sun.com To: ysr at sun.com Cc: Warden, Anthony (IT/UK); hotspot-gc-use at openjdk.java.net Sent: Thu Feb 04 00:07:11 2010 Subject: Re: Fully disable GC - no pauses - run with massive heap untiloutofmemory Hi Anthony, I hope you are well! An alternative is to use an MBean and monitor the occupancy of the eden. When that goes over a certain threshold, you can somehow kill the JVM. In fact, you can be a bit more graceful this way, since you can notice early-ish that the JVM will run out of memory a bit into the future and shutdown all processing gracefully (i.e., let the current Tx complete), while redirecting new requests to a new JVM. Are you going to have any long-lived data in the JVM? Maybe, size the old generation large enough to accommodate it and after initialization do a System.gc() to move all that into the old generation. That way, you'll start processing with a totally empty eden. GC group: should we really be encouraging this? :-) Tony, HS GC Group Y. Srinivas Ramakrishna wrote: > Hello Anthony -- > > On 02/03/10 14:43, anthony.warden at nomura.com wrote: > >> How best to do this please? >> >> Command line options / hack the gc thread etc. >> > > Shape the heap so as to give all (or most) of it to the young gen, > use no survivor spaces. For example, say you want a 32 G heap. > Let's give 4 M to the old gen may be. So you could do:- > > -Xmn32g -Xmx(32*1024+4)m -Xms(32*1204+4)m -XX:SurvivorRatio=1g > > (of course you use the computed values of the expressions in parentheses above.) > > Unfortunately, this will not give you an OOM when the heap gets > full. It'll trigger a scavenge. If you want to give up the ghost at > the first scavenge you can (if you are running on Solaris for example) > write a Dtrace script to detect the invocation of a GC and kill > the JVM process. Don't know if that's the behaviour you seek here. > > -- ramki > > >> >> --------------------------------- >> >> >> >> This e-mail (including any attachments) is confidential, may contain >> proprietary or privileged information and is intended for the named >> recipient(s) only. Unintended recipients are prohibited from taking action >> on the basis of information in this e-mail and must delete all copies. >> Nomura will not accept responsibility or liability for the accuracy or >> completeness of, or the presence of any virus or disabling code in, this >> e-mail. If verification is sought please request a hard copy. Any reference >> to the terms of executed transactions should be treated as preliminary only >> and subject to formal written confirmation by Nomura. Nomura reserves the >> right to monitor e-mail communications through its networks (in accordance >> with applicable laws). No confidentiality or privilege is waived or lost by >> Nomura by any mistransmission of this e-mail. Any reference to "Nomura" is >> a reference to any entity in the Nomura Holdings, Inc. group. Please read >> our Electronic Communications Legal Notice which forms part of this e-mail: >> http://www.Nomura.com/email_disclaimer.htm >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > This e-mail (including any attachments) is confidential, may contain proprietary or privileged information and is intended for the named recipient(s) only. Unintended recipients are prohibited from taking action on the basis of information in this e-mail and must delete all copies. Nomura will not accept responsibility or liability for the accuracy or completeness of, or the presence of any virus or disabling code in, this e-mail. If verification is sought please request a hard copy. Any reference to the terms of executed transactions should be treated as preliminary only and subject to formal written confirmation by Nomura. Nomura reserves the right to monitor e-mail communications through its networks (in accordance with applicable laws). No confidentiality or privilege is waived or lost by Nomura by any mistransmission of this e-mail. Any reference to "Nomura" is a reference to any entity in the Nomura Holdings, Inc. group. Please read our Electronic Communications Legal Notice which forms part of this e-mail: http://www.Nomura.com/email_disclaimer.htm -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20100204/271f8793/attachment-0001.html From tony.printezis at sun.com Wed Feb 3 22:57:05 2010 From: tony.printezis at sun.com (Tony Printezis) Date: Thu, 04 Feb 2010 01:57:05 -0500 Subject: Fully disable GC - no pauses - run with massive heap until outofmemory In-Reply-To: <4B6A1488.7070801@Sun.COM> References: <2E97E78D7F99D64DA5108FE2E9F0E828070EC6F1@LONEV3201.EUROPE.NOM> <4B6A0DA2.7090202@Sun.COM> <4B6A0FAF.50500@sun.com> <4B6A1488.7070801@Sun.COM> Message-ID: <4B6A6FC1.4000702@sun.com> Ramki, Yes, you can set thresholds and get notifications, but I think you're right; said thresholds are only checked before / after a GC. Tony Y. Srinivas Ramakrishna wrote: > On 02/03/10 16:07, Tony Printezis wrote: > >> Hi Anthony, >> >> I hope you are well! >> >> An alternative is to use an MBean and monitor the occupancy of the eden. >> When that goes over a certain threshold, you can somehow kill the JVM. >> > > Is there a threshold monitor that will look at generation occupancy > _before_ a GC which is what you'd want here, right? I was always > under the impression these detected occupancy after a GC and > only then would sound the alarms; but perhaps there are options > that specify precise behaviour in this regard. > > Mandy, cc'd, would know for sure. (Don't know where the spec is.) > -- ramki > > > >> In fact, you can be a bit more graceful this way, since you can notice >> early-ish that the JVM will run out of memory a bit into the future and >> shutdown all processing gracefully (i.e., let the current Tx complete), >> while redirecting new requests to a new JVM. >> >> Are you going to have any long-lived data in the JVM? Maybe, size the >> old generation large enough to accommodate it and after initialization >> do a System.gc() to move all that into the old generation. That way, >> you'll start processing with a totally empty eden. >> >> GC group: should we really be encouraging this? :-) >> >> Tony, HS GC Group >> >> Y. Srinivas Ramakrishna wrote: >> >>> Hello Anthony -- >>> >>> On 02/03/10 14:43, anthony.warden at nomura.com wrote: >>> >>> >>>> How best to do this please? >>>> >>>> Command line options / hack the gc thread etc. >>>> >>>> >>> Shape the heap so as to give all (or most) of it to the young gen, >>> use no survivor spaces. For example, say you want a 32 G heap. >>> Let's give 4 M to the old gen may be. So you could do:- >>> >>> -Xmn32g -Xmx(32*1024+4)m -Xms(32*1204+4)m -XX:SurvivorRatio=1g >>> >>> (of course you use the computed values of the expressions in parentheses above.) >>> >>> Unfortunately, this will not give you an OOM when the heap gets >>> full. It'll trigger a scavenge. If you want to give up the ghost at >>> the first scavenge you can (if you are running on Solaris for example) >>> write a Dtrace script to detect the invocation of a GC and kill >>> the JVM process. Don't know if that's the behaviour you seek here. >>> >>> -- ramki >>> >>> >>> >>>> --------------------------------- >>>> >>>> >>>> >>>> This e-mail (including any attachments) is confidential, may contain >>>> proprietary or privileged information and is intended for the named >>>> recipient(s) only. Unintended recipients are prohibited from taking action >>>> on the basis of information in this e-mail and must delete all copies. >>>> Nomura will not accept responsibility or liability for the accuracy or >>>> completeness of, or the presence of any virus or disabling code in, this >>>> e-mail. If verification is sought please request a hard copy. Any reference >>>> to the terms of executed transactions should be treated as preliminary only >>>> and subject to formal written confirmation by Nomura. Nomura reserves the >>>> right to monitor e-mail communications through its networks (in accordance >>>> with applicable laws). No confidentiality or privilege is waived or lost by >>>> Nomura by any mistransmission of this e-mail. Any reference to "Nomura" is >>>> a reference to any entity in the Nomura Holdings, Inc. group. Please read >>>> our Electronic Communications Legal Notice which forms part of this e-mail: >>>> http://www.Nomura.com/email_disclaimer.htm >>>> >>>> >>>> ------------------------------------------------------------------------ >>>> >>>> _______________________________________________ >>>> hotspot-gc-use mailing list >>>> hotspot-gc-use at openjdk.java.net >>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>>> >>>> >>> _______________________________________________ >>> hotspot-gc-use mailing list >>> hotspot-gc-use at openjdk.java.net >>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>> >>> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > From andrei.pozolotin at gmail.com Wed Feb 17 14:28:33 2010 From: andrei.pozolotin at gmail.com (Andrei Pozolotin) Date: Wed, 17 Feb 2010 16:28:33 -0600 Subject: Incremental CMS: long "[GC [1 CMS-initial-mark:" Message-ID: <4B7C6D91.6000202@gmail.com> *Jon, *hello; I need to convert CMS into incremental CMS to reduce latency, eliminate occasional "concurrent mode failure" - related pauses; and I am running into the following problem: 1) in "normal CMS" - [GC [1 CMS-initial-mark: takes 50 milliseconds 2) in "incremental CMS" - [GC [1 CMS-initial-mark: takes 800 milliseconds both above results are under same no-load conditions, with non-reducible statuc heap object set of 430 MB composed of some 350K complex objects; I can not understand why and how to fix it; I tried a lot of Incremental CMS - related option combinations, still dead end. GC settings, log attached; Thank you for your help, Andrei ++++++++++++++++++++++++++++++++++++++++++++++++ # use server-style JIT compiler wrapper.java.additional.100=-server # do GC logging wrapper.java.additional.110=-verbose:gc wrapper.java.additional.111=-Xloggc:./logs/gc.log wrapper.java.additional.112=-XX:+PrintGCDetails wrapper.java.additional.113=-XX:+PrintGCTimeStamps # Heap Total = PermGen + NewGen + OldGen; wrapper.java.additional.120=-Xms3700m wrapper.java.additional.121=-Xmx3700m # PermGen; wrapper.java.additional.130=-XX:PermSize=50m wrapper.java.additional.131=-XX:MaxPermSize=50m # NewGen; Parallel collector; wrapper.java.additional.140=-XX:NewSize=1200m wrapper.java.additional.141=-XX:MaxNewSize=1200m wrapper.java.additional.142=-XX:-UseAdaptiveSizePolicy wrapper.java.additional.143=-XX:SurvivorRatio=1 wrapper.java.additional.144=-XX:TargetSurvivorRatio=90 wrapper.java.additional.145=-XX:MaxTenuringThreshold=3 wrapper.java.additional.146=-XX:+UseParNewGC wrapper.java.additional.147=-XX:-UseParallelGC wrapper.java.additional.148=-XX:ParallelGCThreads=4 # OldGen; Incremental CMS collector; wrapper.java.additional.170=-XX:-UseParallelOldGC wrapper.java.additional.171=-XX:+UseConcMarkSweepGC wrapper.java.additional.172=-XX:+CMSIncrementalMode wrapper.java.additional.173=-XX:-CMSIncrementalPacing wrapper.java.additional.174=-XX:CMSIncrementalDutyCycle=1 wrapper.java.additional.175=-XX:+CMSParallelRemarkEnabled wrapper.java.additional.176=-XX:+CMSScavengeBeforeRemark wrapper.java.additional.177=-XX:CMSIncrementalOffset=10 wrapper.java.additional.178=-XX:CMSScheduleRemarkEdenPenetration=5 wrapper.java.additional.179=-XX:CMSWaitDuration=60000 wrapper.java.additional.181=-XX:ParallelCMSThreads=1 wrapper.java.additional.182=-XX:PrintCMSStatistics=0 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20100217/78e38c65/attachment-0001.html -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: gc.log Url: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20100217/78e38c65/attachment-0001.ksh From Y.S.Ramakrishna at Sun.COM Wed Feb 17 14:42:34 2010 From: Y.S.Ramakrishna at Sun.COM (Y. Srinivas Ramakrishna) Date: Wed, 17 Feb 2010 14:42:34 -0800 Subject: Incremental CMS: long "[GC [1 CMS-initial-mark:" In-Reply-To: <4B7C6D91.6000202@gmail.com> References: <4B7C6D91.6000202@gmail.com> Message-ID: <4B7C70DA.7070204@Sun.COM> On 02/17/10 14:28, Andrei Pozolotin wrote: > *Jon, *hello; > > I need to convert CMS into incremental CMS to reduce latency, > eliminate occasional "concurrent mode failure" - related pauses; > > and I am running into the following problem: > > 1) in "normal CMS" - [GC [1 CMS-initial-mark: takes 50 milliseconds > > 2) in "incremental CMS" - [GC [1 CMS-initial-mark: takes 800 > milliseconds This is to be expected and is a consequence of how and when the initial mark is done in "normal" vs "incremental". In the former, the initial mark almost immediately follows after a scavenge pause and Eden is empty. In the latter case it normally occurs sometime between two scavenges by which time Eden has objects in it. For various reasons, the young gen is scanned as a source of initial roots to mark from, and for various reasons it is currently the case that this work is done single-threaded. As you will find, the magnitude of this difference typically depends (among other factors) on the size of Eden you specify (being directly proportional to it) and the duty cycle you specify (being inversely proportional to it). There are ways to fix this in the JVM, but no real workaround that i can think of at the command-line level. If this is important, please file a bug on this via your support contacts. thanks! -- ramki > > both above results are under same no-load conditions, > with non-reducible statuc heap object set of 430 MB > composed of some 350K complex objects; > > I can not understand why and how to fix it; > I tried a lot of Incremental CMS - related option combinations, > still dead end. > > GC settings, log attached; > > Thank you for your help, > > Andrei > > ++++++++++++++++++++++++++++++++++++++++++++++++ > > # use server-style JIT compiler > wrapper.java.additional.100=-server > > # do GC logging > wrapper.java.additional.110=-verbose:gc > wrapper.java.additional.111=-Xloggc:./logs/gc.log > wrapper.java.additional.112=-XX:+PrintGCDetails > wrapper.java.additional.113=-XX:+PrintGCTimeStamps > > # Heap Total = PermGen + NewGen + OldGen; > wrapper.java.additional.120=-Xms3700m > wrapper.java.additional.121=-Xmx3700m > > # PermGen; > wrapper.java.additional.130=-XX:PermSize=50m > wrapper.java.additional.131=-XX:MaxPermSize=50m > > # NewGen; Parallel collector; > wrapper.java.additional.140=-XX:NewSize=1200m > wrapper.java.additional.141=-XX:MaxNewSize=1200m > wrapper.java.additional.142=-XX:-UseAdaptiveSizePolicy > wrapper.java.additional.143=-XX:SurvivorRatio=1 > wrapper.java.additional.144=-XX:TargetSurvivorRatio=90 > wrapper.java.additional.145=-XX:MaxTenuringThreshold=3 > wrapper.java.additional.146=-XX:+UseParNewGC > wrapper.java.additional.147=-XX:-UseParallelGC > wrapper.java.additional.148=-XX:ParallelGCThreads=4 > > # OldGen; Incremental CMS collector; > wrapper.java.additional.170=-XX:-UseParallelOldGC > wrapper.java.additional.171=-XX:+UseConcMarkSweepGC > wrapper.java.additional.172=-XX:+CMSIncrementalMode > wrapper.java.additional.173=-XX:-CMSIncrementalPacing > wrapper.java.additional.174=-XX:CMSIncrementalDutyCycle=1 > wrapper.java.additional.175=-XX:+CMSParallelRemarkEnabled > wrapper.java.additional.176=-XX:+CMSScavengeBeforeRemark > wrapper.java.additional.177=-XX:CMSIncrementalOffset=10 > wrapper.java.additional.178=-XX:CMSScheduleRemarkEdenPenetration=5 > wrapper.java.additional.179=-XX:CMSWaitDuration=60000 > wrapper.java.additional.181=-XX:ParallelCMSThreads=1 > wrapper.java.additional.182=-XX:PrintCMSStatistics=0 > > > ------------------------------------------------------------------------ > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From andrei.pozolotin at gmail.com Wed Feb 17 14:57:58 2010 From: andrei.pozolotin at gmail.com (Andrei Pozolotin) Date: Wed, 17 Feb 2010 16:57:58 -0600 Subject: Incremental CMS: long "[GC [1 CMS-initial-mark:" In-Reply-To: <4B7C70DA.7070204@Sun.COM> References: <4B7C6D91.6000202@gmail.com> <4B7C70DA.7070204@Sun.COM> Message-ID: <4B7C7476.6050607@gmail.com> Ramki, hi Thank you for quick response; from reading JDK 6 source I deluded myself that the problem you describe: "... occurs sometime between two scavenges ..." would get fixed by these options: wrapper.java.additional.175=-XX:+CMSParallelRemarkEnabled wrapper.java.additional.176=-XX:+CMSScavengeBeforeRemark wrapper.java.additional.177=-XX:CMSIncrementalOffset=10 wrapper.java.additional.178=-XX:CMSScheduleRemarkEdenPenetration=5 please confirm I was wrong? Cheers, Andrei -------- Original Message -------- Subject: Re: Incremental CMS: long "[GC [1 CMS-initial-mark:" From: Y. Srinivas Ramakrishna To: Andrei Pozolotin Cc: hotspot-gc-use at openjdk.java.net Date: Wed 17 Feb 2010 04:42:34 PM CST > On 02/17/10 14:28, Andrei Pozolotin wrote: >> *Jon, *hello; >> >> I need to convert CMS into incremental CMS to reduce latency, >> eliminate occasional "concurrent mode failure" - related pauses; >> >> and I am running into the following problem: >> >> 1) in "normal CMS" - [GC [1 CMS-initial-mark: takes 50 milliseconds >> >> 2) in "incremental CMS" - [GC [1 CMS-initial-mark: takes 800 >> milliseconds > > This is to be expected and is a consequence of how and when > the initial mark is done in "normal" vs "incremental". > In the former, the initial mark almost immediately follows > after a scavenge pause and Eden is empty. In the latter case > it normally occurs sometime between two scavenges by which time > Eden has objects in it. For various reasons, the young gen > is scanned as a source of initial roots to mark from, > and for various reasons it is currently the case that this > work is done single-threaded. > > As you will find, the magnitude of this difference typically > depends (among other factors) on the size of Eden you specify > (being directly proportional to it) and the duty cycle you > specify (being inversely proportional to it). > > There are ways to fix this in the JVM, but no real workaround that > i can think of at the command-line level. > > If this is important, please file a bug on this via your > support contacts. > > thanks! > -- ramki > >> >> both above results are under same no-load conditions, >> with non-reducible statuc heap object set of 430 MB >> composed of some 350K complex objects; >> >> I can not understand why and how to fix it; >> I tried a lot of Incremental CMS - related option combinations, >> still dead end. >> >> GC settings, log attached; >> >> Thank you for your help, >> >> Andrei >> >> ++++++++++++++++++++++++++++++++++++++++++++++++ >> >> # use server-style JIT compiler >> wrapper.java.additional.100=-server >> >> # do GC logging >> wrapper.java.additional.110=-verbose:gc >> wrapper.java.additional.111=-Xloggc:./logs/gc.log >> wrapper.java.additional.112=-XX:+PrintGCDetails >> wrapper.java.additional.113=-XX:+PrintGCTimeStamps >> >> # Heap Total = PermGen + NewGen + OldGen; >> wrapper.java.additional.120=-Xms3700m >> wrapper.java.additional.121=-Xmx3700m >> >> # PermGen; >> wrapper.java.additional.130=-XX:PermSize=50m >> wrapper.java.additional.131=-XX:MaxPermSize=50m >> >> # NewGen; Parallel collector; >> wrapper.java.additional.140=-XX:NewSize=1200m >> wrapper.java.additional.141=-XX:MaxNewSize=1200m >> wrapper.java.additional.142=-XX:-UseAdaptiveSizePolicy >> wrapper.java.additional.143=-XX:SurvivorRatio=1 >> wrapper.java.additional.144=-XX:TargetSurvivorRatio=90 >> wrapper.java.additional.145=-XX:MaxTenuringThreshold=3 >> wrapper.java.additional.146=-XX:+UseParNewGC >> wrapper.java.additional.147=-XX:-UseParallelGC >> wrapper.java.additional.148=-XX:ParallelGCThreads=4 >> >> # OldGen; Incremental CMS collector; >> wrapper.java.additional.170=-XX:-UseParallelOldGC >> wrapper.java.additional.171=-XX:+UseConcMarkSweepGC >> wrapper.java.additional.172=-XX:+CMSIncrementalMode >> wrapper.java.additional.173=-XX:-CMSIncrementalPacing >> wrapper.java.additional.174=-XX:CMSIncrementalDutyCycle=1 >> wrapper.java.additional.175=-XX:+CMSParallelRemarkEnabled >> wrapper.java.additional.176=-XX:+CMSScavengeBeforeRemark >> wrapper.java.additional.177=-XX:CMSIncrementalOffset=10 >> wrapper.java.additional.178=-XX:CMSScheduleRemarkEdenPenetration=5 >> wrapper.java.additional.179=-XX:CMSWaitDuration=60000 >> wrapper.java.additional.181=-XX:ParallelCMSThreads=1 >> wrapper.java.additional.182=-XX:PrintCMSStatistics=0 >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20100217/2a6e3c0f/attachment.html From Y.S.Ramakrishna at Sun.COM Wed Feb 17 15:31:23 2010 From: Y.S.Ramakrishna at Sun.COM (Y. Srinivas Ramakrishna) Date: Wed, 17 Feb 2010 15:31:23 -0800 Subject: Incremental CMS: long "[GC [1 CMS-initial-mark:" In-Reply-To: <4B7C7476.6050607@gmail.com> References: <4B7C6D91.6000202@gmail.com> <4B7C70DA.7070204@Sun.COM> <4B7C7476.6050607@gmail.com> Message-ID: <4B7C7C4B.3060207@Sun.COM> On 02/17/10 14:57, Andrei Pozolotin wrote: > Ramki, hi > > Thank you for quick response; > > from reading JDK 6 source I deluded myself that the problem you > describe: > > "... occurs sometime between two scavenges ..." > > would get fixed by these options: > > wrapper.java.additional.175=-XX:+CMSParallelRemarkEnabled > wrapper.java.additional.176=-XX:+CMSScavengeBeforeRemark > wrapper.java.additional.177=-XX:CMSIncrementalOffset=10 > wrapper.java.additional.178=-XX:CMSScheduleRemarkEdenPenetration=5 > > please confirm I was wrong? Yes, unfortunately, those are in some sense orthogonal to the way CMS "incrementalizes" its working, here (somewhat relatedly see [1]) For example, CMSScavengeBeforeRemark does indeed schedule a scavenge before a remark pause, but not (unfortunately) before an initial mark pause. Similarly the CMSScheduleRemarkEdenPenetration controls the scheduling of remark pauses, not the initial mark pauses. As for CMSIncrementalOffset, it is just a constant positive delta applied to the ideal starting point for a given duty cycle. So, in fact, any positive value will take you further away from the objective you are trying to achieve here of reducing the occupancy of Eden when initial mark hits. Try G1, that may save you filing an RFE on (i)CMS for this. Honestly though, there is an existing blanket RFE on this: 6412968 CMS: Long initial mark pauses but it has not risen to the top of our list of things to fix. If this is important, please contact your Java support.... -- ramki [1] 6518192 iCMS: smoother, duration-bounded incrementality > Cheers, > > Andrei > > > > -------- Original Message -------- > Subject: Re: Incremental CMS: long "[GC [1 CMS-initial-mark:" > From: Y. Srinivas Ramakrishna > To: Andrei Pozolotin > Cc: hotspot-gc-use at openjdk.java.net > Date: Wed 17 Feb 2010 04:42:34 PM CST >> On 02/17/10 14:28, Andrei Pozolotin wrote: >>> *Jon, *hello; >>> >>> I need to convert CMS into incremental CMS to reduce latency, >>> eliminate occasional "concurrent mode failure" - related pauses; >>> >>> and I am running into the following problem: >>> >>> 1) in "normal CMS" - [GC [1 CMS-initial-mark: takes 50 milliseconds >>> >>> 2) in "incremental CMS" - [GC [1 CMS-initial-mark: takes 800 >>> milliseconds >> >> This is to be expected and is a consequence of how and when >> the initial mark is done in "normal" vs "incremental". >> In the former, the initial mark almost immediately follows >> after a scavenge pause and Eden is empty. In the latter case >> it normally occurs sometime between two scavenges by which time >> Eden has objects in it. For various reasons, the young gen >> is scanned as a source of initial roots to mark from, >> and for various reasons it is currently the case that this >> work is done single-threaded. >> >> As you will find, the magnitude of this difference typically >> depends (among other factors) on the size of Eden you specify >> (being directly proportional to it) and the duty cycle you >> specify (being inversely proportional to it). >> >> There are ways to fix this in the JVM, but no real workaround that >> i can think of at the command-line level. >> >> If this is important, please file a bug on this via your >> support contacts. >> >> thanks! >> -- ramki >> >>> >>> both above results are under same no-load conditions, >>> with non-reducible statuc heap object set of 430 MB >>> composed of some 350K complex objects; >>> >>> I can not understand why and how to fix it; >>> I tried a lot of Incremental CMS - related option combinations, >>> still dead end. >>> >>> GC settings, log attached; >>> >>> Thank you for your help, >>> >>> Andrei >>> >>> ++++++++++++++++++++++++++++++++++++++++++++++++ >>> >>> # use server-style JIT compiler >>> wrapper.java.additional.100=-server >>> >>> # do GC logging >>> wrapper.java.additional.110=-verbose:gc >>> wrapper.java.additional.111=-Xloggc:./logs/gc.log >>> wrapper.java.additional.112=-XX:+PrintGCDetails >>> wrapper.java.additional.113=-XX:+PrintGCTimeStamps >>> >>> # Heap Total = PermGen + NewGen + OldGen; >>> wrapper.java.additional.120=-Xms3700m >>> wrapper.java.additional.121=-Xmx3700m >>> >>> # PermGen; >>> wrapper.java.additional.130=-XX:PermSize=50m >>> wrapper.java.additional.131=-XX:MaxPermSize=50m >>> >>> # NewGen; Parallel collector; >>> wrapper.java.additional.140=-XX:NewSize=1200m >>> wrapper.java.additional.141=-XX:MaxNewSize=1200m >>> wrapper.java.additional.142=-XX:-UseAdaptiveSizePolicy >>> wrapper.java.additional.143=-XX:SurvivorRatio=1 >>> wrapper.java.additional.144=-XX:TargetSurvivorRatio=90 >>> wrapper.java.additional.145=-XX:MaxTenuringThreshold=3 >>> wrapper.java.additional.146=-XX:+UseParNewGC >>> wrapper.java.additional.147=-XX:-UseParallelGC >>> wrapper.java.additional.148=-XX:ParallelGCThreads=4 >>> >>> # OldGen; Incremental CMS collector; >>> wrapper.java.additional.170=-XX:-UseParallelOldGC >>> wrapper.java.additional.171=-XX:+UseConcMarkSweepGC >>> wrapper.java.additional.172=-XX:+CMSIncrementalMode >>> wrapper.java.additional.173=-XX:-CMSIncrementalPacing >>> wrapper.java.additional.174=-XX:CMSIncrementalDutyCycle=1 >>> wrapper.java.additional.175=-XX:+CMSParallelRemarkEnabled >>> wrapper.java.additional.176=-XX:+CMSScavengeBeforeRemark >>> wrapper.java.additional.177=-XX:CMSIncrementalOffset=10 >>> wrapper.java.additional.178=-XX:CMSScheduleRemarkEdenPenetration=5 >>> wrapper.java.additional.179=-XX:CMSWaitDuration=60000 >>> wrapper.java.additional.181=-XX:ParallelCMSThreads=1 >>> wrapper.java.additional.182=-XX:PrintCMSStatistics=0 >>> >>> >>> ------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> hotspot-gc-use mailing list >>> hotspot-gc-use at openjdk.java.net >>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> > From andrei.pozolotin at gmail.com Wed Feb 17 15:42:14 2010 From: andrei.pozolotin at gmail.com (Andrei Pozolotin) Date: Wed, 17 Feb 2010 17:42:14 -0600 Subject: Incremental CMS: long "[GC [1 CMS-initial-mark:" In-Reply-To: <4B7C7C4B.3060207@Sun.COM> References: <4B7C6D91.6000202@gmail.com> <4B7C70DA.7070204@Sun.COM> <4B7C7476.6050607@gmail.com> <4B7C7C4B.3060207@Sun.COM> Message-ID: <4B7C7ED6.4010607@gmail.com> Ramki, hello again; 1) "unfortunately" - got it; thank you; 2) thanks for G1 idea; will try now; 3) it does not seem gchisto / visualvm can see G1 at all; can you please recommend a tool that can help with G1 tuning? Cheers, Andrei -------- Original Message -------- Subject: Re: Incremental CMS: long "[GC [1 CMS-initial-mark:" From: Y. Srinivas Ramakrishna To: Andrei Pozolotin Cc: hotspot-gc-use at openjdk.java.net Date: Wed 17 Feb 2010 05:31:23 PM CST > On 02/17/10 14:57, Andrei Pozolotin wrote: >> Ramki, hi >> >> Thank you for quick response; >> >> from reading JDK 6 source I deluded myself that the problem you >> describe: >> >> "... occurs sometime between two scavenges ..." >> >> would get fixed by these options: >> >> wrapper.java.additional.175=-XX:+CMSParallelRemarkEnabled >> wrapper.java.additional.176=-XX:+CMSScavengeBeforeRemark >> wrapper.java.additional.177=-XX:CMSIncrementalOffset=10 >> wrapper.java.additional.178=-XX:CMSScheduleRemarkEdenPenetration=5 >> >> please confirm I was wrong? > > Yes, unfortunately, those are in some sense orthogonal to > the way CMS "incrementalizes" its working, here (somewhat > relatedly see [1]) > > For example, CMSScavengeBeforeRemark does indeed schedule a scavenge > before a remark pause, but not (unfortunately) before an initial > mark pause. Similarly the CMSScheduleRemarkEdenPenetration > controls the scheduling of remark pauses, not the initial > mark pauses. > > As for CMSIncrementalOffset, it is just a constant positive delta > applied to the ideal starting point for a given duty cycle. > So, in fact, any positive value will take you further away > from the objective you are trying to achieve here of reducing > the occupancy of Eden when initial mark hits. > > Try G1, that may save you filing an RFE on (i)CMS for this. > > Honestly though, there is an existing blanket RFE on this: > 6412968 CMS: Long initial mark pauses > but it has not risen to the top of our list of things to fix. > > If this is important, please contact your Java support.... > > -- ramki > > [1] 6518192 iCMS: smoother, duration-bounded incrementality > > >> Cheers, >> >> Andrei >> >> >> >> -------- Original Message -------- >> Subject: Re: Incremental CMS: long "[GC [1 CMS-initial-mark:" >> From: Y. Srinivas Ramakrishna >> To: Andrei Pozolotin >> Cc: hotspot-gc-use at openjdk.java.net >> Date: Wed 17 Feb 2010 04:42:34 PM CST >>> On 02/17/10 14:28, Andrei Pozolotin wrote: >>>> *Jon, *hello; >>>> >>>> I need to convert CMS into incremental CMS to reduce latency, >>>> eliminate occasional "concurrent mode failure" - related pauses; >>>> >>>> and I am running into the following problem: >>>> >>>> 1) in "normal CMS" - [GC [1 CMS-initial-mark: takes 50 >>>> milliseconds >>>> >>>> 2) in "incremental CMS" - [GC [1 CMS-initial-mark: takes 800 >>>> milliseconds >>> >>> This is to be expected and is a consequence of how and when >>> the initial mark is done in "normal" vs "incremental". >>> In the former, the initial mark almost immediately follows >>> after a scavenge pause and Eden is empty. In the latter case >>> it normally occurs sometime between two scavenges by which time >>> Eden has objects in it. For various reasons, the young gen >>> is scanned as a source of initial roots to mark from, >>> and for various reasons it is currently the case that this >>> work is done single-threaded. >>> >>> As you will find, the magnitude of this difference typically >>> depends (among other factors) on the size of Eden you specify >>> (being directly proportional to it) and the duty cycle you >>> specify (being inversely proportional to it). >>> >>> There are ways to fix this in the JVM, but no real workaround that >>> i can think of at the command-line level. >>> >>> If this is important, please file a bug on this via your >>> support contacts. >>> >>> thanks! >>> -- ramki >>> >>>> >>>> both above results are under same no-load conditions, >>>> with non-reducible statuc heap object set of 430 MB >>>> composed of some 350K complex objects; >>>> >>>> I can not understand why and how to fix it; >>>> I tried a lot of Incremental CMS - related option combinations, >>>> still dead end. >>>> >>>> GC settings, log attached; >>>> >>>> Thank you for your help, >>>> >>>> Andrei >>>> >>>> ++++++++++++++++++++++++++++++++++++++++++++++++ >>>> >>>> # use server-style JIT compiler >>>> wrapper.java.additional.100=-server >>>> >>>> # do GC logging >>>> wrapper.java.additional.110=-verbose:gc >>>> wrapper.java.additional.111=-Xloggc:./logs/gc.log >>>> wrapper.java.additional.112=-XX:+PrintGCDetails >>>> wrapper.java.additional.113=-XX:+PrintGCTimeStamps >>>> >>>> # Heap Total = PermGen + NewGen + OldGen; >>>> wrapper.java.additional.120=-Xms3700m >>>> wrapper.java.additional.121=-Xmx3700m >>>> >>>> # PermGen; >>>> wrapper.java.additional.130=-XX:PermSize=50m >>>> wrapper.java.additional.131=-XX:MaxPermSize=50m >>>> >>>> # NewGen; Parallel collector; >>>> wrapper.java.additional.140=-XX:NewSize=1200m >>>> wrapper.java.additional.141=-XX:MaxNewSize=1200m >>>> wrapper.java.additional.142=-XX:-UseAdaptiveSizePolicy >>>> wrapper.java.additional.143=-XX:SurvivorRatio=1 >>>> wrapper.java.additional.144=-XX:TargetSurvivorRatio=90 >>>> wrapper.java.additional.145=-XX:MaxTenuringThreshold=3 >>>> wrapper.java.additional.146=-XX:+UseParNewGC >>>> wrapper.java.additional.147=-XX:-UseParallelGC >>>> wrapper.java.additional.148=-XX:ParallelGCThreads=4 >>>> >>>> # OldGen; Incremental CMS collector; >>>> wrapper.java.additional.170=-XX:-UseParallelOldGC >>>> wrapper.java.additional.171=-XX:+UseConcMarkSweepGC >>>> wrapper.java.additional.172=-XX:+CMSIncrementalMode >>>> wrapper.java.additional.173=-XX:-CMSIncrementalPacing >>>> wrapper.java.additional.174=-XX:CMSIncrementalDutyCycle=1 >>>> wrapper.java.additional.175=-XX:+CMSParallelRemarkEnabled >>>> wrapper.java.additional.176=-XX:+CMSScavengeBeforeRemark >>>> wrapper.java.additional.177=-XX:CMSIncrementalOffset=10 >>>> wrapper.java.additional.178=-XX:CMSScheduleRemarkEdenPenetration=5 >>>> wrapper.java.additional.179=-XX:CMSWaitDuration=60000 >>>> wrapper.java.additional.181=-XX:ParallelCMSThreads=1 >>>> wrapper.java.additional.182=-XX:PrintCMSStatistics=0 >>>> >>>> >>>> ------------------------------------------------------------------------ >>>> >>>> >>>> _______________________________________________ >>>> hotspot-gc-use mailing list >>>> hotspot-gc-use at openjdk.java.net >>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>> >>> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20100217/7646c6fc/attachment-0001.html From Jon.Masamitsu at Sun.COM Wed Feb 17 22:25:25 2010 From: Jon.Masamitsu at Sun.COM (Jon Masamitsu) Date: Wed, 17 Feb 2010 22:25:25 -0800 Subject: Incremental CMS: long "[GC [1 CMS-initial-mark:" Message-ID: <301DA800-BF22-4E96-B5C1-610C4F19CC96@sun.com> > From: Andrei Pozolotin > Date: February 17, 2010 3:42:14 PM PST > To: ysr at Sun.COM > Cc: hotspot-gc-use at openjdk.java.net > Subject: Re: Incremental CMS: long "[GC [1 CMS-initial-mark:" > > > Ramki, hello again; > > 1) "unfortunately" - got it; thank you; > > 2) thanks for G1 idea; will try now; > > 3) it does not seem gchisto / visualvm can see G1 at all; > can you please recommend a tool that can help with G1 tuning? Yes, sadly, gchisto has not been updated to parse the G1 style of logging. The work is probably in part to add to/change the G1 logging to be more like the other collectors and then to update gchisto. That's my guess and since its still a guess, we're not close to getting there. > > Cheers, > > Andrei -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20100217/dc94faea/attachment.html From andrei.pozolotin at gmail.com Thu Feb 18 07:18:15 2010 From: andrei.pozolotin at gmail.com (Andrei Pozolotin) Date: Thu, 18 Feb 2010 09:18:15 -0600 Subject: Incremental CMS: long "[GC [1 CMS-initial-mark:" In-Reply-To: <301DA800-BF22-4E96-B5C1-610C4F19CC96@sun.com> References: <301DA800-BF22-4E96-B5C1-610C4F19CC96@sun.com> Message-ID: <4B7D5A37.8060504@gmail.com> Jon, Ramki, hi; Thank you for your answers; Andrei. -------- Original Message -------- Subject: Re: Incremental CMS: long "[GC [1 CMS-initial-mark:" From: Jon Masamitsu To: andrei.pozolotin at gmail.com Cc: hotspot-gc-use at openjdk.java.net Date: Thu 18 Feb 2010 12:25:25 AM CST >> *From: *Andrei Pozolotin > > >> *Date: *February 17, 2010 3:42:14 PM PST >> *To: *ysr at Sun.COM >> *Cc: *hotspot-gc-use at openjdk.java.net >> >> *Subject: **Re: Incremental CMS: long "[GC [1 CMS-initial-mark:"* >> >> >> Ramki, hello again; >> >> 1) "unfortunately" - got it; thank you; >> >> 2) thanks for G1 idea; will try now; >> >> 3) it does not seem gchisto / visualvm can see G1 at all; >> can you please recommend a tool that can help with G1 tuning? >> > > Yes, sadly, gchisto has not been updated to parse the G1 style of > logging. The work > is probably in part to add to/change the G1 logging to be more like > the other collectors > and then to update gchisto. That's my guess and since its still a > guess, we're not > close to getting there. >> >> >> Cheers, >> >> Andrei >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20100218/2c65136a/attachment.html From andrei.pozolotin at gmail.com Thu Feb 18 08:56:57 2010 From: andrei.pozolotin at gmail.com (Andrei Pozolotin) Date: Thu, 18 Feb 2010 10:56:57 -0600 Subject: G1 is bad for me Message-ID: <4B7D7159.5000808@gmail.com> Ramki, Jon, hi; I read everyting I could find on G1: http://www.google.com/patents/about?id=j5SpAAAAEBAJ&dq=7,340,494 http://developers.sun.com/learning/javaoneonline/2008/pdf/TS-5419.pdf http://research.sun.com/jtech/pubs/04-g1-paper-ismm.pdf http://blogs.sun.com/sunabl/entry/the_new_g1_collector http://blogs.sun.com/jonthecollector/entry/our_collectors etc, etc, etc, got myself excited - looks great on paper! and I followed Albert's advice, while trying to mimic my CMS settings: http://blogs.sun.com/sunabl/entry/the_new_g1_collector -Xms3700m -Xmx3700m -XX:PermSize=50m -XX:MaxPermSize=50m -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC -XX:G1YoungGenSize=900m -XX:SurvivorRatio=2 -XX:+G1ParallelRSetUpdatingEnabled -XX:+G1ParallelRSetScanningEnabled results: 1) pauses are 1000 milliseconds or more; 2) reproducible crashes after 3 minutes under load; logs attached; am I missing something obvious? like do I need to switch to older / specific JVM build? per Tony, this was already working fine for a neighbour of mine in 2008: http://developers.sun.com/learning/javaoneonline/2008/pdf/TS-5419.pdf Thank you; Andrei -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20100218/44d48184/attachment-0001.html -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: gc.log.3 Url: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20100218/44d48184/attachment-0002.ksh -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: hs_err_pid17694.log Url: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20100218/44d48184/attachment-0003.ksh From Y.S.Ramakrishna at Sun.COM Thu Feb 18 09:37:56 2010 From: Y.S.Ramakrishna at Sun.COM (Y. Srinivas Ramakrishna) Date: Thu, 18 Feb 2010 09:37:56 -0800 Subject: G1 is bad for me In-Reply-To: <4B7D7159.5000808@gmail.com> References: <4B7D7159.5000808@gmail.com> Message-ID: <4B7D7AF4.8010801@Sun.COM> Hi Andrei -- please use the latest hs17 JVM which you can get from JDK 7. There have been many improvements (and bug fixes) since the 6u18/hs16 that you are using below. Most importantly, please add -XX:-ReduceInitialCardMarks to work around a regression introduced in 6u18/hs16, as described in the 6u18 release notes. As re optimal settings (these may be quite different from shapes that helped with CMS for example), i'll let others w/more (tuning and other) experience w/G1 comment. -- ramki On 02/18/10 08:56, Andrei Pozolotin wrote: > Ramki, Jon, hi; > > I read everyting I could find on G1: > http://www.google.com/patents/about?id=j5SpAAAAEBAJ&dq=7,340,494 > http://developers.sun.com/learning/javaoneonline/2008/pdf/TS-5419.pdf > http://research.sun.com/jtech/pubs/04-g1-paper-ismm.pdf > http://blogs.sun.com/sunabl/entry/the_new_g1_collector > http://blogs.sun.com/jonthecollector/entry/our_collectors > etc, etc, etc, > > got myself excited - looks great on paper! > > and I followed Albert's advice, while trying to mimic my CMS settings: > http://blogs.sun.com/sunabl/entry/the_new_g1_collector > > -Xms3700m > -Xmx3700m > -XX:PermSize=50m > -XX:MaxPermSize=50m > -XX:+UnlockExperimentalVMOptions > -XX:+UseG1GC > -XX:G1YoungGenSize=900m > -XX:SurvivorRatio=2 > -XX:+G1ParallelRSetUpdatingEnabled > -XX:+G1ParallelRSetScanningEnabled > > results: > > 1) pauses are 1000 milliseconds or more; > > 2) reproducible crashes after 3 minutes under load; > > logs attached; > > am I missing something obvious? > like do I need to switch to older / specific JVM build? > > per Tony, this was already working fine for a neighbour of mine in 2008: > http://developers.sun.com/learning/javaoneonline/2008/pdf/TS-5419.pdf > > Thank you; > > Andrei > > > ------------------------------------------------------------------------ > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From andrei.pozolotin at gmail.com Thu Feb 18 12:04:24 2010 From: andrei.pozolotin at gmail.com (Andrei Pozolotin) Date: Thu, 18 Feb 2010 14:04:24 -0600 Subject: G1 is bad for me In-Reply-To: <4B7D7AF4.8010801@Sun.COM> References: <4B7D7159.5000808@gmail.com> <4B7D7AF4.8010801@Sun.COM> Message-ID: <4B7D9D48.1020807@gmail.com> Ramki, hello; thank you for the advice - I followed it; results: 1) pauses now are 500 ms instead of 1000 ms; 2) no crashes under production load after 30 minutes; progress! now I am looking for advice how to move pauses into 50 ms range and remove all-4-cpu stop the world 100% cpu load spikes; log attached; cheers, Andrei -------- Original Message -------- Subject: Re: G1 is bad for me From: Y. Srinivas Ramakrishna To: Andrei Pozolotin Cc: hotspot-gc-use at openjdk.java.net Date: Thu 18 Feb 2010 11:37:56 AM CST > Hi Andrei -- please use the latest hs17 JVM which you can get from JDK > 7. There have been > many improvements (and bug fixes) since the 6u18/hs16 that you are > using below. > Most importantly, please add -XX:-ReduceInitialCardMarks to work > around a regression > introduced in 6u18/hs16, as described in the 6u18 release notes. > As re optimal settings (these may be quite different from shapes that > helped with CMS > for example), i'll let others w/more (tuning and other) experience > w/G1 comment. > > -- ramki > > On 02/18/10 08:56, Andrei Pozolotin wrote: >> Ramki, Jon, hi; >> >> I read everyting I could find on G1: >> http://www.google.com/patents/about?id=j5SpAAAAEBAJ&dq=7,340,494 >> >> http://developers.sun.com/learning/javaoneonline/2008/pdf/TS-5419.pdf >> http://research.sun.com/jtech/pubs/04-g1-paper-ismm.pdf >> http://blogs.sun.com/sunabl/entry/the_new_g1_collector >> http://blogs.sun.com/jonthecollector/entry/our_collectors >> etc, etc, etc, >> >> got myself excited - looks great on paper! >> >> and I followed Albert's advice, while trying to mimic my CMS >> settings: >> http://blogs.sun.com/sunabl/entry/the_new_g1_collector >> >> -Xms3700m >> -Xmx3700m >> -XX:PermSize=50m >> -XX:MaxPermSize=50m >> -XX:+UnlockExperimentalVMOptions >> -XX:+UseG1GC >> -XX:G1YoungGenSize=900m >> -XX:SurvivorRatio=2 >> -XX:+G1ParallelRSetUpdatingEnabled >> -XX:+G1ParallelRSetScanningEnabled >> >> results: >> >> 1) pauses are 1000 milliseconds or more; >> >> 2) reproducible crashes after 3 minutes under load; >> >> logs attached; >> >> am I missing something obvious? >> like do I need to switch to older / specific JVM build? >> >> per Tony, this was already working fine for a neighbour of mine >> in 2008: >> >> http://developers.sun.com/learning/javaoneonline/2008/pdf/TS-5419.pdf >> >> Thank you; >> >> Andrei >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20100218/edd9d9a7/attachment-0001.html -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: gc.log Url: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20100218/edd9d9a7/attachment-0001.ksh From chkwok at digibites.nl Fri Feb 19 06:59:53 2010 From: chkwok at digibites.nl (Chi Ho Kwok) Date: Fri, 19 Feb 2010 15:59:53 +0100 Subject: G1 is bad for me In-Reply-To: <4B7D7AF4.8010801@Sun.COM> References: <4B7D7159.5000808@gmail.com> <4B7D7AF4.8010801@Sun.COM> Message-ID: <1b9d6f691002190659i6917bdafo2d61a4f59a833f41@mail.gmail.com> Hi all, A large part of our CPU load is caused by GC, so I'm always interested in new techniques, but the last time I tried G1 (6u14?), the jvm rapidly crashed after startup under load. Is the recommendation to switch to/wait for JDK7 releases for G1, or is G1 on 6u18 (or later?) any good for production usage? I'd assume there will be more updates for JDK6 as JDK7 got extended to up to 10 milestones. CMS with a 32G heap which changes rapidly (~10%/min replaced) is manageable, but we're "losing" 20% capacity because the threshold has to be so low, anything higher than 80% causes new gen promotions to fail during CMS and the app to stop for half a minute. Processing capacity is also reduced by half during a CMS run because the amount of threads has to be high (=6 at the moment) to make it hurry up, or it'll mean either setting the threshold even lower or risk promotion failures :( Chi Ho Kwok On Thu, Feb 18, 2010 at 6:37 PM, Y. Srinivas Ramakrishna < Y.S.Ramakrishna at sun.com> wrote: > Hi Andrei -- please use the latest hs17 JVM which you can get from JDK 7. > There have been > many improvements (and bug fixes) since the 6u18/hs16 that you are using > below. > Most importantly, please add -XX:-ReduceInitialCardMarks to work around a > regression > introduced in 6u18/hs16, as described in the 6u18 release notes. > As re optimal settings (these may be quite different from shapes that > helped with CMS > for example), i'll let others w/more (tuning and other) experience w/G1 > comment. > > -- ramki -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20100219/f7788cbf/attachment.html From andrei.pozolotin at gmail.com Fri Feb 19 07:21:26 2010 From: andrei.pozolotin at gmail.com (Andrei Pozolotin) Date: Fri, 19 Feb 2010 09:21:26 -0600 Subject: G1 is bad for me In-Reply-To: <1b9d6f691002190659i6917bdafo2d61a4f59a833f41@mail.gmail.com> References: <4B7D7159.5000808@gmail.com> <4B7D7AF4.8010801@Sun.COM> <1b9d6f691002190659i6917bdafo2d61a4f59a833f41@mail.gmail.com> Message-ID: <4B7EAC76.6010203@gmail.com> Chi Ho Kwok, hi I can confirm the following: 1) I followed Ramki advice below and deployed JDK 7 b83 with HS17; 2) it does stop crashes in G1 which were happening in JDK 6 u18 HS16; 3) but G1 cpu usage is like 10 times more then CMS under same load, and cpu load pattern is excessively spikey in G1, while much more smooth in CMS; 4) G1 pauses are 200 ms, while CMS pauses are 50 ms under same load; 5) I am using the following ergonomic options: -XX:MaxGCPauseMillis=70 -XX:GCPauseIntervalMillis=700 6) per gc logs, most of G1 200 ms pause is spent in "Remembered Set Rescan" this is presumably because this "helpful ergonomics" can not separate static heap set and keeps scanning it all the time; any advice how to kill this ergonomics and provide manual overrides is much appreciated! thank you Andrei -------- Original Message -------- Subject: Re: G1 is bad for me From: Chi Ho Kwok To: hotspot-gc-use Date: Fri 19 Feb 2010 08:59:53 AM CST > Hi all, > > A large part of our CPU load is caused by GC, so I'm always interested > in new techniques, but the last time I tried G1 (6u14?), the jvm > rapidly crashed after startup under load. Is the recommendation to > switch to/wait for JDK7 releases for G1, or is G1 on 6u18 (or later?) > any good for production usage? I'd assume there will be more updates > for JDK6 as JDK7 got extended to up to 10 milestones. > > CMS with a 32G heap which changes rapidly (~10%/min replaced) is > manageable, but we're "losing" 20% capacity because the threshold has > to be so low, anything higher than 80% causes new gen promotions to > fail during CMS and the app to stop for half a minute. Processing > capacity is also reduced by half during a CMS run because the amount > of threads has to be high (=6 at the moment) to make it hurry up, or > it'll mean either setting the threshold even lower or risk promotion > failures :( > > Chi Ho Kwok > > On Thu, Feb 18, 2010 at 6:37 PM, Y. Srinivas Ramakrishna > > wrote: > > Hi Andrei -- please use the latest hs17 JVM which you can get from > JDK 7. There have been > many improvements (and bug fixes) since the 6u18/hs16 that you are > using below. > Most importantly, please add -XX:-ReduceInitialCardMarks to work > around a regression > introduced in 6u18/hs16, as described in the 6u18 release notes. > As re optimal settings (these may be quite different from shapes > that helped with CMS > for example), i'll let others w/more (tuning and other) experience > w/G1 comment. > > -- ramki > > ------------------------------------------------------------------------ > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20100219/30f3828e/attachment.html From andrei.pozolotin at gmail.com Fri Feb 19 07:50:13 2010 From: andrei.pozolotin at gmail.com (Andrei Pozolotin) Date: Fri, 19 Feb 2010 09:50:13 -0600 Subject: G1 is bad for me In-Reply-To: <1b9d6f691002190659i6917bdafo2d61a4f59a833f41@mail.gmail.com> References: <4B7D7159.5000808@gmail.com> <4B7D7AF4.8010801@Sun.COM> <1b9d6f691002190659i6917bdafo2d61a4f59a833f41@mail.gmail.com> Message-ID: <4B7EB335.5050701@gmail.com> these are CMS settings and log sample, for same reference load ++++++++++++++++++++++++++++++++++++++++++++++++ wrapper.java.additional.100=-server # do GC logging wrapper.java.additional.110=-verbose:gc wrapper.java.additional.111=-Xloggc:./logs/gc.log wrapper.java.additional.112=-XX:+PrintGCDetails wrapper.java.additional.113=-XX:+PrintGCTimeStamps # Heap Total = PermGen + NewGen + OldGen; wrapper.java.additional.120=-Xms3700m wrapper.java.additional.121=-Xmx3700m # PermGen; wrapper.java.additional.130=-XX:PermSize=50m wrapper.java.additional.131=-XX:MaxPermSize=50m # NewGen; Parallel collector; wrapper.java.additional.140=-XX:NewSize=1800m wrapper.java.additional.141=-XX:MaxNewSize=1800m wrapper.java.additional.142=-XX:-UseAdaptiveSizePolicy wrapper.java.additional.143=-XX:SurvivorRatio=1 wrapper.java.additional.144=-XX:TargetSurvivorRatio=90 wrapper.java.additional.145=-XX:MaxTenuringThreshold=2 wrapper.java.additional.146=-XX:+UseParNewGC wrapper.java.additional.147=-XX:-UseParallelGC wrapper.java.additional.148=-XX:ParallelGCThreads=4 # OldGen; Incremental CMS collector; wrapper.java.additional.170=-XX:-UseParallelOldGC wrapper.java.additional.171=-XX:+UseConcMarkSweepGC wrapper.java.additional.172=-XX:+UseCMSInitiatingOccupancyOnly wrapper.java.additional.173=-XX:CMSInitiatingOccupancyFraction=24 wrapper.java.additional.175=-XX:+CMSParallelRemarkEnabled wrapper.java.additional.176=-XX:+CMSScavengeBeforeRemark wrapper.java.additional.179=-XX:CMSWaitDuration=60000 wrapper.java.additional.181=-XX:ParallelCMSThreads=1 wrapper.java.additional.182=-XX:PrintCMSStatistics=0 ++++++++++++++++++++++++++++++++++++++++++++++++ 1426.274: [GC 1426.274: [ParNew: 615748K->1899K(1228800K), 0.0369160 secs] 1048474K->434948K(3174400K), 0.0370710 secs] [Times: user=0.10 sys=0.00, real=0.04 secs] 1440.019: [GC 1440.019: [ParNew: 616299K->1474K(1228800K), 0.0353210 secs] 1049348K->434647K(3174400K), 0.0354860 secs] [Times: user=0.09 sys=0.00, real=0.03 secs] 1452.544: [GC 1452.544: [ParNew: 615874K->1506K(1228800K), 0.0349110 secs] 1049047K->434841K(3174400K), 0.0350570 secs] [Times: user=0.09 sys=0.00, real=0.03 secs] 1467.298: [GC 1467.298: [ParNew: 615906K->1916K(1228800K), 0.0292920 secs] 1049241K->435408K(3174400K), 0.0295150 secs] [Times: user=0.12 sys=0.00, real=0.03 secs] 1480.940: [GC 1480.940: [ParNew: 616316K->1629K(1228800K), 0.0284240 secs] 1049808K->435324K(3174400K), 0.0285660 secs] [Times: user=0.11 sys=0.00, real=0.02 secs] 1490.471: [GC 1490.472: [ParNew: 616029K->2056K(1228800K), 0.0306450 secs] 1049724K->435967K(3174400K), 0.0308110 secs] [Times: user=0.12 sys=0.00, real=0.03 secs] 1503.550: [GC 1503.550: [ParNew: 616456K->2198K(1228800K), 0.0323910 secs] 1050367K->436251K(3174400K), 0.0325430 secs] [Times: user=0.12 sys=0.00, real=0.03 secs] 1514.911: [GC 1514.911: [ParNew: 616598K->1687K(1228800K), 0.0315140 secs] 1050651K->436052K(3174400K), 0.0316790 secs] [Times: user=0.12 sys=0.00, real=0.03 secs] 1524.320: [GC 1524.320: [ParNew: 616087K->1773K(1228800K), 0.0300510 secs] 1050452K->436382K(3174400K), 0.0302330 secs] [Times: user=0.12 sys=0.00, real=0.03 secs] 1529.386: [GC 1529.386: [ParNew: 616173K->1671K(1228800K), 0.0297700 secs] 1050782K->436438K(3174400K), 0.0299210 secs] [Times: user=0.11 sys=0.00, real=0.03 secs] ++++++++++++++++++++++++++++++++++++++++++++++++ -------- Original Message -------- Subject: Re: G1 is bad for me From: Chi Ho Kwok To: hotspot-gc-use Date: Fri 19 Feb 2010 08:59:53 AM CST > Hi all, > > A large part of our CPU load is caused by GC, so I'm always interested > in new techniques, but the last time I tried G1 (6u14?), the jvm > rapidly crashed after startup under load. Is the recommendation to > switch to/wait for JDK7 releases for G1, or is G1 on 6u18 (or later?) > any good for production usage? I'd assume there will be more updates > for JDK6 as JDK7 got extended to up to 10 milestones. > > CMS with a 32G heap which changes rapidly (~10%/min replaced) is > manageable, but we're "losing" 20% capacity because the threshold has > to be so low, anything higher than 80% causes new gen promotions to > fail during CMS and the app to stop for half a minute. Processing > capacity is also reduced by half during a CMS run because the amount > of threads has to be high (=6 at the moment) to make it hurry up, or > it'll mean either setting the threshold even lower or risk promotion > failures :( > > Chi Ho Kwok > > On Thu, Feb 18, 2010 at 6:37 PM, Y. Srinivas Ramakrishna > > wrote: > > Hi Andrei -- please use the latest hs17 JVM which you can get from > JDK 7. There have been > many improvements (and bug fixes) since the 6u18/hs16 that you are > using below. > Most importantly, please add -XX:-ReduceInitialCardMarks to work > around a regression > introduced in 6u18/hs16, as described in the 6u18 release notes. > As re optimal settings (these may be quite different from shapes > that helped with CMS > for example), i'll let others w/more (tuning and other) experience > w/G1 comment. > > -- ramki > > ------------------------------------------------------------------------ > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20100219/3d31e3ac/attachment.html From andrei.pozolotin at gmail.com Fri Feb 19 07:50:18 2010 From: andrei.pozolotin at gmail.com (Andrei Pozolotin) Date: Fri, 19 Feb 2010 09:50:18 -0600 Subject: G1 is bad for me In-Reply-To: <1b9d6f691002190659i6917bdafo2d61a4f59a833f41@mail.gmail.com> References: <4B7D7159.5000808@gmail.com> <4B7D7AF4.8010801@Sun.COM> <1b9d6f691002190659i6917bdafo2d61a4f59a833f41@mail.gmail.com> Message-ID: <4B7EB33A.8030808@gmail.com> these are G1 settings and log sample, for same reference load ++++++++++++++++++++++++++++++++++++++++++++++++ # use server-style JIT compiler wrapper.java.additional.100=-server # do GC logging wrapper.java.additional.110=-verbose:gc wrapper.java.additional.111=-Xloggc:./logs/gc.log wrapper.java.additional.112=-XX:+PrintGCDetails wrapper.java.additional.113=-XX:+PrintGCTimeStamps # Heap Total = PermGen + NewGen + OldGen; wrapper.java.additional.120=-Xms3700m wrapper.java.additional.121=-Xmx3700m # PermGen; wrapper.java.additional.130=-XX:PermSize=50m wrapper.java.additional.131=-XX:MaxPermSize=50m # NewGen; Parallel collector; wrapper.java.additional.150=-XX:+UnlockExperimentalVMOptions wrapper.java.additional.151=-XX:+UseG1GC wrapper.java.additional.152=-XX:-ReduceInitialCardMarks wrapper.java.additional.156=-XX:+G1ParallelRSetUpdatingEnabled wrapper.java.additional.157=-XX:+G1ParallelRSetScanningEnabled wrapper.java.additional.159=-XX:MaxGCPauseMillis=70 wrapper.java.additional.160=-XX:GCPauseIntervalMillis=700 ++++++++++++++++++++++++++++++++++++++++++++++++ 614.106: [GC pause (young), 0.32634900 secs] [Parallel Time: 323.5 ms] [Update RS (Start) (ms): 614110.5 614110.8 614111.4 614110.5] [Update RS (ms): 8.0 7.7 7.2 8.2 Avg: 7.8, Min: 7.2, Max: 8.2] [Processed Buffers : 4 8 6 4 Sum: 22, Avg: 5, Min: 4, Max: 8] [Ext Root Scanning (ms): 3.7 4.0 4.2 3.7 Avg: 3.9, Min: 3.7, Max: 4.2] [Mark Stack Scanning (ms): 0.0 0.0 0.0 0.0 Avg: 0.0, Min: 0.0, Max: 0.0] [Scan-Only Scanning (ms): 0.0 0.0 0.0 0.0 Avg: 0.0, Min: 0.0, Max: 0.0] [Scan-Only Regions : 0 0 0 0 Sum: 0, Avg: 0, Min: 0, Max: 0] [Scan RS (ms): 301.0 301.3 300.7 300.9 Avg: 301.0, Min: 300.7, Max: 301.3] [Object Copy (ms): 10.2 9.9 10.8 10.0 Avg: 10.2, Min: 9.9, Max: 10.8] [Termination (ms): 0.0 0.0 0.0 0.0 Avg: 0.0, Min: 0.0, Max: 0.0] [Other: 0.6 ms] [Clear CT: 0.9 ms] [Other: 2.0 ms] [ 823M->742M(3700M)] [Times: user=1.23 sys=0.01, real=0.33 secs] ++++++++++++++++++++++++++++++++++++++++++++++++ -------- Original Message -------- Subject: Re: G1 is bad for me From: Chi Ho Kwok To: hotspot-gc-use Date: Fri 19 Feb 2010 08:59:53 AM CST > Hi all, > > A large part of our CPU load is caused by GC, so I'm always interested > in new techniques, but the last time I tried G1 (6u14?), the jvm > rapidly crashed after startup under load. Is the recommendation to > switch to/wait for JDK7 releases for G1, or is G1 on 6u18 (or later?) > any good for production usage? I'd assume there will be more updates > for JDK6 as JDK7 got extended to up to 10 milestones. > > CMS with a 32G heap which changes rapidly (~10%/min replaced) is > manageable, but we're "losing" 20% capacity because the threshold has > to be so low, anything higher than 80% causes new gen promotions to > fail during CMS and the app to stop for half a minute. Processing > capacity is also reduced by half during a CMS run because the amount > of threads has to be high (=6 at the moment) to make it hurry up, or > it'll mean either setting the threshold even lower or risk promotion > failures :( > > Chi Ho Kwok > > On Thu, Feb 18, 2010 at 6:37 PM, Y. Srinivas Ramakrishna > > wrote: > > Hi Andrei -- please use the latest hs17 JVM which you can get from > JDK 7. There have been > many improvements (and bug fixes) since the 6u18/hs16 that you are > using below. > Most importantly, please add -XX:-ReduceInitialCardMarks to work > around a regression > introduced in 6u18/hs16, as described in the 6u18 release notes. > As re optimal settings (these may be quite different from shapes > that helped with CMS > for example), i'll let others w/more (tuning and other) experience > w/G1 comment. > > -- ramki > > ------------------------------------------------------------------------ > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20100219/22e46e4e/attachment-0001.html From Jon.Masamitsu at Sun.COM Fri Feb 19 07:53:45 2010 From: Jon.Masamitsu at Sun.COM (Jon Masamitsu) Date: Fri, 19 Feb 2010 07:53:45 -0800 Subject: G1 is bad for me In-Reply-To: <1b9d6f691002190659i6917bdafo2d61a4f59a833f41@mail.gmail.com> References: <4B7D7159.5000808@gmail.com> <4B7D7AF4.8010801@Sun.COM> <1b9d6f691002190659i6917bdafo2d61a4f59a833f41@mail.gmail.com> Message-ID: <4B7EB409.9080006@sun.com> Chi Ho Kwok wrote On 02/19/10 06:59,: > Hi all, > > A large part of our CPU load is caused by GC, so I'm always interested > in new techniques, but the last time I tried G1 (6u14?), the jvm > rapidly crashed after startup under load. Is the recommendation to > switch to/wait for JDK7 releases for G1, or is G1 on 6u18 (or later?) > any good for production usage? I'd assume there will be more updates > for JDK6 as JDK7 got extended to up to 10 milestones. G1 is in 6u18 but it is still an experimental collector. It's definitely better but if you try it add the flag -XX:-ReduceInitialCardMarks to work around regression that escaped in 6u18. > > CMS with a 32G heap which changes rapidly (~10%/min replaced) is > manageable, but we're "losing" 20% capacity because the threshold has > to be so low, anything higher than 80% causes new gen promotions to > fail during CMS and the app to stop for half a minute. Processing > capacity is also reduced by half during a CMS run because the amount > of threads has to be high (=6 at the moment) to make it hurry up, or > it'll mean either setting the threshold even lower or risk promotion > failures :( If you're running into fragmentation, then either you start the CMS cycles early (as it appears you do) or use a larger heap (if you can). There an RFE in jdk 7 (6631166) that helps with fragmentation in some applications. It's not in a jdk 6 yet. If you're a Java-4-business customer, contact your Sun/Oracle support person and talk to them about getting that RFE. If your application is allocating objects at a high rate and CMS has to do lots of processing to keep up (might be the case if you're using 6 GC threads to do the concurrent marking in parallel), then the overhead may just be what's required to collect all that garbage. Again, if you're a Java-4-business customer, talk to your support person to see if you can get some help in analyzing your situation. > > Chi Ho Kwok > > On Thu, Feb 18, 2010 at 6:37 PM, Y. Srinivas Ramakrishna > > wrote: > > Hi Andrei -- please use the latest hs17 JVM which you can get from > JDK 7. There have been > many improvements (and bug fixes) since the 6u18/hs16 that you are > using below. > Most importantly, please add -XX:-ReduceInitialCardMarks to work > around a regression > introduced in 6u18/hs16, as described in the 6u18 release notes. > As re optimal settings (these may be quite different from shapes > that helped with CMS > for example), i'll let others w/more (tuning and other) experience > w/G1 comment. > > -- ramki > >------------------------------------------------------------------------ > >_______________________________________________ >hotspot-gc-use mailing list >hotspot-gc-use at openjdk.java.net >http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > From andrei.pozolotin at gmail.com Mon Feb 22 07:28:45 2010 From: andrei.pozolotin at gmail.com (Andrei Pozolotin) Date: Mon, 22 Feb 2010 09:28:45 -0600 Subject: G1 is bad for me In-Reply-To: <4B7D7AF4.8010801@Sun.COM> References: <4B7D7159.5000808@gmail.com> <4B7D7AF4.8010801@Sun.COM> Message-ID: <4B82A2AD.5060805@gmail.com> Tony, Ramki, Jon, hello; 1) I take it that you silence means that "we are not there yet" with G1; understandable - thank you for your efforts! 2) there are some 82 people on this mail list who seem to be really interested in GC and G1 specifically; can I ask to you to make brief announcements here when JDK 7 weekly build brings some new G1 advances, so we can test it right away? Thank you; Andrei -------- Original Message -------- Subject: Re: G1 is bad for me From: Y. Srinivas Ramakrishna To: Andrei Pozolotin Cc: hotspot-gc-use at openjdk.java.net Date: Thu 18 Feb 2010 11:37:56 AM CST > Hi Andrei -- please use the latest hs17 JVM which you can get from JDK > 7. There have been > many improvements (and bug fixes) since the 6u18/hs16 that you are > using below. > Most importantly, please add -XX:-ReduceInitialCardMarks to work > around a regression > introduced in 6u18/hs16, as described in the 6u18 release notes. > As re optimal settings (these may be quite different from shapes that > helped with CMS > for example), i'll let others w/more (tuning and other) experience > w/G1 comment. > > -- ramki > > On 02/18/10 08:56, Andrei Pozolotin wrote: >> Ramki, Jon, hi; >> >> I read everyting I could find on G1: >> http://www.google.com/patents/about?id=j5SpAAAAEBAJ&dq=7,340,494 >> >> http://developers.sun.com/learning/javaoneonline/2008/pdf/TS-5419.pdf >> http://research.sun.com/jtech/pubs/04-g1-paper-ismm.pdf >> http://blogs.sun.com/sunabl/entry/the_new_g1_collector >> http://blogs.sun.com/jonthecollector/entry/our_collectors >> etc, etc, etc, >> >> got myself excited - looks great on paper! >> >> and I followed Albert's advice, while trying to mimic my CMS >> settings: >> http://blogs.sun.com/sunabl/entry/the_new_g1_collector >> >> -Xms3700m >> -Xmx3700m >> -XX:PermSize=50m >> -XX:MaxPermSize=50m >> -XX:+UnlockExperimentalVMOptions >> -XX:+UseG1GC >> -XX:G1YoungGenSize=900m >> -XX:SurvivorRatio=2 >> -XX:+G1ParallelRSetUpdatingEnabled >> -XX:+G1ParallelRSetScanningEnabled >> >> results: >> >> 1) pauses are 1000 milliseconds or more; >> >> 2) reproducible crashes after 3 minutes under load; >> >> logs attached; >> >> am I missing something obvious? >> like do I need to switch to older / specific JVM build? >> >> per Tony, this was already working fine for a neighbour of mine >> in 2008: >> >> http://developers.sun.com/learning/javaoneonline/2008/pdf/TS-5419.pdf >> >> Thank you; >> >> Andrei >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20100222/23bf3fef/attachment.html From Y.S.Ramakrishna at Sun.COM Mon Feb 22 07:45:00 2010 From: Y.S.Ramakrishna at Sun.COM (Y. Srinivas Ramakrishna) Date: Mon, 22 Feb 2010 07:45:00 -0800 Subject: G1 is bad for me In-Reply-To: <4B82A2AD.5060805@gmail.com> References: <4B7D7159.5000808@gmail.com> <4B7D7AF4.8010801@Sun.COM> <4B82A2AD.5060805@gmail.com> Message-ID: <4B82A67C.4030902@sun.com> Sorry, we haven't responded in depth because we are busy with other high priority stuff. Please follow up with your support contacts for urgent issues. If you subscribe to hotspot-gc-dev at openjdk.java.net you will get all G1 integration notifications; those then usually flow into the next JDK7 build in the subsequent week or two. regards. -- ramki Andrei Pozolotin wrote: > Tony, Ramki, Jon, hello; > > 1) I take it that you silence means that "we are not there yet" with G1; > understandable - thank you for your efforts! > > 2) there are some 82 people on this mail list who seem to be really > interested > in GC and G1 specifically; can I ask to you to make brief > announcements here > when JDK 7 weekly build brings some new G1 advances, so we can test > it right away? > > Thank you; > > Andrei > > > -------- Original Message -------- > Subject: Re: G1 is bad for me > From: Y. Srinivas Ramakrishna > To: Andrei Pozolotin > Cc: hotspot-gc-use at openjdk.java.net > Date: Thu 18 Feb 2010 11:37:56 AM CST >> Hi Andrei -- please use the latest hs17 JVM which you can get from JDK >> 7. There have been >> many improvements (and bug fixes) since the 6u18/hs16 that you are >> using below. >> Most importantly, please add -XX:-ReduceInitialCardMarks to work >> around a regression >> introduced in 6u18/hs16, as described in the 6u18 release notes. >> As re optimal settings (these may be quite different from shapes that >> helped with CMS >> for example), i'll let others w/more (tuning and other) experience >> w/G1 comment. >> >> -- ramki >> >> On 02/18/10 08:56, Andrei Pozolotin wrote: >>> Ramki, Jon, hi; >>> >>> I read everyting I could find on G1: >>> http://www.google.com/patents/about?id=j5SpAAAAEBAJ&dq=7,340,494 >>> >>> http://developers.sun.com/learning/javaoneonline/2008/pdf/TS-5419.pdf >>> http://research.sun.com/jtech/pubs/04-g1-paper-ismm.pdf >>> http://blogs.sun.com/sunabl/entry/the_new_g1_collector >>> http://blogs.sun.com/jonthecollector/entry/our_collectors >>> etc, etc, etc, >>> >>> got myself excited - looks great on paper! >>> >>> and I followed Albert's advice, while trying to mimic my CMS >>> settings: >>> http://blogs.sun.com/sunabl/entry/the_new_g1_collector >>> >>> -Xms3700m >>> -Xmx3700m >>> -XX:PermSize=50m >>> -XX:MaxPermSize=50m >>> -XX:+UnlockExperimentalVMOptions >>> -XX:+UseG1GC >>> -XX:G1YoungGenSize=900m >>> -XX:SurvivorRatio=2 >>> -XX:+G1ParallelRSetUpdatingEnabled >>> -XX:+G1ParallelRSetScanningEnabled >>> >>> results: >>> >>> 1) pauses are 1000 milliseconds or more; >>> >>> 2) reproducible crashes after 3 minutes under load; >>> >>> logs attached; >>> >>> am I missing something obvious? >>> like do I need to switch to older / specific JVM build? >>> >>> per Tony, this was already working fine for a neighbour of mine >>> in 2008: >>> >>> http://developers.sun.com/learning/javaoneonline/2008/pdf/TS-5419.pdf >>> >>> Thank you; >>> >>> Andrei >>> >>> >>> ------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> hotspot-gc-use mailing list >>> hotspot-gc-use at openjdk.java.net >>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> > > From andrei.pozolotin at gmail.com Mon Feb 22 07:53:09 2010 From: andrei.pozolotin at gmail.com (Andrei Pozolotin) Date: Mon, 22 Feb 2010 09:53:09 -0600 Subject: G1 is bad for me In-Reply-To: <4B82A67C.4030902@sun.com> References: <4B7D7159.5000808@gmail.com> <4B7D7AF4.8010801@Sun.COM> <4B82A2AD.5060805@gmail.com> <4B82A67C.4030902@sun.com> Message-ID: <4B82A865.8070501@gmail.com> Ramki, got it; thank you; Andrei -------- Original Message -------- Subject: Re: G1 is bad for me From: Y. Srinivas Ramakrishna To: Andrei Pozolotin Cc: hotspot-gc-use at openjdk.java.net Date: Mon 22 Feb 2010 09:45:00 AM CST > Sorry, we haven't responded in depth because we are busy with other > high priority stuff. Please follow up with your support > contacts for urgent issues. > > If you subscribe to hotspot-gc-dev at openjdk.java.net you will > get all G1 integration notifications; those then usually flow into > the next JDK7 build in the subsequent week or two. > > regards. > -- ramki > > Andrei Pozolotin wrote: >> Tony, Ramki, Jon, hello; >> >> 1) I take it that you silence means that "we are not there yet" >> with G1; >> understandable - thank you for your efforts! >> >> 2) there are some 82 people on this mail list who seem to be really >> interested >> in GC and G1 specifically; can I ask to you to make brief >> announcements here >> when JDK 7 weekly build brings some new G1 advances, so we can test >> it right away? >> >> Thank you; >> >> Andrei >> >> >> -------- Original Message -------- >> Subject: Re: G1 is bad for me >> From: Y. Srinivas Ramakrishna >> To: Andrei Pozolotin >> Cc: hotspot-gc-use at openjdk.java.net >> Date: Thu 18 Feb 2010 11:37:56 AM CST >>> Hi Andrei -- please use the latest hs17 JVM which you can get from JDK >>> 7. There have been >>> many improvements (and bug fixes) since the 6u18/hs16 that you are >>> using below. >>> Most importantly, please add -XX:-ReduceInitialCardMarks to work >>> around a regression >>> introduced in 6u18/hs16, as described in the 6u18 release notes. >>> As re optimal settings (these may be quite different from shapes that >>> helped with CMS >>> for example), i'll let others w/more (tuning and other) experience >>> w/G1 comment. >>> >>> -- ramki >>> >>> On 02/18/10 08:56, Andrei Pozolotin wrote: >>>> Ramki, Jon, hi; >>>> >>>> I read everyting I could find on G1: >>>> http://www.google.com/patents/about?id=j5SpAAAAEBAJ&dq=7,340,494 >>>> >>>> http://developers.sun.com/learning/javaoneonline/2008/pdf/TS-5419.pdf >>>> http://research.sun.com/jtech/pubs/04-g1-paper-ismm.pdf >>>> http://blogs.sun.com/sunabl/entry/the_new_g1_collector >>>> http://blogs.sun.com/jonthecollector/entry/our_collectors >>>> etc, etc, etc, >>>> >>>> got myself excited - looks great on paper! >>>> >>>> and I followed Albert's advice, while trying to mimic my CMS >>>> settings: >>>> http://blogs.sun.com/sunabl/entry/the_new_g1_collector >>>> >>>> -Xms3700m >>>> -Xmx3700m >>>> -XX:PermSize=50m >>>> -XX:MaxPermSize=50m >>>> -XX:+UnlockExperimentalVMOptions >>>> -XX:+UseG1GC >>>> -XX:G1YoungGenSize=900m >>>> -XX:SurvivorRatio=2 >>>> -XX:+G1ParallelRSetUpdatingEnabled >>>> -XX:+G1ParallelRSetScanningEnabled >>>> >>>> results: >>>> >>>> 1) pauses are 1000 milliseconds or more; >>>> >>>> 2) reproducible crashes after 3 minutes under load; >>>> >>>> logs attached; >>>> >>>> am I missing something obvious? >>>> like do I need to switch to older / specific JVM build? >>>> >>>> per Tony, this was already working fine for a neighbour of mine >>>> in 2008: >>>> >>>> http://developers.sun.com/learning/javaoneonline/2008/pdf/TS-5419.pdf >>>> >>>> Thank you; >>>> >>>> Andrei >>>> >>>> >>>> ------------------------------------------------------------------------ >>>> >>>> >>>> _______________________________________________ >>>> hotspot-gc-use mailing list >>>> hotspot-gc-use at openjdk.java.net >>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20100222/a756a77c/attachment.html From andrei.pozolotin at gmail.com Thu Feb 25 13:15:07 2010 From: andrei.pozolotin at gmail.com (Andrei Pozolotin) Date: Thu, 25 Feb 2010 15:15:07 -0600 Subject: Tenured Promotion Composition? Message-ID: <4B86E85B.2090809@gmail.com> Ramki, Jon, hello; 1) Can you please suggest correct way to obtain Tenured Promotion Composition? That is, for each ParNew collection event I would like to get a print out of a class histogram which shows, say top 10 classes responsible for 80% of promotion chunk, by memory footprint; 2) JVM TI does not seem to have this info: http://java.sun.com/javase/6/docs/technotes/guides/jvmti/ 3) JDK 6 command options seem not offer this either: http://www.md.pp.ru/~eu/jdk6options.html ??? Thank you, Andrei -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20100225/ec73eaef/attachment.html From peter.schuller at infidyne.com Sat Feb 27 10:48:10 2010 From: peter.schuller at infidyne.com (Peter Schuller) Date: Sat, 27 Feb 2010 19:48:10 +0100 Subject: g1: remembered set scan costs very high w/ LRU cache type of behavior Message-ID: <5a1151761002271048n798d8e54of9981e88e2b447a8@mail.gmail.com> Hello, I have a very simple test case specifically designed to stress GC:s (while being reasonably representative of some worst-case but plausible production use cases): http://github.com/scode/plrutest/blob/master/src/org/scode/lrutest/LruTest.java It implements a very simple LRU cached using a LinkedHashMap. It inserts entries in this hash map at a pre-defined target rate (100 000 per second). Each insert is key:ed on a random integer between 0 and CACHE_SIZE * 2; thus when the cache reaches steady state 50% of iterations are expected to generate old garbage. It is completely non-generational, and the intent was to see how well the concurrent marking and partially young evacuations handle large heaps. I have run it with various parameters, but the ones relevant to this post are: -verbose:gc -Xms20g \ -Xmx20g \ -XX:+PrintGCTimeStamps \ -XX:+PrintGCDetails \ -XX:+UnlockExperimentalVMOptions \ -XX:+UseG1GC \ -XX:MaxGCPauseMillis=10 \ -XX:GCPauseIntervalMillis=20 \ -XX:G1ConfidencePercent=100 \ -XX:+G1ParallelRSetUpdatingEnabled \ -XX:+G1ParallelRSetScanningEnabled \ -XX:+DisableExplicitGC \ It's being run on a 16 core 64 bit Linux machine with 36 GB of RAM. (I know about there being a bug that affects parallel rset scanning/updating; for the purpose of this test I don't care about that. The observed behavior of very high RS scan costs happens with non-parallel RS scanning as well, only the numbers of higher since it will be running on one core instead of 16.) However, I run into trouble with young evacuations before even reaching the point of generating actual garbage (i.e., before the cache is full). Up to roughly 1 gb of heap used (as reported by GC output), pauses are fairly reasonable, with about 10 ms being spent in the RS scan phase (with parallel rs scanning enabled). At roughly 1 gb heap size the RS scan cost as reported by the GC output, suddenly starts steadly increasing until it reaches at least hundreds of milliseconds with no sign of stopping. Here is some sample output at ~ 1.7 gb heap size: 103.128: [GC pause (young)^C, 0.19716900 secs] [Parallel Time: 193.8 ms] [Update RS (Start) (ms): 103128.7 103128.7 103128.7 103128.7 103128.8 103128.8 103128.8 103128.9 103128.9 103128.9 103129.0 103129.0 103129.1] [Update RS (ms): 1.1 1.2 1.0 0.8 1.2 1.2 1.1 1.0 1.1 1.1 0.9 0.1 0.0 Avg: 0.9, Min: 0.0, Max: 1.2] [Processed Buffers : 1 1 1 1 1 1 1 1 1 1 1 1 0 Sum: 12, Avg: 0, Min: 0, Max: 1] [Ext Root Scanning (ms): 0.2 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Avg: 0.0, Min: 0.0, Max: 0.2] [Mark Stack Scanning (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Avg: 0.0, Min: 0.0, Max: 0.0] [Scan-Only Scanning (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Avg: 0.0, Min: 0.0, Max: 0.0] [Scan-Only Regions : 0 0 0 0 0 0 0 0 0 0 0 0 0 Sum: 0, Avg: 0, Min: 0, Max: 0] [Scan RS (ms): 178.8 178.8 179.1 179.4 178.7 179.2 178.9 178.8 179.2 179.2 178.3 179.9 179.6 Avg: 179.1, Min: 178.3, Max: 179.9] [Object Copy (ms): 10.6 10.5 10.3 10.2 10.6 10.0 10.3 10.4 9.9 9.9 10.9 10.2 10.5 Avg: 10.3, Min: 9.9, Max: 10.9] [Termination (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Avg: 0.0, Min: 0.0, Max: 0.0] [Other: 3.4 ms] [Clear CT: 1.7 ms] [Other: 1.7 ms] [ 1722M->1720M(20480M)] [Times: user=2.44 sys=0.06, real=0.19 secs] Except the RS scan bit, it looks reasonable. Object copy seems to be landing very nicely at around my asked-for 10 ms. RS updating is limited as would be expected given that most writes to the cache would likely happen to old objects. Root/stack scanning is insignificant since it is a small program with a single thread. But RS scan is apparantly very expensive. While I can see how RS scanning would be a significant factor given that all objects created in the young generation (barring any internal to the LinkedHashMap) will be referenced by some pseudo-random object in non-young regions (internal LinkedHashMap structure), I would expect the cost of RS scanning per young evacuation to scale with the size of the young generation. Beyond the edge case of a very very small hash map, I expect the distribution of references into the young generation to be fairly widely spread across the internal structure of the hash map. I can see some possibilities: (1) Maybe the LInkedHashMap happens to be structured such that there is significant locality to the object referenced put into it, such that up to the 1 gb heap size the number of distinct dirty cards is fairly low. The increase in RS scan time I am seeing may simply be the cost converging on an eventual very high cost resulting from poor locality of writes to old regions. In this case, it may be that the remember set simply do not scale well enough period, with many well-spread writes to older generations pointing to young data. (2) Maybe I am right in my thinking that writes to older regions would already be well spread long before the 1gb heapsize, and there is something wrong with the RS scanning itself that causes costs to go up much more than they "should" given the work load. Is there a better explanation? In the case of (1) it seems to me that what I really want is to run G1 in non-generational mode since I would then expect it to evacuate useful regions that does in fact free data (thanks to the statistics gathered by the concurrent marking). In this case the cost of RS scanning, even if it may be high, would at least be a trade-off to actual progress in terms of freeing heap. RIght now, all this RS scan work is being spent on not making any progress. (What are the reasons for non-generational mode no longer being supported? Enabling it asserts later on so it is clearly not by accident. Is it because of the optimization to not to remember set writes for writes in the young generation, that it is not just a switch to turn it off? Are there other problems with running it non-generationally?) In the case of (2) perhaps this indicates a fixable problem with the GC. As an aside, the GC behavior during young generation eviction is, as expected, a lot better when I run it against an immutable LRU cache: http://github.com/scode/plrutest/blob/master/src/org/scode/plrutest/clojure/lrucache.clj (That is just the cache, not a runnable test.) -- / Peter Schuller From Jon.Masamitsu at Sun.COM Sat Feb 27 11:13:23 2010 From: Jon.Masamitsu at Sun.COM (Jon Masamitsu) Date: Sat, 27 Feb 2010 11:13:23 -0800 Subject: g1: remembered set scan costs very high w/ LRU cache type of behavior In-Reply-To: <5a1151761002271048n798d8e54of9981e88e2b447a8@mail.gmail.com> References: <5a1151761002271048n798d8e54of9981e88e2b447a8@mail.gmail.com> Message-ID: <4B896ED3.3040107@sun.com> Peter, Which version of the jdk are you using? A jdk 6.0 update? Or jdk 7.0 build? Thanks. Jon Peter Schuller wrote On 02/27/10 10:48,: >Hello, > >I have a very simple test case specifically designed to stress GC:s >(while being reasonably representative of some worst-case but >plausible production use cases): > > http://github.com/scode/plrutest/blob/master/src/org/scode/lrutest/LruTest.java > >It implements a very simple LRU cached using a LinkedHashMap. It >inserts entries in this hash map at a pre-defined target rate (100 000 >per second). Each insert is key:ed on a random integer between 0 and >CACHE_SIZE * 2; thus when the cache reaches steady state 50% of >iterations are expected to generate old garbage. > >It is completely non-generational, and the intent was to see how well >the concurrent marking and partially young evacuations handle large >heaps. > >I have run it with various parameters, but the ones relevant to this post are: > > -verbose:gc > -Xms20g \ > -Xmx20g \ > -XX:+PrintGCTimeStamps \ > -XX:+PrintGCDetails \ > -XX:+UnlockExperimentalVMOptions \ > -XX:+UseG1GC \ > -XX:MaxGCPauseMillis=10 \ > -XX:GCPauseIntervalMillis=20 \ > -XX:G1ConfidencePercent=100 \ > -XX:+G1ParallelRSetUpdatingEnabled \ > -XX:+G1ParallelRSetScanningEnabled \ > -XX:+DisableExplicitGC \ > >It's being run on a 16 core 64 bit Linux machine with 36 GB of RAM. > >(I know about there being a bug that affects parallel rset >scanning/updating; for the purpose of this test I don't care about >that. The observed behavior of very high RS scan costs happens with >non-parallel RS scanning as well, only the numbers of higher since it >will be running on one core instead of 16.) > >However, I run into trouble with young evacuations before even >reaching the point of generating actual garbage (i.e., before the >cache is full). Up to roughly 1 gb of heap used (as reported by GC >output), pauses are fairly reasonable, with about 10 ms being spent in >the RS scan phase (with parallel rs scanning enabled). At roughly 1 gb >heap size the RS scan cost as reported by the GC output, suddenly >starts steadly increasing until it reaches at least hundreds of >milliseconds with no sign of stopping. Here is some sample output at ~ >1.7 gb heap size: > >103.128: [GC pause (young)^C, 0.19716900 secs] > [Parallel Time: 193.8 ms] > [Update RS (Start) (ms): 103128.7 103128.7 103128.7 103128.7 > 103128.8 103128.8 103128.8 103128.9 103128.9 103128.9 103129.0 >103129.0 103129.1] > [Update RS (ms): 1.1 1.2 1.0 0.8 1.2 1.2 1.1 1.0 1.1 >1.1 0.9 0.1 0.0 > Avg: 0.9, Min: 0.0, Max: 1.2] > [Processed Buffers : 1 1 1 1 1 1 1 1 1 1 1 1 0 > Sum: 12, Avg: 0, Min: 0, Max: 1] > [Ext Root Scanning (ms): 0.2 0.1 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.0 0.0 0.0 0.0 > Avg: 0.0, Min: 0.0, Max: 0.2] > [Mark Stack Scanning (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 >0.0 0.0 0.0 0.0 0.0 0.0 > Avg: 0.0, Min: 0.0, Max: 0.0] > [Scan-Only Scanning (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 >0.0 0.0 0.0 0.0 0.0 0.0 > Avg: 0.0, Min: 0.0, Max: 0.0] > [Scan-Only Regions : 0 0 0 0 0 0 0 0 0 0 0 0 0 > Sum: 0, Avg: 0, Min: 0, Max: 0] > [Scan RS (ms): 178.8 178.8 179.1 179.4 178.7 179.2 178.9 >178.8 179.2 179.2 178.3 179.9 179.6 > Avg: 179.1, Min: 178.3, Max: 179.9] > [Object Copy (ms): 10.6 10.5 10.3 10.2 10.6 10.0 10.3 >10.4 9.9 9.9 10.9 10.2 10.5 > Avg: 10.3, Min: 9.9, Max: 10.9] > [Termination (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 >0.0 0.0 0.0 0.0 > Avg: 0.0, Min: 0.0, Max: 0.0] > [Other: 3.4 ms] > [Clear CT: 1.7 ms] > [Other: 1.7 ms] > [ 1722M->1720M(20480M)] > [Times: user=2.44 sys=0.06, real=0.19 secs] > >Except the RS scan bit, it looks reasonable. Object copy seems to be >landing very nicely at around my asked-for 10 ms. RS updating is >limited as would be expected given that most writes to the cache would >likely happen to old objects. Root/stack scanning is insignificant >since it is a small program with a single thread. > >But RS scan is apparantly very expensive. While I can see how RS >scanning would be a significant factor given that all objects created >in the young generation (barring any internal to the LinkedHashMap) >will be referenced by some pseudo-random object in non-young regions >(internal LinkedHashMap structure), I would expect the cost of RS >scanning per young evacuation to scale with the size of the young >generation. Beyond the edge case of a very very small hash map, I >expect the distribution of references into the young generation to be >fairly widely spread across the internal structure of the hash map. > >I can see some possibilities: > >(1) Maybe the LInkedHashMap happens to be structured such that there >is significant locality to the object referenced put into it, such >that up to the 1 gb heap size the number of distinct dirty cards is >fairly low. The increase in RS scan time I am seeing may simply be the >cost converging on an eventual very high cost resulting from poor >locality of writes to old regions. In this case, it may be that the >remember set simply do not scale well enough period, with many >well-spread writes to older generations pointing to young data. > >(2) Maybe I am right in my thinking that writes to older regions would >already be well spread long before the 1gb heapsize, and there is >something wrong with the RS scanning itself that causes costs to go up >much more than they "should" given the work load. > >Is there a better explanation? > >In the case of (1) it seems to me that what I really want is to run G1 >in non-generational mode since I would then expect it to evacuate >useful regions that does in fact free data (thanks to the statistics >gathered by the concurrent marking). In this case the cost of RS >scanning, even if it may be high, would at least be a trade-off to >actual progress in terms of freeing heap. RIght now, all this RS scan >work is being spent on not making any progress. > >(What are the reasons for non-generational mode no longer being >supported? Enabling it asserts later on so it is clearly not by >accident. Is it because of the optimization to not to remember set >writes for writes in the young generation, that it is not just a >switch to turn it off? Are there other problems with running it >non-generationally?) > >In the case of (2) perhaps this indicates a fixable problem with the GC. > >As an aside, the GC behavior during young generation eviction is, as >expected, a lot better when I run it against an immutable LRU cache: > > http://github.com/scode/plrutest/blob/master/src/org/scode/plrutest/clojure/lrucache.clj > >(That is just the cache, not a runnable test.) > > > From peter.schuller at infidyne.com Sat Feb 27 12:19:55 2010 From: peter.schuller at infidyne.com (Peter Schuller) Date: Sat, 27 Feb 2010 21:19:55 +0100 Subject: g1: remembered set scan costs very high w/ LRU cache type of behavior In-Reply-To: <4B896ED3.3040107@sun.com> References: <5a1151761002271048n798d8e54of9981e88e2b447a8@mail.gmail.com> <4B896ED3.3040107@sun.com> Message-ID: <5a1151761002271219j3f241f9aj6bdd5b2a724ceebe@mail.gmail.com> > Which version of the jdk are you using? ?A jdk 6.0 update? ?Or > jdk 7.0 build? Apologies, I meant to include that. I'm using: java version "1.7.0-ea" Java(TM) SE Runtime Environment (build 1.7.0-ea-b84) Java HotSpot(TM) 64-Bit Server VM (build 17.0-b09, mixed mode) -- / Peter Schuller From Jon.Masamitsu at Sun.COM Sat Feb 27 13:12:27 2010 From: Jon.Masamitsu at Sun.COM (Jon Masamitsu) Date: Sat, 27 Feb 2010 13:12:27 -0800 Subject: g1: remembered set scan costs very high w/ LRU cache type of behavior In-Reply-To: <5a1151761002271219j3f241f9aj6bdd5b2a724ceebe@mail.gmail.com> References: <5a1151761002271048n798d8e54of9981e88e2b447a8@mail.gmail.com> <4B896ED3.3040107@sun.com> <5a1151761002271219j3f241f9aj6bdd5b2a724ceebe@mail.gmail.com> Message-ID: <4B898ABB.4000409@sun.com> Peter Schuller wrote On 02/27/10 12:19,: >>Which version of the jdk are you using? ? A jdk 6.0 update? ? Or >>jdk 7.0 build? >> >> > >Apologies, I meant to include that. I'm using: > >java version "1.7.0-ea" >Java(TM) SE Runtime Environment (build 1.7.0-ea-b84) >Java HotSpot(TM) 64-Bit Server VM (build 17.0-b09, mixed mode) > > > Peter, The problem you're seeing may be addressed by some work in progress that is being done under CR 6923991: G1: improve scalability of RSet scanning I don't think that work has been released yet. If it hasn't, with your permission, we'll try your test program with it and see how well it does. By the way, thank you very much for giving G1 a look. Especially for the details in your mail. Jon From peter.schuller at infidyne.com Sat Feb 27 15:18:31 2010 From: peter.schuller at infidyne.com (Peter Schuller) Date: Sun, 28 Feb 2010 00:18:31 +0100 Subject: g1: remembered set scan costs very high w/ LRU cache type of behavior In-Reply-To: <4B898ABB.4000409@sun.com> References: <5a1151761002271048n798d8e54of9981e88e2b447a8@mail.gmail.com> <4B896ED3.3040107@sun.com> <5a1151761002271219j3f241f9aj6bdd5b2a724ceebe@mail.gmail.com> <4B898ABB.4000409@sun.com> Message-ID: <5a1151761002271518q40d95865o311699ef66764d32@mail.gmail.com> > The problem you're seeing may be addressed by some > work in progress that is being done under CR > > 6923991: G1: improve scalability of RSet scanning > > I don't think that work has been released yet. Well, if this commit constitutes the work: http://hg.openjdk.java.net/jdk7/hotspot/hotspot/rev/0414c1049f15 Then I suspect I have it, because the JDK comes from: jdk-7-ea-bin-b84-linux-x64-18_feb_2010.bin Which is 7 days after the commit happened. I'll see about building from source to maks sure. However, reading about it my initial impression that it is probably not the same issue as it seems to be about many-core scalability of the RS scanning. In my case I see poor performance even with a single core, and to the extent it is a scaling issue I get the impression it is a function of the number of RS tracked writes rather than the number of CPU cores. > If it hasn't, > with your permission, we'll try your test program with > it and see how well it does. Of course. Btw, I discovered something else. I have previously been doing some testing on JDK:s build from the jdk 1.7 bsdport on FreeBSD and had stability issues where the JVM would suddenly exit with no message when I ran my clojure persistent LRU test with G1. I just hit this behavior on the Linux host earlier today, with the snapshot binary mentioned above, which means I can no longer chalk it up to a FreeBSD specific issue. I doubt I can do much to narrow it down to a very small test case, but I could tidy up the test case a bit and make that available, and perhaps produce an executable .jar with clojure + the test case if that helps. I suspect (based on speculation rather than evidence) that the allocation and write patterns being generated by the use of persistent/immutable data structures is key to triggering it. That is, unless there is an actual System.exit() somewhere in the codebase... but I highly doubt there would be code that does this as a function of some behavior which depends on the garbage collector choice. -- / Peter Schuller From peter.schuller at infidyne.com Sat Feb 27 16:17:18 2010 From: peter.schuller at infidyne.com (Peter Schuller) Date: Sun, 28 Feb 2010 01:17:18 +0100 Subject: g1: remembered set scan costs very high w/ LRU cache type of behavior In-Reply-To: <5a1151761002271518q40d95865o311699ef66764d32@mail.gmail.com> References: <5a1151761002271048n798d8e54of9981e88e2b447a8@mail.gmail.com> <4B896ED3.3040107@sun.com> <5a1151761002271219j3f241f9aj6bdd5b2a724ceebe@mail.gmail.com> <4B898ABB.4000409@sun.com> <5a1151761002271518q40d95865o311699ef66764d32@mail.gmail.com> Message-ID: <5a1151761002271617u38322a93u18cddb88407b381c@mail.gmail.com> > I'll see about building from source to maks sure. I see the same behavior (RS scan times begin growing at ~ 1 gb heap) with a JDK7 built from sources cloned from hg just now (an hour or so ago). -- / Peter Schuller From Jon.Masamitsu at Sun.COM Sat Feb 27 18:46:10 2010 From: Jon.Masamitsu at Sun.COM (Jon Masamitsu) Date: Sat, 27 Feb 2010 18:46:10 -0800 Subject: g1: remembered set scan costs very high w/ LRU cache type of behavior In-Reply-To: <5a1151761002271617u38322a93u18cddb88407b381c@mail.gmail.com> References: <5a1151761002271048n798d8e54of9981e88e2b447a8@mail.gmail.com> <4B896ED3.3040107@sun.com> <5a1151761002271219j3f241f9aj6bdd5b2a724ceebe@mail.gmail.com> <4B898ABB.4000409@sun.com> <5a1151761002271518q40d95865o311699ef66764d32@mail.gmail.com> <5a1151761002271617u38322a93u18cddb88407b381c@mail.gmail.com> Message-ID: <4B89D8F2.5040208@sun.com> Peter Schuller wrote On 02/27/10 16:17,: >>I'll see about building from source to maks sure. >> >> > >I see the same behavior (RS scan times begin growing at ~ 1 gb heap) >with a JDK7 built from sources cloned from hg just now (an hour or so >ago). > > > Thanks for doing the experiment. We'll have to take a closer look.