From y.s.ramakrishna at sun.com Tue Feb 2 04:51:26 2010 From: y.s.ramakrishna at sun.com (y.s.ramakrishna at sun.com) Date: Tue, 02 Feb 2010 04:51:26 +0000 Subject: hg: jdk7/hotspot-gc/hotspot: 6904516: More object array barrier fixes, following up on 6906727 Message-ID: <20100202045138.28F5241A57@hg.openjdk.java.net> Changeset: 6484c4ee11cb Author: ysr Date: 2010-02-01 17:29 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/6484c4ee11cb 6904516: More object array barrier fixes, following up on 6906727 Summary: Fixed missing pre-barrier calls for G1, modified C1 to call pre- and correct post-barrier interfaces, deleted obsolete interface, (temporarily) disabled redundant deferred barrier in BacktraceBuilder. Reviewed-by: coleenp, jmasa, kvn, never ! src/share/vm/c1/c1_Runtime1.cpp ! src/share/vm/classfile/javaClasses.cpp ! src/share/vm/gc_implementation/g1/g1CollectedHeap.cpp ! src/share/vm/memory/barrierSet.hpp ! src/share/vm/memory/barrierSet.inline.hpp ! src/share/vm/runtime/stubRoutines.cpp From Y.S.Ramakrishna at Sun.COM Tue Feb 2 05:45:05 2010 From: Y.S.Ramakrishna at Sun.COM (Y. Srinivas Ramakrishna) Date: Mon, 01 Feb 2010 21:45:05 -0800 Subject: hg: jdk7/hotspot-gc/hotspot: 6904516: More object array barrier fixes, following up on 6906727 In-Reply-To: <20100202045138.28F5241A57@hg.openjdk.java.net> References: <20100202045138.28F5241A57@hg.openjdk.java.net> Message-ID: <4B67BBE1.40600@sun.com> This changeset was also reviewed by John Cuthbertson (johnc). -- ramki Y.S.Ramakrishna at Sun.COM wrote: > Changeset: 6484c4ee11cb > Author: ysr > Date: 2010-02-01 17:29 -0800 > URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/6484c4ee11cb > > 6904516: More object array barrier fixes, following up on 6906727 > Summary: Fixed missing pre-barrier calls for G1, modified C1 to call pre- and correct post-barrier interfaces, deleted obsolete interface, (temporarily) disabled redundant deferred barrier in BacktraceBuilder. > Reviewed-by: coleenp, jmasa, kvn, never > > ! src/share/vm/c1/c1_Runtime1.cpp > ! src/share/vm/classfile/javaClasses.cpp > ! src/share/vm/gc_implementation/g1/g1CollectedHeap.cpp > ! src/share/vm/memory/barrierSet.hpp > ! src/share/vm/memory/barrierSet.inline.hpp > ! src/share/vm/runtime/stubRoutines.cpp > From Y.S.Ramakrishna at Sun.COM Wed Feb 3 23:58:26 2010 From: Y.S.Ramakrishna at Sun.COM (Y. Srinivas Ramakrishna) Date: Wed, 03 Feb 2010 15:58:26 -0800 Subject: Fully disable GC - no pauses - run with massive heap until outofmemory In-Reply-To: <2E97E78D7F99D64DA5108FE2E9F0E828070EC6F1@LONEV3201.EUROPE.NOM> References: <2E97E78D7F99D64DA5108FE2E9F0E828070EC6F1@LONEV3201.EUROPE.NOM> Message-ID: <4B6A0DA2.7090202@Sun.COM> Hello Anthony -- On 02/03/10 14:43, anthony.warden at nomura.com wrote: > How best to do this please? > > Command line options / hack the gc thread etc. Shape the heap so as to give all (or most) of it to the young gen, use no survivor spaces. For example, say you want a 32 G heap. Let's give 4 M to the old gen may be. So you could do:- -Xmn32g -Xmx(32*1024+4)m -Xms(32*1204+4)m -XX:SurvivorRatio=1g (of course you use the computed values of the expressions in parentheses above.) Unfortunately, this will not give you an OOM when the heap gets full. It'll trigger a scavenge. If you want to give up the ghost at the first scavenge you can (if you are running on Solaris for example) write a Dtrace script to detect the invocation of a GC and kill the JVM process. Don't know if that's the behaviour you seek here. -- ramki > > > > --------------------------------- > > > > This e-mail (including any attachments) is confidential, may contain > proprietary or privileged information and is intended for the named > recipient(s) only. Unintended recipients are prohibited from taking action > on the basis of information in this e-mail and must delete all copies. > Nomura will not accept responsibility or liability for the accuracy or > completeness of, or the presence of any virus or disabling code in, this > e-mail. If verification is sought please request a hard copy. Any reference > to the terms of executed transactions should be treated as preliminary only > and subject to formal written confirmation by Nomura. Nomura reserves the > right to monitor e-mail communications through its networks (in accordance > with applicable laws). No confidentiality or privilege is waived or lost by > Nomura by any mistransmission of this e-mail. Any reference to "Nomura" is > a reference to any entity in the Nomura Holdings, Inc. group. Please read > our Electronic Communications Legal Notice which forms part of this e-mail: > http://www.Nomura.com/email_disclaimer.htm > > > ------------------------------------------------------------------------ > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From Y.S.Ramakrishna at Sun.COM Thu Feb 4 00:03:52 2010 From: Y.S.Ramakrishna at Sun.COM (Y. Srinivas Ramakrishna) Date: Wed, 03 Feb 2010 16:03:52 -0800 Subject: Fully disable GC - no pauses - run with massive heap until outofmemory In-Reply-To: <4B6A0DA2.7090202@Sun.COM> References: <2E97E78D7F99D64DA5108FE2E9F0E828070EC6F1@LONEV3201.EUROPE.NOM> <4B6A0DA2.7090202@Sun.COM> Message-ID: <4B6A0EE8.2080006@Sun.COM> On 02/03/10 15:58, Y. Srinivas Ramakrishna wrote: > Hello Anthony -- > > On 02/03/10 14:43, anthony.warden at nomura.com wrote: >> How best to do this please? >> >> Command line options / hack the gc thread etc. > > Shape the heap so as to give all (or most) of it to the young gen, > use no survivor spaces. For example, say you want a 32 G heap. > Let's give 4 M to the old gen may be. So you could do:- > > -Xmn32g -Xmx(32*1024+4)m -Xms(32*1204+4)m -XX:SurvivorRatio=1g i meant 1024 instead of 1204 above. Sorry for the typo. -- ramki > > (of course you use the computed values of the expressions in parentheses above.) > > Unfortunately, this will not give you an OOM when the heap gets > full. It'll trigger a scavenge. If you want to give up the ghost at > the first scavenge you can (if you are running on Solaris for example) > write a Dtrace script to detect the invocation of a GC and kill > the JVM process. Don't know if that's the behaviour you seek here. > > -- ramki > >> >> >> --------------------------------- >> >> >> >> This e-mail (including any attachments) is confidential, may contain >> proprietary or privileged information and is intended for the named >> recipient(s) only. Unintended recipients are prohibited from taking action >> on the basis of information in this e-mail and must delete all copies. >> Nomura will not accept responsibility or liability for the accuracy or >> completeness of, or the presence of any virus or disabling code in, this >> e-mail. If verification is sought please request a hard copy. Any reference >> to the terms of executed transactions should be treated as preliminary only >> and subject to formal written confirmation by Nomura. Nomura reserves the >> right to monitor e-mail communications through its networks (in accordance >> with applicable laws). No confidentiality or privilege is waived or lost by >> Nomura by any mistransmission of this e-mail. Any reference to "Nomura" is >> a reference to any entity in the Nomura Holdings, Inc. group. Please read >> our Electronic Communications Legal Notice which forms part of this e-mail: >> http://www.Nomura.com/email_disclaimer.htm >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From tony.printezis at sun.com Thu Feb 4 00:07:11 2010 From: tony.printezis at sun.com (Tony Printezis) Date: Wed, 03 Feb 2010 19:07:11 -0500 Subject: Fully disable GC - no pauses - run with massive heap until outofmemory In-Reply-To: <4B6A0DA2.7090202@Sun.COM> References: <2E97E78D7F99D64DA5108FE2E9F0E828070EC6F1@LONEV3201.EUROPE.NOM> <4B6A0DA2.7090202@Sun.COM> Message-ID: <4B6A0FAF.50500@sun.com> Hi Anthony, I hope you are well! An alternative is to use an MBean and monitor the occupancy of the eden. When that goes over a certain threshold, you can somehow kill the JVM. In fact, you can be a bit more graceful this way, since you can notice early-ish that the JVM will run out of memory a bit into the future and shutdown all processing gracefully (i.e., let the current Tx complete), while redirecting new requests to a new JVM. Are you going to have any long-lived data in the JVM? Maybe, size the old generation large enough to accommodate it and after initialization do a System.gc() to move all that into the old generation. That way, you'll start processing with a totally empty eden. GC group: should we really be encouraging this? :-) Tony, HS GC Group Y. Srinivas Ramakrishna wrote: > Hello Anthony -- > > On 02/03/10 14:43, anthony.warden at nomura.com wrote: > >> How best to do this please? >> >> Command line options / hack the gc thread etc. >> > > Shape the heap so as to give all (or most) of it to the young gen, > use no survivor spaces. For example, say you want a 32 G heap. > Let's give 4 M to the old gen may be. So you could do:- > > -Xmn32g -Xmx(32*1024+4)m -Xms(32*1204+4)m -XX:SurvivorRatio=1g > > (of course you use the computed values of the expressions in parentheses above.) > > Unfortunately, this will not give you an OOM when the heap gets > full. It'll trigger a scavenge. If you want to give up the ghost at > the first scavenge you can (if you are running on Solaris for example) > write a Dtrace script to detect the invocation of a GC and kill > the JVM process. Don't know if that's the behaviour you seek here. > > -- ramki > > >> >> --------------------------------- >> >> >> >> This e-mail (including any attachments) is confidential, may contain >> proprietary or privileged information and is intended for the named >> recipient(s) only. Unintended recipients are prohibited from taking action >> on the basis of information in this e-mail and must delete all copies. >> Nomura will not accept responsibility or liability for the accuracy or >> completeness of, or the presence of any virus or disabling code in, this >> e-mail. If verification is sought please request a hard copy. Any reference >> to the terms of executed transactions should be treated as preliminary only >> and subject to formal written confirmation by Nomura. Nomura reserves the >> right to monitor e-mail communications through its networks (in accordance >> with applicable laws). No confidentiality or privilege is waived or lost by >> Nomura by any mistransmission of this e-mail. Any reference to "Nomura" is >> a reference to any entity in the Nomura Holdings, Inc. group. Please read >> our Electronic Communications Legal Notice which forms part of this e-mail: >> http://www.Nomura.com/email_disclaimer.htm >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From john.cuthbertson at sun.com Thu Feb 4 00:11:00 2010 From: john.cuthbertson at sun.com (john.cuthbertson at sun.com) Date: Thu, 04 Feb 2010 00:11:00 +0000 Subject: hg: jdk7/hotspot-gc/hotspot: 6914402: G1: assert(!is_young_card(cached_ptr), "shouldn't get a card in young region") Message-ID: <20100204001121.AAD8D41CE6@hg.openjdk.java.net> Changeset: deada8912c54 Author: johnc Date: 2010-02-02 18:39 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/deada8912c54 6914402: G1: assert(!is_young_card(cached_ptr),"shouldn't get a card in young region") Summary: Invalid assert. Filter cards evicted from the card count cache instead. Reviewed-by: apetrusenko, tonyp ! src/share/vm/gc_implementation/g1/concurrentG1Refine.cpp From Y.S.Ramakrishna at Sun.COM Thu Feb 4 00:27:52 2010 From: Y.S.Ramakrishna at Sun.COM (Y. Srinivas Ramakrishna) Date: Wed, 03 Feb 2010 16:27:52 -0800 Subject: Fully disable GC - no pauses - run with massive heap until outofmemory In-Reply-To: <4B6A0FAF.50500@sun.com> References: <2E97E78D7F99D64DA5108FE2E9F0E828070EC6F1@LONEV3201.EUROPE.NOM> <4B6A0DA2.7090202@Sun.COM> <4B6A0FAF.50500@sun.com> Message-ID: <4B6A1488.7070801@Sun.COM> On 02/03/10 16:07, Tony Printezis wrote: > Hi Anthony, > > I hope you are well! > > An alternative is to use an MBean and monitor the occupancy of the eden. > When that goes over a certain threshold, you can somehow kill the JVM. Is there a threshold monitor that will look at generation occupancy _before_ a GC which is what you'd want here, right? I was always under the impression these detected occupancy after a GC and only then would sound the alarms; but perhaps there are options that specify precise behaviour in this regard. Mandy, cc'd, would know for sure. (Don't know where the spec is.) -- ramki > In fact, you can be a bit more graceful this way, since you can notice > early-ish that the JVM will run out of memory a bit into the future and > shutdown all processing gracefully (i.e., let the current Tx complete), > while redirecting new requests to a new JVM. > > Are you going to have any long-lived data in the JVM? Maybe, size the > old generation large enough to accommodate it and after initialization > do a System.gc() to move all that into the old generation. That way, > you'll start processing with a totally empty eden. > > GC group: should we really be encouraging this? :-) > > Tony, HS GC Group > > Y. Srinivas Ramakrishna wrote: >> Hello Anthony -- >> >> On 02/03/10 14:43, anthony.warden at nomura.com wrote: >> >>> How best to do this please? >>> >>> Command line options / hack the gc thread etc. >>> >> Shape the heap so as to give all (or most) of it to the young gen, >> use no survivor spaces. For example, say you want a 32 G heap. >> Let's give 4 M to the old gen may be. So you could do:- >> >> -Xmn32g -Xmx(32*1024+4)m -Xms(32*1204+4)m -XX:SurvivorRatio=1g >> >> (of course you use the computed values of the expressions in parentheses above.) >> >> Unfortunately, this will not give you an OOM when the heap gets >> full. It'll trigger a scavenge. If you want to give up the ghost at >> the first scavenge you can (if you are running on Solaris for example) >> write a Dtrace script to detect the invocation of a GC and kill >> the JVM process. Don't know if that's the behaviour you seek here. >> >> -- ramki >> >> >>> --------------------------------- >>> >>> >>> >>> This e-mail (including any attachments) is confidential, may contain >>> proprietary or privileged information and is intended for the named >>> recipient(s) only. Unintended recipients are prohibited from taking action >>> on the basis of information in this e-mail and must delete all copies. >>> Nomura will not accept responsibility or liability for the accuracy or >>> completeness of, or the presence of any virus or disabling code in, this >>> e-mail. If verification is sought please request a hard copy. Any reference >>> to the terms of executed transactions should be treated as preliminary only >>> and subject to formal written confirmation by Nomura. Nomura reserves the >>> right to monitor e-mail communications through its networks (in accordance >>> with applicable laws). No confidentiality or privilege is waived or lost by >>> Nomura by any mistransmission of this e-mail. Any reference to "Nomura" is >>> a reference to any entity in the Nomura Holdings, Inc. group. Please read >>> our Electronic Communications Legal Notice which forms part of this e-mail: >>> http://www.Nomura.com/email_disclaimer.htm >>> >>> >>> ------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> hotspot-gc-use mailing list >>> hotspot-gc-use at openjdk.java.net >>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From tony.printezis at sun.com Thu Feb 4 06:57:05 2010 From: tony.printezis at sun.com (Tony Printezis) Date: Thu, 04 Feb 2010 01:57:05 -0500 Subject: Fully disable GC - no pauses - run with massive heap until outofmemory In-Reply-To: <4B6A1488.7070801@Sun.COM> References: <2E97E78D7F99D64DA5108FE2E9F0E828070EC6F1@LONEV3201.EUROPE.NOM> <4B6A0DA2.7090202@Sun.COM> <4B6A0FAF.50500@sun.com> <4B6A1488.7070801@Sun.COM> Message-ID: <4B6A6FC1.4000702@sun.com> Ramki, Yes, you can set thresholds and get notifications, but I think you're right; said thresholds are only checked before / after a GC. Tony Y. Srinivas Ramakrishna wrote: > On 02/03/10 16:07, Tony Printezis wrote: > >> Hi Anthony, >> >> I hope you are well! >> >> An alternative is to use an MBean and monitor the occupancy of the eden. >> When that goes over a certain threshold, you can somehow kill the JVM. >> > > Is there a threshold monitor that will look at generation occupancy > _before_ a GC which is what you'd want here, right? I was always > under the impression these detected occupancy after a GC and > only then would sound the alarms; but perhaps there are options > that specify precise behaviour in this regard. > > Mandy, cc'd, would know for sure. (Don't know where the spec is.) > -- ramki > > > >> In fact, you can be a bit more graceful this way, since you can notice >> early-ish that the JVM will run out of memory a bit into the future and >> shutdown all processing gracefully (i.e., let the current Tx complete), >> while redirecting new requests to a new JVM. >> >> Are you going to have any long-lived data in the JVM? Maybe, size the >> old generation large enough to accommodate it and after initialization >> do a System.gc() to move all that into the old generation. That way, >> you'll start processing with a totally empty eden. >> >> GC group: should we really be encouraging this? :-) >> >> Tony, HS GC Group >> >> Y. Srinivas Ramakrishna wrote: >> >>> Hello Anthony -- >>> >>> On 02/03/10 14:43, anthony.warden at nomura.com wrote: >>> >>> >>>> How best to do this please? >>>> >>>> Command line options / hack the gc thread etc. >>>> >>>> >>> Shape the heap so as to give all (or most) of it to the young gen, >>> use no survivor spaces. For example, say you want a 32 G heap. >>> Let's give 4 M to the old gen may be. So you could do:- >>> >>> -Xmn32g -Xmx(32*1024+4)m -Xms(32*1204+4)m -XX:SurvivorRatio=1g >>> >>> (of course you use the computed values of the expressions in parentheses above.) >>> >>> Unfortunately, this will not give you an OOM when the heap gets >>> full. It'll trigger a scavenge. If you want to give up the ghost at >>> the first scavenge you can (if you are running on Solaris for example) >>> write a Dtrace script to detect the invocation of a GC and kill >>> the JVM process. Don't know if that's the behaviour you seek here. >>> >>> -- ramki >>> >>> >>> >>>> --------------------------------- >>>> >>>> >>>> >>>> This e-mail (including any attachments) is confidential, may contain >>>> proprietary or privileged information and is intended for the named >>>> recipient(s) only. Unintended recipients are prohibited from taking action >>>> on the basis of information in this e-mail and must delete all copies. >>>> Nomura will not accept responsibility or liability for the accuracy or >>>> completeness of, or the presence of any virus or disabling code in, this >>>> e-mail. If verification is sought please request a hard copy. Any reference >>>> to the terms of executed transactions should be treated as preliminary only >>>> and subject to formal written confirmation by Nomura. Nomura reserves the >>>> right to monitor e-mail communications through its networks (in accordance >>>> with applicable laws). No confidentiality or privilege is waived or lost by >>>> Nomura by any mistransmission of this e-mail. Any reference to "Nomura" is >>>> a reference to any entity in the Nomura Holdings, Inc. group. Please read >>>> our Electronic Communications Legal Notice which forms part of this e-mail: >>>> http://www.Nomura.com/email_disclaimer.htm >>>> >>>> >>>> ------------------------------------------------------------------------ >>>> >>>> _______________________________________________ >>>> hotspot-gc-use mailing list >>>> hotspot-gc-use at openjdk.java.net >>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>>> >>>> >>> _______________________________________________ >>> hotspot-gc-use mailing list >>> hotspot-gc-use at openjdk.java.net >>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>> >>> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From atulksh at hotmail.com Thu Feb 4 20:37:35 2010 From: atulksh at hotmail.com (atulksh) Date: Thu, 4 Feb 2010 12:37:35 -0800 (PST) Subject: long unaccounted pause during concurrent mark phase of ParNew cycle Message-ID: <27459365.post@talk.nabble.com> Hi, We have an application that is freezing (pausing) for a very long (tens of seconds) time during the concurrent mark phase of the ParNew cycle. This pause is consistently seen when there is good amount of activity on the server. We tried to upgrade the JVM (JRE 6) from Update 02 to Update 17 keeping all the parameters same, but the pauses did not go away (they became more consistent and slightly longer). We even tried to enable CompressedOops option, thinking that that would reduce memory requirements, but that has not helped. We are running on 64bit SUSE 10.1 platform with 32Gb of RAM and 8 core machine. 26g is allocated to the java process. The full set of VM parameters are here -Xmx26g -Xms26g -server -XX:+UseConcMarkSweepGC -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:CMSInitiatingOccupancyFraction=70 -XX:-CMSParallelRemarkEnabled The nature of the pause is like this. All the application threads that are being processed seem to stop responding. New requests that come to our app server are not processed and get queued. When the pause ends then everything frees up and seems to go on normally henceforth. In the GC you see a gap between the ParNews. The following are two examples of the pauses. Notice the pause from 10:13:06 to 10:13:35 (29 seconds) in the first one and 11:29:04 to 11:29:17 (13 seconds) in the second one. These pauses only occur in the concurrent mark phase. They do not seem to occur at any other time or anywhere else in the CMS cycle. .... Thu Feb 4 10:12:59 2010: [GC [1 CMS-initial-mark: 18976170K(27092608K)] 18989786K(27245952K), 0.0059140 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] Thu Feb 4 10:12:59 2010: [CMS-concurrent-mark-start] Thu Feb 4 10:12:59 2010 [18,980,009 K]: [GC 42035.293: [ParNew Desired survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: 1833064 bytes, 1833064 total - age 2: 18472 bytes, 1851536 total : 148315K->3839K(153344K), 0.0400230 secs] 19124486K->18980009K(27245952K), 0.0401580 secs] [Times: user=0.25 sys=0.00, real=0.04 secs] Thu Feb 4 10:12:59 2010 [18,979,036 K]: [GC 42035.415: [ParNew Desired survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: 2068376 bytes, 2068376 total - age 2: 23792 bytes, 2092168 total - age 3: 18472 bytes, 2110640 total : 140159K->2866K(153344K), 0.0352200 secs] 19116329K->18979036K(27245952K), 0.0353520 secs] [Times: user=0.29 sys=0.00, real=0.03 secs] Thu Feb 4 10:12:59 2010 [18,979,103 K]: [GC 42035.535: [ParNew Desired survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: 1782608 bytes, 1782608 total - age 2: 22032 bytes, 1804640 total - age 3: 23792 bytes, 1828432 total - age 4: 18472 bytes, 1846904 total : 139186K->2933K(153344K), 0.0498480 secs] 19115356K->18979103K(27245952K), 0.0499850 secs] [Times: user=0.24 sys=0.00, real=0.05 secs] Thu Feb 4 10:12:59 2010 [18,979,181 K]: [GC 42035.673: [ParNew Desired survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: 2161000 bytes, 2161000 total - age 2: 22032 bytes, 2183032 total - age 3: 22032 bytes, 2205064 total - age 4: 23792 bytes, 2228856 total : 139253K->2992K(153344K), 0.0400420 secs] 19115423K->18979181K(27245952K), 0.0401730 secs] [Times: user=0.26 sys=0.00, real=0.04 secs] Thu Feb 4 10:13:06 2010 [18,979,876 K]: [GC 42042.243: [ParNew Desired survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: 2113272 bytes, 2113272 total - age 2: 23792 bytes, 2137064 total - age 3: 22032 bytes, 2159096 total - age 4: 22032 bytes, 2181128 total : 138855K->3662K(153344K), 0.0465500 secs] 19115044K->18979876K(27245952K), 0.0466840 secs] [Times: user=0.27 sys=0.00, real=0.04 secs] Thu Feb 4 10:13:35 2010 [18,979,051 K]: [GC 42071.809: [ParNew Desired survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: 219056 bytes, 219056 total - age 2: 1867632 bytes, 2086688 total - age 3: 23792 bytes, 2110480 total - age 4: 22032 bytes, 2132512 total : 139982K->2814K(153344K), 0.0426920 secs] 19116196K->18979051K(27245952K), 0.0428200 secs] [Times: user=0.28 sys=0.00, real=0.05 secs] Thu Feb 4 10:13:35 2010: [CMS-concurrent-mark: 36.272/36.715 secs] [Times: user=56.34 sys=0.34, real=36.71 secs] .... .... Thu Feb 4 11:28:59 2010: [GC [1 CMS-initial-mark: 19012252K(27092608K)] 19032719K(27245952K), 0.0279440 secs] [Times: user=0.02 sys=0.00, real=0.03 secs] Thu Feb 4 11:28:59 2010: [CMS-concurrent-mark-start] Thu Feb 4 11:28:59 2010 [19,154,661 K]: [GC 46595.614: [ParNew Desired survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: 17431976 bytes, 17431976 total : 153344K->17024K(153344K), 0.9259390 secs] 19165596K->19154661K(27245952K), 0.9260890 secs] [Times: user=1.23 sys=0.05, real=0.93 secs] Thu Feb 4 11:29:00 2010 [19,199,696 K]: [GC 46596.973: [ParNew Desired survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: 17448432 bytes, 17448432 total : 153344K->17024K(153344K), 0.3054460 secs] 19290981K->19199696K(27245952K), 0.3055930 secs] [Times: user=0.79 sys=0.07, real=0.31 secs] Thu Feb 4 11:29:01 2010 [19,229,358 K]: [GC 46597.722: [ParNew Desired survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: 10331408 bytes, 10331408 total : 153344K->17024K(153344K), 0.1449040 secs] 19336016K->19229358K(27245952K), 0.1450630 secs] [Times: user=0.59 sys=0.02, real=0.15 secs] Thu Feb 4 11:29:02 2010 [19,240,015 K]: [GC 46598.038: [ParNew Desired survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: 17354120 bytes, 17354120 total : 153344K->17024K(153344K), 0.0935440 secs] 19365678K->19240015K(27245952K), 0.0936870 secs] [Times: user=0.59 sys=0.00, real=0.10 secs] Thu Feb 4 11:29:02 2010 [19,302,509 K]: [GC 46598.347: [ParNew Desired survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: 17432360 bytes, 17432360 total : 151519K->17024K(153344K), 0.4619430 secs] 19374510K->19302509K(27245952K), 0.4620890 secs] [Times: user=1.19 sys=0.02, real=0.46 secs] Thu Feb 4 11:29:03 2010 [19,329,594 K]: [GC 46599.030: [ParNew Desired survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: 11933296 bytes, 11933296 total : 153344K->17024K(153344K), 0.1492670 secs] 19438829K->19329594K(27245952K), 0.1494200 secs] [Times: user=0.77 sys=0.03, real=0.15 secs] Thu Feb 4 11:29:03 2010 [19,337,176 K]: [GC 46599.423: [ParNew Desired survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: 5719032 bytes, 5719032 total : 153344K->12852K(153344K), 0.0675180 secs] 19465914K->19337176K(27245952K), 0.0676670 secs] [Times: user=0.46 sys=0.02, real=0.07 secs] Thu Feb 4 11:29:03 2010 [19,338,256 K]: [GC 46599.754: [ParNew Desired survivor size 8716288 bytes, new threshold 2 (max 4) - age 1: 6783616 bytes, 6783616 total - age 2: 5570080 bytes, 12353696 total : 149172K->13931K(153344K), 0.0650520 secs] 19473496K->19338256K(27245952K), 0.0652060 secs] [Times: user=0.42 sys=0.01, real=0.07 secs] Thu Feb 4 11:29:04 2010 [19,340,264 K]: [GC 46600.144: [ParNew Desired survivor size 8716288 bytes, new threshold 2 (max 4) - age 1: 2190136 bytes, 2190136 total - age 2: 6762648 bytes, 8952784 total : 150251K->10375K(153344K), 0.0709470 secs] 19474576K->19340264K(27245952K), 0.0710920 secs] [Times: user=0.48 sys=0.01, real=0.07 secs] Thu Feb 4 11:29:17 2010 [19,345,928 K]: [GC 46613.628: [ParNew Desired survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: 760488 bytes, 760488 total - age 2: 2173472 bytes, 2933960 total : 146695K->9279K(153344K), 0.0689390 secs] 19476584K->19345928K(27245952K), 0.0691070 secs] [Times: user=0.45 sys=0.00, real=0.07 secs] Thu Feb 4 11:29:17 2010: [CMS-concurrent-mark: 15.853/18.260 secs] [Times: user=29.26 sys=0.55, real=18.26 secs] Thu Feb 4 11:29:17 2010: [CMS-concurrent-preclean-start] .... Any idea what may causing these pauses and what possible JVM tuning would be neccessary to avoid these pauses? Your help greatly appreciated. Thank you very much. Atul -- View this message in context: http://old.nabble.com/long-unaccounted-pause-during-concurrent-mark-phase-of-ParNew-cycle-tp27459365p27459365.html Sent from the OpenJDK Hotspot Garbage Collection mailing list archive at Nabble.com. From Jon.Masamitsu at Sun.COM Thu Feb 4 21:34:44 2010 From: Jon.Masamitsu at Sun.COM (Jon Masamitsu) Date: Thu, 04 Feb 2010 13:34:44 -0800 Subject: long unaccounted pause during concurrent mark phase of ParNew cycle In-Reply-To: <27459365.post@talk.nabble.com> References: <27459365.post@talk.nabble.com> Message-ID: <4B6B3D74.90403@Sun.COM> Atul, Do you mean the 36 secs here? > Thu Feb 4 10:13:35 2010: [CMS-concurrent-mark: 36.272/36.715 secs] [Times: > user=56.34 sys=0.34, real=36.71 secs] That is a concurrent phase (i.e., the application threads are still running although they have to share the hardware threads with the concurrent GC threads). Jon On 02/04/10 12:37, atulksh wrote: > Hi, > We have an application that is freezing (pausing) for a very long (tens of > seconds) time during the concurrent mark phase of the ParNew cycle. This > pause is consistently seen when there is good amount of activity on the > server. We tried to upgrade the JVM (JRE 6) from Update 02 to Update 17 > keeping all the parameters same, but the pauses did not go away (they became > more consistent and slightly longer). We even tried to enable CompressedOops > option, thinking that that would reduce memory requirements, but that has > not helped. > > We are running on 64bit SUSE 10.1 platform with 32Gb of RAM and 8 core > machine. 26g is allocated to the java process. The full set of VM parameters > are here > > > -Xmx26g > -Xms26g > -server > -XX:+UseConcMarkSweepGC > -XX:PermSize=256m > -XX:MaxPermSize=256m > -XX:+PrintTenuringDistribution > -XX:+PrintGCDetails > -XX:+PrintGCTimeStamps > -XX:CMSInitiatingOccupancyFraction=70 > -XX:-CMSParallelRemarkEnabled > > The nature of the pause is like this. All the application threads that are > being processed seem to stop responding. New requests that come to our app > server are not processed and get queued. When the pause ends then everything > frees up and seems to go on normally henceforth. In the GC you see a gap > between the ParNews. The following are two examples of the pauses. Notice > the pause from 10:13:06 to 10:13:35 (29 seconds) in the first one and > 11:29:04 to 11:29:17 (13 seconds) in the second one. These pauses only occur > in the concurrent mark phase. They do not seem to occur at any other time or > anywhere else in the CMS cycle. > > .... > Thu Feb 4 10:12:59 2010: [GC [1 CMS-initial-mark: 18976170K(27092608K)] > 18989786K(27245952K), 0.0059140 secs] [Times: user=0.01 sys=0.00, real=0.01 > secs] > Thu Feb 4 10:12:59 2010: [CMS-concurrent-mark-start] > Thu Feb 4 10:12:59 2010 [18,980,009 K]: [GC 42035.293: [ParNew Desired > survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: 1833064 > bytes, 1833064 total - age 2: 18472 bytes, 1851536 total : > 148315K->3839K(153344K), 0.0400230 secs] 19124486K->18980009K(27245952K), > 0.0401580 secs] [Times: user=0.25 sys=0.00, real=0.04 secs] > Thu Feb 4 10:12:59 2010 [18,979,036 K]: [GC 42035.415: [ParNew Desired > survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: 2068376 > bytes, 2068376 total - age 2: 23792 bytes, 2092168 total - age > 3: 18472 bytes, 2110640 total : 140159K->2866K(153344K), 0.0352200 > secs] 19116329K->18979036K(27245952K), 0.0353520 secs] [Times: user=0.29 > sys=0.00, real=0.03 secs] > Thu Feb 4 10:12:59 2010 [18,979,103 K]: [GC 42035.535: [ParNew Desired > survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: 1782608 > bytes, 1782608 total - age 2: 22032 bytes, 1804640 total - age > 3: 23792 bytes, 1828432 total - age 4: 18472 bytes, > 1846904 total : 139186K->2933K(153344K), 0.0498480 secs] > 19115356K->18979103K(27245952K), 0.0499850 secs] [Times: user=0.24 sys=0.00, > real=0.05 secs] > Thu Feb 4 10:12:59 2010 [18,979,181 K]: [GC 42035.673: [ParNew Desired > survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: 2161000 > bytes, 2161000 total - age 2: 22032 bytes, 2183032 total - age > 3: 22032 bytes, 2205064 total - age 4: 23792 bytes, > 2228856 total : 139253K->2992K(153344K), 0.0400420 secs] > 19115423K->18979181K(27245952K), 0.0401730 secs] [Times: user=0.26 sys=0.00, > real=0.04 secs] > Thu Feb 4 10:13:06 2010 [18,979,876 K]: [GC 42042.243: [ParNew Desired > survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: 2113272 > bytes, 2113272 total - age 2: 23792 bytes, 2137064 total - age > 3: 22032 bytes, 2159096 total - age 4: 22032 bytes, > 2181128 total : 138855K->3662K(153344K), 0.0465500 secs] > 19115044K->18979876K(27245952K), 0.0466840 secs] [Times: user=0.27 sys=0.00, > real=0.04 secs] > Thu Feb 4 10:13:35 2010 [18,979,051 K]: [GC 42071.809: [ParNew Desired > survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: 219056 > bytes, 219056 total - age 2: 1867632 bytes, 2086688 total - age > 3: 23792 bytes, 2110480 total - age 4: 22032 bytes, > 2132512 total : 139982K->2814K(153344K), 0.0426920 secs] > 19116196K->18979051K(27245952K), 0.0428200 secs] [Times: user=0.28 sys=0.00, > real=0.05 secs] > Thu Feb 4 10:13:35 2010: [CMS-concurrent-mark: 36.272/36.715 secs] [Times: > user=56.34 sys=0.34, real=36.71 secs] > .... > > > .... > Thu Feb 4 11:28:59 2010: [GC [1 CMS-initial-mark: 19012252K(27092608K)] > 19032719K(27245952K), 0.0279440 secs] [Times: user=0.02 sys=0.00, real=0.03 > secs] > Thu Feb 4 11:28:59 2010: [CMS-concurrent-mark-start] > Thu Feb 4 11:28:59 2010 [19,154,661 K]: [GC 46595.614: [ParNew Desired > survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: 17431976 > bytes, 17431976 total : 153344K->17024K(153344K), 0.9259390 secs] > 19165596K->19154661K(27245952K), 0.9260890 secs] [Times: user=1.23 sys=0.05, > real=0.93 secs] > Thu Feb 4 11:29:00 2010 [19,199,696 K]: [GC 46596.973: [ParNew Desired > survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: 17448432 > bytes, 17448432 total : 153344K->17024K(153344K), 0.3054460 secs] > 19290981K->19199696K(27245952K), 0.3055930 secs] [Times: user=0.79 sys=0.07, > real=0.31 secs] > Thu Feb 4 11:29:01 2010 [19,229,358 K]: [GC 46597.722: [ParNew Desired > survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: 10331408 > bytes, 10331408 total : 153344K->17024K(153344K), 0.1449040 secs] > 19336016K->19229358K(27245952K), 0.1450630 secs] [Times: user=0.59 sys=0.02, > real=0.15 secs] > Thu Feb 4 11:29:02 2010 [19,240,015 K]: [GC 46598.038: [ParNew Desired > survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: 17354120 > bytes, 17354120 total : 153344K->17024K(153344K), 0.0935440 secs] > 19365678K->19240015K(27245952K), 0.0936870 secs] [Times: user=0.59 sys=0.00, > real=0.10 secs] > Thu Feb 4 11:29:02 2010 [19,302,509 K]: [GC 46598.347: [ParNew Desired > survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: 17432360 > bytes, 17432360 total : 151519K->17024K(153344K), 0.4619430 secs] > 19374510K->19302509K(27245952K), 0.4620890 secs] [Times: user=1.19 sys=0.02, > real=0.46 secs] > Thu Feb 4 11:29:03 2010 [19,329,594 K]: [GC 46599.030: [ParNew Desired > survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: 11933296 > bytes, 11933296 total : 153344K->17024K(153344K), 0.1492670 secs] > 19438829K->19329594K(27245952K), 0.1494200 secs] [Times: user=0.77 sys=0.03, > real=0.15 secs] > Thu Feb 4 11:29:03 2010 [19,337,176 K]: [GC 46599.423: [ParNew Desired > survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: 5719032 > bytes, 5719032 total : 153344K->12852K(153344K), 0.0675180 secs] > 19465914K->19337176K(27245952K), 0.0676670 secs] [Times: user=0.46 sys=0.02, > real=0.07 secs] > Thu Feb 4 11:29:03 2010 [19,338,256 K]: [GC 46599.754: [ParNew Desired > survivor size 8716288 bytes, new threshold 2 (max 4) - age 1: 6783616 > bytes, 6783616 total - age 2: 5570080 bytes, 12353696 total : > 149172K->13931K(153344K), 0.0650520 secs] 19473496K->19338256K(27245952K), > 0.0652060 secs] [Times: user=0.42 sys=0.01, real=0.07 secs] > Thu Feb 4 11:29:04 2010 [19,340,264 K]: [GC 46600.144: [ParNew Desired > survivor size 8716288 bytes, new threshold 2 (max 4) - age 1: 2190136 > bytes, 2190136 total - age 2: 6762648 bytes, 8952784 total : > 150251K->10375K(153344K), 0.0709470 secs] 19474576K->19340264K(27245952K), > 0.0710920 secs] [Times: user=0.48 sys=0.01, real=0.07 secs] > Thu Feb 4 11:29:17 2010 [19,345,928 K]: [GC 46613.628: [ParNew Desired > survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: 760488 > bytes, 760488 total - age 2: 2173472 bytes, 2933960 total : > 146695K->9279K(153344K), 0.0689390 secs] 19476584K->19345928K(27245952K), > 0.0691070 secs] [Times: user=0.45 sys=0.00, real=0.07 secs] > Thu Feb 4 11:29:17 2010: [CMS-concurrent-mark: 15.853/18.260 secs] [Times: > user=29.26 sys=0.55, real=18.26 secs] > Thu Feb 4 11:29:17 2010: [CMS-concurrent-preclean-start] > .... > > Any idea what may causing these pauses and what possible JVM tuning would be > neccessary to avoid these pauses? Your help greatly appreciated. > > Thank you very much. > > Atul From atulksh at hotmail.com Thu Feb 4 23:03:03 2010 From: atulksh at hotmail.com (atulksh) Date: Thu, 4 Feb 2010 15:03:03 -0800 (PST) Subject: long unaccounted pause during concurrent mark phase of ParNew cycle In-Reply-To: <4B6B3D74.90403@Sun.COM> References: <27459365.post@talk.nabble.com> <4B6B3D74.90403@Sun.COM> Message-ID: <27461205.post@talk.nabble.com> Hi Jon, Thanks for your response. Actually the 36 seconds that it is reporting that it took for the concurrent mark phase is correct. But the time I am mentioning is in between two ParNew cycles during the concurrent mark phase. Thu Feb 4 10:13:06 2010 [18,979,876 K]: [GC 42042.243: [ParNew Desired survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: 2113272 bytes, 2113272 total - age 2: 23792 bytes, 2137064 total - age 3: 22032 bytes, 2159096 total - age 4: 22032 bytes, 2181128 total : 138855K->3662K(153344K), 0.0465500 secs] 19115044K->18979876K(27245952K), 0.0466840 secs] [Times: user=0.27 sys=0.00, real=0.04 secs] Thu Feb 4 10:13:35 2010 [18,979,051 K]: [GC 42071.809: [ParNew Desired survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: 219056 bytes, 219056 total - age 2: 1867632 bytes, 2086688 total - age 3: 23792 bytes, 2110480 total - age 4: 22032 bytes, 2132512 total : 139982K->2814K(153344K), 0.0426920 secs] 19116196K->18979051K(27245952K), 0.0428200 secs] [Times: user=0.28 sys=0.00, real=0.05 secs] Notice that the two ParNews happened 29 seconds apart. During this period what we see is our app is hung and not accepting any new requests. It is as though the application is frozen for that amount of time. We noticed this because our users complained about the app not responding quickly sometimes. After analysing our logs, we realized this was the point where that happened. Otherwise this would just look like normal GC activity. One more thing that we see is in our GC logs these are the only times when there is such a long gap between two ParNew cycles. Otherwise we get ParNew pretty much every second, at a few seconds apart at max. Atul Jon Masamitsu wrote: > > Atul, > > Do you mean the 36 secs here? > > > Thu Feb 4 10:13:35 2010: [CMS-concurrent-mark: 36.272/36.715 secs] > [Times: > > user=56.34 sys=0.34, real=36.71 secs] > > That is a concurrent phase (i.e., the application threads > are still running although they have to share the hardware > threads with the concurrent GC threads). > > Jon > > On 02/04/10 12:37, atulksh wrote: >> Hi, >> We have an application that is freezing (pausing) for a very long (tens >> of >> seconds) time during the concurrent mark phase of the ParNew cycle. This >> pause is consistently seen when there is good amount of activity on the >> server. We tried to upgrade the JVM (JRE 6) from Update 02 to Update 17 >> keeping all the parameters same, but the pauses did not go away (they >> became >> more consistent and slightly longer). We even tried to enable >> CompressedOops >> option, thinking that that would reduce memory requirements, but that has >> not helped. >> >> We are running on 64bit SUSE 10.1 platform with 32Gb of RAM and 8 core >> machine. 26g is allocated to the java process. The full set of VM >> parameters >> are here >> >> >> -Xmx26g >> -Xms26g >> -server >> -XX:+UseConcMarkSweepGC >> -XX:PermSize=256m >> -XX:MaxPermSize=256m >> -XX:+PrintTenuringDistribution >> -XX:+PrintGCDetails >> -XX:+PrintGCTimeStamps >> -XX:CMSInitiatingOccupancyFraction=70 >> -XX:-CMSParallelRemarkEnabled >> >> The nature of the pause is like this. All the application threads that >> are >> being processed seem to stop responding. New requests that come to our >> app >> server are not processed and get queued. When the pause ends then >> everything >> frees up and seems to go on normally henceforth. In the GC you see a gap >> between the ParNews. The following are two examples of the pauses. Notice >> the pause from 10:13:06 to 10:13:35 (29 seconds) in the first one and >> 11:29:04 to 11:29:17 (13 seconds) in the second one. These pauses only >> occur >> in the concurrent mark phase. They do not seem to occur at any other time >> or >> anywhere else in the CMS cycle. >> >> .... >> Thu Feb 4 10:12:59 2010: [GC [1 CMS-initial-mark: 18976170K(27092608K)] >> 18989786K(27245952K), 0.0059140 secs] [Times: user=0.01 sys=0.00, >> real=0.01 >> secs] >> Thu Feb 4 10:12:59 2010: [CMS-concurrent-mark-start] >> Thu Feb 4 10:12:59 2010 [18,980,009 K]: [GC 42035.293: [ParNew Desired >> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >> 1833064 >> bytes, 1833064 total - age 2: 18472 bytes, 1851536 total : >> 148315K->3839K(153344K), 0.0400230 secs] 19124486K->18980009K(27245952K), >> 0.0401580 secs] [Times: user=0.25 sys=0.00, real=0.04 secs] >> Thu Feb 4 10:12:59 2010 [18,979,036 K]: [GC 42035.415: [ParNew Desired >> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >> 2068376 >> bytes, 2068376 total - age 2: 23792 bytes, 2092168 total - >> age >> 3: 18472 bytes, 2110640 total : 140159K->2866K(153344K), >> 0.0352200 >> secs] 19116329K->18979036K(27245952K), 0.0353520 secs] [Times: user=0.29 >> sys=0.00, real=0.03 secs] >> Thu Feb 4 10:12:59 2010 [18,979,103 K]: [GC 42035.535: [ParNew Desired >> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >> 1782608 >> bytes, 1782608 total - age 2: 22032 bytes, 1804640 total - >> age >> 3: 23792 bytes, 1828432 total - age 4: 18472 bytes, >> 1846904 total : 139186K->2933K(153344K), 0.0498480 secs] >> 19115356K->18979103K(27245952K), 0.0499850 secs] [Times: user=0.24 >> sys=0.00, >> real=0.05 secs] >> Thu Feb 4 10:12:59 2010 [18,979,181 K]: [GC 42035.673: [ParNew Desired >> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >> 2161000 >> bytes, 2161000 total - age 2: 22032 bytes, 2183032 total - >> age >> 3: 22032 bytes, 2205064 total - age 4: 23792 bytes, >> 2228856 total : 139253K->2992K(153344K), 0.0400420 secs] >> 19115423K->18979181K(27245952K), 0.0401730 secs] [Times: user=0.26 >> sys=0.00, >> real=0.04 secs] >> Thu Feb 4 10:13:06 2010 [18,979,876 K]: [GC 42042.243: [ParNew Desired >> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >> 2113272 >> bytes, 2113272 total - age 2: 23792 bytes, 2137064 total - >> age >> 3: 22032 bytes, 2159096 total - age 4: 22032 bytes, >> 2181128 total : 138855K->3662K(153344K), 0.0465500 secs] >> 19115044K->18979876K(27245952K), 0.0466840 secs] [Times: user=0.27 >> sys=0.00, >> real=0.04 secs] >> Thu Feb 4 10:13:35 2010 [18,979,051 K]: [GC 42071.809: [ParNew Desired >> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >> 219056 >> bytes, 219056 total - age 2: 1867632 bytes, 2086688 total - >> age >> 3: 23792 bytes, 2110480 total - age 4: 22032 bytes, >> 2132512 total : 139982K->2814K(153344K), 0.0426920 secs] >> 19116196K->18979051K(27245952K), 0.0428200 secs] [Times: user=0.28 >> sys=0.00, >> real=0.05 secs] >> Thu Feb 4 10:13:35 2010: [CMS-concurrent-mark: 36.272/36.715 secs] >> [Times: >> user=56.34 sys=0.34, real=36.71 secs] >> .... >> >> >> .... >> Thu Feb 4 11:28:59 2010: [GC [1 CMS-initial-mark: 19012252K(27092608K)] >> 19032719K(27245952K), 0.0279440 secs] [Times: user=0.02 sys=0.00, >> real=0.03 >> secs] >> Thu Feb 4 11:28:59 2010: [CMS-concurrent-mark-start] >> Thu Feb 4 11:28:59 2010 [19,154,661 K]: [GC 46595.614: [ParNew Desired >> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >> 17431976 >> bytes, 17431976 total : 153344K->17024K(153344K), 0.9259390 secs] >> 19165596K->19154661K(27245952K), 0.9260890 secs] [Times: user=1.23 >> sys=0.05, >> real=0.93 secs] >> Thu Feb 4 11:29:00 2010 [19,199,696 K]: [GC 46596.973: [ParNew Desired >> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >> 17448432 >> bytes, 17448432 total : 153344K->17024K(153344K), 0.3054460 secs] >> 19290981K->19199696K(27245952K), 0.3055930 secs] [Times: user=0.79 >> sys=0.07, >> real=0.31 secs] >> Thu Feb 4 11:29:01 2010 [19,229,358 K]: [GC 46597.722: [ParNew Desired >> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >> 10331408 >> bytes, 10331408 total : 153344K->17024K(153344K), 0.1449040 secs] >> 19336016K->19229358K(27245952K), 0.1450630 secs] [Times: user=0.59 >> sys=0.02, >> real=0.15 secs] >> Thu Feb 4 11:29:02 2010 [19,240,015 K]: [GC 46598.038: [ParNew Desired >> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >> 17354120 >> bytes, 17354120 total : 153344K->17024K(153344K), 0.0935440 secs] >> 19365678K->19240015K(27245952K), 0.0936870 secs] [Times: user=0.59 >> sys=0.00, >> real=0.10 secs] >> Thu Feb 4 11:29:02 2010 [19,302,509 K]: [GC 46598.347: [ParNew Desired >> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >> 17432360 >> bytes, 17432360 total : 151519K->17024K(153344K), 0.4619430 secs] >> 19374510K->19302509K(27245952K), 0.4620890 secs] [Times: user=1.19 >> sys=0.02, >> real=0.46 secs] >> Thu Feb 4 11:29:03 2010 [19,329,594 K]: [GC 46599.030: [ParNew Desired >> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >> 11933296 >> bytes, 11933296 total : 153344K->17024K(153344K), 0.1492670 secs] >> 19438829K->19329594K(27245952K), 0.1494200 secs] [Times: user=0.77 >> sys=0.03, >> real=0.15 secs] >> Thu Feb 4 11:29:03 2010 [19,337,176 K]: [GC 46599.423: [ParNew Desired >> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >> 5719032 >> bytes, 5719032 total : 153344K->12852K(153344K), 0.0675180 secs] >> 19465914K->19337176K(27245952K), 0.0676670 secs] [Times: user=0.46 >> sys=0.02, >> real=0.07 secs] >> Thu Feb 4 11:29:03 2010 [19,338,256 K]: [GC 46599.754: [ParNew Desired >> survivor size 8716288 bytes, new threshold 2 (max 4) - age 1: >> 6783616 >> bytes, 6783616 total - age 2: 5570080 bytes, 12353696 total : >> 149172K->13931K(153344K), 0.0650520 secs] >> 19473496K->19338256K(27245952K), >> 0.0652060 secs] [Times: user=0.42 sys=0.01, real=0.07 secs] >> Thu Feb 4 11:29:04 2010 [19,340,264 K]: [GC 46600.144: [ParNew Desired >> survivor size 8716288 bytes, new threshold 2 (max 4) - age 1: >> 2190136 >> bytes, 2190136 total - age 2: 6762648 bytes, 8952784 total : >> 150251K->10375K(153344K), 0.0709470 secs] >> 19474576K->19340264K(27245952K), >> 0.0710920 secs] [Times: user=0.48 sys=0.01, real=0.07 secs] >> Thu Feb 4 11:29:17 2010 [19,345,928 K]: [GC 46613.628: [ParNew Desired >> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >> 760488 >> bytes, 760488 total - age 2: 2173472 bytes, 2933960 total : >> 146695K->9279K(153344K), 0.0689390 secs] 19476584K->19345928K(27245952K), >> 0.0691070 secs] [Times: user=0.45 sys=0.00, real=0.07 secs] >> Thu Feb 4 11:29:17 2010: [CMS-concurrent-mark: 15.853/18.260 secs] >> [Times: >> user=29.26 sys=0.55, real=18.26 secs] >> Thu Feb 4 11:29:17 2010: [CMS-concurrent-preclean-start] >> .... >> >> Any idea what may causing these pauses and what possible JVM tuning would >> be >> neccessary to avoid these pauses? Your help greatly appreciated. >> >> Thank you very much. >> >> Atul > > -- View this message in context: http://old.nabble.com/long-unaccounted-pause-during-concurrent-mark-phase-of-ParNew-cycle-tp27459365p27461205.html Sent from the OpenJDK Hotspot Garbage Collection mailing list archive at Nabble.com. From atulksh at hotmail.com Fri Feb 5 04:02:15 2010 From: atulksh at hotmail.com (atulksh) Date: Thu, 4 Feb 2010 20:02:15 -0800 (PST) Subject: long unaccounted pause during concurrent mark phase of ParNew cycle In-Reply-To: <4B6B3D74.90403@Sun.COM> References: <27459365.post@talk.nabble.com> <4B6B3D74.90403@Sun.COM> Message-ID: <27463304.post@talk.nabble.com> Hi Jon, I have some more (probably more precise) information for you. I have been able to reproduce this issue in our test environment (up till now we were only seeing it in our production setup). Along with the parameters that I have specified above I added the following. After loading up the app server (it is Tomcat 6.0.14) I got a pause during the CMS cycle of about 32 seconds. The way I noticed the pause is like this. I kept on running a script that does some basic page refresh (small data page) in the application continously. This is like a ping to the server. Then I parallely do operations that load up the memory in the application server. These are the real hits that users do in whom memory requirement is usually high and that drives the app server to the CMS threshold. In my case once the CMS kicked in, I saw that the period where the applilcation was in the "pause" no requests of my script were processed. They were just queued up to Tomcat - meaning Apache got those requests and logged that they were sent to Tomcat. But Tomcat did not register of receiving them. Then after the app went out of the "pause" then everything came back and all these pending requests got processed - like the app just had been unfrozen. During this freeze, what I see in the GC log is that the CMS cycle is almost at the end of the concurrent mark phase. It is doing the GC is doing the "perm space scanning". The pause exactly coincides with the time reported for the "perm space scanning". I am attaching the full gc log snippt right from the start of the CMS cycle. Below is just the place where the pause appears. 28676.784: [GC 28676.784: [ParNew Desired survivor size 4358144 bytes, new threshold 2 (max 4) - age 1: 3742608 bytes, 3742608 total - age 2: 3451152 bytes, 7193760 total : 76672K->8512K(76672K), 0.2045430 secs] 20262376K->20197311K(27254464K), 0.2048030 secs] [Times: user=0.77 sys=0.01, real=0.21 secs] Total time for which application threads were stopped: 0.2055080 seconds Finished perm space scanning in 2th thread: 0.367 sec Finished perm space scanning in 0th thread: 0.362 sec Application time: 0.1481860 seconds 28708.626: [GC 28708.627: [ParNew Desired survivor size 4358144 bytes, new threshold 2 (max 4) - age 1: 2079760 bytes, 2079760 total - age 2: 3657248 bytes, 5737008 total : 76672K->6795K(76672K), 0.1937080 secs] 20265471K->20198965K(27254464K), 0.1939480 secs] [Times: user=0.74 sys=0.00, real=0.20 secs] Total time for which application threads were stopped: 0.1948820 seconds Finished perm space scanning in 3th thread: 32.153 sec Finished perm space scanning in 1th thread: 32.097 sec Finished work stealing in 1th thread: 0.000 sec Finished work stealing in 3th thread: 0.002 sec Finished work stealing in 2th thread: 31.791 sec Finished work stealing in 0th thread: 31.791 sec 28708.825: [CMS-concurrent-mark: 88.906/158.952 secs] (CMS-concurrent-mark yielded 454 times) [Times: user=544.33 sys=4.59, real=158.95 secs] What is "perm space scanning" and is this where the gc would freeze the application (STW type). Is this a bug or something that is expected? Please let me know. This has become quite critical for our application. Please see the full gc log file attached. Your help very much appreciated. Thank you, Atul Jon Masamitsu wrote: > > Atul, > > Do you mean the 36 secs here? > > > Thu Feb 4 10:13:35 2010: [CMS-concurrent-mark: 36.272/36.715 secs] > [Times: > > user=56.34 sys=0.34, real=36.71 secs] > > That is a concurrent phase (i.e., the application threads > are still running although they have to share the hardware > threads with the concurrent GC threads). > > Jon > > On 02/04/10 12:37, atulksh wrote: >> Hi, >> We have an application that is freezing (pausing) for a very long (tens >> of >> seconds) time during the concurrent mark phase of the ParNew cycle. This >> pause is consistently seen when there is good amount of activity on the >> server. We tried to upgrade the JVM (JRE 6) from Update 02 to Update 17 >> keeping all the parameters same, but the pauses did not go away (they >> became >> more consistent and slightly longer). We even tried to enable >> CompressedOops >> option, thinking that that would reduce memory requirements, but that has >> not helped. >> >> We are running on 64bit SUSE 10.1 platform with 32Gb of RAM and 8 core >> machine. 26g is allocated to the java process. The full set of VM >> parameters >> are here >> >> >> -Xmx26g >> -Xms26g >> -server >> -XX:+UseConcMarkSweepGC >> -XX:PermSize=256m >> -XX:MaxPermSize=256m >> -XX:+PrintTenuringDistribution >> -XX:+PrintGCDetails >> -XX:+PrintGCTimeStamps >> -XX:CMSInitiatingOccupancyFraction=70 >> -XX:-CMSParallelRemarkEnabled >> >> The nature of the pause is like this. All the application threads that >> are >> being processed seem to stop responding. New requests that come to our >> app >> server are not processed and get queued. When the pause ends then >> everything >> frees up and seems to go on normally henceforth. In the GC you see a gap >> between the ParNews. The following are two examples of the pauses. Notice >> the pause from 10:13:06 to 10:13:35 (29 seconds) in the first one and >> 11:29:04 to 11:29:17 (13 seconds) in the second one. These pauses only >> occur >> in the concurrent mark phase. They do not seem to occur at any other time >> or >> anywhere else in the CMS cycle. >> >> .... >> Thu Feb 4 10:12:59 2010: [GC [1 CMS-initial-mark: 18976170K(27092608K)] >> 18989786K(27245952K), 0.0059140 secs] [Times: user=0.01 sys=0.00, >> real=0.01 >> secs] >> Thu Feb 4 10:12:59 2010: [CMS-concurrent-mark-start] >> Thu Feb 4 10:12:59 2010 [18,980,009 K]: [GC 42035.293: [ParNew Desired >> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >> 1833064 >> bytes, 1833064 total - age 2: 18472 bytes, 1851536 total : >> 148315K->3839K(153344K), 0.0400230 secs] 19124486K->18980009K(27245952K), >> 0.0401580 secs] [Times: user=0.25 sys=0.00, real=0.04 secs] >> Thu Feb 4 10:12:59 2010 [18,979,036 K]: [GC 42035.415: [ParNew Desired >> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >> 2068376 >> bytes, 2068376 total - age 2: 23792 bytes, 2092168 total - >> age >> 3: 18472 bytes, 2110640 total : 140159K->2866K(153344K), >> 0.0352200 >> secs] 19116329K->18979036K(27245952K), 0.0353520 secs] [Times: user=0.29 >> sys=0.00, real=0.03 secs] >> Thu Feb 4 10:12:59 2010 [18,979,103 K]: [GC 42035.535: [ParNew Desired >> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >> 1782608 >> bytes, 1782608 total - age 2: 22032 bytes, 1804640 total - >> age >> 3: 23792 bytes, 1828432 total - age 4: 18472 bytes, >> 1846904 total : 139186K->2933K(153344K), 0.0498480 secs] >> 19115356K->18979103K(27245952K), 0.0499850 secs] [Times: user=0.24 >> sys=0.00, >> real=0.05 secs] >> Thu Feb 4 10:12:59 2010 [18,979,181 K]: [GC 42035.673: [ParNew Desired >> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >> 2161000 >> bytes, 2161000 total - age 2: 22032 bytes, 2183032 total - >> age >> 3: 22032 bytes, 2205064 total - age 4: 23792 bytes, >> 2228856 total : 139253K->2992K(153344K), 0.0400420 secs] >> 19115423K->18979181K(27245952K), 0.0401730 secs] [Times: user=0.26 >> sys=0.00, >> real=0.04 secs] >> Thu Feb 4 10:13:06 2010 [18,979,876 K]: [GC 42042.243: [ParNew Desired >> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >> 2113272 >> bytes, 2113272 total - age 2: 23792 bytes, 2137064 total - >> age >> 3: 22032 bytes, 2159096 total - age 4: 22032 bytes, >> 2181128 total : 138855K->3662K(153344K), 0.0465500 secs] >> 19115044K->18979876K(27245952K), 0.0466840 secs] [Times: user=0.27 >> sys=0.00, >> real=0.04 secs] >> Thu Feb 4 10:13:35 2010 [18,979,051 K]: [GC 42071.809: [ParNew Desired >> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >> 219056 >> bytes, 219056 total - age 2: 1867632 bytes, 2086688 total - >> age >> 3: 23792 bytes, 2110480 total - age 4: 22032 bytes, >> 2132512 total : 139982K->2814K(153344K), 0.0426920 secs] >> 19116196K->18979051K(27245952K), 0.0428200 secs] [Times: user=0.28 >> sys=0.00, >> real=0.05 secs] >> Thu Feb 4 10:13:35 2010: [CMS-concurrent-mark: 36.272/36.715 secs] >> [Times: >> user=56.34 sys=0.34, real=36.71 secs] >> .... >> >> >> .... >> Thu Feb 4 11:28:59 2010: [GC [1 CMS-initial-mark: 19012252K(27092608K)] >> 19032719K(27245952K), 0.0279440 secs] [Times: user=0.02 sys=0.00, >> real=0.03 >> secs] >> Thu Feb 4 11:28:59 2010: [CMS-concurrent-mark-start] >> Thu Feb 4 11:28:59 2010 [19,154,661 K]: [GC 46595.614: [ParNew Desired >> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >> 17431976 >> bytes, 17431976 total : 153344K->17024K(153344K), 0.9259390 secs] >> 19165596K->19154661K(27245952K), 0.9260890 secs] [Times: user=1.23 >> sys=0.05, >> real=0.93 secs] >> Thu Feb 4 11:29:00 2010 [19,199,696 K]: [GC 46596.973: [ParNew Desired >> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >> 17448432 >> bytes, 17448432 total : 153344K->17024K(153344K), 0.3054460 secs] >> 19290981K->19199696K(27245952K), 0.3055930 secs] [Times: user=0.79 >> sys=0.07, >> real=0.31 secs] >> Thu Feb 4 11:29:01 2010 [19,229,358 K]: [GC 46597.722: [ParNew Desired >> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >> 10331408 >> bytes, 10331408 total : 153344K->17024K(153344K), 0.1449040 secs] >> 19336016K->19229358K(27245952K), 0.1450630 secs] [Times: user=0.59 >> sys=0.02, >> real=0.15 secs] >> Thu Feb 4 11:29:02 2010 [19,240,015 K]: [GC 46598.038: [ParNew Desired >> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >> 17354120 >> bytes, 17354120 total : 153344K->17024K(153344K), 0.0935440 secs] >> 19365678K->19240015K(27245952K), 0.0936870 secs] [Times: user=0.59 >> sys=0.00, >> real=0.10 secs] >> Thu Feb 4 11:29:02 2010 [19,302,509 K]: [GC 46598.347: [ParNew Desired >> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >> 17432360 >> bytes, 17432360 total : 151519K->17024K(153344K), 0.4619430 secs] >> 19374510K->19302509K(27245952K), 0.4620890 secs] [Times: user=1.19 >> sys=0.02, >> real=0.46 secs] >> Thu Feb 4 11:29:03 2010 [19,329,594 K]: [GC 46599.030: [ParNew Desired >> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >> 11933296 >> bytes, 11933296 total : 153344K->17024K(153344K), 0.1492670 secs] >> 19438829K->19329594K(27245952K), 0.1494200 secs] [Times: user=0.77 >> sys=0.03, >> real=0.15 secs] >> Thu Feb 4 11:29:03 2010 [19,337,176 K]: [GC 46599.423: [ParNew Desired >> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >> 5719032 >> bytes, 5719032 total : 153344K->12852K(153344K), 0.0675180 secs] >> 19465914K->19337176K(27245952K), 0.0676670 secs] [Times: user=0.46 >> sys=0.02, >> real=0.07 secs] >> Thu Feb 4 11:29:03 2010 [19,338,256 K]: [GC 46599.754: [ParNew Desired >> survivor size 8716288 bytes, new threshold 2 (max 4) - age 1: >> 6783616 >> bytes, 6783616 total - age 2: 5570080 bytes, 12353696 total : >> 149172K->13931K(153344K), 0.0650520 secs] >> 19473496K->19338256K(27245952K), >> 0.0652060 secs] [Times: user=0.42 sys=0.01, real=0.07 secs] >> Thu Feb 4 11:29:04 2010 [19,340,264 K]: [GC 46600.144: [ParNew Desired >> survivor size 8716288 bytes, new threshold 2 (max 4) - age 1: >> 2190136 >> bytes, 2190136 total - age 2: 6762648 bytes, 8952784 total : >> 150251K->10375K(153344K), 0.0709470 secs] >> 19474576K->19340264K(27245952K), >> 0.0710920 secs] [Times: user=0.48 sys=0.01, real=0.07 secs] >> Thu Feb 4 11:29:17 2010 [19,345,928 K]: [GC 46613.628: [ParNew Desired >> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >> 760488 >> bytes, 760488 total - age 2: 2173472 bytes, 2933960 total : >> 146695K->9279K(153344K), 0.0689390 secs] 19476584K->19345928K(27245952K), >> 0.0691070 secs] [Times: user=0.45 sys=0.00, real=0.07 secs] >> Thu Feb 4 11:29:17 2010: [CMS-concurrent-mark: 15.853/18.260 secs] >> [Times: >> user=29.26 sys=0.55, real=18.26 secs] >> Thu Feb 4 11:29:17 2010: [CMS-concurrent-preclean-start] >> .... >> >> Any idea what may causing these pauses and what possible JVM tuning would >> be >> neccessary to avoid these pauses? Your help greatly appreciated. >> >> Thank you very much. >> >> Atul > > http://old.nabble.com/file/p27463304/long.pause.in.cms.gc.log long.pause.in.cms.gc.log -- View this message in context: http://old.nabble.com/long-unaccounted-pause-during-concurrent-mark-phase-of-ParNew-cycle-tp27459365p27463304.html Sent from the OpenJDK Hotspot Garbage Collection mailing list archive at Nabble.com. From Y.S.Ramakrishna at Sun.COM Fri Feb 5 06:05:49 2010 From: Y.S.Ramakrishna at Sun.COM (Y. Srinivas Ramakrishna) Date: Thu, 04 Feb 2010 22:05:49 -0800 Subject: long unaccounted pause during concurrent mark phase of ParNew cycle In-Reply-To: <27463304.post@talk.nabble.com> References: <27459365.post@talk.nabble.com> <4B6B3D74.90403@Sun.COM> <27463304.post@talk.nabble.com> Message-ID: <4B6BB53D.5010800@sun.com> Hello Atul -- Could you add -XX:-CMSConcurrentMTEnabled to the mix and see if it affects the behaviour in any manner? PS: I notice that you are using -XX:-CMSParallelRemarkEnabled. What was the reason for using that flag? thanks. -- ramki atulksh wrote: > Hi Jon, > I have some more (probably more precise) information for you. I have been > able to reproduce this issue in our test environment (up till now we were > only seeing it in our production setup). Along with the parameters that I > have specified above I added the following. > > > > After loading up the app server (it is Tomcat 6.0.14) I got a pause during > the CMS cycle of about 32 seconds. The way I noticed the pause is like this. > I kept on running a script that does some basic page refresh (small data > page) in the application continously. This is like a ping to the server. > Then I parallely do operations that load up the memory in the application > server. These are the real hits that users do in whom memory requirement is > usually high and that drives the app server to the CMS threshold. > > In my case once the CMS kicked in, I saw that the period where the > applilcation was in the "pause" no requests of my script were processed. > They were just queued up to Tomcat - meaning Apache got those requests and > logged that they were sent to Tomcat. But Tomcat did not register of > receiving them. Then after the app went out of the "pause" then everything > came back and all these pending requests got processed - like the app just > had been unfrozen. > > During this freeze, what I see in the GC log is that the CMS cycle is almost > at the end of the concurrent mark phase. It is doing the GC is doing the > "perm space scanning". The pause exactly coincides with the time reported > for the "perm space scanning". I am attaching the full gc log snippt right > from the start of the CMS cycle. Below is just the place where the pause > appears. > > 28676.784: [GC 28676.784: [ParNew > Desired survivor size 4358144 bytes, new threshold 2 (max 4) > - age 1: 3742608 bytes, 3742608 total > - age 2: 3451152 bytes, 7193760 total > : 76672K->8512K(76672K), 0.2045430 secs] 20262376K->20197311K(27254464K), > 0.2048030 secs] [Times: user=0.77 sys=0.01, real=0.21 secs] > Total time for which application threads were stopped: 0.2055080 seconds > Finished perm space scanning in 2th thread: 0.367 sec > Finished perm space scanning in 0th thread: 0.362 sec > Application time: 0.1481860 seconds > 28708.626: [GC 28708.627: [ParNew > Desired survivor size 4358144 bytes, new threshold 2 (max 4) > - age 1: 2079760 bytes, 2079760 total > - age 2: 3657248 bytes, 5737008 total > : 76672K->6795K(76672K), 0.1937080 secs] 20265471K->20198965K(27254464K), > 0.1939480 secs] [Times: user=0.74 sys=0.00, real=0.20 secs] > Total time for which application threads were stopped: 0.1948820 seconds > Finished perm space scanning in 3th thread: 32.153 sec > Finished perm space scanning in 1th thread: 32.097 sec > Finished work stealing in 1th thread: 0.000 sec > Finished work stealing in 3th thread: 0.002 sec > Finished work stealing in 2th thread: 31.791 sec > Finished work stealing in 0th thread: 31.791 sec > 28708.825: [CMS-concurrent-mark: 88.906/158.952 secs] (CMS-concurrent-mark > yielded 454 times) > [Times: user=544.33 sys=4.59, real=158.95 secs] > > What is "perm space scanning" and is this where the gc would freeze the > application (STW type). Is this a bug or something that is expected? > > Please let me know. This has become quite critical for our application. > Please see the full gc log file attached. Your help very much appreciated. > > Thank you, > > Atul > > > Jon Masamitsu wrote: >> Atul, >> >> Do you mean the 36 secs here? >> >> > Thu Feb 4 10:13:35 2010: [CMS-concurrent-mark: 36.272/36.715 secs] >> [Times: >> > user=56.34 sys=0.34, real=36.71 secs] >> >> That is a concurrent phase (i.e., the application threads >> are still running although they have to share the hardware >> threads with the concurrent GC threads). >> >> Jon >> >> On 02/04/10 12:37, atulksh wrote: >>> Hi, >>> We have an application that is freezing (pausing) for a very long (tens >>> of >>> seconds) time during the concurrent mark phase of the ParNew cycle. This >>> pause is consistently seen when there is good amount of activity on the >>> server. We tried to upgrade the JVM (JRE 6) from Update 02 to Update 17 >>> keeping all the parameters same, but the pauses did not go away (they >>> became >>> more consistent and slightly longer). We even tried to enable >>> CompressedOops >>> option, thinking that that would reduce memory requirements, but that has >>> not helped. >>> >>> We are running on 64bit SUSE 10.1 platform with 32Gb of RAM and 8 core >>> machine. 26g is allocated to the java process. The full set of VM >>> parameters >>> are here >>> >>> >>> -Xmx26g >>> -Xms26g >>> -server >>> -XX:+UseConcMarkSweepGC >>> -XX:PermSize=256m >>> -XX:MaxPermSize=256m >>> -XX:+PrintTenuringDistribution >>> -XX:+PrintGCDetails >>> -XX:+PrintGCTimeStamps >>> -XX:CMSInitiatingOccupancyFraction=70 >>> -XX:-CMSParallelRemarkEnabled >>> >>> The nature of the pause is like this. All the application threads that >>> are >>> being processed seem to stop responding. New requests that come to our >>> app >>> server are not processed and get queued. When the pause ends then >>> everything >>> frees up and seems to go on normally henceforth. In the GC you see a gap >>> between the ParNews. The following are two examples of the pauses. Notice >>> the pause from 10:13:06 to 10:13:35 (29 seconds) in the first one and >>> 11:29:04 to 11:29:17 (13 seconds) in the second one. These pauses only >>> occur >>> in the concurrent mark phase. They do not seem to occur at any other time >>> or >>> anywhere else in the CMS cycle. >>> >>> .... >>> Thu Feb 4 10:12:59 2010: [GC [1 CMS-initial-mark: 18976170K(27092608K)] >>> 18989786K(27245952K), 0.0059140 secs] [Times: user=0.01 sys=0.00, >>> real=0.01 >>> secs] >>> Thu Feb 4 10:12:59 2010: [CMS-concurrent-mark-start] >>> Thu Feb 4 10:12:59 2010 [18,980,009 K]: [GC 42035.293: [ParNew Desired >>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>> 1833064 >>> bytes, 1833064 total - age 2: 18472 bytes, 1851536 total : >>> 148315K->3839K(153344K), 0.0400230 secs] 19124486K->18980009K(27245952K), >>> 0.0401580 secs] [Times: user=0.25 sys=0.00, real=0.04 secs] >>> Thu Feb 4 10:12:59 2010 [18,979,036 K]: [GC 42035.415: [ParNew Desired >>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>> 2068376 >>> bytes, 2068376 total - age 2: 23792 bytes, 2092168 total - >>> age >>> 3: 18472 bytes, 2110640 total : 140159K->2866K(153344K), >>> 0.0352200 >>> secs] 19116329K->18979036K(27245952K), 0.0353520 secs] [Times: user=0.29 >>> sys=0.00, real=0.03 secs] >>> Thu Feb 4 10:12:59 2010 [18,979,103 K]: [GC 42035.535: [ParNew Desired >>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>> 1782608 >>> bytes, 1782608 total - age 2: 22032 bytes, 1804640 total - >>> age >>> 3: 23792 bytes, 1828432 total - age 4: 18472 bytes, >>> 1846904 total : 139186K->2933K(153344K), 0.0498480 secs] >>> 19115356K->18979103K(27245952K), 0.0499850 secs] [Times: user=0.24 >>> sys=0.00, >>> real=0.05 secs] >>> Thu Feb 4 10:12:59 2010 [18,979,181 K]: [GC 42035.673: [ParNew Desired >>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>> 2161000 >>> bytes, 2161000 total - age 2: 22032 bytes, 2183032 total - >>> age >>> 3: 22032 bytes, 2205064 total - age 4: 23792 bytes, >>> 2228856 total : 139253K->2992K(153344K), 0.0400420 secs] >>> 19115423K->18979181K(27245952K), 0.0401730 secs] [Times: user=0.26 >>> sys=0.00, >>> real=0.04 secs] >>> Thu Feb 4 10:13:06 2010 [18,979,876 K]: [GC 42042.243: [ParNew Desired >>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>> 2113272 >>> bytes, 2113272 total - age 2: 23792 bytes, 2137064 total - >>> age >>> 3: 22032 bytes, 2159096 total - age 4: 22032 bytes, >>> 2181128 total : 138855K->3662K(153344K), 0.0465500 secs] >>> 19115044K->18979876K(27245952K), 0.0466840 secs] [Times: user=0.27 >>> sys=0.00, >>> real=0.04 secs] >>> Thu Feb 4 10:13:35 2010 [18,979,051 K]: [GC 42071.809: [ParNew Desired >>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>> 219056 >>> bytes, 219056 total - age 2: 1867632 bytes, 2086688 total - >>> age >>> 3: 23792 bytes, 2110480 total - age 4: 22032 bytes, >>> 2132512 total : 139982K->2814K(153344K), 0.0426920 secs] >>> 19116196K->18979051K(27245952K), 0.0428200 secs] [Times: user=0.28 >>> sys=0.00, >>> real=0.05 secs] >>> Thu Feb 4 10:13:35 2010: [CMS-concurrent-mark: 36.272/36.715 secs] >>> [Times: >>> user=56.34 sys=0.34, real=36.71 secs] >>> .... >>> >>> >>> .... >>> Thu Feb 4 11:28:59 2010: [GC [1 CMS-initial-mark: 19012252K(27092608K)] >>> 19032719K(27245952K), 0.0279440 secs] [Times: user=0.02 sys=0.00, >>> real=0.03 >>> secs] >>> Thu Feb 4 11:28:59 2010: [CMS-concurrent-mark-start] >>> Thu Feb 4 11:28:59 2010 [19,154,661 K]: [GC 46595.614: [ParNew Desired >>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>> 17431976 >>> bytes, 17431976 total : 153344K->17024K(153344K), 0.9259390 secs] >>> 19165596K->19154661K(27245952K), 0.9260890 secs] [Times: user=1.23 >>> sys=0.05, >>> real=0.93 secs] >>> Thu Feb 4 11:29:00 2010 [19,199,696 K]: [GC 46596.973: [ParNew Desired >>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>> 17448432 >>> bytes, 17448432 total : 153344K->17024K(153344K), 0.3054460 secs] >>> 19290981K->19199696K(27245952K), 0.3055930 secs] [Times: user=0.79 >>> sys=0.07, >>> real=0.31 secs] >>> Thu Feb 4 11:29:01 2010 [19,229,358 K]: [GC 46597.722: [ParNew Desired >>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>> 10331408 >>> bytes, 10331408 total : 153344K->17024K(153344K), 0.1449040 secs] >>> 19336016K->19229358K(27245952K), 0.1450630 secs] [Times: user=0.59 >>> sys=0.02, >>> real=0.15 secs] >>> Thu Feb 4 11:29:02 2010 [19,240,015 K]: [GC 46598.038: [ParNew Desired >>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>> 17354120 >>> bytes, 17354120 total : 153344K->17024K(153344K), 0.0935440 secs] >>> 19365678K->19240015K(27245952K), 0.0936870 secs] [Times: user=0.59 >>> sys=0.00, >>> real=0.10 secs] >>> Thu Feb 4 11:29:02 2010 [19,302,509 K]: [GC 46598.347: [ParNew Desired >>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>> 17432360 >>> bytes, 17432360 total : 151519K->17024K(153344K), 0.4619430 secs] >>> 19374510K->19302509K(27245952K), 0.4620890 secs] [Times: user=1.19 >>> sys=0.02, >>> real=0.46 secs] >>> Thu Feb 4 11:29:03 2010 [19,329,594 K]: [GC 46599.030: [ParNew Desired >>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>> 11933296 >>> bytes, 11933296 total : 153344K->17024K(153344K), 0.1492670 secs] >>> 19438829K->19329594K(27245952K), 0.1494200 secs] [Times: user=0.77 >>> sys=0.03, >>> real=0.15 secs] >>> Thu Feb 4 11:29:03 2010 [19,337,176 K]: [GC 46599.423: [ParNew Desired >>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>> 5719032 >>> bytes, 5719032 total : 153344K->12852K(153344K), 0.0675180 secs] >>> 19465914K->19337176K(27245952K), 0.0676670 secs] [Times: user=0.46 >>> sys=0.02, >>> real=0.07 secs] >>> Thu Feb 4 11:29:03 2010 [19,338,256 K]: [GC 46599.754: [ParNew Desired >>> survivor size 8716288 bytes, new threshold 2 (max 4) - age 1: >>> 6783616 >>> bytes, 6783616 total - age 2: 5570080 bytes, 12353696 total : >>> 149172K->13931K(153344K), 0.0650520 secs] >>> 19473496K->19338256K(27245952K), >>> 0.0652060 secs] [Times: user=0.42 sys=0.01, real=0.07 secs] >>> Thu Feb 4 11:29:04 2010 [19,340,264 K]: [GC 46600.144: [ParNew Desired >>> survivor size 8716288 bytes, new threshold 2 (max 4) - age 1: >>> 2190136 >>> bytes, 2190136 total - age 2: 6762648 bytes, 8952784 total : >>> 150251K->10375K(153344K), 0.0709470 secs] >>> 19474576K->19340264K(27245952K), >>> 0.0710920 secs] [Times: user=0.48 sys=0.01, real=0.07 secs] >>> Thu Feb 4 11:29:17 2010 [19,345,928 K]: [GC 46613.628: [ParNew Desired >>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>> 760488 >>> bytes, 760488 total - age 2: 2173472 bytes, 2933960 total : >>> 146695K->9279K(153344K), 0.0689390 secs] 19476584K->19345928K(27245952K), >>> 0.0691070 secs] [Times: user=0.45 sys=0.00, real=0.07 secs] >>> Thu Feb 4 11:29:17 2010: [CMS-concurrent-mark: 15.853/18.260 secs] >>> [Times: >>> user=29.26 sys=0.55, real=18.26 secs] >>> Thu Feb 4 11:29:17 2010: [CMS-concurrent-preclean-start] >>> .... >>> >>> Any idea what may causing these pauses and what possible JVM tuning would >>> be >>> neccessary to avoid these pauses? Your help greatly appreciated. >>> >>> Thank you very much. >>> >>> Atul >> > http://old.nabble.com/file/p27463304/long.pause.in.cms.gc.log > long.pause.in.cms.gc.log From Jon.Masamitsu at Sun.COM Fri Feb 5 16:08:48 2010 From: Jon.Masamitsu at Sun.COM (Jon Masamitsu) Date: Fri, 05 Feb 2010 08:08:48 -0800 Subject: long unaccounted pause during concurrent mark phase of ParNew cycle In-Reply-To: <27463304.post@talk.nabble.com> References: <27459365.post@talk.nabble.com> <4B6B3D74.90403@Sun.COM> <27463304.post@talk.nabble.com> Message-ID: <4B6C4290.4000907@Sun.COM> Atul, You mentioned that you were on jdk6 update 2 initially. Did you see this behavior with earlier releases? Jon On 02/04/10 20:02, atulksh wrote: > Hi Jon, > I have some more (probably more precise) information for you. I have been > able to reproduce this issue in our test environment (up till now we were > only seeing it in our production setup). Along with the parameters that I > have specified above I added the following. > > > > After loading up the app server (it is Tomcat 6.0.14) I got a pause during > the CMS cycle of about 32 seconds. The way I noticed the pause is like this. > I kept on running a script that does some basic page refresh (small data > page) in the application continously. This is like a ping to the server. > Then I parallely do operations that load up the memory in the application > server. These are the real hits that users do in whom memory requirement is > usually high and that drives the app server to the CMS threshold. > > In my case once the CMS kicked in, I saw that the period where the > applilcation was in the "pause" no requests of my script were processed. > They were just queued up to Tomcat - meaning Apache got those requests and > logged that they were sent to Tomcat. But Tomcat did not register of > receiving them. Then after the app went out of the "pause" then everything > came back and all these pending requests got processed - like the app just > had been unfrozen. > > During this freeze, what I see in the GC log is that the CMS cycle is almost > at the end of the concurrent mark phase. It is doing the GC is doing the > "perm space scanning". The pause exactly coincides with the time reported > for the "perm space scanning". I am attaching the full gc log snippt right > from the start of the CMS cycle. Below is just the place where the pause > appears. > > 28676.784: [GC 28676.784: [ParNew > Desired survivor size 4358144 bytes, new threshold 2 (max 4) > - age 1: 3742608 bytes, 3742608 total > - age 2: 3451152 bytes, 7193760 total > : 76672K->8512K(76672K), 0.2045430 secs] 20262376K->20197311K(27254464K), > 0.2048030 secs] [Times: user=0.77 sys=0.01, real=0.21 secs] > Total time for which application threads were stopped: 0.2055080 seconds > Finished perm space scanning in 2th thread: 0.367 sec > Finished perm space scanning in 0th thread: 0.362 sec > Application time: 0.1481860 seconds > 28708.626: [GC 28708.627: [ParNew > Desired survivor size 4358144 bytes, new threshold 2 (max 4) > - age 1: 2079760 bytes, 2079760 total > - age 2: 3657248 bytes, 5737008 total > : 76672K->6795K(76672K), 0.1937080 secs] 20265471K->20198965K(27254464K), > 0.1939480 secs] [Times: user=0.74 sys=0.00, real=0.20 secs] > Total time for which application threads were stopped: 0.1948820 seconds > Finished perm space scanning in 3th thread: 32.153 sec > Finished perm space scanning in 1th thread: 32.097 sec > Finished work stealing in 1th thread: 0.000 sec > Finished work stealing in 3th thread: 0.002 sec > Finished work stealing in 2th thread: 31.791 sec > Finished work stealing in 0th thread: 31.791 sec > 28708.825: [CMS-concurrent-mark: 88.906/158.952 secs] (CMS-concurrent-mark > yielded 454 times) > [Times: user=544.33 sys=4.59, real=158.95 secs] > > What is "perm space scanning" and is this where the gc would freeze the > application (STW type). Is this a bug or something that is expected? > > Please let me know. This has become quite critical for our application. > Please see the full gc log file attached. Your help very much appreciated. > > Thank you, > > Atul > > > Jon Masamitsu wrote: >> Atul, >> >> Do you mean the 36 secs here? >> >> > Thu Feb 4 10:13:35 2010: [CMS-concurrent-mark: 36.272/36.715 secs] >> [Times: >> > user=56.34 sys=0.34, real=36.71 secs] >> >> That is a concurrent phase (i.e., the application threads >> are still running although they have to share the hardware >> threads with the concurrent GC threads). >> >> Jon >> >> On 02/04/10 12:37, atulksh wrote: >>> Hi, >>> We have an application that is freezing (pausing) for a very long (tens >>> of >>> seconds) time during the concurrent mark phase of the ParNew cycle. This >>> pause is consistently seen when there is good amount of activity on the >>> server. We tried to upgrade the JVM (JRE 6) from Update 02 to Update 17 >>> keeping all the parameters same, but the pauses did not go away (they >>> became >>> more consistent and slightly longer). We even tried to enable >>> CompressedOops >>> option, thinking that that would reduce memory requirements, but that has >>> not helped. >>> >>> We are running on 64bit SUSE 10.1 platform with 32Gb of RAM and 8 core >>> machine. 26g is allocated to the java process. The full set of VM >>> parameters >>> are here >>> >>> >>> -Xmx26g >>> -Xms26g >>> -server >>> -XX:+UseConcMarkSweepGC >>> -XX:PermSize=256m >>> -XX:MaxPermSize=256m >>> -XX:+PrintTenuringDistribution >>> -XX:+PrintGCDetails >>> -XX:+PrintGCTimeStamps >>> -XX:CMSInitiatingOccupancyFraction=70 >>> -XX:-CMSParallelRemarkEnabled >>> >>> The nature of the pause is like this. All the application threads that >>> are >>> being processed seem to stop responding. New requests that come to our >>> app >>> server are not processed and get queued. When the pause ends then >>> everything >>> frees up and seems to go on normally henceforth. In the GC you see a gap >>> between the ParNews. The following are two examples of the pauses. Notice >>> the pause from 10:13:06 to 10:13:35 (29 seconds) in the first one and >>> 11:29:04 to 11:29:17 (13 seconds) in the second one. These pauses only >>> occur >>> in the concurrent mark phase. They do not seem to occur at any other time >>> or >>> anywhere else in the CMS cycle. >>> >>> .... >>> Thu Feb 4 10:12:59 2010: [GC [1 CMS-initial-mark: 18976170K(27092608K)] >>> 18989786K(27245952K), 0.0059140 secs] [Times: user=0.01 sys=0.00, >>> real=0.01 >>> secs] >>> Thu Feb 4 10:12:59 2010: [CMS-concurrent-mark-start] >>> Thu Feb 4 10:12:59 2010 [18,980,009 K]: [GC 42035.293: [ParNew Desired >>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>> 1833064 >>> bytes, 1833064 total - age 2: 18472 bytes, 1851536 total : >>> 148315K->3839K(153344K), 0.0400230 secs] 19124486K->18980009K(27245952K), >>> 0.0401580 secs] [Times: user=0.25 sys=0.00, real=0.04 secs] >>> Thu Feb 4 10:12:59 2010 [18,979,036 K]: [GC 42035.415: [ParNew Desired >>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>> 2068376 >>> bytes, 2068376 total - age 2: 23792 bytes, 2092168 total - >>> age >>> 3: 18472 bytes, 2110640 total : 140159K->2866K(153344K), >>> 0.0352200 >>> secs] 19116329K->18979036K(27245952K), 0.0353520 secs] [Times: user=0.29 >>> sys=0.00, real=0.03 secs] >>> Thu Feb 4 10:12:59 2010 [18,979,103 K]: [GC 42035.535: [ParNew Desired >>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>> 1782608 >>> bytes, 1782608 total - age 2: 22032 bytes, 1804640 total - >>> age >>> 3: 23792 bytes, 1828432 total - age 4: 18472 bytes, >>> 1846904 total : 139186K->2933K(153344K), 0.0498480 secs] >>> 19115356K->18979103K(27245952K), 0.0499850 secs] [Times: user=0.24 >>> sys=0.00, >>> real=0.05 secs] >>> Thu Feb 4 10:12:59 2010 [18,979,181 K]: [GC 42035.673: [ParNew Desired >>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>> 2161000 >>> bytes, 2161000 total - age 2: 22032 bytes, 2183032 total - >>> age >>> 3: 22032 bytes, 2205064 total - age 4: 23792 bytes, >>> 2228856 total : 139253K->2992K(153344K), 0.0400420 secs] >>> 19115423K->18979181K(27245952K), 0.0401730 secs] [Times: user=0.26 >>> sys=0.00, >>> real=0.04 secs] >>> Thu Feb 4 10:13:06 2010 [18,979,876 K]: [GC 42042.243: [ParNew Desired >>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>> 2113272 >>> bytes, 2113272 total - age 2: 23792 bytes, 2137064 total - >>> age >>> 3: 22032 bytes, 2159096 total - age 4: 22032 bytes, >>> 2181128 total : 138855K->3662K(153344K), 0.0465500 secs] >>> 19115044K->18979876K(27245952K), 0.0466840 secs] [Times: user=0.27 >>> sys=0.00, >>> real=0.04 secs] >>> Thu Feb 4 10:13:35 2010 [18,979,051 K]: [GC 42071.809: [ParNew Desired >>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>> 219056 >>> bytes, 219056 total - age 2: 1867632 bytes, 2086688 total - >>> age >>> 3: 23792 bytes, 2110480 total - age 4: 22032 bytes, >>> 2132512 total : 139982K->2814K(153344K), 0.0426920 secs] >>> 19116196K->18979051K(27245952K), 0.0428200 secs] [Times: user=0.28 >>> sys=0.00, >>> real=0.05 secs] >>> Thu Feb 4 10:13:35 2010: [CMS-concurrent-mark: 36.272/36.715 secs] >>> [Times: >>> user=56.34 sys=0.34, real=36.71 secs] >>> .... >>> >>> >>> .... >>> Thu Feb 4 11:28:59 2010: [GC [1 CMS-initial-mark: 19012252K(27092608K)] >>> 19032719K(27245952K), 0.0279440 secs] [Times: user=0.02 sys=0.00, >>> real=0.03 >>> secs] >>> Thu Feb 4 11:28:59 2010: [CMS-concurrent-mark-start] >>> Thu Feb 4 11:28:59 2010 [19,154,661 K]: [GC 46595.614: [ParNew Desired >>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>> 17431976 >>> bytes, 17431976 total : 153344K->17024K(153344K), 0.9259390 secs] >>> 19165596K->19154661K(27245952K), 0.9260890 secs] [Times: user=1.23 >>> sys=0.05, >>> real=0.93 secs] >>> Thu Feb 4 11:29:00 2010 [19,199,696 K]: [GC 46596.973: [ParNew Desired >>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>> 17448432 >>> bytes, 17448432 total : 153344K->17024K(153344K), 0.3054460 secs] >>> 19290981K->19199696K(27245952K), 0.3055930 secs] [Times: user=0.79 >>> sys=0.07, >>> real=0.31 secs] >>> Thu Feb 4 11:29:01 2010 [19,229,358 K]: [GC 46597.722: [ParNew Desired >>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>> 10331408 >>> bytes, 10331408 total : 153344K->17024K(153344K), 0.1449040 secs] >>> 19336016K->19229358K(27245952K), 0.1450630 secs] [Times: user=0.59 >>> sys=0.02, >>> real=0.15 secs] >>> Thu Feb 4 11:29:02 2010 [19,240,015 K]: [GC 46598.038: [ParNew Desired >>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>> 17354120 >>> bytes, 17354120 total : 153344K->17024K(153344K), 0.0935440 secs] >>> 19365678K->19240015K(27245952K), 0.0936870 secs] [Times: user=0.59 >>> sys=0.00, >>> real=0.10 secs] >>> Thu Feb 4 11:29:02 2010 [19,302,509 K]: [GC 46598.347: [ParNew Desired >>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>> 17432360 >>> bytes, 17432360 total : 151519K->17024K(153344K), 0.4619430 secs] >>> 19374510K->19302509K(27245952K), 0.4620890 secs] [Times: user=1.19 >>> sys=0.02, >>> real=0.46 secs] >>> Thu Feb 4 11:29:03 2010 [19,329,594 K]: [GC 46599.030: [ParNew Desired >>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>> 11933296 >>> bytes, 11933296 total : 153344K->17024K(153344K), 0.1492670 secs] >>> 19438829K->19329594K(27245952K), 0.1494200 secs] [Times: user=0.77 >>> sys=0.03, >>> real=0.15 secs] >>> Thu Feb 4 11:29:03 2010 [19,337,176 K]: [GC 46599.423: [ParNew Desired >>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>> 5719032 >>> bytes, 5719032 total : 153344K->12852K(153344K), 0.0675180 secs] >>> 19465914K->19337176K(27245952K), 0.0676670 secs] [Times: user=0.46 >>> sys=0.02, >>> real=0.07 secs] >>> Thu Feb 4 11:29:03 2010 [19,338,256 K]: [GC 46599.754: [ParNew Desired >>> survivor size 8716288 bytes, new threshold 2 (max 4) - age 1: >>> 6783616 >>> bytes, 6783616 total - age 2: 5570080 bytes, 12353696 total : >>> 149172K->13931K(153344K), 0.0650520 secs] >>> 19473496K->19338256K(27245952K), >>> 0.0652060 secs] [Times: user=0.42 sys=0.01, real=0.07 secs] >>> Thu Feb 4 11:29:04 2010 [19,340,264 K]: [GC 46600.144: [ParNew Desired >>> survivor size 8716288 bytes, new threshold 2 (max 4) - age 1: >>> 2190136 >>> bytes, 2190136 total - age 2: 6762648 bytes, 8952784 total : >>> 150251K->10375K(153344K), 0.0709470 secs] >>> 19474576K->19340264K(27245952K), >>> 0.0710920 secs] [Times: user=0.48 sys=0.01, real=0.07 secs] >>> Thu Feb 4 11:29:17 2010 [19,345,928 K]: [GC 46613.628: [ParNew Desired >>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>> 760488 >>> bytes, 760488 total - age 2: 2173472 bytes, 2933960 total : >>> 146695K->9279K(153344K), 0.0689390 secs] 19476584K->19345928K(27245952K), >>> 0.0691070 secs] [Times: user=0.45 sys=0.00, real=0.07 secs] >>> Thu Feb 4 11:29:17 2010: [CMS-concurrent-mark: 15.853/18.260 secs] >>> [Times: >>> user=29.26 sys=0.55, real=18.26 secs] >>> Thu Feb 4 11:29:17 2010: [CMS-concurrent-preclean-start] >>> .... >>> >>> Any idea what may causing these pauses and what possible JVM tuning would >>> be >>> neccessary to avoid these pauses? Your help greatly appreciated. >>> >>> Thank you very much. >>> >>> Atul >> > http://old.nabble.com/file/p27463304/long.pause.in.cms.gc.log > long.pause.in.cms.gc.log From atulksh at hotmail.com Fri Feb 5 16:42:27 2010 From: atulksh at hotmail.com (atulksh) Date: Fri, 5 Feb 2010 08:42:27 -0800 (PST) Subject: long unaccounted pause during concurrent mark phase of ParNew cycle In-Reply-To: <4B6BB53D.5010800@sun.com> References: <27459365.post@talk.nabble.com> <4B6B3D74.90403@Sun.COM> <27463304.post@talk.nabble.com> <4B6BB53D.5010800@sun.com> Message-ID: <27470522.post@talk.nabble.com> Hi Ramki, I will give as try with that option as you have asked. If I disable the Concurrent Multi Threading in CMS with this flag is it going to not multi-thread any phase in general? We have been using -XX:-CMSParallelRemarkEnabled in our production environment for a long time. After we had upgraded to Update 2 we had seen crashes which were attributed to the remark phase. Using this flag had stopped those crashes. This issue was eventually fixed in Update 4 but after we had tried Update 4 we got more frequent crashes and since we had never tried to update our VM. Now that we have been seeing these pauses, we have been trying to upgrade to Update 17. What we see is our pause behavior did not change a bit from Update 2 to Update 17, so it looks like that is something inherent and has not changed. Did you checkout my last response where I have given more details about our pause? It looks like our pause is coinciding with the perm space scanning occurring at the end of the concurrent mark phase. Any insights as to what normally goes on in the GC at that time (is that STW phase) and if yes what can we do about it? I am happy to provide you with any further information you may want. Your help and insights greatly appreciated. Thank you very much. Atul y.s.ramakrishna wrote: > > Hello Atul -- > > Could you add -XX:-CMSConcurrentMTEnabled to the mix > and see if it affects the behaviour in any manner? > > PS: I notice that you are using -XX:-CMSParallelRemarkEnabled. > What was the reason for using that flag? > > thanks. > -- ramki > > atulksh wrote: >> Hi Jon, >> I have some more (probably more precise) information for you. I have been >> able to reproduce this issue in our test environment (up till now we were >> only seeing it in our production setup). Along with the parameters that I >> have specified above I added the following. >> >> >> >> After loading up the app server (it is Tomcat 6.0.14) I got a pause >> during >> the CMS cycle of about 32 seconds. The way I noticed the pause is like >> this. >> I kept on running a script that does some basic page refresh (small data >> page) in the application continously. This is like a ping to the server. >> Then I parallely do operations that load up the memory in the application >> server. These are the real hits that users do in whom memory requirement >> is >> usually high and that drives the app server to the CMS threshold. >> >> In my case once the CMS kicked in, I saw that the period where the >> applilcation was in the "pause" no requests of my script were processed. >> They were just queued up to Tomcat - meaning Apache got those requests >> and >> logged that they were sent to Tomcat. But Tomcat did not register of >> receiving them. Then after the app went out of the "pause" then >> everything >> came back and all these pending requests got processed - like the app >> just >> had been unfrozen. >> >> During this freeze, what I see in the GC log is that the CMS cycle is >> almost >> at the end of the concurrent mark phase. It is doing the GC is doing the >> "perm space scanning". The pause exactly coincides with the time reported >> for the "perm space scanning". I am attaching the full gc log snippt >> right >> from the start of the CMS cycle. Below is just the place where the pause >> appears. >> >> 28676.784: [GC 28676.784: [ParNew >> Desired survivor size 4358144 bytes, new threshold 2 (max 4) >> - age 1: 3742608 bytes, 3742608 total >> - age 2: 3451152 bytes, 7193760 total >> : 76672K->8512K(76672K), 0.2045430 secs] 20262376K->20197311K(27254464K), >> 0.2048030 secs] [Times: user=0.77 sys=0.01, real=0.21 secs] >> Total time for which application threads were stopped: 0.2055080 seconds >> Finished perm space scanning in 2th thread: 0.367 sec >> Finished perm space scanning in 0th thread: 0.362 sec >> Application time: 0.1481860 seconds >> 28708.626: [GC 28708.627: [ParNew >> Desired survivor size 4358144 bytes, new threshold 2 (max 4) >> - age 1: 2079760 bytes, 2079760 total >> - age 2: 3657248 bytes, 5737008 total >> : 76672K->6795K(76672K), 0.1937080 secs] 20265471K->20198965K(27254464K), >> 0.1939480 secs] [Times: user=0.74 sys=0.00, real=0.20 secs] >> Total time for which application threads were stopped: 0.1948820 seconds >> Finished perm space scanning in 3th thread: 32.153 sec >> Finished perm space scanning in 1th thread: 32.097 sec >> Finished work stealing in 1th thread: 0.000 sec >> Finished work stealing in 3th thread: 0.002 sec >> Finished work stealing in 2th thread: 31.791 sec >> Finished work stealing in 0th thread: 31.791 sec >> 28708.825: [CMS-concurrent-mark: 88.906/158.952 secs] >> (CMS-concurrent-mark >> yielded 454 times) >> [Times: user=544.33 sys=4.59, real=158.95 secs] >> >> What is "perm space scanning" and is this where the gc would freeze the >> application (STW type). Is this a bug or something that is expected? >> >> Please let me know. This has become quite critical for our application. >> Please see the full gc log file attached. Your help very much >> appreciated. >> >> Thank you, >> >> Atul >> >> >> Jon Masamitsu wrote: >>> Atul, >>> >>> Do you mean the 36 secs here? >>> >>> > Thu Feb 4 10:13:35 2010: [CMS-concurrent-mark: 36.272/36.715 secs] >>> [Times: >>> > user=56.34 sys=0.34, real=36.71 secs] >>> >>> That is a concurrent phase (i.e., the application threads >>> are still running although they have to share the hardware >>> threads with the concurrent GC threads). >>> >>> Jon >>> >>> On 02/04/10 12:37, atulksh wrote: >>>> Hi, >>>> We have an application that is freezing (pausing) for a very long (tens >>>> of >>>> seconds) time during the concurrent mark phase of the ParNew cycle. >>>> This >>>> pause is consistently seen when there is good amount of activity on the >>>> server. We tried to upgrade the JVM (JRE 6) from Update 02 to Update 17 >>>> keeping all the parameters same, but the pauses did not go away (they >>>> became >>>> more consistent and slightly longer). We even tried to enable >>>> CompressedOops >>>> option, thinking that that would reduce memory requirements, but that >>>> has >>>> not helped. >>>> >>>> We are running on 64bit SUSE 10.1 platform with 32Gb of RAM and 8 core >>>> machine. 26g is allocated to the java process. The full set of VM >>>> parameters >>>> are here >>>> >>>> >>>> -Xmx26g >>>> -Xms26g >>>> -server >>>> -XX:+UseConcMarkSweepGC >>>> -XX:PermSize=256m >>>> -XX:MaxPermSize=256m >>>> -XX:+PrintTenuringDistribution >>>> -XX:+PrintGCDetails >>>> -XX:+PrintGCTimeStamps >>>> -XX:CMSInitiatingOccupancyFraction=70 >>>> -XX:-CMSParallelRemarkEnabled >>>> >>>> The nature of the pause is like this. All the application threads that >>>> are >>>> being processed seem to stop responding. New requests that come to our >>>> app >>>> server are not processed and get queued. When the pause ends then >>>> everything >>>> frees up and seems to go on normally henceforth. In the GC you see a >>>> gap >>>> between the ParNews. The following are two examples of the pauses. >>>> Notice >>>> the pause from 10:13:06 to 10:13:35 (29 seconds) in the first one and >>>> 11:29:04 to 11:29:17 (13 seconds) in the second one. These pauses only >>>> occur >>>> in the concurrent mark phase. They do not seem to occur at any other >>>> time >>>> or >>>> anywhere else in the CMS cycle. >>>> >>>> .... >>>> Thu Feb 4 10:12:59 2010: [GC [1 CMS-initial-mark: >>>> 18976170K(27092608K)] >>>> 18989786K(27245952K), 0.0059140 secs] [Times: user=0.01 sys=0.00, >>>> real=0.01 >>>> secs] >>>> Thu Feb 4 10:12:59 2010: [CMS-concurrent-mark-start] >>>> Thu Feb 4 10:12:59 2010 [18,980,009 K]: [GC 42035.293: [ParNew Desired >>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>> 1833064 >>>> bytes, 1833064 total - age 2: 18472 bytes, 1851536 total : >>>> 148315K->3839K(153344K), 0.0400230 secs] >>>> 19124486K->18980009K(27245952K), >>>> 0.0401580 secs] [Times: user=0.25 sys=0.00, real=0.04 secs] >>>> Thu Feb 4 10:12:59 2010 [18,979,036 K]: [GC 42035.415: [ParNew Desired >>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>> 2068376 >>>> bytes, 2068376 total - age 2: 23792 bytes, 2092168 total - >>>> age >>>> 3: 18472 bytes, 2110640 total : 140159K->2866K(153344K), >>>> 0.0352200 >>>> secs] 19116329K->18979036K(27245952K), 0.0353520 secs] [Times: >>>> user=0.29 >>>> sys=0.00, real=0.03 secs] >>>> Thu Feb 4 10:12:59 2010 [18,979,103 K]: [GC 42035.535: [ParNew Desired >>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>> 1782608 >>>> bytes, 1782608 total - age 2: 22032 bytes, 1804640 total - >>>> age >>>> 3: 23792 bytes, 1828432 total - age 4: 18472 bytes, >>>> 1846904 total : 139186K->2933K(153344K), 0.0498480 secs] >>>> 19115356K->18979103K(27245952K), 0.0499850 secs] [Times: user=0.24 >>>> sys=0.00, >>>> real=0.05 secs] >>>> Thu Feb 4 10:12:59 2010 [18,979,181 K]: [GC 42035.673: [ParNew Desired >>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>> 2161000 >>>> bytes, 2161000 total - age 2: 22032 bytes, 2183032 total - >>>> age >>>> 3: 22032 bytes, 2205064 total - age 4: 23792 bytes, >>>> 2228856 total : 139253K->2992K(153344K), 0.0400420 secs] >>>> 19115423K->18979181K(27245952K), 0.0401730 secs] [Times: user=0.26 >>>> sys=0.00, >>>> real=0.04 secs] >>>> Thu Feb 4 10:13:06 2010 [18,979,876 K]: [GC 42042.243: [ParNew Desired >>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>> 2113272 >>>> bytes, 2113272 total - age 2: 23792 bytes, 2137064 total - >>>> age >>>> 3: 22032 bytes, 2159096 total - age 4: 22032 bytes, >>>> 2181128 total : 138855K->3662K(153344K), 0.0465500 secs] >>>> 19115044K->18979876K(27245952K), 0.0466840 secs] [Times: user=0.27 >>>> sys=0.00, >>>> real=0.04 secs] >>>> Thu Feb 4 10:13:35 2010 [18,979,051 K]: [GC 42071.809: [ParNew Desired >>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>> 219056 >>>> bytes, 219056 total - age 2: 1867632 bytes, 2086688 total - >>>> age >>>> 3: 23792 bytes, 2110480 total - age 4: 22032 bytes, >>>> 2132512 total : 139982K->2814K(153344K), 0.0426920 secs] >>>> 19116196K->18979051K(27245952K), 0.0428200 secs] [Times: user=0.28 >>>> sys=0.00, >>>> real=0.05 secs] >>>> Thu Feb 4 10:13:35 2010: [CMS-concurrent-mark: 36.272/36.715 secs] >>>> [Times: >>>> user=56.34 sys=0.34, real=36.71 secs] >>>> .... >>>> >>>> >>>> .... >>>> Thu Feb 4 11:28:59 2010: [GC [1 CMS-initial-mark: >>>> 19012252K(27092608K)] >>>> 19032719K(27245952K), 0.0279440 secs] [Times: user=0.02 sys=0.00, >>>> real=0.03 >>>> secs] >>>> Thu Feb 4 11:28:59 2010: [CMS-concurrent-mark-start] >>>> Thu Feb 4 11:28:59 2010 [19,154,661 K]: [GC 46595.614: [ParNew Desired >>>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>>> 17431976 >>>> bytes, 17431976 total : 153344K->17024K(153344K), 0.9259390 secs] >>>> 19165596K->19154661K(27245952K), 0.9260890 secs] [Times: user=1.23 >>>> sys=0.05, >>>> real=0.93 secs] >>>> Thu Feb 4 11:29:00 2010 [19,199,696 K]: [GC 46596.973: [ParNew Desired >>>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>>> 17448432 >>>> bytes, 17448432 total : 153344K->17024K(153344K), 0.3054460 secs] >>>> 19290981K->19199696K(27245952K), 0.3055930 secs] [Times: user=0.79 >>>> sys=0.07, >>>> real=0.31 secs] >>>> Thu Feb 4 11:29:01 2010 [19,229,358 K]: [GC 46597.722: [ParNew Desired >>>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>>> 10331408 >>>> bytes, 10331408 total : 153344K->17024K(153344K), 0.1449040 secs] >>>> 19336016K->19229358K(27245952K), 0.1450630 secs] [Times: user=0.59 >>>> sys=0.02, >>>> real=0.15 secs] >>>> Thu Feb 4 11:29:02 2010 [19,240,015 K]: [GC 46598.038: [ParNew Desired >>>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>>> 17354120 >>>> bytes, 17354120 total : 153344K->17024K(153344K), 0.0935440 secs] >>>> 19365678K->19240015K(27245952K), 0.0936870 secs] [Times: user=0.59 >>>> sys=0.00, >>>> real=0.10 secs] >>>> Thu Feb 4 11:29:02 2010 [19,302,509 K]: [GC 46598.347: [ParNew Desired >>>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>>> 17432360 >>>> bytes, 17432360 total : 151519K->17024K(153344K), 0.4619430 secs] >>>> 19374510K->19302509K(27245952K), 0.4620890 secs] [Times: user=1.19 >>>> sys=0.02, >>>> real=0.46 secs] >>>> Thu Feb 4 11:29:03 2010 [19,329,594 K]: [GC 46599.030: [ParNew Desired >>>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>>> 11933296 >>>> bytes, 11933296 total : 153344K->17024K(153344K), 0.1492670 secs] >>>> 19438829K->19329594K(27245952K), 0.1494200 secs] [Times: user=0.77 >>>> sys=0.03, >>>> real=0.15 secs] >>>> Thu Feb 4 11:29:03 2010 [19,337,176 K]: [GC 46599.423: [ParNew Desired >>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>> 5719032 >>>> bytes, 5719032 total : 153344K->12852K(153344K), 0.0675180 secs] >>>> 19465914K->19337176K(27245952K), 0.0676670 secs] [Times: user=0.46 >>>> sys=0.02, >>>> real=0.07 secs] >>>> Thu Feb 4 11:29:03 2010 [19,338,256 K]: [GC 46599.754: [ParNew Desired >>>> survivor size 8716288 bytes, new threshold 2 (max 4) - age 1: >>>> 6783616 >>>> bytes, 6783616 total - age 2: 5570080 bytes, 12353696 total : >>>> 149172K->13931K(153344K), 0.0650520 secs] >>>> 19473496K->19338256K(27245952K), >>>> 0.0652060 secs] [Times: user=0.42 sys=0.01, real=0.07 secs] >>>> Thu Feb 4 11:29:04 2010 [19,340,264 K]: [GC 46600.144: [ParNew Desired >>>> survivor size 8716288 bytes, new threshold 2 (max 4) - age 1: >>>> 2190136 >>>> bytes, 2190136 total - age 2: 6762648 bytes, 8952784 total : >>>> 150251K->10375K(153344K), 0.0709470 secs] >>>> 19474576K->19340264K(27245952K), >>>> 0.0710920 secs] [Times: user=0.48 sys=0.01, real=0.07 secs] >>>> Thu Feb 4 11:29:17 2010 [19,345,928 K]: [GC 46613.628: [ParNew Desired >>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>> 760488 >>>> bytes, 760488 total - age 2: 2173472 bytes, 2933960 total : >>>> 146695K->9279K(153344K), 0.0689390 secs] >>>> 19476584K->19345928K(27245952K), >>>> 0.0691070 secs] [Times: user=0.45 sys=0.00, real=0.07 secs] >>>> Thu Feb 4 11:29:17 2010: [CMS-concurrent-mark: 15.853/18.260 secs] >>>> [Times: >>>> user=29.26 sys=0.55, real=18.26 secs] >>>> Thu Feb 4 11:29:17 2010: [CMS-concurrent-preclean-start] >>>> .... >>>> >>>> Any idea what may causing these pauses and what possible JVM tuning >>>> would >>>> be >>>> neccessary to avoid these pauses? Your help greatly appreciated. >>>> >>>> Thank you very much. >>>> >>>> Atul >>> >> http://old.nabble.com/file/p27463304/long.pause.in.cms.gc.log >> long.pause.in.cms.gc.log > > > -- View this message in context: http://old.nabble.com/long-unaccounted-pause-during-concurrent-mark-phase-of-ParNew-cycle-tp27459365p27470522.html Sent from the OpenJDK Hotspot Garbage Collection mailing list archive at Nabble.com. From atulksh at hotmail.com Fri Feb 5 16:47:51 2010 From: atulksh at hotmail.com (atulksh) Date: Fri, 5 Feb 2010 08:47:51 -0800 (PST) Subject: long unaccounted pause during concurrent mark phase of ParNew cycle In-Reply-To: <4B6C4290.4000907@Sun.COM> References: <27459365.post@talk.nabble.com> <4B6B3D74.90403@Sun.COM> <27463304.post@talk.nabble.com> <4B6C4290.4000907@Sun.COM> Message-ID: <27470597.post@talk.nabble.com> Hi Jon, We are currently still using Update 2 on our servers. We see the same kind of pause behavior on Update 2 as well as Update 17. The pause behavior does not seem to change between these two versions. Thank you, Atul Jon Masamitsu wrote: > > Atul, > > You mentioned that you were on jdk6 update 2 initially. > Did you see this behavior with earlier releases? > > Jon > > On 02/04/10 20:02, atulksh wrote: >> Hi Jon, >> I have some more (probably more precise) information for you. I have been >> able to reproduce this issue in our test environment (up till now we were >> only seeing it in our production setup). Along with the parameters that I >> have specified above I added the following. >> >> >> >> After loading up the app server (it is Tomcat 6.0.14) I got a pause >> during >> the CMS cycle of about 32 seconds. The way I noticed the pause is like >> this. >> I kept on running a script that does some basic page refresh (small data >> page) in the application continously. This is like a ping to the server. >> Then I parallely do operations that load up the memory in the application >> server. These are the real hits that users do in whom memory requirement >> is >> usually high and that drives the app server to the CMS threshold. >> >> In my case once the CMS kicked in, I saw that the period where the >> applilcation was in the "pause" no requests of my script were processed. >> They were just queued up to Tomcat - meaning Apache got those requests >> and >> logged that they were sent to Tomcat. But Tomcat did not register of >> receiving them. Then after the app went out of the "pause" then >> everything >> came back and all these pending requests got processed - like the app >> just >> had been unfrozen. >> >> During this freeze, what I see in the GC log is that the CMS cycle is >> almost >> at the end of the concurrent mark phase. It is doing the GC is doing the >> "perm space scanning". The pause exactly coincides with the time reported >> for the "perm space scanning". I am attaching the full gc log snippt >> right >> from the start of the CMS cycle. Below is just the place where the pause >> appears. >> >> 28676.784: [GC 28676.784: [ParNew >> Desired survivor size 4358144 bytes, new threshold 2 (max 4) >> - age 1: 3742608 bytes, 3742608 total >> - age 2: 3451152 bytes, 7193760 total >> : 76672K->8512K(76672K), 0.2045430 secs] 20262376K->20197311K(27254464K), >> 0.2048030 secs] [Times: user=0.77 sys=0.01, real=0.21 secs] >> Total time for which application threads were stopped: 0.2055080 seconds >> Finished perm space scanning in 2th thread: 0.367 sec >> Finished perm space scanning in 0th thread: 0.362 sec >> Application time: 0.1481860 seconds >> 28708.626: [GC 28708.627: [ParNew >> Desired survivor size 4358144 bytes, new threshold 2 (max 4) >> - age 1: 2079760 bytes, 2079760 total >> - age 2: 3657248 bytes, 5737008 total >> : 76672K->6795K(76672K), 0.1937080 secs] 20265471K->20198965K(27254464K), >> 0.1939480 secs] [Times: user=0.74 sys=0.00, real=0.20 secs] >> Total time for which application threads were stopped: 0.1948820 seconds >> Finished perm space scanning in 3th thread: 32.153 sec >> Finished perm space scanning in 1th thread: 32.097 sec >> Finished work stealing in 1th thread: 0.000 sec >> Finished work stealing in 3th thread: 0.002 sec >> Finished work stealing in 2th thread: 31.791 sec >> Finished work stealing in 0th thread: 31.791 sec >> 28708.825: [CMS-concurrent-mark: 88.906/158.952 secs] >> (CMS-concurrent-mark >> yielded 454 times) >> [Times: user=544.33 sys=4.59, real=158.95 secs] >> >> What is "perm space scanning" and is this where the gc would freeze the >> application (STW type). Is this a bug or something that is expected? >> >> Please let me know. This has become quite critical for our application. >> Please see the full gc log file attached. Your help very much >> appreciated. >> >> Thank you, >> >> Atul >> >> >> Jon Masamitsu wrote: >>> Atul, >>> >>> Do you mean the 36 secs here? >>> >>> > Thu Feb 4 10:13:35 2010: [CMS-concurrent-mark: 36.272/36.715 secs] >>> [Times: >>> > user=56.34 sys=0.34, real=36.71 secs] >>> >>> That is a concurrent phase (i.e., the application threads >>> are still running although they have to share the hardware >>> threads with the concurrent GC threads). >>> >>> Jon >>> >>> On 02/04/10 12:37, atulksh wrote: >>>> Hi, >>>> We have an application that is freezing (pausing) for a very long (tens >>>> of >>>> seconds) time during the concurrent mark phase of the ParNew cycle. >>>> This >>>> pause is consistently seen when there is good amount of activity on the >>>> server. We tried to upgrade the JVM (JRE 6) from Update 02 to Update 17 >>>> keeping all the parameters same, but the pauses did not go away (they >>>> became >>>> more consistent and slightly longer). We even tried to enable >>>> CompressedOops >>>> option, thinking that that would reduce memory requirements, but that >>>> has >>>> not helped. >>>> >>>> We are running on 64bit SUSE 10.1 platform with 32Gb of RAM and 8 core >>>> machine. 26g is allocated to the java process. The full set of VM >>>> parameters >>>> are here >>>> >>>> >>>> -Xmx26g >>>> -Xms26g >>>> -server >>>> -XX:+UseConcMarkSweepGC >>>> -XX:PermSize=256m >>>> -XX:MaxPermSize=256m >>>> -XX:+PrintTenuringDistribution >>>> -XX:+PrintGCDetails >>>> -XX:+PrintGCTimeStamps >>>> -XX:CMSInitiatingOccupancyFraction=70 >>>> -XX:-CMSParallelRemarkEnabled >>>> >>>> The nature of the pause is like this. All the application threads that >>>> are >>>> being processed seem to stop responding. New requests that come to our >>>> app >>>> server are not processed and get queued. When the pause ends then >>>> everything >>>> frees up and seems to go on normally henceforth. In the GC you see a >>>> gap >>>> between the ParNews. The following are two examples of the pauses. >>>> Notice >>>> the pause from 10:13:06 to 10:13:35 (29 seconds) in the first one and >>>> 11:29:04 to 11:29:17 (13 seconds) in the second one. These pauses only >>>> occur >>>> in the concurrent mark phase. They do not seem to occur at any other >>>> time >>>> or >>>> anywhere else in the CMS cycle. >>>> >>>> .... >>>> Thu Feb 4 10:12:59 2010: [GC [1 CMS-initial-mark: >>>> 18976170K(27092608K)] >>>> 18989786K(27245952K), 0.0059140 secs] [Times: user=0.01 sys=0.00, >>>> real=0.01 >>>> secs] >>>> Thu Feb 4 10:12:59 2010: [CMS-concurrent-mark-start] >>>> Thu Feb 4 10:12:59 2010 [18,980,009 K]: [GC 42035.293: [ParNew Desired >>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>> 1833064 >>>> bytes, 1833064 total - age 2: 18472 bytes, 1851536 total : >>>> 148315K->3839K(153344K), 0.0400230 secs] >>>> 19124486K->18980009K(27245952K), >>>> 0.0401580 secs] [Times: user=0.25 sys=0.00, real=0.04 secs] >>>> Thu Feb 4 10:12:59 2010 [18,979,036 K]: [GC 42035.415: [ParNew Desired >>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>> 2068376 >>>> bytes, 2068376 total - age 2: 23792 bytes, 2092168 total - >>>> age >>>> 3: 18472 bytes, 2110640 total : 140159K->2866K(153344K), >>>> 0.0352200 >>>> secs] 19116329K->18979036K(27245952K), 0.0353520 secs] [Times: >>>> user=0.29 >>>> sys=0.00, real=0.03 secs] >>>> Thu Feb 4 10:12:59 2010 [18,979,103 K]: [GC 42035.535: [ParNew Desired >>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>> 1782608 >>>> bytes, 1782608 total - age 2: 22032 bytes, 1804640 total - >>>> age >>>> 3: 23792 bytes, 1828432 total - age 4: 18472 bytes, >>>> 1846904 total : 139186K->2933K(153344K), 0.0498480 secs] >>>> 19115356K->18979103K(27245952K), 0.0499850 secs] [Times: user=0.24 >>>> sys=0.00, >>>> real=0.05 secs] >>>> Thu Feb 4 10:12:59 2010 [18,979,181 K]: [GC 42035.673: [ParNew Desired >>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>> 2161000 >>>> bytes, 2161000 total - age 2: 22032 bytes, 2183032 total - >>>> age >>>> 3: 22032 bytes, 2205064 total - age 4: 23792 bytes, >>>> 2228856 total : 139253K->2992K(153344K), 0.0400420 secs] >>>> 19115423K->18979181K(27245952K), 0.0401730 secs] [Times: user=0.26 >>>> sys=0.00, >>>> real=0.04 secs] >>>> Thu Feb 4 10:13:06 2010 [18,979,876 K]: [GC 42042.243: [ParNew Desired >>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>> 2113272 >>>> bytes, 2113272 total - age 2: 23792 bytes, 2137064 total - >>>> age >>>> 3: 22032 bytes, 2159096 total - age 4: 22032 bytes, >>>> 2181128 total : 138855K->3662K(153344K), 0.0465500 secs] >>>> 19115044K->18979876K(27245952K), 0.0466840 secs] [Times: user=0.27 >>>> sys=0.00, >>>> real=0.04 secs] >>>> Thu Feb 4 10:13:35 2010 [18,979,051 K]: [GC 42071.809: [ParNew Desired >>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>> 219056 >>>> bytes, 219056 total - age 2: 1867632 bytes, 2086688 total - >>>> age >>>> 3: 23792 bytes, 2110480 total - age 4: 22032 bytes, >>>> 2132512 total : 139982K->2814K(153344K), 0.0426920 secs] >>>> 19116196K->18979051K(27245952K), 0.0428200 secs] [Times: user=0.28 >>>> sys=0.00, >>>> real=0.05 secs] >>>> Thu Feb 4 10:13:35 2010: [CMS-concurrent-mark: 36.272/36.715 secs] >>>> [Times: >>>> user=56.34 sys=0.34, real=36.71 secs] >>>> .... >>>> >>>> >>>> .... >>>> Thu Feb 4 11:28:59 2010: [GC [1 CMS-initial-mark: >>>> 19012252K(27092608K)] >>>> 19032719K(27245952K), 0.0279440 secs] [Times: user=0.02 sys=0.00, >>>> real=0.03 >>>> secs] >>>> Thu Feb 4 11:28:59 2010: [CMS-concurrent-mark-start] >>>> Thu Feb 4 11:28:59 2010 [19,154,661 K]: [GC 46595.614: [ParNew Desired >>>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>>> 17431976 >>>> bytes, 17431976 total : 153344K->17024K(153344K), 0.9259390 secs] >>>> 19165596K->19154661K(27245952K), 0.9260890 secs] [Times: user=1.23 >>>> sys=0.05, >>>> real=0.93 secs] >>>> Thu Feb 4 11:29:00 2010 [19,199,696 K]: [GC 46596.973: [ParNew Desired >>>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>>> 17448432 >>>> bytes, 17448432 total : 153344K->17024K(153344K), 0.3054460 secs] >>>> 19290981K->19199696K(27245952K), 0.3055930 secs] [Times: user=0.79 >>>> sys=0.07, >>>> real=0.31 secs] >>>> Thu Feb 4 11:29:01 2010 [19,229,358 K]: [GC 46597.722: [ParNew Desired >>>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>>> 10331408 >>>> bytes, 10331408 total : 153344K->17024K(153344K), 0.1449040 secs] >>>> 19336016K->19229358K(27245952K), 0.1450630 secs] [Times: user=0.59 >>>> sys=0.02, >>>> real=0.15 secs] >>>> Thu Feb 4 11:29:02 2010 [19,240,015 K]: [GC 46598.038: [ParNew Desired >>>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>>> 17354120 >>>> bytes, 17354120 total : 153344K->17024K(153344K), 0.0935440 secs] >>>> 19365678K->19240015K(27245952K), 0.0936870 secs] [Times: user=0.59 >>>> sys=0.00, >>>> real=0.10 secs] >>>> Thu Feb 4 11:29:02 2010 [19,302,509 K]: [GC 46598.347: [ParNew Desired >>>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>>> 17432360 >>>> bytes, 17432360 total : 151519K->17024K(153344K), 0.4619430 secs] >>>> 19374510K->19302509K(27245952K), 0.4620890 secs] [Times: user=1.19 >>>> sys=0.02, >>>> real=0.46 secs] >>>> Thu Feb 4 11:29:03 2010 [19,329,594 K]: [GC 46599.030: [ParNew Desired >>>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>>> 11933296 >>>> bytes, 11933296 total : 153344K->17024K(153344K), 0.1492670 secs] >>>> 19438829K->19329594K(27245952K), 0.1494200 secs] [Times: user=0.77 >>>> sys=0.03, >>>> real=0.15 secs] >>>> Thu Feb 4 11:29:03 2010 [19,337,176 K]: [GC 46599.423: [ParNew Desired >>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>> 5719032 >>>> bytes, 5719032 total : 153344K->12852K(153344K), 0.0675180 secs] >>>> 19465914K->19337176K(27245952K), 0.0676670 secs] [Times: user=0.46 >>>> sys=0.02, >>>> real=0.07 secs] >>>> Thu Feb 4 11:29:03 2010 [19,338,256 K]: [GC 46599.754: [ParNew Desired >>>> survivor size 8716288 bytes, new threshold 2 (max 4) - age 1: >>>> 6783616 >>>> bytes, 6783616 total - age 2: 5570080 bytes, 12353696 total : >>>> 149172K->13931K(153344K), 0.0650520 secs] >>>> 19473496K->19338256K(27245952K), >>>> 0.0652060 secs] [Times: user=0.42 sys=0.01, real=0.07 secs] >>>> Thu Feb 4 11:29:04 2010 [19,340,264 K]: [GC 46600.144: [ParNew Desired >>>> survivor size 8716288 bytes, new threshold 2 (max 4) - age 1: >>>> 2190136 >>>> bytes, 2190136 total - age 2: 6762648 bytes, 8952784 total : >>>> 150251K->10375K(153344K), 0.0709470 secs] >>>> 19474576K->19340264K(27245952K), >>>> 0.0710920 secs] [Times: user=0.48 sys=0.01, real=0.07 secs] >>>> Thu Feb 4 11:29:17 2010 [19,345,928 K]: [GC 46613.628: [ParNew Desired >>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>> 760488 >>>> bytes, 760488 total - age 2: 2173472 bytes, 2933960 total : >>>> 146695K->9279K(153344K), 0.0689390 secs] >>>> 19476584K->19345928K(27245952K), >>>> 0.0691070 secs] [Times: user=0.45 sys=0.00, real=0.07 secs] >>>> Thu Feb 4 11:29:17 2010: [CMS-concurrent-mark: 15.853/18.260 secs] >>>> [Times: >>>> user=29.26 sys=0.55, real=18.26 secs] >>>> Thu Feb 4 11:29:17 2010: [CMS-concurrent-preclean-start] >>>> .... >>>> >>>> Any idea what may causing these pauses and what possible JVM tuning >>>> would >>>> be >>>> neccessary to avoid these pauses? Your help greatly appreciated. >>>> >>>> Thank you very much. >>>> >>>> Atul >>> >> http://old.nabble.com/file/p27463304/long.pause.in.cms.gc.log >> long.pause.in.cms.gc.log > > -- View this message in context: http://old.nabble.com/long-unaccounted-pause-during-concurrent-mark-phase-of-ParNew-cycle-tp27459365p27470597.html Sent from the OpenJDK Hotspot Garbage Collection mailing list archive at Nabble.com. From Jon.Masamitsu at Sun.COM Fri Feb 5 17:00:17 2010 From: Jon.Masamitsu at Sun.COM (Jon Masamitsu) Date: Fri, 05 Feb 2010 09:00:17 -0800 Subject: long unaccounted pause during concurrent mark phase of ParNew cycle In-Reply-To: <27470597.post@talk.nabble.com> References: <27459365.post@talk.nabble.com> <4B6B3D74.90403@Sun.COM> <27463304.post@talk.nabble.com> <4B6C4290.4000907@Sun.COM> <27470597.post@talk.nabble.com> Message-ID: <4B6C4EA1.6010008@Sun.COM> On 02/05/10 08:47, atulksh wrote: > Hi Jon, > We are currently still using Update 2 on our servers. We see the same kind > of pause behavior on Update 2 as well as Update 17. The pause behavior does > not seem to change between these two versions. Did you notice this behavior after upgrading to jdk6 update 2 (i.e., did it work OK on a jdk5 release or earlier)? > > Thank you, > > Atul > > > Jon Masamitsu wrote: >> Atul, >> >> You mentioned that you were on jdk6 update 2 initially. >> Did you see this behavior with earlier releases? >> >> Jon >> >> On 02/04/10 20:02, atulksh wrote: >>> Hi Jon, >>> I have some more (probably more precise) information for you. I have been >>> able to reproduce this issue in our test environment (up till now we were >>> only seeing it in our production setup). Along with the parameters that I >>> have specified above I added the following. >>> >>> >>> >>> After loading up the app server (it is Tomcat 6.0.14) I got a pause >>> during >>> the CMS cycle of about 32 seconds. The way I noticed the pause is like >>> this. >>> I kept on running a script that does some basic page refresh (small data >>> page) in the application continously. This is like a ping to the server. >>> Then I parallely do operations that load up the memory in the application >>> server. These are the real hits that users do in whom memory requirement >>> is >>> usually high and that drives the app server to the CMS threshold. >>> >>> In my case once the CMS kicked in, I saw that the period where the >>> applilcation was in the "pause" no requests of my script were processed. >>> They were just queued up to Tomcat - meaning Apache got those requests >>> and >>> logged that they were sent to Tomcat. But Tomcat did not register of >>> receiving them. Then after the app went out of the "pause" then >>> everything >>> came back and all these pending requests got processed - like the app >>> just >>> had been unfrozen. >>> >>> During this freeze, what I see in the GC log is that the CMS cycle is >>> almost >>> at the end of the concurrent mark phase. It is doing the GC is doing the >>> "perm space scanning". The pause exactly coincides with the time reported >>> for the "perm space scanning". I am attaching the full gc log snippt >>> right >>> from the start of the CMS cycle. Below is just the place where the pause >>> appears. >>> >>> 28676.784: [GC 28676.784: [ParNew >>> Desired survivor size 4358144 bytes, new threshold 2 (max 4) >>> - age 1: 3742608 bytes, 3742608 total >>> - age 2: 3451152 bytes, 7193760 total >>> : 76672K->8512K(76672K), 0.2045430 secs] 20262376K->20197311K(27254464K), >>> 0.2048030 secs] [Times: user=0.77 sys=0.01, real=0.21 secs] >>> Total time for which application threads were stopped: 0.2055080 seconds >>> Finished perm space scanning in 2th thread: 0.367 sec >>> Finished perm space scanning in 0th thread: 0.362 sec >>> Application time: 0.1481860 seconds >>> 28708.626: [GC 28708.627: [ParNew >>> Desired survivor size 4358144 bytes, new threshold 2 (max 4) >>> - age 1: 2079760 bytes, 2079760 total >>> - age 2: 3657248 bytes, 5737008 total >>> : 76672K->6795K(76672K), 0.1937080 secs] 20265471K->20198965K(27254464K), >>> 0.1939480 secs] [Times: user=0.74 sys=0.00, real=0.20 secs] >>> Total time for which application threads were stopped: 0.1948820 seconds >>> Finished perm space scanning in 3th thread: 32.153 sec >>> Finished perm space scanning in 1th thread: 32.097 sec >>> Finished work stealing in 1th thread: 0.000 sec >>> Finished work stealing in 3th thread: 0.002 sec >>> Finished work stealing in 2th thread: 31.791 sec >>> Finished work stealing in 0th thread: 31.791 sec >>> 28708.825: [CMS-concurrent-mark: 88.906/158.952 secs] >>> (CMS-concurrent-mark >>> yielded 454 times) >>> [Times: user=544.33 sys=4.59, real=158.95 secs] >>> >>> What is "perm space scanning" and is this where the gc would freeze the >>> application (STW type). Is this a bug or something that is expected? >>> >>> Please let me know. This has become quite critical for our application. >>> Please see the full gc log file attached. Your help very much >>> appreciated. >>> >>> Thank you, >>> >>> Atul >>> >>> >>> Jon Masamitsu wrote: >>>> Atul, >>>> >>>> Do you mean the 36 secs here? >>>> >>>> > Thu Feb 4 10:13:35 2010: [CMS-concurrent-mark: 36.272/36.715 secs] >>>> [Times: >>>> > user=56.34 sys=0.34, real=36.71 secs] >>>> >>>> That is a concurrent phase (i.e., the application threads >>>> are still running although they have to share the hardware >>>> threads with the concurrent GC threads). >>>> >>>> Jon >>>> >>>> On 02/04/10 12:37, atulksh wrote: >>>>> Hi, >>>>> We have an application that is freezing (pausing) for a very long (tens >>>>> of >>>>> seconds) time during the concurrent mark phase of the ParNew cycle. >>>>> This >>>>> pause is consistently seen when there is good amount of activity on the >>>>> server. We tried to upgrade the JVM (JRE 6) from Update 02 to Update 17 >>>>> keeping all the parameters same, but the pauses did not go away (they >>>>> became >>>>> more consistent and slightly longer). We even tried to enable >>>>> CompressedOops >>>>> option, thinking that that would reduce memory requirements, but that >>>>> has >>>>> not helped. >>>>> >>>>> We are running on 64bit SUSE 10.1 platform with 32Gb of RAM and 8 core >>>>> machine. 26g is allocated to the java process. The full set of VM >>>>> parameters >>>>> are here >>>>> >>>>> >>>>> -Xmx26g >>>>> -Xms26g >>>>> -server >>>>> -XX:+UseConcMarkSweepGC >>>>> -XX:PermSize=256m >>>>> -XX:MaxPermSize=256m >>>>> -XX:+PrintTenuringDistribution >>>>> -XX:+PrintGCDetails >>>>> -XX:+PrintGCTimeStamps >>>>> -XX:CMSInitiatingOccupancyFraction=70 >>>>> -XX:-CMSParallelRemarkEnabled >>>>> >>>>> The nature of the pause is like this. All the application threads that >>>>> are >>>>> being processed seem to stop responding. New requests that come to our >>>>> app >>>>> server are not processed and get queued. When the pause ends then >>>>> everything >>>>> frees up and seems to go on normally henceforth. In the GC you see a >>>>> gap >>>>> between the ParNews. The following are two examples of the pauses. >>>>> Notice >>>>> the pause from 10:13:06 to 10:13:35 (29 seconds) in the first one and >>>>> 11:29:04 to 11:29:17 (13 seconds) in the second one. These pauses only >>>>> occur >>>>> in the concurrent mark phase. They do not seem to occur at any other >>>>> time >>>>> or >>>>> anywhere else in the CMS cycle. >>>>> >>>>> .... >>>>> Thu Feb 4 10:12:59 2010: [GC [1 CMS-initial-mark: >>>>> 18976170K(27092608K)] >>>>> 18989786K(27245952K), 0.0059140 secs] [Times: user=0.01 sys=0.00, >>>>> real=0.01 >>>>> secs] >>>>> Thu Feb 4 10:12:59 2010: [CMS-concurrent-mark-start] >>>>> Thu Feb 4 10:12:59 2010 [18,980,009 K]: [GC 42035.293: [ParNew Desired >>>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>>> 1833064 >>>>> bytes, 1833064 total - age 2: 18472 bytes, 1851536 total : >>>>> 148315K->3839K(153344K), 0.0400230 secs] >>>>> 19124486K->18980009K(27245952K), >>>>> 0.0401580 secs] [Times: user=0.25 sys=0.00, real=0.04 secs] >>>>> Thu Feb 4 10:12:59 2010 [18,979,036 K]: [GC 42035.415: [ParNew Desired >>>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>>> 2068376 >>>>> bytes, 2068376 total - age 2: 23792 bytes, 2092168 total - >>>>> age >>>>> 3: 18472 bytes, 2110640 total : 140159K->2866K(153344K), >>>>> 0.0352200 >>>>> secs] 19116329K->18979036K(27245952K), 0.0353520 secs] [Times: >>>>> user=0.29 >>>>> sys=0.00, real=0.03 secs] >>>>> Thu Feb 4 10:12:59 2010 [18,979,103 K]: [GC 42035.535: [ParNew Desired >>>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>>> 1782608 >>>>> bytes, 1782608 total - age 2: 22032 bytes, 1804640 total - >>>>> age >>>>> 3: 23792 bytes, 1828432 total - age 4: 18472 bytes, >>>>> 1846904 total : 139186K->2933K(153344K), 0.0498480 secs] >>>>> 19115356K->18979103K(27245952K), 0.0499850 secs] [Times: user=0.24 >>>>> sys=0.00, >>>>> real=0.05 secs] >>>>> Thu Feb 4 10:12:59 2010 [18,979,181 K]: [GC 42035.673: [ParNew Desired >>>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>>> 2161000 >>>>> bytes, 2161000 total - age 2: 22032 bytes, 2183032 total - >>>>> age >>>>> 3: 22032 bytes, 2205064 total - age 4: 23792 bytes, >>>>> 2228856 total : 139253K->2992K(153344K), 0.0400420 secs] >>>>> 19115423K->18979181K(27245952K), 0.0401730 secs] [Times: user=0.26 >>>>> sys=0.00, >>>>> real=0.04 secs] >>>>> Thu Feb 4 10:13:06 2010 [18,979,876 K]: [GC 42042.243: [ParNew Desired >>>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>>> 2113272 >>>>> bytes, 2113272 total - age 2: 23792 bytes, 2137064 total - >>>>> age >>>>> 3: 22032 bytes, 2159096 total - age 4: 22032 bytes, >>>>> 2181128 total : 138855K->3662K(153344K), 0.0465500 secs] >>>>> 19115044K->18979876K(27245952K), 0.0466840 secs] [Times: user=0.27 >>>>> sys=0.00, >>>>> real=0.04 secs] >>>>> Thu Feb 4 10:13:35 2010 [18,979,051 K]: [GC 42071.809: [ParNew Desired >>>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>>> 219056 >>>>> bytes, 219056 total - age 2: 1867632 bytes, 2086688 total - >>>>> age >>>>> 3: 23792 bytes, 2110480 total - age 4: 22032 bytes, >>>>> 2132512 total : 139982K->2814K(153344K), 0.0426920 secs] >>>>> 19116196K->18979051K(27245952K), 0.0428200 secs] [Times: user=0.28 >>>>> sys=0.00, >>>>> real=0.05 secs] >>>>> Thu Feb 4 10:13:35 2010: [CMS-concurrent-mark: 36.272/36.715 secs] >>>>> [Times: >>>>> user=56.34 sys=0.34, real=36.71 secs] >>>>> .... >>>>> >>>>> >>>>> .... >>>>> Thu Feb 4 11:28:59 2010: [GC [1 CMS-initial-mark: >>>>> 19012252K(27092608K)] >>>>> 19032719K(27245952K), 0.0279440 secs] [Times: user=0.02 sys=0.00, >>>>> real=0.03 >>>>> secs] >>>>> Thu Feb 4 11:28:59 2010: [CMS-concurrent-mark-start] >>>>> Thu Feb 4 11:28:59 2010 [19,154,661 K]: [GC 46595.614: [ParNew Desired >>>>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>>>> 17431976 >>>>> bytes, 17431976 total : 153344K->17024K(153344K), 0.9259390 secs] >>>>> 19165596K->19154661K(27245952K), 0.9260890 secs] [Times: user=1.23 >>>>> sys=0.05, >>>>> real=0.93 secs] >>>>> Thu Feb 4 11:29:00 2010 [19,199,696 K]: [GC 46596.973: [ParNew Desired >>>>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>>>> 17448432 >>>>> bytes, 17448432 total : 153344K->17024K(153344K), 0.3054460 secs] >>>>> 19290981K->19199696K(27245952K), 0.3055930 secs] [Times: user=0.79 >>>>> sys=0.07, >>>>> real=0.31 secs] >>>>> Thu Feb 4 11:29:01 2010 [19,229,358 K]: [GC 46597.722: [ParNew Desired >>>>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>>>> 10331408 >>>>> bytes, 10331408 total : 153344K->17024K(153344K), 0.1449040 secs] >>>>> 19336016K->19229358K(27245952K), 0.1450630 secs] [Times: user=0.59 >>>>> sys=0.02, >>>>> real=0.15 secs] >>>>> Thu Feb 4 11:29:02 2010 [19,240,015 K]: [GC 46598.038: [ParNew Desired >>>>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>>>> 17354120 >>>>> bytes, 17354120 total : 153344K->17024K(153344K), 0.0935440 secs] >>>>> 19365678K->19240015K(27245952K), 0.0936870 secs] [Times: user=0.59 >>>>> sys=0.00, >>>>> real=0.10 secs] >>>>> Thu Feb 4 11:29:02 2010 [19,302,509 K]: [GC 46598.347: [ParNew Desired >>>>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>>>> 17432360 >>>>> bytes, 17432360 total : 151519K->17024K(153344K), 0.4619430 secs] >>>>> 19374510K->19302509K(27245952K), 0.4620890 secs] [Times: user=1.19 >>>>> sys=0.02, >>>>> real=0.46 secs] >>>>> Thu Feb 4 11:29:03 2010 [19,329,594 K]: [GC 46599.030: [ParNew Desired >>>>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>>>> 11933296 >>>>> bytes, 11933296 total : 153344K->17024K(153344K), 0.1492670 secs] >>>>> 19438829K->19329594K(27245952K), 0.1494200 secs] [Times: user=0.77 >>>>> sys=0.03, >>>>> real=0.15 secs] >>>>> Thu Feb 4 11:29:03 2010 [19,337,176 K]: [GC 46599.423: [ParNew Desired >>>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>>> 5719032 >>>>> bytes, 5719032 total : 153344K->12852K(153344K), 0.0675180 secs] >>>>> 19465914K->19337176K(27245952K), 0.0676670 secs] [Times: user=0.46 >>>>> sys=0.02, >>>>> real=0.07 secs] >>>>> Thu Feb 4 11:29:03 2010 [19,338,256 K]: [GC 46599.754: [ParNew Desired >>>>> survivor size 8716288 bytes, new threshold 2 (max 4) - age 1: >>>>> 6783616 >>>>> bytes, 6783616 total - age 2: 5570080 bytes, 12353696 total : >>>>> 149172K->13931K(153344K), 0.0650520 secs] >>>>> 19473496K->19338256K(27245952K), >>>>> 0.0652060 secs] [Times: user=0.42 sys=0.01, real=0.07 secs] >>>>> Thu Feb 4 11:29:04 2010 [19,340,264 K]: [GC 46600.144: [ParNew Desired >>>>> survivor size 8716288 bytes, new threshold 2 (max 4) - age 1: >>>>> 2190136 >>>>> bytes, 2190136 total - age 2: 6762648 bytes, 8952784 total : >>>>> 150251K->10375K(153344K), 0.0709470 secs] >>>>> 19474576K->19340264K(27245952K), >>>>> 0.0710920 secs] [Times: user=0.48 sys=0.01, real=0.07 secs] >>>>> Thu Feb 4 11:29:17 2010 [19,345,928 K]: [GC 46613.628: [ParNew Desired >>>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>>> 760488 >>>>> bytes, 760488 total - age 2: 2173472 bytes, 2933960 total : >>>>> 146695K->9279K(153344K), 0.0689390 secs] >>>>> 19476584K->19345928K(27245952K), >>>>> 0.0691070 secs] [Times: user=0.45 sys=0.00, real=0.07 secs] >>>>> Thu Feb 4 11:29:17 2010: [CMS-concurrent-mark: 15.853/18.260 secs] >>>>> [Times: >>>>> user=29.26 sys=0.55, real=18.26 secs] >>>>> Thu Feb 4 11:29:17 2010: [CMS-concurrent-preclean-start] >>>>> .... >>>>> >>>>> Any idea what may causing these pauses and what possible JVM tuning >>>>> would >>>>> be >>>>> neccessary to avoid these pauses? Your help greatly appreciated. >>>>> >>>>> Thank you very much. >>>>> >>>>> Atul >>> http://old.nabble.com/file/p27463304/long.pause.in.cms.gc.log >>> long.pause.in.cms.gc.log >> > From atulksh at hotmail.com Fri Feb 5 17:10:08 2010 From: atulksh at hotmail.com (atulksh) Date: Fri, 5 Feb 2010 09:10:08 -0800 (PST) Subject: long unaccounted pause during concurrent mark phase of ParNew cycle In-Reply-To: <4B6C4EA1.6010008@Sun.COM> References: <27459365.post@talk.nabble.com> <4B6B3D74.90403@Sun.COM> <27463304.post@talk.nabble.com> <4B6C4290.4000907@Sun.COM> <27470597.post@talk.nabble.com> <4B6C4EA1.6010008@Sun.COM> Message-ID: <27470926.post@talk.nabble.com> I see. Sorry for not getting your previous question. Actually before Update 02 we did not see this issue. But that was a very different situation altogether since our servers did not have that much memory allocated to them. Only recently we have seen this issue come to surface because perhaps it has now become very noticeable. One correlation that we have found is that the pauses occur when there is significant load on the system (in terms of user activity - not necessarily user hits). When the system is relatively less uses the pauses are also relatively less. Thank you, Atul Jon Masamitsu wrote: > > > > On 02/05/10 08:47, atulksh wrote: >> Hi Jon, >> We are currently still using Update 2 on our servers. We see the same >> kind >> of pause behavior on Update 2 as well as Update 17. The pause behavior >> does >> not seem to change between these two versions. > > Did you notice this behavior after upgrading to jdk6 update 2 (i.e., > did it work OK on a jdk5 release or earlier)? > >> >> Thank you, >> >> Atul >> >> >> Jon Masamitsu wrote: >>> Atul, >>> >>> You mentioned that you were on jdk6 update 2 initially. >>> Did you see this behavior with earlier releases? >>> >>> Jon >>> >>> On 02/04/10 20:02, atulksh wrote: >>>> Hi Jon, >>>> I have some more (probably more precise) information for you. I have >>>> been >>>> able to reproduce this issue in our test environment (up till now we >>>> were >>>> only seeing it in our production setup). Along with the parameters that >>>> I >>>> have specified above I added the following. >>>> >>>> >>>> >>>> After loading up the app server (it is Tomcat 6.0.14) I got a pause >>>> during >>>> the CMS cycle of about 32 seconds. The way I noticed the pause is like >>>> this. >>>> I kept on running a script that does some basic page refresh (small >>>> data >>>> page) in the application continously. This is like a ping to the >>>> server. >>>> Then I parallely do operations that load up the memory in the >>>> application >>>> server. These are the real hits that users do in whom memory >>>> requirement >>>> is >>>> usually high and that drives the app server to the CMS threshold. >>>> >>>> In my case once the CMS kicked in, I saw that the period where the >>>> applilcation was in the "pause" no requests of my script were >>>> processed. >>>> They were just queued up to Tomcat - meaning Apache got those requests >>>> and >>>> logged that they were sent to Tomcat. But Tomcat did not register of >>>> receiving them. Then after the app went out of the "pause" then >>>> everything >>>> came back and all these pending requests got processed - like the app >>>> just >>>> had been unfrozen. >>>> >>>> During this freeze, what I see in the GC log is that the CMS cycle is >>>> almost >>>> at the end of the concurrent mark phase. It is doing the GC is doing >>>> the >>>> "perm space scanning". The pause exactly coincides with the time >>>> reported >>>> for the "perm space scanning". I am attaching the full gc log snippt >>>> right >>>> from the start of the CMS cycle. Below is just the place where the >>>> pause >>>> appears. >>>> >>>> 28676.784: [GC 28676.784: [ParNew >>>> Desired survivor size 4358144 bytes, new threshold 2 (max 4) >>>> - age 1: 3742608 bytes, 3742608 total >>>> - age 2: 3451152 bytes, 7193760 total >>>> : 76672K->8512K(76672K), 0.2045430 secs] >>>> 20262376K->20197311K(27254464K), >>>> 0.2048030 secs] [Times: user=0.77 sys=0.01, real=0.21 secs] >>>> Total time for which application threads were stopped: 0.2055080 >>>> seconds >>>> Finished perm space scanning in 2th thread: 0.367 sec >>>> Finished perm space scanning in 0th thread: 0.362 sec >>>> Application time: 0.1481860 seconds >>>> 28708.626: [GC 28708.627: [ParNew >>>> Desired survivor size 4358144 bytes, new threshold 2 (max 4) >>>> - age 1: 2079760 bytes, 2079760 total >>>> - age 2: 3657248 bytes, 5737008 total >>>> : 76672K->6795K(76672K), 0.1937080 secs] >>>> 20265471K->20198965K(27254464K), >>>> 0.1939480 secs] [Times: user=0.74 sys=0.00, real=0.20 secs] >>>> Total time for which application threads were stopped: 0.1948820 >>>> seconds >>>> Finished perm space scanning in 3th thread: 32.153 sec >>>> Finished perm space scanning in 1th thread: 32.097 sec >>>> Finished work stealing in 1th thread: 0.000 sec >>>> Finished work stealing in 3th thread: 0.002 sec >>>> Finished work stealing in 2th thread: 31.791 sec >>>> Finished work stealing in 0th thread: 31.791 sec >>>> 28708.825: [CMS-concurrent-mark: 88.906/158.952 secs] >>>> (CMS-concurrent-mark >>>> yielded 454 times) >>>> [Times: user=544.33 sys=4.59, real=158.95 secs] >>>> >>>> What is "perm space scanning" and is this where the gc would freeze the >>>> application (STW type). Is this a bug or something that is expected? >>>> >>>> Please let me know. This has become quite critical for our application. >>>> Please see the full gc log file attached. Your help very much >>>> appreciated. >>>> >>>> Thank you, >>>> >>>> Atul >>>> >>>> >>>> Jon Masamitsu wrote: >>>>> Atul, >>>>> >>>>> Do you mean the 36 secs here? >>>>> >>>>> > Thu Feb 4 10:13:35 2010: [CMS-concurrent-mark: 36.272/36.715 secs] >>>>> [Times: >>>>> > user=56.34 sys=0.34, real=36.71 secs] >>>>> >>>>> That is a concurrent phase (i.e., the application threads >>>>> are still running although they have to share the hardware >>>>> threads with the concurrent GC threads). >>>>> >>>>> Jon >>>>> >>>>> On 02/04/10 12:37, atulksh wrote: >>>>>> Hi, >>>>>> We have an application that is freezing (pausing) for a very long >>>>>> (tens >>>>>> of >>>>>> seconds) time during the concurrent mark phase of the ParNew cycle. >>>>>> This >>>>>> pause is consistently seen when there is good amount of activity on >>>>>> the >>>>>> server. We tried to upgrade the JVM (JRE 6) from Update 02 to Update >>>>>> 17 >>>>>> keeping all the parameters same, but the pauses did not go away (they >>>>>> became >>>>>> more consistent and slightly longer). We even tried to enable >>>>>> CompressedOops >>>>>> option, thinking that that would reduce memory requirements, but that >>>>>> has >>>>>> not helped. >>>>>> >>>>>> We are running on 64bit SUSE 10.1 platform with 32Gb of RAM and 8 >>>>>> core >>>>>> machine. 26g is allocated to the java process. The full set of VM >>>>>> parameters >>>>>> are here >>>>>> >>>>>> >>>>>> -Xmx26g >>>>>> -Xms26g >>>>>> -server >>>>>> -XX:+UseConcMarkSweepGC >>>>>> -XX:PermSize=256m >>>>>> -XX:MaxPermSize=256m >>>>>> -XX:+PrintTenuringDistribution >>>>>> -XX:+PrintGCDetails >>>>>> -XX:+PrintGCTimeStamps >>>>>> -XX:CMSInitiatingOccupancyFraction=70 >>>>>> -XX:-CMSParallelRemarkEnabled >>>>>> >>>>>> The nature of the pause is like this. All the application threads >>>>>> that >>>>>> are >>>>>> being processed seem to stop responding. New requests that come to >>>>>> our >>>>>> app >>>>>> server are not processed and get queued. When the pause ends then >>>>>> everything >>>>>> frees up and seems to go on normally henceforth. In the GC you see a >>>>>> gap >>>>>> between the ParNews. The following are two examples of the pauses. >>>>>> Notice >>>>>> the pause from 10:13:06 to 10:13:35 (29 seconds) in the first one and >>>>>> 11:29:04 to 11:29:17 (13 seconds) in the second one. These pauses >>>>>> only >>>>>> occur >>>>>> in the concurrent mark phase. They do not seem to occur at any other >>>>>> time >>>>>> or >>>>>> anywhere else in the CMS cycle. >>>>>> >>>>>> .... >>>>>> Thu Feb 4 10:12:59 2010: [GC [1 CMS-initial-mark: >>>>>> 18976170K(27092608K)] >>>>>> 18989786K(27245952K), 0.0059140 secs] [Times: user=0.01 sys=0.00, >>>>>> real=0.01 >>>>>> secs] >>>>>> Thu Feb 4 10:12:59 2010: [CMS-concurrent-mark-start] >>>>>> Thu Feb 4 10:12:59 2010 [18,980,009 K]: [GC 42035.293: [ParNew >>>>>> Desired >>>>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>>>> 1833064 >>>>>> bytes, 1833064 total - age 2: 18472 bytes, 1851536 total >>>>>> : >>>>>> 148315K->3839K(153344K), 0.0400230 secs] >>>>>> 19124486K->18980009K(27245952K), >>>>>> 0.0401580 secs] [Times: user=0.25 sys=0.00, real=0.04 secs] >>>>>> Thu Feb 4 10:12:59 2010 [18,979,036 K]: [GC 42035.415: [ParNew >>>>>> Desired >>>>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>>>> 2068376 >>>>>> bytes, 2068376 total - age 2: 23792 bytes, 2092168 total >>>>>> - >>>>>> age >>>>>> 3: 18472 bytes, 2110640 total : 140159K->2866K(153344K), >>>>>> 0.0352200 >>>>>> secs] 19116329K->18979036K(27245952K), 0.0353520 secs] [Times: >>>>>> user=0.29 >>>>>> sys=0.00, real=0.03 secs] >>>>>> Thu Feb 4 10:12:59 2010 [18,979,103 K]: [GC 42035.535: [ParNew >>>>>> Desired >>>>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>>>> 1782608 >>>>>> bytes, 1782608 total - age 2: 22032 bytes, 1804640 total >>>>>> - >>>>>> age >>>>>> 3: 23792 bytes, 1828432 total - age 4: 18472 bytes, >>>>>> 1846904 total : 139186K->2933K(153344K), 0.0498480 secs] >>>>>> 19115356K->18979103K(27245952K), 0.0499850 secs] [Times: user=0.24 >>>>>> sys=0.00, >>>>>> real=0.05 secs] >>>>>> Thu Feb 4 10:12:59 2010 [18,979,181 K]: [GC 42035.673: [ParNew >>>>>> Desired >>>>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>>>> 2161000 >>>>>> bytes, 2161000 total - age 2: 22032 bytes, 2183032 total >>>>>> - >>>>>> age >>>>>> 3: 22032 bytes, 2205064 total - age 4: 23792 bytes, >>>>>> 2228856 total : 139253K->2992K(153344K), 0.0400420 secs] >>>>>> 19115423K->18979181K(27245952K), 0.0401730 secs] [Times: user=0.26 >>>>>> sys=0.00, >>>>>> real=0.04 secs] >>>>>> Thu Feb 4 10:13:06 2010 [18,979,876 K]: [GC 42042.243: [ParNew >>>>>> Desired >>>>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>>>> 2113272 >>>>>> bytes, 2113272 total - age 2: 23792 bytes, 2137064 total >>>>>> - >>>>>> age >>>>>> 3: 22032 bytes, 2159096 total - age 4: 22032 bytes, >>>>>> 2181128 total : 138855K->3662K(153344K), 0.0465500 secs] >>>>>> 19115044K->18979876K(27245952K), 0.0466840 secs] [Times: user=0.27 >>>>>> sys=0.00, >>>>>> real=0.04 secs] >>>>>> Thu Feb 4 10:13:35 2010 [18,979,051 K]: [GC 42071.809: [ParNew >>>>>> Desired >>>>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>>>> 219056 >>>>>> bytes, 219056 total - age 2: 1867632 bytes, 2086688 total >>>>>> - >>>>>> age >>>>>> 3: 23792 bytes, 2110480 total - age 4: 22032 bytes, >>>>>> 2132512 total : 139982K->2814K(153344K), 0.0426920 secs] >>>>>> 19116196K->18979051K(27245952K), 0.0428200 secs] [Times: user=0.28 >>>>>> sys=0.00, >>>>>> real=0.05 secs] >>>>>> Thu Feb 4 10:13:35 2010: [CMS-concurrent-mark: 36.272/36.715 secs] >>>>>> [Times: >>>>>> user=56.34 sys=0.34, real=36.71 secs] >>>>>> .... >>>>>> >>>>>> >>>>>> .... >>>>>> Thu Feb 4 11:28:59 2010: [GC [1 CMS-initial-mark: >>>>>> 19012252K(27092608K)] >>>>>> 19032719K(27245952K), 0.0279440 secs] [Times: user=0.02 sys=0.00, >>>>>> real=0.03 >>>>>> secs] >>>>>> Thu Feb 4 11:28:59 2010: [CMS-concurrent-mark-start] >>>>>> Thu Feb 4 11:28:59 2010 [19,154,661 K]: [GC 46595.614: [ParNew >>>>>> Desired >>>>>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>>>>> 17431976 >>>>>> bytes, 17431976 total : 153344K->17024K(153344K), 0.9259390 secs] >>>>>> 19165596K->19154661K(27245952K), 0.9260890 secs] [Times: user=1.23 >>>>>> sys=0.05, >>>>>> real=0.93 secs] >>>>>> Thu Feb 4 11:29:00 2010 [19,199,696 K]: [GC 46596.973: [ParNew >>>>>> Desired >>>>>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>>>>> 17448432 >>>>>> bytes, 17448432 total : 153344K->17024K(153344K), 0.3054460 secs] >>>>>> 19290981K->19199696K(27245952K), 0.3055930 secs] [Times: user=0.79 >>>>>> sys=0.07, >>>>>> real=0.31 secs] >>>>>> Thu Feb 4 11:29:01 2010 [19,229,358 K]: [GC 46597.722: [ParNew >>>>>> Desired >>>>>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>>>>> 10331408 >>>>>> bytes, 10331408 total : 153344K->17024K(153344K), 0.1449040 secs] >>>>>> 19336016K->19229358K(27245952K), 0.1450630 secs] [Times: user=0.59 >>>>>> sys=0.02, >>>>>> real=0.15 secs] >>>>>> Thu Feb 4 11:29:02 2010 [19,240,015 K]: [GC 46598.038: [ParNew >>>>>> Desired >>>>>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>>>>> 17354120 >>>>>> bytes, 17354120 total : 153344K->17024K(153344K), 0.0935440 secs] >>>>>> 19365678K->19240015K(27245952K), 0.0936870 secs] [Times: user=0.59 >>>>>> sys=0.00, >>>>>> real=0.10 secs] >>>>>> Thu Feb 4 11:29:02 2010 [19,302,509 K]: [GC 46598.347: [ParNew >>>>>> Desired >>>>>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>>>>> 17432360 >>>>>> bytes, 17432360 total : 151519K->17024K(153344K), 0.4619430 secs] >>>>>> 19374510K->19302509K(27245952K), 0.4620890 secs] [Times: user=1.19 >>>>>> sys=0.02, >>>>>> real=0.46 secs] >>>>>> Thu Feb 4 11:29:03 2010 [19,329,594 K]: [GC 46599.030: [ParNew >>>>>> Desired >>>>>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>>>>> 11933296 >>>>>> bytes, 11933296 total : 153344K->17024K(153344K), 0.1492670 secs] >>>>>> 19438829K->19329594K(27245952K), 0.1494200 secs] [Times: user=0.77 >>>>>> sys=0.03, >>>>>> real=0.15 secs] >>>>>> Thu Feb 4 11:29:03 2010 [19,337,176 K]: [GC 46599.423: [ParNew >>>>>> Desired >>>>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>>>> 5719032 >>>>>> bytes, 5719032 total : 153344K->12852K(153344K), 0.0675180 secs] >>>>>> 19465914K->19337176K(27245952K), 0.0676670 secs] [Times: user=0.46 >>>>>> sys=0.02, >>>>>> real=0.07 secs] >>>>>> Thu Feb 4 11:29:03 2010 [19,338,256 K]: [GC 46599.754: [ParNew >>>>>> Desired >>>>>> survivor size 8716288 bytes, new threshold 2 (max 4) - age 1: >>>>>> 6783616 >>>>>> bytes, 6783616 total - age 2: 5570080 bytes, 12353696 total >>>>>> : >>>>>> 149172K->13931K(153344K), 0.0650520 secs] >>>>>> 19473496K->19338256K(27245952K), >>>>>> 0.0652060 secs] [Times: user=0.42 sys=0.01, real=0.07 secs] >>>>>> Thu Feb 4 11:29:04 2010 [19,340,264 K]: [GC 46600.144: [ParNew >>>>>> Desired >>>>>> survivor size 8716288 bytes, new threshold 2 (max 4) - age 1: >>>>>> 2190136 >>>>>> bytes, 2190136 total - age 2: 6762648 bytes, 8952784 total >>>>>> : >>>>>> 150251K->10375K(153344K), 0.0709470 secs] >>>>>> 19474576K->19340264K(27245952K), >>>>>> 0.0710920 secs] [Times: user=0.48 sys=0.01, real=0.07 secs] >>>>>> Thu Feb 4 11:29:17 2010 [19,345,928 K]: [GC 46613.628: [ParNew >>>>>> Desired >>>>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>>>> 760488 >>>>>> bytes, 760488 total - age 2: 2173472 bytes, 2933960 total >>>>>> : >>>>>> 146695K->9279K(153344K), 0.0689390 secs] >>>>>> 19476584K->19345928K(27245952K), >>>>>> 0.0691070 secs] [Times: user=0.45 sys=0.00, real=0.07 secs] >>>>>> Thu Feb 4 11:29:17 2010: [CMS-concurrent-mark: 15.853/18.260 secs] >>>>>> [Times: >>>>>> user=29.26 sys=0.55, real=18.26 secs] >>>>>> Thu Feb 4 11:29:17 2010: [CMS-concurrent-preclean-start] >>>>>> .... >>>>>> >>>>>> Any idea what may causing these pauses and what possible JVM tuning >>>>>> would >>>>>> be >>>>>> neccessary to avoid these pauses? Your help greatly appreciated. >>>>>> >>>>>> Thank you very much. >>>>>> >>>>>> Atul >>>> http://old.nabble.com/file/p27463304/long.pause.in.cms.gc.log >>>> long.pause.in.cms.gc.log >>> >> > > -- View this message in context: http://old.nabble.com/long-unaccounted-pause-during-concurrent-mark-phase-of-ParNew-cycle-tp27459365p27470926.html Sent from the OpenJDK Hotspot Garbage Collection mailing list archive at Nabble.com. From atulksh at hotmail.com Fri Feb 5 17:13:24 2010 From: atulksh at hotmail.com (atulksh) Date: Fri, 5 Feb 2010 09:13:24 -0800 (PST) Subject: long unaccounted pause during concurrent mark phase of ParNew cycle In-Reply-To: <27470926.post@talk.nabble.com> References: <27459365.post@talk.nabble.com> <4B6B3D74.90403@Sun.COM> <27463304.post@talk.nabble.com> <4B6C4290.4000907@Sun.COM> <27470597.post@talk.nabble.com> <4B6C4EA1.6010008@Sun.COM> <27470926.post@talk.nabble.com> Message-ID: <27470979.post@talk.nabble.com> Hi Jon, Also to give a bit more information, we have been using Update 2 on our servers since more than 2 years now. In that period we upgraded our servers from 8Gb to 32Gb (current). Thank you, Atul atulksh wrote: > > I see. Sorry for not getting your previous question. Actually before > Update 02 we did not see this issue. But that was a very different > situation altogether since our servers did not have that much memory > allocated to them. Only recently we have seen this issue come to surface > because perhaps it has now become very noticeable. > > One correlation that we have found is that the pauses occur when there is > significant load on the system (in terms of user activity - not > necessarily user hits). When the system is relatively less uses the pauses > are also relatively less. > > Thank you, > > Atul > > > Jon Masamitsu wrote: >> >> >> >> On 02/05/10 08:47, atulksh wrote: >>> Hi Jon, >>> We are currently still using Update 2 on our servers. We see the same >>> kind >>> of pause behavior on Update 2 as well as Update 17. The pause behavior >>> does >>> not seem to change between these two versions. >> >> Did you notice this behavior after upgrading to jdk6 update 2 (i.e., >> did it work OK on a jdk5 release or earlier)? >> >>> >>> Thank you, >>> >>> Atul >>> >>> >>> Jon Masamitsu wrote: >>>> Atul, >>>> >>>> You mentioned that you were on jdk6 update 2 initially. >>>> Did you see this behavior with earlier releases? >>>> >>>> Jon >>>> >>>> On 02/04/10 20:02, atulksh wrote: >>>>> Hi Jon, >>>>> I have some more (probably more precise) information for you. I have >>>>> been >>>>> able to reproduce this issue in our test environment (up till now we >>>>> were >>>>> only seeing it in our production setup). Along with the parameters >>>>> that I >>>>> have specified above I added the following. >>>>> >>>>> >>>>> >>>>> After loading up the app server (it is Tomcat 6.0.14) I got a pause >>>>> during >>>>> the CMS cycle of about 32 seconds. The way I noticed the pause is like >>>>> this. >>>>> I kept on running a script that does some basic page refresh (small >>>>> data >>>>> page) in the application continously. This is like a ping to the >>>>> server. >>>>> Then I parallely do operations that load up the memory in the >>>>> application >>>>> server. These are the real hits that users do in whom memory >>>>> requirement >>>>> is >>>>> usually high and that drives the app server to the CMS threshold. >>>>> >>>>> In my case once the CMS kicked in, I saw that the period where the >>>>> applilcation was in the "pause" no requests of my script were >>>>> processed. >>>>> They were just queued up to Tomcat - meaning Apache got those requests >>>>> and >>>>> logged that they were sent to Tomcat. But Tomcat did not register of >>>>> receiving them. Then after the app went out of the "pause" then >>>>> everything >>>>> came back and all these pending requests got processed - like the app >>>>> just >>>>> had been unfrozen. >>>>> >>>>> During this freeze, what I see in the GC log is that the CMS cycle is >>>>> almost >>>>> at the end of the concurrent mark phase. It is doing the GC is doing >>>>> the >>>>> "perm space scanning". The pause exactly coincides with the time >>>>> reported >>>>> for the "perm space scanning". I am attaching the full gc log snippt >>>>> right >>>>> from the start of the CMS cycle. Below is just the place where the >>>>> pause >>>>> appears. >>>>> >>>>> 28676.784: [GC 28676.784: [ParNew >>>>> Desired survivor size 4358144 bytes, new threshold 2 (max 4) >>>>> - age 1: 3742608 bytes, 3742608 total >>>>> - age 2: 3451152 bytes, 7193760 total >>>>> : 76672K->8512K(76672K), 0.2045430 secs] >>>>> 20262376K->20197311K(27254464K), >>>>> 0.2048030 secs] [Times: user=0.77 sys=0.01, real=0.21 secs] >>>>> Total time for which application threads were stopped: 0.2055080 >>>>> seconds >>>>> Finished perm space scanning in 2th thread: 0.367 sec >>>>> Finished perm space scanning in 0th thread: 0.362 sec >>>>> Application time: 0.1481860 seconds >>>>> 28708.626: [GC 28708.627: [ParNew >>>>> Desired survivor size 4358144 bytes, new threshold 2 (max 4) >>>>> - age 1: 2079760 bytes, 2079760 total >>>>> - age 2: 3657248 bytes, 5737008 total >>>>> : 76672K->6795K(76672K), 0.1937080 secs] >>>>> 20265471K->20198965K(27254464K), >>>>> 0.1939480 secs] [Times: user=0.74 sys=0.00, real=0.20 secs] >>>>> Total time for which application threads were stopped: 0.1948820 >>>>> seconds >>>>> Finished perm space scanning in 3th thread: 32.153 sec >>>>> Finished perm space scanning in 1th thread: 32.097 sec >>>>> Finished work stealing in 1th thread: 0.000 sec >>>>> Finished work stealing in 3th thread: 0.002 sec >>>>> Finished work stealing in 2th thread: 31.791 sec >>>>> Finished work stealing in 0th thread: 31.791 sec >>>>> 28708.825: [CMS-concurrent-mark: 88.906/158.952 secs] >>>>> (CMS-concurrent-mark >>>>> yielded 454 times) >>>>> [Times: user=544.33 sys=4.59, real=158.95 secs] >>>>> >>>>> What is "perm space scanning" and is this where the gc would freeze >>>>> the >>>>> application (STW type). Is this a bug or something that is expected? >>>>> >>>>> Please let me know. This has become quite critical for our >>>>> application. >>>>> Please see the full gc log file attached. Your help very much >>>>> appreciated. >>>>> >>>>> Thank you, >>>>> >>>>> Atul >>>>> >>>>> >>>>> Jon Masamitsu wrote: >>>>>> Atul, >>>>>> >>>>>> Do you mean the 36 secs here? >>>>>> >>>>>> > Thu Feb 4 10:13:35 2010: [CMS-concurrent-mark: 36.272/36.715 >>>>>> secs] >>>>>> [Times: >>>>>> > user=56.34 sys=0.34, real=36.71 secs] >>>>>> >>>>>> That is a concurrent phase (i.e., the application threads >>>>>> are still running although they have to share the hardware >>>>>> threads with the concurrent GC threads). >>>>>> >>>>>> Jon >>>>>> >>>>>> On 02/04/10 12:37, atulksh wrote: >>>>>>> Hi, >>>>>>> We have an application that is freezing (pausing) for a very long >>>>>>> (tens >>>>>>> of >>>>>>> seconds) time during the concurrent mark phase of the ParNew cycle. >>>>>>> This >>>>>>> pause is consistently seen when there is good amount of activity on >>>>>>> the >>>>>>> server. We tried to upgrade the JVM (JRE 6) from Update 02 to Update >>>>>>> 17 >>>>>>> keeping all the parameters same, but the pauses did not go away >>>>>>> (they >>>>>>> became >>>>>>> more consistent and slightly longer). We even tried to enable >>>>>>> CompressedOops >>>>>>> option, thinking that that would reduce memory requirements, but >>>>>>> that >>>>>>> has >>>>>>> not helped. >>>>>>> >>>>>>> We are running on 64bit SUSE 10.1 platform with 32Gb of RAM and 8 >>>>>>> core >>>>>>> machine. 26g is allocated to the java process. The full set of VM >>>>>>> parameters >>>>>>> are here >>>>>>> >>>>>>> >>>>>>> -Xmx26g >>>>>>> -Xms26g >>>>>>> -server >>>>>>> -XX:+UseConcMarkSweepGC >>>>>>> -XX:PermSize=256m >>>>>>> -XX:MaxPermSize=256m >>>>>>> -XX:+PrintTenuringDistribution >>>>>>> -XX:+PrintGCDetails >>>>>>> -XX:+PrintGCTimeStamps >>>>>>> -XX:CMSInitiatingOccupancyFraction=70 >>>>>>> -XX:-CMSParallelRemarkEnabled >>>>>>> >>>>>>> The nature of the pause is like this. All the application threads >>>>>>> that >>>>>>> are >>>>>>> being processed seem to stop responding. New requests that come to >>>>>>> our >>>>>>> app >>>>>>> server are not processed and get queued. When the pause ends then >>>>>>> everything >>>>>>> frees up and seems to go on normally henceforth. In the GC you see a >>>>>>> gap >>>>>>> between the ParNews. The following are two examples of the pauses. >>>>>>> Notice >>>>>>> the pause from 10:13:06 to 10:13:35 (29 seconds) in the first one >>>>>>> and >>>>>>> 11:29:04 to 11:29:17 (13 seconds) in the second one. These pauses >>>>>>> only >>>>>>> occur >>>>>>> in the concurrent mark phase. They do not seem to occur at any other >>>>>>> time >>>>>>> or >>>>>>> anywhere else in the CMS cycle. >>>>>>> >>>>>>> .... >>>>>>> Thu Feb 4 10:12:59 2010: [GC [1 CMS-initial-mark: >>>>>>> 18976170K(27092608K)] >>>>>>> 18989786K(27245952K), 0.0059140 secs] [Times: user=0.01 sys=0.00, >>>>>>> real=0.01 >>>>>>> secs] >>>>>>> Thu Feb 4 10:12:59 2010: [CMS-concurrent-mark-start] >>>>>>> Thu Feb 4 10:12:59 2010 [18,980,009 K]: [GC 42035.293: [ParNew >>>>>>> Desired >>>>>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>>>>> 1833064 >>>>>>> bytes, 1833064 total - age 2: 18472 bytes, 1851536 >>>>>>> total : >>>>>>> 148315K->3839K(153344K), 0.0400230 secs] >>>>>>> 19124486K->18980009K(27245952K), >>>>>>> 0.0401580 secs] [Times: user=0.25 sys=0.00, real=0.04 secs] >>>>>>> Thu Feb 4 10:12:59 2010 [18,979,036 K]: [GC 42035.415: [ParNew >>>>>>> Desired >>>>>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>>>>> 2068376 >>>>>>> bytes, 2068376 total - age 2: 23792 bytes, 2092168 >>>>>>> total - >>>>>>> age >>>>>>> 3: 18472 bytes, 2110640 total : 140159K->2866K(153344K), >>>>>>> 0.0352200 >>>>>>> secs] 19116329K->18979036K(27245952K), 0.0353520 secs] [Times: >>>>>>> user=0.29 >>>>>>> sys=0.00, real=0.03 secs] >>>>>>> Thu Feb 4 10:12:59 2010 [18,979,103 K]: [GC 42035.535: [ParNew >>>>>>> Desired >>>>>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>>>>> 1782608 >>>>>>> bytes, 1782608 total - age 2: 22032 bytes, 1804640 >>>>>>> total - >>>>>>> age >>>>>>> 3: 23792 bytes, 1828432 total - age 4: 18472 bytes, >>>>>>> 1846904 total : 139186K->2933K(153344K), 0.0498480 secs] >>>>>>> 19115356K->18979103K(27245952K), 0.0499850 secs] [Times: user=0.24 >>>>>>> sys=0.00, >>>>>>> real=0.05 secs] >>>>>>> Thu Feb 4 10:12:59 2010 [18,979,181 K]: [GC 42035.673: [ParNew >>>>>>> Desired >>>>>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>>>>> 2161000 >>>>>>> bytes, 2161000 total - age 2: 22032 bytes, 2183032 >>>>>>> total - >>>>>>> age >>>>>>> 3: 22032 bytes, 2205064 total - age 4: 23792 bytes, >>>>>>> 2228856 total : 139253K->2992K(153344K), 0.0400420 secs] >>>>>>> 19115423K->18979181K(27245952K), 0.0401730 secs] [Times: user=0.26 >>>>>>> sys=0.00, >>>>>>> real=0.04 secs] >>>>>>> Thu Feb 4 10:13:06 2010 [18,979,876 K]: [GC 42042.243: [ParNew >>>>>>> Desired >>>>>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>>>>> 2113272 >>>>>>> bytes, 2113272 total - age 2: 23792 bytes, 2137064 >>>>>>> total - >>>>>>> age >>>>>>> 3: 22032 bytes, 2159096 total - age 4: 22032 bytes, >>>>>>> 2181128 total : 138855K->3662K(153344K), 0.0465500 secs] >>>>>>> 19115044K->18979876K(27245952K), 0.0466840 secs] [Times: user=0.27 >>>>>>> sys=0.00, >>>>>>> real=0.04 secs] >>>>>>> Thu Feb 4 10:13:35 2010 [18,979,051 K]: [GC 42071.809: [ParNew >>>>>>> Desired >>>>>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>>>>> 219056 >>>>>>> bytes, 219056 total - age 2: 1867632 bytes, 2086688 >>>>>>> total - >>>>>>> age >>>>>>> 3: 23792 bytes, 2110480 total - age 4: 22032 bytes, >>>>>>> 2132512 total : 139982K->2814K(153344K), 0.0426920 secs] >>>>>>> 19116196K->18979051K(27245952K), 0.0428200 secs] [Times: user=0.28 >>>>>>> sys=0.00, >>>>>>> real=0.05 secs] >>>>>>> Thu Feb 4 10:13:35 2010: [CMS-concurrent-mark: 36.272/36.715 secs] >>>>>>> [Times: >>>>>>> user=56.34 sys=0.34, real=36.71 secs] >>>>>>> .... >>>>>>> >>>>>>> >>>>>>> .... >>>>>>> Thu Feb 4 11:28:59 2010: [GC [1 CMS-initial-mark: >>>>>>> 19012252K(27092608K)] >>>>>>> 19032719K(27245952K), 0.0279440 secs] [Times: user=0.02 sys=0.00, >>>>>>> real=0.03 >>>>>>> secs] >>>>>>> Thu Feb 4 11:28:59 2010: [CMS-concurrent-mark-start] >>>>>>> Thu Feb 4 11:28:59 2010 [19,154,661 K]: [GC 46595.614: [ParNew >>>>>>> Desired >>>>>>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>>>>>> 17431976 >>>>>>> bytes, 17431976 total : 153344K->17024K(153344K), 0.9259390 secs] >>>>>>> 19165596K->19154661K(27245952K), 0.9260890 secs] [Times: user=1.23 >>>>>>> sys=0.05, >>>>>>> real=0.93 secs] >>>>>>> Thu Feb 4 11:29:00 2010 [19,199,696 K]: [GC 46596.973: [ParNew >>>>>>> Desired >>>>>>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>>>>>> 17448432 >>>>>>> bytes, 17448432 total : 153344K->17024K(153344K), 0.3054460 secs] >>>>>>> 19290981K->19199696K(27245952K), 0.3055930 secs] [Times: user=0.79 >>>>>>> sys=0.07, >>>>>>> real=0.31 secs] >>>>>>> Thu Feb 4 11:29:01 2010 [19,229,358 K]: [GC 46597.722: [ParNew >>>>>>> Desired >>>>>>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>>>>>> 10331408 >>>>>>> bytes, 10331408 total : 153344K->17024K(153344K), 0.1449040 secs] >>>>>>> 19336016K->19229358K(27245952K), 0.1450630 secs] [Times: user=0.59 >>>>>>> sys=0.02, >>>>>>> real=0.15 secs] >>>>>>> Thu Feb 4 11:29:02 2010 [19,240,015 K]: [GC 46598.038: [ParNew >>>>>>> Desired >>>>>>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>>>>>> 17354120 >>>>>>> bytes, 17354120 total : 153344K->17024K(153344K), 0.0935440 secs] >>>>>>> 19365678K->19240015K(27245952K), 0.0936870 secs] [Times: user=0.59 >>>>>>> sys=0.00, >>>>>>> real=0.10 secs] >>>>>>> Thu Feb 4 11:29:02 2010 [19,302,509 K]: [GC 46598.347: [ParNew >>>>>>> Desired >>>>>>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>>>>>> 17432360 >>>>>>> bytes, 17432360 total : 151519K->17024K(153344K), 0.4619430 secs] >>>>>>> 19374510K->19302509K(27245952K), 0.4620890 secs] [Times: user=1.19 >>>>>>> sys=0.02, >>>>>>> real=0.46 secs] >>>>>>> Thu Feb 4 11:29:03 2010 [19,329,594 K]: [GC 46599.030: [ParNew >>>>>>> Desired >>>>>>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>>>>>> 11933296 >>>>>>> bytes, 11933296 total : 153344K->17024K(153344K), 0.1492670 secs] >>>>>>> 19438829K->19329594K(27245952K), 0.1494200 secs] [Times: user=0.77 >>>>>>> sys=0.03, >>>>>>> real=0.15 secs] >>>>>>> Thu Feb 4 11:29:03 2010 [19,337,176 K]: [GC 46599.423: [ParNew >>>>>>> Desired >>>>>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>>>>> 5719032 >>>>>>> bytes, 5719032 total : 153344K->12852K(153344K), 0.0675180 secs] >>>>>>> 19465914K->19337176K(27245952K), 0.0676670 secs] [Times: user=0.46 >>>>>>> sys=0.02, >>>>>>> real=0.07 secs] >>>>>>> Thu Feb 4 11:29:03 2010 [19,338,256 K]: [GC 46599.754: [ParNew >>>>>>> Desired >>>>>>> survivor size 8716288 bytes, new threshold 2 (max 4) - age 1: >>>>>>> 6783616 >>>>>>> bytes, 6783616 total - age 2: 5570080 bytes, 12353696 >>>>>>> total : >>>>>>> 149172K->13931K(153344K), 0.0650520 secs] >>>>>>> 19473496K->19338256K(27245952K), >>>>>>> 0.0652060 secs] [Times: user=0.42 sys=0.01, real=0.07 secs] >>>>>>> Thu Feb 4 11:29:04 2010 [19,340,264 K]: [GC 46600.144: [ParNew >>>>>>> Desired >>>>>>> survivor size 8716288 bytes, new threshold 2 (max 4) - age 1: >>>>>>> 2190136 >>>>>>> bytes, 2190136 total - age 2: 6762648 bytes, 8952784 >>>>>>> total : >>>>>>> 150251K->10375K(153344K), 0.0709470 secs] >>>>>>> 19474576K->19340264K(27245952K), >>>>>>> 0.0710920 secs] [Times: user=0.48 sys=0.01, real=0.07 secs] >>>>>>> Thu Feb 4 11:29:17 2010 [19,345,928 K]: [GC 46613.628: [ParNew >>>>>>> Desired >>>>>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>>>>> 760488 >>>>>>> bytes, 760488 total - age 2: 2173472 bytes, 2933960 >>>>>>> total : >>>>>>> 146695K->9279K(153344K), 0.0689390 secs] >>>>>>> 19476584K->19345928K(27245952K), >>>>>>> 0.0691070 secs] [Times: user=0.45 sys=0.00, real=0.07 secs] >>>>>>> Thu Feb 4 11:29:17 2010: [CMS-concurrent-mark: 15.853/18.260 secs] >>>>>>> [Times: >>>>>>> user=29.26 sys=0.55, real=18.26 secs] >>>>>>> Thu Feb 4 11:29:17 2010: [CMS-concurrent-preclean-start] >>>>>>> .... >>>>>>> >>>>>>> Any idea what may causing these pauses and what possible JVM tuning >>>>>>> would >>>>>>> be >>>>>>> neccessary to avoid these pauses? Your help greatly appreciated. >>>>>>> >>>>>>> Thank you very much. >>>>>>> >>>>>>> Atul >>>>> http://old.nabble.com/file/p27463304/long.pause.in.cms.gc.log >>>>> long.pause.in.cms.gc.log >>>> >>> >> >> > > -- View this message in context: http://old.nabble.com/long-unaccounted-pause-during-concurrent-mark-phase-of-ParNew-cycle-tp27459365p27470979.html Sent from the OpenJDK Hotspot Garbage Collection mailing list archive at Nabble.com. From Y.S.Ramakrishna at Sun.COM Fri Feb 5 19:14:19 2010 From: Y.S.Ramakrishna at Sun.COM (Y. Srinivas Ramakrishna) Date: Fri, 05 Feb 2010 11:14:19 -0800 Subject: long unaccounted pause during concurrent mark phase of ParNew cycle In-Reply-To: <27470522.post@talk.nabble.com> References: <27459365.post@talk.nabble.com> <4B6B3D74.90403@Sun.COM> <27463304.post@talk.nabble.com> <4B6BB53D.5010800@sun.com> <27470522.post@talk.nabble.com> Message-ID: <4B6C6E0B.7070304@sun.com> atulksh wrote: > Hi Ramki, > I will give as try with that option as you have asked. If I disable the > Concurrent Multi Threading in CMS with this flag is it going to not > multi-thread any phase in general? No, that flag affects just the MT-ness of the concurrent phases. > > We have been using -XX:-CMSParallelRemarkEnabled in our production > environment for a long time. After we had upgraded to Update 2 we had seen > crashes which were attributed to the remark phase. Using this flag had > stopped those crashes. This issue was eventually fixed in Update 4 but after > we had tried Update 4 we got more frequent crashes and since we had never > tried to update our VM. I believe you had run into 6722113 and cousins, which were fixed in 6u14. Certainly in 6u18. > > Now that we have been seeing these pauses, we have been trying to upgrade to > Update 17. What we see is our pause behavior did not change a bit from > Update 2 to Update 17, so it looks like that is something inherent and has > not changed. I see. Yes, it does look like behaviour that has not been reported before and which we were not independently aware of. > > Did you checkout my last response where I have given more details about our > pause? It looks like our pause is coinciding with the perm space scanning > occurring at the end of the concurrent mark phase. Any insights as to what > normally goes on in the GC at that time (is that STW phase) and if yes what > can we do about it? See my private response to you. No this is not a STW pause. Something is preventing a foreground collection from happening perhaps because of the CMS activity locking out a foreground collection. Sans space in the young gen to allocate, the application appears to stall. > > I am happy to provide you with any further information you may want. Your > help and insights greatly appreciated. Yes, thanks. As I noted earlier: (1) a gc log was not attached with yr original report (you said one was, but you may have perhaps forgotten to attach it or the openjdk mailing list may have stripped it. Send it to me and/or Jon directly. Also the other info I requested in that email. Thanks. -- ramki > > Thank you very much. > > Atul > > y.s.ramakrishna wrote: >> Hello Atul -- >> >> Could you add -XX:-CMSConcurrentMTEnabled to the mix >> and see if it affects the behaviour in any manner? >> >> PS: I notice that you are using -XX:-CMSParallelRemarkEnabled. >> What was the reason for using that flag? >> >> thanks. >> -- ramki >> >> atulksh wrote: >>> Hi Jon, >>> I have some more (probably more precise) information for you. I have been >>> able to reproduce this issue in our test environment (up till now we were >>> only seeing it in our production setup). Along with the parameters that I >>> have specified above I added the following. >>> >>> >>> >>> After loading up the app server (it is Tomcat 6.0.14) I got a pause >>> during >>> the CMS cycle of about 32 seconds. The way I noticed the pause is like >>> this. >>> I kept on running a script that does some basic page refresh (small data >>> page) in the application continously. This is like a ping to the server. >>> Then I parallely do operations that load up the memory in the application >>> server. These are the real hits that users do in whom memory requirement >>> is >>> usually high and that drives the app server to the CMS threshold. >>> >>> In my case once the CMS kicked in, I saw that the period where the >>> applilcation was in the "pause" no requests of my script were processed. >>> They were just queued up to Tomcat - meaning Apache got those requests >>> and >>> logged that they were sent to Tomcat. But Tomcat did not register of >>> receiving them. Then after the app went out of the "pause" then >>> everything >>> came back and all these pending requests got processed - like the app >>> just >>> had been unfrozen. >>> >>> During this freeze, what I see in the GC log is that the CMS cycle is >>> almost >>> at the end of the concurrent mark phase. It is doing the GC is doing the >>> "perm space scanning". The pause exactly coincides with the time reported >>> for the "perm space scanning". I am attaching the full gc log snippt >>> right >>> from the start of the CMS cycle. Below is just the place where the pause >>> appears. >>> >>> 28676.784: [GC 28676.784: [ParNew >>> Desired survivor size 4358144 bytes, new threshold 2 (max 4) >>> - age 1: 3742608 bytes, 3742608 total >>> - age 2: 3451152 bytes, 7193760 total >>> : 76672K->8512K(76672K), 0.2045430 secs] 20262376K->20197311K(27254464K), >>> 0.2048030 secs] [Times: user=0.77 sys=0.01, real=0.21 secs] >>> Total time for which application threads were stopped: 0.2055080 seconds >>> Finished perm space scanning in 2th thread: 0.367 sec >>> Finished perm space scanning in 0th thread: 0.362 sec >>> Application time: 0.1481860 seconds >>> 28708.626: [GC 28708.627: [ParNew >>> Desired survivor size 4358144 bytes, new threshold 2 (max 4) >>> - age 1: 2079760 bytes, 2079760 total >>> - age 2: 3657248 bytes, 5737008 total >>> : 76672K->6795K(76672K), 0.1937080 secs] 20265471K->20198965K(27254464K), >>> 0.1939480 secs] [Times: user=0.74 sys=0.00, real=0.20 secs] >>> Total time for which application threads were stopped: 0.1948820 seconds >>> Finished perm space scanning in 3th thread: 32.153 sec >>> Finished perm space scanning in 1th thread: 32.097 sec >>> Finished work stealing in 1th thread: 0.000 sec >>> Finished work stealing in 3th thread: 0.002 sec >>> Finished work stealing in 2th thread: 31.791 sec >>> Finished work stealing in 0th thread: 31.791 sec >>> 28708.825: [CMS-concurrent-mark: 88.906/158.952 secs] >>> (CMS-concurrent-mark >>> yielded 454 times) >>> [Times: user=544.33 sys=4.59, real=158.95 secs] >>> >>> What is "perm space scanning" and is this where the gc would freeze the >>> application (STW type). Is this a bug or something that is expected? >>> >>> Please let me know. This has become quite critical for our application. >>> Please see the full gc log file attached. Your help very much >>> appreciated. >>> >>> Thank you, >>> >>> Atul >>> >>> >>> Jon Masamitsu wrote: >>>> Atul, >>>> >>>> Do you mean the 36 secs here? >>>> >>>> > Thu Feb 4 10:13:35 2010: [CMS-concurrent-mark: 36.272/36.715 secs] >>>> [Times: >>>> > user=56.34 sys=0.34, real=36.71 secs] >>>> >>>> That is a concurrent phase (i.e., the application threads >>>> are still running although they have to share the hardware >>>> threads with the concurrent GC threads). >>>> >>>> Jon >>>> >>>> On 02/04/10 12:37, atulksh wrote: >>>>> Hi, >>>>> We have an application that is freezing (pausing) for a very long (tens >>>>> of >>>>> seconds) time during the concurrent mark phase of the ParNew cycle. >>>>> This >>>>> pause is consistently seen when there is good amount of activity on the >>>>> server. We tried to upgrade the JVM (JRE 6) from Update 02 to Update 17 >>>>> keeping all the parameters same, but the pauses did not go away (they >>>>> became >>>>> more consistent and slightly longer). We even tried to enable >>>>> CompressedOops >>>>> option, thinking that that would reduce memory requirements, but that >>>>> has >>>>> not helped. >>>>> >>>>> We are running on 64bit SUSE 10.1 platform with 32Gb of RAM and 8 core >>>>> machine. 26g is allocated to the java process. The full set of VM >>>>> parameters >>>>> are here >>>>> >>>>> >>>>> -Xmx26g >>>>> -Xms26g >>>>> -server >>>>> -XX:+UseConcMarkSweepGC >>>>> -XX:PermSize=256m >>>>> -XX:MaxPermSize=256m >>>>> -XX:+PrintTenuringDistribution >>>>> -XX:+PrintGCDetails >>>>> -XX:+PrintGCTimeStamps >>>>> -XX:CMSInitiatingOccupancyFraction=70 >>>>> -XX:-CMSParallelRemarkEnabled >>>>> >>>>> The nature of the pause is like this. All the application threads that >>>>> are >>>>> being processed seem to stop responding. New requests that come to our >>>>> app >>>>> server are not processed and get queued. When the pause ends then >>>>> everything >>>>> frees up and seems to go on normally henceforth. In the GC you see a >>>>> gap >>>>> between the ParNews. The following are two examples of the pauses. >>>>> Notice >>>>> the pause from 10:13:06 to 10:13:35 (29 seconds) in the first one and >>>>> 11:29:04 to 11:29:17 (13 seconds) in the second one. These pauses only >>>>> occur >>>>> in the concurrent mark phase. They do not seem to occur at any other >>>>> time >>>>> or >>>>> anywhere else in the CMS cycle. >>>>> >>>>> .... >>>>> Thu Feb 4 10:12:59 2010: [GC [1 CMS-initial-mark: >>>>> 18976170K(27092608K)] >>>>> 18989786K(27245952K), 0.0059140 secs] [Times: user=0.01 sys=0.00, >>>>> real=0.01 >>>>> secs] >>>>> Thu Feb 4 10:12:59 2010: [CMS-concurrent-mark-start] >>>>> Thu Feb 4 10:12:59 2010 [18,980,009 K]: [GC 42035.293: [ParNew Desired >>>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>>> 1833064 >>>>> bytes, 1833064 total - age 2: 18472 bytes, 1851536 total : >>>>> 148315K->3839K(153344K), 0.0400230 secs] >>>>> 19124486K->18980009K(27245952K), >>>>> 0.0401580 secs] [Times: user=0.25 sys=0.00, real=0.04 secs] >>>>> Thu Feb 4 10:12:59 2010 [18,979,036 K]: [GC 42035.415: [ParNew Desired >>>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>>> 2068376 >>>>> bytes, 2068376 total - age 2: 23792 bytes, 2092168 total - >>>>> age >>>>> 3: 18472 bytes, 2110640 total : 140159K->2866K(153344K), >>>>> 0.0352200 >>>>> secs] 19116329K->18979036K(27245952K), 0.0353520 secs] [Times: >>>>> user=0.29 >>>>> sys=0.00, real=0.03 secs] >>>>> Thu Feb 4 10:12:59 2010 [18,979,103 K]: [GC 42035.535: [ParNew Desired >>>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>>> 1782608 >>>>> bytes, 1782608 total - age 2: 22032 bytes, 1804640 total - >>>>> age >>>>> 3: 23792 bytes, 1828432 total - age 4: 18472 bytes, >>>>> 1846904 total : 139186K->2933K(153344K), 0.0498480 secs] >>>>> 19115356K->18979103K(27245952K), 0.0499850 secs] [Times: user=0.24 >>>>> sys=0.00, >>>>> real=0.05 secs] >>>>> Thu Feb 4 10:12:59 2010 [18,979,181 K]: [GC 42035.673: [ParNew Desired >>>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>>> 2161000 >>>>> bytes, 2161000 total - age 2: 22032 bytes, 2183032 total - >>>>> age >>>>> 3: 22032 bytes, 2205064 total - age 4: 23792 bytes, >>>>> 2228856 total : 139253K->2992K(153344K), 0.0400420 secs] >>>>> 19115423K->18979181K(27245952K), 0.0401730 secs] [Times: user=0.26 >>>>> sys=0.00, >>>>> real=0.04 secs] >>>>> Thu Feb 4 10:13:06 2010 [18,979,876 K]: [GC 42042.243: [ParNew Desired >>>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>>> 2113272 >>>>> bytes, 2113272 total - age 2: 23792 bytes, 2137064 total - >>>>> age >>>>> 3: 22032 bytes, 2159096 total - age 4: 22032 bytes, >>>>> 2181128 total : 138855K->3662K(153344K), 0.0465500 secs] >>>>> 19115044K->18979876K(27245952K), 0.0466840 secs] [Times: user=0.27 >>>>> sys=0.00, >>>>> real=0.04 secs] >>>>> Thu Feb 4 10:13:35 2010 [18,979,051 K]: [GC 42071.809: [ParNew Desired >>>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>>> 219056 >>>>> bytes, 219056 total - age 2: 1867632 bytes, 2086688 total - >>>>> age >>>>> 3: 23792 bytes, 2110480 total - age 4: 22032 bytes, >>>>> 2132512 total : 139982K->2814K(153344K), 0.0426920 secs] >>>>> 19116196K->18979051K(27245952K), 0.0428200 secs] [Times: user=0.28 >>>>> sys=0.00, >>>>> real=0.05 secs] >>>>> Thu Feb 4 10:13:35 2010: [CMS-concurrent-mark: 36.272/36.715 secs] >>>>> [Times: >>>>> user=56.34 sys=0.34, real=36.71 secs] >>>>> .... >>>>> >>>>> >>>>> .... >>>>> Thu Feb 4 11:28:59 2010: [GC [1 CMS-initial-mark: >>>>> 19012252K(27092608K)] >>>>> 19032719K(27245952K), 0.0279440 secs] [Times: user=0.02 sys=0.00, >>>>> real=0.03 >>>>> secs] >>>>> Thu Feb 4 11:28:59 2010: [CMS-concurrent-mark-start] >>>>> Thu Feb 4 11:28:59 2010 [19,154,661 K]: [GC 46595.614: [ParNew Desired >>>>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>>>> 17431976 >>>>> bytes, 17431976 total : 153344K->17024K(153344K), 0.9259390 secs] >>>>> 19165596K->19154661K(27245952K), 0.9260890 secs] [Times: user=1.23 >>>>> sys=0.05, >>>>> real=0.93 secs] >>>>> Thu Feb 4 11:29:00 2010 [19,199,696 K]: [GC 46596.973: [ParNew Desired >>>>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>>>> 17448432 >>>>> bytes, 17448432 total : 153344K->17024K(153344K), 0.3054460 secs] >>>>> 19290981K->19199696K(27245952K), 0.3055930 secs] [Times: user=0.79 >>>>> sys=0.07, >>>>> real=0.31 secs] >>>>> Thu Feb 4 11:29:01 2010 [19,229,358 K]: [GC 46597.722: [ParNew Desired >>>>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>>>> 10331408 >>>>> bytes, 10331408 total : 153344K->17024K(153344K), 0.1449040 secs] >>>>> 19336016K->19229358K(27245952K), 0.1450630 secs] [Times: user=0.59 >>>>> sys=0.02, >>>>> real=0.15 secs] >>>>> Thu Feb 4 11:29:02 2010 [19,240,015 K]: [GC 46598.038: [ParNew Desired >>>>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>>>> 17354120 >>>>> bytes, 17354120 total : 153344K->17024K(153344K), 0.0935440 secs] >>>>> 19365678K->19240015K(27245952K), 0.0936870 secs] [Times: user=0.59 >>>>> sys=0.00, >>>>> real=0.10 secs] >>>>> Thu Feb 4 11:29:02 2010 [19,302,509 K]: [GC 46598.347: [ParNew Desired >>>>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>>>> 17432360 >>>>> bytes, 17432360 total : 151519K->17024K(153344K), 0.4619430 secs] >>>>> 19374510K->19302509K(27245952K), 0.4620890 secs] [Times: user=1.19 >>>>> sys=0.02, >>>>> real=0.46 secs] >>>>> Thu Feb 4 11:29:03 2010 [19,329,594 K]: [GC 46599.030: [ParNew Desired >>>>> survivor size 8716288 bytes, new threshold 1 (max 4) - age 1: >>>>> 11933296 >>>>> bytes, 11933296 total : 153344K->17024K(153344K), 0.1492670 secs] >>>>> 19438829K->19329594K(27245952K), 0.1494200 secs] [Times: user=0.77 >>>>> sys=0.03, >>>>> real=0.15 secs] >>>>> Thu Feb 4 11:29:03 2010 [19,337,176 K]: [GC 46599.423: [ParNew Desired >>>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>>> 5719032 >>>>> bytes, 5719032 total : 153344K->12852K(153344K), 0.0675180 secs] >>>>> 19465914K->19337176K(27245952K), 0.0676670 secs] [Times: user=0.46 >>>>> sys=0.02, >>>>> real=0.07 secs] >>>>> Thu Feb 4 11:29:03 2010 [19,338,256 K]: [GC 46599.754: [ParNew Desired >>>>> survivor size 8716288 bytes, new threshold 2 (max 4) - age 1: >>>>> 6783616 >>>>> bytes, 6783616 total - age 2: 5570080 bytes, 12353696 total : >>>>> 149172K->13931K(153344K), 0.0650520 secs] >>>>> 19473496K->19338256K(27245952K), >>>>> 0.0652060 secs] [Times: user=0.42 sys=0.01, real=0.07 secs] >>>>> Thu Feb 4 11:29:04 2010 [19,340,264 K]: [GC 46600.144: [ParNew Desired >>>>> survivor size 8716288 bytes, new threshold 2 (max 4) - age 1: >>>>> 2190136 >>>>> bytes, 2190136 total - age 2: 6762648 bytes, 8952784 total : >>>>> 150251K->10375K(153344K), 0.0709470 secs] >>>>> 19474576K->19340264K(27245952K), >>>>> 0.0710920 secs] [Times: user=0.48 sys=0.01, real=0.07 secs] >>>>> Thu Feb 4 11:29:17 2010 [19,345,928 K]: [GC 46613.628: [ParNew Desired >>>>> survivor size 8716288 bytes, new threshold 4 (max 4) - age 1: >>>>> 760488 >>>>> bytes, 760488 total - age 2: 2173472 bytes, 2933960 total : >>>>> 146695K->9279K(153344K), 0.0689390 secs] >>>>> 19476584K->19345928K(27245952K), >>>>> 0.0691070 secs] [Times: user=0.45 sys=0.00, real=0.07 secs] >>>>> Thu Feb 4 11:29:17 2010: [CMS-concurrent-mark: 15.853/18.260 secs] >>>>> [Times: >>>>> user=29.26 sys=0.55, real=18.26 secs] >>>>> Thu Feb 4 11:29:17 2010: [CMS-concurrent-preclean-start] >>>>> .... >>>>> >>>>> Any idea what may causing these pauses and what possible JVM tuning >>>>> would >>>>> be >>>>> neccessary to avoid these pauses? Your help greatly appreciated. >>>>> >>>>> Thank you very much. >>>>> >>>>> Atul >>> http://old.nabble.com/file/p27463304/long.pause.in.cms.gc.log >>> long.pause.in.cms.gc.log >> >> > From John.Coomes at sun.com Sat Feb 6 01:31:16 2010 From: John.Coomes at sun.com (John Coomes) Date: Fri, 5 Feb 2010 17:31:16 -0800 Subject: review request (M) - 4396719 mark sweep stack overflow Message-ID: <19308.50788.240081.902763@sun.com> I'd like reviews of the fix for 4396719 Mark Sweep stack overflow on deeply nested Object arrays I split the review into two parts in an attempt to make it easier. The first part fixes the bug by using an explicit stack/queue for object arrays: http://cr.openjdk.java.net/~jcoomes/4396719-objarray/ The second part adds a size parameter to the TaskQueue class; the new objArray queue used by par compact doesn't need to be as big as the default queue size, particularly in 64-bit. http://cr.openjdk.java.net/~jcoomes/4396719-tqsize/ Performance testing is still in progress. -John From john.coomes at sun.com Sat Feb 6 02:36:26 2010 From: john.coomes at sun.com (john.coomes at sun.com) Date: Sat, 06 Feb 2010 02:36:26 +0000 Subject: hg: jdk7/hotspot-gc: Added tag jdk7-b82 for changeset e1176f86805f Message-ID: <20100206023626.578A841FE1@hg.openjdk.java.net> Changeset: 6880a3af9add Author: mikejwre Date: 2010-02-04 11:19 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/rev/6880a3af9add Added tag jdk7-b82 for changeset e1176f86805f ! .hgtags From john.coomes at sun.com Sat Feb 6 02:36:32 2010 From: john.coomes at sun.com (john.coomes at sun.com) Date: Sat, 06 Feb 2010 02:36:32 +0000 Subject: hg: jdk7/hotspot-gc/corba: Added tag jdk7-b82 for changeset 1e8c1bfad1ab Message-ID: <20100206023637.9224641FE2@hg.openjdk.java.net> Changeset: fde0df7a2384 Author: mikejwre Date: 2010-02-04 11:19 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/corba/rev/fde0df7a2384 Added tag jdk7-b82 for changeset 1e8c1bfad1ab ! .hgtags From john.coomes at sun.com Sat Feb 6 02:39:59 2010 From: john.coomes at sun.com (john.coomes at sun.com) Date: Sat, 06 Feb 2010 02:39:59 +0000 Subject: hg: jdk7/hotspot-gc/jaxp: Added tag jdk7-b82 for changeset c876ad22e4bf Message-ID: <20100206023959.C098041FE3@hg.openjdk.java.net> Changeset: 309a0a7fc6ce Author: mikejwre Date: 2010-02-04 11:19 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jaxp/rev/309a0a7fc6ce Added tag jdk7-b82 for changeset c876ad22e4bf ! .hgtags From john.coomes at sun.com Sat Feb 6 02:40:07 2010 From: john.coomes at sun.com (john.coomes at sun.com) Date: Sat, 06 Feb 2010 02:40:07 +0000 Subject: hg: jdk7/hotspot-gc/jaxws: Added tag jdk7-b82 for changeset 31573ae8eed1 Message-ID: <20100206024008.366FF41FE4@hg.openjdk.java.net> Changeset: 371e3ded591d Author: mikejwre Date: 2010-02-04 11:19 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jaxws/rev/371e3ded591d Added tag jdk7-b82 for changeset 31573ae8eed1 ! .hgtags From john.coomes at sun.com Sat Feb 6 02:40:20 2010 From: john.coomes at sun.com (john.coomes at sun.com) Date: Sat, 06 Feb 2010 02:40:20 +0000 Subject: hg: jdk7/hotspot-gc/jdk: 4 new changesets Message-ID: <20100206024204.6528341FE6@hg.openjdk.java.net> Changeset: b250e6c5a9e5 Author: andrew Date: 2010-01-28 21:01 +0000 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/b250e6c5a9e5 6921068: Remove javadoc builds warnings from specdefault tag Summary: Ignore specdefault tag to avoid javadoc warnings Reviewed-by: darcy, ohair ! make/docs/Makefile Changeset: b3b10d45778a Author: andrew Date: 2010-01-29 02:38 +0000 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/b3b10d45778a 6917466: Should set bootclasspath for javadoc in jdk build Summary: javadoc needs to reference the newly built classes so as not to hit APIs which differ between the boot JDK and the source files of the JDK being built. Reviewed-by: ohair ! make/common/shared/Defs-java.gmk ! make/javax/swing/beaninfo/SwingBeans.gmk Changeset: 69ef657320ad Author: ohair Date: 2010-01-29 09:05 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/69ef657320ad Merge Changeset: 9027c6b9d7e2 Author: mikejwre Date: 2010-02-04 11:19 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/9027c6b9d7e2 Added tag jdk7-b82 for changeset 69ef657320ad ! .hgtags From john.coomes at sun.com Sat Feb 6 02:48:07 2010 From: john.coomes at sun.com (john.coomes at sun.com) Date: Sat, 06 Feb 2010 02:48:07 +0000 Subject: hg: jdk7/hotspot-gc/langtools: Added tag jdk7-b82 for changeset 47003a3622f6 Message-ID: <20100206024814.5940441FE8@hg.openjdk.java.net> Changeset: c9f4ae1f1480 Author: mikejwre Date: 2010-02-04 11:19 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/langtools/rev/c9f4ae1f1480 Added tag jdk7-b82 for changeset 47003a3622f6 ! .hgtags From Igor.Veresov at Sun.COM Mon Feb 8 04:00:47 2010 From: Igor.Veresov at Sun.COM (Igor Veresov) Date: Sun, 07 Feb 2010 20:00:47 -0800 Subject: review request (M) - 4396719 mark sweep stack overflow In-Reply-To: <19308.50788.240081.902763@sun.com> References: <19308.50788.240081.902763@sun.com> Message-ID: <4B6F8C6F.20101@sun.com> Reviewed. igor On 2/5/10 5:31 PM, John Coomes wrote: > I'd like reviews of the fix for > > 4396719 Mark Sweep stack overflow on deeply nested Object arrays > > I split the review into two parts in an attempt to make it easier. > The first part fixes the bug by using an explicit stack/queue for > object arrays: > > http://cr.openjdk.java.net/~jcoomes/4396719-objarray/ > > The second part adds a size parameter to the TaskQueue class; the new > objArray queue used by par compact doesn't need to be as big as the > default queue size, particularly in 64-bit. > > http://cr.openjdk.java.net/~jcoomes/4396719-tqsize/ > > Performance testing is still in progress. > > -John > From tony.printezis at sun.com Mon Feb 8 17:04:14 2010 From: tony.printezis at sun.com (Tony Printezis) Date: Mon, 08 Feb 2010 12:04:14 -0500 Subject: CRR: 6802453: G1: hr()->is_in_reserved(from),"Precondition." (S) Message-ID: <4B70440E.3040807@sun.com> Hi all, I'm opening this for code review, after testing it extensively: http://cr.openjdk.java.net/~tonyp/6802453/webrev.0/ The change looks bigger than it really is, given that I had to move par_expand() further down in the file so that it is public. Tony -- --------------------------------------------------------------------- | Tony Printezis, Staff Engineer | Sun Microsystems Inc. | | | MS UBUR02-311 | | e-mail: tony.printezis at sun.com | 35 Network Drive | | office: +1 781 442 0998 (x20998) | Burlington, MA 01803-2756, USA | --------------------------------------------------------------------- e-mail client: Thunderbird (Linux) From Igor.Veresov at Sun.COM Mon Feb 8 18:59:14 2010 From: Igor.Veresov at Sun.COM (Igor Veresov) Date: Mon, 08 Feb 2010 10:59:14 -0800 Subject: CRR: 6802453: G1: hr()->is_in_reserved(from), "Precondition." (S) In-Reply-To: <4B70440E.3040807@sun.com> References: <4B70440E.3040807@sun.com> Message-ID: <4B705F02.5050800@sun.com> Looks good. igor On 2/8/10 9:04 AM, Tony Printezis wrote: > Hi all, > > I'm opening this for code review, after testing it extensively: > > http://cr.openjdk.java.net/~tonyp/6802453/webrev.0/ > > The change looks bigger than it really is, given that I had to move > par_expand() further down in the file so that it is public. > > Tony > From John.Cuthbertson at Sun.COM Mon Feb 8 19:05:24 2010 From: John.Cuthbertson at Sun.COM (john cuthbertson - Sun Microsystems) Date: Mon, 08 Feb 2010 11:05:24 -0800 Subject: CRR: 6802453: G1: hr()->is_in_reserved(from), "Precondition." (S) In-Reply-To: <4B70440E.3040807@sun.com> References: <4B70440E.3040807@sun.com> Message-ID: <4B706074.4060402@sun.com> Hi Tony, The change looks good to me. JohnC On 02/08/10 09:04, Tony Printezis wrote: > Hi all, > > I'm opening this for code review, after testing it extensively: > > http://cr.openjdk.java.net/~tonyp/6802453/webrev.0/ > > The change looks bigger than it really is, given that I had to move > par_expand() further down in the file so that it is public. > > Tony > From Igor.Veresov at Sun.COM Mon Feb 8 19:20:48 2010 From: Igor.Veresov at Sun.COM (Igor Veresov) Date: Mon, 08 Feb 2010 11:20:48 -0800 Subject: Fwd: Request for review(M): 6923991: G1: improve scalability of RSet scanning Message-ID: <4B706410.50003@sun.com> -------- Original Message -------- Subject: Request for review(M): 6923991: G1: improve scalability of RSet scanning Date: Fri, 05 Feb 2010 20:29:15 -0800 From: Igor Veresov To: g1 at sun.com G1 showed poor load balancing and high contention behavior during RSet scanning on CMT machines as well as on many-core x86. Previously implemented exponential skipping had unacceptable performance when it came to a fairly large processor count. The fix is to implement block-based work stealing in the rset scanning phase. It was also required to get rid of copying during this phase and postpone it until the main copying phase, where we have a fine-grain load balancing on the object granularity. I've also made the size of the sparse PRT entry and number of entries in the fine-grain PRT depend on the region size. Webrev: http://cr.openjdk.java.net/~iveresov/6923991/webrev.00/ igor From tony.printezis at sun.com Mon Feb 8 19:27:35 2010 From: tony.printezis at sun.com (Tony Printezis) Date: Mon, 08 Feb 2010 14:27:35 -0500 Subject: CRR: 6802453: G1: hr()->is_in_reserved(from), "Precondition." (S) In-Reply-To: <4B70440E.3040807@sun.com> References: <4B70440E.3040807@sun.com> Message-ID: <4B7065A7.1050305@sun.com> Hi all, All set thanks to Igor and John Cuthbertson. I'll push this first thing tomorrow morning... Tony Tony Printezis wrote: > Hi all, > > I'm opening this for code review, after testing it extensively: > > http://cr.openjdk.java.net/~tonyp/6802453/webrev.0/ > > The change looks bigger than it really is, given that I had to move > par_expand() further down in the file so that it is public. > > Tony > -- --------------------------------------------------------------------- | Tony Printezis, Staff Engineer | Sun Microsystems Inc. | | | MS UBUR02-311 | | e-mail: tony.printezis at sun.com | 35 Network Drive | | office: +1 781 442 0998 (x20998) | Burlington, MA 01803-2756, USA | --------------------------------------------------------------------- e-mail client: Thunderbird (Linux) From john.cuthbertson at sun.com Tue Feb 9 07:58:03 2010 From: john.cuthbertson at sun.com (john.cuthbertson at sun.com) Date: Tue, 09 Feb 2010 07:58:03 +0000 Subject: hg: jdk7/hotspot-gc/hotspot: 27 new changesets Message-ID: <20100209075936.B257C4181E@hg.openjdk.java.net> Changeset: 9e1637a04678 Author: twisti Date: 2010-01-28 08:36 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/9e1637a04678 6920970: Zero build fixes after 6849984 and 6911204 Summary: Two recent commits broke the build on Zero. Reviewed-by: twisti Contributed-by: Gary Benson ! src/cpu/zero/vm/interpreter_zero.cpp ! src/cpu/zero/vm/sharedRuntime_zero.cpp Changeset: 8d9bfe6a446b Author: never Date: 2010-01-28 16:28 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/8d9bfe6a446b 6920346: G1: "must avoid base_memory and AliasIdxTop" Reviewed-by: kvn ! src/share/vm/opto/memnode.hpp Changeset: 0e14bd797dad Author: never Date: 2010-01-28 20:41 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/0e14bd797dad 6792161: assert("No dead instructions after post-alloc") Reviewed-by: kvn ! src/share/vm/opto/ifg.cpp + test/compiler/6792161/Test6792161.java Changeset: ba263cfb7611 Author: twisti Date: 2010-01-29 12:13 +0100 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/ba263cfb7611 6917766: JSR 292 needs its own deopt handler Summary: We need to introduce a new MH deopt handler so we can easily determine if the deopt happened at a MH call site or not. Reviewed-by: never, jrose ! src/cpu/sparc/vm/c1_LIRAssembler_sparc.cpp ! src/cpu/sparc/vm/frame_sparc.cpp ! src/cpu/x86/vm/c1_LIRAssembler_x86.cpp ! src/cpu/x86/vm/frame_x86.cpp ! src/cpu/x86/vm/frame_x86.inline.hpp ! src/share/vm/asm/codeBuffer.hpp ! src/share/vm/c1/c1_Compilation.cpp ! src/share/vm/c1/c1_LIRAssembler.hpp ! src/share/vm/code/nmethod.cpp ! src/share/vm/code/nmethod.hpp ! src/share/vm/opto/output.cpp ! src/share/vm/runtime/deoptimization.cpp ! src/share/vm/runtime/frame.cpp ! src/share/vm/runtime/frame.hpp ! src/share/vm/runtime/sharedRuntime.cpp Changeset: 24128c2ffa87 Author: twisti Date: 2010-01-29 08:33 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/24128c2ffa87 6921339: backout 6917766 Reviewed-by: mr ! src/cpu/sparc/vm/c1_LIRAssembler_sparc.cpp ! src/cpu/sparc/vm/frame_sparc.cpp ! src/cpu/x86/vm/c1_LIRAssembler_x86.cpp ! src/cpu/x86/vm/frame_x86.cpp ! src/cpu/x86/vm/frame_x86.inline.hpp ! src/share/vm/asm/codeBuffer.hpp ! src/share/vm/c1/c1_Compilation.cpp ! src/share/vm/c1/c1_LIRAssembler.hpp ! src/share/vm/code/nmethod.cpp ! src/share/vm/code/nmethod.hpp ! src/share/vm/opto/output.cpp ! src/share/vm/runtime/deoptimization.cpp ! src/share/vm/runtime/frame.cpp ! src/share/vm/runtime/frame.hpp ! src/share/vm/runtime/sharedRuntime.cpp Changeset: 5f24d0319e54 Author: kvn Date: 2010-01-29 09:27 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/5f24d0319e54 4360113: Evict nmethods when code cache gets full Summary: Speculatively unload the oldest nmethods when code cache gets full. Reviewed-by: never, kvn Contributed-by: eric.caspole at amd.com ! src/share/vm/ci/ciEnv.cpp ! src/share/vm/code/codeCache.cpp ! src/share/vm/code/codeCache.hpp ! src/share/vm/code/nmethod.cpp ! src/share/vm/code/nmethod.hpp ! src/share/vm/compiler/compileBroker.cpp ! src/share/vm/compiler/compileBroker.hpp ! src/share/vm/includeDB_compiler2 ! src/share/vm/includeDB_core ! src/share/vm/oops/methodOop.cpp ! src/share/vm/oops/methodOop.hpp ! src/share/vm/opto/output.cpp ! src/share/vm/runtime/compilationPolicy.cpp ! src/share/vm/runtime/globals.hpp ! src/share/vm/runtime/sharedRuntime.cpp ! src/share/vm/runtime/sweeper.cpp ! src/share/vm/runtime/sweeper.hpp ! src/share/vm/runtime/vm_operations.cpp ! src/share/vm/runtime/vm_operations.hpp Changeset: 844a9d73ec22 Author: never Date: 2010-01-29 22:51 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/844a9d73ec22 6916644: C2 compiler crash on x86 Reviewed-by: kvn, twisti ! src/share/vm/adlc/output_c.cpp ! src/share/vm/adlc/output_h.cpp ! src/share/vm/opto/machnode.hpp ! src/share/vm/opto/matcher.cpp + test/compiler/6916644/Test6916644.java Changeset: 18a389214829 Author: twisti Date: 2010-02-01 19:29 +0100 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/18a389214829 6921352: JSR 292 needs its own deopt handler Summary: We need to introduce a new MH deopt handler so we can easily determine if the deopt happened at a MH call site or not. Reviewed-by: never, jrose ! src/cpu/sparc/vm/c1_LIRAssembler_sparc.cpp ! src/cpu/sparc/vm/frame_sparc.cpp ! src/cpu/x86/vm/c1_LIRAssembler_x86.cpp ! src/cpu/x86/vm/frame_x86.cpp ! src/cpu/x86/vm/frame_x86.hpp ! src/cpu/x86/vm/frame_x86.inline.hpp ! src/share/vm/asm/codeBuffer.hpp ! src/share/vm/c1/c1_Compilation.cpp ! src/share/vm/c1/c1_LIRAssembler.hpp ! src/share/vm/code/nmethod.cpp ! src/share/vm/code/nmethod.hpp ! src/share/vm/opto/output.cpp ! src/share/vm/runtime/deoptimization.cpp ! src/share/vm/runtime/frame.cpp ! src/share/vm/runtime/sharedRuntime.cpp Changeset: 5fcfaa1ad96f Author: twisti Date: 2010-02-01 23:18 +0100 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/5fcfaa1ad96f 6921799: JSR 292 call sites should not be fixed-up Summary: MethodHandle invoke call sites should not be fixed-up by SharedRuntime::fixup_callers_callsite as c2i/i2c adapters are used to implement MethodHandle actions. Reviewed-by: kvn, never ! src/share/vm/runtime/sharedRuntime.cpp Changeset: 87684f1a88b5 Author: kvn Date: 2010-02-01 16:49 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/87684f1a88b5 6614597: Performance variability in jvm2008 xml.validation Summary: Fix incorrect marking of methods as not compilable. Reviewed-by: never ! src/cpu/sparc/vm/interp_masm_sparc.cpp ! src/cpu/sparc/vm/interp_masm_sparc.hpp ! src/cpu/x86/vm/c1_LIRAssembler_x86.cpp ! src/cpu/x86/vm/interp_masm_x86_32.cpp ! src/cpu/x86/vm/interp_masm_x86_32.hpp ! src/cpu/x86/vm/interp_masm_x86_64.cpp ! src/cpu/x86/vm/interp_masm_x86_64.hpp ! src/share/vm/ci/ciMethod.cpp ! src/share/vm/code/dependencies.cpp ! src/share/vm/code/nmethod.cpp ! src/share/vm/oops/methodDataOop.hpp ! src/share/vm/oops/methodOop.cpp ! src/share/vm/opto/doCall.cpp ! src/share/vm/opto/parseHelper.cpp ! src/share/vm/opto/runtime.cpp ! src/share/vm/runtime/deoptimization.cpp ! src/share/vm/runtime/deoptimization.hpp ! src/share/vm/runtime/globals.hpp Changeset: 74c848d437ab Author: never Date: 2010-02-03 12:28 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/74c848d437ab 6921922: fix for 6911204 breaks tagged stack interpreter Reviewed-by: kvn ! src/share/vm/runtime/globals.hpp ! src/share/vm/runtime/sharedRuntime.cpp ! src/share/vm/runtime/sharedRuntime.hpp Changeset: cef333a48af6 Author: kvn Date: 2010-02-03 15:03 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/cef333a48af6 6923043: failed nightly tests which use -XX:+PrintCompilation -Xcomp -XX:CompileOnly Summary: Print "made not compilable" line only for deoptimizations. Reviewed-by: never ! src/share/vm/compiler/compileBroker.cpp ! src/share/vm/oops/methodOop.cpp ! src/share/vm/oops/methodOop.hpp Changeset: e8443c7be117 Author: never Date: 2010-02-03 15:56 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/e8443c7be117 6921969: optimize 64 long multiply for case with high bits zero Reviewed-by: never, twisti, kvn, rasbold Contributed-by: Hiroshi Yamauchi ! src/cpu/x86/vm/x86_32.ad + test/compiler/6921969/TestMultiplyLongHiZero.java Changeset: 0fce83e8dc0e Author: never Date: 2010-02-03 18:33 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/0fce83e8dc0e Merge Changeset: c1f1137b3575 Author: twisti Date: 2010-02-04 03:34 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/c1f1137b3575 Merge ! src/share/vm/runtime/globals.hpp Changeset: c028504fdaa6 Author: never Date: 2010-02-04 11:16 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/c028504fdaa6 6921992: failure in verify scheduling after 6792161 Reviewed-by: kvn ! src/share/vm/opto/ifg.cpp Changeset: 6deeaebad47a Author: dcubed Date: 2010-02-01 17:35 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/6deeaebad47a 6902182: 4/4 Starting with jdwp agent should not incur performance penalty Summary: Rename can_post_exceptions support to can_post_on_exceptions. Add support for should_post_on_exceptions flag to permit per JavaThread optimizations. Reviewed-by: never, kvn, dcubed Contributed-by: tom.deneau at amd.com ! src/cpu/sparc/vm/c1_LIRAssembler_sparc.cpp ! src/cpu/x86/vm/c1_LIRAssembler_x86.cpp ! src/share/vm/c1/c1_LIRGenerator.cpp ! src/share/vm/c1/c1_Runtime1.cpp ! src/share/vm/ci/ciEnv.cpp ! src/share/vm/ci/ciEnv.hpp ! src/share/vm/interpreter/interpreterRuntime.cpp ! src/share/vm/opto/graphKit.cpp ! src/share/vm/opto/graphKit.hpp ! src/share/vm/opto/parse2.cpp ! src/share/vm/opto/runtime.cpp ! src/share/vm/prims/jvmtiEventController.cpp ! src/share/vm/prims/jvmtiExport.cpp ! src/share/vm/prims/jvmtiExport.hpp ! src/share/vm/prims/jvmtiManageCapabilities.cpp ! src/share/vm/prims/jvmtiThreadState.hpp ! src/share/vm/runtime/sharedRuntime.cpp ! src/share/vm/runtime/thread.cpp ! src/share/vm/runtime/thread.hpp Changeset: 0fc941df6fb7 Author: dcubed Date: 2010-02-02 10:37 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/0fc941df6fb7 6918421: 1/1 in-process JVM now ignores preset Windows unhandled exception filter Summary: Add support for chaining Windows UnhandledExceptionFilter handlers Reviewed-by: kamg, dholmes, never, acorn, ikrylov ! src/os/windows/vm/os_windows.cpp Changeset: f19bf22685cc Author: dcubed Date: 2010-02-02 11:08 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/f19bf22685cc Merge ! src/os/windows/vm/os_windows.cpp Changeset: 7f8790caccb0 Author: apangin Date: 2010-02-04 15:50 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/7f8790caccb0 Merge ! src/cpu/sparc/vm/c1_LIRAssembler_sparc.cpp ! src/cpu/x86/vm/c1_LIRAssembler_x86.cpp ! src/share/vm/ci/ciEnv.cpp ! src/share/vm/opto/runtime.cpp ! src/share/vm/runtime/sharedRuntime.cpp Changeset: 359445e739ac Author: mikejwre Date: 2010-01-21 11:12 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/359445e739ac Added tag jdk7-b80 for changeset 3003ddd1d433 ! .hgtags Changeset: 10f901469941 Author: trims Date: 2010-01-22 14:57 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/10f901469941 Merge Changeset: 1f9b07674480 Author: trims Date: 2010-01-22 15:01 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/1f9b07674480 6919437: Bump the HS17 build number to 08 Summary: Update the HS17 build number to 08 Reviewed-by: jcoomes ! make/hotspot_version Changeset: 1999f5b12482 Author: mikejwre Date: 2010-01-28 11:26 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/1999f5b12482 Added tag jdk7-b81 for changeset 1f9b07674480 ! .hgtags Changeset: dba18cabafec Author: trims Date: 2010-02-05 12:26 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/dba18cabafec Merge Changeset: ff3232b68fbb Author: trims Date: 2010-02-05 12:27 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/ff3232b68fbb 6921473: Bump the HS17 build number to 09 Summary: Update the HS17 build number to 09 Reviewed-by: jcoomes ! make/hotspot_version Changeset: 230fac611b50 Author: johnc Date: 2010-02-08 09:58 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/230fac611b50 Merge ! src/share/vm/c1/c1_Runtime1.cpp ! src/share/vm/includeDB_core From antonios.printezis at sun.com Wed Feb 10 02:57:28 2010 From: antonios.printezis at sun.com (antonios.printezis at sun.com) Date: Wed, 10 Feb 2010 02:57:28 +0000 Subject: hg: jdk7/hotspot-gc/hotspot: 2 new changesets Message-ID: <20100210025739.9DB0741931@hg.openjdk.java.net> Changeset: 38836cf1d8d2 Author: tonyp Date: 2010-02-05 11:05 -0500 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/38836cf1d8d2 6920977: G1: guarantee(k == probe->klass(),"klass should be in dictionary") fails Summary: the guarantee is too strict and the test will fail (incorrectly) if the class is not in the system dictionary but in the placeholders. Reviewed-by: acorn, phh ! src/share/vm/classfile/loaderConstraints.cpp ! src/share/vm/classfile/loaderConstraints.hpp ! src/share/vm/classfile/systemDictionary.cpp ! src/share/vm/includeDB_core Changeset: 9eee977dd1a9 Author: tonyp Date: 2010-02-08 14:23 -0500 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/9eee977dd1a9 6802453: G1: hr()->is_in_reserved(from),"Precondition." Summary: The operations of re-using a RSet component and expanding the same RSet component were not mutually exlusive, and this could lead to RSets getting corrupted and entries being dropped. Reviewed-by: iveresov, johnc ! src/share/vm/gc_implementation/g1/heapRegionRemSet.cpp From john.cuthbertson at sun.com Wed Feb 10 08:46:13 2010 From: john.cuthbertson at sun.com (john.cuthbertson at sun.com) Date: Wed, 10 Feb 2010 08:46:13 +0000 Subject: hg: jdk7/hotspot-gc/hotspot: 6782663: Data produced by PrintGCApplicationConcurrentTime and PrintGCApplicationStoppedTime is not accurate. Message-ID: <20100210084616.C00FF41983@hg.openjdk.java.net> Changeset: 8859772195c6 Author: johnc Date: 2010-02-09 13:56 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/8859772195c6 6782663: Data produced by PrintGCApplicationConcurrentTime and PrintGCApplicationStoppedTime is not accurate. Summary: Update and display the timers associated with these flags for all safepoints. Reviewed-by: ysr, jcoomes ! src/share/vm/runtime/vmThread.cpp ! src/share/vm/services/runtimeService.cpp From John.Coomes at sun.com Thu Feb 11 00:41:52 2010 From: John.Coomes at sun.com (John Coomes) Date: Wed, 10 Feb 2010 16:41:52 -0800 Subject: review request (M) - 4396719 mark sweep stack overflow In-Reply-To: <4B6F8C6F.20101@sun.com> References: <19308.50788.240081.902763@sun.com> <4B6F8C6F.20101@sun.com> Message-ID: <19315.21072.592798.497788@sun.com> Igor Veresov (Igor.Veresov at Sun.COM) wrote: > Reviewed. Many thanks! -John > On 2/5/10 5:31 PM, John Coomes wrote: > > I'd like reviews of the fix for > > > > 4396719 Mark Sweep stack overflow on deeply nested Object arrays > > > > I split the review into two parts in an attempt to make it easier. > > The first part fixes the bug by using an explicit stack/queue for > > object arrays: > > > > http://cr.openjdk.java.net/~jcoomes/4396719-objarray/ > > > > The second part adds a size parameter to the TaskQueue class; the new > > objArray queue used by par compact doesn't need to be as big as the > > default queue size, particularly in 64-bit. > > > > http://cr.openjdk.java.net/~jcoomes/4396719-tqsize/ > > > > Performance testing is still in progress. > > > > -John > > > From igor.veresov at sun.com Fri Feb 12 01:46:24 2010 From: igor.veresov at sun.com (igor.veresov at sun.com) Date: Fri, 12 Feb 2010 01:46:24 +0000 Subject: hg: jdk7/hotspot-gc/hotspot: 6923991: G1: improve scalability of RSet scanning Message-ID: <20100212014628.570B441C17@hg.openjdk.java.net> Changeset: 0414c1049f15 Author: iveresov Date: 2010-02-11 15:52 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/0414c1049f15 6923991: G1: improve scalability of RSet scanning Summary: Implemented block-based work stealing. Moved copying during the rset scanning phase to the main copying phase. Made the size of rset table depend on the region size. Reviewed-by: apetrusenko, tonyp ! src/share/vm/gc_implementation/g1/g1CollectedHeap.cpp ! src/share/vm/gc_implementation/g1/g1CollectedHeap.hpp ! src/share/vm/gc_implementation/g1/g1CollectorPolicy.cpp ! src/share/vm/gc_implementation/g1/g1OopClosures.hpp ! src/share/vm/gc_implementation/g1/g1OopClosures.inline.hpp ! src/share/vm/gc_implementation/g1/g1RemSet.cpp ! src/share/vm/gc_implementation/g1/g1_globals.hpp ! src/share/vm/gc_implementation/g1/g1_specialized_oop_closures.hpp ! src/share/vm/gc_implementation/g1/heapRegionRemSet.cpp ! src/share/vm/gc_implementation/g1/heapRegionRemSet.hpp ! src/share/vm/gc_implementation/g1/sparsePRT.cpp ! src/share/vm/gc_implementation/g1/sparsePRT.hpp ! src/share/vm/memory/cardTableModRefBS.hpp ! src/share/vm/utilities/globalDefinitions.hpp From john.coomes at sun.com Sun Feb 14 20:07:54 2010 From: john.coomes at sun.com (john.coomes at sun.com) Date: Sun, 14 Feb 2010 20:07:54 +0000 Subject: hg: jdk7/hotspot-gc: Added tag jdk7-b83 for changeset 6880a3af9add Message-ID: <20100214200754.274754207E@hg.openjdk.java.net> Changeset: 2f3ea057d1ad Author: mikejwre Date: 2010-02-12 13:25 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/rev/2f3ea057d1ad Added tag jdk7-b83 for changeset 6880a3af9add ! .hgtags From john.coomes at sun.com Sun Feb 14 20:08:09 2010 From: john.coomes at sun.com (john.coomes at sun.com) Date: Sun, 14 Feb 2010 20:08:09 +0000 Subject: hg: jdk7/hotspot-gc/corba: Added tag jdk7-b83 for changeset fde0df7a2384 Message-ID: <20100214200814.2C0554207F@hg.openjdk.java.net> Changeset: 68c8961a82e4 Author: mikejwre Date: 2010-02-12 13:25 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/corba/rev/68c8961a82e4 Added tag jdk7-b83 for changeset fde0df7a2384 ! .hgtags From john.coomes at sun.com Sun Feb 14 20:12:16 2010 From: john.coomes at sun.com (john.coomes at sun.com) Date: Sun, 14 Feb 2010 20:12:16 +0000 Subject: hg: jdk7/hotspot-gc/jaxp: Added tag jdk7-b83 for changeset 309a0a7fc6ce Message-ID: <20100214201217.9FB1642081@hg.openjdk.java.net> Changeset: c664aff74b41 Author: mikejwre Date: 2010-02-12 13:25 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jaxp/rev/c664aff74b41 Added tag jdk7-b83 for changeset 309a0a7fc6ce ! .hgtags From john.coomes at sun.com Sun Feb 14 20:12:33 2010 From: john.coomes at sun.com (john.coomes at sun.com) Date: Sun, 14 Feb 2010 20:12:33 +0000 Subject: hg: jdk7/hotspot-gc/jaxws: Added tag jdk7-b83 for changeset 371e3ded591d Message-ID: <20100214201233.F158642082@hg.openjdk.java.net> Changeset: 8bc02839eee4 Author: mikejwre Date: 2010-02-12 13:25 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jaxws/rev/8bc02839eee4 Added tag jdk7-b83 for changeset 371e3ded591d ! .hgtags From john.coomes at sun.com Sun Feb 14 20:12:51 2010 From: john.coomes at sun.com (john.coomes at sun.com) Date: Sun, 14 Feb 2010 20:12:51 +0000 Subject: hg: jdk7/hotspot-gc/jdk: Added tag jdk7-b83 for changeset 9027c6b9d7e2 Message-ID: <20100214201425.5847442083@hg.openjdk.java.net> Changeset: 9b82fa0505bc Author: mikejwre Date: 2010-02-12 13:25 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/9b82fa0505bc Added tag jdk7-b83 for changeset 9027c6b9d7e2 ! .hgtags From john.coomes at sun.com Sun Feb 14 20:17:27 2010 From: john.coomes at sun.com (john.coomes at sun.com) Date: Sun, 14 Feb 2010 20:17:27 +0000 Subject: hg: jdk7/hotspot-gc/langtools: Added tag jdk7-b83 for changeset c9f4ae1f1480 Message-ID: <20100214201748.0AB3842085@hg.openjdk.java.net> Changeset: 2edcb5dc642d Author: mikejwre Date: 2010-02-12 13:25 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/langtools/rev/2edcb5dc642d Added tag jdk7-b83 for changeset c9f4ae1f1480 ! .hgtags From john.cuthbertson at sun.com Wed Feb 17 03:46:48 2010 From: john.cuthbertson at sun.com (john.cuthbertson at sun.com) Date: Wed, 17 Feb 2010 03:46:48 +0000 Subject: hg: jdk7/hotspot-gc/hotspot: 11 new changesets Message-ID: <20100217034724.1B50D4242C@hg.openjdk.java.net> Changeset: 455df1b81409 Author: kamg Date: 2010-02-08 13:49 -0500 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/455df1b81409 6587322: dtrace probe object__alloc doesn't fire in some situations on amd64 Summary: Fix misplaced probe point Reviewed-by: rasbold, phh Contributed-by: neojia at gmail.com ! src/cpu/x86/vm/templateTable_x86_64.cpp Changeset: 95d21201c29a Author: apangin Date: 2010-02-11 10:48 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/95d21201c29a Merge Changeset: 3f5b7efb9642 Author: never Date: 2010-02-05 11:07 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/3f5b7efb9642 6920293: OptimizeStringConcat causing core dumps Reviewed-by: kvn, twisti ! src/os_cpu/solaris_x86/vm/os_solaris_x86.cpp ! src/share/vm/code/nmethod.cpp ! src/share/vm/opto/stringopts.cpp ! src/share/vm/runtime/sharedRuntime.cpp Changeset: 576e77447e3c Author: kvn Date: 2010-02-07 12:15 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/576e77447e3c 6923002: assert(false,"this call site should not be polymorphic") Summary: Clear the total count when a receiver information is cleared. Reviewed-by: never, jrose ! src/cpu/sparc/vm/c1_LIRAssembler_sparc.cpp ! src/cpu/sparc/vm/interp_masm_sparc.cpp ! src/cpu/sparc/vm/sharedRuntime_sparc.cpp ! src/cpu/x86/vm/c1_LIRAssembler_x86.cpp ! src/cpu/x86/vm/interp_masm_x86_32.cpp ! src/cpu/x86/vm/interp_masm_x86_64.cpp ! src/share/vm/ci/ciMethod.cpp ! src/share/vm/oops/methodDataOop.hpp ! src/share/vm/opto/doCall.cpp ! src/share/vm/opto/runtime.cpp ! src/share/vm/runtime/arguments.cpp Changeset: f516d5d7a019 Author: kvn Date: 2010-02-08 12:20 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/f516d5d7a019 6910605: C2: NullPointerException/ClassCaseException is thrown when C2 with DeoptimizeALot is used Summary: Set the reexecute bit for runtime calls _new_array_Java when they used for _multianewarray bytecode. Reviewed-by: never ! src/share/vm/code/pcDesc.cpp ! src/share/vm/opto/graphKit.cpp ! src/share/vm/opto/parse3.cpp + test/compiler/6910605/Test.java Changeset: f70b0d9ab095 Author: kvn Date: 2010-02-09 01:31 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/f70b0d9ab095 6910618: C2: Error: assert(d->is_oop(),"JVM_ArrayCopy: dst not an oop") Summary: Mark in PcDesc call sites which return oop and save the result oop across objects reallocation during deoptimization. Reviewed-by: never ! src/share/vm/c1/c1_IR.hpp ! src/share/vm/code/debugInfoRec.cpp ! src/share/vm/code/debugInfoRec.hpp ! src/share/vm/code/nmethod.cpp ! src/share/vm/code/pcDesc.hpp ! src/share/vm/code/scopeDesc.cpp ! src/share/vm/code/scopeDesc.hpp ! src/share/vm/includeDB_core ! src/share/vm/opto/output.cpp ! src/share/vm/prims/jvmtiCodeBlobEvents.cpp ! src/share/vm/runtime/deoptimization.cpp + test/compiler/6910618/Test.java Changeset: 4ee1c645110e Author: kvn Date: 2010-02-09 10:21 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/4ee1c645110e 6924097: assert((_type == Type::MEMORY) == (_adr_type != 0),"adr_type for memory phis only") Summary: Use PhiNode::make_blank(r, n) method to construct the phi. Reviewed-by: never ! src/share/vm/opto/loopopts.cpp Changeset: e3a4305c6bc3 Author: kvn Date: 2010-02-12 08:54 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/e3a4305c6bc3 6925249: assert(last_sp < (intptr_t*) interpreter_frame_monitor_begin(),"bad tos") Summary: Fix assert since top deoptimized frame has last_sp == interpreter_frame_monitor_begin if there are no expressions. Reviewed-by: twisti ! src/cpu/x86/vm/frame_x86.inline.hpp ! src/share/vm/runtime/deoptimization.cpp ! src/share/vm/runtime/frame.cpp ! src/share/vm/runtime/vframeArray.cpp Changeset: c09ee209b65c Author: kvn Date: 2010-02-12 10:34 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/c09ee209b65c 6926048: Improve Zero performance Summary: Make Zero figure out result types in a similar way to C++ interpreter implementation. Reviewed-by: kvn Contributed-by: gbenson at redhat.com ! src/cpu/zero/vm/cppInterpreter_zero.cpp ! src/cpu/zero/vm/cppInterpreter_zero.hpp Changeset: 7b4415a18c8a Author: kvn Date: 2010-02-12 15:27 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/7b4415a18c8a Merge ! src/cpu/sparc/vm/c1_LIRAssembler_sparc.cpp ! src/cpu/x86/vm/c1_LIRAssembler_x86.cpp ! src/share/vm/includeDB_core ! src/share/vm/opto/graphKit.cpp ! src/share/vm/opto/runtime.cpp ! src/share/vm/runtime/arguments.cpp ! src/share/vm/runtime/sharedRuntime.cpp Changeset: 58add740c4ee Author: johnc Date: 2010-02-16 14:11 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/58add740c4ee Merge ! src/share/vm/includeDB_core From Y.S.Ramakrishna at Sun.COM Wed Feb 17 22:42:34 2010 From: Y.S.Ramakrishna at Sun.COM (Y. Srinivas Ramakrishna) Date: Wed, 17 Feb 2010 14:42:34 -0800 Subject: Incremental CMS: long "[GC [1 CMS-initial-mark:" In-Reply-To: <4B7C6D91.6000202@gmail.com> References: <4B7C6D91.6000202@gmail.com> Message-ID: <4B7C70DA.7070204@Sun.COM> On 02/17/10 14:28, Andrei Pozolotin wrote: > *Jon, *hello; > > I need to convert CMS into incremental CMS to reduce latency, > eliminate occasional "concurrent mode failure" - related pauses; > > and I am running into the following problem: > > 1) in "normal CMS" - [GC [1 CMS-initial-mark: takes 50 milliseconds > > 2) in "incremental CMS" - [GC [1 CMS-initial-mark: takes 800 > milliseconds This is to be expected and is a consequence of how and when the initial mark is done in "normal" vs "incremental". In the former, the initial mark almost immediately follows after a scavenge pause and Eden is empty. In the latter case it normally occurs sometime between two scavenges by which time Eden has objects in it. For various reasons, the young gen is scanned as a source of initial roots to mark from, and for various reasons it is currently the case that this work is done single-threaded. As you will find, the magnitude of this difference typically depends (among other factors) on the size of Eden you specify (being directly proportional to it) and the duty cycle you specify (being inversely proportional to it). There are ways to fix this in the JVM, but no real workaround that i can think of at the command-line level. If this is important, please file a bug on this via your support contacts. thanks! -- ramki > > both above results are under same no-load conditions, > with non-reducible statuc heap object set of 430 MB > composed of some 350K complex objects; > > I can not understand why and how to fix it; > I tried a lot of Incremental CMS - related option combinations, > still dead end. > > GC settings, log attached; > > Thank you for your help, > > Andrei > > ++++++++++++++++++++++++++++++++++++++++++++++++ > > # use server-style JIT compiler > wrapper.java.additional.100=-server > > # do GC logging > wrapper.java.additional.110=-verbose:gc > wrapper.java.additional.111=-Xloggc:./logs/gc.log > wrapper.java.additional.112=-XX:+PrintGCDetails > wrapper.java.additional.113=-XX:+PrintGCTimeStamps > > # Heap Total = PermGen + NewGen + OldGen; > wrapper.java.additional.120=-Xms3700m > wrapper.java.additional.121=-Xmx3700m > > # PermGen; > wrapper.java.additional.130=-XX:PermSize=50m > wrapper.java.additional.131=-XX:MaxPermSize=50m > > # NewGen; Parallel collector; > wrapper.java.additional.140=-XX:NewSize=1200m > wrapper.java.additional.141=-XX:MaxNewSize=1200m > wrapper.java.additional.142=-XX:-UseAdaptiveSizePolicy > wrapper.java.additional.143=-XX:SurvivorRatio=1 > wrapper.java.additional.144=-XX:TargetSurvivorRatio=90 > wrapper.java.additional.145=-XX:MaxTenuringThreshold=3 > wrapper.java.additional.146=-XX:+UseParNewGC > wrapper.java.additional.147=-XX:-UseParallelGC > wrapper.java.additional.148=-XX:ParallelGCThreads=4 > > # OldGen; Incremental CMS collector; > wrapper.java.additional.170=-XX:-UseParallelOldGC > wrapper.java.additional.171=-XX:+UseConcMarkSweepGC > wrapper.java.additional.172=-XX:+CMSIncrementalMode > wrapper.java.additional.173=-XX:-CMSIncrementalPacing > wrapper.java.additional.174=-XX:CMSIncrementalDutyCycle=1 > wrapper.java.additional.175=-XX:+CMSParallelRemarkEnabled > wrapper.java.additional.176=-XX:+CMSScavengeBeforeRemark > wrapper.java.additional.177=-XX:CMSIncrementalOffset=10 > wrapper.java.additional.178=-XX:CMSScheduleRemarkEdenPenetration=5 > wrapper.java.additional.179=-XX:CMSWaitDuration=60000 > wrapper.java.additional.181=-XX:ParallelCMSThreads=1 > wrapper.java.additional.182=-XX:PrintCMSStatistics=0 > > > ------------------------------------------------------------------------ > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From Y.S.Ramakrishna at Sun.COM Wed Feb 17 23:31:23 2010 From: Y.S.Ramakrishna at Sun.COM (Y. Srinivas Ramakrishna) Date: Wed, 17 Feb 2010 15:31:23 -0800 Subject: Incremental CMS: long "[GC [1 CMS-initial-mark:" In-Reply-To: <4B7C7476.6050607@gmail.com> References: <4B7C6D91.6000202@gmail.com> <4B7C70DA.7070204@Sun.COM> <4B7C7476.6050607@gmail.com> Message-ID: <4B7C7C4B.3060207@Sun.COM> On 02/17/10 14:57, Andrei Pozolotin wrote: > Ramki, hi > > Thank you for quick response; > > from reading JDK 6 source I deluded myself that the problem you > describe: > > "... occurs sometime between two scavenges ..." > > would get fixed by these options: > > wrapper.java.additional.175=-XX:+CMSParallelRemarkEnabled > wrapper.java.additional.176=-XX:+CMSScavengeBeforeRemark > wrapper.java.additional.177=-XX:CMSIncrementalOffset=10 > wrapper.java.additional.178=-XX:CMSScheduleRemarkEdenPenetration=5 > > please confirm I was wrong? Yes, unfortunately, those are in some sense orthogonal to the way CMS "incrementalizes" its working, here (somewhat relatedly see [1]) For example, CMSScavengeBeforeRemark does indeed schedule a scavenge before a remark pause, but not (unfortunately) before an initial mark pause. Similarly the CMSScheduleRemarkEdenPenetration controls the scheduling of remark pauses, not the initial mark pauses. As for CMSIncrementalOffset, it is just a constant positive delta applied to the ideal starting point for a given duty cycle. So, in fact, any positive value will take you further away from the objective you are trying to achieve here of reducing the occupancy of Eden when initial mark hits. Try G1, that may save you filing an RFE on (i)CMS for this. Honestly though, there is an existing blanket RFE on this: 6412968 CMS: Long initial mark pauses but it has not risen to the top of our list of things to fix. If this is important, please contact your Java support.... -- ramki [1] 6518192 iCMS: smoother, duration-bounded incrementality > Cheers, > > Andrei > > > > -------- Original Message -------- > Subject: Re: Incremental CMS: long "[GC [1 CMS-initial-mark:" > From: Y. Srinivas Ramakrishna > To: Andrei Pozolotin > Cc: hotspot-gc-use at openjdk.java.net > Date: Wed 17 Feb 2010 04:42:34 PM CST >> On 02/17/10 14:28, Andrei Pozolotin wrote: >>> *Jon, *hello; >>> >>> I need to convert CMS into incremental CMS to reduce latency, >>> eliminate occasional "concurrent mode failure" - related pauses; >>> >>> and I am running into the following problem: >>> >>> 1) in "normal CMS" - [GC [1 CMS-initial-mark: takes 50 milliseconds >>> >>> 2) in "incremental CMS" - [GC [1 CMS-initial-mark: takes 800 >>> milliseconds >> >> This is to be expected and is a consequence of how and when >> the initial mark is done in "normal" vs "incremental". >> In the former, the initial mark almost immediately follows >> after a scavenge pause and Eden is empty. In the latter case >> it normally occurs sometime between two scavenges by which time >> Eden has objects in it. For various reasons, the young gen >> is scanned as a source of initial roots to mark from, >> and for various reasons it is currently the case that this >> work is done single-threaded. >> >> As you will find, the magnitude of this difference typically >> depends (among other factors) on the size of Eden you specify >> (being directly proportional to it) and the duty cycle you >> specify (being inversely proportional to it). >> >> There are ways to fix this in the JVM, but no real workaround that >> i can think of at the command-line level. >> >> If this is important, please file a bug on this via your >> support contacts. >> >> thanks! >> -- ramki >> >>> >>> both above results are under same no-load conditions, >>> with non-reducible statuc heap object set of 430 MB >>> composed of some 350K complex objects; >>> >>> I can not understand why and how to fix it; >>> I tried a lot of Incremental CMS - related option combinations, >>> still dead end. >>> >>> GC settings, log attached; >>> >>> Thank you for your help, >>> >>> Andrei >>> >>> ++++++++++++++++++++++++++++++++++++++++++++++++ >>> >>> # use server-style JIT compiler >>> wrapper.java.additional.100=-server >>> >>> # do GC logging >>> wrapper.java.additional.110=-verbose:gc >>> wrapper.java.additional.111=-Xloggc:./logs/gc.log >>> wrapper.java.additional.112=-XX:+PrintGCDetails >>> wrapper.java.additional.113=-XX:+PrintGCTimeStamps >>> >>> # Heap Total = PermGen + NewGen + OldGen; >>> wrapper.java.additional.120=-Xms3700m >>> wrapper.java.additional.121=-Xmx3700m >>> >>> # PermGen; >>> wrapper.java.additional.130=-XX:PermSize=50m >>> wrapper.java.additional.131=-XX:MaxPermSize=50m >>> >>> # NewGen; Parallel collector; >>> wrapper.java.additional.140=-XX:NewSize=1200m >>> wrapper.java.additional.141=-XX:MaxNewSize=1200m >>> wrapper.java.additional.142=-XX:-UseAdaptiveSizePolicy >>> wrapper.java.additional.143=-XX:SurvivorRatio=1 >>> wrapper.java.additional.144=-XX:TargetSurvivorRatio=90 >>> wrapper.java.additional.145=-XX:MaxTenuringThreshold=3 >>> wrapper.java.additional.146=-XX:+UseParNewGC >>> wrapper.java.additional.147=-XX:-UseParallelGC >>> wrapper.java.additional.148=-XX:ParallelGCThreads=4 >>> >>> # OldGen; Incremental CMS collector; >>> wrapper.java.additional.170=-XX:-UseParallelOldGC >>> wrapper.java.additional.171=-XX:+UseConcMarkSweepGC >>> wrapper.java.additional.172=-XX:+CMSIncrementalMode >>> wrapper.java.additional.173=-XX:-CMSIncrementalPacing >>> wrapper.java.additional.174=-XX:CMSIncrementalDutyCycle=1 >>> wrapper.java.additional.175=-XX:+CMSParallelRemarkEnabled >>> wrapper.java.additional.176=-XX:+CMSScavengeBeforeRemark >>> wrapper.java.additional.177=-XX:CMSIncrementalOffset=10 >>> wrapper.java.additional.178=-XX:CMSScheduleRemarkEdenPenetration=5 >>> wrapper.java.additional.179=-XX:CMSWaitDuration=60000 >>> wrapper.java.additional.181=-XX:ParallelCMSThreads=1 >>> wrapper.java.additional.182=-XX:PrintCMSStatistics=0 >>> >>> >>> ------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> hotspot-gc-use mailing list >>> hotspot-gc-use at openjdk.java.net >>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> > _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From atulksh at hotmail.com Thu Feb 18 01:16:40 2010 From: atulksh at hotmail.com (atulksh) Date: Wed, 17 Feb 2010 17:16:40 -0800 (PST) Subject: long unaccounted pause during concurrent mark phase of ParNew cycle In-Reply-To: <27461205.post@talk.nabble.com> References: <27459365.post@talk.nabble.com> <4B6B3D74.90403@Sun.COM> <27461205.post@talk.nabble.com> Message-ID: <27633143.post@talk.nabble.com> We have been able to workaround this issue with the help of Sun engineers (especially Ramki - thank you very much). I wanted to update this thread with that information so it can help anybody else who may bump up into such a problem. What we have been able to identify as yet, is there is some sort of thread synchronization issue that happens during the concurrent mark phase in CMS cycle (especially near end of the concurrent mark cycle), where the CMS threads do not yield control to the gc threads who want to do a scavenge (young generation collection). The young generation is full and needs a scavenge, the scavenge cannot start because it is waiting for the CMS threads to relinquish control, and all application threads wait for the scavenge to happen. The application goes in a frozen state. Eventually - after tens of seconds - this wait is broken (do not know how) and the scavenge happens and the world is comes to normality and the CMS cycle continues without problems henceforth. This happens pretty much in every CMS cycle. The above only happens if there are more than 1 concurrent marking threads enabled - which in our cases happened only on multi-core machines with 4 or 8 cores. 8 core machines showed this problem much more prominantly than 4 core machines (where at least in a test environment we could reproduce this problem on after manually bumping the ParallelCMSThreads parameter to 4). The more the number of concurrent CMS threads the more likelihood the problem will occur. Our workaround for the issue was to use the -XX:-CMSConcurrentMTEnabled to disable the use of concurrent CMS threads. So far we have not seen any significant degradation in the gc activity due to disabling concurrent MT in CMs with the flag. There is an issue in Sun's bug database for the garbage collector that points to such an issue. Sun engineers are trying to fix this issue as the bug indicates. http://bugs.sun.com/view_bug.do?bug_id=6692906 -- View this message in context: http://old.nabble.com/long-unaccounted-pause-during-concurrent-mark-phase-of-ParNew-cycle-tp27459365p27633143.html Sent from the OpenJDK Hotspot Garbage Collection mailing list archive at Nabble.com. From Jon.Masamitsu at Sun.COM Thu Feb 18 06:25:25 2010 From: Jon.Masamitsu at Sun.COM (Jon Masamitsu) Date: Wed, 17 Feb 2010 22:25:25 -0800 Subject: Incremental CMS: long "[GC [1 CMS-initial-mark:" Message-ID: <301DA800-BF22-4E96-B5C1-610C4F19CC96@sun.com> > From: Andrei Pozolotin > Date: February 17, 2010 3:42:14 PM PST > To: ysr at Sun.COM > Cc: hotspot-gc-use at openjdk.java.net > Subject: Re: Incremental CMS: long "[GC [1 CMS-initial-mark:" > > > Ramki, hello again; > > 1) "unfortunately" - got it; thank you; > > 2) thanks for G1 idea; will try now; > > 3) it does not seem gchisto / visualvm can see G1 at all; > can you please recommend a tool that can help with G1 tuning? Yes, sadly, gchisto has not been updated to parse the G1 style of logging. The work is probably in part to add to/change the G1 logging to be more like the other collectors and then to update gchisto. That's my guess and since its still a guess, we're not close to getting there. > > Cheers, > > Andrei -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From Y.S.Ramakrishna at Sun.COM Thu Feb 18 17:37:56 2010 From: Y.S.Ramakrishna at Sun.COM (Y. Srinivas Ramakrishna) Date: Thu, 18 Feb 2010 09:37:56 -0800 Subject: G1 is bad for me In-Reply-To: <4B7D7159.5000808@gmail.com> References: <4B7D7159.5000808@gmail.com> Message-ID: <4B7D7AF4.8010801@Sun.COM> Hi Andrei -- please use the latest hs17 JVM which you can get from JDK 7. There have been many improvements (and bug fixes) since the 6u18/hs16 that you are using below. Most importantly, please add -XX:-ReduceInitialCardMarks to work around a regression introduced in 6u18/hs16, as described in the 6u18 release notes. As re optimal settings (these may be quite different from shapes that helped with CMS for example), i'll let others w/more (tuning and other) experience w/G1 comment. -- ramki On 02/18/10 08:56, Andrei Pozolotin wrote: > Ramki, Jon, hi; > > I read everyting I could find on G1: > http://www.google.com/patents/about?id=j5SpAAAAEBAJ&dq=7,340,494 > http://developers.sun.com/learning/javaoneonline/2008/pdf/TS-5419.pdf > http://research.sun.com/jtech/pubs/04-g1-paper-ismm.pdf > http://blogs.sun.com/sunabl/entry/the_new_g1_collector > http://blogs.sun.com/jonthecollector/entry/our_collectors > etc, etc, etc, > > got myself excited - looks great on paper! > > and I followed Albert's advice, while trying to mimic my CMS settings: > http://blogs.sun.com/sunabl/entry/the_new_g1_collector > > -Xms3700m > -Xmx3700m > -XX:PermSize=50m > -XX:MaxPermSize=50m > -XX:+UnlockExperimentalVMOptions > -XX:+UseG1GC > -XX:G1YoungGenSize=900m > -XX:SurvivorRatio=2 > -XX:+G1ParallelRSetUpdatingEnabled > -XX:+G1ParallelRSetScanningEnabled > > results: > > 1) pauses are 1000 milliseconds or more; > > 2) reproducible crashes after 3 minutes under load; > > logs attached; > > am I missing something obvious? > like do I need to switch to older / specific JVM build? > > per Tony, this was already working fine for a neighbour of mine in 2008: > http://developers.sun.com/learning/javaoneonline/2008/pdf/TS-5419.pdf > > Thank you; > > Andrei > > > ------------------------------------------------------------------------ > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From Jon.Masamitsu at Sun.COM Fri Feb 19 15:53:45 2010 From: Jon.Masamitsu at Sun.COM (Jon Masamitsu) Date: Fri, 19 Feb 2010 07:53:45 -0800 Subject: G1 is bad for me In-Reply-To: <1b9d6f691002190659i6917bdafo2d61a4f59a833f41@mail.gmail.com> References: <4B7D7159.5000808@gmail.com> <4B7D7AF4.8010801@Sun.COM> <1b9d6f691002190659i6917bdafo2d61a4f59a833f41@mail.gmail.com> Message-ID: <4B7EB409.9080006@sun.com> Chi Ho Kwok wrote On 02/19/10 06:59,: > Hi all, > > A large part of our CPU load is caused by GC, so I'm always interested > in new techniques, but the last time I tried G1 (6u14?), the jvm > rapidly crashed after startup under load. Is the recommendation to > switch to/wait for JDK7 releases for G1, or is G1 on 6u18 (or later?) > any good for production usage? I'd assume there will be more updates > for JDK6 as JDK7 got extended to up to 10 milestones. G1 is in 6u18 but it is still an experimental collector. It's definitely better but if you try it add the flag -XX:-ReduceInitialCardMarks to work around regression that escaped in 6u18. > > CMS with a 32G heap which changes rapidly (~10%/min replaced) is > manageable, but we're "losing" 20% capacity because the threshold has > to be so low, anything higher than 80% causes new gen promotions to > fail during CMS and the app to stop for half a minute. Processing > capacity is also reduced by half during a CMS run because the amount > of threads has to be high (=6 at the moment) to make it hurry up, or > it'll mean either setting the threshold even lower or risk promotion > failures :( If you're running into fragmentation, then either you start the CMS cycles early (as it appears you do) or use a larger heap (if you can). There an RFE in jdk 7 (6631166) that helps with fragmentation in some applications. It's not in a jdk 6 yet. If you're a Java-4-business customer, contact your Sun/Oracle support person and talk to them about getting that RFE. If your application is allocating objects at a high rate and CMS has to do lots of processing to keep up (might be the case if you're using 6 GC threads to do the concurrent marking in parallel), then the overhead may just be what's required to collect all that garbage. Again, if you're a Java-4-business customer, talk to your support person to see if you can get some help in analyzing your situation. > > Chi Ho Kwok > > On Thu, Feb 18, 2010 at 6:37 PM, Y. Srinivas Ramakrishna > > wrote: > > Hi Andrei -- please use the latest hs17 JVM which you can get from > JDK 7. There have been > many improvements (and bug fixes) since the 6u18/hs16 that you are > using below. > Most importantly, please add -XX:-ReduceInitialCardMarks to work > around a regression > introduced in 6u18/hs16, as described in the 6u18 release notes. > As re optimal settings (these may be quite different from shapes > that helped with CMS > for example), i'll let others w/more (tuning and other) experience > w/G1 comment. > > -- ramki > >------------------------------------------------------------------------ > >_______________________________________________ >hotspot-gc-use mailing list >hotspot-gc-use at openjdk.java.net >http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From Y.S.Ramakrishna at Sun.COM Mon Feb 22 15:45:00 2010 From: Y.S.Ramakrishna at Sun.COM (Y. Srinivas Ramakrishna) Date: Mon, 22 Feb 2010 07:45:00 -0800 Subject: G1 is bad for me In-Reply-To: <4B82A2AD.5060805@gmail.com> References: <4B7D7159.5000808@gmail.com> <4B7D7AF4.8010801@Sun.COM> <4B82A2AD.5060805@gmail.com> Message-ID: <4B82A67C.4030902@sun.com> Sorry, we haven't responded in depth because we are busy with other high priority stuff. Please follow up with your support contacts for urgent issues. If you subscribe to hotspot-gc-dev at openjdk.java.net you will get all G1 integration notifications; those then usually flow into the next JDK7 build in the subsequent week or two. regards. -- ramki Andrei Pozolotin wrote: > Tony, Ramki, Jon, hello; > > 1) I take it that you silence means that "we are not there yet" with G1; > understandable - thank you for your efforts! > > 2) there are some 82 people on this mail list who seem to be really > interested > in GC and G1 specifically; can I ask to you to make brief > announcements here > when JDK 7 weekly build brings some new G1 advances, so we can test > it right away? > > Thank you; > > Andrei > > > -------- Original Message -------- > Subject: Re: G1 is bad for me > From: Y. Srinivas Ramakrishna > To: Andrei Pozolotin > Cc: hotspot-gc-use at openjdk.java.net > Date: Thu 18 Feb 2010 11:37:56 AM CST >> Hi Andrei -- please use the latest hs17 JVM which you can get from JDK >> 7. There have been >> many improvements (and bug fixes) since the 6u18/hs16 that you are >> using below. >> Most importantly, please add -XX:-ReduceInitialCardMarks to work >> around a regression >> introduced in 6u18/hs16, as described in the 6u18 release notes. >> As re optimal settings (these may be quite different from shapes that >> helped with CMS >> for example), i'll let others w/more (tuning and other) experience >> w/G1 comment. >> >> -- ramki >> >> On 02/18/10 08:56, Andrei Pozolotin wrote: >>> Ramki, Jon, hi; >>> >>> I read everyting I could find on G1: >>> http://www.google.com/patents/about?id=j5SpAAAAEBAJ&dq=7,340,494 >>> >>> http://developers.sun.com/learning/javaoneonline/2008/pdf/TS-5419.pdf >>> http://research.sun.com/jtech/pubs/04-g1-paper-ismm.pdf >>> http://blogs.sun.com/sunabl/entry/the_new_g1_collector >>> http://blogs.sun.com/jonthecollector/entry/our_collectors >>> etc, etc, etc, >>> >>> got myself excited - looks great on paper! >>> >>> and I followed Albert's advice, while trying to mimic my CMS >>> settings: >>> http://blogs.sun.com/sunabl/entry/the_new_g1_collector >>> >>> -Xms3700m >>> -Xmx3700m >>> -XX:PermSize=50m >>> -XX:MaxPermSize=50m >>> -XX:+UnlockExperimentalVMOptions >>> -XX:+UseG1GC >>> -XX:G1YoungGenSize=900m >>> -XX:SurvivorRatio=2 >>> -XX:+G1ParallelRSetUpdatingEnabled >>> -XX:+G1ParallelRSetScanningEnabled >>> >>> results: >>> >>> 1) pauses are 1000 milliseconds or more; >>> >>> 2) reproducible crashes after 3 minutes under load; >>> >>> logs attached; >>> >>> am I missing something obvious? >>> like do I need to switch to older / specific JVM build? >>> >>> per Tony, this was already working fine for a neighbour of mine >>> in 2008: >>> >>> http://developers.sun.com/learning/javaoneonline/2008/pdf/TS-5419.pdf >>> >>> Thank you; >>> >>> Andrei >>> >>> >>> ------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> hotspot-gc-use mailing list >>> hotspot-gc-use at openjdk.java.net >>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> > > _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From antonios.printezis at sun.com Wed Feb 24 10:05:59 2010 From: antonios.printezis at sun.com (antonios.printezis at sun.com) Date: Wed, 24 Feb 2010 10:05:59 +0000 Subject: hg: jdk7/hotspot-gc/hotspot: 2 new changesets Message-ID: <20100224100606.1B47C4152D@hg.openjdk.java.net> Changeset: b81f3572f355 Author: tonyp Date: 2010-02-23 23:13 -0500 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/b81f3572f355 6928059: G1: command line parameter renaming Summary: Rename G1 parameters to make them more consistent. Reviewed-by: jmasa, johnc ! src/share/vm/gc_implementation/g1/concurrentG1Refine.cpp ! src/share/vm/gc_implementation/g1/concurrentG1Refine.hpp ! src/share/vm/gc_implementation/g1/concurrentG1RefineThread.cpp ! src/share/vm/gc_implementation/g1/concurrentMark.cpp ! src/share/vm/gc_implementation/g1/g1CollectedHeap.cpp ! src/share/vm/gc_implementation/g1/g1CollectorPolicy.cpp ! src/share/vm/gc_implementation/g1/g1MMUTracker.cpp ! src/share/vm/gc_implementation/g1/g1MMUTracker.hpp ! src/share/vm/gc_implementation/g1/g1RemSet.cpp ! src/share/vm/gc_implementation/g1/g1_globals.hpp Changeset: 1c72304f1885 Author: tonyp Date: 2010-02-23 23:14 -0500 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/1c72304f1885 6928073: G1: use existing command line parameters for marking cycle initiation Summary: replace the combination of the G1SteadyStateUsed / G1SteadyStateUsedDelta parameteres to decide the marking initiation threshold and instead use InitiatingHeapOccupancyPercent. Reviewed-by: ysr, johnc ! src/share/vm/gc_implementation/g1/g1CollectorPolicy.cpp ! src/share/vm/gc_implementation/g1/g1_globals.hpp ! src/share/vm/runtime/arguments.cpp ! src/share/vm/runtime/globals.hpp From jon.masamitsu at sun.com Wed Feb 24 20:07:32 2010 From: jon.masamitsu at sun.com (jon.masamitsu at sun.com) Date: Wed, 24 Feb 2010 20:07:32 +0000 Subject: hg: jdk7/hotspot-gc/hotspot: 6928081: G1: rename parameters common with CMS Message-ID: <20100224200738.8BB6A415FF@hg.openjdk.java.net> Changeset: 5f1f51edaff6 Author: jmasa Date: 2010-02-24 07:00 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/5f1f51edaff6 6928081: G1: rename parameters common with CMS Summary: Rename marking stack sizing flags to be common between G1 and CMS Reviewed-by: ysr, tonyp ! src/share/vm/gc_implementation/concurrentMarkSweep/cmsAdaptiveSizePolicy.cpp ! src/share/vm/gc_implementation/concurrentMarkSweep/concurrentMarkSweepGeneration.cpp ! src/share/vm/gc_implementation/g1/concurrentMark.cpp ! src/share/vm/gc_implementation/g1/g1_globals.hpp ! src/share/vm/includeDB_core ! src/share/vm/runtime/arguments.cpp ! src/share/vm/runtime/arguments.hpp ! src/share/vm/runtime/globals.hpp ! src/share/vm/utilities/taskqueue.hpp From antonios.printezis at sun.com Thu Feb 25 02:09:01 2010 From: antonios.printezis at sun.com (antonios.printezis at sun.com) Date: Thu, 25 Feb 2010 02:09:01 +0000 Subject: hg: jdk7/hotspot-gc/hotspot: 6928065: G1: use existing command line parameters to set the young generation size Message-ID: <20100225020904.900FD41677@hg.openjdk.java.net> Changeset: a1c410de27e4 Author: tonyp Date: 2010-02-24 14:56 -0500 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/hotspot/rev/a1c410de27e4 6928065: G1: use existing command line parameters to set the young generation size Summary: see synopsis Reviewed-by: johnc, jmasa ! src/share/vm/gc_implementation/g1/g1CollectorPolicy.cpp ! src/share/vm/gc_implementation/g1/g1_globals.hpp From Paul.Hohensee at Sun.COM Thu Feb 25 21:34:19 2010 From: Paul.Hohensee at Sun.COM (Paul Hohensee) Date: Thu, 25 Feb 2010 16:34:19 -0500 Subject: Pls review 6923123 (XXXS) Message-ID: <4B86ECDB.6090904@sun.com> 6923123: Hotspot refuses to start when -Xmx4m or -Xms4m is specified. Webrev here http://cr.openjdk.java.net/~phh/6923123/webrev.00/ This fixes a problem with the client heap ergo work that went into hs16. See 6887571 for a description of that work. The problem occurred because the default value for NewSize was increased from a minimum of 1m on a per-platform basis to 4m for all platforms. In CollectorPolicy::initialize_size_policy() in collectorPolicy.cpp, a check is made that the minimum heap size be greater than NewSize. With NewSize = 4m and a heap <= 4m, the check fails and the vm refuses to start. Setting NewSize back to 1m fixes the problem. It doesn't change the initial heap size computed by the unified heap ergo calculation in arguments.cpp in either set_heap_size() or set_cms_and_parnew_gc_flags(). Thanks, Paul From john.coomes at sun.com Fri Feb 26 04:34:19 2010 From: john.coomes at sun.com (john.coomes at sun.com) Date: Fri, 26 Feb 2010 04:34:19 +0000 Subject: hg: jdk7/hotspot-gc: Added tag jdk7-b84 for changeset 2f3ea057d1ad Message-ID: <20100226043419.BC26141831@hg.openjdk.java.net> Changeset: cf26288a114b Author: mikejwre Date: 2010-02-18 13:31 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/rev/cf26288a114b Added tag jdk7-b84 for changeset 2f3ea057d1ad ! .hgtags From john.coomes at sun.com Fri Feb 26 04:34:24 2010 From: john.coomes at sun.com (john.coomes at sun.com) Date: Fri, 26 Feb 2010 04:34:24 +0000 Subject: hg: jdk7/hotspot-gc/corba: Added tag jdk7-b84 for changeset 68c8961a82e4 Message-ID: <20100226043428.54C5341832@hg.openjdk.java.net> Changeset: c67a9df7bc0c Author: mikejwre Date: 2010-02-18 13:31 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/corba/rev/c67a9df7bc0c Added tag jdk7-b84 for changeset 68c8961a82e4 ! .hgtags From john.coomes at sun.com Fri Feb 26 04:39:19 2010 From: john.coomes at sun.com (john.coomes at sun.com) Date: Fri, 26 Feb 2010 04:39:19 +0000 Subject: hg: jdk7/hotspot-gc/jaxp: 4 new changesets Message-ID: <20100226043919.A340D41836@hg.openjdk.java.net> Changeset: df2e196a5f01 Author: ohair Date: 2010-02-03 16:44 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jaxp/rev/df2e196a5f01 6923146: Upgrade to JAXP 1.4.3 Reviewed-by: darcy ! jaxp.properties Changeset: bf7c7f2825ef Author: lana Date: 2010-02-08 23:58 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jaxp/rev/bf7c7f2825ef Merge Changeset: 32c0cf01d555 Author: lana Date: 2010-02-14 23:35 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jaxp/rev/32c0cf01d555 Merge Changeset: 6c0ccabb430d Author: mikejwre Date: 2010-02-18 13:31 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jaxp/rev/6c0ccabb430d Added tag jdk7-b84 for changeset 32c0cf01d555 ! .hgtags From john.coomes at sun.com Fri Feb 26 04:39:24 2010 From: john.coomes at sun.com (john.coomes at sun.com) Date: Fri, 26 Feb 2010 04:39:24 +0000 Subject: hg: jdk7/hotspot-gc/jaxws: Added tag jdk7-b84 for changeset 8bc02839eee4 Message-ID: <20100226043925.2930F41837@hg.openjdk.java.net> Changeset: 8424512588ff Author: mikejwre Date: 2010-02-18 13:31 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jaxws/rev/8424512588ff Added tag jdk7-b84 for changeset 8bc02839eee4 ! .hgtags From john.coomes at sun.com Fri Feb 26 04:42:04 2010 From: john.coomes at sun.com (john.coomes at sun.com) Date: Fri, 26 Feb 2010 04:42:04 +0000 Subject: hg: jdk7/hotspot-gc/jdk: 48 new changesets Message-ID: <20100226045725.A326D4183C@hg.openjdk.java.net> Changeset: dca3a251a001 Author: xuelei Date: 2010-01-20 21:38 +0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/dca3a251a001 6862064: incorrect implementation of PKIXParameters.clone() Reviewed-by: weijun, mullan ! src/share/classes/java/security/cert/PKIXParameters.java Changeset: b19cd193245e Author: dcubed Date: 2010-01-20 12:09 -0700 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/b19cd193245e 6580131: 3/4 CompiledMethodLoad events don't produce the expected extra notifications to describe inlining Summary: Add support for additional implementation specific info to the JVM/TI CompiledMethodLoad event via the compile_info parameter. Reviewed-by: never, ohair, tbell, tdeneau Contributed-by: Vasanth Venkatachalam ! make/common/shared/Sanity.gmk ! make/java/jvm/Makefile ! make/mkdemo/jvmti/Makefile ! make/mkdemo/jvmti/README.txt + make/mkdemo/jvmti/compiledMethodLoad/Makefile + src/share/demo/jvmti/compiledMethodLoad/README.txt + src/share/demo/jvmti/compiledMethodLoad/compiledMethodLoad.c + src/share/demo/jvmti/compiledMethodLoad/sample.makefile.txt ! src/share/demo/jvmti/index.html + src/share/javavm/export/jvmticmlr.h + test/demo/jvmti/compiledMethodLoad/CompiledMethodLoadTest.java ! test/demo/jvmti/heapTracker/HeapTrackerTest.java ! test/demo/jvmti/hprof/CpuTimesDefineClassTest.java ! test/demo/jvmti/hprof/CpuTimesTest.java ! test/demo/jvmti/minst/MinstTest.java ! test/demo/jvmti/mtrace/TraceJFrame.java Changeset: 117b245b5bb9 Author: vinnie Date: 2010-01-21 23:59 +0000 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/117b245b5bb9 6763530: Cannot decode PublicKey (Proider SunPKCS11, curve prime256v1) Reviewed-by: andrew ! src/share/classes/sun/security/pkcs11/P11ECKeyFactory.java ! src/share/classes/sun/security/pkcs11/P11Key.java Changeset: c94ac5522d01 Author: vinnie Date: 2010-01-22 00:02 +0000 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/c94ac5522d01 Merge Changeset: e67bf9abc6a5 Author: chegar Date: 2010-01-25 15:41 +0000 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/e67bf9abc6a5 6707289: InterfaceAddress.getNetworkPrefixLength() does not conform to Javadoc Reviewed-by: michaelm ! src/windows/native/java/net/NetworkInterface_winXP.c + test/java/net/InterfaceAddress/NetworkPrefixLength.java Changeset: 558f2a424bfa Author: weijun Date: 2010-01-26 17:03 +0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/558f2a424bfa 6919610: KeyTabInputStream uses static field for per-instance value Reviewed-by: mullan ! src/share/classes/sun/security/krb5/internal/ktab/KeyTabInputStream.java + test/sun/security/krb5/ktab/KeyTabIndex.java Changeset: f544825d0976 Author: jccollet Date: 2010-01-26 11:39 +0100 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/f544825d0976 6919185: test/closed/sun/net/ftp/FtpTests fails to compile Summary: Fixed a couple of regressions in FtpClient and updated the test. Reviewed-by: chegar ! src/share/classes/sun/net/ftp/impl/FtpClient.java Changeset: 8df0ffac7f4d Author: chegar Date: 2010-01-27 16:11 +0000 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/8df0ffac7f4d 6905552: libnet/nio portability issues Reviewed-by: alanb ! src/share/native/java/net/net_util.c ! src/solaris/native/java/net/Inet4AddressImpl.c ! src/solaris/native/java/net/Inet6AddressImpl.c ! src/solaris/native/java/net/NetworkInterface.c ! src/solaris/native/java/net/PlainDatagramSocketImpl.c ! src/solaris/native/java/net/net_util_md.c ! src/solaris/native/java/net/net_util_md.h ! src/solaris/native/sun/net/spi/SdpProvider.c ! src/solaris/native/sun/nio/ch/Net.c ! src/solaris/native/sun/nio/ch/SctpNet.c Changeset: 4192f6edbbf4 Author: ptisnovs Date: 2010-01-27 17:47 +0100 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/4192f6edbbf4 6920143: test/java/awt/TestArea/UsingWithMouse.java needs realSync() Summary: Added small delay to make sure that TextArea animation have finished Reviewed-by: anthony ! test/java/awt/TextArea/UsingWithMouse/SelectionAutoscrollTest.java Changeset: 0126effcc249 Author: lana Date: 2010-01-27 14:46 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/0126effcc249 Merge Changeset: 946b30073247 Author: sherman Date: 2010-01-27 19:39 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/946b30073247 6920732: opensource test/java/nio/charset Summary: move the test cases to openjdk Reviewed-by: martin + test/java/nio/charset/Charset/AvailableCharsetNames.java + test/java/nio/charset/Charset/CharsetContainmentTest.java + test/java/nio/charset/Charset/Contains.java + test/java/nio/charset/Charset/Default.java + test/java/nio/charset/Charset/EmptyCharsetName.java + test/java/nio/charset/Charset/EncDec.java + test/java/nio/charset/Charset/IllegalCharsetName.java + test/java/nio/charset/Charset/NIOCharsetAvailabilityTest.java + test/java/nio/charset/Charset/NullCharsetName.java + test/java/nio/charset/Charset/RegisteredCharsets.java + test/java/nio/charset/Charset/default.sh + test/java/nio/charset/CharsetDecoder/AverageMax.java + test/java/nio/charset/CharsetDecoder/EmptyInput.java + test/java/nio/charset/CharsetEncoder/CanEncode.java + test/java/nio/charset/CharsetEncoder/Flush.java + test/java/nio/charset/RemovingSunIO/SunioAlias.java + test/java/nio/charset/RemovingSunIO/TestCOMP.java + test/java/nio/charset/RemovingSunIO/TestUnmappableForLength.java + test/java/nio/charset/coders/BashCache.java + test/java/nio/charset/coders/BashStreams.java + test/java/nio/charset/coders/Check.java + test/java/nio/charset/coders/CheckSJISMappingProp.sh + test/java/nio/charset/coders/Errors.java + test/java/nio/charset/coders/FullRead.java + test/java/nio/charset/coders/IOCoders.java + test/java/nio/charset/coders/IsLegalReplacement.java + test/java/nio/charset/coders/ResetISO2022JP.java + test/java/nio/charset/coders/SJISPropTest.java + test/java/nio/charset/coders/StreamTimeout.java + test/java/nio/charset/coders/Surrogate.java + test/java/nio/charset/coders/Surrogates.java + test/java/nio/charset/coders/Util.java + test/java/nio/charset/coders/ref.shift_jis + test/java/nio/charset/coders/ref.windows-31j + test/java/nio/charset/spi/FooCharset.java + test/java/nio/charset/spi/FooProvider.java + test/java/nio/charset/spi/Test.java + test/java/nio/charset/spi/basic.sh + test/java/nio/charset/spi/charsetProvider.sp + test/java/nio/charset/spi/default-pol Changeset: 7dadd2951a8b Author: andrew Date: 2010-02-02 10:55 +0000 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/7dadd2951a8b 6921740: Eliminate warnings from sun.io converters and allow compiling with JAVAC_MAX_WARNINGS=true Summary: Fix sun.io converters unchecked and cast warnings produced by -Xlint:all Reviewed-by: alanb, sherman ! make/java/sun_nio/Makefile ! src/share/classes/sun/io/ByteToCharUTF8.java ! src/share/classes/sun/io/CharToByteUnicode.java ! src/share/classes/sun/io/CharacterEncoding.java ! src/share/classes/sun/io/Converters.java ! src/share/classes/sun/nio/cs/AbstractCharsetProvider.java Changeset: e6ab5fabaf7e Author: weijun Date: 2010-02-03 17:04 +0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/e6ab5fabaf7e 6922482: keytool's help on -file always shows 'output file' Reviewed-by: wetmore ! src/share/classes/sun/security/tools/KeyTool.java + test/sun/security/tools/keytool/file-in-help.sh Changeset: a39e899aa5dc Author: sherman Date: 2010-02-05 00:10 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/a39e899aa5dc 6919132: Regex \P{Lu} selects half of a surrogate pari Summary: To use StartS for complement category/block class Reviewed-by: martin, okutsu ! src/share/classes/java/util/regex/Pattern.java ! test/java/util/regex/RegExTest.java Changeset: 7136683a33d2 Author: wetmore Date: 2010-02-05 17:05 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/7136683a33d2 6923976: TestProviderLeak.java is using too small of an initial heap under newer Hotspot (b79+) Reviewed-by: ohair ! test/com/sun/crypto/provider/KeyFactory/TestProviderLeak.java Changeset: 445b9928fb70 Author: sherman Date: 2010-02-06 09:26 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/445b9928fb70 6923692: java/classes_util TEST_BUG:ReadZip.java fails when Summary: to create the test file at test.dir Reviewed-by: alanb ! test/java/util/zip/ZipFile/ReadZip.java Changeset: e79d95ea2e81 Author: lana Date: 2010-02-08 23:59 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/e79d95ea2e81 Merge Changeset: 7e8c77ae401a Author: rkennke Date: 2010-02-02 16:38 +0100 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/7e8c77ae401a 6888734: PIT: regression test fails when java.security.manager is enabled Summary: Load FontManager instance in privileged block to avoid AccessControlException Reviewed-by: igor, tdv ! src/share/classes/sun/font/FontManagerFactory.java + test/java/awt/PrintJob/Security/SecurityDialogTest.java + test/java/awt/PrintJob/Security/policy Changeset: cedd0cdd5b9a Author: rkennke Date: 2010-02-03 10:02 +0100 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/cedd0cdd5b9a 6896335: GraphicsEnvironment.getDefaultScreenDevice() throws UnsatisfiedLinkError in headless mode Summary: Use local ge variable instead of localEnv field in GE. Reviewed-by: igor, prr ! src/share/classes/java/awt/GraphicsEnvironment.java + test/java/awt/GraphicsEnvironment/TestGetDefScreenDevice.java Changeset: 58d014485a29 Author: rkennke Date: 2010-02-07 11:07 +0100 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/58d014485a29 6904882: java.awt.Font.createFont() causes AccessControlException if executed with "-Djava.security.manager" Summary: Perform FontUtilities initialization in privileged block Reviewed-by: igor, prr ! src/share/classes/sun/font/FontUtilities.java + test/java/awt/FontClass/FontPrivilege.java Changeset: 89b0235188b5 Author: lana Date: 2010-02-09 00:00 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/89b0235188b5 Merge Changeset: 31a3f28f3326 Author: dcherepanov Date: 2009-12-23 01:22 +0300 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/31a3f28f3326 6908299: Missed changes for 6664512 during the merge with 6879044 Reviewed-by: mchung, art ! src/share/classes/sun/util/logging/PlatformLogger.java Changeset: 7b65af04d43c Author: dav Date: 2009-12-22 17:28 +0300 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/7b65af04d43c 6893325: JComboBox and dragging to an item outside the bounds of the containing JFrame is not selecting that Reviewed-by: art, dcherepanov ! src/solaris/classes/sun/awt/X11/XWindowPeer.java Changeset: 26280d1705b2 Author: dcherepanov Date: 2009-12-28 20:35 +0300 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/26280d1705b2 6857363: deadlock caused by sun.awt.X11.XTrayIconPeer$Tooltip.display Reviewed-by: ant, art ! src/solaris/classes/sun/awt/X11/InfoWindow.java Changeset: fd5bf5955e37 Author: uta Date: 2009-12-24 17:19 +0300 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/fd5bf5955e37 4874070: invoking DragSource's startDrag with an Image renders no image on drag Reviewed-by: art, denis, alexp ! make/sun/awt/FILES_c_windows.gmk ! make/sun/awt/Makefile ! make/sun/awt/make.depend ! src/share/classes/javax/swing/TransferHandler.java ! src/share/classes/sun/awt/dnd/SunDragSourceContextPeer.java ! src/windows/classes/sun/awt/windows/WDataTransferer.java ! src/windows/classes/sun/awt/windows/WDragSourceContextPeer.java ! src/windows/classes/sun/awt/windows/WToolkit.java ! src/windows/lib/flavormap.properties ! src/windows/native/sun/windows/awt.h + src/windows/native/sun/windows/awt_DCHolder.cpp + src/windows/native/sun/windows/awt_DCHolder.h ! src/windows/native/sun/windows/awt_DnDDS.cpp ! src/windows/native/sun/windows/awt_DnDDS.h ! src/windows/native/sun/windows/awt_DnDDT.cpp ! src/windows/native/sun/windows/awt_DnDDT.h ! src/windows/native/sun/windows/awt_Toolkit.cpp ! src/windows/native/sun/windows/awt_Toolkit.h + src/windows/native/sun/windows/awt_ole.cpp + src/windows/native/sun/windows/awt_ole.h + test/java/awt/dnd/DnDFileGroupDescriptor/DnDFileGroupDescriptor.html + test/java/awt/dnd/DnDFileGroupDescriptor/DnDFileGroupDescriptor.java + test/java/awt/dnd/DnDFileGroupDescriptor/DnDTarget.java + test/java/awt/dnd/ImageDecoratedDnD/DnDSource.java + test/java/awt/dnd/ImageDecoratedDnD/DnDTarget.java + test/java/awt/dnd/ImageDecoratedDnD/ImageDecoratedDnD.html + test/java/awt/dnd/ImageDecoratedDnD/ImageDecoratedDnD.java + test/java/awt/dnd/ImageDecoratedDnD/ImageGenerator.java + test/java/awt/dnd/ImageDecoratedDnD/MyCursor.java + test/java/awt/dnd/ImageDecoratedDnDInOut/DnDSource.java + test/java/awt/dnd/ImageDecoratedDnDInOut/DnDTarget.java + test/java/awt/dnd/ImageDecoratedDnDInOut/ImageDecoratedDnDInOut.html + test/java/awt/dnd/ImageDecoratedDnDInOut/ImageDecoratedDnDInOut.java + test/java/awt/dnd/ImageDecoratedDnDInOut/ImageGenerator.java + test/java/awt/dnd/ImageDecoratedDnDInOut/MyCursor.java + test/java/awt/dnd/ImageDecoratedDnDNegative/DnDSource.java + test/java/awt/dnd/ImageDecoratedDnDNegative/DnDTarget.java + test/java/awt/dnd/ImageDecoratedDnDNegative/ImageDecoratedDnDNegative.html + test/java/awt/dnd/ImageDecoratedDnDNegative/ImageDecoratedDnDNegative.java + test/java/awt/dnd/ImageDecoratedDnDNegative/ImageGenerator.java + test/java/awt/dnd/ImageDecoratedDnDNegative/MyCursor.java Changeset: 4799006d0171 Author: uta Date: 2010-01-13 17:10 +0300 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/4799006d0171 Merge Changeset: f154d4943a1a Author: uta Date: 2010-01-14 17:56 +0300 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/f154d4943a1a 6916867: Fastdebug build failed after CR 4874070 fix putback. Reviewed-by: art, dcherepanov ! src/windows/native/sun/windows/awt_DnDDS.cpp Changeset: 3cc5eff94552 Author: dcherepanov Date: 2010-01-20 01:33 +0300 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/3cc5eff94552 6660258: Java application stops Windows logout/shutdown (regression in 1.5.0_14) Reviewed-by: anthony, art, uta ! src/windows/native/sun/windows/awt_Component.cpp ! src/windows/native/sun/windows/awtmsg.h Changeset: 0f92194cd798 Author: dcherepanov Date: 2010-01-22 19:47 +0300 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/0f92194cd798 6756774: fstdebug jvm fails with assertion (result != deleted_handle(),"Used a deleted global handle.") Reviewed-by: art, anthony ! src/windows/native/sun/windows/awt_MenuItem.cpp Changeset: d7c4baff3f96 Author: lana Date: 2010-01-28 18:24 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/d7c4baff3f96 Merge - make/java/redist/FILES.gmk ! make/sun/awt/Makefile - make/sun/nio/FILES_java.gmk - src/share/classes/sun/dyn/util/BytecodeSignature.java - src/solaris/classes/sun/nio/ch/SctpSocketDispatcher.java Changeset: 66c193082586 Author: yan Date: 2010-02-08 17:02 +0300 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/66c193082586 6882912: Strange behaviours when typing @ or ' Summary: Eliminate stray lines without a proper unicode->regularKeyID pair (e.g. as there is no "arrow left" unicode, should be no pair) Reviewed-by: rupashka ! src/share/classes/sun/awt/ExtendedKeyCodes.java Changeset: b51982678191 Author: lana Date: 2010-02-09 00:02 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/b51982678191 Merge Changeset: de7807599a9b Author: peytoia Date: 2009-12-15 14:50 +0900 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/de7807599a9b 5047314: [Col] Collator.compare() runs indefinitely for a certain set of Thai strings Reviewed-by: okutsu ! src/share/classes/java/text/CollationElementIterator.java + test/java/text/Collator/Bug5047314.java Changeset: 3efbbc00ac5f Author: lana Date: 2009-12-16 16:25 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/3efbbc00ac5f Merge - make/tools/CharsetMapping/DoubleByte-X.java - make/tools/CharsetMapping/SingleByte-X.java - src/share/classes/sun/awt/ComponentAccessor.java - src/share/classes/sun/awt/WindowAccessor.java - src/share/classes/sun/security/provider/IdentityDatabase.java - src/share/classes/sun/security/provider/SystemIdentity.java - src/share/classes/sun/security/provider/SystemSigner.java - src/share/classes/sun/security/x509/X500Signer.java - src/share/classes/sun/security/x509/X509Cert.java - src/share/classes/sun/tools/jar/JarVerifierStream.java - src/share/classes/sun/util/CoreResourceBundleControl-XLocales.java - src/share/classes/sun/util/LocaleDataMetaInfo-XLocales.java - test/java/util/Formatter/Basic-X.java - test/sun/tools/native2ascii/test2 - test/tools/launcher/SolarisDataModel.sh - test/tools/launcher/SolarisRunpath.sh - test/tools/launcher/libraryCaller.c - test/tools/launcher/libraryCaller.h - test/tools/launcher/libraryCaller.java Changeset: e2f7e92c30f1 Author: peterz Date: 2009-12-21 19:26 +0300 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/e2f7e92c30f1 6860433: [Nimbus] Code to set a single slider's thumb background doesn't work as specified Reviewed-by: rupashka ! src/share/classes/javax/swing/plaf/nimbus/Defaults.template ! src/share/classes/javax/swing/plaf/nimbus/NimbusLookAndFeel.java + test/javax/swing/plaf/nimbus/ColorCustomizationTest.java Changeset: fffd21bc5657 Author: peterz Date: 2009-12-25 17:47 +0300 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/fffd21bc5657 6860438: [Nimbus] Code to globally set slider's thumb background doesn't work as specified Reviewed-by: rupashka ! src/share/classes/javax/swing/MultiUIDefaults.java + test/javax/swing/MultiUIDefaults/4300666/bug4300666.html + test/javax/swing/MultiUIDefaults/4300666/bug4300666.java + test/javax/swing/MultiUIDefaults/4331767/bug4331767.java + test/javax/swing/MultiUIDefaults/Test6860438.java Changeset: e9ccd1dd6923 Author: andrew Date: 2010-01-08 12:51 +0000 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/e9ccd1dd6923 6584033: (tz) wrong buffer length in TimeZone_md.c Summary: Add testcase for this bug Reviewed-by: darcy, okutsu + test/java/util/TimeZone/TimeZoneDatePermissionCheck.java + test/java/util/TimeZone/TimeZoneDatePermissionCheck.sh Changeset: b129d0f7be40 Author: peytoia Date: 2010-01-13 15:40 +0900 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/b129d0f7be40 6868503: RuleBasedBreakIterator is inefficient Reviewed-by: okutsu ! src/share/classes/java/text/RuleBasedBreakIterator.java Changeset: 9c5a24050392 Author: malenkov Date: 2010-01-21 21:45 +0300 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/9c5a24050392 4922835: DOC: Statement javadoc should indicate that target and methodName cannot be null Reviewed-by: peterz ! src/share/classes/java/beans/Expression.java ! src/share/classes/java/beans/Statement.java Changeset: eba0ff97a252 Author: malenkov Date: 2010-01-21 21:53 +0300 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/eba0ff97a252 4968536: DOC: Javadoc for java.beans.Encoder.getPersistenceDelegate is incomplete Reviewed-by: peterz ! src/share/classes/java/beans/Encoder.java Changeset: 3c61edecf44f Author: okutsu Date: 2010-01-26 15:42 +0900 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/3c61edecf44f 6912866: (date) java.util.Date.before / after may be expensive Reviewed-by: peytoia ! src/share/classes/java/util/Date.java Changeset: e7127f3fa2f4 Author: peterz Date: 2010-01-28 17:06 +0300 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/e7127f3fa2f4 6912118: Incosistency in several SynthUI classes between inherited specs ofupdate() and paint() methods Reviewed-by: rupashka ! src/share/classes/javax/swing/plaf/synth/SynthButtonUI.java ! src/share/classes/javax/swing/plaf/synth/SynthColorChooserUI.java ! src/share/classes/javax/swing/plaf/synth/SynthComboBoxUI.java ! src/share/classes/javax/swing/plaf/synth/SynthDesktopIconUI.java ! src/share/classes/javax/swing/plaf/synth/SynthDesktopPaneUI.java ! src/share/classes/javax/swing/plaf/synth/SynthEditorPaneUI.java ! src/share/classes/javax/swing/plaf/synth/SynthInternalFrameUI.java ! src/share/classes/javax/swing/plaf/synth/SynthLabelUI.java ! src/share/classes/javax/swing/plaf/synth/SynthListUI.java ! src/share/classes/javax/swing/plaf/synth/SynthMenuBarUI.java ! src/share/classes/javax/swing/plaf/synth/SynthMenuItemUI.java ! src/share/classes/javax/swing/plaf/synth/SynthMenuUI.java ! src/share/classes/javax/swing/plaf/synth/SynthOptionPaneUI.java ! src/share/classes/javax/swing/plaf/synth/SynthPanelUI.java ! src/share/classes/javax/swing/plaf/synth/SynthPopupMenuUI.java ! src/share/classes/javax/swing/plaf/synth/SynthProgressBarUI.java ! src/share/classes/javax/swing/plaf/synth/SynthRootPaneUI.java ! src/share/classes/javax/swing/plaf/synth/SynthScrollBarUI.java ! src/share/classes/javax/swing/plaf/synth/SynthScrollPaneUI.java ! src/share/classes/javax/swing/plaf/synth/SynthSeparatorUI.java ! src/share/classes/javax/swing/plaf/synth/SynthSliderUI.java ! src/share/classes/javax/swing/plaf/synth/SynthSpinnerUI.java ! src/share/classes/javax/swing/plaf/synth/SynthSplitPaneUI.java ! src/share/classes/javax/swing/plaf/synth/SynthTabbedPaneUI.java ! src/share/classes/javax/swing/plaf/synth/SynthTableHeaderUI.java ! src/share/classes/javax/swing/plaf/synth/SynthTableUI.java ! src/share/classes/javax/swing/plaf/synth/SynthTextAreaUI.java ! src/share/classes/javax/swing/plaf/synth/SynthTextFieldUI.java ! src/share/classes/javax/swing/plaf/synth/SynthToolBarUI.java ! src/share/classes/javax/swing/plaf/synth/SynthToolTipUI.java ! src/share/classes/javax/swing/plaf/synth/SynthTreeUI.java ! src/share/classes/javax/swing/plaf/synth/SynthViewportUI.java Changeset: 4eb3a8c95c24 Author: malenkov Date: 2010-01-28 20:49 +0300 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/4eb3a8c95c24 6412286: DOC: LTP: Unspecified NPE in java.beans.DefaultPersistenceDelegate.instantiate method Reviewed-by: peterz ! src/share/classes/java/beans/DefaultPersistenceDelegate.java ! src/share/classes/java/beans/PersistenceDelegate.java Changeset: 3283bb8c1bcb Author: lana Date: 2010-01-28 16:12 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/3283bb8c1bcb Merge - make/java/redist/FILES.gmk - make/sun/nio/FILES_java.gmk - src/share/classes/sun/dyn/util/BytecodeSignature.java - src/solaris/classes/sun/nio/ch/SctpSocketDispatcher.java Changeset: 3913691b3021 Author: malenkov Date: 2010-02-05 10:36 +0300 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/3913691b3021 6921057: REGRESSION: persistence delegate issue on Windows and Linux against 5.u23b03/6u17b11 Reviewed-by: peterz, art ! src/share/classes/java/beans/Introspector.java ! test/java/beans/Introspector/6380849/TestBeanInfo.java ! test/java/beans/Introspector/Test5102804.java ! test/java/beans/XMLEncoder/Test4646747.java Changeset: 5c0c2882eed2 Author: lana Date: 2010-02-09 00:05 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/5c0c2882eed2 Merge Changeset: 7cb9388bb1a1 Author: lana Date: 2010-02-14 23:38 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/7cb9388bb1a1 Merge Changeset: a9b4fde406d4 Author: mikejwre Date: 2010-02-18 13:31 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/jdk/rev/a9b4fde406d4 Added tag jdk7-b84 for changeset 7cb9388bb1a1 ! .hgtags From john.coomes at sun.com Fri Feb 26 05:09:07 2010 From: john.coomes at sun.com (john.coomes at sun.com) Date: Fri, 26 Feb 2010 05:09:07 +0000 Subject: hg: jdk7/hotspot-gc/langtools: 16 new changesets Message-ID: <20100226050954.604DC41841@hg.openjdk.java.net> Changeset: f23b985beb78 Author: jjg Date: 2010-01-19 14:28 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/langtools/rev/f23b985beb78 6917067: refactor type annotations code from TransTypes into new TypeAnnotations class Reviewed-by: jjg, darcy Contributed-by: mali at csail.mit.edu, mernst at cs.washington.edu + src/share/classes/com/sun/tools/javac/code/TypeAnnotations.java ! src/share/classes/com/sun/tools/javac/comp/TransTypes.java Changeset: 0eaf89e08564 Author: jjg Date: 2010-01-20 16:12 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/langtools/rev/0eaf89e08564 6918127: improve handling of TypeAnnotationPosition fields Reviewed-by: jjg, darcy Contributed-by: mali at csail.mit.edu, mernst at cs.washington.edu ! src/share/classes/com/sun/tools/classfile/ExtendedAnnotation.java ! src/share/classes/com/sun/tools/javac/code/TypeAnnotationPosition.java ! src/share/classes/com/sun/tools/javac/jvm/Code.java ! src/share/classes/com/sun/tools/javac/jvm/Gen.java Changeset: da0e3e2dd3ef Author: jjg Date: 2010-01-26 11:15 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/langtools/rev/da0e3e2dd3ef 6919944: incorrect position given for duplicate annotation value error Reviewed-by: darcy ! src/share/classes/com/sun/tools/javac/comp/Check.java ! test/tools/javac/typeAnnotations/failures/common/arrayclass/DuplicateAnnotationValue.java ! test/tools/javac/typeAnnotations/failures/common/arrayclass/DuplicateAnnotationValue.out ! test/tools/javac/typeAnnotations/failures/common/arrays/DuplicateAnnotationValue.java ! test/tools/javac/typeAnnotations/failures/common/arrays/DuplicateAnnotationValue.out ! test/tools/javac/typeAnnotations/failures/common/innertypeparams/DuplicateAnnotationValue.java ! test/tools/javac/typeAnnotations/failures/common/innertypeparams/DuplicateAnnotationValue.out ! test/tools/javac/typeAnnotations/failures/common/newarray/DuplicateAnnotationValue.java ! test/tools/javac/typeAnnotations/failures/common/newarray/DuplicateAnnotationValue.out ! test/tools/javac/typeAnnotations/failures/common/parambounds/DuplicateAnnotationValue.java ! test/tools/javac/typeAnnotations/failures/common/parambounds/DuplicateAnnotationValue.out ! test/tools/javac/typeAnnotations/failures/common/receiver/DuplicateAnnotationValue.java ! test/tools/javac/typeAnnotations/failures/common/receiver/DuplicateAnnotationValue.out ! test/tools/javac/typeAnnotations/failures/common/typeArgs/DuplicateAnnotationValue.java ! test/tools/javac/typeAnnotations/failures/common/typeArgs/DuplicateAnnotationValue.out ! test/tools/javac/typeAnnotations/failures/common/typeparams/DuplicateAnnotationValue.java ! test/tools/javac/typeAnnotations/failures/common/typeparams/DuplicateAnnotationValue.out ! test/tools/javac/typeAnnotations/failures/common/wildcards/DuplicateAnnotationValue.java ! test/tools/javac/typeAnnotations/failures/common/wildcards/DuplicateAnnotationValue.out Changeset: 59167312ed4e Author: jjg Date: 2010-01-26 11:23 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/langtools/rev/59167312ed4e 6917130: should test that annotations that have been optimized away are not emitted to classfile Reviewed-by: jjg, darcy Contributed-by: mali at csail.mit.edu, mernst at cs.washington.edu + test/tools/javac/typeAnnotations/classfile/DeadCode.java Changeset: ff7a01f9eff3 Author: lana Date: 2010-01-27 14:46 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/langtools/rev/ff7a01f9eff3 Merge Changeset: 699ecefbdd4e Author: jjg Date: 2010-01-29 16:06 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/langtools/rev/699ecefbdd4e 6919889: assorted position errors in compiler syntax trees Reviewed-by: darcy ! src/share/classes/com/sun/tools/javac/code/Flags.java ! src/share/classes/com/sun/tools/javac/parser/JavacParser.java ! src/share/classes/com/sun/tools/javac/tree/TreeInfo.java ! src/share/classes/com/sun/tools/javac/tree/TreeMaker.java + test/tools/javac/T6654037.java ! test/tools/javac/generics/diamond/neg/Neg01.out ! test/tools/javac/generics/diamond/neg/Neg02.out ! test/tools/javac/generics/diamond/neg/Neg03.out ! test/tools/javac/generics/diamond/neg/Neg04.out + test/tools/javac/treepostests/TreePosTest.java Changeset: 8e638442522a Author: jjg Date: 2010-01-29 16:54 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/langtools/rev/8e638442522a 6499119: Created package-info class file modeled improperly 6920317: package-info.java file has to be specified on the javac cmdline, else it will not be avail. Reviewed-by: darcy ! src/share/classes/com/sun/tools/javac/code/Symbol.java ! src/share/classes/com/sun/tools/javac/comp/Enter.java ! src/share/classes/com/sun/tools/javac/comp/Lower.java ! src/share/classes/com/sun/tools/javac/processing/JavacProcessingEnvironment.java + test/tools/javac/processing/6499119/ClassProcessor.java + test/tools/javac/processing/6499119/package-info.java + test/tools/javac/processing/T6920317.java Changeset: 732510cc3538 Author: jjg Date: 2010-02-01 17:05 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/langtools/rev/732510cc3538 6919986: [308] change size of type_index (of CLASS_EXTENDS and THROWS) from byte to short Reviewed-by: darcy, jjg Contributed-by: mali at csail.mit.edu, mernst at cs.washington.edu ! src/share/classes/com/sun/tools/classfile/ExtendedAnnotation.java ! src/share/classes/com/sun/tools/javac/code/TypeAnnotations.java ! src/share/classes/com/sun/tools/javac/jvm/ClassReader.java ! src/share/classes/com/sun/tools/javac/jvm/ClassWriter.java Changeset: b0a68258360a Author: jjg Date: 2010-02-02 10:56 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/langtools/rev/b0a68258360a 6918625: handle annotations on array class literals Reviewed-by: jjg, darcy Contributed-by: mali at csail.mit.edu, mernst at cs.washington.edu ! src/share/classes/com/sun/tools/classfile/ClassWriter.java ! src/share/classes/com/sun/tools/javac/code/TypeAnnotationPosition.java ! src/share/classes/com/sun/tools/javap/AnnotationWriter.java + test/tools/javap/typeAnnotations/ArrayClassLiterals2.java Changeset: 41ed86f86585 Author: jjg Date: 2010-02-03 11:28 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/langtools/rev/41ed86f86585 6922429: extend tree position test waiver Reviewed-by: darcy ! test/tools/javac/treepostests/TreePosTest.java Changeset: f65d652cb6af Author: jjg Date: 2010-02-03 11:33 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/langtools/rev/f65d652cb6af 6922300: [308] populate the reference_info for type annotations targeting primitive class literals Reviewed-by: darcy, jjg Contributed-by: mali at csail.mit.edu, mernst at cs.washington.edu ! src/share/classes/com/sun/tools/javac/jvm/Gen.java Changeset: 4c844e609d81 Author: jjg Date: 2010-02-03 16:58 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/langtools/rev/4c844e609d81 6921979: add test program to verify annotations are attached to nodes as expected Reviewed-by: darcy + test/tools/javac/treeannotests/AnnoTreeTests.java + test/tools/javac/treeannotests/DA.java + test/tools/javac/treeannotests/TA.java + test/tools/javac/treeannotests/Test.java + test/tools/javac/treeannotests/TestProcessor.java Changeset: 4b4e282a3146 Author: jjg Date: 2010-02-04 10:14 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/langtools/rev/4b4e282a3146 6923080: TreeScanner.visitNewClass should scan tree.typeargs Reviewed-by: darcy ! src/share/classes/com/sun/tools/javac/code/TypeAnnotations.java ! src/share/classes/com/sun/tools/javac/comp/MemberEnter.java ! src/share/classes/com/sun/tools/javac/processing/JavacProcessingEnvironment.java ! src/share/classes/com/sun/tools/javac/tree/TreeScanner.java + test/tools/javac/tree/T6923080.java + test/tools/javac/tree/TreeScannerTest.java Changeset: 56a46d079264 Author: lana Date: 2010-02-08 23:59 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/langtools/rev/56a46d079264 Merge Changeset: d9cd5b8286e4 Author: lana Date: 2010-02-14 23:39 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/langtools/rev/d9cd5b8286e4 Merge Changeset: 75d5bd12eb86 Author: mikejwre Date: 2010-02-18 13:31 -0800 URL: http://hg.openjdk.java.net/jdk7/hotspot-gc/langtools/rev/75d5bd12eb86 Added tag jdk7-b84 for changeset d9cd5b8286e4 ! .hgtags From peter.schuller at infidyne.com Sat Feb 27 18:48:10 2010 From: peter.schuller at infidyne.com (Peter Schuller) Date: Sat, 27 Feb 2010 19:48:10 +0100 Subject: g1: remembered set scan costs very high w/ LRU cache type of behavior Message-ID: <5a1151761002271048n798d8e54of9981e88e2b447a8@mail.gmail.com> Hello, I have a very simple test case specifically designed to stress GC:s (while being reasonably representative of some worst-case but plausible production use cases): http://github.com/scode/plrutest/blob/master/src/org/scode/lrutest/LruTest.java It implements a very simple LRU cached using a LinkedHashMap. It inserts entries in this hash map at a pre-defined target rate (100 000 per second). Each insert is key:ed on a random integer between 0 and CACHE_SIZE * 2; thus when the cache reaches steady state 50% of iterations are expected to generate old garbage. It is completely non-generational, and the intent was to see how well the concurrent marking and partially young evacuations handle large heaps. I have run it with various parameters, but the ones relevant to this post are: -verbose:gc -Xms20g \ -Xmx20g \ -XX:+PrintGCTimeStamps \ -XX:+PrintGCDetails \ -XX:+UnlockExperimentalVMOptions \ -XX:+UseG1GC \ -XX:MaxGCPauseMillis=10 \ -XX:GCPauseIntervalMillis=20 \ -XX:G1ConfidencePercent=100 \ -XX:+G1ParallelRSetUpdatingEnabled \ -XX:+G1ParallelRSetScanningEnabled \ -XX:+DisableExplicitGC \ It's being run on a 16 core 64 bit Linux machine with 36 GB of RAM. (I know about there being a bug that affects parallel rset scanning/updating; for the purpose of this test I don't care about that. The observed behavior of very high RS scan costs happens with non-parallel RS scanning as well, only the numbers of higher since it will be running on one core instead of 16.) However, I run into trouble with young evacuations before even reaching the point of generating actual garbage (i.e., before the cache is full). Up to roughly 1 gb of heap used (as reported by GC output), pauses are fairly reasonable, with about 10 ms being spent in the RS scan phase (with parallel rs scanning enabled). At roughly 1 gb heap size the RS scan cost as reported by the GC output, suddenly starts steadly increasing until it reaches at least hundreds of milliseconds with no sign of stopping. Here is some sample output at ~ 1.7 gb heap size: 103.128: [GC pause (young)^C, 0.19716900 secs] [Parallel Time: 193.8 ms] [Update RS (Start) (ms): 103128.7 103128.7 103128.7 103128.7 103128.8 103128.8 103128.8 103128.9 103128.9 103128.9 103129.0 103129.0 103129.1] [Update RS (ms): 1.1 1.2 1.0 0.8 1.2 1.2 1.1 1.0 1.1 1.1 0.9 0.1 0.0 Avg: 0.9, Min: 0.0, Max: 1.2] [Processed Buffers : 1 1 1 1 1 1 1 1 1 1 1 1 0 Sum: 12, Avg: 0, Min: 0, Max: 1] [Ext Root Scanning (ms): 0.2 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Avg: 0.0, Min: 0.0, Max: 0.2] [Mark Stack Scanning (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Avg: 0.0, Min: 0.0, Max: 0.0] [Scan-Only Scanning (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Avg: 0.0, Min: 0.0, Max: 0.0] [Scan-Only Regions : 0 0 0 0 0 0 0 0 0 0 0 0 0 Sum: 0, Avg: 0, Min: 0, Max: 0] [Scan RS (ms): 178.8 178.8 179.1 179.4 178.7 179.2 178.9 178.8 179.2 179.2 178.3 179.9 179.6 Avg: 179.1, Min: 178.3, Max: 179.9] [Object Copy (ms): 10.6 10.5 10.3 10.2 10.6 10.0 10.3 10.4 9.9 9.9 10.9 10.2 10.5 Avg: 10.3, Min: 9.9, Max: 10.9] [Termination (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Avg: 0.0, Min: 0.0, Max: 0.0] [Other: 3.4 ms] [Clear CT: 1.7 ms] [Other: 1.7 ms] [ 1722M->1720M(20480M)] [Times: user=2.44 sys=0.06, real=0.19 secs] Except the RS scan bit, it looks reasonable. Object copy seems to be landing very nicely at around my asked-for 10 ms. RS updating is limited as would be expected given that most writes to the cache would likely happen to old objects. Root/stack scanning is insignificant since it is a small program with a single thread. But RS scan is apparantly very expensive. While I can see how RS scanning would be a significant factor given that all objects created in the young generation (barring any internal to the LinkedHashMap) will be referenced by some pseudo-random object in non-young regions (internal LinkedHashMap structure), I would expect the cost of RS scanning per young evacuation to scale with the size of the young generation. Beyond the edge case of a very very small hash map, I expect the distribution of references into the young generation to be fairly widely spread across the internal structure of the hash map. I can see some possibilities: (1) Maybe the LInkedHashMap happens to be structured such that there is significant locality to the object referenced put into it, such that up to the 1 gb heap size the number of distinct dirty cards is fairly low. The increase in RS scan time I am seeing may simply be the cost converging on an eventual very high cost resulting from poor locality of writes to old regions. In this case, it may be that the remember set simply do not scale well enough period, with many well-spread writes to older generations pointing to young data. (2) Maybe I am right in my thinking that writes to older regions would already be well spread long before the 1gb heapsize, and there is something wrong with the RS scanning itself that causes costs to go up much more than they "should" given the work load. Is there a better explanation? In the case of (1) it seems to me that what I really want is to run G1 in non-generational mode since I would then expect it to evacuate useful regions that does in fact free data (thanks to the statistics gathered by the concurrent marking). In this case the cost of RS scanning, even if it may be high, would at least be a trade-off to actual progress in terms of freeing heap. RIght now, all this RS scan work is being spent on not making any progress. (What are the reasons for non-generational mode no longer being supported? Enabling it asserts later on so it is clearly not by accident. Is it because of the optimization to not to remember set writes for writes in the young generation, that it is not just a switch to turn it off? Are there other problems with running it non-generationally?) In the case of (2) perhaps this indicates a fixable problem with the GC. As an aside, the GC behavior during young generation eviction is, as expected, a lot better when I run it against an immutable LRU cache: http://github.com/scode/plrutest/blob/master/src/org/scode/plrutest/clojure/lrucache.clj (That is just the cache, not a runnable test.) -- / Peter Schuller _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From Jon.Masamitsu at Sun.COM Sat Feb 27 19:13:23 2010 From: Jon.Masamitsu at Sun.COM (Jon Masamitsu) Date: Sat, 27 Feb 2010 11:13:23 -0800 Subject: g1: remembered set scan costs very high w/ LRU cache type of behavior In-Reply-To: <5a1151761002271048n798d8e54of9981e88e2b447a8@mail.gmail.com> References: <5a1151761002271048n798d8e54of9981e88e2b447a8@mail.gmail.com> Message-ID: <4B896ED3.3040107@sun.com> Peter, Which version of the jdk are you using? A jdk 6.0 update? Or jdk 7.0 build? Thanks. Jon Peter Schuller wrote On 02/27/10 10:48,: >Hello, > >I have a very simple test case specifically designed to stress GC:s >(while being reasonably representative of some worst-case but >plausible production use cases): > > http://github.com/scode/plrutest/blob/master/src/org/scode/lrutest/LruTest.java > >It implements a very simple LRU cached using a LinkedHashMap. It >inserts entries in this hash map at a pre-defined target rate (100 000 >per second). Each insert is key:ed on a random integer between 0 and >CACHE_SIZE * 2; thus when the cache reaches steady state 50% of >iterations are expected to generate old garbage. > >It is completely non-generational, and the intent was to see how well >the concurrent marking and partially young evacuations handle large >heaps. > >I have run it with various parameters, but the ones relevant to this post are: > > -verbose:gc > -Xms20g \ > -Xmx20g \ > -XX:+PrintGCTimeStamps \ > -XX:+PrintGCDetails \ > -XX:+UnlockExperimentalVMOptions \ > -XX:+UseG1GC \ > -XX:MaxGCPauseMillis=10 \ > -XX:GCPauseIntervalMillis=20 \ > -XX:G1ConfidencePercent=100 \ > -XX:+G1ParallelRSetUpdatingEnabled \ > -XX:+G1ParallelRSetScanningEnabled \ > -XX:+DisableExplicitGC \ > >It's being run on a 16 core 64 bit Linux machine with 36 GB of RAM. > >(I know about there being a bug that affects parallel rset >scanning/updating; for the purpose of this test I don't care about >that. The observed behavior of very high RS scan costs happens with >non-parallel RS scanning as well, only the numbers of higher since it >will be running on one core instead of 16.) > >However, I run into trouble with young evacuations before even >reaching the point of generating actual garbage (i.e., before the >cache is full). Up to roughly 1 gb of heap used (as reported by GC >output), pauses are fairly reasonable, with about 10 ms being spent in >the RS scan phase (with parallel rs scanning enabled). At roughly 1 gb >heap size the RS scan cost as reported by the GC output, suddenly >starts steadly increasing until it reaches at least hundreds of >milliseconds with no sign of stopping. Here is some sample output at ~ >1.7 gb heap size: > >103.128: [GC pause (young)^C, 0.19716900 secs] > [Parallel Time: 193.8 ms] > [Update RS (Start) (ms): 103128.7 103128.7 103128.7 103128.7 > 103128.8 103128.8 103128.8 103128.9 103128.9 103128.9 103129.0 >103129.0 103129.1] > [Update RS (ms): 1.1 1.2 1.0 0.8 1.2 1.2 1.1 1.0 1.1 >1.1 0.9 0.1 0.0 > Avg: 0.9, Min: 0.0, Max: 1.2] > [Processed Buffers : 1 1 1 1 1 1 1 1 1 1 1 1 0 > Sum: 12, Avg: 0, Min: 0, Max: 1] > [Ext Root Scanning (ms): 0.2 0.1 0.0 0.0 0.0 0.0 0.0 0.0 > 0.0 0.0 0.0 0.0 0.0 > Avg: 0.0, Min: 0.0, Max: 0.2] > [Mark Stack Scanning (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 >0.0 0.0 0.0 0.0 0.0 0.0 > Avg: 0.0, Min: 0.0, Max: 0.0] > [Scan-Only Scanning (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 >0.0 0.0 0.0 0.0 0.0 0.0 > Avg: 0.0, Min: 0.0, Max: 0.0] > [Scan-Only Regions : 0 0 0 0 0 0 0 0 0 0 0 0 0 > Sum: 0, Avg: 0, Min: 0, Max: 0] > [Scan RS (ms): 178.8 178.8 179.1 179.4 178.7 179.2 178.9 >178.8 179.2 179.2 178.3 179.9 179.6 > Avg: 179.1, Min: 178.3, Max: 179.9] > [Object Copy (ms): 10.6 10.5 10.3 10.2 10.6 10.0 10.3 >10.4 9.9 9.9 10.9 10.2 10.5 > Avg: 10.3, Min: 9.9, Max: 10.9] > [Termination (ms): 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 >0.0 0.0 0.0 0.0 > Avg: 0.0, Min: 0.0, Max: 0.0] > [Other: 3.4 ms] > [Clear CT: 1.7 ms] > [Other: 1.7 ms] > [ 1722M->1720M(20480M)] > [Times: user=2.44 sys=0.06, real=0.19 secs] > >Except the RS scan bit, it looks reasonable. Object copy seems to be >landing very nicely at around my asked-for 10 ms. RS updating is >limited as would be expected given that most writes to the cache would >likely happen to old objects. Root/stack scanning is insignificant >since it is a small program with a single thread. > >But RS scan is apparantly very expensive. While I can see how RS >scanning would be a significant factor given that all objects created >in the young generation (barring any internal to the LinkedHashMap) >will be referenced by some pseudo-random object in non-young regions >(internal LinkedHashMap structure), I would expect the cost of RS >scanning per young evacuation to scale with the size of the young >generation. Beyond the edge case of a very very small hash map, I >expect the distribution of references into the young generation to be >fairly widely spread across the internal structure of the hash map. > >I can see some possibilities: > >(1) Maybe the LInkedHashMap happens to be structured such that there >is significant locality to the object referenced put into it, such >that up to the 1 gb heap size the number of distinct dirty cards is >fairly low. The increase in RS scan time I am seeing may simply be the >cost converging on an eventual very high cost resulting from poor >locality of writes to old regions. In this case, it may be that the >remember set simply do not scale well enough period, with many >well-spread writes to older generations pointing to young data. > >(2) Maybe I am right in my thinking that writes to older regions would >already be well spread long before the 1gb heapsize, and there is >something wrong with the RS scanning itself that causes costs to go up >much more than they "should" given the work load. > >Is there a better explanation? > >In the case of (1) it seems to me that what I really want is to run G1 >in non-generational mode since I would then expect it to evacuate >useful regions that does in fact free data (thanks to the statistics >gathered by the concurrent marking). In this case the cost of RS >scanning, even if it may be high, would at least be a trade-off to >actual progress in terms of freeing heap. RIght now, all this RS scan >work is being spent on not making any progress. > >(What are the reasons for non-generational mode no longer being >supported? Enabling it asserts later on so it is clearly not by >accident. Is it because of the optimization to not to remember set >writes for writes in the young generation, that it is not just a >switch to turn it off? Are there other problems with running it >non-generationally?) > >In the case of (2) perhaps this indicates a fixable problem with the GC. > >As an aside, the GC behavior during young generation eviction is, as >expected, a lot better when I run it against an immutable LRU cache: > > http://github.com/scode/plrutest/blob/master/src/org/scode/plrutest/clojure/lrucache.clj > >(That is just the cache, not a runnable test.) > > > _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From peter.schuller at infidyne.com Sat Feb 27 20:19:55 2010 From: peter.schuller at infidyne.com (Peter Schuller) Date: Sat, 27 Feb 2010 21:19:55 +0100 Subject: g1: remembered set scan costs very high w/ LRU cache type of behavior In-Reply-To: <4B896ED3.3040107@sun.com> References: <5a1151761002271048n798d8e54of9981e88e2b447a8@mail.gmail.com> <4B896ED3.3040107@sun.com> Message-ID: <5a1151761002271219j3f241f9aj6bdd5b2a724ceebe@mail.gmail.com> > Which version of the jdk are you using? ?A jdk 6.0 update? ?Or > jdk 7.0 build? Apologies, I meant to include that. I'm using: java version "1.7.0-ea" Java(TM) SE Runtime Environment (build 1.7.0-ea-b84) Java HotSpot(TM) 64-Bit Server VM (build 17.0-b09, mixed mode) -- / Peter Schuller _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From Jon.Masamitsu at Sun.COM Sat Feb 27 21:12:27 2010 From: Jon.Masamitsu at Sun.COM (Jon Masamitsu) Date: Sat, 27 Feb 2010 13:12:27 -0800 Subject: g1: remembered set scan costs very high w/ LRU cache type of behavior In-Reply-To: <5a1151761002271219j3f241f9aj6bdd5b2a724ceebe@mail.gmail.com> References: <5a1151761002271048n798d8e54of9981e88e2b447a8@mail.gmail.com> <4B896ED3.3040107@sun.com> <5a1151761002271219j3f241f9aj6bdd5b2a724ceebe@mail.gmail.com> Message-ID: <4B898ABB.4000409@sun.com> Peter Schuller wrote On 02/27/10 12:19,: >>Which version of the jdk are you using? ? A jdk 6.0 update? ? Or >>jdk 7.0 build? >> >> > >Apologies, I meant to include that. I'm using: > >java version "1.7.0-ea" >Java(TM) SE Runtime Environment (build 1.7.0-ea-b84) >Java HotSpot(TM) 64-Bit Server VM (build 17.0-b09, mixed mode) > > > Peter, The problem you're seeing may be addressed by some work in progress that is being done under CR 6923991: G1: improve scalability of RSet scanning I don't think that work has been released yet. If it hasn't, with your permission, we'll try your test program with it and see how well it does. By the way, thank you very much for giving G1 a look. Especially for the details in your mail. Jon _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From peter.schuller at infidyne.com Sat Feb 27 23:18:31 2010 From: peter.schuller at infidyne.com (Peter Schuller) Date: Sun, 28 Feb 2010 00:18:31 +0100 Subject: g1: remembered set scan costs very high w/ LRU cache type of behavior In-Reply-To: <4B898ABB.4000409@sun.com> References: <5a1151761002271048n798d8e54of9981e88e2b447a8@mail.gmail.com> <4B896ED3.3040107@sun.com> <5a1151761002271219j3f241f9aj6bdd5b2a724ceebe@mail.gmail.com> <4B898ABB.4000409@sun.com> Message-ID: <5a1151761002271518q40d95865o311699ef66764d32@mail.gmail.com> > The problem you're seeing may be addressed by some > work in progress that is being done under CR > > 6923991: G1: improve scalability of RSet scanning > > I don't think that work has been released yet. Well, if this commit constitutes the work: http://hg.openjdk.java.net/jdk7/hotspot/hotspot/rev/0414c1049f15 Then I suspect I have it, because the JDK comes from: jdk-7-ea-bin-b84-linux-x64-18_feb_2010.bin Which is 7 days after the commit happened. I'll see about building from source to maks sure. However, reading about it my initial impression that it is probably not the same issue as it seems to be about many-core scalability of the RS scanning. In my case I see poor performance even with a single core, and to the extent it is a scaling issue I get the impression it is a function of the number of RS tracked writes rather than the number of CPU cores. > If it hasn't, > with your permission, we'll try your test program with > it and see how well it does. Of course. Btw, I discovered something else. I have previously been doing some testing on JDK:s build from the jdk 1.7 bsdport on FreeBSD and had stability issues where the JVM would suddenly exit with no message when I ran my clojure persistent LRU test with G1. I just hit this behavior on the Linux host earlier today, with the snapshot binary mentioned above, which means I can no longer chalk it up to a FreeBSD specific issue. I doubt I can do much to narrow it down to a very small test case, but I could tidy up the test case a bit and make that available, and perhaps produce an executable .jar with clojure + the test case if that helps. I suspect (based on speculation rather than evidence) that the allocation and write patterns being generated by the use of persistent/immutable data structures is key to triggering it. That is, unless there is an actual System.exit() somewhere in the codebase... but I highly doubt there would be code that does this as a function of some behavior which depends on the garbage collector choice. -- / Peter Schuller _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From peter.schuller at infidyne.com Sun Feb 28 00:17:18 2010 From: peter.schuller at infidyne.com (Peter Schuller) Date: Sun, 28 Feb 2010 01:17:18 +0100 Subject: g1: remembered set scan costs very high w/ LRU cache type of behavior In-Reply-To: <5a1151761002271518q40d95865o311699ef66764d32@mail.gmail.com> References: <5a1151761002271048n798d8e54of9981e88e2b447a8@mail.gmail.com> <4B896ED3.3040107@sun.com> <5a1151761002271219j3f241f9aj6bdd5b2a724ceebe@mail.gmail.com> <4B898ABB.4000409@sun.com> <5a1151761002271518q40d95865o311699ef66764d32@mail.gmail.com> Message-ID: <5a1151761002271617u38322a93u18cddb88407b381c@mail.gmail.com> > I'll see about building from source to maks sure. I see the same behavior (RS scan times begin growing at ~ 1 gb heap) with a JDK7 built from sources cloned from hg just now (an hour or so ago). -- / Peter Schuller _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From Jon.Masamitsu at Sun.COM Sun Feb 28 02:46:10 2010 From: Jon.Masamitsu at Sun.COM (Jon Masamitsu) Date: Sat, 27 Feb 2010 18:46:10 -0800 Subject: g1: remembered set scan costs very high w/ LRU cache type of behavior In-Reply-To: <5a1151761002271617u38322a93u18cddb88407b381c@mail.gmail.com> References: <5a1151761002271048n798d8e54of9981e88e2b447a8@mail.gmail.com> <4B896ED3.3040107@sun.com> <5a1151761002271219j3f241f9aj6bdd5b2a724ceebe@mail.gmail.com> <4B898ABB.4000409@sun.com> <5a1151761002271518q40d95865o311699ef66764d32@mail.gmail.com> <5a1151761002271617u38322a93u18cddb88407b381c@mail.gmail.com> Message-ID: <4B89D8F2.5040208@sun.com> Peter Schuller wrote On 02/27/10 16:17,: >>I'll see about building from source to maks sure. >> >> > >I see the same behavior (RS scan times begin growing at ~ 1 gb heap) >with a JDK7 built from sources cloned from hg just now (an hour or so >ago). > > > Thanks for doing the experiment. We'll have to take a closer look. _______________________________________________ hotspot-gc-use mailing list hotspot-gc-use at openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use