From bengt.rutisson at oracle.com Mon May 7 14:23:06 2012 From: bengt.rutisson at oracle.com (Bengt Rutisson) Date: Mon, 07 May 2012 23:23:06 +0200 Subject: Will parsers break if we start logging the GC cause as part of the PrintGC logging? Message-ID: <4FA83D3A.1000703@oracle.com> Hi all, I have a webrev out for a change that will add the GC cause to all "Full GC logging". See: http://permalink.gmane.org/gmane.comp.java.openjdk.hotspot.gc.devel/4527 The extra information was intentionally just added to full GCs since this logging already had information for System.gc() calls so we figured that any parsers out there would have to handle this information anyway. It was requested to add the information about the GC cause also to CMS collections. If I start down that path I think I could just as well add the GC cause to all GC logging. If we break any parsers we will probably break them already when we add the cause to CMS GCs. Not sure what the best way to handle this is. Some suggestions: (1) Only add cause to Full GCs (as in my change now) (2) Only add cause to Full GCs and CMS GCs (as I think is what was suggested) (3) Add cause to all GCs (probably the proper but kind of risky way) (4) Only do (1) but file CRs for (2) and (3) Any thoughts? It is really a choice between getting interesting information and risking breaking existing GC log parsers. Here is the latest webrev: http://cr.openjdk.java.net/~brutisso/7166894/webrev.01/ Thanks, Bengt From vitalyd at gmail.com Mon May 7 14:24:54 2012 From: vitalyd at gmail.com (Vitaly Davidovich) Date: Mon, 7 May 2012 17:24:54 -0400 Subject: Will parsers break if we start logging the GC cause as part of the PrintGC logging? In-Reply-To: <4FA83D3A.1000703@oracle.com> References: <4FA83D3A.1000703@oracle.com> Message-ID: Hi Bengt, Another option would be to enable the extra logging/new format via a VM argument? I know there are already tons of them so this is probably undesirable, but may provide at least a transition period for customers to upgrade their parsers. Sent from my phone On May 7, 2012 5:19 PM, "Bengt Rutisson" wrote: > > Hi all, > > I have a webrev out for a change that will add the GC cause to all "Full > GC logging". See: > > http://permalink.gmane.org/gmane.comp.java.openjdk.hotspot.gc.devel/4527 > > The extra information was intentionally just added to full GCs since > this logging already had information for System.gc() calls so we figured > that any parsers out there would have to handle this information anyway. > > It was requested to add the information about the GC cause also to CMS > collections. If I start down that path I think I could just as well add > the GC cause to all GC logging. If we break any parsers we will probably > break them already when we add the cause to CMS GCs. > > Not sure what the best way to handle this is. Some suggestions: > > (1) Only add cause to Full GCs (as in my change now) > (2) Only add cause to Full GCs and CMS GCs (as I think is what was > suggested) > (3) Add cause to all GCs (probably the proper but kind of risky way) > (4) Only do (1) but file CRs for (2) and (3) > > Any thoughts? It is really a choice between getting interesting > information and risking breaking existing GC log parsers. > > Here is the latest webrev: > http://cr.openjdk.java.net/~brutisso/7166894/webrev.01/ > > Thanks, > Bengt > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120507/024a53f7/attachment.html From bengt.rutisson at oracle.com Mon May 7 14:37:15 2012 From: bengt.rutisson at oracle.com (Bengt Rutisson) Date: Mon, 07 May 2012 23:37:15 +0200 Subject: Will parsers break if we start logging the GC cause as part of the PrintGC logging? In-Reply-To: References: <4FA83D3A.1000703@oracle.com> Message-ID: <4FA8408B.3080001@oracle.com> Hi Vitaly, On 2012-05-07 23:24, Vitaly Davidovich wrote: > > Hi Bengt, > > Another option would be to enable the extra logging/new format via a > VM argument? I know there are already tons of them so this is probably > undesirable, but may provide at least a transition period for > customers to upgrade their parsers. > Agreed, but I would really like to avoid more VM arguments. If we feel uncomfortable with this change I think we should just not do it at this point in time. There is a project in planning for a more general logging framework for Hotspot. Once that is in place I am sure we will do a major update to the GC logging. At that time I guess many parsers will break, so if we feel that the proposed change is high risk at the moment I think we should postpone it to the major logging overhaul. I know there has been a JEP written for the new logging framework but that JEP is still awaiting publication. Hopefully it will be available soon. Thanks, Bengt > Sent from my phone > > On May 7, 2012 5:19 PM, "Bengt Rutisson" > wrote: > > > Hi all, > > I have a webrev out for a change that will add the GC cause to all > "Full > GC logging". See: > > http://permalink.gmane.org/gmane.comp.java.openjdk.hotspot.gc.devel/4527 > > The extra information was intentionally just added to full GCs since > this logging already had information for System.gc() calls so we > figured > that any parsers out there would have to handle this information > anyway. > > It was requested to add the information about the GC cause also to CMS > collections. If I start down that path I think I could just as > well add > the GC cause to all GC logging. If we break any parsers we will > probably > break them already when we add the cause to CMS GCs. > > Not sure what the best way to handle this is. Some suggestions: > > (1) Only add cause to Full GCs (as in my change now) > (2) Only add cause to Full GCs and CMS GCs (as I think is what was > suggested) > (3) Add cause to all GCs (probably the proper but kind of risky way) > (4) Only do (1) but file CRs for (2) and (3) > > Any thoughts? It is really a choice between getting interesting > information and risking breaking existing GC log parsers. > > Here is the latest webrev: > http://cr.openjdk.java.net/~brutisso/7166894/webrev.01/ > > > Thanks, > Bengt > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120507/b47faf78/attachment.html From vitalyd at gmail.com Mon May 7 14:49:05 2012 From: vitalyd at gmail.com (Vitaly Davidovich) Date: Mon, 7 May 2012 17:49:05 -0400 Subject: Will parsers break if we start logging the GC cause as part of the PrintGC logging? In-Reply-To: <4FA8408B.3080001@oracle.com> References: <4FA83D3A.1000703@oracle.com> <4FA8408B.3080001@oracle.com> Message-ID: Sounds good and makes sense - thanks. Sent from my phone On May 7, 2012 5:32 PM, "Bengt Rutisson" wrote: > > Hi Vitaly, > > On 2012-05-07 23:24, Vitaly Davidovich wrote: > > Hi Bengt, > > Another option would be to enable the extra logging/new format via a VM > argument? I know there are already tons of them so this is probably > undesirable, but may provide at least a transition period for customers to > upgrade their parsers. > > > Agreed, but I would really like to avoid more VM arguments. If we feel > uncomfortable with this change I think we should just not do it at this > point in time. > > There is a project in planning for a more general logging framework for > Hotspot. Once that is in place I am sure we will do a major update to the > GC logging. At that time I guess many parsers will break, so if we feel > that the proposed change is high risk at the moment I think we should > postpone it to the major logging overhaul. > > I know there has been a JEP written for the new logging framework but that > JEP is still awaiting publication. Hopefully it will be available soon. > > Thanks, > Bengt > > Sent from my phone > On May 7, 2012 5:19 PM, "Bengt Rutisson" > wrote: > >> >> Hi all, >> >> I have a webrev out for a change that will add the GC cause to all "Full >> GC logging". See: >> >> http://permalink.gmane.org/gmane.comp.java.openjdk.hotspot.gc.devel/4527 >> >> The extra information was intentionally just added to full GCs since >> this logging already had information for System.gc() calls so we figured >> that any parsers out there would have to handle this information anyway. >> >> It was requested to add the information about the GC cause also to CMS >> collections. If I start down that path I think I could just as well add >> the GC cause to all GC logging. If we break any parsers we will probably >> break them already when we add the cause to CMS GCs. >> >> Not sure what the best way to handle this is. Some suggestions: >> >> (1) Only add cause to Full GCs (as in my change now) >> (2) Only add cause to Full GCs and CMS GCs (as I think is what was >> suggested) >> (3) Add cause to all GCs (probably the proper but kind of risky way) >> (4) Only do (1) but file CRs for (2) and (3) >> >> Any thoughts? It is really a choice between getting interesting >> information and risking breaking existing GC log parsers. >> >> Here is the latest webrev: >> http://cr.openjdk.java.net/~brutisso/7166894/webrev.01/ >> >> Thanks, >> Bengt >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120507/c9acdaea/attachment.html From ysr1729 at gmail.com Mon May 7 22:52:49 2012 From: ysr1729 at gmail.com (Srinivas Ramakrishna) Date: Mon, 7 May 2012 22:52:49 -0700 Subject: Will parsers break if we start logging the GC cause as part of the PrintGC logging? In-Reply-To: References: <4FA83D3A.1000703@oracle.com> Message-ID: I'd tend to agree with Vitaly, since that puts the choice in the hands of the users, at least until they figure out how to fix their parsers... You could enable the option by default, but the existence of the option allows a user to switch off the feature if they just can't deal with it. That said, as a user I wouldn't lose any sleep over it... We'll just fix our parsers if/ when the change comes along. I'd strongly suggest making the change for all collectors, rather than for only a small subset... Thanks for this addition, it'll be generally useful.... And I'd say have it for each collection, not just full collections... -- Ramki Sent from my iPhone On May 7, 2012, at 2:24 PM, Vitaly Davidovich wrote: > Hi Bengt, > > Another option would be to enable the extra logging/new format via a VM argument? I know there are already tons of them so this is probably undesirable, but may provide at least a transition period for customers to upgrade their parsers. > > Sent from my phone > > On May 7, 2012 5:19 PM, "Bengt Rutisson" wrote: > > Hi all, > > I have a webrev out for a change that will add the GC cause to all "Full > GC logging". See: > > http://permalink.gmane.org/gmane.comp.java.openjdk.hotspot.gc.devel/4527 > > The extra information was intentionally just added to full GCs since > this logging already had information for System.gc() calls so we figured > that any parsers out there would have to handle this information anyway. > > It was requested to add the information about the GC cause also to CMS > collections. If I start down that path I think I could just as well add > the GC cause to all GC logging. If we break any parsers we will probably > break them already when we add the cause to CMS GCs. > > Not sure what the best way to handle this is. Some suggestions: > > (1) Only add cause to Full GCs (as in my change now) > (2) Only add cause to Full GCs and CMS GCs (as I think is what was > suggested) > (3) Add cause to all GCs (probably the proper but kind of risky way) > (4) Only do (1) but file CRs for (2) and (3) > > Any thoughts? It is really a choice between getting interesting > information and risking breaking existing GC log parsers. > > Here is the latest webrev: > http://cr.openjdk.java.net/~brutisso/7166894/webrev.01/ > > Thanks, > Bengt > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120507/8877e874/attachment-0001.html From rednaxelafx at gmail.com Mon May 7 23:30:02 2012 From: rednaxelafx at gmail.com (Krystal Mok) Date: Tue, 8 May 2012 14:30:02 +0800 Subject: Will parsers break if we start logging the GC cause as part of the PrintGC logging? In-Reply-To: <4FA8408B.3080001@oracle.com> References: <4FA83D3A.1000703@oracle.com> <4FA8408B.3080001@oracle.com> Message-ID: Hi Bengt, Comments inline below: On Tue, May 8, 2012 at 5:37 AM, Bengt Rutisson wrote: > > Agreed, but I would really like to avoid more VM arguments. If we feel > uncomfortable with this change I think we should just not do it at this > point in time. > > Yes, I tend to agree with this approach. And that's why I didn't submit a patch of PrintGCReason to the upstream. It's not gonna be fun to fix parsing logic twice in a row. The change in G1 was okay because there aren't a whole bunch of parsers in the wild that could parse G1 logs. > There is a project in planning for a more general logging framework for > Hotspot. Once that is in place I am sure we will do a major update to the > GC logging. At that time I guess many parsers will break, so if we feel > that the proposed change is high risk at the moment I think we should > postpone it to the major logging overhaul. > > And I've been holding back my new parser work waiting for this... > I know there has been a JEP written for the new logging framework but that > JEP is still awaiting publication. Hopefully it will be available soon. > > Thanks, > Bengt > > Thanks, Kris -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120508/f97bba67/attachment.html From bengt.rutisson at oracle.com Tue May 8 03:42:27 2012 From: bengt.rutisson at oracle.com (Bengt Rutisson) Date: Tue, 08 May 2012 12:42:27 +0200 Subject: Will parsers break if we start logging the GC cause as part of the PrintGC logging? In-Reply-To: References: <4FA83D3A.1000703@oracle.com> Message-ID: <4FA8F893.604@oracle.com> Thanks for the feedback everyone! I think I'll do as Ramki suggested. Add the GC cause to all collectors, but add a switch for users to turn it off. I am not too happy about adding another command line option, but is seems more consistent to do the change for all collectors. I'll do the changes and post a new webrev on the hotspot-gc-dev list. Vitaly, I have been experimenting a bit with your suggestion of a stack local object to do the string concatenation. This will be even more valuable if I do the change for all collectors, so I'll probably include that in the new webrev as well. Thanks, Bengt On 2012-05-08 07:52, Srinivas Ramakrishna wrote: > I'd tend to agree with Vitaly, since that puts the choice in the hands > of the users, at least until they figure out how to fix their > parsers... You could enable the option by default, but the existence > of the option allows a user to switch off the feature if they just > can't deal with it. That said, as a user I wouldn't lose any sleep > over it... We'll just fix our parsers if/ when the change comes along. > > I'd strongly suggest making the change for all collectors, rather than > for only a small subset... > > Thanks for this addition, it'll be generally useful.... And I'd say > have it for each collection, not just full collections... > > -- Ramki > > Sent from my iPhone > > On May 7, 2012, at 2:24 PM, Vitaly Davidovich > wrote: > >> Hi Be! ngt, >> >> Another option would be to enable the extra logging/new format via a >> VM argument? I know there are already tons of them so this is >> probably undesirable, but may provide at least a transition period >> for customers to upgrade their parsers. >> >> Sent from my phone >> >> On May 7, 2012 5:19 PM, "Bengt Rutisson" > > wrote: >> >> >> Hi all, >> >> I have a webrev out for a change that will add the GC cause to >> all "Full >> GC logging". See: >> >> http://permalink.gmane.org/gmane.comp.java.openjdk.hotspot.gc.devel/4527 >> >> The extra information was intentionally just added to full GCs since >> this logging already had information for System.gc() calls so we >> figured >> that any parsers out there would have to handle this information >> anyway. >> >> It was requested to add the information about the GC cause also >> to CMS >> collections. If I start down that path I think I could just as >> well add >> the GC cause to all GC logging. If we break any parsers we will >> probably >> break them already when we add the cause to CMS GCs. >> >> Not sure what the best way to handle this is. Some suggestions: >> >> (1) Only add cause to Full GCs (as in my change now) >> (2) Only add cause to Full GCs and CMS GCs (as I think is what was >> suggested) >> (3) Add cause to all GCs (probably the proper but kind of risky way) >> (4) Only do (1) but file CRs for (2) and (3) >> >> Any thoughts? It is really a choice between getting interesting >> information and risking breaking existing GC log parsers. >> >> Here is the latest webrev: >> http://cr.openjdk.java.net/~brutisso/7166894/webrev.01/ >> >> >> Thanks, >> Bengt >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120508/88b42be9/attachment.html From vitalyd at gmail.com Tue May 8 06:09:11 2012 From: vitalyd at gmail.com (Vitaly Davidovich) Date: Tue, 8 May 2012 09:09:11 -0400 Subject: Will parsers break if we start logging the GC cause as part of the PrintGC logging? In-Reply-To: <4FA8F893.604@oracle.com> References: <4FA83D3A.1000703@oracle.com> <4FA8F893.604@oracle.com> Message-ID: Hi Bengt, Sounds good - looking forward to seeing what you come up with. Let me also echo others that this is a great addition, so thanks for doing it. Thanks Sent from my phone On May 8, 2012 6:38 AM, "Bengt Rutisson" wrote: > > Thanks for the feedback everyone! > > I think I'll do as Ramki suggested. Add the GC cause to all collectors, > but add a switch for users to turn it off. I am not too happy about adding > another command line option, but is seems more consistent to do the change > for all collectors. > > I'll do the changes and post a new webrev on the hotspot-gc-dev list. > > Vitaly, I have been experimenting a bit with your suggestion of a stack > local object to do the string concatenation. This will be even more > valuable if I do the change for all collectors, so I'll probably include > that in the new webrev as well. > > Thanks, > Bengt > > On 2012-05-08 07:52, Srinivas Ramakrishna wrote: > > I'd tend to agree with Vitaly, since that puts the choice in the hands of > the users, at least until they figure out how to fix their parsers... You > could enable the option by default, but the existence of the option allows > a user to switch off the feature if they just can't deal with it. That > said, as a user I wouldn't lose any sleep over it... We'll just fix our > parsers if/ when the change comes along. > > I'd strongly suggest making the change for all collectors, rather than > for only a small subset... > > Thanks for this addition, it'll be generally useful.... And I'd say have > it for each collection, not just full collections... > > -- Ramki > > Sent from my iPhone > > On May 7, 2012, at 2:24 PM, Vitaly Davidovich wrote: > > Hi Be! ngt, > > Another option would be to enable the extra logging/new format via a VM > argument? I know there are already tons of them so this is probably > undesirable, but may provide at least a transition period for customers to > upgrade their parsers. > > Sent from my phone > On May 7, 2012 5:19 PM, "Bengt Rutisson" > wrote: > >> >> Hi all, >> >> I have a webrev out for a change that will add the GC cause to all "Full >> GC logging". See: >> >> http://permalink.gmane.org/gmane.comp.java.openjdk.hotspot.gc.devel/4527 >> >> The extra information was intentionally just added to full GCs since >> this logging already had information for System.gc() calls so we figured >> that any parsers out there would have to handle this information anyway. >> >> It was requested to add the information about the GC cause also to CMS >> collections. If I start down that path I think I could just as well add >> the GC cause to all GC logging. If we break any parsers we will probably >> break them already when we add the cause to CMS GCs. >> >> Not sure what the best way to handle this is. Some suggestions: >> >> (1) Only add cause to Full GCs (as in my change now) >> (2) Only add cause to Full GCs and CMS GCs (as I think is what was >> suggested) >> (3) Add cause to all GCs (probably the proper but kind of risky way) >> (4) Only do (1) but file CRs for (2) and (3) >> >> Any thoughts? It is really a choice between getting interesting >> information and risking breaking existing GC log parsers. >> >> Here is the latest webrev: >> http://cr.openjdk.java.net/~brutisso/7166894/webrev.01/ >> >> Thanks, >> Bengt >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120508/aa4b6ad0/attachment-0001.html From misha.dmitriev at oracle.com Tue May 8 12:56:48 2012 From: misha.dmitriev at oracle.com (Misha Dmitriev) Date: Tue, 8 May 2012 12:56:48 -0700 Subject: Will parsers break if we start logging the GC cause as part of the PrintGC logging? In-Reply-To: References: Message-ID: <3B4FD499-9900-46ED-97E0-B222B2AD4690@oracle.com> > Message: 2 > Date: Tue, 08 May 2012 12:42:27 +0200 > From: Bengt Rutisson > Subject: Re: Will parsers break if we start logging the GC cause as > part of the PrintGC logging? > To: Srinivas Ramakrishna > Cc: "hotspot-gc-use at openjdk.java.net" > > Message-ID: <4FA8F893.604 at oracle.com> > Content-Type: text/plain; charset="utf-8" > > > Thanks for the feedback everyone! > > I think I'll do as Ramki suggested. Add the GC cause to all collectors, > but add a switch for users to turn it off. I am not too happy about > adding another command line option, but is seems more consistent to do > the change for all collectors. I agree that the "enable by default, and add a flag to disable" is the best choice. Speaking about HotSpot VM options/flags. I think the biggest problem is not that there are so many flags (I've seen applications with many more), but the fact that the documentation for them is difficult to find, and some flags are always missing from it. If 'java -Xhelp' and 'java -XXhelp' (well, the second option does not even exist) gave me all the available flags for sure, managing them would be much easier. Misha From ysr1729 at gmail.com Tue May 8 14:07:11 2012 From: ysr1729 at gmail.com (Srinivas Ramakrishna) Date: Tue, 8 May 2012 14:07:11 -0700 Subject: Will parsers break if we start logging the GC cause as part of the PrintGC logging? In-Reply-To: <3B4FD499-9900-46ED-97E0-B222B2AD4690@oracle.com> References: <3B4FD499-9900-46ED-97E0-B222B2AD4690@oracle.com> Message-ID: Hi Misha -- I think a slightly more glorified version of -XX:+PrintFlagsFinal will perhaps be sufficient, which also includes a dump of the documentation string usually associated with the JVM flags. I agree with you that something like that would be both "self-documenting" (if you will) and stay current with the installation at hand without grovelling through source code (or out of date documentation). It would also make power users inn data centers hugely happy i think... -- ramki On Tue, May 8, 2012 at 12:56 PM, Misha Dmitriev wrote: > > Message: 2 > > Date: Tue, 08 May 2012 12:42:27 +0200 > > From: Bengt Rutisson > > Subject: Re: Will parsers break if we start logging the GC cause as > > part of the PrintGC logging? > > To: Srinivas Ramakrishna > > Cc: "hotspot-gc-use at openjdk.java.net" > > > > Message-ID: <4FA8F893.604 at oracle.com> > > Content-Type: text/plain; charset="utf-8" > > > > > > Thanks for the feedback everyone! > > > > I think I'll do as Ramki suggested. Add the GC cause to all collectors, > > but add a switch for users to turn it off. I am not too happy about > > adding another command line option, but is seems more consistent to do > > the change for all collectors. > > I agree that the "enable by default, and add a flag to disable" is the > best choice. > > Speaking about HotSpot VM options/flags. I think the biggest problem is > not that there are so many flags (I've seen applications with many more), > but the fact that the documentation for them is difficult to find, and some > flags are always missing from it. If 'java -Xhelp' and 'java -XXhelp' > (well, the second option does not even exist) gave me all the available > flags for sure, managing them would be much easier. > > Misha > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120508/e3fdd266/attachment.html From emanuel.amaral.couto at gmail.com Tue May 8 17:32:04 2012 From: emanuel.amaral.couto at gmail.com (Emanuel Couto) Date: Wed, 9 May 2012 01:32:04 +0100 Subject: ParallelGC does not allocate all memory it could have Message-ID: Hi, I have a question regarding ParallelGC. I have used the following parameters to execute a program that is creating objects of random size until no more heap is available: java -XX:+UseParallelGC -XX:+UseParallelOldGC -Xmx256m -jar I noticed that when it is not possible to allocate more memory in the tenured generation and it is full, the JVM does not allocate the maximum possible memory for the young generation, thus throwing an OutOfMemoryException before the heap is full. When the OutOfMemoryException is thrown the tenured generation allocates 170MB, which is maximum size according to visualvm. The young generation only allocates 20MB, but it had 85MB reserved. This behavior does not happen with SerialGC or ConcurrentGC. -- Emanuel Couto From jon.masamitsu at oracle.com Tue May 8 21:24:59 2012 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Tue, 08 May 2012 21:24:59 -0700 Subject: ParallelGC does not allocate all memory it could have In-Reply-To: References: Message-ID: <4FA9F19B.3090106@oracle.com> Emanuel, Run you test with -XX:+PrintGCDetails and -XX:+PrintHeapAtGC and send the output for at least the last half dozen GC's. Also, have you test program print out the size of the allocation that failed with the OutOfMemory and send that. Jon On 5/8/2012 5:32 PM, Emanuel Couto wrote: > Hi, > > I have a question regarding ParallelGC. I have used the following > parameters to execute a program that is creating objects of random > size until no more heap is available: > > java -XX:+UseParallelGC -XX:+UseParallelOldGC -Xmx256m -jar > > I noticed that when it is not possible to allocate more memory in the > tenured generation and it is full, the JVM does not allocate the > maximum possible memory for the young generation, thus throwing an > OutOfMemoryException before the heap is full. > > When the OutOfMemoryException is thrown the tenured generation > allocates 170MB, which is maximum size according to visualvm. The > young generation only allocates 20MB, but it had 85MB reserved. > > This behavior does not happen with SerialGC or ConcurrentGC. > From dhd at exnet.com Tue May 8 23:47:01 2012 From: dhd at exnet.com (Damon Hart-Davis) Date: Wed, 9 May 2012 07:47:01 +0100 Subject: MinHeapFreeRatio / MaxHeapFreeRatio In-Reply-To: References: Message-ID: Hi, Note: I should have been clear that memory is not given back to the OS even when the heap is (say) 60%+ free and I force GCs. Rgds Damon On 9 May 2012, at 06:46, Damon Hart-Davis wrote: > Hi, > > First time on this list, so hello! I see a few familiar names from days of yore! > > I am running my favourite (Tomcat/Web) app in a very constrained memory environment (a SheevaPlug) along with all the usual Internet server junk (sendmail, ntp, sshd, etc) and I am aiming to squeeze it into the even tighter space of a Raspberry Pi in due course. > > As such, I want the JVM to give memory back to the OS whenever possible. In code I set a target to keep about 25% of the heap free, and so for example I stop cacheing some stuff when below that, and cache more vigorously above that. > > Here are some relevant options: > > CATALINA_OPTS="-Xmx100m -Xms64m" > # Cap size of non-(main-)heap components. > CATALINA_OPTS="$CATALINA_OPTS -XX:MaxPermSize=48m" > # Trim thread stack size. > CATALINA_OPTS="$CATALINA_OPTS -Xss256k" > # Keep the new generation well within the target 25% free... > CATALINA_OPTS="$CATALINA_OPTS -XX:NewRatio=5" > # Be aggressive about giving memory back to the system above target 25% free. > CATALINA_OPTS="$CATALINA_OPTS -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=26" > # Run in incremental GC mode to minimise pauses. > CATALINA_OPTS="$CATALINA_OPTS -Xincgc" > > I see some evidence that the free ratios are being observed by when the heap is expanded, but I have never seen any variant of JDK 6 or 7, including the Oracle embedded 6, actually give memory back to the OS with this app. > > Am I still doing something wrong? B^> > > Rgds > > Damon > From dhd at exnet.com Tue May 8 22:46:45 2012 From: dhd at exnet.com (Damon Hart-Davis) Date: Wed, 9 May 2012 06:46:45 +0100 Subject: MinHeapFreeRatio / MaxHeapFreeRatio In-Reply-To: References: Message-ID: Hi, First time on this list, so hello! I see a few familiar names from days of yore! I am running my favourite (Tomcat/Web) app in a very constrained memory environment (a SheevaPlug) along with all the usual Internet server junk (sendmail, ntp, sshd, etc) and I am aiming to squeeze it into the even tighter space of a Raspberry Pi in due course. As such, I want the JVM to give memory back to the OS whenever possible. In code I set a target to keep about 25% of the heap free, and so for example I stop cacheing some stuff when below that, and cache more vigorously above that. Here are some relevant options: CATALINA_OPTS="-Xmx100m -Xms64m" # Cap size of non-(main-)heap components. CATALINA_OPTS="$CATALINA_OPTS -XX:MaxPermSize=48m" # Trim thread stack size. CATALINA_OPTS="$CATALINA_OPTS -Xss256k" # Keep the new generation well within the target 25% free... CATALINA_OPTS="$CATALINA_OPTS -XX:NewRatio=5" # Be aggressive about giving memory back to the system above target 25% free. CATALINA_OPTS="$CATALINA_OPTS -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=26" # Run in incremental GC mode to minimise pauses. CATALINA_OPTS="$CATALINA_OPTS -Xincgc" I see some evidence that the free ratios are being observed by when the heap is expanded, but I have never seen any variant of JDK 6 or 7, including the Oracle embedded 6, actually give memory back to the OS with this app. Am I still doing something wrong? B^> Rgds Damon From jon.masamitsu at oracle.com Wed May 9 02:27:03 2012 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Wed, 09 May 2012 02:27:03 -0700 Subject: MinHeapFreeRatio / MaxHeapFreeRatio In-Reply-To: References: Message-ID: <4FAA3867.2020808@oracle.com> Damon, Do you turn on ExplicitGCInvokesConcurrent? Jon On 5/8/2012 11:47 PM, Damon Hart-Davis wrote: > Hi, > > Note: I should have been clear that memory is not given back to the OS even when the heap is (say) 60%+ free and I force GCs. > > Rgds > > Damon > > > On 9 May 2012, at 06:46, Damon Hart-Davis wrote: > >> Hi, >> >> First time on this list, so hello! I see a few familiar names from days of yore! >> >> I am running my favourite (Tomcat/Web) app in a very constrained memory environment (a SheevaPlug) along with all the usual Internet server junk (sendmail, ntp, sshd, etc) and I am aiming to squeeze it into the even tighter space of a Raspberry Pi in due course. >> >> As such, I want the JVM to give memory back to the OS whenever possible. In code I set a target to keep about 25% of the heap free, and so for example I stop cacheing some stuff when below that, and cache more vigorously above that. >> >> Here are some relevant options: >> >> CATALINA_OPTS="-Xmx100m -Xms64m" >> # Cap size of non-(main-)heap components. >> CATALINA_OPTS="$CATALINA_OPTS -XX:MaxPermSize=48m" >> # Trim thread stack size. >> CATALINA_OPTS="$CATALINA_OPTS -Xss256k" >> # Keep the new generation well within the target 25% free... >> CATALINA_OPTS="$CATALINA_OPTS -XX:NewRatio=5" >> # Be aggressive about giving memory back to the system above target 25% free. >> CATALINA_OPTS="$CATALINA_OPTS -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=26" >> # Run in incremental GC mode to minimise pauses. >> CATALINA_OPTS="$CATALINA_OPTS -Xincgc" >> >> I see some evidence that the free ratios are being observed by when the heap is expanded, but I have never seen any variant of JDK 6 or 7, including the Oracle embedded 6, actually give memory back to the OS with this app. >> >> Am I still doing something wrong? B^> >> >> Rgds >> >> Damon >> > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From dhd at exnet.com Wed May 9 02:34:14 2012 From: dhd at exnet.com (Damon Hart-Davis) Date: Wed, 9 May 2012 10:34:14 +0100 Subject: MinHeapFreeRatio / MaxHeapFreeRatio In-Reply-To: <4FAA3867.2020808@oracle.com> References: <4FAA3867.2020808@oracle.com> Message-ID: <733AEBA7-7CE7-451E-A4A5-D053E65CE48F@exnet.com> Hi, No, because when I explicitly call System.gc() I want it to try as hard as it can, even to the point of stopping the world... Rgds Damon On 9 May 2012, at 10:27, Jon Masamitsu wrote: > Damon, > > Do you turn on ExplicitGCInvokesConcurrent? > > Jon > From jon.masamitsu at oracle.com Thu May 10 09:31:34 2012 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Thu, 10 May 2012 09:31:34 -0700 Subject: MinHeapFreeRatio / MaxHeapFreeRatio In-Reply-To: <733AEBA7-7CE7-451E-A4A5-D053E65CE48F@exnet.com> References: <4FAA3867.2020808@oracle.com> <733AEBA7-7CE7-451E-A4A5-D053E65CE48F@exnet.com> Message-ID: <4FABED66.1080308@oracle.com> Damon, From inspection of the code I think that CMS does not do a resizing of the heap after it does a System.gc(). That resizing would be the code that adjusts the generations for MinHeapFreeRatio / MaxHeapFreeRatio and returns memory to the OS. I've created CR 7167923 to investigate this bug. Jon On 5/9/2012 2:34 AM, Damon Hart-Davis wrote: > Hi, > > No, because when I explicitly call System.gc() I want it to try as hard as it can, even to the point of stopping the world... > > Rgds > > Damon > > > On 9 May 2012, at 10:27, Jon Masamitsu wrote: > >> Damon, >> >> Do you turn on ExplicitGCInvokesConcurrent? >> >> Jon >> From rednaxelafx at gmail.com Thu May 10 09:41:48 2012 From: rednaxelafx at gmail.com (Krystal Mok) Date: Fri, 11 May 2012 00:41:48 +0800 Subject: MinHeapFreeRatio / MaxHeapFreeRatio In-Reply-To: <4FABED66.1080308@oracle.com> References: <4FAA3867.2020808@oracle.com> <733AEBA7-7CE7-451E-A4A5-D053E65CE48F@exnet.com> <4FABED66.1080308@oracle.com> Message-ID: Hi Jon, No, CMS doesn't do shrinking at all. The interface is there in the generational GC framework, but the implementation in CMS is left empty. I tried to implement shrinking in CMS once, but I ran out of time completing it, and found out that the original demand from our internal customer was supposed to be using ParallelGC, so I gave up. Resizing generations to meet MinHeapFreeRatio / MaxHeapFreeRatio means shrinking has to be implemented in CMS, right? - Kris On Fri, May 11, 2012 at 12:31 AM, Jon Masamitsu wrote: > Damon, > > From inspection of the code I think that CMS does not do a > resizing of the heap after it does a System.gc(). That resizing > would be the code that adjusts the generations for > MinHeapFreeRatio / MaxHeapFreeRatio and returns > memory to the OS. > > I've created CR 7167923 to investigate this bug. > > Jon > > On 5/9/2012 2:34 AM, Damon Hart-Davis wrote: > > Hi, > > > > No, because when I explicitly call System.gc() I want it to try as hard > as it can, even to the point of stopping the world... > > > > Rgds > > > > Damon > > > > > > On 9 May 2012, at 10:27, Jon Masamitsu wrote: > > > >> Damon, > >> > >> Do you turn on ExplicitGCInvokesConcurrent? > >> > >> Jon > >> > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120511/a4f6e1b4/attachment.html From dhd at exnet.com Thu May 10 09:43:10 2012 From: dhd at exnet.com (Damon Hart-Davis) Date: Thu, 10 May 2012 17:43:10 +0100 Subject: MinHeapFreeRatio / MaxHeapFreeRatio In-Reply-To: <4FABED66.1080308@oracle.com> References: <4FAA3867.2020808@oracle.com> <733AEBA7-7CE7-451E-A4A5-D053E65CE48F@exnet.com> <4FABED66.1080308@oracle.com> Message-ID: <0C54993A-E20C-4170-B6C7-77B4EBDE5064@exnet.com> Thanks! Damon On 10 May 2012, at 17:31, Jon Masamitsu wrote: > Damon, > > From inspection of the code I think that CMS does not do a > resizing of the heap after it does a System.gc(). That resizing > would be the code that adjusts the generations for > MinHeapFreeRatio / MaxHeapFreeRatio and returns > memory to the OS. > > I've created CR 7167923 to investigate this bug. > > Jon > > On 5/9/2012 2:34 AM, Damon Hart-Davis wrote: >> Hi, >> >> No, because when I explicitly call System.gc() I want it to try as hard as it can, even to the point of stopping the world... >> >> Rgds >> >> Damon >> >> >> On 9 May 2012, at 10:27, Jon Masamitsu wrote: >> >>> Damon, >>> >>> Do you turn on ExplicitGCInvokesConcurrent? >>> >>> Jon >>> > From dhd at exnet.com Thu May 10 10:00:46 2012 From: dhd at exnet.com (Damon Hart-Davis) Date: Thu, 10 May 2012 18:00:46 +0100 Subject: MinHeapFreeRatio / MaxHeapFreeRatio In-Reply-To: References: <4FAA3867.2020808@oracle.com> <733AEBA7-7CE7-451E-A4A5-D053E65CE48F@exnet.com> <4FABED66.1080308@oracle.com> Message-ID: <58DB97C4-92D0-478A-8D94-424596E1318A@exnet.com> Hi, FWIW, I don't recall (though I'm not totally sure) the JVM ever giving memory back to the OS even when I tried (for other reasons) the serial collector in SE 6 Embedded. Rgds Damon On 10 May 2012, at 17:41, Krystal Mok wrote: > Hi Jon, > > No, CMS doesn't do shrinking at all. The interface is there in the generational GC framework, but the implementation in CMS is left empty. I tried to implement shrinking in CMS once, but I ran out of time completing it, and found out that the original demand from our internal customer was supposed to be using ParallelGC, so I gave up. > > Resizing generations to meet MinHeapFreeRatio / MaxHeapFreeRatio means shrinking has to be implemented in CMS, right? > > - Kris > > On Fri, May 11, 2012 at 12:31 AM, Jon Masamitsu wrote: > Damon, > > From inspection of the code I think that CMS does not do a > resizing of the heap after it does a System.gc(). That resizing > would be the code that adjusts the generations for > MinHeapFreeRatio / MaxHeapFreeRatio and returns > memory to the OS. > > I've created CR 7167923 to investigate this bug. > > Jon > > On 5/9/2012 2:34 AM, Damon Hart-Davis wrote: > > Hi, > > > > No, because when I explicitly call System.gc() I want it to try as hard as it can, even to the point of stopping the world... > > > > Rgds > > > > Damon > > > > > > On 9 May 2012, at 10:27, Jon Masamitsu wrote: > > > >> Damon, > >> > >> Do you turn on ExplicitGCInvokesConcurrent? > >> > >> Jon > >> > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From jon.masamitsu at oracle.com Thu May 10 10:35:11 2012 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Thu, 10 May 2012 10:35:11 -0700 Subject: MinHeapFreeRatio / MaxHeapFreeRatio In-Reply-To: References: <4FAA3867.2020808@oracle.com> <733AEBA7-7CE7-451E-A4A5-D053E65CE48F@exnet.com> <4FABED66.1080308@oracle.com> Message-ID: <4FABFC4F.1000705@oracle.com> On 5/10/2012 9:41 AM, Krystal Mok wrote: > Hi Jon, > > No, CMS doesn't do shrinking at all. The interface is there in the > generational GC framework, but the implementation in CMS is left empty. I > tried to implement shrinking in CMS once, but I ran out of time completing > it, and found out that the original demand from our internal customer was > supposed to be using ParallelGC, so I gave up. > > Resizing generations to meet MinHeapFreeRatio / MaxHeapFreeRatio means > shrinking has to be implemented in CMS, right? Not quite. Damon was using System.gc() to try to resize and System.gc() (not using the concurrent collection but the STW collection) uses the MarkSweep code. That code can use the compute_new_size() and shrink the heap. You're right about the CompactibleFreeListSpace used by CMS where the free space is on a free list. Live objects at the end of the generation make it impossible to shrink the generation. Jon > - Kris > > On Fri, May 11, 2012 at 12:31 AM, Jon Masamitsuwrote: > >> Damon, >> >> From inspection of the code I think that CMS does not do a >> resizing of the heap after it does a System.gc(). That resizing >> would be the code that adjusts the generations for >> MinHeapFreeRatio / MaxHeapFreeRatio and returns >> memory to the OS. >> >> I've created CR 7167923 to investigate this bug. >> >> Jon >> >> On 5/9/2012 2:34 AM, Damon Hart-Davis wrote: >>> Hi, >>> >>> No, because when I explicitly call System.gc() I want it to try as hard >> as it can, even to the point of stopping the world... >>> Rgds >>> >>> Damon >>> >>> >>> On 9 May 2012, at 10:27, Jon Masamitsu wrote: >>> >>>> Damon, >>>> >>>> Do you turn on ExplicitGCInvokesConcurrent? >>>> >>>> Jon >>>> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> From jon.masamitsu at oracle.com Thu May 10 10:40:58 2012 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Thu, 10 May 2012 10:40:58 -0700 Subject: MinHeapFreeRatio / MaxHeapFreeRatio In-Reply-To: <58DB97C4-92D0-478A-8D94-424596E1318A@exnet.com> References: <4FAA3867.2020808@oracle.com> <733AEBA7-7CE7-451E-A4A5-D053E65CE48F@exnet.com> <4FABED66.1080308@oracle.com> <58DB97C4-92D0-478A-8D94-424596E1318A@exnet.com> Message-ID: <4FABFDAA.9000008@oracle.com> Damon, It's been years since I looked but the Serial collector used to return space to the OS. At least on Solaris. I've heard complaints that it doesn't work on windows. I've watched the Serial collector work with jvisualvm and the heap size shrinks but I couldn't tell if the space got returned. I also saw that ParallelGC does not shrink back the heap the way I thought it should. I think there may be a bug in PSVirtualspace. Jon On 5/10/2012 10:00 AM, Damon Hart-Davis wrote: > Hi, > > FWIW, I don't recall (though I'm not totally sure) the JVM ever giving memory back to the OS even when I tried (for other reasons) the serial collector in SE 6 Embedded. > > Rgds > > Damon > > > On 10 May 2012, at 17:41, Krystal Mok wrote: > >> Hi Jon, >> >> No, CMS doesn't do shrinking at all. The interface is there in the generational GC framework, but the implementation in CMS is left empty. I tried to implement shrinking in CMS once, but I ran out of time completing it, and found out that the original demand from our internal customer was supposed to be using ParallelGC, so I gave up. >> >> Resizing generations to meet MinHeapFreeRatio / MaxHeapFreeRatio means shrinking has to be implemented in CMS, right? >> >> - Kris >> >> On Fri, May 11, 2012 at 12:31 AM, Jon Masamitsu wrote: >> Damon, >> >> From inspection of the code I think that CMS does not do a >> resizing of the heap after it does a System.gc(). That resizing >> would be the code that adjusts the generations for >> MinHeapFreeRatio / MaxHeapFreeRatio and returns >> memory to the OS. >> >> I've created CR 7167923 to investigate this bug. >> >> Jon >> >> On 5/9/2012 2:34 AM, Damon Hart-Davis wrote: >>> Hi, >>> >>> No, because when I explicitly call System.gc() I want it to try as hard as it can, even to the point of stopping the world... >>> >>> Rgds >>> >>> Damon >>> >>> >>> On 9 May 2012, at 10:27, Jon Masamitsu wrote: >>> >>>> Damon, >>>> >>>> Do you turn on ExplicitGCInvokesConcurrent? >>>> >>>> Jon >>>> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From dhd at exnet.com Thu May 10 10:49:56 2012 From: dhd at exnet.com (Damon Hart-Davis) Date: Thu, 10 May 2012 18:49:56 +0100 Subject: MinHeapFreeRatio / MaxHeapFreeRatio In-Reply-To: <4FABFC4F.1000705@oracle.com> References: <4FAA3867.2020808@oracle.com> <733AEBA7-7CE7-451E-A4A5-D053E65CE48F@exnet.com> <4FABED66.1080308@oracle.com> <4FABFC4F.1000705@oracle.com> Message-ID: <83753E3A-32A8-4E40-8CAC-34F2881A6495@exnet.com> Hi, I's be happy if the JVM gave memory back to the OS either following a STW triggered by System.gc() or a CMS failure, or opportunistically in the normal course of GC at least to the extent that trailing live objects would constrain it. The latter would presumably be slightly less toxic in terms of fruitless pauses, etc, if it could be done. Rgds Damon On 10 May 2012, at 18:35, Jon Masamitsu wrote: > > > On 5/10/2012 9:41 AM, Krystal Mok wrote: >> Hi Jon, >> >> No, CMS doesn't do shrinking at all. The interface is there in the >> generational GC framework, but the implementation in CMS is left empty. I >> tried to implement shrinking in CMS once, but I ran out of time completing >> it, and found out that the original demand from our internal customer was >> supposed to be using ParallelGC, so I gave up. >> >> Resizing generations to meet MinHeapFreeRatio / MaxHeapFreeRatio means >> shrinking has to be implemented in CMS, right? > > Not quite. Damon was using System.gc() to try to resize and > System.gc() (not using the concurrent > collection but the STW collection) uses the MarkSweep code. That code > can use the compute_new_size() > and shrink the heap. > > You're right about the CompactibleFreeListSpace used by CMS where the > free space is on > a free list. Live objects at the end of the generation make it > impossible to shrink the > generation. > > Jon >> - Kris >> >> On Fri, May 11, 2012 at 12:31 AM, Jon Masamitsuwrote: >> >>> Damon, >>> >>> From inspection of the code I think that CMS does not do a >>> resizing of the heap after it does a System.gc(). That resizing >>> would be the code that adjusts the generations for >>> MinHeapFreeRatio / MaxHeapFreeRatio and returns >>> memory to the OS. >>> >>> I've created CR 7167923 to investigate this bug. >>> >>> Jon >>> >>> On 5/9/2012 2:34 AM, Damon Hart-Davis wrote: >>>> Hi, >>>> >>>> No, because when I explicitly call System.gc() I want it to try as hard >>> as it can, even to the point of stopping the world... >>>> Rgds >>>> >>>> Damon >>>> >>>> >>>> On 9 May 2012, at 10:27, Jon Masamitsu wrote: >>>> >>>>> Damon, >>>>> >>>>> Do you turn on ExplicitGCInvokesConcurrent? >>>>> >>>>> Jon >>>>> >>> _______________________________________________ >>> hotspot-gc-use mailing list >>> hotspot-gc-use at openjdk.java.net >>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>> > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > From holger.hoffstaette at googlemail.com Thu May 10 11:03:35 2012 From: holger.hoffstaette at googlemail.com (=?ISO-8859-1?Q?Holger_Hoffst=E4tte?=) Date: Thu, 10 May 2012 20:03:35 +0200 Subject: MinHeapFreeRatio / MaxHeapFreeRatio In-Reply-To: <4FABFDAA.9000008@oracle.com> References: <4FAA3867.2020808@oracle.com> <733AEBA7-7CE7-451E-A4A5-D053E65CE48F@exnet.com> <4FABED66.1080308@oracle.com> <58DB97C4-92D0-478A-8D94-424596E1318A@exnet.com> <4FABFDAA.9000008@oracle.com> Message-ID: <4FAC02F7.4040703@googlemail.com> On 10.05.2012 19:40, Jon Masamitsu wrote: > It's been years since I looked but the Serial collector used to > return space to the OS. At least on Solaris. I've heard complaints > that it doesn't work on windows. I can assure you that it does; the settings work exactly as expected, and heap is returned to the OS. :) cheers Holger From dhd at exnet.com Thu May 10 11:33:14 2012 From: dhd at exnet.com (Damon Hart-Davis) Date: Thu, 10 May 2012 19:33:14 +0100 Subject: MinHeapFreeRatio / MaxHeapFreeRatio In-Reply-To: <4FAC02F7.4040703@googlemail.com> References: <4FAA3867.2020808@oracle.com> <733AEBA7-7CE7-451E-A4A5-D053E65CE48F@exnet.com> <4FABED66.1080308@oracle.com> <58DB97C4-92D0-478A-8D94-424596E1318A@exnet.com> <4FABFDAA.9000008@oracle.com> <4FAC02F7.4040703@googlemail.com> Message-ID: Hi, Well, it's good to know that it *can* work! Rgds Damon On 10 May 2012, at 19:03, Holger Hoffst?tte wrote: > On 10.05.2012 19:40, Jon Masamitsu wrote: >> It's been years since I looked but the Serial collector used to >> return space to the OS. At least on Solaris. I've heard complaints >> that it doesn't work on windows. > > I can assure you that it does; the settings work exactly as expected, and > heap is returned to the OS. :) > > cheers > Holger > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > From rednaxelafx at gmail.com Thu May 10 21:24:24 2012 From: rednaxelafx at gmail.com (Krystal Mok) Date: Fri, 11 May 2012 12:24:24 +0800 Subject: MinHeapFreeRatio / MaxHeapFreeRatio In-Reply-To: <4FABFC4F.1000705@oracle.com> References: <4FAA3867.2020808@oracle.com> <733AEBA7-7CE7-451E-A4A5-D053E65CE48F@exnet.com> <4FABED66.1080308@oracle.com> <4FABFC4F.1000705@oracle.com> Message-ID: On Fri, May 11, 2012 at 1:35 AM, Jon Masamitsu wrote: > > Not quite. Damon was using System.gc() to try to resize and System.gc() > (not using the concurrent > collection but the STW collection) uses the MarkSweep code. That code can > use the compute_new_size() > and shrink the heap. > > Oops, should have made myself clearer. I meant shrinking is part of the generational GC framework's interface, but the implementation in CMS generation is empty: void ConcurrentMarkSweepGeneration::shrink_by(size_t bytes) { assert_locked_or_safepoint(Heap_lock); assert_lock_strong(freelistLock()); // XXX Fix when compaction is implemented. warning("Shrinking of CMS not yet implemented"); return; } And ConcurrentMarkSweepGeneration::compute_new_size() currently handles expansion but not shrinking. So even if one were to resize the space at the end of a STW full GC, the CMS old / perm gen still could handle shrinking. In fact, my original motivation to implement shrinking in CMS (generation, not concurrent collection) was almost the same as what Damon is asking for, that is to release memory back to OS after a full GC. - Kris > You're right about the CompactibleFreeListSpace used by CMS where the free > space is on > a free list. Live objects at the end of the generation make it impossible > to shrink the > generation. > > Jon > > - Kris >> >> On Fri, May 11, 2012 at 12:31 AM, Jon Masamitsu> oracle.com >wrote: >> >> Damon, >>> >>> From inspection of the code I think that CMS does not do a >>> resizing of the heap after it does a System.gc(). That resizing >>> would be the code that adjusts the generations for >>> MinHeapFreeRatio / MaxHeapFreeRatio and returns >>> memory to the OS. >>> >>> I've created CR 7167923 to investigate this bug. >>> >>> Jon >>> >>> On 5/9/2012 2:34 AM, Damon Hart-Davis wrote: >>> >>>> Hi, >>>> >>>> No, because when I explicitly call System.gc() I want it to try as hard >>>> >>> as it can, even to the point of stopping the world... >>> >>>> Rgds >>>> >>>> Damon >>>> >>>> >>>> On 9 May 2012, at 10:27, Jon Masamitsu wrote: >>>> >>>> Damon, >>>>> >>>>> Do you turn on ExplicitGCInvokesConcurrent? >>>>> >>>>> Jon >>>>> >>>>> ______________________________**_________________ >>> hotspot-gc-use mailing list >>> hotspot-gc-use at openjdk.java.**net >>> http://mail.openjdk.java.net/**mailman/listinfo/hotspot-gc-**use >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120511/7007a183/attachment.html From ysr1729 at gmail.com Thu May 10 23:05:13 2012 From: ysr1729 at gmail.com (Srinivas Ramakrishna) Date: Thu, 10 May 2012 23:05:13 -0700 Subject: MinHeapFreeRatio / MaxHeapFreeRatio In-Reply-To: References: <4FAA3867.2020808@oracle.com> <733AEBA7-7CE7-451E-A4A5-D053E65CE48F@exnet.com> <4FABED66.1080308@oracle.com> <4FABFC4F.1000705@oracle.com> Message-ID: Jon, Kris -- I agree that there is an easily fixed bug here which could be fixed by (more or less) having the CMS generation basically do after a stop-world collection what TenuredGeneration does, modulo appropriate parameters (CMS needs more headroom etc. which are really different free ratio defaults). Looks like the expansion/shrinking/resizing really belonged to a Policy class which would take a generation as a parameter and resize it appropriately based on the state of the generation and policy parameters (which could be obtained from the generation). That would allow for sharing of the code without the kind of duplication that we have today with small modifications at various places... -- ramki On Thu, May 10, 2012 at 9:24 PM, Krystal Mok wrote: > > On Fri, May 11, 2012 at 1:35 AM, Jon Masamitsu wrote: > >> >> Not quite. Damon was using System.gc() to try to resize and System.gc() >> (not using the concurrent >> collection but the STW collection) uses the MarkSweep code. That code >> can use the compute_new_size() >> and shrink the heap. >> >> Oops, should have made myself clearer. > I meant shrinking is part of the generational GC framework's interface, > but the implementation in CMS generation is empty: > > void ConcurrentMarkSweepGeneration::shrink_by(size_t bytes) { > assert_locked_or_safepoint(Heap_lock); > assert_lock_strong(freelistLock()); > // XXX Fix when compaction is implemented. > warning("Shrinking of CMS not yet implemented"); > return; > } > > And ConcurrentMarkSweepGeneration::compute_new_size() currently handles > expansion but not shrinking. > > So even if one were to resize the space at the end of a STW full GC, the > CMS old / perm gen still could handle shrinking. > In fact, my original motivation to implement shrinking in CMS (generation, > not concurrent collection) was almost the same as what Damon is asking for, > that is to release memory back to OS after a full GC. > > - Kris > > >> You're right about the CompactibleFreeListSpace used by CMS where the >> free space is on >> a free list. Live objects at the end of the generation make it >> impossible to shrink the >> generation. >> >> Jon >> >> - Kris >>> >>> On Fri, May 11, 2012 at 12:31 AM, Jon Masamitsu>> oracle.com >wrote: >>> >>> Damon, >>>> >>>> From inspection of the code I think that CMS does not do a >>>> resizing of the heap after it does a System.gc(). That resizing >>>> would be the code that adjusts the generations for >>>> MinHeapFreeRatio / MaxHeapFreeRatio and returns >>>> memory to the OS. >>>> >>>> I've created CR 7167923 to investigate this bug. >>>> >>>> Jon >>>> >>>> On 5/9/2012 2:34 AM, Damon Hart-Davis wrote: >>>> >>>>> Hi, >>>>> >>>>> No, because when I explicitly call System.gc() I want it to try as hard >>>>> >>>> as it can, even to the point of stopping the world... >>>> >>>>> Rgds >>>>> >>>>> Damon >>>>> >>>>> >>>>> On 9 May 2012, at 10:27, Jon Masamitsu wrote: >>>>> >>>>> Damon, >>>>>> >>>>>> Do you turn on ExplicitGCInvokesConcurrent? >>>>>> >>>>>> Jon >>>>>> >>>>>> ______________________________**_________________ >>>> hotspot-gc-use mailing list >>>> hotspot-gc-use at openjdk.java.**net >>>> http://mail.openjdk.java.net/**mailman/listinfo/hotspot-gc-**use >>>> >>>> > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120510/8c3a7c8c/attachment.html From jon.masamitsu at oracle.com Fri May 11 10:16:53 2012 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Fri, 11 May 2012 10:16:53 -0700 Subject: MinHeapFreeRatio / MaxHeapFreeRatio In-Reply-To: References: <4FAA3867.2020808@oracle.com> <733AEBA7-7CE7-451E-A4A5-D053E65CE48F@exnet.com> <4FABED66.1080308@oracle.com> <4FABFC4F.1000705@oracle.com> Message-ID: <4FAD4985.1030208@oracle.com> Ramki, Do you remember if there was originally a call to do_collection() in the CMS do_compaction_work()? It currently has a call to GenMarkSweep::invoke_at_safepoint() to do the collection. For shrinking the CMS generation I had in mind looking at the free chunk at the end of the generation and shrinking the generation according to policy (the FreeRatio's) and the size of that free chunk. After a mark-sweep-compact the free chunk at the end of the generation would be everything free. So in this case the current interfaces would work if CMS::compute_new_size() were just called at the end of do_compaction_work(). And if CMS::compute_new_size() were fixed. Do you believe? Jon On 05/10/12 23:05, Srinivas Ramakrishna wrote: > Jon, Kris -- I agree that there is an easily fixed bug here which > could be fixed by (more or less) having the CMS generation > basically do after a stop-world collection what TenuredGeneration > does, modulo appropriate parameters > (CMS needs more headroom etc. which are really different free ratio > defaults). Looks like the > expansion/shrinking/resizing really belonged to a Policy class which > would take a generation > as a parameter and resize it appropriately based on the state of the > generation and policy > parameters (which could be obtained from the generation). That would > allow for sharing of > the code without the kind of duplication that we have today with small > modifications at various > places... > > -- ramki > > On Thu, May 10, 2012 at 9:24 PM, Krystal Mok > wrote: > > > On Fri, May 11, 2012 at 1:35 AM, Jon Masamitsu > > wrote: > > > Not quite. Damon was using System.gc() to try to resize and > System.gc() (not using the concurrent > collection but the STW collection) uses the MarkSweep code. > That code can use the compute_new_size() > and shrink the heap. > > Oops, should have made myself clearer. > I meant shrinking is part of the generational GC framework's > interface, but the implementation in CMS generation is empty: > > void ConcurrentMarkSweepGeneration::shrink_by(size_t bytes) { > assert_locked_or_safepoint(Heap_lock); > assert_lock_strong(freelistLock()); > // XXX Fix when compaction is implemented. > warning("Shrinking of CMS not yet implemented"); > return; > } > > And ConcurrentMarkSweepGeneration::compute_new_size() currently > handles expansion but not shrinking. > > So even if one were to resize the space at the end of a STW full > GC, the CMS old / perm gen still could handle shrinking. > In fact, my original motivation to implement shrinking in CMS > (generation, not concurrent collection) was almost the same as > what Damon is asking for, that is to release memory back to OS > after a full GC. > > - Kris > > You're right about the CompactibleFreeListSpace used by CMS > where the free space is on > a free list. Live objects at the end of the generation make > it impossible to shrink the > generation. > > Jon > > - Kris > > On Fri, May 11, 2012 at 12:31 AM, Jon > Masamitsu >wrote: > > Damon, > > From inspection of the code I think that CMS does not > do a > resizing of the heap after it does a System.gc(). > That resizing > would be the code that adjusts the generations for > MinHeapFreeRatio / MaxHeapFreeRatio and returns > memory to the OS. > > I've created CR 7167923 to investigate this bug. > > Jon > > On 5/9/2012 2:34 AM, Damon Hart-Davis wrote: > > Hi, > > No, because when I explicitly call System.gc() I > want it to try as hard > > as it can, even to the point of stopping the world... > > Rgds > > Damon > > > On 9 May 2012, at 10:27, Jon Masamitsu wrote: > > Damon, > > Do you turn on ExplicitGCInvokesConcurrent? > > Jon > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120511/9fc3d643/attachment-0001.html From ysr1729 at gmail.com Fri May 11 10:49:18 2012 From: ysr1729 at gmail.com (Srinivas Ramakrishna) Date: Fri, 11 May 2012 10:49:18 -0700 Subject: MinHeapFreeRatio / MaxHeapFreeRatio In-Reply-To: <4FAD4985.1030208@oracle.com> References: <4FAA3867.2020808@oracle.com> <733AEBA7-7CE7-451E-A4A5-D053E65CE48F@exnet.com> <4FABED66.1080308@oracle.com> <4FABFC4F.1000705@oracle.com> <4FAD4985.1030208@oracle.com> Message-ID: On Fri, May 11, 2012 at 10:16 AM, Jon Masamitsu wrote: > Ramki, > > Do you remember if there was originally a call to > do_collection() in the CMS do_compaction_work()? > It currently has a call to GenMarkSweep::invoke_at_safepoint() > to do the collection. > :-) Sorry i can't remember off-hand, but if i were to grok through the hg history i might remember when and why that was done... I'll try and take a look at the code over the weekend... > > For shrinking the CMS generation I had in mind looking > at the free chunk at the end of the generation and shrinking > the generation according to policy (the FreeRatio's) and > the size of that free chunk. After a mark-sweep-compact > the free chunk at the end of the generation would be > everything free. So in this case the current interfaces > would work if CMS::compute_new_size() were just > called at the end of do_compaction_work(). And if > CMS::compute_new_size() were fixed. Do you believe? > Completely! I think this makes sense. BTW you may have some of the "last big chunk" code in the adaptive sizing work for CMS that was put away in the attic a while ago (it might need the cobwebs dusted off as some of the original assumptions and interfaces may have changed.) -- ramki > Jon > > > > On 05/10/12 23:05, Srinivas Ramakrishna wrote: > > Jon, Kris -- I agree that there is an easily fixed bug here which could be > fixed by (more or less) having the CMS generation > basically do after a stop-world collection what TenuredGeneration does, > modulo appropriate parameters > (CMS needs more headroom etc. which are really different free ratio > defaults). Looks like the > expansion/shrinking/resizing really belonged to a Policy class which would > take a generation > as a parameter and resize it appropriately based on the state of the > generation and policy > parameters (which could be obtained from the generation). That would allow > for sharing of > the code without the kind of duplication that we have today with small > modifications at various > places... > > -- ramki > > On Thu, May 10, 2012 at 9:24 PM, Krystal Mok wrote: > >> >> On Fri, May 11, 2012 at 1:35 AM, Jon Masamitsu > > wrote: >> >>> >>> Not quite. Damon was using System.gc() to try to resize and >>> System.gc() (not using the concurrent >>> collection but the STW collection) uses the MarkSweep code. That code >>> can use the compute_new_size() >>> and shrink the heap. >>> >>> Oops, should have made myself clearer. >> I meant shrinking is part of the generational GC framework's interface, >> but the implementation in CMS generation is empty: >> >> void ConcurrentMarkSweepGeneration::shrink_by(size_t bytes) { >> assert_locked_or_safepoint(Heap_lock); >> assert_lock_strong(freelistLock()); >> // XXX Fix when compaction is implemented. >> warning("Shrinking of CMS not yet implemented"); >> return; >> } >> >> And ConcurrentMarkSweepGeneration::compute_new_size() currently handles >> expansion but not shrinking. >> >> So even if one were to resize the space at the end of a STW full GC, >> the CMS old / perm gen still could handle shrinking. >> In fact, my original motivation to implement shrinking in CMS >> (generation, not concurrent collection) was almost the same as what Damon >> is asking for, that is to release memory back to OS after a full GC. >> >> - Kris >> >> >>> You're right about the CompactibleFreeListSpace used by CMS where the >>> free space is on >>> a free list. Live objects at the end of the generation make it >>> impossible to shrink the >>> generation. >>> >>> Jon >>> >>> - Kris >>>> >>>> On Fri, May 11, 2012 at 12:31 AM, Jon Masamitsu< >>>> jon.masamitsu at oracle.com>wrote: >>>> >>>> Damon, >>>>> >>>>> From inspection of the code I think that CMS does not do a >>>>> resizing of the heap after it does a System.gc(). That resizing >>>>> would be the code that adjusts the generations for >>>>> MinHeapFreeRatio / MaxHeapFreeRatio and returns >>>>> memory to the OS. >>>>> >>>>> I've created CR 7167923 to investigate this bug. >>>>> >>>>> Jon >>>>> >>>>> On 5/9/2012 2:34 AM, Damon Hart-Davis wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> No, because when I explicitly call System.gc() I want it to try as >>>>>> hard >>>>>> >>>>> as it can, even to the point of stopping the world... >>>>> >>>>>> Rgds >>>>>> >>>>>> Damon >>>>>> >>>>>> >>>>>> On 9 May 2012, at 10:27, Jon Masamitsu wrote: >>>>>> >>>>>> Damon, >>>>>>> >>>>>>> Do you turn on ExplicitGCInvokesConcurrent? >>>>>>> >>>>>>> Jon >>>>>>> >>>>>>> _______________________________________________ >>>>> hotspot-gc-use mailing list >>>>> hotspot-gc-use at openjdk.java.net >>>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>>>> >>>>> >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120511/230ec11b/attachment.html From jon.masamitsu at oracle.com Fri May 11 16:32:10 2012 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Fri, 11 May 2012 16:32:10 -0700 Subject: MinHeapFreeRatio / MaxHeapFreeRatio In-Reply-To: References: <4FAA3867.2020808@oracle.com> <733AEBA7-7CE7-451E-A4A5-D053E65CE48F@exnet.com> <4FABED66.1080308@oracle.com> <4FABFC4F.1000705@oracle.com> <4FAD4985.1030208@oracle.com> Message-ID: <4FADA17A.7000907@oracle.com> On 5/11/2012 10:49 AM, Srinivas Ramakrishna wrote: > On Fri, May 11, 2012 at 10:16 AM, Jon Masamitsuwrote: > >> Ramki, >> >> Do you remember if there was originally a call to >> do_collection() in the CMS do_compaction_work()? >> It currently has a call to GenMarkSweep::invoke_at_safepoint() >> to do the collection. >> > :-) Sorry i can't remember off-hand, but if i were to grok through the hg > history i might remember > when and why that was done... I'll try and take a look at the code over the > weekend... Not worth groveling through hg history. I was just curious. My memory says that we used to call do_collection() and it did the resizing. Then, as you say, we realized the resizing done by TenuredGeneration::compute_new_size() wasn't what we really wanted. But really, that is neither here nor there. Jon > >> For shrinking the CMS generation I had in mind looking >> at the free chunk at the end of the generation and shrinking >> the generation according to policy (the FreeRatio's) and >> the size of that free chunk. After a mark-sweep-compact >> the free chunk at the end of the generation would be >> everything free. So in this case the current interfaces >> would work if CMS::compute_new_size() were just >> called at the end of do_compaction_work(). And if >> CMS::compute_new_size() were fixed. Do you believe? >> > Completely! I think this makes sense. BTW you may have some of the > "last big chunk" code in the adaptive sizing work for CMS that was put away > in the attic a while ago (it might need the cobwebs dusted off as some of > the > original assumptions and interfaces may have changed.) > > -- ramki > > >> Jon >> >> >> >> On 05/10/12 23:05, Srinivas Ramakrishna wrote: >> >> Jon, Kris -- I agree that there is an easily fixed bug here which could be >> fixed by (more or less) having the CMS generation >> basically do after a stop-world collection what TenuredGeneration does, >> modulo appropriate parameters >> (CMS needs more headroom etc. which are really different free ratio >> defaults). Looks like the >> expansion/shrinking/resizing really belonged to a Policy class which would >> take a generation >> as a parameter and resize it appropriately based on the state of the >> generation and policy >> parameters (which could be obtained from the generation). That would allow >> for sharing of >> the code without the kind of duplication that we have today with small >> modifications at various >> places... >> >> -- ramki >> >> On Thu, May 10, 2012 at 9:24 PM, Krystal Mokwrote: >> >>> On Fri, May 11, 2012 at 1:35 AM, Jon Masamitsu>>> wrote: >>>> Not quite. Damon was using System.gc() to try to resize and >>>> System.gc() (not using the concurrent >>>> collection but the STW collection) uses the MarkSweep code. That code >>>> can use the compute_new_size() >>>> and shrink the heap. >>>> >>>> Oops, should have made myself clearer. >>> I meant shrinking is part of the generational GC framework's interface, >>> but the implementation in CMS generation is empty: >>> >>> void ConcurrentMarkSweepGeneration::shrink_by(size_t bytes) { >>> assert_locked_or_safepoint(Heap_lock); >>> assert_lock_strong(freelistLock()); >>> // XXX Fix when compaction is implemented. >>> warning("Shrinking of CMS not yet implemented"); >>> return; >>> } >>> >>> And ConcurrentMarkSweepGeneration::compute_new_size() currently handles >>> expansion but not shrinking. >>> >>> So even if one were to resize the space at the end of a STW full GC, >>> the CMS old / perm gen still could handle shrinking. >>> In fact, my original motivation to implement shrinking in CMS >>> (generation, not concurrent collection) was almost the same as what Damon >>> is asking for, that is to release memory back to OS after a full GC. >>> >>> - Kris >>> >>> >>>> You're right about the CompactibleFreeListSpace used by CMS where the >>>> free space is on >>>> a free list. Live objects at the end of the generation make it >>>> impossible to shrink the >>>> generation. >>>> >>>> Jon >>>> >>>> - Kris >>>>> On Fri, May 11, 2012 at 12:31 AM, Jon Masamitsu< >>>>> jon.masamitsu at oracle.com>wrote: >>>>> >>>>> Damon, >>>>>> From inspection of the code I think that CMS does not do a >>>>>> resizing of the heap after it does a System.gc(). That resizing >>>>>> would be the code that adjusts the generations for >>>>>> MinHeapFreeRatio / MaxHeapFreeRatio and returns >>>>>> memory to the OS. >>>>>> >>>>>> I've created CR 7167923 to investigate this bug. >>>>>> >>>>>> Jon >>>>>> >>>>>> On 5/9/2012 2:34 AM, Damon Hart-Davis wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> No, because when I explicitly call System.gc() I want it to try as >>>>>>> hard >>>>>>> >>>>>> as it can, even to the point of stopping the world... >>>>>> >>>>>>> Rgds >>>>>>> >>>>>>> Damon >>>>>>> >>>>>>> >>>>>>> On 9 May 2012, at 10:27, Jon Masamitsu wrote: >>>>>>> >>>>>>> Damon, >>>>>>>> Do you turn on ExplicitGCInvokesConcurrent? >>>>>>>> >>>>>>>> Jon >>>>>>>> >>>>>>>> _______________________________________________ >>>>>> hotspot-gc-use mailing list >>>>>> hotspot-gc-use at openjdk.java.net >>>>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>>>>> >>>>>> >>> _______________________________________________ >>> hotspot-gc-use mailing list >>> hotspot-gc-use at openjdk.java.net >>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>> >>> From dhd at exnet.com Tue May 15 12:47:13 2012 From: dhd at exnet.com (Damon Hart-Davis) Date: Tue, 15 May 2012 20:47:13 +0100 Subject: MinHeapFreeRatio / MaxHeapFreeRatio Message-ID: Hi, Is there anything I can do to help things along? I know that nothing will happen for a while, but still, I hate just to let things slip away once something has had your attention! Rgds Damon From jon.masamitsu at oracle.com Wed May 16 10:14:48 2012 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Wed, 16 May 2012 10:14:48 -0700 Subject: MinHeapFreeRatio / MaxHeapFreeRatio In-Reply-To: References: Message-ID: <4FB3E088.4000707@oracle.com> Damon, Do you want to build a VM with the change and see if it does what you want? Jon On 5/15/2012 12:47 PM, Damon Hart-Davis wrote: > Hi, > > Is there anything I can do to help things along? I know that nothing will happen for a while, but still, I hate just to let things slip away once something has had your attention! > > Rgds > > Damon > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From dhd at exnet.com Wed May 16 13:15:52 2012 From: dhd at exnet.com (Damon Hart-Davis) Date: Wed, 16 May 2012 21:15:52 +0100 Subject: MinHeapFreeRatio / MaxHeapFreeRatio In-Reply-To: <4FB3E088.4000707@oracle.com> References: <4FB3E088.4000707@oracle.com> Message-ID: Hi, I have none of the machinery to do a build (I have never built OpenJDK) and my target machine (SheevaPlug) is too small to run a build without turning off production services, so I could try but it would probably take me days to get started and fire up a whole new machine so as not to disturb the old one! Also it was suggested to me by a colleague of yours at Oracle (hush...!) that OpenJDK7 doesn't actually run very well on ARMv5, which is why I'm actually running the Embedded JRE on there. So in other words I don't think I can help very much with building what I'm actually using, at least not quickly, through I'm very happy to try out a binary or three and report behaviour if that is any use! And if I just have to wait, I'm happy to... B^> Rgds Damon On 16 May 2012, at 18:14, Jon Masamitsu wrote: > Damon, > > Do you want to build a VM with the change and see if > it does what you want? > > Jon > > On 5/15/2012 12:47 PM, Damon Hart-Davis wrote: >> Hi, >> >> Is there anything I can do to help things along? I know that nothing will happen for a while, but still, I hate just to let things slip away once something has had your attention! >> >> Rgds >> >> Damon >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > From carfield at carfield.com.hk Mon May 21 08:18:28 2012 From: carfield at carfield.com.hk (Carfield Yim) Date: Mon, 21 May 2012 23:18:28 +0800 Subject: MinHeapFreeRatio / MaxHeapFreeRatio In-Reply-To: References: Message-ID: HI, I have to include -XX:+UseParNewGC if I need MaxHeapFreeRatio working, maybe you can try on it. On Wed, May 9, 2012 at 1:46 PM, Damon Hart-Davis wrote: > Hi, > > First time on this list, so hello! ?I see a few familiar names from days of yore! > > I am running my favourite (Tomcat/Web) app in a very constrained memory environment (a SheevaPlug) along with all the usual Internet server junk (sendmail, ?ntp, sshd, etc) and I am aiming to squeeze it into the even tighter space of a Raspberry Pi in due course. > > As such, I want the JVM to give memory back to the OS whenever possible. ?In code I set a target to keep about 25% of the heap free, and so for example I stop cacheing some stuff when below that, and cache more vigorously above that. > > Here are some relevant options: > > CATALINA_OPTS="-Xmx100m -Xms64m" > # Cap size of non-(main-)heap components. > CATALINA_OPTS="$CATALINA_OPTS -XX:MaxPermSize=48m" > # Trim thread stack size. > CATALINA_OPTS="$CATALINA_OPTS -Xss256k" > # Keep the new generation well within the target 25% free... > CATALINA_OPTS="$CATALINA_OPTS -XX:NewRatio=5" > # Be aggressive about giving memory back to the system above target 25% free. > CATALINA_OPTS="$CATALINA_OPTS -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=26" > # Run in incremental GC mode to minimise pauses. > CATALINA_OPTS="$CATALINA_OPTS -Xincgc" > > I see some evidence that the free ratios are being observed by when the heap is expanded, but I have never seen any variant of JDK 6 or 7, including the Oracle embedded 6, actually give memory back to the OS with this app. > > Am I still doing something wrong? ?B^> > > Rgds > > Damon > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From dhd at exnet.com Mon May 21 08:23:38 2012 From: dhd at exnet.com (Damon Hart-Davis) Date: Mon, 21 May 2012 16:23:38 +0100 Subject: MinHeapFreeRatio / MaxHeapFreeRatio In-Reply-To: References: Message-ID: <34EA45B4-7423-4C37-AAA1-1C9437AAA3A0@exnet.com> Hi, Interesting! Is that likely to be a good idea on a uniprocessor, and have you been using it with concgc/incgc? Rgds Damon On 21 May 2012, at 16:18, Carfield Yim wrote: > HI, I have to include -XX:+UseParNewGC if I need MaxHeapFreeRatio > working, maybe you can try on it. > > On Wed, May 9, 2012 at 1:46 PM, Damon Hart-Davis wrote: >> Hi, >> >> First time on this list, so hello! I see a few familiar names from days of yore! >> >> I am running my favourite (Tomcat/Web) app in a very constrained memory environment (a SheevaPlug) along with all the usual Internet server junk (sendmail, ntp, sshd, etc) and I am aiming to squeeze it into the even tighter space of a Raspberry Pi in due course. >> >> As such, I want the JVM to give memory back to the OS whenever possible. In code I set a target to keep about 25% of the heap free, and so for example I stop cacheing some stuff when below that, and cache more vigorously above that. >> >> Here are some relevant options: >> >> CATALINA_OPTS="-Xmx100m -Xms64m" >> # Cap size of non-(main-)heap components. >> CATALINA_OPTS="$CATALINA_OPTS -XX:MaxPermSize=48m" >> # Trim thread stack size. >> CATALINA_OPTS="$CATALINA_OPTS -Xss256k" >> # Keep the new generation well within the target 25% free... >> CATALINA_OPTS="$CATALINA_OPTS -XX:NewRatio=5" >> # Be aggressive about giving memory back to the system above target 25% free. >> CATALINA_OPTS="$CATALINA_OPTS -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=26" >> # Run in incremental GC mode to minimise pauses. >> CATALINA_OPTS="$CATALINA_OPTS -Xincgc" >> >> I see some evidence that the free ratios are being observed by when the heap is expanded, but I have never seen any variant of JDK 6 or 7, including the Oracle embedded 6, actually give memory back to the OS with this app. >> >> Am I still doing something wrong? B^> >> >> Rgds >> >> Damon >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > From gbowyer at fastmail.co.uk Tue May 22 12:05:53 2012 From: gbowyer at fastmail.co.uk (Greg Bowyer) Date: Tue, 22 May 2012 12:05:53 -0700 Subject: Stupid question Message-ID: <4FBBE391.3040600@fastmail.co.uk> Not sure if this is the right venue for this, or if I am insane I have been playing with the openjdk code with a view to lock the heap in memory (or at least suggest to the OS that it wants to be locked in memory). My use case is for java processes that are greedy in memory and typically have a virtual size larger than physical ram. This is not where an end user allocates a java heap beyond ram but rather things like Lucene / Cassandra, where typically the JVM heap is large but limited to say 1/4 of the total physical ram and the rest of the process virtual size is taken up with mmap()'d files. There are java projects that currently do this with a call out via JNA / JNI to mlockall() Asking the OS to use the MAP_LOCKED flag in the mmap calls in os_linux.cpp effectively does an mlock / mlockall which I think means that when the OS chooses pages to page out; then it should (for some measure of should) avoid paging out the JVM heap. This means that horrors between CMS and paging do not cause hateful pauses (hopefully) The thing I cant understand is that when I start a new VM with this code (attached) it appears to lock the full size of the heap (-Xmx) (even though the given memory is not used by the VM), this seems to make the entire space resident. any ideas ? -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: locked_heap.patch Url: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120522/bdb4aa8c/locked_heap.patch From vitalyd at gmail.com Tue May 22 12:27:08 2012 From: vitalyd at gmail.com (Vitaly Davidovich) Date: Tue, 22 May 2012 15:27:08 -0400 Subject: Stupid question In-Reply-To: <4FBBE391.3040600@fastmail.co.uk> References: <4FBBE391.3040600@fastmail.co.uk> Message-ID: Hi Greg, As an aside, in os::commit_memory, I think the tertiary condition that sets the flags is inverted - you want MAP_LOCKED if LockHeapInMemory is set, I believe. As to your question, what args are being passed to the above function in your run? I take it -Xms is less than Xmx in your trial? Sent from my phone On May 22, 2012 3:07 PM, "Greg Bowyer" wrote: > Not sure if this is the right venue for this, or if I am insane > > I have been playing with the openjdk code with a view to lock the heap in > memory (or at least suggest to the OS that it wants to be locked in memory). > > My use case is for java processes that are greedy in memory and typically > have a virtual size larger than physical ram. This is not where an end user > allocates a java heap beyond ram but rather things like Lucene / Cassandra, > where typically the JVM heap is large but limited to say 1/4 of the total > physical ram and the rest of the process virtual size is taken up with > mmap()'d files. > > There are java projects that currently do this with a call out via JNA / > JNI to mlockall() > > Asking the OS to use the MAP_LOCKED flag in the mmap calls in os_linux.cpp > effectively does an mlock / mlockall which I think means that when the OS > chooses pages to page out; then it should (for some measure of should) > avoid paging out the JVM heap. > > This means that horrors between CMS and paging do not cause hateful pauses > (hopefully) > > The thing I cant understand is that when I start a new VM with this code > (attached) it appears to lock the full size of the heap (-Xmx) (even though > the given memory is not used by the VM), this seems to make the entire > space resident. > > any ideas ? > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120522/1c544755/attachment.html From gbowyer at fastmail.co.uk Tue May 22 13:08:44 2012 From: gbowyer at fastmail.co.uk (Greg Bowyer) Date: Tue, 22 May 2012 13:08:44 -0700 Subject: Stupid question In-Reply-To: References: <4FBBE391.3040600@fastmail.co.uk> Message-ID: <4FBBF24C.1070408@fastmail.co.uk> Yes thats basically it java -Xms512m -Xmx9g test On 22/05/12 12:27, Vitaly Davidovich wrote: > > Hi Greg, > > As an aside, in os::commit_memory, I think the tertiary condition that > sets the flags is inverted - you want MAP_LOCKED if LockHeapInMemory > is set, I believe. > > As to your question, what args are being passed to the above function > in your run? I take it -Xms is less than Xmx in your trial? > > Sent from my phone > > On May 22, 2012 3:07 PM, "Greg Bowyer" > wrote: > > Not sure if this is the right venue for this, or if I am insane > > I have been playing with the openjdk code with a view to lock the > heap in memory (or at least suggest to the OS that it wants to be > locked in memory). > > My use case is for java processes that are greedy in memory and > typically have a virtual size larger than physical ram. This is > not where an end user allocates a java heap beyond ram but rather > things like Lucene / Cassandra, where typically the JVM heap is > large but limited to say 1/4 of the total physical ram and the > rest of the process virtual size is taken up with mmap()'d files. > > There are java projects that currently do this with a call out via > JNA / JNI to mlockall() > > Asking the OS to use the MAP_LOCKED flag in the mmap calls in > os_linux.cpp effectively does an mlock / mlockall which I think > means that when the OS chooses pages to page out; then it should > (for some measure of should) avoid paging out the JVM heap. > > This means that horrors between CMS and paging do not cause > hateful pauses (hopefully) > > The thing I cant understand is that when I start a new VM with > this code (attached) it appears to lock the full size of the heap > (-Xmx) (even though the given memory is not used by the VM), this > seems to make the entire space resident. > > any ideas ? > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120522/95d4ce95/attachment.html From vitalyd at gmail.com Tue May 22 13:35:53 2012 From: vitalyd at gmail.com (Vitaly Davidovich) Date: Tue, 22 May 2012 16:35:53 -0400 Subject: Stupid question In-Reply-To: <4FBBF24C.1070408@fastmail.co.uk> References: <4FBBE391.3040600@fastmail.co.uk> <4FBBF24C.1070408@fastmail.co.uk> Message-ID: can you add a print statement to commit_memory() to see what arguments are being passed to it? I'm sure someone on this mailing list will have a better suggestion, but that's what I'd try just to establish that some code is indeed trying to expand the heap to its max allotted size. Sent from my phone On May 22, 2012 4:08 PM, "Greg Bowyer" wrote: > ** > Yes thats basically it > > java -Xms512m -Xmx9g test > > On 22/05/12 12:27, Vitaly Davidovich wrote: > > Hi Greg, > > As an aside, in os::commit_memory, I think the tertiary condition that > sets the flags is inverted - you want MAP_LOCKED if LockHeapInMemory is > set, I believe. > > As to your question, what args are being passed to the above function in > your run? I take it -Xms is less than Xmx in your trial? > > Sent from my phone > On May 22, 2012 3:07 PM, "Greg Bowyer" wrote: > >> Not sure if this is the right venue for this, or if I am insane >> >> I have been playing with the openjdk code with a view to lock the heap in >> memory (or at least suggest to the OS that it wants to be locked in memory). >> >> My use case is for java processes that are greedy in memory and typically >> have a virtual size larger than physical ram. This is not where an end user >> allocates a java heap beyond ram but rather things like Lucene / Cassandra, >> where typically the JVM heap is large but limited to say 1/4 of the total >> physical ram and the rest of the process virtual size is taken up with >> mmap()'d files. >> >> There are java projects that currently do this with a call out via JNA / >> JNI to mlockall() >> >> Asking the OS to use the MAP_LOCKED flag in the mmap calls in >> os_linux.cpp effectively does an mlock / mlockall which I think means that >> when the OS chooses pages to page out; then it should (for some measure of >> should) avoid paging out the JVM heap. >> >> This means that horrors between CMS and paging do not cause hateful >> pauses (hopefully) >> >> The thing I cant understand is that when I start a new VM with this code >> (attached) it appears to lock the full size of the heap (-Xmx) (even though >> the given memory is not used by the VM), this seems to make the entire >> space resident. >> >> any ideas ? >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120522/c1ccfb9a/attachment-0001.html From gbowyer at fastmail.co.uk Tue May 22 14:27:11 2012 From: gbowyer at fastmail.co.uk (Greg Bowyer) Date: Tue, 22 May 2012 14:27:11 -0700 Subject: Stupid question In-Reply-To: References: <4FBBE391.3040600@fastmail.co.uk> <4FBBF24C.1070408@fastmail.co.uk> Message-ID: <4FBC04AF.8000609@fastmail.co.uk> Done, I get this The "Leaky program" bit is stupid simple test program that slowly leaks memory, mostly to test the jvm simply Printf's for commint memory and anon_mmap added I notice that we get this one -> anon_mmap addr=0x5bae00000 bytes=9749659648 fixed=0 greg at localhost ~/projects/openjdk/jdk7/build/linux-amd64-fastdebug/j2sdk-image $ ./bin/java -Xms512m -Xmx9g -XX:+LockHeapInMemory test VM option '+LockHeapInMemory' commit_memory addr=0x7f2d66c49000 bytes=12288 anon_mmap addr=(nil) bytes=50331648 fixed=0 commit_memory addr=0x7f2d60bb5000 bytes=2555904 anon_mmap addr=(nil) bytes=786432 fixed=0 commit_memory addr=0x7f2d66b89000 bytes=40960 anon_mmap addr=0x5bae00000 bytes=9749659648 fixed=0 anon_mmap addr=(nil) bytes=19046400 fixed=0 commit_memory addr=0x7f2d60bb4000 bytes=4096 commit_memory addr=0x740000000 bytes=178913280 commit_memory addr=0x7f2d605b4000 bytes=352256 commit_memory addr=0x5c0000000 bytes=357957632 anon_mmap addr=(nil) bytes=12582912 fixed=0 commit_memory addr=0x7f2d5f9b4000 bytes=700416 commit_memory addr=0x7f2d5ed8b000 bytes=700416 commit_memory addr=0x5bae00000 bytes=21757952 anon_mmap addr=(nil) bytes=167936 fixed=0 commit_memory addr=0x7f2d5f98b000 bytes=45056 commit_memory addr=0x7f2d5ed62000 bytes=45056 commit_memory addr=0x7f2d5d7e5000 bytes=12288 commit_memory addr=0x7f2d5d6e4000 bytes=12288 commit_memory addr=0x7f2d5d3f5000 bytes=12288 commit_memory addr=0x7f2d5d2f4000 bytes=12288 commit_memory addr=0x7f2d5d1f3000 bytes=12288 commit_memory addr=0x7f2d5d0f2000 bytes=12288 Leaking program is slowly leaking 1 Leaking program is slowly leaking 2 Leaking program is slowly leaking 3 .... ^Ccommit_memory addr=0x7f2d5cef0000 bytes=12288 On 22/05/12 13:35, Vitaly Davidovich wrote: > > can you add a print statement to commit_memory() to see what arguments > are being passed to it? I'm sure someone on this mailing list will > have a better suggestion, but that's what I'd try just to establish > that some code is indeed trying to expand the heap to its max allotted > size. > > Sent from my phone > > On May 22, 2012 4:08 PM, "Greg Bowyer" > wrote: > > Yes thats basically it > > java -Xms512m -Xmx9g test > > On 22/05/12 12:27, Vitaly Davidovich wrote: >> >> Hi Greg, >> >> As an aside, in os::commit_memory, I think the tertiary condition >> that sets the flags is inverted - you want MAP_LOCKED if >> LockHeapInMemory is set, I believe. >> >> As to your question, what args are being passed to the above >> function in your run? I take it -Xms is less than Xmx in your trial? >> >> Sent from my phone >> >> On May 22, 2012 3:07 PM, "Greg Bowyer" > > wrote: >> >> Not sure if this is the right venue for this, or if I am insane >> >> I have been playing with the openjdk code with a view to lock >> the heap in memory (or at least suggest to the OS that it >> wants to be locked in memory). >> >> My use case is for java processes that are greedy in memory >> and typically have a virtual size larger than physical ram. >> This is not where an end user allocates a java heap beyond >> ram but rather things like Lucene / Cassandra, where >> typically the JVM heap is large but limited to say 1/4 of the >> total physical ram and the rest of the process virtual size >> is taken up with mmap()'d files. >> >> There are java projects that currently do this with a call >> out via JNA / JNI to mlockall() >> >> Asking the OS to use the MAP_LOCKED flag in the mmap calls in >> os_linux.cpp effectively does an mlock / mlockall which I >> think means that when the OS chooses pages to page out; then >> it should (for some measure of should) avoid paging out the >> JVM heap. >> >> This means that horrors between CMS and paging do not cause >> hateful pauses (hopefully) >> >> The thing I cant understand is that when I start a new VM >> with this code (attached) it appears to lock the full size of >> the heap (-Xmx) (even though the given memory is not used by >> the VM), this seems to make the entire space resident. >> >> any ideas ? >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120522/5b2798ae/attachment.html From gbowyer at fastmail.co.uk Tue May 22 14:35:09 2012 From: gbowyer at fastmail.co.uk (Greg Bowyer) Date: Tue, 22 May 2012 14:35:09 -0700 Subject: Stupid question In-Reply-To: <4FBC04AF.8000609@fastmail.co.uk> References: <4FBBE391.3040600@fastmail.co.uk> <4FBBF24C.1070408@fastmail.co.uk> <4FBC04AF.8000609@fastmail.co.uk> Message-ID: <4FBC068D.3080500@fastmail.co.uk> To clarify slightly I see this here through /proc/meminfo it does not look like the used pages are locked, just unevictable greg at localhost ~ $ cat /proc/meminfo MemTotal: 12286752 kB MemFree: 209512 kB Buffers: 21612 kB Cached: 487636 kB SwapCached: 352 kB Active: 1573468 kB Inactive: 435696 kB Active(anon): 1325624 kB Inactive(anon): 178984 kB Active(file): 247844 kB Inactive(file): 256712 kB Unevictable: 9602276 kB Mlocked: 246888 kB the process itself suggests that its fully wired PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31019 greg 20 0 9523m 9.2g 11m S 0 78.4 0:03.96 java On 22/05/12 14:27, Greg Bowyer wrote: > Done, I get this > > The "Leaky program" bit is stupid simple test program that slowly > leaks memory, mostly to test the jvm simply > > Printf's for commint memory and anon_mmap added > > I notice that we get this one -> anon_mmap addr=0x5bae00000 > bytes=9749659648 fixed=0 > > greg at localhost > ~/projects/openjdk/jdk7/build/linux-amd64-fastdebug/j2sdk-image $ > ./bin/java -Xms512m -Xmx9g -XX:+LockHeapInMemory test > VM option '+LockHeapInMemory' > commit_memory addr=0x7f2d66c49000 bytes=12288 > anon_mmap addr=(nil) bytes=50331648 fixed=0 > commit_memory addr=0x7f2d60bb5000 bytes=2555904 > anon_mmap addr=(nil) bytes=786432 fixed=0 > commit_memory addr=0x7f2d66b89000 bytes=40960 > anon_mmap addr=0x5bae00000 bytes=9749659648 fixed=0 > anon_mmap addr=(nil) bytes=19046400 fixed=0 > commit_memory addr=0x7f2d60bb4000 bytes=4096 > commit_memory addr=0x740000000 bytes=178913280 > commit_memory addr=0x7f2d605b4000 bytes=352256 > commit_memory addr=0x5c0000000 bytes=357957632 > anon_mmap addr=(nil) bytes=12582912 fixed=0 > commit_memory addr=0x7f2d5f9b4000 bytes=700416 > commit_memory addr=0x7f2d5ed8b000 bytes=700416 > commit_memory addr=0x5bae00000 bytes=21757952 > anon_mmap addr=(nil) bytes=167936 fixed=0 > commit_memory addr=0x7f2d5f98b000 bytes=45056 > commit_memory addr=0x7f2d5ed62000 bytes=45056 > commit_memory addr=0x7f2d5d7e5000 bytes=12288 > commit_memory addr=0x7f2d5d6e4000 bytes=12288 > commit_memory addr=0x7f2d5d3f5000 bytes=12288 > commit_memory addr=0x7f2d5d2f4000 bytes=12288 > commit_memory addr=0x7f2d5d1f3000 bytes=12288 > commit_memory addr=0x7f2d5d0f2000 bytes=12288 > Leaking program is slowly leaking 1 > Leaking program is slowly leaking 2 > Leaking program is slowly leaking 3 > .... > ^Ccommit_memory addr=0x7f2d5cef0000 bytes=12288 > > On 22/05/12 13:35, Vitaly Davidovich wrote: >> >> can you add a print statement to commit_memory() to see what >> arguments are being passed to it? I'm sure someone on this mailing >> list will have a better suggestion, but that's what I'd try just to >> establish that some code is indeed trying to expand the heap to its >> max allotted size. >> >> Sent from my phone >> >> On May 22, 2012 4:08 PM, "Greg Bowyer" > > wrote: >> >> Yes thats basically it >> >> java -Xms512m -Xmx9g test >> >> On 22/05/12 12:27, Vitaly Davidovich wrote: >>> >>> Hi Greg, >>> >>> As an aside, in os::commit_memory, I think the tertiary >>> condition that sets the flags is inverted - you want MAP_LOCKED >>> if LockHeapInMemory is set, I believe. >>> >>> As to your question, what args are being passed to the above >>> function in your run? I take it -Xms is less than Xmx in your trial? >>> >>> Sent from my phone >>> >>> On May 22, 2012 3:07 PM, "Greg Bowyer" >> > wrote: >>> >>> Not sure if this is the right venue for this, or if I am insane >>> >>> I have been playing with the openjdk code with a view to >>> lock the heap in memory (or at least suggest to the OS that >>> it wants to be locked in memory). >>> >>> My use case is for java processes that are greedy in memory >>> and typically have a virtual size larger than physical ram. >>> This is not where an end user allocates a java heap beyond >>> ram but rather things like Lucene / Cassandra, where >>> typically the JVM heap is large but limited to say 1/4 of >>> the total physical ram and the rest of the process virtual >>> size is taken up with mmap()'d files. >>> >>> There are java projects that currently do this with a call >>> out via JNA / JNI to mlockall() >>> >>> Asking the OS to use the MAP_LOCKED flag in the mmap calls >>> in os_linux.cpp effectively does an mlock / mlockall which I >>> think means that when the OS chooses pages to page out; then >>> it should (for some measure of should) avoid paging out the >>> JVM heap. >>> >>> This means that horrors between CMS and paging do not cause >>> hateful pauses (hopefully) >>> >>> The thing I cant understand is that when I start a new VM >>> with this code (attached) it appears to lock the full size >>> of the heap (-Xmx) (even though the given memory is not used >>> by the VM), this seems to make the entire space resident. >>> >>> any ideas ? >>> >>> _______________________________________________ >>> hotspot-gc-use mailing list >>> hotspot-gc-use at openjdk.java.net >>> >>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120522/aeac79cb/attachment-0001.html From vitalyd at gmail.com Tue May 22 15:29:04 2012 From: vitalyd at gmail.com (Vitaly Davidovich) Date: Tue, 22 May 2012 18:29:04 -0400 Subject: Stupid question In-Reply-To: References: <4FBBE391.3040600@fastmail.co.uk> Message-ID: Look under /proc//status for VmLck count. In your output, I think Unevictable is how mlock()'d pages are reported, which looks inline with your patch. Sent from my phone On May 22, 2012 5:35 PM, "Greg Bowyer" wrote: -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120522/146f3b50/attachment.html From gbowyer at fastmail.co.uk Tue May 22 16:19:24 2012 From: gbowyer at fastmail.co.uk (Greg Bowyer) Date: Tue, 22 May 2012 16:19:24 -0700 Subject: Stupid question In-Reply-To: <4FBC068D.3080500@fastmail.co.uk> References: <4FBBE391.3040600@fastmail.co.uk> <4FBBF24C.1070408@fastmail.co.uk> <4FBC04AF.8000609@fastmail.co.uk> <4FBC068D.3080500@fastmail.co.uk> Message-ID: <4FBC1EFC.1050908@fastmail.co.uk> So the more I think about this the more it occurs to me that its obvious that the mmap() is for the full heap size and that my mlock is in the wrong place I am thinking that this should be moved up to os::malloc since hotspot/src/share/vm/memory/allocation.inline.hpp really does the allocation and the os macros, but then the space for parallel new does use os::commit rather than os::malloc ? On 22/05/12 14:35, Greg Bowyer wrote: > To clarify slightly I see this here > > through /proc/meminfo it does not look like the used pages are locked, > just unevictable > > greg at localhost ~ $ cat /proc/meminfo > MemTotal: 12286752 kB > MemFree: 209512 kB > Buffers: 21612 kB > Cached: 487636 kB > SwapCached: 352 kB > Active: 1573468 kB > Inactive: 435696 kB > Active(anon): 1325624 kB > Inactive(anon): 178984 kB > Active(file): 247844 kB > Inactive(file): 256712 kB > Unevictable: 9602276 kB > Mlocked: 246888 kB > > the process itself suggests that its fully wired > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 31019 greg 20 0 9523m 9.2g 11m S 0 78.4 0:03.96 java > > On 22/05/12 14:27, Greg Bowyer wrote: >> Done, I get this >> >> The "Leaky program" bit is stupid simple test program that slowly >> leaks memory, mostly to test the jvm simply >> >> Printf's for commint memory and anon_mmap added >> >> I notice that we get this one -> anon_mmap addr=0x5bae00000 >> bytes=9749659648 fixed=0 >> >> greg at localhost >> ~/projects/openjdk/jdk7/build/linux-amd64-fastdebug/j2sdk-image $ >> ./bin/java -Xms512m -Xmx9g -XX:+LockHeapInMemory test >> VM option '+LockHeapInMemory' >> commit_memory addr=0x7f2d66c49000 bytes=12288 >> anon_mmap addr=(nil) bytes=50331648 fixed=0 >> commit_memory addr=0x7f2d60bb5000 bytes=2555904 >> anon_mmap addr=(nil) bytes=786432 fixed=0 >> commit_memory addr=0x7f2d66b89000 bytes=40960 >> anon_mmap addr=0x5bae00000 bytes=9749659648 fixed=0 >> anon_mmap addr=(nil) bytes=19046400 fixed=0 >> commit_memory addr=0x7f2d60bb4000 bytes=4096 >> commit_memory addr=0x740000000 bytes=178913280 >> commit_memory addr=0x7f2d605b4000 bytes=352256 >> commit_memory addr=0x5c0000000 bytes=357957632 >> anon_mmap addr=(nil) bytes=12582912 fixed=0 >> commit_memory addr=0x7f2d5f9b4000 bytes=700416 >> commit_memory addr=0x7f2d5ed8b000 bytes=700416 >> commit_memory addr=0x5bae00000 bytes=21757952 >> anon_mmap addr=(nil) bytes=167936 fixed=0 >> commit_memory addr=0x7f2d5f98b000 bytes=45056 >> commit_memory addr=0x7f2d5ed62000 bytes=45056 >> commit_memory addr=0x7f2d5d7e5000 bytes=12288 >> commit_memory addr=0x7f2d5d6e4000 bytes=12288 >> commit_memory addr=0x7f2d5d3f5000 bytes=12288 >> commit_memory addr=0x7f2d5d2f4000 bytes=12288 >> commit_memory addr=0x7f2d5d1f3000 bytes=12288 >> commit_memory addr=0x7f2d5d0f2000 bytes=12288 >> Leaking program is slowly leaking 1 >> Leaking program is slowly leaking 2 >> Leaking program is slowly leaking 3 >> .... >> ^Ccommit_memory addr=0x7f2d5cef0000 bytes=12288 >> >> On 22/05/12 13:35, Vitaly Davidovich wrote: >>> >>> can you add a print statement to commit_memory() to see what >>> arguments are being passed to it? I'm sure someone on this mailing >>> list will have a better suggestion, but that's what I'd try just to >>> establish that some code is indeed trying to expand the heap to its >>> max allotted size. >>> >>> Sent from my phone >>> >>> On May 22, 2012 4:08 PM, "Greg Bowyer" >> > wrote: >>> >>> Yes thats basically it >>> >>> java -Xms512m -Xmx9g test >>> >>> On 22/05/12 12:27, Vitaly Davidovich wrote: >>>> >>>> Hi Greg, >>>> >>>> As an aside, in os::commit_memory, I think the tertiary >>>> condition that sets the flags is inverted - you want MAP_LOCKED >>>> if LockHeapInMemory is set, I believe. >>>> >>>> As to your question, what args are being passed to the above >>>> function in your run? I take it -Xms is less than Xmx in your >>>> trial? >>>> >>>> Sent from my phone >>>> >>>> On May 22, 2012 3:07 PM, "Greg Bowyer" >>> > wrote: >>>> >>>> Not sure if this is the right venue for this, or if I am insane >>>> >>>> I have been playing with the openjdk code with a view to >>>> lock the heap in memory (or at least suggest to the OS that >>>> it wants to be locked in memory). >>>> >>>> My use case is for java processes that are greedy in memory >>>> and typically have a virtual size larger than physical ram. >>>> This is not where an end user allocates a java heap beyond >>>> ram but rather things like Lucene / Cassandra, where >>>> typically the JVM heap is large but limited to say 1/4 of >>>> the total physical ram and the rest of the process virtual >>>> size is taken up with mmap()'d files. >>>> >>>> There are java projects that currently do this with a call >>>> out via JNA / JNI to mlockall() >>>> >>>> Asking the OS to use the MAP_LOCKED flag in the mmap calls >>>> in os_linux.cpp effectively does an mlock / mlockall which >>>> I think means that when the OS chooses pages to page out; >>>> then it should (for some measure of should) avoid paging >>>> out the JVM heap. >>>> >>>> This means that horrors between CMS and paging do not cause >>>> hateful pauses (hopefully) >>>> >>>> The thing I cant understand is that when I start a new VM >>>> with this code (attached) it appears to lock the full size >>>> of the heap (-Xmx) (even though the given memory is not >>>> used by the VM), this seems to make the entire space resident. >>>> >>>> any ideas ? >>>> >>>> _______________________________________________ >>>> hotspot-gc-use mailing list >>>> hotspot-gc-use at openjdk.java.net >>>> >>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120522/d0abb5d3/attachment.html From Peter.B.Kessler at Oracle.COM Tue May 22 16:38:31 2012 From: Peter.B.Kessler at Oracle.COM (Peter B. Kessler) Date: Tue, 22 May 2012 16:38:31 -0700 Subject: Stupid question In-Reply-To: <4FBC04AF.8000609@fastmail.co.uk> References: <4FBBE391.3040600@fastmail.co.uk> <4FBBF24C.1070408@fastmail.co.uk> <4FBC04AF.8000609@fastmail.co.uk> Message-ID: <4FBC2377.7080108@Oracle.COM> If you add -XX:+PrintGCDetails to your command line, when the program exits it will print out where the various pieces of the Java heap are, which might help you connect the mmaps with parts of the heap. E.g., $ $Deployed/JDK-7/bin/java -d64 -Xms512m -Xmx9g -XX:+PrintGCDetails -cp $Play/Java HelloWorld Hello world! Heap PSYoungGen total 154112K, used 5284K [0xfffffd7f3a600000, 0xfffffd7f45200000, 0xfffffd7ffa600000) eden space 132096K, 4% used [0xfffffd7f3a600000,0xfffffd7f3ab29028,0xfffffd7f42700000) from space 22016K, 0% used [0xfffffd7f43c80000,0xfffffd7f43c80000,0xfffffd7f45200000) to space 22016K, 0% used [0xfffffd7f42700000,0xfffffd7f42700000,0xfffffd7f43c80000) PSOldGen total 350208K, used 0K [0xfffffd7dba600000, 0xfffffd7dcfc00000, 0xfffffd7f3a600000) object space 350208K, 0% used [0xfffffd7dba600000,0xfffffd7dba600000,0xfffffd7dcfc00000) PSPermGen total 22528K, used 2538K [0xfffffd7db5400000, 0xfffffd7db6a00000, 0xfffffd7dba600000) object space 22528K, 11% used [0xfffffd7db5400000,0xfffffd7db567a868,0xfffffd7db6a00000) The generation triples are [start address, committed to address, end address). The space triples are [start address, current top address, end address), but I don't think you need those here. With the -XX:+UseParallelGC collector, the generations are perm, old, young. Other collectors lay the generations out differently. You can also use those addresses to find the pieces of the Java heap in a pmap. ... peter Greg Bowyer wrote: > Done, I get this > > The "Leaky program" bit is stupid simple test program that slowly leaks > memory, mostly to test the jvm simply > > Printf's for commint memory and anon_mmap added > > I notice that we get this one -> anon_mmap addr=0x5bae00000 bytes=9749659648 fixed=0 > > greg at localhost > ~/projects/openjdk/jdk7/build/linux-amd64-fastdebug/j2sdk-image $ ./bin/java -Xms512m -Xmx9g -XX:+LockHeapInMemory test > VM option '+LockHeapInMemory' > commit_memory addr=0x7f2d66c49000 bytes=12288 > anon_mmap addr=(nil) bytes=50331648 fixed=0 > commit_memory addr=0x7f2d60bb5000 bytes=2555904 > anon_mmap addr=(nil) bytes=786432 fixed=0 > commit_memory addr=0x7f2d66b89000 bytes=40960 > anon_mmap addr=0x5bae00000 bytes=9749659648 fixed=0 > anon_mmap addr=(nil) bytes=19046400 fixed=0 > commit_memory addr=0x7f2d60bb4000 bytes=4096 > commit_memory addr=0x740000000 bytes=178913280 > commit_memory addr=0x7f2d605b4000 bytes=352256 > commit_memory addr=0x5c0000000 bytes=357957632 > anon_mmap addr=(nil) bytes=12582912 fixed=0 > commit_memory addr=0x7f2d5f9b4000 bytes=700416 > commit_memory addr=0x7f2d5ed8b000 bytes=700416 > commit_memory addr=0x5bae00000 bytes=21757952 > anon_mmap addr=(nil) bytes=167936 fixed=0 > commit_memory addr=0x7f2d5f98b000 bytes=45056 > commit_memory addr=0x7f2d5ed62000 bytes=45056 > commit_memory addr=0x7f2d5d7e5000 bytes=12288 > commit_memory addr=0x7f2d5d6e4000 bytes=12288 > commit_memory addr=0x7f2d5d3f5000 bytes=12288 > commit_memory addr=0x7f2d5d2f4000 bytes=12288 > commit_memory addr=0x7f2d5d1f3000 bytes=12288 > commit_memory addr=0x7f2d5d0f2000 bytes=12288 > Leaking program is slowly leaking 1 > Leaking program is slowly leaking 2 > Leaking program is slowly leaking 3 > .... > ^Ccommit_memory addr=0x7f2d5cef0000 bytes=12288 > > On 22/05/12 13:35, Vitaly Davidovich wrote: >> >> can you add a print statement to commit_memory() to see what arguments >> are being passed to it? I'm sure someone on this mailing list will >> have a better suggestion, but that's what I'd try just to establish >> that some code is indeed trying to expand the heap to its max allotted >> size. >> >> Sent from my phone >> >> On May 22, 2012 4:08 PM, "Greg Bowyer" > > wrote: >> >> Yes thats basically it >> >> java -Xms512m -Xmx9g test >> >> On 22/05/12 12:27, Vitaly Davidovich wrote: >>> >>> Hi Greg, >>> >>> As an aside, in os::commit_memory, I think the tertiary condition >>> that sets the flags is inverted - you want MAP_LOCKED if >>> LockHeapInMemory is set, I believe. >>> >>> As to your question, what args are being passed to the above >>> function in your run? I take it -Xms is less than Xmx in your trial? >>> >>> Sent from my phone >>> >>> On May 22, 2012 3:07 PM, "Greg Bowyer" >> > wrote: >>> >>> Not sure if this is the right venue for this, or if I am insane >>> >>> I have been playing with the openjdk code with a view to lock >>> the heap in memory (or at least suggest to the OS that it >>> wants to be locked in memory). >>> >>> My use case is for java processes that are greedy in memory >>> and typically have a virtual size larger than physical ram. >>> This is not where an end user allocates a java heap beyond >>> ram but rather things like Lucene / Cassandra, where >>> typically the JVM heap is large but limited to say 1/4 of the >>> total physical ram and the rest of the process virtual size >>> is taken up with mmap()'d files. >>> >>> There are java projects that currently do this with a call >>> out via JNA / JNI to mlockall() >>> >>> Asking the OS to use the MAP_LOCKED flag in the mmap calls in >>> os_linux.cpp effectively does an mlock / mlockall which I >>> think means that when the OS chooses pages to page out; then >>> it should (for some measure of should) avoid paging out the >>> JVM heap. >>> >>> This means that horrors between CMS and paging do not cause >>> hateful pauses (hopefully) >>> >>> The thing I cant understand is that when I start a new VM >>> with this code (attached) it appears to lock the full size of >>> the heap (-Xmx) (even though the given memory is not used by >>> the VM), this seems to make the entire space resident. >>> >>> any ideas ? >>> >>> _______________________________________________ >>> hotspot-gc-use mailing list >>> hotspot-gc-use at openjdk.java.net >>> >>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>> >> > > > ------------------------------------------------------------------------ > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From gbowyer at fastmail.co.uk Tue May 22 16:50:14 2012 From: gbowyer at fastmail.co.uk (Greg Bowyer) Date: Tue, 22 May 2012 16:50:14 -0700 Subject: Stupid question In-Reply-To: <4FBC2377.7080108@Oracle.COM> References: <4FBBE391.3040600@fastmail.co.uk> <4FBBF24C.1070408@fastmail.co.uk> <4FBC04AF.8000609@fastmail.co.uk> <4FBC2377.7080108@Oracle.COM> Message-ID: <4FBC2636.2080003@fastmail.co.uk> Awesome thats really nice to know that said its really easy to spot the mmap segment for the heap since its the same size as its limit and its 9g :) -- Greg On 22/05/12 16:38, Peter B. Kessler wrote: > If you add -XX:+PrintGCDetails to your command line, when the program exits it will print out where the various pieces of the Java heap are, which might help you connect the mmaps with parts of the heap. E.g., > > $ $Deployed/JDK-7/bin/java -d64 -Xms512m -Xmx9g -XX:+PrintGCDetails -cp $Play/Java HelloWorld > Hello world! > Heap > PSYoungGen total 154112K, used 5284K [0xfffffd7f3a600000, 0xfffffd7f45200000, 0xfffffd7ffa600000) > eden space 132096K, 4% used [0xfffffd7f3a600000,0xfffffd7f3ab29028,0xfffffd7f42700000) > from space 22016K, 0% used [0xfffffd7f43c80000,0xfffffd7f43c80000,0xfffffd7f45200000) > to space 22016K, 0% used [0xfffffd7f42700000,0xfffffd7f42700000,0xfffffd7f43c80000) > PSOldGen total 350208K, used 0K [0xfffffd7dba600000, 0xfffffd7dcfc00000, 0xfffffd7f3a600000) > object space 350208K, 0% used [0xfffffd7dba600000,0xfffffd7dba600000,0xfffffd7dcfc00000) > PSPermGen total 22528K, used 2538K [0xfffffd7db5400000, 0xfffffd7db6a00000, 0xfffffd7dba600000) > object space 22528K, 11% used [0xfffffd7db5400000,0xfffffd7db567a868,0xfffffd7db6a00000) > > The generation triples are [start address, committed to address, end address). The space triples are [start address, current top address, end address), but I don't think you need those here. With the -XX:+UseParallelGC collector, the generations are perm, old, young. Other collectors lay the generations out differently. You can also use those addresses to find the pieces of the Java heap in a pmap. > > ... peter > > Greg Bowyer wrote: >> Done, I get this >> >> The "Leaky program" bit is stupid simple test program that slowly leaks >> memory, mostly to test the jvm simply >> >> Printf's for commint memory and anon_mmap added >> >> I notice that we get this one -> anon_mmap addr=0x5bae00000 bytes=9749659648 fixed=0 >> >> greg at localhost >> ~/projects/openjdk/jdk7/build/linux-amd64-fastdebug/j2sdk-image $ ./bin/java -Xms512m -Xmx9g -XX:+LockHeapInMemory test >> VM option '+LockHeapInMemory' >> commit_memory addr=0x7f2d66c49000 bytes=12288 >> anon_mmap addr=(nil) bytes=50331648 fixed=0 >> commit_memory addr=0x7f2d60bb5000 bytes=2555904 >> anon_mmap addr=(nil) bytes=786432 fixed=0 >> commit_memory addr=0x7f2d66b89000 bytes=40960 >> anon_mmap addr=0x5bae00000 bytes=9749659648 fixed=0 >> anon_mmap addr=(nil) bytes=19046400 fixed=0 >> commit_memory addr=0x7f2d60bb4000 bytes=4096 >> commit_memory addr=0x740000000 bytes=178913280 >> commit_memory addr=0x7f2d605b4000 bytes=352256 >> commit_memory addr=0x5c0000000 bytes=357957632 >> anon_mmap addr=(nil) bytes=12582912 fixed=0 >> commit_memory addr=0x7f2d5f9b4000 bytes=700416 >> commit_memory addr=0x7f2d5ed8b000 bytes=700416 >> commit_memory addr=0x5bae00000 bytes=21757952 >> anon_mmap addr=(nil) bytes=167936 fixed=0 >> commit_memory addr=0x7f2d5f98b000 bytes=45056 >> commit_memory addr=0x7f2d5ed62000 bytes=45056 >> commit_memory addr=0x7f2d5d7e5000 bytes=12288 >> commit_memory addr=0x7f2d5d6e4000 bytes=12288 >> commit_memory addr=0x7f2d5d3f5000 bytes=12288 >> commit_memory addr=0x7f2d5d2f4000 bytes=12288 >> commit_memory addr=0x7f2d5d1f3000 bytes=12288 >> commit_memory addr=0x7f2d5d0f2000 bytes=12288 >> Leaking program is slowly leaking 1 >> Leaking program is slowly leaking 2 >> Leaking program is slowly leaking 3 >> .... >> ^Ccommit_memory addr=0x7f2d5cef0000 bytes=12288 >> >> On 22/05/12 13:35, Vitaly Davidovich wrote: >>> can you add a print statement to commit_memory() to see what arguments >>> are being passed to it? I'm sure someone on this mailing list will >>> have a better suggestion, but that's what I'd try just to establish >>> that some code is indeed trying to expand the heap to its max allotted >>> size. >>> >>> Sent from my phone >>> >>> On May 22, 2012 4:08 PM, "Greg Bowyer">> > wrote: >>> >>> Yes thats basically it >>> >>> java -Xms512m -Xmx9g test >>> >>> On 22/05/12 12:27, Vitaly Davidovich wrote: >>>> Hi Greg, >>>> >>>> As an aside, in os::commit_memory, I think the tertiary condition >>>> that sets the flags is inverted - you want MAP_LOCKED if >>>> LockHeapInMemory is set, I believe. >>>> >>>> As to your question, what args are being passed to the above >>>> function in your run? I take it -Xms is less than Xmx in your trial? >>>> >>>> Sent from my phone >>>> >>>> On May 22, 2012 3:07 PM, "Greg Bowyer">>> > wrote: >>>> >>>> Not sure if this is the right venue for this, or if I am insane >>>> >>>> I have been playing with the openjdk code with a view to lock >>>> the heap in memory (or at least suggest to the OS that it >>>> wants to be locked in memory). >>>> >>>> My use case is for java processes that are greedy in memory >>>> and typically have a virtual size larger than physical ram. >>>> This is not where an end user allocates a java heap beyond >>>> ram but rather things like Lucene / Cassandra, where >>>> typically the JVM heap is large but limited to say 1/4 of the >>>> total physical ram and the rest of the process virtual size >>>> is taken up with mmap()'d files. >>>> >>>> There are java projects that currently do this with a call >>>> out via JNA / JNI to mlockall() >>>> >>>> Asking the OS to use the MAP_LOCKED flag in the mmap calls in >>>> os_linux.cpp effectively does an mlock / mlockall which I >>>> think means that when the OS chooses pages to page out; then >>>> it should (for some measure of should) avoid paging out the >>>> JVM heap. >>>> >>>> This means that horrors between CMS and paging do not cause >>>> hateful pauses (hopefully) >>>> >>>> The thing I cant understand is that when I start a new VM >>>> with this code (attached) it appears to lock the full size of >>>> the heap (-Xmx) (even though the given memory is not used by >>>> the VM), this seems to make the entire space resident. >>>> >>>> any ideas ? >>>> >>>> _______________________________________________ >>>> hotspot-gc-use mailing list >>>> hotspot-gc-use at openjdk.java.net >>>> >>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>>> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From rednaxelafx at gmail.com Tue May 22 16:56:59 2012 From: rednaxelafx at gmail.com (Krystal Mok) Date: Wed, 23 May 2012 07:56:59 +0800 Subject: Stupid question In-Reply-To: <4FBC2377.7080108@Oracle.COM> References: <4FBBE391.3040600@fastmail.co.uk> <4FBBF24C.1070408@fastmail.co.uk> <4FBC04AF.8000609@fastmail.co.uk> <4FBC2377.7080108@Oracle.COM> Message-ID: <3EEC5ADE-DE2E-40CE-B0DA-A0B06328DE28@gmail.com> Hi Greg, When -XX:+PrintGCDetails is set, you can also get the heap layout with kill -3 which prints the same information to stdout as Peter mentioned (after the thread dump), and it's good for when you don't want to wait until program exit to get the heap layout. - Kris On 2012-5-23, at 7:38, "Peter B. Kessler" wrote: > If you add -XX:+PrintGCDetails to your command line, when the program exits it will print out where the various pieces of the Java heap are, which might help you connect the mmaps with parts of the heap. E.g., > > $ $Deployed/JDK-7/bin/java -d64 -Xms512m -Xmx9g -XX:+PrintGCDetails -cp $Play/Java HelloWorld > Hello world! > Heap > PSYoungGen total 154112K, used 5284K [0xfffffd7f3a600000, 0xfffffd7f45200000, 0xfffffd7ffa600000) > eden space 132096K, 4% used [0xfffffd7f3a600000,0xfffffd7f3ab29028,0xfffffd7f42700000) > from space 22016K, 0% used [0xfffffd7f43c80000,0xfffffd7f43c80000,0xfffffd7f45200000) > to space 22016K, 0% used [0xfffffd7f42700000,0xfffffd7f42700000,0xfffffd7f43c80000) > PSOldGen total 350208K, used 0K [0xfffffd7dba600000, 0xfffffd7dcfc00000, 0xfffffd7f3a600000) > object space 350208K, 0% used [0xfffffd7dba600000,0xfffffd7dba600000,0xfffffd7dcfc00000) > PSPermGen total 22528K, used 2538K [0xfffffd7db5400000, 0xfffffd7db6a00000, 0xfffffd7dba600000) > object space 22528K, 11% used [0xfffffd7db5400000,0xfffffd7db567a868,0xfffffd7db6a00000) > > The generation triples are [start address, committed to address, end address). The space triples are [start address, current top address, end address), but I don't think you need those here. With the -XX:+UseParallelGC collector, the generations are perm, old, young. Other collectors lay the generations out differently. You can also use those addresses to find the pieces of the Java heap in a pmap. > > ... peter > > Greg Bowyer wrote: >> Done, I get this >> >> The "Leaky program" bit is stupid simple test program that slowly leaks >> memory, mostly to test the jvm simply >> >> Printf's for commint memory and anon_mmap added >> >> I notice that we get this one -> anon_mmap addr=0x5bae00000 bytes=9749659648 fixed=0 >> >> greg at localhost >> ~/projects/openjdk/jdk7/build/linux-amd64-fastdebug/j2sdk-image $ ./bin/java -Xms512m -Xmx9g -XX:+LockHeapInMemory test >> VM option '+LockHeapInMemory' >> commit_memory addr=0x7f2d66c49000 bytes=12288 >> anon_mmap addr=(nil) bytes=50331648 fixed=0 >> commit_memory addr=0x7f2d60bb5000 bytes=2555904 >> anon_mmap addr=(nil) bytes=786432 fixed=0 >> commit_memory addr=0x7f2d66b89000 bytes=40960 >> anon_mmap addr=0x5bae00000 bytes=9749659648 fixed=0 >> anon_mmap addr=(nil) bytes=19046400 fixed=0 >> commit_memory addr=0x7f2d60bb4000 bytes=4096 >> commit_memory addr=0x740000000 bytes=178913280 >> commit_memory addr=0x7f2d605b4000 bytes=352256 >> commit_memory addr=0x5c0000000 bytes=357957632 >> anon_mmap addr=(nil) bytes=12582912 fixed=0 >> commit_memory addr=0x7f2d5f9b4000 bytes=700416 >> commit_memory addr=0x7f2d5ed8b000 bytes=700416 >> commit_memory addr=0x5bae00000 bytes=21757952 >> anon_mmap addr=(nil) bytes=167936 fixed=0 >> commit_memory addr=0x7f2d5f98b000 bytes=45056 >> commit_memory addr=0x7f2d5ed62000 bytes=45056 >> commit_memory addr=0x7f2d5d7e5000 bytes=12288 >> commit_memory addr=0x7f2d5d6e4000 bytes=12288 >> commit_memory addr=0x7f2d5d3f5000 bytes=12288 >> commit_memory addr=0x7f2d5d2f4000 bytes=12288 >> commit_memory addr=0x7f2d5d1f3000 bytes=12288 >> commit_memory addr=0x7f2d5d0f2000 bytes=12288 >> Leaking program is slowly leaking 1 >> Leaking program is slowly leaking 2 >> Leaking program is slowly leaking 3 >> .... >> ^Ccommit_memory addr=0x7f2d5cef0000 bytes=12288 >> >> On 22/05/12 13:35, Vitaly Davidovich wrote: >>> >>> can you add a print statement to commit_memory() to see what arguments >>> are being passed to it? I'm sure someone on this mailing list will >>> have a better suggestion, but that's what I'd try just to establish >>> that some code is indeed trying to expand the heap to its max allotted >>> size. >>> >>> Sent from my phone >>> >>> On May 22, 2012 4:08 PM, "Greg Bowyer" >> > wrote: >>> >>> Yes thats basically it >>> >>> java -Xms512m -Xmx9g test >>> >>> On 22/05/12 12:27, Vitaly Davidovich wrote: >>>> >>>> Hi Greg, >>>> >>>> As an aside, in os::commit_memory, I think the tertiary condition >>>> that sets the flags is inverted - you want MAP_LOCKED if >>>> LockHeapInMemory is set, I believe. >>>> >>>> As to your question, what args are being passed to the above >>>> function in your run? I take it -Xms is less than Xmx in your trial? >>>> >>>> Sent from my phone >>>> >>>> On May 22, 2012 3:07 PM, "Greg Bowyer" >>> > wrote: >>>> >>>> Not sure if this is the right venue for this, or if I am insane >>>> >>>> I have been playing with the openjdk code with a view to lock >>>> the heap in memory (or at least suggest to the OS that it >>>> wants to be locked in memory). >>>> >>>> My use case is for java processes that are greedy in memory >>>> and typically have a virtual size larger than physical ram. >>>> This is not where an end user allocates a java heap beyond >>>> ram but rather things like Lucene / Cassandra, where >>>> typically the JVM heap is large but limited to say 1/4 of the >>>> total physical ram and the rest of the process virtual size >>>> is taken up with mmap()'d files. >>>> >>>> There are java projects that currently do this with a call >>>> out via JNA / JNI to mlockall() >>>> >>>> Asking the OS to use the MAP_LOCKED flag in the mmap calls in >>>> os_linux.cpp effectively does an mlock / mlockall which I >>>> think means that when the OS chooses pages to page out; then >>>> it should (for some measure of should) avoid paging out the >>>> JVM heap. >>>> >>>> This means that horrors between CMS and paging do not cause >>>> hateful pauses (hopefully) >>>> >>>> The thing I cant understand is that when I start a new VM >>>> with this code (attached) it appears to lock the full size of >>>> the heap (-Xmx) (even though the given memory is not used by >>>> the VM), this seems to make the entire space resident. >>>> >>>> any ideas ? >>>> >>>> _______________________________________________ >>>> hotspot-gc-use mailing list >>>> hotspot-gc-use at openjdk.java.net >>>> >>>> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >>>> >>> >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From jon.masamitsu at oracle.com Wed May 23 11:39:53 2012 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Wed, 23 May 2012 11:39:53 -0700 Subject: MinHeapFreeRatio / MaxHeapFreeRatio In-Reply-To: <4FABED66.1080308@oracle.com> References: <4FAA3867.2020808@oracle.com> <733AEBA7-7CE7-451E-A4A5-D053E65CE48F@exnet.com> <4FABED66.1080308@oracle.com> Message-ID: <4FBD2EF9.1020605@oracle.com> Damon, I looked at the CMS code more closely and think I was mistaken before. The CMS code that does the System.gc() does eventually call the code that does resizing. I'll have to look elsewhere for the problem. Jon On 5/10/2012 9:31 AM, Jon Masamitsu wrote: > Damon, > > From inspection of the code I think that CMS does not do a > resizing of the heap after it does a System.gc(). That resizing > would be the code that adjusts the generations for > MinHeapFreeRatio / MaxHeapFreeRatio and returns > memory to the OS. > > I've created CR 7167923 to investigate this bug. > > Jon > > On 5/9/2012 2:34 AM, Damon Hart-Davis wrote: >> Hi, >> >> No, because when I explicitly call System.gc() I want it to try as hard as it can, even to the point of stopping the world... >> >> Rgds >> >> Damon >> >> >> On 9 May 2012, at 10:27, Jon Masamitsu wrote: >> >>> Damon, >>> >>> Do you turn on ExplicitGCInvokesConcurrent? >>> >>> Jon >>> > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use