From Sharang.Patil at owmessaging.com Fri Apr 12 10:02:37 2013 From: Sharang.Patil at owmessaging.com (Sharang Patil) Date: Fri, 12 Apr 2013 22:32:37 +0530 Subject: JDK6 core observed Message-ID: <001e01ce379f$8e265e50$aa731af0$@Patil@owmessaging.com> Hi, I had a query regarding the GC tuning and JVM crashing and need your help in understanding the scenarios better. Following are the details. Let me know if you need any additional details. java version "1.6.0_26" Java(TM) SE Runtime Environment (build 1.6.0_26-b03) Java HotSpot(TM) Server VM (build 20.1-b02, mixed mode) Platform - SunOS 5.10 Generic_142909-17 sun4v sparc SUNW,Sun-Fire-T200 Primary Memory - 8GB No of Processors - 32 Following are the JVM args we tried for the concurrent 350 users for 2 hours Jmeter run and it worked fine. For 350 threads [-J-server -J-d64 -Xms6g -Xmx6g -Xss1m -J-noclassgc -J-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:NewSize=1280m -XX:MaxNewSize=1280m -XX:+CMSClassUnloadingEnabled -XX:+PrintTenuringDistribution -Xloggc:log/details.pipe -XX:+PrintHeapAtGC -Dmail.smtp.auth=true -XX:TargetSurvivorRatio=60 -XX:CMSInitiatingOccupancyFraction=30 -XX:SurvivorRatio=8 -XX:MaxGCPauseMillis=5000 -XX:GCTimeRatio=19] When we tried with 400 threads for 2 hours run, there is a core generated with the JVM log file as # Problematic frame: # V [libjvm.so+0x84bc2c] int objArrayKlass::oop_oop_iterate_nv(oopDesc*,ParScanWithoutBarrierClosure*)+0x 1fc I have attached the JVM log file and GC log file for this case. Thanks, Sharang The information and attachments contained in this email are intended for use by the named recipient only and are confidential, privileged and/or protected by intellectual property rights. You are hereby notified that any use, dissemination, disclosure, distribution, copying or storage of this information is strictly prohibited. If you have received this information in error, please notify the sender by return email and delete the electronic transmission, including all attachments, from your system. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20130412/4ffdf32d/attachment-0001.html -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: hs_err_pid11774.log Url: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20130412/4ffdf32d/hs_err_pid11774-0001.log -------------- next part -------------- A non-text attachment was scrubbed... Name: details.pipe_400Threads_04122013 Type: application/octet-stream Size: 2134 bytes Desc: not available Url : http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20130412/4ffdf32d/details-0001.pipe_400Threads_04122013 From sirinath at sakrio.com Fri Apr 19 01:39:21 2013 From: sirinath at sakrio.com (Suminda Dharmasena) Date: Fri, 19 Apr 2013 14:09:21 +0530 Subject: javax GC API In-Reply-To: References: Message-ID: Another potential annotations is: - GCUsing(GCSystem) - all objects created in the specified scope will be GCed using specified GC - annotations need to support static references I like to know if there are others thinking in the same line. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20130419/29708921/attachment.html From sirinath at sakrio.com Fri Apr 19 02:10:40 2013 From: sirinath at sakrio.com (Suminda Dharmasena) Date: Fri, 19 Apr 2013 14:40:40 +0530 Subject: javax GC API In-Reply-To: References: Message-ID: Another potential tag to ensure GC does not happen during the execution of a code block: - GCSuspend - a synchronous object can be used so that the GC threads do not run while the code block is executing Of course some revision to current annotations will be needed to accommodate this. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20130419/6cda7abb/attachment.html From sirinath at sakrio.com Fri Apr 19 02:43:29 2013 From: sirinath at sakrio.com (Suminda Dharmasena) Date: Fri, 19 Apr 2013 15:13:29 +0530 Subject: javax GC API In-Reply-To: References: Message-ID: Also an annotation to GC immediately when reference count reaches 0: - GCImmediate -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20130419/0e0fa325/attachment.html From sirinath at sakrio.com Fri Apr 19 02:59:32 2013 From: sirinath at sakrio.com (Suminda Dharmasena) Date: Fri, 19 Apr 2013 15:29:32 +0530 Subject: javax GC API In-Reply-To: <31A020B9-F7F2-4DC4-9EE8-E8ABF1D25A69@kodewerk.com> References: <31A020B9-F7F2-4DC4-9EE8-E8ABF1D25A69@kodewerk.com> Message-ID: Hi, Not creating an object will not help. Objects created before can be GCed. The block will suspend GC block until the block exists or until just before OutOfMemory exception is raised. Also this can allow API based GC and processing of GC related annotations. There are times where you cannot afford to have a sizable GC pause. S -- Suminda Sirinath Salpitikorala Dharmasena, B.Sc. Comp. & I.S. (Hon.) Lond., P.G.Dip. Ind. Maths. J'Pura, MIEEE, MACM, CEO Sakr??! ? *Address*: 6G ? 1st Lane ? Pagoda Road ? Nugegoda 10250 ? Sri Lanka. ? *Tele*: +94-(0)11-5 864614 / 5 875614 / 2 825908 ? *Web*: http://www.sakrio.com ? This email is subjected to the email Terms of Use and Disclaimer: http://www.sakrio.com/email-legal. Please read this first. -- On 19 April 2013 15:20, Kirk Pepperdine wrote: > Hi Suminda, > > With all due respect, I'm not sure that it's a great idea to pass control > of the garbage collectors over to the programmer. In this case, if you were > to ensure that GC did not happen during the execution of a block you are in > danger of filling heap. A better way to prevent GC is to not create objects. > > Regards, > Kirk > > On 2013-04-19, at 11:10 AM, Suminda Dharmasena > wrote: > > Another potential tag to ensure GC does not happen during the execution of > a code block: > > - GCSuspend - a synchronous object can be used so that the GC threads > do not run while the code block is executing > > Of course some revision to current annotations will be needed > to accommodate this. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20130419/a26258ab/attachment.html From sirinath at sakrio.com Fri Apr 19 03:04:50 2013 From: sirinath at sakrio.com (Suminda Dharmasena) Date: Fri, 19 Apr 2013 15:34:50 +0530 Subject: javax GC API In-Reply-To: References: <31A020B9-F7F2-4DC4-9EE8-E8ABF1D25A69@kodewerk.com> Message-ID: Also if the programmer fails to wisely GC using API / Annotations then the fallback is the default GC mechanism. This way there will be no memory leaks. Also objects which are GCed will be objects that can be GCed. Live / referenced objects which escape the explicit GC context will not be GCed. The objective is that: - The GC check happens at define points in code - GC happens at define points if possible, else fall back to the system GC mechanism -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20130419/1d87061a/attachment.html From kirk at kodewerk.com Fri Apr 19 02:50:11 2013 From: kirk at kodewerk.com (Kirk Pepperdine) Date: Fri, 19 Apr 2013 11:50:11 +0200 Subject: javax GC API In-Reply-To: References: Message-ID: <31A020B9-F7F2-4DC4-9EE8-E8ABF1D25A69@kodewerk.com> Hi Suminda, With all due respect, I'm not sure that it's a great idea to pass control of the garbage collectors over to the programmer. In this case, if you were to ensure that GC did not happen during the execution of a block you are in danger of filling heap. A better way to prevent GC is to not create objects. Regards, Kirk On 2013-04-19, at 11:10 AM, Suminda Dharmasena wrote: > Another potential tag to ensure GC does not happen during the execution of a code block: > GCSuspend - a synchronous object can be used so that the GC threads do not run while the code block is executing > Of course some revision to current annotations will be needed to accommodate this. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20130419/39198e1c/attachment.html From sirinath at sakrio.com Fri Apr 19 01:06:22 2013 From: sirinath at sakrio.com (Suminda Dharmasena) Date: Fri, 19 Apr 2013 13:36:22 +0530 Subject: javax GC API Message-ID: Hi, Any thoughts plans to add API accessibility to the GC subsystems and the ability to add our own GC. Looking for the comments from the community and Oracle. In addition I like to propose introducing the following annotations for more predictable GC overheads: - GCAtBlockEnd - GCAtBlockExit - for loops the GC happens when the - GCAtLastReference - GCAtEndOfScope - GCOnAssignment - GCOnReturn - GCOnGCOfContainingObject - GC on GC of aggregate / composing object - GCOnStatementCompletion - GC parameters and return value (if not assigned) of a call when the function returns. Need to support statement level annotations. - GCOnLastResort - Only GC at last resort before outofmemory exception - class level - ExcludeFinalising - prevent finalizers running on the object - class level - GCUsingCurrentThread - GCUsingDefaultThread - GCUsingNewThread(Priority = n) To do this annotations need to be extended to local variables and for some application at the statement / block level. At each point the objects pointed by the handle are GCed if there is a possibility (reference count becomes 0). If a reference to an object is specified to be Suminda -- Suminda Sirinath Salpitikorala Dharmasena, B.Sc. Comp. & I.S. (Hon.) Lond., P.G.Dip. Ind. Maths. J'Pura, MIEEE, MACM, CEO Sakr??! ? *Address*: 6G ? 1st Lane ? Pagoda Road ? Nugegoda 10250 ? Sri Lanka. ? *Tele*: +94-(0)11-5 864614 / 5 875614 / 2 825908 ? *Web*: http://www.sakrio.com ? This email is subjected to the email Terms of Use and Disclaimer: http://www.sakrio.com/email-legal. Please read this first. -- -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20130419/4ed25cca/attachment.html From sirinath at sakrio.com Mon Apr 22 09:27:42 2013 From: sirinath at sakrio.com (Suminda Dharmasena) Date: Mon, 22 Apr 2013 21:57:42 +0530 Subject: javax GC API In-Reply-To: References: <31A020B9-F7F2-4DC4-9EE8-E8ABF1D25A69@kodewerk.com> Message-ID: Hi, Proper escape analysis can eliminate the need of GC. I cannot understand why Oracle (Sun before oracle) choose to carry the burden of the GC. This creates: - Memory footprint - Processing footprint - GC pause With proper escape analysis and allocating in the stack GC can be eliminated in most cases. Also finalising can be eliminated. Annotations can be used what point exactly the resource is released. This will take only fraction of nanoseconds but this also will be known. Also collection can happen in a deferred mode as this is what needs to be GCed. Ideally this should be reserved for IO pause applications which can run the GC thread on pause. Suminda -- Suminda Sirinath Salpitikorala Dharmasena, B.Sc. Comp. & I.S. (Hon.) Lond., P.G.Dip. Ind. Maths. J'Pura, MIEEE, MACM, CEO Sakr??! ? *Address*: 6G ? 1st Lane ? Pagoda Road ? Nugegoda 10250 ? Sri Lanka. ? *Tele*: +94-(0)11-5 864614 / 5 875614 / 2 825908 ? *Web*: http://www.sakrio.com ? This email is subjected to the email Terms of Use and Disclaimer: http://www.sakrio.com/email-legal. Please read this first. -- On 22 April 2013 19:45, Ryan Gardner wrote: > On Fri, Apr 19, 2013 at 6:04 AM, Suminda Dharmasena wrote: > >> Also if the programmer fails to wisely GC using API / Annotations then >> the fallback is the default GC mechanism. This way there will be no memory >> leaks. Also objects which are GCed will be objects that can be GCed. Live / >> referenced objects which escape the explicit GC context will not be GCed. >> >> The objective is that: >> >> - The GC check happens at define points in code >> - GC happens at define points if possible, else fall back to the >> system GC mechanism >> >> How will this work with multithreaded environments like a web server? > > The stop-the-world phases of GC by definition stop more than just one > thread? > > Also this seems very JVM vendor specific - I can't imagine the Azul guys > (who market their JVM as if GC pauses don't exist at all on their VM) would > care about this, nor would the IBM guys likely care. > > The specific annotations proposed also do not follow the best practice for > naming annotations - they explicitly define a specific behavior to take as > a result of the annotation being applied. > > I'm just a community member, and not a hotspot engineer or contributor - > but I can wholeheartedly say that I would never use these annotations and I > would never use code from any open source project that littered their > project with these annotations. > > > >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20130422/efc5812f/attachment.html From nadav.wiener+hotspot at gmail.com Mon Apr 29 01:05:55 2013 From: nadav.wiener+hotspot at gmail.com (Nadav Wiener) Date: Mon, 29 Apr 2013 11:05:55 +0300 Subject: Can CMS maximum free chunk size provide advance warning before full GC? Message-ID: For the context of a soft real time system that should not pause for more than 200ms, we're looking for a way to have an advance warning before a Full GC is imminent. We realize we might not be able to avoid it, but we'd like to fail over to another node before the system stalls. We've been able to come up with a scheme that will provide us with an advance warning, ahead of imminent full GC that may cause the system to stall for several seconds (which we need to avoid). What we've been able to come up with relies on CMS free list statistics: -XX:PrintFLSStatistics=1. This prints free list statistics into the GC log after every GC cycle, including young GC, so the information is available at short intervals, and will appear even more frequently during intervals of high memory allocation rate. It probably costs a little in terms of performance, but our working assumption is that we can afford it. The output to the log looks like so: Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 382153298 Max Chunk Size: 382064598 Number of Blocks: 28 Av. Block Size: 13648332 Tree Height: 8 In particular, the maximum free chunk size is 382064598 words. With 64-bit words this should amount to just below 2915MB. This number has been decreasing very slowly, at a rate of roughly 1MB per hour. It is our understanding that so long as the maximum free chunk size is larger than the young generation (assuming no humungous object allocation), every object promotion should succeed. Recently, we've run a several-days-long stress tests, and have been seeing that CMS was able to maintain maximum chunk sizes upward of 94% of total old region space. The maximum free chunk size appears to be decreasing at a rate of less than 1MB/hour, which should be fine -- according to this we won't be hitting full GC any time soon, and the servers will likely be down for maintenance more frequently than full GC can occur. In a previous test, at a time when the system was less memory efficient, we've been able to run the system for a good 10 hours. During the first hour, the maximum free chunk size has decreased to 100MB, where it stayed for over 8 hours. During the last 40 minutes of the run, the maximum free chunk size has decreased at a steady rate towards 0, when a full GC occurred -- this was very encouraging, because for that workload we seemed to be able to get a 40 minute advance warning (when the chunk size started a steady decline towards 0). **My question to you**: assuming this all reflects a prolonged peak workload (workload at any given point in time in production will only be lower), does this sound like a valid approach? To what degree of reliability do you reckon we should be able to count on the maximum free chunk size statistic from the GC log? We are definitely open for suggestions, but request that they be limited to solutions available on HotSpot (No Azul for us, at least for now). Also, G1 by itself is no solution unless we can come up with a similar metric that will give us advance warning before Full GCs, or any GCs that significantly exceed our SLA (and these can occasionally occur). -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20130429/1bac0731/attachment.html From jon.masamitsu at oracle.com Mon Apr 29 11:53:38 2013 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Mon, 29 Apr 2013 11:53:38 -0700 Subject: Can CMS maximum free chunk size provide advance warning before full GC? In-Reply-To: References: Message-ID: <517EC1B2.90001@oracle.com> On 4/29/13 1:05 AM, Nadav Wiener wrote: > For the context of a soft real time system that should not pause for > more than 200ms, we're looking for a way to have an advance warning > before a Full GC is imminent. We realize we might not be able to avoid > it, but we'd like to fail over to another node before the system stalls. > > We've been able to come up with a scheme that will provide us with an > advance warning, ahead of imminent full GC that may cause the system > to stall for several seconds (which we need to avoid). > > What we've been able to come up with relies on CMS free list > statistics: -XX:PrintFLSStatistics=1. This prints free > list statistics into the GC log after every GC cycle, including young > GC, so the information is available at short intervals, and will > appear even more frequently during intervals of high memory allocation > rate. It probably costs a little in terms of performance, but our > working assumption is that we can afford it. > > The output to the log looks like so: > > Statistics for BinaryTreeDictionary: > ------------------------------------ > Total Free Space: 382153298 > Max Chunk Size: 382064598 > Number of Blocks: 28 > Av. Block Size: 13648332 > Tree Height: 8 > > In particular, the maximum free chunk size is 382064598 words. With > 64-bit words this should amount to just below 2915MB. This number has > been decreasing very slowly, at a rate of roughly 1MB per hour. > > It is our understanding that so long as the maximum free chunk size is > larger than the young generation (assuming no humungous object > allocation), every object promotion should succeed. To a very large degree this is correct. There are circumstances under which an object promoted from the young generation into the CMS generation will require more space in the CMS generation than it did in the young generation. I don't think this happens to a significant extent. > > Recently, we've run a several-days-long stress tests, and have been > seeing that CMS was able to maintain maximum chunk sizes upward of 94% > of total old region space. The maximum free chunk size appears to be > decreasing at a rate of less than 1MB/hour, which should be fine -- > according to this we won't be hitting full GC any time soon, and the > servers will likely be down for maintenance more frequently than full > GC can occur. > > In a previous test, at a time when the system was less memory > efficient, we've been able to run the system for a good 10 hours. > During the first hour, the maximum free chunk size has decreased to > 100MB, where it stayed for over 8 hours. During the last 40 minutes of > the run, the maximum free chunk size has decreased at a steady rate > towards 0, when a full GC occurred -- this was very encouraging, > because for that workload we seemed to be able to get a 40 minute > advance warning (when the chunk size started a steady decline towards 0). Out of curiosity, when you say "less memory efficient" do you mean that the previous system actually used more objects in the Java heap? Or did you do something to make the objects of fewer sizes? Meaning, if before you had objects of sizes N, N+1, N+2, N+3, now you pad them out to all be of size N+3. > > **My question to you**: assuming this all reflects a prolonged peak > workload (workload at any given point in time in production will only > be lower), does this sound like a valid approach? To what degree of > reliability do you reckon we should be able to count on the maximum > free chunk size statistic from the GC log? > The maximum free chunk size is exact at the time GC prints it, but it can be stale by the time you read it and make your decisions. > We are definitely open for suggestions, but request that they be > limited to solutions available on HotSpot (No Azul for us, at least > for now). Also, G1 by itself is no solution unless we can come up with > a similar metric that will give us advance warning before Full GCs, or > any GCs that significantly exceed our SLA (and these can occasionally > occur). I think that the use of maximum free chunk size as your metric is a good choice. It is very conservative (which sounds like what you want) and not subject to odd mixtures of object sizes. For G1 I think you could use the number of completely free regions. I don't know if it is printed in any of the logs currently but it is probably a metric we maintain (or could easily). If the number of completely free regions decreases over time, it could signal that a full GC is coming. Jon > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20130429/a1e2031d/attachment.html From bernd-2013 at eckenfels.net Mon Apr 29 14:53:05 2013 From: bernd-2013 at eckenfels.net (Bernd Eckenfels) Date: Mon, 29 Apr 2013 23:53:05 +0200 Subject: Can CMS maximum free chunk size provide advance warning before full GC? In-Reply-To: <517EC1B2.90001@oracle.com> References: <517EC1B2.90001@oracle.com> Message-ID: Am 29.04.2013, 20:53 Uhr, schrieb Jon Masamitsu : > I think that the use of maximum free chunk size as your metric is a good > choice. It is very conservative (which sounds like what you want) and > not subject to odd mixtures of object sizes. > > For G1 I think you could use the number of completely free regions. > I don't know if it is printed in any of the logs currently but it is > probably > a metric we maintain (or could easily). If the number of completely free > regions decreases over time, it could signal that a full GC is coming. And it would be great to have both in the (Same) MXBean Attribute for application self monitoring. Gruss Bernd -- http://bernd.eckenfels.net