From anjul_narain at yahoo.com Tue Oct 4 11:35:39 2011 From: anjul_narain at yahoo.com (Anjul Narain Varma) Date: Tue, 4 Oct 2011 11:35:39 -0700 (PDT) Subject: OutOfMemoryError : Out of swap space Message-ID: <1317753339.20158.YahooMailNeo@web114419.mail.gq1.yahoo.com> Hello, ? I am intermittently getting the below Out of Memory error for a Java application running on Windows 2003 with 4 GB RAM. ? jvm 2??? | 2011/10/03 02:08:00 | # jvm 2??? | 2011/10/03 02:08:00 | # An unexpected error has been detected by Java Runtime Environment: jvm 2??? | 2011/10/03 02:08:00 | # jvm 2??? | 2011/10/03 02:08:00 | # java.lang.OutOfMemoryError: requested 4096000 bytes for GrET in C:\BUILD_AREA\jdk6_07\hotspot\src\share\vm\utilities\growableArray.cpp. Out of swap space? jvm 2??? | 2011/10/03 02:08:00 | # jvm 2??? | 2011/10/03 02:08:00 | #? Internal Error (allocation.inline.hpp:42), pid=3224, tid=3964 jvm 2??? | 2011/10/03 02:08:00 | #? Error: GrET in C:\BUILD_AREA\jdk6_07\hotspot\src\share\vm\utilities\growableArray.cpp jvm 2??? | 2011/10/03 02:08:00 | # JDK version: 1.6.0_07 ? JVM?settings?are: ?-Xmx16m -Xmx1600m. ? Reviewing the bug reports, it seems to have been fixed in jdk 1.6.0_21, however this is production machine and we don't want to go for an upgrade presently. One of the workaround we came across was to make the -Xms=-Xmx value? Is there any way to find what?value would work for us? Please note, that we don't have any other process(s) running on this box except for the java application. ? Thank you. Anjul Varma -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20111004/cec4b75a/attachment.html From lawrence.chow at oracle.com Wed Oct 5 12:03:26 2011 From: lawrence.chow at oracle.com (lawrence.chow at oracle.com) Date: Wed, 5 Oct 2011 12:03:26 -0700 (PDT) Subject: Auto Reply: hotspot-gc-use Digest, Vol 44, Issue 2 Message-ID: Lawrence Chow will be out of the office on 10/02/11 through 10/06/11 Lawrence will return to the office on Friday, 10/07/11. Please contact Matt.Miller at oracle.com, Terry.Statt at oracle.com, or Mary.McCarthy at oracle.com if assistance is needed from a Java collaborator in my absence. From y.s.ramakrishna at oracle.com Thu Oct 6 09:50:36 2011 From: y.s.ramakrishna at oracle.com (Y. S. Ramakrishna) Date: Thu, 06 Oct 2011 09:50:36 -0700 Subject: OutOfMemoryError : Out of swap space In-Reply-To: <1317753339.20158.YahooMailNeo@web114419.mail.gq1.yahoo.com> References: <1317753339.20158.YahooMailNeo@web114419.mail.gq1.yahoo.com> Message-ID: <4E8DDC5C.7090902@oracle.com> You are likely out of C heap. If you have native code you might want to check if there's a leak. The JVM also allocates out of native heap and it is possible that a bug is causing a leak of C heap space. You could see if you could run with a 64-bit JVM and then monitor the C heap usage to see if it shows evidence of unbounded growth. Xms=Xmx will likely not make any difference in this case, although a smaller Xmx might allow you to run longer by possibly freeing up more virtual address space for the C heap. good luck. -- ramki On 10/04/11 11:35, Anjul Narain Varma wrote: > Hello, > > I am intermittently getting the below Out of Memory error for a Java > application running on Windows 2003 with 4 GB RAM. > > jvm 2 | 2011/10/03 02:08:00 | # > jvm 2 | 2011/10/03 02:08:00 | # An unexpected error has been detected > by Java Runtime Environment: > jvm 2 | 2011/10/03 02:08:00 | # > jvm 2 | 2011/10/03 02:08:00 | # java.lang.OutOfMemoryError: requested > 4096000 bytes for GrET in > C:\BUILD_AREA\jdk6_07\hotspot\src\share\vm\utilities\growableArray.cpp. > Out of swap space? > jvm 2 | 2011/10/03 02:08:00 | # > jvm 2 | 2011/10/03 02:08:00 | # Internal Error > (allocation.inline.hpp:42), pid=3224, tid=3964 > jvm 2 | 2011/10/03 02:08:00 | # Error: GrET in > C:\BUILD_AREA\jdk6_07\hotspot\src\share\vm\utilities\growableArray.cpp > jvm 2 | 2011/10/03 02:08:00 | # > JDK version: 1.6.0_07 > > JVM settings are: > -Xmx16m -Xmx1600m. > > Reviewing the bug reports, it seems to have been fixed in jdk 1.6.0_21, > however this is production machine and we don't want to go for an > upgrade presently. > One of the workaround we came across was to make the -Xms=-Xmx value? Is > there any way to find what value would work for us? > Please note, that we don't have any other process(s) running on this box > except for the java application. > > Thank you. > Anjul Varma > > > ------------------------------------------------------------------------ > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use From bdimitri at cisco.com Fri Oct 28 01:22:03 2011 From: bdimitri at cisco.com (Bogdan Dimitriu (bdimitri)) Date: Fri, 28 Oct 2011 10:22:03 +0200 Subject: OutOfMemoryError : Out of swap space In-Reply-To: <1317753339.20158.YahooMailNeo@web114419.mail.gq1.yahoo.com> References: <1317753339.20158.YahooMailNeo@web114419.mail.gq1.yahoo.com> Message-ID: Are you using any non-JDK libraries that make native calls? If so check if these don't have an bugs that leak native memory in those areas. Bogdan From: hotspot-gc-use-bounces at openjdk.java.net [mailto:hotspot-gc-use-bounces at openjdk.java.net] On Behalf Of Anjul Narain Varma Sent: 04 October 2011 19:36 To: hotspot-gc-use at openjdk.java.net Subject: OutOfMemoryError : Out of swap space Hello, I am intermittently getting the below Out of Memory error for a Java application running on Windows 2003 with 4 GB RAM. jvm 2 | 2011/10/03 02:08:00 | # jvm 2 | 2011/10/03 02:08:00 | # An unexpected error has been detected by Java Runtime Environment: jvm 2 | 2011/10/03 02:08:00 | # jvm 2 | 2011/10/03 02:08:00 | # java.lang.OutOfMemoryError: requested 4096000 bytes for GrET in C:\BUILD_AREA\jdk6_07\hotspot\src\share\vm\utilities\growableArray.cpp. Out of swap space? jvm 2 | 2011/10/03 02:08:00 | # jvm 2 | 2011/10/03 02:08:00 | # Internal Error (allocation.inline.hpp:42), pid=3224, tid=3964 jvm 2 | 2011/10/03 02:08:00 | # Error: GrET in C:\BUILD_AREA\jdk6_07\hotspot\src\share\vm\utilities\growableArray.cpp jvm 2 | 2011/10/03 02:08:00 | # JDK version: 1.6.0_07 JVM settings are: -Xmx16m -Xmx1600m. Reviewing the bug reports, it seems to have been fixed in jdk 1.6.0_21, however this is production machine and we don't want to go for an upgrade presently. One of the workaround we came across was to make the -Xms=-Xmx value? Is there any way to find what value would work for us? Please note, that we don't have any other process(s) running on this box except for the java application. Thank you. Anjul Varma -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20111028/398ecea6/attachment.html From jon at siliconcircus.com Mon Oct 31 06:06:54 2011 From: jon at siliconcircus.com (Jon Bright) Date: Mon, 31 Oct 2011 14:06:54 +0100 Subject: Identifying concurrent mode failures caused by fragmentation Message-ID: <4EAE9D6E.1060807@siliconcircus.com> Hi, We have an application running with a 6GB heap (complete parameters below). Mostly it has a fairly low turnover of memory use, but on occasion, it will come under some pressure as it reloads a large in-memory data set from a database. Sometimes in this situation, we'll see a concurrent mode failure. Here's one failure: 20021.464: [GC 20021.465: [ParNew: 13093K->3939K(76672K), 0.0569240 secs]20021.522: [CMS20023.747: [CMS-concurrent-mark: 11.403/29.029 secs] [Times: user=41.11 sys=1.03, real=29.03 secs] (concurrent mode failure): 3873922K->2801744K(6206272K), 30.7900180 secs] 3886215K->2801744K(6282944K), [CMS Perm : 142884K->142834K(524288K)] icms_dc=33 , 30.8473830 secs] [Times: user=30.26 sys=0.71, real=30.85 secs] Total time for which application threads were stopped: 30.8484460 seconds (I've attached a lengthier log including the previous and subsequent CMS collection.) Am I correct in thinking that this failure can basically only be caused by fragmentation? Both young and old seem to have plenty of space. There doesn't seem to be any sign that the tenured generation would run out of space before CMS completes. Fragmentation is the only remaining cause that occurs to me. We're running with 1.6.0_11, although this will be upgraded to 1.6.0_26 tomorrow. I realise our current version is ancient - I'm not really looking for help on the problem itself, just for advice on whether the log line above indicates fragmentation. Thanks Jon Bright The parameters we have set are: -server -Xmx6144M -Xms6144M -XX:MaxPermSize=512m -XX:PermSize=512m -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing -XX:SoftRefLRUPolicyMSPerMB=3 -XX:CMSIncrementalSafetyFactor=30 -XX:+PrintGCDetails -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCTimeStamps -Xloggc:/home/tbmx/log/gc_`date +%Y%m%d%H%M`.log -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: gc_20111031_jonbright.log Url: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20111031/474c9fe8/gc_20111031_jonbright-0001.log From jon.masamitsu at oracle.com Mon Oct 31 19:43:25 2011 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Mon, 31 Oct 2011 19:43:25 -0700 Subject: Identifying concurrent mode failures caused by fragmentation In-Reply-To: <4EAE9D6E.1060807@siliconcircus.com> References: <4EAE9D6E.1060807@siliconcircus.com> Message-ID: <4EAF5CCD.5030004@oracle.com> Jon, I haven't looked at the longer log but in general I've found the information in the GC logs inadequate to figure out if the problem is fragmentation. But more important, there has been some good work in recent versions of hotspot so that we're more successful at combating fragmentation. Try the latest release and see if it helps (u26 should be good enough). Jon On 10/31/11 06:06, Jon Bright wrote: > Hi, > > We have an application running with a 6GB heap (complete parameters > below). Mostly it has a fairly low turnover of memory use, but on > occasion, it will come under some pressure as it reloads a large > in-memory data set from a database. > > Sometimes in this situation, we'll see a concurrent mode failure. > Here's one failure: > > 20021.464: [GC 20021.465: [ParNew: 13093K->3939K(76672K), 0.0569240 > secs]20021.522: [CMS20023.747: [CMS-concurrent-mark: 11.403/29.029 > secs] [Times: user=41.11 sys=1.03, real=29.03 secs] > (concurrent mode failure): 3873922K->2801744K(6206272K), 30.7900180 > secs] 3886215K->2801744K(6282944K), [CMS Perm : > 142884K->142834K(524288K)] icms_dc=33 , 30.8473830 secs] [Times: > user=30.26 sys=0.71, real=30.85 secs] > Total time for which application threads were stopped: 30.8484460 seconds > > (I've attached a lengthier log including the previous and subsequent > CMS collection.) > > Am I correct in thinking that this failure can basically only be > caused by fragmentation? Both young and old seem to have plenty of > space. There doesn't seem to be any sign that the tenured generation > would run out of space before CMS completes. Fragmentation is the > only remaining cause that occurs to me. > > We're running with 1.6.0_11, although this will be upgraded to > 1.6.0_26 tomorrow. I realise our current version is ancient - I'm not > really looking for help on the problem itself, just for advice on > whether the log line above indicates fragmentation. > > Thanks > > Jon Bright > > > > The parameters we have set are: > > -server > -Xmx6144M > -Xms6144M > -XX:MaxPermSize=512m > -XX:PermSize=512m > -XX:+UseConcMarkSweepGC > -XX:+CMSIncrementalMode > -XX:+CMSIncrementalPacing > -XX:SoftRefLRUPolicyMSPerMB=3 > -XX:CMSIncrementalSafetyFactor=30 > -XX:+PrintGCDetails > -XX:+PrintGCApplicationStoppedTime > -XX:+PrintGCApplicationConcurrentTime > -XX:+PrintGCTimeStamps > -Xloggc:/home/tbmx/log/gc_`date +%Y%m%d%H%M`.log > > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20111031/b28d5b97/attachment.html From rednaxelafx at gmail.com Mon Oct 31 20:21:20 2011 From: rednaxelafx at gmail.com (Krystal Mok) Date: Tue, 1 Nov 2011 11:21:20 +0800 Subject: Identifying concurrent mode failures caused by fragmentation In-Reply-To: <4EAE9D6E.1060807@siliconcircus.com> References: <4EAE9D6E.1060807@siliconcircus.com> Message-ID: Hi Jon, It might be helpful to set -XX:PrintFLSStatistics to a value greater than zero, to get the stats of FreeListSpace so that you'd know the size of the biggest fragment. The GC log produced by -XX:+PrintGCDetails doesn't give enough information on fragmentation. Here's an example of using -XX:PrintFLSStatistics=1: https://gist.github.com/1329783 It does make the GC log messier, and some of the GC log parsing tools won't cope with this, but you get to know how bad the fragmentation is. Anyway, it looks like you're using CMS in incremental mode. This mode should be obsolete in JDK6 already. Is there a good reason for you to be using it? If not, I'd suggest turning it off, though, no matter if you're upgrading your JDK or not. Regards, Kris Mok On Mon, Oct 31, 2011 at 9:06 PM, Jon Bright wrote: > Hi, > > We have an application running with a 6GB heap (complete parameters > below). Mostly it has a fairly low turnover of memory use, but on > occasion, it will come under some pressure as it reloads a large in-memory > data set from a database. > > Sometimes in this situation, we'll see a concurrent mode failure. Here's > one failure: > > 20021.464: [GC 20021.465: [ParNew: 13093K->3939K(76672K), 0.0569240 > secs]20021.522: [CMS20023.747: [CMS-concurrent-mark: 11.403/29.029 secs] > [Times: user=41.11 sys=1.03, real=29.03 secs] > (concurrent mode failure): 3873922K->2801744K(6206272K), 30.7900180 secs] > 3886215K->2801744K(6282944K), [CMS Perm : 142884K->142834K(524288K)] > icms_dc=33 , 30.8473830 secs] [Times: user=30.26 sys=0.71, real=30.85 secs] > Total time for which application threads were stopped: 30.8484460 seconds > > (I've attached a lengthier log including the previous and subsequent CMS > collection.) > > Am I correct in thinking that this failure can basically only be caused by > fragmentation? Both young and old seem to have plenty of space. There > doesn't seem to be any sign that the tenured generation would run out of > space before CMS completes. Fragmentation is the only remaining cause that > occurs to me. > > We're running with 1.6.0_11, although this will be upgraded to 1.6.0_26 > tomorrow. I realise our current version is ancient - I'm not really > looking for help on the problem itself, just for advice on whether the log > line above indicates fragmentation. > > Thanks > > Jon Bright > > > > The parameters we have set are: > > -server > -Xmx6144M > -Xms6144M > -XX:MaxPermSize=512m > -XX:PermSize=512m > -XX:+UseConcMarkSweepGC > -XX:+CMSIncrementalMode > -XX:+CMSIncrementalPacing > -XX:SoftRefLRUPolicyMSPerMB=3 > -XX:**CMSIncrementalSafetyFactor=30 > -XX:+PrintGCDetails > -XX:+**PrintGCApplicationStoppedTime > -XX:+**PrintGCApplicationConcurrentTi**me > -XX:+PrintGCTimeStamps > -Xloggc:/home/tbmx/log/gc_`**date +%Y%m%d%H%M`.log > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20111101/50c254a5/attachment.html From jon.masamitsu at oracle.com Mon Oct 31 21:10:49 2011 From: jon.masamitsu at oracle.com (Jon Masamitsu) Date: Mon, 31 Oct 2011 21:10:49 -0700 Subject: Identifying concurrent mode failures caused by fragmentation In-Reply-To: <4EAF5CCD.5030004@oracle.com> References: <4EAE9D6E.1060807@siliconcircus.com> <4EAF5CCD.5030004@oracle.com> Message-ID: <4EAF7149.6020008@oracle.com> Jon, I noticed that you are using the incremental mode of CMS. May I ask why you chose to turn that on? Jon On 10/31/2011 7:43 PM, Jon Masamitsu wrote: > Jon, > > I haven't looked at the longer log but in general I've found the > information in the GC logs inadequate to figure out if the > problem is fragmentation. But more important, there has > been some good work in recent versions of hotspot so that > we're more successful at combating fragmentation. Try > the latest release and see if it helps (u26 should be good > enough). > > Jon > > On 10/31/11 06:06, Jon Bright wrote: >> Hi, >> >> We have an application running with a 6GB heap (complete parameters >> below). Mostly it has a fairly low turnover of memory use, but on >> occasion, it will come under some pressure as it reloads a large >> in-memory data set from a database. >> >> Sometimes in this situation, we'll see a concurrent mode failure. >> Here's one failure: >> >> 20021.464: [GC 20021.465: [ParNew: 13093K->3939K(76672K), 0.0569240 >> secs]20021.522: [CMS20023.747: [CMS-concurrent-mark: 11.403/29.029 >> secs] [Times: user=41.11 sys=1.03, real=29.03 secs] >> (concurrent mode failure): 3873922K->2801744K(6206272K), 30.7900180 >> secs] 3886215K->2801744K(6282944K), [CMS Perm : >> 142884K->142834K(524288K)] icms_dc=33 , 30.8473830 secs] [Times: >> user=30.26 sys=0.71, real=30.85 secs] >> Total time for which application threads were stopped: 30.8484460 >> seconds >> >> (I've attached a lengthier log including the previous and subsequent >> CMS collection.) >> >> Am I correct in thinking that this failure can basically only be >> caused by fragmentation? Both young and old seem to have plenty of >> space. There doesn't seem to be any sign that the tenured generation >> would run out of space before CMS completes. Fragmentation is the >> only remaining cause that occurs to me. >> >> We're running with 1.6.0_11, although this will be upgraded to >> 1.6.0_26 tomorrow. I realise our current version is ancient - I'm >> not really looking for help on the problem itself, just for advice on >> whether the log line above indicates fragmentation. >> >> Thanks >> >> Jon Bright >> >> >> >> The parameters we have set are: >> >> -server >> -Xmx6144M >> -Xms6144M >> -XX:MaxPermSize=512m >> -XX:PermSize=512m >> -XX:+UseConcMarkSweepGC >> -XX:+CMSIncrementalMode >> -XX:+CMSIncrementalPacing >> -XX:SoftRefLRUPolicyMSPerMB=3 >> -XX:CMSIncrementalSafetyFactor=30 >> -XX:+PrintGCDetails >> -XX:+PrintGCApplicationStoppedTime >> -XX:+PrintGCApplicationConcurrentTime >> -XX:+PrintGCTimeStamps >> -Xloggc:/home/tbmx/log/gc_`date +%Y%m%d%H%M`.log >> >> >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20111031/40c89aab/attachment.html