From Jacob.Kessler at Sun.COM Fri Jun 12 13:05:29 2009 From: Jacob.Kessler at Sun.COM (Jacob Kessler) Date: Fri, 12 Jun 2009 13:05:29 -0700 Subject: Classes not unloading correctly? Message-ID: <4A32B509.8070107@sun.com> I'm working on the GlassFish scripting project, trying to solve an issue with the PermGen around deploying and undeploying applications. We use an embedded JRuby interpreter to host each deployed Ruby application, and JRuby makes extensive use of JIT-ed classes to improve performance. A typical JRuby instance takes roughly 20MB of permgen. We'd like for that space to be reclaimed once the application is undeployed, since losing 20MB of permgen each time an application is deployed puts a fairly hard limit on the number of redeploys that the server can take before it has to be restarted. We started out using -XX:+UseConcMarkSweepGC and -XX:+CMSClassUnloadingEnabled, and with the aid of a memory analysis tool (YourKit), we confirmed that some classes were being unloaded (not a significant number, though), and we tracked down and fixed two things that were improperly holding references to the classloader that contained JRuby. After fixing those, though, we ended up in a situation where YourKit was reporting the classes as still in memory after a forced full collection, but as having no paths to the GC roots. This situation seemed able to persist indefinitely (so, beyond time for the finalizer queue to drain). Does anyone have any ideas on what might be preventing the classloader from collecting? I'm using $java -version java version "1.6.0_07" Java(TM) SE Runtime Environment (build 1.6.0_07-b06) Java HotSpot(TM) 64-Bit Server VM (build 10.0-b23, mixed mode) Thank you for any help with this. From Y.S.Ramakrishna at Sun.COM Fri Jun 12 13:07:04 2009 From: Y.S.Ramakrishna at Sun.COM (Y. Srinivas Ramakrishna) Date: Fri, 12 Jun 2009 13:07:04 -0700 Subject: Classes not unloading correctly? In-Reply-To: <4A32B509.8070107@sun.com> References: <4A32B509.8070107@sun.com> Message-ID: <4A32B568.5060300@sun.com> Hi Jacob -- could you check if the same behaviour obtains with the -client JIT-compiler as well. It is possible that this is related to 4957990. I'll contact you off-line with a JVM that has a tentative fix for that bug to see if it makes any difference to this problem. -- ramki Jacob Kessler wrote: > I'm working on the GlassFish scripting project, trying to solve an issue > with the PermGen around deploying and undeploying applications. > > We use an embedded JRuby interpreter to host each deployed Ruby > application, and JRuby makes extensive use of JIT-ed classes to improve > performance. A typical JRuby instance takes roughly 20MB of permgen. > We'd like for that space to be reclaimed once the application is > undeployed, since losing 20MB of permgen each time an application is > deployed puts a fairly hard limit on the number of redeploys that the > server can take before it has to be restarted. > > We started out using -XX:+UseConcMarkSweepGC and > -XX:+CMSClassUnloadingEnabled, and with the aid of a memory analysis > tool (YourKit), we confirmed that some classes were being unloaded (not > a significant number, though), and we tracked down and fixed two things > that were improperly holding references to the classloader that > contained JRuby. After fixing those, though, we ended up in a situation > where YourKit was reporting the classes as still in memory after a > forced full collection, but as having no paths to the GC roots. This > situation seemed able to persist indefinitely (so, beyond time for the > finalizer queue to drain). Does anyone have any ideas on what might be > preventing the classloader from collecting? > > I'm using > $java -version > java version "1.6.0_07" > Java(TM) SE Runtime Environment (build 1.6.0_07-b06) > Java HotSpot(TM) 64-Bit Server VM (build 10.0-b23, mixed mode) > > Thank you for any help with this. > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > From Antonios.Printezis at sun.com Fri Jun 12 13:42:07 2009 From: Antonios.Printezis at sun.com (Tony Printezis) Date: Fri, 12 Jun 2009 16:42:07 -0400 Subject: Classes not unloading correctly? In-Reply-To: <4A32B509.8070107@sun.com> References: <4A32B509.8070107@sun.com> Message-ID: <4A32BD9F.3020703@sun.com> Great to see a second Kessler on this list! Welcome, Jacob. Tony Jacob Kessler wrote: > I'm working on the GlassFish scripting project, trying to solve an issue > with the PermGen around deploying and undeploying applications. > > We use an embedded JRuby interpreter to host each deployed Ruby > application, and JRuby makes extensive use of JIT-ed classes to improve > performance. A typical JRuby instance takes roughly 20MB of permgen. > We'd like for that space to be reclaimed once the application is > undeployed, since losing 20MB of permgen each time an application is > deployed puts a fairly hard limit on the number of redeploys that the > server can take before it has to be restarted. > > We started out using -XX:+UseConcMarkSweepGC and > -XX:+CMSClassUnloadingEnabled, and with the aid of a memory analysis > tool (YourKit), we confirmed that some classes were being unloaded (not > a significant number, though), and we tracked down and fixed two things > that were improperly holding references to the classloader that > contained JRuby. After fixing those, though, we ended up in a situation > where YourKit was reporting the classes as still in memory after a > forced full collection, but as having no paths to the GC roots. This > situation seemed able to persist indefinitely (so, beyond time for the > finalizer queue to drain). Does anyone have any ideas on what might be > preventing the classloader from collecting? > > I'm using > $java -version > java version "1.6.0_07" > Java(TM) SE Runtime Environment (build 1.6.0_07-b06) > Java HotSpot(TM) 64-Bit Server VM (build 10.0-b23, mixed mode) > > Thank you for any help with this. > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > -- --------------------------------------------------------------------- | Tony Printezis, Staff Engineer | Sun Microsystems Inc. | | | MS UBUR02-311 | | e-mail: tony.printezis at sun.com | 35 Network Drive | | office: +1 781 442 0998 (x20998) | Burlington, MA 01803-2756, USA | --------------------------------------------------------------------- e-mail client: Thunderbird (Linux) From java at stolsvik.com Sat Jun 13 17:49:39 2009 From: java at stolsvik.com (=?UTF-8?Q?Endre_St=C3=B8lsvik?=) Date: Sun, 14 Jun 2009 02:49:39 +0200 Subject: Classes not unloading correctly? In-Reply-To: <4A32B509.8070107@sun.com> References: <4A32B509.8070107@sun.com> Message-ID: <1501fdf40906131749x3b04d096l820bad94ce195388@mail.gmail.com> This might be totally off base, but this wouldn't by any chance be the "classic" problem that e.g. Tomcat experiences when reloading a webapp, where static ThreadLocals carrying a value whose class's classloader again references the ThreadLocal - keeping the ClassLoader and its loaded classes from being reclaimed? http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6254531 http://www.jroller.com/tackline/entry/fixing_threadlocal This problem would not have been there if the threadpool that served the webapp wasn't recycled, but instead being dumped and recreated on reload. I've been told that ephemerons could also have been used to solve the problem, had they existed in java..! As mentioned, this is a long shot, but aspects of the mentioned problem looked similar. Endre. On Fri, Jun 12, 2009 at 22:05, Jacob Kessler wrote: > I'm working on the GlassFish scripting project, trying to solve an issue > with the PermGen around deploying and undeploying applications. > > We use an embedded JRuby interpreter to host each deployed Ruby > application, and JRuby makes extensive use of JIT-ed classes to improve > performance. A typical JRuby instance takes roughly 20MB of permgen. > We'd like for that space to be reclaimed once the application is > undeployed, since losing 20MB of permgen each time an application is > deployed puts a fairly hard limit on the number of redeploys that the > server can take before it has to be restarted. > > We started out using -XX:+UseConcMarkSweepGC and > -XX:+CMSClassUnloadingEnabled, and with the aid of a memory analysis > tool (YourKit), we confirmed that some classes were being unloaded (not > a significant number, though), and we tracked down and fixed two things > that were improperly holding references to the classloader that > contained JRuby. After fixing those, though, we ended up in a situation > where YourKit was reporting the classes as still in memory after a > forced full collection, but as having no paths to the GC roots. This > situation seemed able to persist indefinitely (so, beyond time for the > finalizer queue to drain). Does anyone have any ideas on what might be > preventing the classloader from collecting? > > I'm using > $java -version > java version "1.6.0_07" > Java(TM) SE Runtime Environment (build 1.6.0_07-b06) > Java HotSpot(TM) 64-Bit Server VM (build 10.0-b23, mixed mode) > > Thank you for any help with this. > > _______________________________________________ > hotspot-gc-use mailing list > hotspot-gc-use at openjdk.java.net > http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20090614/d45b6a22/attachment.html From Jacob.Kessler at Sun.COM Tue Jun 23 12:27:37 2009 From: Jacob.Kessler at Sun.COM (Jacob Kessler) Date: Tue, 23 Jun 2009 12:27:37 -0700 Subject: Classes not unloading correctly? In-Reply-To: <4A32B568.5060300@sun.com> References: <4A32B509.8070107@sun.com> <4A32B568.5060300@sun.com> Message-ID: <4A412CA9.3030901@sun.com> While I don't think that I can check the behavior with -client (no easy access to 32-bit machines), allowing it to run with the permgen settings recommended in the bug report did allow class unloading to happen, which makes a strong argument for that being the issue. I'd be interested in trying the tentative fix out to see if that fixes things. Y. Srinivas Ramakrishna wrote: > Hi Jacob -- could you check if the same behaviour obtains with the > -client JIT-compiler as > well. It is possible that this is related to 4957990. I'll contact you > off-line with > a JVM that has a tentative fix for that bug to see if it makes any > difference to this > problem. > > -- ramki > > Jacob Kessler wrote: >> I'm working on the GlassFish scripting project, trying to solve an >> issue with the PermGen around deploying and undeploying applications. >> >> We use an embedded JRuby interpreter to host each deployed Ruby >> application, and JRuby makes extensive use of JIT-ed classes to >> improve performance. A typical JRuby instance takes roughly 20MB of >> permgen. We'd like for that space to be reclaimed once the >> application is undeployed, since losing 20MB of permgen each time an >> application is deployed puts a fairly hard limit on the number of >> redeploys that the server can take before it has to be restarted. >> >> We started out using -XX:+UseConcMarkSweepGC and >> -XX:+CMSClassUnloadingEnabled, and with the aid of a memory analysis >> tool (YourKit), we confirmed that some classes were being unloaded >> (not a significant number, though), and we tracked down and fixed two >> things that were improperly holding references to the classloader >> that contained JRuby. After fixing those, though, we ended up in a >> situation where YourKit was reporting the classes as still in memory >> after a forced full collection, but as having no paths to the GC >> roots. This situation seemed able to persist indefinitely (so, beyond >> time for the finalizer queue to drain). Does anyone have any ideas on >> what might be preventing the classloader from collecting? >> >> I'm using >> $java -version >> java version "1.6.0_07" >> Java(TM) SE Runtime Environment (build 1.6.0_07-b06) >> Java HotSpot(TM) 64-Bit Server VM (build 10.0-b23, mixed mode) >> >> Thank you for any help with this. >> >> _______________________________________________ >> hotspot-gc-use mailing list >> hotspot-gc-use at openjdk.java.net >> http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use >> >