PermGen Collection Issues

[Martin Hare-Robertson]

Hi,

I am hitting a "java.lang.OutOfMemoryError: PermGen space" in a situation where I think a great deal of the perm gen is actually eligible for collection.

I am running Tomcat 6 using a 32 bit Hotspot JVM (1.6.0_07). I have had some trouble in the past with classloader leaks when I reload webapps within Tomcat. I have therefore been doing some testing to explicitly reload my webapp many times and then use heap dumps to track down any classloader leaks. I have a script for testing the reloading of my webapp which does the following:

1) Submit 5 login requests to ensure that the webapp is loaded and working.
2) Reload the webapp.
3) Wait 30 seconds to give threads from the old webapp sufficient time to terminate.
4) Goto 1)

I have fixed a number of bugs and now think that my webapp isn't leaking any classloaders. However, when I run this script I find that OutOfMemoryErrors get thrown after ~30 iterations. However, the heap dump which was made at the time when the OutOfMemoryError is thrown shows that all of the old Classloaders are only weakly referenced (according to Eclipse MAT).

This suggests to me that something has gone wrong with the garbage collection that it has thrown an OutOfMemoryError when there was memory which was only weakly referenced which should have been freed.

I ran jstat to record the GC activity and the perm gen ends up stuck at 100% full and the last GC cause is always one of "Permanent Generation Full" and "Last ditch collection".

My GC command line options are as follows:
-Xms600m
-Xmx600m
-XX:+UseMembar
-XX:+UseConcMarkSweepGC
-XX:+UseParNewGC
-XX:CMSInitiatingOccupancyFraction=80
-XX:+UseCMSInitiatingOccupancyOnly
-XX:PermSize=120m
-XX:MaxPermSize=120m
-XX:NewSize=128m
-XX:SurvivorRatio=8
-Xss120k

My issue seems similar to this old bug (http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6545719) which claims to have been fixed in 6u4.

When exactly is a "java.lang.OutOfMemoryError: PermGen space" error allowed to be thrown? Presumably this occurs when a thread attempts to allocate into the perm gen and the perm gen collector is unable to free up sufficient space? When the perm gen is collected would you expect all garbage to be collected or does the collector quit early? I have attached an example graph showing the perm gen occupancy which I am seeing during a test. This seems to show only a small amount of perm gen being freed at each collection.


MartinHR
___

Here is the complete jstat output from my test run:


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120426/69fe168d/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: jstat.out
Type: application/octet-stream
Size: 543377 bytes
Desc: jstat.out
Url : http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120426/69fe168d/jstat-0001.out 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: perm.png
Type: image/png
Size: 2338 bytes
Desc: perm.png
Url : http://mail.openjdk.java.net/pipermail/hotspot-gc-use/attachments/20120426/69fe168d/perm-0001.png