PermGen problem jmap/jhat leads to dead end

[Keith McNeill]

We're having a perm gen problem (maybe also code cache).  Our product 
generates bytecodes and then loads and optionally unloads them.  Running 
stability tests where we repeatably load & unload our generated 
applications we have encountered problems running out of perm gen 
space.  A few years ago we had similar problems but by using jmap/jhat 
we were able to debug and fix this.

This time using jmap/jhat to analyze the problem has lead to a dead end.

In particular we see:

1) There are no instances left of the classes that we load/unload
2) The class objects of the classes that we load/unload hang around, 
they don't seem to get GC'ed
3) The ClassLoader of the loaded classes is hanging around.  Doing a 
"Reference Chains from Rootset" in jhat results in 0 chains.  Unless 
jhat is lying then the classes/classloader should be able to be GC'ed.  
Right?
4) Using jmap -permstat I see the classloaders in question are marked as 
'dead'.  This means that it should be able to be GC'ed
5) Tried Current GC, default GC, G1 GC and basically get the same 
errors.  The G1 GC might error out with code cache problems instead.
6) The IBM JVM in the same scenario seems to be happy
7) If we load the classes (Class.forName()) but don't instantiate them, 
PermGen seems happier but we then run out of Code Cache.  I'm not sure 
if the problems are related.

The tests are run on linux & linux64.  We are using:  
-XX:MaxPermSize=128m -Xms800m -Xmx1500m -XX:+CMSClassUnloadingEnabled 
and appropriate JVM args for the different GC's

Given #3,4 & #6 above I'm thinking that it maybe a hotspot problem. 

What should I try next to track this problem down?

Thanks,

Keith McNeill