PermGen problem jmap/jhat leads to dead end

[Keith McNeill]

java version "1.6.0_16"
Java(TM) SE Runtime Environment (build 1.6.0_16-b01)
Java HotSpot(TM) Server VM (build 14.2-b01, mixed mode)


Y. Srinivas Ramakrishna wrote:
> What's the version of the HotSpot JVM you are running with? (See 
> 4957990.)
>
> -- ramki
>
> On 12/14/09 13:54, Keith McNeill wrote:
>>
>> We're having a perm gen problem (maybe also code cache).  Our product 
>> generates bytecodes and then loads and optionally unloads them.  
>> Running stability tests where we repeatably load & unload our 
>> generated applications we have encountered problems running out of 
>> perm gen space.  A few years ago we had similar problems but by using 
>> jmap/jhat we were able to debug and fix this.
>>
>> This time using jmap/jhat to analyze the problem has lead to a dead end.
>>
>> In particular we see:
>>
>> 1) There are no instances left of the classes that we load/unload
>> 2) The class objects of the classes that we load/unload hang around, 
>> they don't seem to get GC'ed
>> 3) The ClassLoader of the loaded classes is hanging around.  Doing a 
>> "Reference Chains from Rootset" in jhat results in 0 chains.  Unless 
>> jhat is lying then the classes/classloader should be able to be 
>> GC'ed.  Right?
>> 4) Using jmap -permstat I see the classloaders in question are marked 
>> as 'dead'.  This means that it should be able to be GC'ed
>> 5) Tried Current GC, default GC, G1 GC and basically get the same 
>> errors.  The G1 GC might error out with code cache problems instead.
>> 6) The IBM JVM in the same scenario seems to be happy
>> 7) If we load the classes (Class.forName()) but don't instantiate 
>> them, PermGen seems happier but we then run out of Code Cache.  I'm 
>> not sure if the problems are related.
>>
>> The tests are run on linux & linux64.  We are using:  
>> -XX:MaxPermSize=128m -Xms800m -Xmx1500m -XX:+CMSClassUnloadingEnabled 
>> and appropriate JVM args for the different GC's
>>
>> Given #3,4 & #6 above I'm thinking that it maybe a hotspot problem.
>> What should I try next to track this problem down?
>>
>> Thanks,
>>
>> Keith McNeill
>>
>>
>>
>>
>>
>