the amazing tales of the search for the invisible man! or, where's my gc root

[Jed Wesley-Smith]

all,

I am writing to this list in some desperation hoping for some expert 
advice. We (the JIRA development team at Atlassian) have been hunting 
memory leaks for some weeks and in the process have tracked down and 
removed every possible reference to a large section of memory (a plugin 
framework) that we could find. Starting with all strong references and 
proceeding to remove soft and weak references - even things like 
clearing the java.lang.reflect.Proxy cache - and even Finalizer 
references until both YourKit, Eclipse MAT, JProfiler and jhat all 
report that the memory in question is dead and should be collectable, 
but inexplicably _the JVM still holds on to it_. There are no JNI Global 
references either, yet this memory remains uncollectable!

This happens for the 1.5 and 1.6 JVMs on Linux and Mac, and the IBM 1.6 
JDK on Linux.

So my question is, how on earth do I search for what is referencing this 
uncollectable memory? Are there any other tools that can help find why 
this memory is not collected? Can I query the VM directly somehow?

I fear this is a JVM GC bug as no known memory analysis tool can find 
the heap root (i.e. according to "the rules" there is no heap root). Are 
there any known GC memory leaks caused by ClassLoaders being dropped for 
instance?

The application is creating and disposing of a lot of ClassLoaders via 
OSGi (Apache Felix) with Spring OSGi. It creates a lot of 
java.lang.reflect.Proxy class instances.

We have written this up and added an example heap dump here:
http://jira.atlassian.com/browse/JRA-16932

Having come to the end of our tethers here, if anyone can help in any 
way it would be massively appreciated.

cheers,
Jed Wesley-Smith
JIRA Team @ Atlassian