the amazing tales of the search for the invisible man! or, where's my gc root

[kirk]

Hi Jed,

I've had a quick look at the heap dump. I'm having a little trouble 
understanding what is in there. What I can see is a large number of 
java.lang.reflect.Method objects being held. There seems to be two 
competing patterns of references holding onto these objects. I've 
attached some screenshots rather than use words.

The scary thing is that the references include ClassLoader.scl, 
JDK12Hooks.systemClassLoader as well as Apache commons logging 
LogFactory. With this type of the complex entanglement it would seem 
unlikely that these objects would ever be collected. The other pattern 
also includes the spiders web of references. It also includes 
UberspecImpl and a whole bunch of static collections. IME, static 
collections are involved in the vast majority of leaks I've diagnosed.

Interestingly enough the a portion of the 2nd largest consumer of memory 
is also tangled up in the JDK12Hooks. Random sampling leads me to AST 
parse trees and "no reference". Looks like much of this is tied up with 
Velocity. In fact the largest consumer of memory at 24% is char[]. I'm 
failing to find anything that is not tied up with Velocity (AST parsing).

Needs more investigation. Be interesting to run a test with generations 
turned on. NetBeans generations is a true count unlike that provided by 
YourKit.

Regards,
Kirk


Jed Wesley-Smith wrote:
> Classes as well. We end up getting an OOME although the profilers 
> report only a third of the heap is reachable.
>
> Although I indicated we saw this on the IBM jdk analysis of that dump 
> showed a completely different issue that apparently may not be a 
> problem (due to reflection optimisation on that jdk) - the dead 
> objects appear to have been correctly cleared. We are reproducing this 
> to verify.
>
> Additionally we tried running with -client on the sun jvms as we saw a 
> bug that might have caused it reported against server only but without 
> success.
>
> cheers,
> jed.
>
> On 16/04/2009, at 12:51 AM, Tony Printezis 
> <Antonios.Printezis at sun.com> wrote:
>
>> OK, I'll bite.
>>
>> When you say: "a large section of memory (a plugin framework)" do you 
>> mean only objects in the young / old gen, or also classes in the perm 
>> gen?
>>
>> How do you know that said memory is not being reclaimed? Do you 
>> eventually get an OOM?
>>
>> Given that it happens with two different JVMs (I assume you use 
>> HotSpot on Linux and Mac, as well as the IBM JDK), it's unlikely to 
>> be a GC bug, as both JVMs would need to have the same bug. Not 
>> impossible, but unlikely, IMHO.
>>
>> Tony
>>
>> Jed Wesley-Smith wrote:
>>> all,
>>>
>>> I am writing to this list in some desperation hoping for some expert 
>>> advice. We (the JIRA development team at Atlassian) have been 
>>> hunting memory leaks for some weeks and in the process have tracked 
>>> down and removed every possible reference to a large section of 
>>> memory (a plugin framework) that we could find. Starting with all 
>>> strong references and proceeding to remove soft and weak references 
>>> - even things like clearing the java.lang.reflect.Proxy cache - and 
>>> even Finalizer references until both YourKit, Eclipse MAT, JProfiler 
>>> and jhat all report that the memory in question is dead and should 
>>> be collectable, but inexplicably _the JVM still holds on to it_. 
>>> There are no JNI Global references either, yet this memory remains 
>>> uncollectable!
>>>
>>> This happens for the 1.5 and 1.6 JVMs on Linux and Mac, and the IBM 
>>> 1.6 JDK on Linux.
>>>
>>> So my question is, how on earth do I search for what is referencing 
>>> this uncollectable memory? Are there any other tools that can help 
>>> find why this memory is not collected? Can I query the VM directly 
>>> somehow?
>>>
>>> I fear this is a JVM GC bug as no known memory analysis tool can 
>>> find the heap root (i.e. according to "the rules" there is no heap 
>>> root). Are there any known GC memory leaks caused by ClassLoaders 
>>> being dropped for instance?
>>>
>>> The application is creating and disposing of a lot of ClassLoaders 
>>> via OSGi (Apache Felix) with Spring OSGi. It creates a lot of 
>>> java.lang.reflect.Proxy class instances.
>>>
>>> We have written this up and added an example heap dump here:
>>> http://jira.atlassian.com/browse/JRA-16932
>>>
>>> Having come to the end of our tethers here, if anyone can help in 
>>> any way it would be massively appreciated.
>>>
>>> cheers,
>>> Jed Wesley-Smith
>>> JIRA Team @ Atlassian
>>
>> -- 
>> ---------------------------------------------------------------------
>> | Tony Printezis, Staff Engineer   | Sun Microsystems Inc.          |
>> |                                  | MS UBUR02-311                  |
>> | e-mail: tony.printezis at sun.com   | 35 Network Drive               |
>> | office: +1 781 442 0998 (x20998) | Burlington, MA 01803-2756, USA |
>> ---------------------------------------------------------------------
>> e-mail client: Thunderbird (Linux)
>>
>>
>