Project Jigsaw, Apache Tomcat and RMI related memory leaks

[Mark Thomas]

On 19/04/2016 16:25, Roger Riggs wrote:
> Hi Mark,
> 
> It may take a bit of time to unwind and find a good solution.

Hi Roger,

Agreed. Your response has actually been a huge help. It gave me a few
more things to look at and I have been able break the problem down
significantly.

I need to work on this some more before I come back with a much clearer
statement of the problem. I hope to be in a position to do that later
today or tomorrow.

I'll probably do that in a new thread.

Thanks for the pointers.

Mark


> 
> On 4/18/16 11:58 AM, Mark Thomas wrote:
>> Hi,
>>
>> The Apache Tomcat community was asked by Rory O'Donnell for feedback on
>> JDK 9 + Project Jigsaw. Having provided that feedback we were directed
>> here so I have reproduced that feedback below.
>>
>>
>> I've started testing Tomcat trunk with JDK 9 + Project Jigsaw and it
>> looks like we are going to hit a bunch of problems related to Tomcat's
>> memory leak protection.
>>
>> The short version is there are lots of JDK calls that can result in a
>> reference being retained to the current class loader. If that class
>> loader is the web application class loader it often ends up being pinned
>> in memory. This triggers a memory leak when the web application is
>> reloaded since the web application class loader is not eligible for GC.
>>
>> Tomcat generally uses reflection to find these problems. It then does
>> one of two things:
>> - If the JRE provides an API the application developer should have used
>> to clean up the reference, Tomcat does this for them and then logs a
>> very loud error message telling the developer they need to fix their app.
>> - If there is nothing the developer could have done to avoid the
>> problem, Tomcat cleans it up. Usually this is via reflection again.
>>
>> Reporting this second class of issues as JRE bugs has been on my TODO
>> list for a long time. It looks like Java 9 is going to bump this to the
>> top of the list.
>>
>> The first problem we have hit is related to RMI. The memory leak is
>> triggered by a call to:
>> java.rmi.registry.Registry.bind() or
>> java.rmi.registry.Registry.rebind()
>>
>> The problem occurs when an object of a class loaded by the web
>> application class loader is bound to the RMI registry.
>>
>> There is no standard API (that I have found) that completely removes all
>> references. In particular
>> java.rmi.registry.Registry.unbind()
>> doesn't help.
> unbind just removes the mapping from name to an exported object;
> it does not un-export the object.
> Unless it was exported as permanent, the normal GC/distributed GC should
> clear it.
> 
> I would think the normal application shutdown should both remove the
> binding
> and unexport it (perhaps forcibly).
> 
> But you are working on the case where the application doesn't behave
> properly.
> 
> If you had a way to discover the exported remote objects, the normal
> forcible unexport mechanism
> should be sufficient.  Handing out references to exports objects would
> need some kind of security
> check.
> 
> It might be worth investigating an official shutdown mechanism though
> linking it to a classloader
> seems a bit special purpose.
> 
> $.02, Roger
> 
>> The code Tomcat uses to clean this up is at [1]. Essentially, we need to
>> remove any reference to the web application's class loader from the RMI
>> caches.
>>
>> With JDK 9 this fails with:
>> java.lang.reflect.InaccessibleObjectException: Unable to make member of
>> class sun.rmi.transport.Target accessible:  module java.rmi does not
>> export sun.rmi.transport to unnamed module ...
>>
>> I took a look at the JDK 9 API but I could not find a way to bypass
>> this.
>>
>> Possible solutions include:
>> 1. Some way that allows us to continue to use reflection as per the
>> current code.
>>
>> 2. A new method somewhere in the RMI API that clears all references
>> associated with a given class loader from the cache.
>>
>> 3. Modify Registry.unbind() so it removes all references.
>>
>> 4. Something else...
>>
>> I do have a concern with 3 on its own that, while that would allow
>> applications to clear their own references, it would mean Tomcat would
>> have no way to check for the problem.
>>
>> Ideally, I'd like to see the API extended so a) applications are able to
>> clean up after themselves and b) Tomcat can check for leaked references
>> and generate error messages for the ones found.
>>
>> Any and all suggestions gratefully received.
>>
>> Thanks,
>>
>> Mark
>>
>> [1]
>> http://svn.apache.org/viewvc/tomcat/trunk/java/org/apache/catalina/loader/WebappClassLoaderBase.java?view=annotate#l2214
>>
>