Project Jigsaw, Apache Tomcat and RMI related memory leaks

[Roger Riggs]

Hi Mark,

It may take a bit of time to unwind and find a good solution.

On 4/18/16 11:58 AM, Mark Thomas wrote:
> Hi,
>
> The Apache Tomcat community was asked by Rory O'Donnell for feedback on
> JDK 9 + Project Jigsaw. Having provided that feedback we were directed
> here so I have reproduced that feedback below.
>
>
> I've started testing Tomcat trunk with JDK 9 + Project Jigsaw and it
> looks like we are going to hit a bunch of problems related to Tomcat's
> memory leak protection.
>
> The short version is there are lots of JDK calls that can result in a
> reference being retained to the current class loader. If that class
> loader is the web application class loader it often ends up being pinned
> in memory. This triggers a memory leak when the web application is
> reloaded since the web application class loader is not eligible for GC.
>
> Tomcat generally uses reflection to find these problems. It then does
> one of two things:
> - If the JRE provides an API the application developer should have used
> to clean up the reference, Tomcat does this for them and then logs a
> very loud error message telling the developer they need to fix their app.
> - If there is nothing the developer could have done to avoid the
> problem, Tomcat cleans it up. Usually this is via reflection again.
>
> Reporting this second class of issues as JRE bugs has been on my TODO
> list for a long time. It looks like Java 9 is going to bump this to the
> top of the list.
>
> The first problem we have hit is related to RMI. The memory leak is
> triggered by a call to:
> java.rmi.registry.Registry.bind() or
> java.rmi.registry.Registry.rebind()
>
> The problem occurs when an object of a class loaded by the web
> application class loader is bound to the RMI registry.
>
> There is no standard API (that I have found) that completely removes all
> references. In particular
> java.rmi.registry.Registry.unbind()
> doesn't help.
unbind just removes the mapping from name to an exported object;
it does not un-export the object.
Unless it was exported as permanent, the normal GC/distributed GC should 
clear it.

I would think the normal application shutdown should both remove the binding
and unexport it (perhaps forcibly).

But you are working on the case where the application doesn't behave 
properly.

If you had a way to discover the exported remote objects, the normal 
forcible unexport mechanism
should be sufficient.  Handing out references to exports objects would 
need some kind of security
check.

It might be worth investigating an official shutdown mechanism though 
linking it to a classloader
seems a bit special purpose.

$.02, Roger

> The code Tomcat uses to clean this up is at [1]. Essentially, we need to
> remove any reference to the web application's class loader from the RMI
> caches.
>
> With JDK 9 this fails with:
> java.lang.reflect.InaccessibleObjectException: Unable to make member of
> class sun.rmi.transport.Target accessible:  module java.rmi does not
> export sun.rmi.transport to unnamed module ...
>
> I took a look at the JDK 9 API but I could not find a way to bypass
> this.
>
> Possible solutions include:
> 1. Some way that allows us to continue to use reflection as per the
> current code.
>
> 2. A new method somewhere in the RMI API that clears all references
> associated with a given class loader from the cache.
>
> 3. Modify Registry.unbind() so it removes all references.
>
> 4. Something else...
>
> I do have a concern with 3 on its own that, while that would allow
> applications to clear their own references, it would mean Tomcat would
> have no way to check for the problem.
>
> Ideally, I'd like to see the API extended so a) applications are able to
> clean up after themselves and b) Tomcat can check for leaked references
> and generate error messages for the ones found.
>
> Any and all suggestions gratefully received.
>
> Thanks,
>
> Mark
>
> [1]
> http://svn.apache.org/viewvc/tomcat/trunk/java/org/apache/catalina/loader/WebappClassLoaderBase.java?view=annotate#l2214