Metaspace leak with instrumentation.retransform

[David Holmes]

Hi,

On 23/05/2023 10:25 pm, Jean-Philippe Bempel wrote:
> Hi all,
> 
> We have just identified a Metaspace leak in a very specific case when
> a class has a method using a try-with-resources construct (or similar
> with try-catch) and re-transforming this class in a loop. It is
> reproducible from jdk8 to jdk20.
> Here the steps to reproduce:
> 
>   1. create a java file with following content:
> 
> public class RetransformLeak {
>      public static void main(String[] args) throws Exception {
>          new MyClass();
>          while (true) {
>              Thread.sleep(1000);
>          }
>      }
> }
> 
> class MyClass {
>      private static void writeFile() {

I don't see this ever getting called. Is some code missing?

>          TWR var0 = new TWR();
>          try {
>              var0.process();
>          } catch (Throwable var4) {
>              try {
>                  var0.close();
>              } catch (Throwable var3) {
>                  var4.addSuppressed(var3);

If commenting this removes the problem then it seems likely the actual 
Throwable being thrown as var3 is holding onto some references which 
keeps the old class alive. But as close() does nothing I can't see how 
it can throw anything. ??

>              }
>              throw var4;
>          }
>          var0.close();
> //        try (TWR twr = new TWR()) {
> //            twr.process();
> //        }
>      }
> 
>      static class TWR implements AutoCloseable {
>          public void process() {}
>          @Override
>          public void close() {}
>      }
> }
> 
> 2. compile it: javac RetransformLeak.java
> 3. create a java file Agent.java with the following content that will
> be our java agent performing re-transformation:
> 
> public class Agent {
>      public static void premain(String arg, Instrumentation inst) {
>          new Thread(() -> retransformLoop(inst, arg)).start();
>      }
> 
>      private static void retransformLoop(Instrumentation
> instrumentation, String className) {
>          Class<?> classToRetransform = null;
>          while (true) {
>              if (classToRetransform == null) {
>                  for (Class<?> clazz : instrumentation.getAllLoadedClasses()) {
>                      if (clazz.getName().equals(className)) {
>                          System.out.println("found class: " + className);
>                          classToRetransform = clazz;
>                          break;
>                      }
>                  }
>              }
>              if (classToRetransform != null) {
>                  try {
>                      instrumentation.retransformClasses(classToRetransform);

What ClassfileTransformer is actually in use here?

>                      //Thread.sleep(1);
>                  } catch (Exception e) {
>                      throw new RuntimeException(e);
>                  }
>              }
>          }
>      }
> }
> 
>   4. Compile it: javac Agent.java
>   5. create a Manifest.txt file for the java agent:
> 
> Premain-Class: Agent
> Can-Retransform-Classes: true
> 
>   6. create java agent jar: jar cfm agent.jar Manifest.txt Agent.class
>   7. execute the RetransformLeak class with a max metaspace size:
> java -javaagent:agent.jar=MyClass -XX:MaxMetaspaceSize=128M -cp .
> RetransformLeak
> 
> output:
> found class: MyClass
> Exception in thread "Thread-0" java.lang.OutOfMemoryError
>    at java.instrument/sun.instrument.InstrumentationImpl.retransformClasses0(Native
> Method)
>    at java.instrument/sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:169)
>    at Agent.retransformLoop(Agent.java:22)
>    at Agent.lambda$premain$0(Agent.java:5)
>    at java.base/java.lang.Thread.run(Thread.java:1623)
> 
> If you comment the line:
>                  var4.addSuppressed(var3);
> in MyClass#writeFile method, no OOME will be thrown and Metaspace will
> remain stable.

It sounds to me like this isn't a leak as such but rather the exception 
keeps things alive. But we need the missing details.

Cheers,
David

> You can also directly use a try-with-resources construct to reproduce
> the leak but I have decomposed it with try catch to be able to
> pinpoint more precisely which bytecode may generate the leak.
> 
> I can file a bug in OpenJDK jira if needed.
> Thanks
> Jean-Philippe Bempel