Metaspace leak with instrumentation.retransform

[Jean-Philippe Bempel]

Hi all,

We have just identified a Metaspace leak in a very specific case when
a class has a method using a try-with-resources construct (or similar
with try-catch) and re-transforming this class in a loop. It is
reproducible from jdk8 to jdk20.
Here the steps to reproduce:

 1. create a java file with following content:

public class RetransformLeak {
    public static void main(String[] args) throws Exception {
        new MyClass();
        while (true) {
            Thread.sleep(1000);
        }
    }
}

class MyClass {
    private static void writeFile() {
        TWR var0 = new TWR();
        try {
            var0.process();
        } catch (Throwable var4) {
            try {
                var0.close();
            } catch (Throwable var3) {
                var4.addSuppressed(var3);
            }
            throw var4;
        }
        var0.close();
//        try (TWR twr = new TWR()) {
//            twr.process();
//        }
    }

    static class TWR implements AutoCloseable {
        public void process() {}
        @Override
        public void close() {}
    }
}

2. compile it: javac RetransformLeak.java
3. create a java file Agent.java with the following content that will
be our java agent performing re-transformation:

public class Agent {
    public static void premain(String arg, Instrumentation inst) {
        new Thread(() -> retransformLoop(inst, arg)).start();
    }

    private static void retransformLoop(Instrumentation
instrumentation, String className) {
        Class<?> classToRetransform = null;
        while (true) {
            if (classToRetransform == null) {
                for (Class<?> clazz : instrumentation.getAllLoadedClasses()) {
                    if (clazz.getName().equals(className)) {
                        System.out.println("found class: " + className);
                        classToRetransform = clazz;
                        break;
                    }
                }
            }
            if (classToRetransform != null) {
                try {
                    instrumentation.retransformClasses(classToRetransform);
                    //Thread.sleep(1);
                } catch (Exception e) {
                    throw new RuntimeException(e);
                }
            }
        }
    }
}

 4. Compile it: javac Agent.java
 5. create a Manifest.txt file for the java agent:

Premain-Class: Agent
Can-Retransform-Classes: true

 6. create java agent jar: jar cfm agent.jar Manifest.txt Agent.class
 7. execute the RetransformLeak class with a max metaspace size:
java -javaagent:agent.jar=MyClass -XX:MaxMetaspaceSize=128M -cp .
RetransformLeak

output:
found class: MyClass
Exception in thread "Thread-0" java.lang.OutOfMemoryError
  at java.instrument/sun.instrument.InstrumentationImpl.retransformClasses0(Native
Method)
  at java.instrument/sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:169)
  at Agent.retransformLoop(Agent.java:22)
  at Agent.lambda$premain$0(Agent.java:5)
  at java.base/java.lang.Thread.run(Thread.java:1623)

If you comment the line:
                var4.addSuppressed(var3);
in MyClass#writeFile method, no OOME will be thrown and Metaspace will
remain stable.
You can also directly use a try-with-resources construct to reproduce
the leak but I have decomposed it with try catch to be able to
pinpoint more precisely which bytecode may generate the leak.

I can file a bug in OpenJDK jira if needed.
Thanks
Jean-Philippe Bempel