Metaspace leak with instrumentation.retransform

[Thomas Stüfe]

On Wed, May 24, 2023 at 7:03 AM David Holmes <david.holmes at oracle.com>
wrote:

> Hi,
>
> On 23/05/2023 10:25 pm, Jean-Philippe Bempel wrote:
> > Hi all,
> >
> > We have just identified a Metaspace leak in a very specific case when
> > a class has a method using a try-with-resources construct (or similar
> > with try-catch) and re-transforming this class in a loop. It is
> > reproducible from jdk8 to jdk20.
> > Here the steps to reproduce:
> >
> >   1. create a java file with following content:
> >
> > public class RetransformLeak {
> >      public static void main(String[] args) throws Exception {
> >          new MyClass();
> >          while (true) {
> >              Thread.sleep(1000);
> >          }
> >      }
> > }
> >
> > class MyClass {
> >      private static void writeFile() {
>
> I don't see this ever getting called. Is some code missing?
>
> >          TWR var0 = new TWR();
> >          try {
> >              var0.process();
> >          } catch (Throwable var4) {
> >              try {
> >                  var0.close();
> >              } catch (Throwable var3) {
> >                  var4.addSuppressed(var3);
>
> If commenting this removes the problem then it seems likely the actual
> Throwable being thrown as var3 is holding onto some references which
> keeps the old class alive. But as close() does nothing I can't see how
> it can throw anything. ??
>
> >              }
> >              throw var4;
> >          }
> >          var0.close();
> > //        try (TWR twr = new TWR()) {
> > //            twr.process();
> > //        }
> >      }
> >
> >      static class TWR implements AutoCloseable {
> >          public void process() {}
> >          @Override
> >          public void close() {}
> >      }
> > }
> >
> > 2. compile it: javac RetransformLeak.java
> > 3. create a java file Agent.java with the following content that will
> > be our java agent performing re-transformation:
> >
> > public class Agent {
> >      public static void premain(String arg, Instrumentation inst) {
> >          new Thread(() -> retransformLoop(inst, arg)).start();
> >      }
> >
> >      private static void retransformLoop(Instrumentation
> > instrumentation, String className) {
> >          Class<?> classToRetransform = null;
> >          while (true) {
> >              if (classToRetransform == null) {
> >                  for (Class<?> clazz :
> instrumentation.getAllLoadedClasses()) {
> >                      if (clazz.getName().equals(className)) {
> >                          System.out.println("found class: " + className);
> >                          classToRetransform = clazz;
> >                          break;
> >                      }
> >                  }
> >              }
> >              if (classToRetransform != null) {
> >                  try {
> >
> instrumentation.retransformClasses(classToRetransform);
>
> What ClassfileTransformer is actually in use here?
>
> >                      //Thread.sleep(1);
> >                  } catch (Exception e) {
> >                      throw new RuntimeException(e);
> >                  }
> >              }
> >          }
> >      }
> > }
> >
> >   4. Compile it: javac Agent.java
> >   5. create a Manifest.txt file for the java agent:
> >
> > Premain-Class: Agent
> > Can-Retransform-Classes: true
> >
> >   6. create java agent jar: jar cfm agent.jar Manifest.txt Agent.class
> >   7. execute the RetransformLeak class with a max metaspace size:
> > java -javaagent:agent.jar=MyClass -XX:MaxMetaspaceSize=128M -cp .
> > RetransformLeak
> >
> > output:
> > found class: MyClass
> > Exception in thread "Thread-0" java.lang.OutOfMemoryError
> >    at
> java.instrument/sun.instrument.InstrumentationImpl.retransformClasses0(Native
> > Method)
> >    at
> java.instrument/sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:169)
> >    at Agent.retransformLoop(Agent.java:22)
> >    at Agent.lambda$premain$0(Agent.java:5)
> >    at java.base/java.lang.Thread.run(Thread.java:1623)
> >
> > If you comment the line:
> >                  var4.addSuppressed(var3);
> > in MyClass#writeFile method, no OOME will be thrown and Metaspace will
> > remain stable.
>
> It sounds to me like this isn't a leak as such but rather the exception
> keeps things alive. But we need the missing details.
>
>
Yes, that would be helpful.

Cheers,
> David
>
> > You can also directly use a try-with-resources construct to reproduce
> > the leak but I have decomposed it with try catch to be able to
> > pinpoint more precisely which bytecode may generate the leak.
> >
> > I can file a bug in OpenJDK jira if needed.
> > Thanks
> > Jean-Philippe Bempel
>


My first thought was that this is not a leak but a case of class
redefinition, where the bytecode is rewritten. The space for the old
bytecode gets salvaged and would be re-used for metaspace allocations from
the same loader, save that said re-use never happens for some reason. The
output of "jcmd VM.metaspace" at the end of the program run would be
helpful too.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/hotspot-dev/attachments/20230524/cbbad561/attachment-0001.htm>