jmx-dev Using the JMX monitor the application will cause the GC time to grow longer

[Hanyu King]

Hi all.

I found that Using the JMX monitor the application will cause the GC time
to grow longer
Problem Description

When you use jmx to remotely monitor your application (such as VisualVM),
after a while (a few days), you will find that the application's GC time
will be longer and longer.

This problem exists in jdk7, jdk8, other versions have not been tried (the
following code is based on jdk7).
Pronlem Causes

When using the jmx monitor, the
javax.management.remote.rmi.RMIConnectionImpl#unwrap method will be called,
and each time a new ClassLoader (CombinedClassLoader) is created. When the
newly created ClassLoader loads the class, first find out if the class has
been loaded from the SystemDictionary according to the ClassLoader object
and the class name. If not, it will find or load it, and then add it to the
SystemDictionary. Because we create a ClassLoader object every time, the
SystemDictionary will definitely not find it, so every time we will put the
new ClassLoader object and the loaded class into the SystemDictionary.

Because our monitoring system(such as VisualVM) returns data in real time,
we constantly create new ClassLoader loading classes. As the time gets
longer, there will be more and more elements in the SystemDictionary. At
this point, the bucket's linked list may be very long, and the efficiency
of the search will be low.

When a young gc occurs, the SystemDictionary is scanned when a strong
reference is processed, so when there are more and more elements in the
SystemDictionary, the young gc will become slower and slower.
Debug Process

When you use jmx to monitor your application, the code goes to the
RMIConnectionImpl implementation class. When it goes to the unwrap method,
it creates a new ClassLoader each time, and then sets the ClassLoader to
the ClassLoader of the current thread context. code show as below.


javax.management.remote.rmi.RMIConnectionImpl#unwrap(

      final MarshalledObject<?> mo,

                              final ClassLoader cl1,

                              final ClassLoader cl2,

                              final Class<T> wrappedClass)



try {

          ClassLoader orderCL = AccessController.doPrivileged(

              new PrivilegedExceptionAction<ClassLoader>() {

                  public ClassLoader run() throws Exception {

                      return new
CombinedClassLoader(Thread.currentThread().getContextClassLoader(),

                              new OrderClassLoaders(cl1, cl2));

                  }

              }

          );

          return unwrap(mo, orderCL, wrappedClass);

 } catch (PrivilegedActionException pe) {






javax.management.remote.rmi.RMIConnectionImpl#unwrap(

      final MarshalledObject<?> mo,

                              final ClassLoader cl,

                              final Class<T> wrappedClass)





final ClassLoader old = AccessController.doPrivileged(new SetCcl(cl));

try {

  return wrappedClass.cast(mo.get());

} catch (ClassNotFoundException cnfe) {

  throw new UnmarshalException(cnfe.toString(), cnfe);

} finally {

  AccessController.doPrivileged(new SetCcl(old));

}



This newly created ClassLoader will eventually be used in the
sun.rmi.server.LoaderHandler#loadClass(java.net.URL[], java.lang.String)
method. code show as below.


ClassLoader var2 = getRMIContextClassLoader();

      if (loaderLog.isLoggable(Log.VERBOSE)) {

          loaderLog.log(Log.VERBOSE, "(thread context class loader: " +
var2 + ")");

      }

      SecurityManager var3 = System.getSecurityManager();

      if (var3 == null) {

          try {

              Class var11 = loadClassForName(var1, false, var2);

              if (loaderLog.isLoggable(Log.VERBOSE)) {

                  loaderLog.log(Log.VERBOSE, "class \"" + var1 + "\" found
via " + "thread context class loader " + "(no security manager: codebase
disabled), " + "defined by " + var11.getClassLoader());

              }

              return var11;

          } catch (ClassNotFoundException var8) {

              if (loaderLog.isLoggable(Log.BRIEF)) {

                  loaderLog.log(Log.BRIEF, "class \"" + var1 + "\" not
found via " + "thread context class loader " + "(no security manager:
codebase disabled)", var8);

              }

              throw new ClassNotFoundException(var8.getMessage() + " (no
security manager: RMI class loader disabled)", var8.getException());

          }

      }


private static ClassLoader getRMIContextClassLoader() {

      return Thread.currentThread().getContextClassLoader();

}


private static Class<?> loadClassForName(String var0, boolean var1,
ClassLoader var2) throws ClassNotFoundException {

      if (var2 == null) {

          ReflectUtil.checkPackageAccess(var0);

      }

      return Class.forName(var0, var1, var2);

}

Finally, Class.forName will be called. At this point, the loader is the
newly created CombinedClassLoader. After the load is successful, it will be
placed in the SystemDictionary.
SystemDictionary

1. The data structure is a hashtable

2. The default bucket size is 1009 and does not support expansion

3. The way to resolve conflicts is to use a linked list
Young gc scan SystemDictionary code as follows

http://hg.openjdk.java.net/jdk7u/jdk7u/hotspot/file/69f46e2dbd83/src/share/vm/memory/sharedHeap.cpp



if
(!_process_strong_tasks->is_task_claimed(SH_PS_SystemDictionary_oops_do)) {

  if (so & SO_AllClasses) {

    SystemDictionary::oops_do(roots);

  } else if (so & SO_SystemClasses) {

    SystemDictionary::always_strong_oops_do(roots);

  }

}





http://hg.openjdk.java.net/jdk7u/jdk7u/hotspot/file/69f46e2dbd83/src/share/vm/classfile/dictionary.cpp

void Dictionary::always_strong_classes_do(OopClosure* blk) {

// Follow all system classes and temporary placeholders in dictionary

for (int index = 0; index < table_size(); index++) {

  for (DictionaryEntry *probe = bucket(index);

                        probe != NULL;

                        probe = probe->next()) {

    klassOop e = probe->klass();

    oop class_loader = probe->loader();

    if (is_strongly_reachable(class_loader, e)) {

      blk->do_oop((oop*)probe->klass_addr());

      if (class_loader != NULL) {

        blk->do_oop(probe->loader_addr());

      }

      probe->protection_domain_set_oops_do(blk);

    }

  }

}

}
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.java.net/pipermail/jmx-dev/attachments/20190124/301b23ec/attachment-0001.html>