AnonymousClassLoader in JRuby
Rémi Forax
forax at univ-mlv.fr
Sat Apr 26 14:39:33 PDT 2008
Charles Oliver Nutter a écrit :
> I've managed to wire the AnonymousClassLoader into JRuby. A patch is
> attached, that will probably be rather confusing to anyone unfamiliar
> with JRuby.
>
> ClassCache is basically a smart class-loading utility that can be used
> to share loaded classes across JRuby instances without keeping hard
> references to them and preventing them from unloading.
>
> AnonymousClassLoader is an all-static utility class that wraps the
> logic of instantiating "one-shot" classloaders for loading bodies of
> code. In this case, it either uses java.dyn.AnonymousClassLoader or
> our own "OneShotClassLoader" which is not anonymous but which is used
> for only a single class.
>
> The code generally appears to work about the same with
> java.dyn.AnonymousClassLoader available, which is good. But I'm having
> trouble quantifying the benefit to JRuby. I'd like to be able to show
> how it helps, but memory profiles look practically the same between
> the version using an OneShotClassLoader per method body and the
> version using AnonymousClassLoader per method body. I have not done a
> lot of work to isolate the cost of loading, but it seems to be
> unnoticeable in my simple benchmark which defines 100_000 Ruby methods
> and forces them to JIT.
>
> So the runtime benefits may not be so great. The practical benefits,
> such as being able to chuck byte[] into AnonymousClassLoader without
> decorating class names and ensuring uniqueness, I have not yet
> utilized in this code. Those benefits may show how writing class
> caches like the one in JRuby are made a lot easier. For the moment, in
> order to allow JRuby to support both MLVM work and pre-JDK7 JVMs, this
> patch still does name-mangling to ensure methods are unique.
>
> I think part of my confusion is that originally I desired a class
> loader for which I could have a single instance I would throw *many*
> byte[] at, and they'd all be loaded using that classloader but without
> hard references and without the overhead of a classloader-per-method.
> But the AnonymousClassLoader interface appears to one exactly one
> byte[] per instance, though it does have considerably reduced
> in-memory cost per AnonymousClassLoader instance.
>
> Basically, what I've been looking for to make my life easier and
> memory costs lower is:
>
> ClassLoader cl = someClassLoader....;
> Class first = cl.defineClass(firstClass);
> Class second = cl.defineClass(secondClass);
> first = null; // and at some point I would expect the class to GC
>
> Is this the purpose of AnonymousClassLoader? Am I doing it wrong? I'll
> be poking around for example code now.
In my opinion, there is one AnonymousClassloader by host class to share
security check.
Using the BGGA syntax (or something close) :
class A {
void f() {
{int => int} closure = {int x =>
x+2;
};
...
}
}
is translated to:
class A {
void f() {
Class<?> closureClass = loader.loadClass("A$Closure1");
IntIntClosure closure = closureClass.newInstance();
...
}
static class A.Closure1 {
public int invoke(int value) {
return value * 2;
}
}
private static final RubyAnonymousClassLoader loader =
new RubyAnonymousClassLoader(A.class);
}
and RubyAnonymousClassLoader is an anoonymous classloader +
a weak cache.
public class RubyAnonymousClassLoader {
private final AnonymousClassLoader loader;
private final HashMap<String, WeakReference<Class<?>> cache = ...
public RubyAnonymousClassLoader( Class<?> hostClass) {
this.loader = new AnonymousClassLoader(hostClass);
}
public synchronized void loadClass(Class<?>) {
// use the cache here
// if cache miss, use the anonymous loader
}
}
Else, why do you use reflection in org.jruby.util.AnonymousClassLoader ?
>
> - Charlie
> ------------------------------------------------------------------------
>
> Index: src/org/jruby/runtime/MethodFactory.java
> ===================================================================
> --- src/org/jruby/runtime/MethodFactory.java (revision 6575)
> +++ src/org/jruby/runtime/MethodFactory.java (working copy)
> @@ -74,13 +74,33 @@
> * @param classLoader The classloader to use for searching for and
> * dynamically loading code.
> * @return A new MethodFactory.
> + * @deprecated Not compatible with DVM/JDK7 AnonymousClassLoader
> */
> public static MethodFactory createFactory(ClassLoader classLoader) {
> if (reflection) return new ReflectionMethodFactory();
> - if (dumping) return new DumpingInvocationMethodFactory(dumpingPath, classLoader);
>
> return new InvocationMethodFactory(classLoader);
> }
> +
> + /**
> + * Based on optional properties, create a new MethodFactory. By default,
> + * this will create a code-generation-based InvocationMethodFactory. If
> + * security restricts code generation, ReflectionMethodFactory will be used.
> + * If we are dumping class definitions, DumpingInvocationMethodFactory will
> + * be used. See MethodFactory's static initializer for more details.
> + *
> + * This version
> + *
> + * @param hostClass The class to use as "host" for the new invoker, or the
> + * class from which to get a classloader to use for searching for and
> + * dynamically loading code.
> + * @return A new MethodFactory.
> + */
> + public static MethodFactory createFactory(Class hostClass) {
> + if (reflection) return new ReflectionMethodFactory();
> +
> + return new InvocationMethodFactory(hostClass);
> + }
>
> /**
> * Get a new method handle based on the target JRuby-compiled method.
> Index: src/org/jruby/internal/runtime/methods/InvocationMethodFactory.java
> ===================================================================
> --- src/org/jruby/internal/runtime/methods/InvocationMethodFactory.java (revision 6575)
> +++ src/org/jruby/internal/runtime/methods/InvocationMethodFactory.java (working copy)
> @@ -54,6 +54,7 @@
> import org.jruby.runtime.ThreadContext;
> import org.jruby.runtime.Visibility;
> import org.jruby.runtime.builtin.IRubyObject;
> +import org.jruby.util.AnonymousClassLoader;
> import org.jruby.util.CodegenUtils;
> import static org.jruby.util.CodegenUtils.*;
> import static java.lang.System.*;
> @@ -152,8 +153,8 @@
> /** The lvar index of the passed-in Block on the call */
> public static final int BLOCK_INDEX = 6;
>
> - /** The classloader to use for code loading */
> - protected JRubyClassLoader classLoader;
> + /** The class to use as host for anonymous classloading */
> + protected Class hostClass;
>
> /**
> * Whether this factory has seen undefined methods already. This is used to
> @@ -169,14 +170,26 @@
> *
> * @param classLoader The classloader to use, or to wrap if it is not a
> * JRubyClassLoader instance.
> + * @deprecated Do not use this version anymore, since it is not compatible
> + * with DVM/JDK7 anonymous classloading.
> */
> public InvocationMethodFactory(ClassLoader classLoader) {
> - if (classLoader instanceof JRubyClassLoader) {
> - this.classLoader = (JRubyClassLoader)classLoader;
> - } else {
> - this.classLoader = new JRubyClassLoader(classLoader);
> - }
> + hostClass = InvocationMethodFactory.class;
> }
> +
> + /**
> + * Construct a new InvocationMethodFactory using the class to host loaded
> + * code. Created classes are either constructed using the DVM/JDK7
> + * AnonymousClassLoader or using a "one shot" classloader for all
> + * construction.
> + *
> + * @param classLoader The classloader to use, or to wrap if it is not a
> + * JRubyClassLoader instance.
> + * @see AnonymousClassLoader
> + */
> + public InvocationMethodFactory(Class hostClass) {
> + this.hostClass = hostClass;
> + }
>
> /**
> * Use code generation to provide a method handle for a compiled Ruby method.
> @@ -189,7 +202,7 @@
> String sup = COMPILED_SUPER_CLASS;
> Class scriptClass = scriptObject.getClass();
> String mname = scriptClass.getName() + "Invoker" + method + arity;
> - synchronized (classLoader) {
> + synchronized (hostClass) {
> Class generatedClass = tryClass(mname);
>
> try {
> @@ -291,10 +304,7 @@
>
> if (DEBUG) out.println("Binding multiple: " + desc1.declaringClassName + "." + javaMethodName);
>
> - String generatedClassName = CodegenUtils.getAnnotatedBindingClassName(javaMethodName, desc1.declaringClassName, desc1.isStatic, desc1.actualRequired, desc1.optional, true);
> - String generatedClassPath = generatedClassName.replace('.', '/');
> -
> - synchronized (classLoader) {
> + synchronized (hostClass) {
>
> try {
> Class c = getAnnotatedMethodClass(descs);
> @@ -366,7 +376,7 @@
> String generatedClassName = CodegenUtils.getAnnotatedBindingClassName(javaMethodName, desc1.declaringClassName, desc1.isStatic, desc1.actualRequired, desc1.optional, true);
> String generatedClassPath = generatedClassName.replace('.', '/');
>
> - synchronized (classLoader) {
> + synchronized (hostClass) {
> Class c = tryClass(generatedClassName);
>
> int min = Integer.MAX_VALUE;
> @@ -485,10 +495,7 @@
> public DynamicMethod getAnnotatedMethod(RubyModule implementationClass, JavaMethodDescriptor desc) {
> String javaMethodName = desc.name;
>
> - String generatedClassName = CodegenUtils.getAnnotatedBindingClassName(javaMethodName, desc.declaringClassName, desc.isStatic, desc.actualRequired, desc.optional, false);
> - String generatedClassPath = generatedClassName.replace('.', '/');
> -
> - synchronized (classLoader) {
> + synchronized (hostClass) {
> try {
> Class c = getAnnotatedMethodClass(desc);
>
> @@ -518,7 +525,7 @@
> String generatedClassName = CodegenUtils.getAnnotatedBindingClassName(javaMethodName, desc.declaringClassName, desc.isStatic, desc.actualRequired, desc.optional, false);
> String generatedClassPath = generatedClassName.replace('.', '/');
>
> - synchronized (classLoader) {
> + synchronized (hostClass) {
> Class c = tryClass(generatedClassName);
>
> if (c == null) {
> @@ -586,7 +593,7 @@
> String generatedClassName = type.getName() + "Invoker";
> String generatedClassPath = typePath + "Invoker";
>
> - synchronized (classLoader) {
> + synchronized (hostClass) {
> Class c = tryClass(generatedClassName);
>
> try {
> @@ -990,10 +997,10 @@
> private Class tryClass(String name) {
> try {
> Class c = null;
> - if (classLoader == null) {
> - c = Class.forName(name, true, classLoader);
> + if (hostClass == null) {
> + c = Class.forName(name, true, null);
> } else {
> - c = classLoader.loadClass(name);
> + c = hostClass.getClassLoader().loadClass(name);
> }
>
> if (c != null && seenUndefinedClasses) {
> @@ -1008,7 +1015,7 @@
> }
> }
>
> - protected Class endCall(ClassWriter cw, MethodVisitor mv, String name) {
> + protected Class endCall(ClassWriter cw, MethodVisitor mv, String name) throws ClassNotFoundException {
> endMethod(mv);
> return endClass(cw, name);
> }
> @@ -1018,12 +1025,12 @@
> mv.visitEnd();
> }
>
> - protected Class endClass(ClassWriter cw, String name) {
> + protected Class endClass(ClassWriter cw, String name) throws ClassNotFoundException {
> cw.visitEnd();
> byte[] code = cw.toByteArray();
> CheckClassAdapter.verify(new ClassReader(code), false, new PrintWriter(System.err));
>
> - return classLoader.defineClass(name, code);
> + return AnonymousClassLoader.loadClass(name, code, hostClass, JRubyClassLoader.class.getProtectionDomain());
> }
>
> private void loadArgument(MethodVisitor mv, int argsIndex, int argIndex) {
> Index: src/org/jruby/internal/runtime/methods/DumpingInvocationMethodFactory.java
> ===================================================================
> --- src/org/jruby/internal/runtime/methods/DumpingInvocationMethodFactory.java (revision 6575)
> +++ src/org/jruby/internal/runtime/methods/DumpingInvocationMethodFactory.java (working copy)
> @@ -30,7 +30,6 @@
> import java.io.File;
> import java.io.FileOutputStream;
>
> -import org.jruby.Ruby;
> import org.objectweb.asm.ClassWriter;
>
> /**
> @@ -44,10 +43,16 @@
>
> private String dumpPath;
>
> + @Deprecated
> public DumpingInvocationMethodFactory(String path, ClassLoader classLoader) {
> super(classLoader);
> this.dumpPath = path;
> }
> +
> + public DumpingInvocationMethodFactory(String path, Class hostClass) {
> + super(hostClass);
> + this.dumpPath = path;
> + }
>
> @Override
> protected Class endClass(ClassWriter cw, String name) {
> @@ -62,6 +67,6 @@
> fos.close();
> } catch(Exception e) {
> }
> - return classLoader.defineClass(name, code);
> + return null; //classLoader.defineClass(name, code);
> }
> }// DumpingInvocationMethodFactory
> Index: src/org/jruby/javasupport/util/RuntimeHelpers.java
> ===================================================================
> --- src/org/jruby/javasupport/util/RuntimeHelpers.java (revision 6575)
> +++ src/org/jruby/javasupport/util/RuntimeHelpers.java (working copy)
> @@ -134,7 +134,7 @@
> scope.determineModule();
> scope.setArities(required, optional, rest);
>
> - MethodFactory factory = MethodFactory.createFactory(compiledClass.getClassLoader());
> + MethodFactory factory = MethodFactory.createFactory(compiledClass);
> DynamicMethod method;
>
> if (name.equals("initialize") || visibility == Visibility.MODULE_FUNCTION) {
> @@ -191,7 +191,7 @@
> scope.determineModule();
> scope.setArities(required, optional, rest);
>
> - MethodFactory factory = MethodFactory.createFactory(compiledClass.getClassLoader());
> + MethodFactory factory = MethodFactory.createFactory(compiledClass);
> DynamicMethod method;
>
> method = factory.getCompiledMethod(rubyClass, javaName,
> Index: src/org/jruby/RubyModule.java
> ===================================================================
> --- src/org/jruby/RubyModule.java (revision 6575)
> +++ src/org/jruby/RubyModule.java (working copy)
> @@ -462,7 +462,7 @@
> // FIXME: This is probably not very efficient, since it loads all methods for each call
> boolean foundMethod = false;
> for (Method method : clazz.getDeclaredMethods()) {
> - if (method.getName().equals(name) && defineAnnotatedMethod(method, MethodFactory.createFactory(getRuntime().getJRubyClassLoader()))) {
> + if (method.getName().equals(name) && defineAnnotatedMethod(method, MethodFactory.createFactory(Ruby.class))) {
> foundMethod = true;
> }
> }
> @@ -605,7 +605,7 @@
> } catch (Throwable t) {
> if (DEBUG) System.out.println("Could not find it!");
> // fallback on non-pregenerated logic
> - MethodFactory methodFactory = MethodFactory.createFactory(getRuntime().getJRubyClassLoader());
> + MethodFactory methodFactory = MethodFactory.createFactory(Ruby.class);
>
> MethodClumper clumper = new MethodClumper();
> clumper.clump(clazz);
> @@ -638,7 +638,7 @@
> }
>
> private void defineAnnotatedMethodsIndexed(Class clazz) {
> - MethodFactory methodFactory = MethodFactory.createFactory(getRuntime().getJRubyClassLoader());
> + MethodFactory methodFactory = MethodFactory.createFactory(Ruby.class);
> methodFactory.defineIndexedAnnotatedMethods(this, clazz, methodDefiningCallback);
> }
>
> Index: src/org/jruby/util/ClassCache.java
> ===================================================================
> --- src/org/jruby/util/ClassCache.java (revision 6575)
> +++ src/org/jruby/util/ClassCache.java (working copy)
> @@ -2,8 +2,6 @@
>
> import java.lang.ref.ReferenceQueue;
> import java.lang.ref.WeakReference;
> -import java.net.URL;
> -import java.net.URLClassLoader;
> import java.security.ProtectionDomain;
> import java.util.Map;
> import java.util.concurrent.ConcurrentHashMap;
> @@ -13,6 +11,13 @@
> * multiple runtimes (or whole JVM).
> */
> public class ClassCache<T> {
> + private final ReferenceQueue referenceQueue = new ReferenceQueue();
> + private final Map<Object, KeyedClassReference> cache =
> + new ConcurrentHashMap<Object, KeyedClassReference>();
> + private final ClassLoader classLoader;
> + private final int max;
> + private final ProtectionDomain defaultDomain;
> +
> /**
> * The ClassLoader this class cache will use for any classes generated through it. It is
> * assumed that the classloader provided will be a parent loader of any runtime using it.
> @@ -21,6 +26,7 @@
> public ClassCache(ClassLoader classLoader, int max) {
> this.classLoader = classLoader;
> this.max = max;
> + this.defaultDomain = classLoader.getClass().getProtectionDomain();
> }
>
> public ClassCache(ClassLoader classLoader) {
> @@ -46,34 +52,6 @@
> }
> }
>
> - private static class OneShotClassLoader extends URLClassLoader {
> - private final static ProtectionDomain DEFAULT_DOMAIN =
> - JRubyClassLoader.class.getProtectionDomain();
> -
> - public OneShotClassLoader(ClassLoader parent) {
> - super(new URL[0], parent);
> - }
> -
> - // Change visibility so others can see it
> - public void addURL(URL url) {
> - super.addURL(url);
> - }
> -
> - public Class<?> defineClass(String name, byte[] bytes) {
> - return super.defineClass(name, bytes, 0, bytes.length, DEFAULT_DOMAIN);
> - }
> -
> - public Class<?> defineClass(String name, byte[] bytes, ProtectionDomain domain) {
> - return super.defineClass(name, bytes, 0, bytes.length, domain);
> - }
> - }
> -
> - private ReferenceQueue referenceQueue = new ReferenceQueue();
> - private Map<Object, KeyedClassReference> cache =
> - new ConcurrentHashMap<Object, KeyedClassReference>();
> - private ClassLoader classLoader;
> - private int max;
> -
> public ClassLoader getClassLoader() {
> return classLoader;
> }
> @@ -91,8 +69,7 @@
> if (weakRef == null || contents == null) {
> if (isFull()) return null;
>
> - OneShotClassLoader oneShotCL = new OneShotClassLoader(getClassLoader());
> - contents = (Class<T>)oneShotCL.defineClass(classGenerator.name(), classGenerator.bytecode());
> + contents = (Class<T>)AnonymousClassLoader.loadClass(classGenerator.name(), classGenerator.bytecode(), ClassCache.class, defaultDomain);
>
> cache.put(key, new KeyedClassReference(key, contents, referenceQueue));
> }
> Index: src/org/jruby/util/AnonymousClassLoader.java
> ===================================================================
> --- src/org/jruby/util/AnonymousClassLoader.java (revision 0)
> +++ src/org/jruby/util/AnonymousClassLoader.java (revision 0)
> @@ -0,0 +1,84 @@
> +package org.jruby.util;
> +
> +import java.lang.reflect.Constructor;
> +import java.lang.reflect.Method;
> +import java.net.URL;
> +import java.net.URLClassLoader;
> +import java.security.ProtectionDomain;
> +
> +public class AnonymousClassLoader {
> + private static final boolean DEBUG = true;
> +
> + private static final Constructor ANON_CL_CONSTRUCTOR;
> + private static final Method ANON_CL_LOADCLASS;
> + private static final boolean ANONYMOUS_SUPPORTED;
> +
> + static {
> + Exception exception = null;
> +
> + Class cls = null;
> + Constructor constructor = null;
> + Method method = null;
> +
> + try {
> + cls = Class.forName("java.dyn.AnonymousClassLoader");
> + constructor = cls.getConstructor(Class.class, byte[].class);
> + method = cls.getMethod("loadClass");
> + } catch (Exception e) {
> + exception = e;
> + }
> +
> + if (method == null && DEBUG) {
> + System.err.println("Could not load AnonymousClassLoader:");
> + exception.printStackTrace();
> + ANON_CL_CONSTRUCTOR = null;
> + ANON_CL_LOADCLASS = null;
> + ANONYMOUS_SUPPORTED = false;
> + } else {
> + ANON_CL_CONSTRUCTOR = constructor;
> + ANON_CL_LOADCLASS = method;
> + ANONYMOUS_SUPPORTED = false;
> + }
> + }
> +
> + public static Class loadClass(String name, byte[] bytecode, Class hostClass, ProtectionDomain domain) throws ClassNotFoundException {
> + Class result;
> +
> + if (ANONYMOUS_SUPPORTED) {
> + try {
> + Object anonCL = ANON_CL_CONSTRUCTOR.newInstance(hostClass, bytecode);
> + result = (Class)ANON_CL_LOADCLASS.invoke(anonCL);
> + } catch (Exception e) {
> + throw new ClassNotFoundException("Could not instantiate AnonymousClassLoader", e);
> + }
> + } else {
> + OneShotClassLoader oneShotCL = new OneShotClassLoader(hostClass.getClassLoader(), domain);
> + result = (Class)oneShotCL.defineClass(name, bytecode);
> + }
> +
> + return result;
> + }
> +
> + private static class OneShotClassLoader extends URLClassLoader {
> + private final ProtectionDomain defaultDomain;
> +
> + public OneShotClassLoader(ClassLoader parent, ProtectionDomain domain) {
> + super(new URL[0], parent);
> + defaultDomain = domain;
> + }
> +
> + // Change visibility so others can see it
> + public void addURL(URL url) {
> + super.addURL(url);
> + }
> +
> + public Class<?> defineClass(String name, byte[] bytes) {
> + return super.defineClass(name, bytes, 0, bytes.length, defaultDomain);
> + }
> +
> + public Class<?> defineClass(String name, byte[] bytes, ProtectionDomain domain) {
> + return super.defineClass(name, bytes, 0, bytes.length, domain);
> + }
> + }
> +}
> +
> Index: src/org/jruby/anno/InvokerGenerator.java
> ===================================================================
> --- src/org/jruby/anno/InvokerGenerator.java (revision 6575)
> +++ src/org/jruby/anno/InvokerGenerator.java (working copy)
> @@ -7,7 +7,6 @@
> import java.util.Map;
> import org.jruby.RubyModule.MethodClumper;
> import org.jruby.internal.runtime.methods.DumpingInvocationMethodFactory;
> -import org.jruby.util.JRubyClassLoader;
>
> public class InvokerGenerator {
> public static final boolean DEBUG = false;
> @@ -26,7 +25,7 @@
> br.close();
> }
>
> - DumpingInvocationMethodFactory dumper = new DumpingInvocationMethodFactory(args[1], new JRubyClassLoader(ClassLoader.getSystemClassLoader()));
> + DumpingInvocationMethodFactory dumper = new DumpingInvocationMethodFactory(args[1], InvokerGenerator.class);
>
> for (String name : classNames) {
> MethodClumper clumper = new MethodClumper();
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> mlvm-dev mailing list
> mlvm-dev at openjdk.java.net
> http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev
>
More information about the mlvm-dev
mailing list