Patch review: fix infinite loop in anonymousClassLoader.checkHostClass

[Rémi Forax]

Hi all.

There is a potential infite loop in the current code of the 
AnonymousClassLoader.
In checkHostClass, the caller and the callee need to be normalized to 
their top level classes,
but outer is not updated in the loop.
The attachment is a test case to reproduce the bug.

I propose the following patch:

diff --git a/src/share/classes/java/dyn/AnonymousClassLoader.java 
b/src/share/classes/java/dyn/AnonymousClassLoader.java
--- a/src/share/classes/java/dyn/AnonymousClassLoader.java
+++ b/src/share/classes/java/dyn/AnonymousClassLoader.java
@@ -275,6 +275,14 @@ class AnonymousClassLoader {
         this.classFileCP = classFileCP;
     }
 
+    private static Class<?> getTopLevelClass(Class<?> clazz) {
+      for(Class<?> outer = clazz.getDeclaringClass(); outer != null;
+          outer = outer.getDeclaringClass()) {
+        clazz = outer;
+      }
+      return clazz;
+    }
+   
     private static Class checkHostClass(Class hostClass) {
         // called only from the constructor
         // does a context-sensitive check on caller class
@@ -299,10 +307,8 @@ class AnonymousClassLoader {
             return hostClass;
 
         // normalize caller and callee to their top-level classes:
-        for (Class outer = caller.getDeclaringClass(); outer != null;
-             caller = outer) { }
-        for (Class outer = callee.getDeclaringClass(); outer != null;
-             callee = outer) { }
+        caller = getTopLevelClass(caller);
+        callee = getTopLevelClass(callee);
         if (caller == callee)
             return caller;
 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: AnonymousCheckHostClassInfiniteLoop.java
Type: text/x-java
Size: 327 bytes
Desc: not available
Url : http://mail.openjdk.java.net/pipermail/mlvm-dev/attachments/20080418/07b65863/attachment.bin