More performance explorations

[John Rose]

On Jun 13, 2011, at 8:37 AM, Ola Bini wrote:

> On 2011-06-13 10.14, Ola Bini wrote:
>> I did a rebuild of MLVM+bsdport today, and I now see the missing class
>> crash again on my Mac too (to clarify, this was gone from my Mac for a
>> while, but came back. This means it's consistent with my JDK7 build on
>> Linux).
> 
> Oh my. I also did a rebuild of the JDK7 on Linux. Guess what? This error
> is now gone from that source... So now the below problem only happens on
> bsdport. *sigh*. So not very consistent then...

I have reproduced this error on my mac.  It happens on both 32- and 64-bit JVMs.  It is insensitive to stack size (60k .. 40m).

With -Xint the error does not occur.  With -XX:+PrintCompilation, here are the last 10 compilations that occur before the error:

   1574  111   !         seph.lang.bim.BaseBase::invoke (23 bytes)   made not entrant
   1574  113    b        seph.lang.Runtime$2::isTrue (2 bytes)
   1574  114    b        seph.lang.bim.NumberBase::minus (11 bytes)
   1575  115    b        seph.lang.Number::minus (9 bytes)
   1578  116    b        gnu.math.Numeric::sub (7 bytes)
   1582  114             seph.lang.bim.NumberBase::minus (11 bytes)   made not entrant
   1587  117    b        seph.lang.LexicalScope$One::assign (55 bytes)
   1643  118    b        seph$gen$abstraction$37$odd?::argument_0_0 (121 bytes)
   1647  119    b        seph$gen$abstraction$38$even?::argument_0_0 (121 bytes)
   1650  118             seph$gen$abstraction$37$odd?::argument_0_0 (121 bytes)   made not entrant

Excluding compilation of odd? and even? makes the error go away:
  java ... -XX:CompileCommand='exclude,*'{odd,even}'*::*' ...

The basic failure is here:
  Caused by: java.lang.NoClassDefFoundError: seph/lang/SephObject
	at java.lang.invoke.MethodHandle.invokeExact(MethodHandle.java)
	at seph$gen$abstraction$38$even?.argument_0_0(<eval>:7)

It is evidently a class loader scoping problem, either in the 292 implementation or in Seph itself.

Here is a command line that exhibits the bug:
  $JAVA_HOME/bin/java -Xbatch -XX:+PrintCompilation -cp lib/seph.jar:build/classes/test:lib/release/asm-all-4.0.jar:$JUNIT4_JAR org.junit.runner.JUnitCore seph.lang.code.BasicSanityTest

Moving seph.jar onto the BCP works around the bug:
  $JAVA_HOME/bin/java ... -Xbootclasspath/a:lib/seph.jar ...

In other words, somebody (probably the code for argument_0_0) is looking on the BCP loader for the Seph runtime.

Looking at the various dumps, I do not see what is the content of argument_0_0, but AbstractionCompiler.java indicates that it simply mentions SephObject as a type.  If this type shows up on an invokedynamic (or MH.invokeExact), it will be lazily reified to SephObject.class as a part of a MethodType, on first execution.  This is probably what causes the NCDF error we are seeing.

The class containing argument_0_0 is apparently loaded here in AbstractionCompiler.java:
  abstractionClass = seph.lang.Runtime.LOADER.defineClass(className, classBytes);

This defines the dynamically loaded code in a singular SephClassLoader.  I have a question:  Why doesn't the SephClassLoader delegate lookups to its parent?  It seems to me that if code in that loader needs to reify SephObject, it must somehow access the parent of SephClassLoader.  Is this really happening?  I suggest putting tracing code into SephClassLoader to see whether it is being asked about SephObject just before the error happens.

One thing that suggests it is 292's fault is the way -Xint hides the bug.  It seems to me that if it is a class loader config problem, -Xint would make no difference.  It is as if the interpreter is using one class loader while the compiled code is using a different one.

This would be easier to debug if we could boil this failure down to a simpler case, with a single MH.invoke call whose signature mentions the non-global type (SephObject in this case).  Something like this:
  class GeneratedC { ... invoke((LocalB)null) ... }
  java LocalA
  class LocalA { ... new MyClassLoader(LocalA.getClassLoader()).loadClass(...bytes for GeneratedC...)

BTW, the -Xbatch command makes for a more deterministic interaction between the compile queue (which is concurrent) and the application.

-- John