Implementing a Smalltalk debugger with JSR292

[Mark Roos]

One of the key parts of Smalltalk is the 'live' debugger.  Unlike the 
general dynamic language features which
are well supported by the additions from JSR292 the debugger requires 
support which may not have been considered
as necessary to support dynamic languages.  So we were not sure we would 
be able to provide that portion
of the Smalltalk experience on the jvm.

The good news is that we were able to implement almost all of the 
Smalltalk debugger features using only the
services provided in the released jdk7.  I thought I would take a moment 
to describe how we  did it to both
demonstrate the approach and to solicit suggestions for improvements.

The Smalltalk debugger is 'live' in that it exists as a separate thread 
within the same process/memory space as the
thread being debugged.  This allows one to manipulate and inspect all 
objects from the same viewpoint as the
debugged thread.  Smalltalk offers the ability to inspect all instances of 
a class(type),  all references to a specific
object,  the variables on all levels of the stack, senders and 
implementers of methods, and the ability to single step
through method sends.  There is also the ability to restart a thread from 
any level of the stack but we opted to 
wait on the coro patch before implementing this ( I also don't use it as 
it can have quite a few side effects ).

The approach we took has two facets, we ( mainly oscar ) coded a C++ jvmti 
agent with a JNI interface which allowed
us to call some JVMTI apis from within the jvm being debugged and we added 
some logic to the callsite to handle the
stepping.

Implementers and senders of methods is handled via reflection on the 
classes and methods present so that was easy.

To support all instances and all references requires heap inspection which 
we get from using the jvmti heap functions.
This had some issues with some of the support classes for invoke dynamic 
but we were able to use a two pass tagging
approach to make sure we found all of the references to our objects.  This 
has to find objects both in arrays and in
instance vars.

Inspecting the stack was straight forward once we filtered the stack trace 
to only have our method sends present.  As
an option one can also inspect the full jvm trace.   Using the jvmti 
variable access api allows the locating the variable 
which is then placed into a static field of out debugger support class. 
This field is then access by Smallltalk via a 
primitive (in Smalltalk a primitive is the way we share with the 
underlying environment).  Once we have this object
we can manipulate its instance vars from the Smalltalk side as well.

When an error is thrown the thread is suspended ( we added some jvmti 
thread management apis just to get away
from the deprecated methods) and a new thread is launched with an instance 
of the debugger and a pointer to the
thread to debug.  At this point one can only inspect the thread locals and 
anything else in the object memory.  The
thread is not restartable so we kill it ( by sending ThreadDeath ).

But it the error is a halt or breakpoint we can then step the thread 
along.  We tried this with jvmti but is was broken
and seems to add quite a bit of delay to everything.  Plus its a callback 
approach which looked like a lot of work.  So
instead we tweaked the call site logic to add a debug check. I liked the 
way this worked a lot.

For a dynamic look up we already have a callsite with a target of one or 
more GWTs to select the implementation
which matches the receiver class.  What we have to do to implement a 
debugger is to place before the first GWT
a test which determines if this is the time to suspend.  Unfortunately 
GWTs are added to the end so we need a 
way to keep the test at the beginning (thanks to John and Réme suggestion) 
we can simply have a callsite have a target which
is another callsite.  The first callsite points at the test logic and the 
second gets the GWT chain.  One nice thing
is that we can revert to the single site version using a debug flag.

The  code to get the initial target for the bootstrap callsite looks like:
  private void setBootstrapTarget(MethodHandle mh){
    // get the appropriate initial call site sequence
    if( RtDebugger._debugEnable){
      // for debugging we need to have a sequence of methodHandles that is 
always the first
      // code executed when a callsite is invoked. This checks to see if 
we should
      // hold here for a debug step or continue.
      _realSite = new MutableCallSite(mh);  // this is the extra call site 
to hold the gwts
      MethodHandle invoker = _realSite.dynamicInvoker();
      MethodHandles.Lookup lookup=MethodHandles.lookup();
      MethodHandle debugEntry= null; 
      MethodType mt=MethodType.methodType(void.class, RtObject.class);
      try {
        debugEntry = lookup.findStatic(RtDebugger.class, "debugEntry", 
mt); 
      }
      catch (Throwable e) {
        e.printStackTrace();
      } 
      invoker = MethodHandles.foldArguments(invoker , debugEntry);  // 
does nothing except suspend if necessary 
      _realSite.setTarget(mh); 
      this.setTarget(invoker); 
    }else{
      // normal behavior is to just use the RtCallSite to hold the GWT 
chain as its target
    this.setTarget(mh);
    }
  }

The debug test code tests for the correct thread and for the stack depth ( 
jvmti api) and looks like:
  public static void debugEntry(RtObject arg){
    if(_debugThread == Thread.currentThread()){
      if(getStackSize(_debugThread) <= _debugDepth){
      // send debugger update notice
      notifyDebugStep();
      suspendThreadPrim(_debugThread);
      }
    }
  }

As you can see it did not take much code ( lots of thinking though ) to 
implement a stepping debugger.

regards
mark





-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.openjdk.java.net/pipermail/mlvm-dev/attachments/20111126/5e7e847b/attachment.html