Scala and JSR 292

Sat Mar 28 01:49:28 PDT 2009

Hi John,

I had another thought that I wanted to add to my previous email. It
seems to me that one strategy the Scala compiler might be able to take
would be to actually provide an implementation of every non-final
method that it would otherwise inherit. Then in the static initializer
of each class, it could look around and set things up so the proper
implementation is invoked. So this way I think it could pick up
changes to existing methods caused by changes in traits. If so, then
the only problem remaining would be the ability to call new methods
that didn't exist before they were added to a trait the class mixes
in. So this sounds a bit like what interface injection is for. But
what would be nice is if the interface generated for each trait could
be somehow marked as "injectable", so that even though the class
already implements the interface, the class gets a chance to provide
implementations for those new methods. I'm a bit concerned about the
restriction that you can only invoke those new methods on the
interface, not the class, though. If that's the case I'm not sure it
really solves the problem completely.

Thanks.

Bill

On Fri, Mar 27, 2009 at 5:43 PM, Bill Venners <bill at artima.com> wrote:
> Hi John,
>
> You suggested we continue this on the mailing list, so here we go. For
> the rest of you, the context is trying to see if the JVM could make it
> easier to change a Scala trait without forcing a recompile of any
> class that mixes in that trait. I've included the previous part of the
> conversation below.
>
> I wanted to ask a few questions about interface injection. First, if a
> class already implements an interface, I assume the JVM won't try to
> do the injection process. Is that correct?
>
> I see your blog says that "If a method is injected with an interface,
> it is not accessible except via that interface itself." So I think for
> Scala to use this to solve the binary compatibility problem, every
> method call would need to be done via an injectable interface. Here's
> the scenario. In Version 1, we have:
>
> trait Fred
> class Bob with Fred
>
> This is compiled and works fine. Neither Bob nor Fred have any methods
> right now. If you call toString on a Bob instance, it would return
> something like "Bob at 955a65".
>
> trait Fred {
>  override def toString = "hello, world!"
> }
>
> Now only Fred is recompiled. What will happen right now is that when
> you call toString on a Bob instance it will just return a default
> toString, such as "Bob at 955a65", but if you recompile Bob and call
> toString again, you'll get "hello, world". So a changed trait can
> change a method that exists already on the class.
>
> So that's why it seems like to use interface injection to solve this
> problem, all calls to any class would need to allow the class to
> dynamically pick up a change on a trait. Thus any invocation would
> need to go through some interface that can be injected?
>
> I guess the realization I had was that this isn't just a method
> missing behavior, because the method could be there already, but with
> the wrong implementation. It almost seems like the first time any
> class is used, Scala would need to be able to given a chance to add
> new methods and change the implementation of existing methods based on
> changes to traits the class implements. The class will already
> implement an interface with the name of each trait, so it will already
> implement all those interfaces. The problem is the behavior of the
> methods declared in the class itself may need to be changed, and also
> some extra methods that aren't currently declard in the class may need
> to be added.
>
> Bill
>
> On Fri, Mar 20, 2009 at 12:19 AM, John Rose <John.Rose at sun.com> wrote:
>> On Mar 12, 2009, at 9:50 AM, Bill Venners wrote:
>>
>>> Hi John,
>>>
>>> Several people (such as Mark Reinhold and Joe Darcy) have suggested we
>>> contact you about a couple ways Scala could potentially benefit from
>>> VM support, because people felt it fit in with JSR 292. I'm not sure
>>> if Martin has spoken with you already about this. I wanted to make
>>> sure the conversation got started.
>>
>> I'm delighted to hear from you!
>>
>> Do you mind if we move this conversation to mlvm-dev at openjdk.java.net?  It
>> will be interesting to others there.
>>
>>> The main concern I believe is that in Scala, when you add a new
>>> concrete method to a trait, you need to recompile classes that mix in
>>> that trait. In other words it is a binary incompatible change to add a
>>> new concrete method to an existing trait. The reason is that the Scala
>>> compiler awards each class that mixes in a trait "forwarding methods"
>>> whose signatures match any concrete methods provided in that trait.
>>> These forwarding methods allow the method to be invoked on the class,
>>> and forward the call to a static method defined elsewhere that
>>> provides the implementation. (That's my understanding of how it works
>>> anyway. Martin will correct me if needed.) So if you add a concrete
>>> method to a trait, you also need to add a "forwarding method" to the
>>> classes that mix it in. Well if the VM would provide some kind of plan
>>> B if it finds it is trying to invoke a method on a class that isn't
>>> defined in that class, to allow the class itself to provide an
>>> implementation (kind of like "method missing" in dynamic languages),
>>> then this would allow the Scala compiler to at least make this a
>>> binary compatible change. Those methods might not invoke as
>>> efficiently, but it wouldn't break code.
>>
>> Interface injection is designed to help with this sort of thing.
>>
>> Mapping this facility to traits would be, I think, pretty easy:  Abstract
>> and concrete methods for trait T are declared in an injectable interface TI.
>>  Concrete trait methods would be placed in an abstract class (perhaps in
>> TI.Impl inner to the interface, just to be tidy).  The trait interface has
>> an injector function which binds to a given concrete target class X,
>> presumably one which Scala's type system has ensured has the desired methods
>> (in some form).  In any case, the injector function matches up each target
>> class X with the appropriate methods taken from X or from TI.Impl, as the
>> case may be.
>>
>> Method handles are used to specify the mix-and-match.  Since they hide
>> whether they are static or non-static, the decision to take from X or
>> TI.Impl is a late-bound decision.  Also, adapter logic can be folded in,
>> e.g., to box or unbox arguments if the signatures don't match exactly.  And
>> of course the names don't need to match; method handles hide method names
>> too.
>>
>> The effect is not of "method not found" but "interface not found", but it
>> amounts to the same functionality.
>>
>> Refs:
>>  http://blogs.sun.com/jrose/entry/interface_injection_in_the_vm
>>  http://wikis.sun.com/display/mlvm/InterfaceInjection
>>
>>
>>> Is that something that sounds like it might fit in with JSR 292?
>>
>> We are starting a prototype of this, and the 292 EG has discussed it
>> briefly, but has not yet taken it up seriously.
>>
>>> One other issue that is less critical is simply that Scala makes
>>> programmers very efficient at generating class files, because for
>>> every anonymous function you get an anonymous class. In a couple
>>> months of part time work I ended up with 12,000 class files of test
>>> code. So there's a scaling issue here. One solution is the Scala
>>> compiler could simply generate a JAR file, so you get one file. That's
>>> probably a fine solution actually, but I heard JSR 292 may include
>>> something like a method pointer. If there were a way that a compiler
>>> could stick little "private classes" inside one class file to
>>> represent some of those functions, then that could help reduce the
>>> class file proliferation you get with Scala. This isn't really a big
>>> deal, but I figured it is at least worthwhile to find out what those
>>> method pointers are intended for, and see if they could be of any help
>>> to Scala also.
>>
>> Method handles are pure functions, and so hide naming, static-ness, and
>> access restriction.  (Access is checked when the handle is made, not when it
>> is called as in reflection.)  With adapters, they can hide minor differences
>> in signatures also.  Finally, they can be curried.  They are thus ideal for
>> all sorts of small behavioral units.  Any class X can build any number and
>> variety of method handles by defining private static methods and reifying
>> them as needed as method handles.  The eventual user of the handle sees
>> nothing about the original method, and has no special access rights beyond
>> calling the method via the handle.
>>
>> The initial implementation of method handles has been under review for a few
>> weeks and will soon be available (for x86/32bit only, at the beginning) in
>> the OpenJDK JDK 7 builds.  The changeset under review, FYI, is:
>>  http://cr.openjdk.java.net/~jrose/6655638/
>>
>> Meanwhile, the JSR 292 EG is seriously analysing the proposed APIs.  It is
>> important that this work for all JVMs, not just Hotspot.
>>
>> Best wishes,
>> -- John
>>
>>> Thanks.
>>>
>>> Bill
>>> ----
>>> Bill Venners
>>> Artima, Inc.
>>> http://www.artima.com
>>
>>
>