[nestmates] javac does not desugar a protected method reference declared in super class in a different package

[Mandy Chung]

While assessing the compatibility concern when converting 
LambdaMetaFactory to hidden nestmate class, I'm getting to think whether 
we should consider backporting JDK-8227415 how javac determines if a 
protected method reference should be desugared to a static method, or 
whether looking for an alternative solution to allow existing compiled 
class to work.

Here is the example (see JDK-8227415 for details):

A simple case where p.Sub1 extends q.I (in two different packages)
and q.I::readFile is a protected method:

class Sub1 extends I {
     public void test(Path outputDir) {
         // the lambda proxy class generated for this::readFile method ref
         // is not a subclass of q.I
         test(this::readFile, outputDir);
     }
     private void test(Function<Path,String> func, Path path) {
         func.apply(path);
     }
}

p.Sub1 is a subclass of q.I whereas the lambda proxy class is not a 
subclass of q.I.WhenLambdaMetaFactory is converted to define lambda 
proxy class as a hidden nestmate via Lookup::defineHiddenClass,IAE would 
be thrown as it fails to access q.I::readFile.

Today it happens to work because lambda proxy class is defined as VM 
anonymous class which gets special access.  JEP 181 defines that 
nestmates allows access to private members in classes or interfaces in 
the same nest but not protected members in a different package as what 
VM anonymous class allows. So this javac adjustment could be made as a 
follow-up of JEP 181.

Another case:

class T {
     :
     private static void m(Function<Path,String> func, Path path) {
         func.apply(path);
     }

     public static class Sub extends I {
         public void test(Path outputDir) {
             // this method reference is desugared to a static method:
             // REF_invokeSpecial p/T$Sub.lambda$test$0:(Ljava/nio/file/Path;)Ljava/lang/String;
             // as Sub is in package p, a different package from its its superclass
             m(super::readFile, outputDir);

             // this compiles to indy LMF with:
             // REF_invokeVirtual q/I.readFile:(Ljava/nio/file/Path;)Ljava/lang/String;
             //
             // p.T is not a subclass of q.I and so it has no access to q.I::readFile
             m(this::readFile, outputDir);
         }
     }
}

In p.T.Sub::test, super::readFile will get desugared to a static method 
in p.T.Sub to invoke q.I::readFile.   However, this::readFile does not 
get desugared. When p.T.m invokes such a protected method reference, p.T 
has no access to q.I::readFile.

Srikanath has fixed this javac issue [1] in the nestmates branch. It's a 
compatibility risk that impacts existing compiled class if such a bridge 
method is missing and they will need recompiled.

A potential alternative solution that may invoke lambda proxy class to 
invoke the protected method via MethodHandle by calling Lookup::in and 
Lookup::findSpecial.

I don't know how high this compatibility concern is.  We only hit one 
JDK test failure due to this issue though. Do you consider fixing javac 
causes high compatibility risk?

If not, I do think we should consider backporting JDK-8227415 to 
existing releases to allow developers to prepare for the hidden classes 
in advance.

What do you think?

thanks
Mandy
[1] http://hg.openjdk.java.net/valhalla/valhalla/rev/15515009b454