From brian.goetz at oracle.com  Mon Jan 11 19:35:33 2016
From: brian.goetz at oracle.com (Brian Goetz)
Date: Mon, 11 Jan 2016 14:35:33 -0500
Subject: Test message, please ignore.
Message-ID: <56940405.1090209@oracle.com>

This is a test message.  Please ignore.



From brian.goetz at oracle.com  Wed Jan 20 19:56:22 2016
From: brian.goetz at oracle.com (Brian Goetz)
Date: Wed, 20 Jan 2016 14:56:22 -0500
Subject: Nestmates
Message-ID: <569FE666.9070103@oracle.com>

This topic is at the complete opposite end of the spectrum from topics 
we've been discussing so far.  It's mostly an implementation story, and 
of particular interest to the compiler and VM implementers here.


Background
----------

Since Java 1.1, the rules for accessibility when inner classes are 
involved at the language level are not fully aligned with those at the 
VM level.  In particular, private and protected access from and to inner 
classes is stricter in the VM than in the language, meaning that in 
these cases, the static compiler emits an access bridge (access$000) 
which effectively downgrades the accessed member's accessibility to 
package.

Access bridges have some disadvantages.  They're ugly, but that's not a 
really big deal.  They're imprecise; they allow wider-than-necessary 
access to the member.  Again, this is not a huge deal on its own.  But 
the real problem is the complexity of the compiler implementation when 
we add generic specialization to the story.

Specialization adds a new category of cross-class accesses that are 
allowed at the language level but not at the VM level, which would 
dramatically increase the need for, and complexity of, accessibility 
bridges.  For example:

class Foo<any T> {
     private T t;

     void m(Foo<int> foo) {
         int i = foo.t;
     }
}

Now we execute:

     Foo<long> fl = ...
     Foo<int> fi = ...
     fl.m(fi)

The spirit of the language rules clearly allow the access from Foo<long> 
to Foo<int>.t -- they are in the "same class".  But at the VM level, 
Foo<int> and Foo<long> are different classes, so the access from 
Foo<long> to a private member of Foo<int> is disallowed.

One reason that this increases the complexity, and not just the number, 
of accessibility bridges is that bridges are (currently) static methods; 
if they represent instance methods, we pass the receiver as the first 
argument.  For access between inner classes, this is fine, but when it 
comes to access between specializations, this breeds new complexity -- 
because the method signature of the accessor needs to be specialized 
based on the type parameters of the receiver.  This interaction means 
the current static-accessor solution would need its own special, ad-hoc 
treatment in specialization, adding to the complexity of specialization.

More generally, this situation arises in any case where a single logical 
unit of encapsulation at the source level is split into multiple runtime 
classes (inner classes, specialization classes, synthetic helper 
classes.)  We propose to address this problem more generally, by 
providing a mechanism where language compilers can indicate that 
multiple runtime classes live in the same unit of encapsulation.  We do 
so by (a) adding metadata to classes to indicate which classes belong in 
the same encapsulation unit and (b) relaxing some VM accessibility rules 
to bring them more in alignment with the language level rules.


Overview
--------

Our proposed strategy is to reify the relationship between classes that 
are members of the same _nest_.  Nestmate-ness can then be considered in 
access control decisions (JVMS 5.4.4).

Classes that derive from a common source class form a _nest_, and two 
classes in the same nest are called _nestmates_.  Nestmate-ness is an 
equivalence relation (reflexive, symmetric, and transitive.)  Nestmates 
of a class C include C's inner classes, synthetic classes generated as 
part of translating C, and specializations thereof.

Since nestmate-ness is an equivalence relation, it forms a partition 
over classes, and we can nominate a canonical member for each partition. 
  We nominate the "top" (outermost lexically enclosing) class in the 
nest as the canonical member; this is the top-level source class from 
which all other nestmates derive.

This makes it easy to calculate nestmate-ness for two classes C and D; C 
and D are nestmates if their "top" class is the same.

Example
-------

class Top<any T> {
     class A<any U> { }
         class B<V> { }
     }

     <any T> void genericMethod() { }
}

When we compile this, we get:
    Top.class                   // Top
    Top$A.class                 // Inner class Top.A
    Top$A$B.class               // Inner class Top.A.B
    Top$Any.class               // Wildcard interface for Top
    Top$A$Any.class             // Wildcard interface for Top.A
    Top$genericMethod.class     // Holder class for generic method

The explicit classes Top, Top.A, and Top.A.B, the synthetic $Any 
classes, and the synthetic holder class for genericMethod, along with 
all of their specializations, form a nest.  The top member of this nest 
is Top.

Since nestmates all derive from a common top-level class, they are by 
definition in the same package and module.  A class can be in only one 
nest at once.


Runtime Representation
----------------------

We represent nestmate-ness with two new attributes -- one in the top 
member, which describes all the members of the nest, and one in each 
member, which requests access to the nest.

     NestTop {
         u2 name_index;
         u4 length;
         u2 child_count;
         u2 childClazz[child_count];
     }

     NestChild {
         u2 name_index;
         u4 length;
         u2 topClazz;
     }

If a class has a NestTop attribute, its nest top is itself. If a class 
has a NestChild attribute, its nest top is the class named via topClazz. 
If a class is a specialization of another class, its nest top is the 
nest top of the class for which it is a specialization.

When loading a class with a NestChild attribute, the VM can verify that 
the requested nest permits it as a member, and reject the class if the 
child and top do not agree.

The NestTop attribute can enumerate all inner classes and synthetic 
classes, but cannot enumerate all specializations thereof. When creating 
a specialization of a class, the VM records the specialization as being 
a member of whatever nest the template class was a member of.


Semantics
---------

The accessibility rules here are strictly additions; nestmate-ness 
creates additional accessibility over and above the existing rules.

Informally:
   - A class can access the private members of its nestmates;
   - A class can access protected members inherited by its nestmates.

This is slightly broader than the language semantics (but still less 
broad than what we do today with access bridges.)  The static compiler 
can continue to enforce the same rules, and the VM will allow these 
accesses without bridges.  (We could make the proposal match the 
language semantics more closely at the cost of additional complexity, 
but its not clear this is worthwhile.)

For private access, we can add the following to 5.4.4:
   - A class C may access a private member D.R if C and D are nestmates.

The rules for protected members are more complicated.  5.4.3.{2,3} first 
resolve the true owner of the member, and feed that to 5.4.4; this 
process throws away some needed information.  We would augment 
5.4.3.{2,3} as follows:
  - When performing member resolution from class C on member D.R, we 
remember both D (the target class) and E (the resolved class) and make 
them both available to 5.4.4.

We then adjust 5.4.4 accordingly, by adding:
  - If R is protected, and C and D are nestmates, and E is accessible to 
D, then access is allowed.


Examples
--------

For private fields, we generate access bridges whenever an inner class 
accesses a private member (field or method) of the enclosing class, or 
of another inner class in the same nest.

In the classes below, the accesses shown are all permitted by the 
language spec (child to parent, sibling to sibling, sibling to child of 
sibling, etc), and the ones requiring access bridges are noted.

     class Foo {
         public static Foo aFoo;
         public static Inner1 aInner1;
         public static Inner1.Inner2 aInner2;
         public static Inner3 aInner3;

         private int foo;

         class Inner1 {
             private int inner1;

             class Inner2 {
                 private int inner2;
             }

             void m() {
                 int i = aFoo.foo           // bridge
                       + aInner1.inner1
                       + aInner2.inner2     // bridge
                       + aInner3.inner3;    // bridge
             }
         }

         class Inner3 {
             private int inner3;

             void m() {
                 int i = aFoo.foo           // bridge
                       + aInner1.inner1     // bridge
                       + aInner2.inner2     // bridge
                       + aInner3.inner3;
             }
         }
     }

For protected members, the situation is more subtle.

     /* package p1 */
     public class Sup {
         protected int pro;
     }

     /* package p2 */
     public class Sub extends p1.Sup {
         void test() {
             ... pro ... //no bridge (invokespecial)
         }

         class Inner {
             void test() {
                 ... sub.pro ... // bridge generated in Sub
             }
         }
     }

Here, the VM rules allow Sub to access protected members of Sup, but for 
accesses from Sub.Inner or Sibling to Sub.pro to succeed, Sub provides 
an access bridge (which effectively makes Sub.pro package-visible 
throughout package p2.)

The rules outlined eliminate access bridges in all of these cases.


Interaction with defineAnonymousClass
-------------------------------------

Nestmate-ness also potentially connects nicely with 
Unsafe.defineAnonymousClass.  The intuitive notion of dAC is, when you 
load anonymous class C with a host class of H, that C is being "injected 
into" H -- access control decisions for C are made using H's 
credentials.  With a formal notion of nestmateness, we can bring 
additional predictability to dAC by saying that C is injected into H's 
nest.



From vlad.ureche at gmail.com  Thu Jan 21 23:10:10 2016
From: vlad.ureche at gmail.com (Vlad Ureche)
Date: Fri, 22 Jan 2016 00:10:10 +0100
Subject: Nestmates
In-Reply-To: <569FE666.9070103@oracle.com>
References: <569FE666.9070103@oracle.com>
Message-ID: <CAKBs1_r1VUh-5-uLHcXQYntdY_6C-QbJW-oh21CSGh2ttnacdw@mail.gmail.com>

Thank you for sharing these insights Brian!

I think I understand the problem and the solution, but let me ask three
questions to make sure I understood well:

1) The NestTop attribute must contain the child classes (except
specializations and lambdas, which are added dynamically), right? Is this
for security, so another class could not pose as a NestChild to access
private data? What about allowing the NestTop attribute to say "anyone who
wants to nest here is welcome to do so"?
2) Why did you choose to have symmetry and transitivity? I understand that
having an equivalence relation allows partitioning, but it's not clear to
me why partitioning is important in this case.
3) Why is the NestChild limited to a single top class?

These questions stem from pondering whether we can use the nestmates
mechanism to implement Scala's enclosing-entity-private access specifiers
(e.g. a variable in class List can be private[scala.collection.List] or
private[scala.collection] or private[scala])... Still, I don't think this
can be done at the granularity required by Scala, so we'll continue to have
name-mangled accessors where necessary :(

Thanks,
Vlad

From brian.goetz at oracle.com  Fri Jan 22 01:25:44 2016
From: brian.goetz at oracle.com (Brian Goetz)
Date: Thu, 21 Jan 2016 20:25:44 -0500
Subject: Nestmates
In-Reply-To: <CAKBs1_r1VUh-5-uLHcXQYntdY_6C-QbJW-oh21CSGh2ttnacdw@mail.gmail.com>
References: <569FE666.9070103@oracle.com>
	<CAKBs1_r1VUh-5-uLHcXQYntdY_6C-QbJW-oh21CSGh2ttnacdw@mail.gmail.com>
Message-ID: <56A18518.1050204@oracle.com>

Certainly we would like it to be useful beyond Java. Extrapolating from 
Java's situation, we've told the users that a "class" is everything 
between the outermost { ... }, and defined encapsulation rules based on 
this notion of class boundary.  For various expedient reasons, at 
various points in Java's evolution (inner classes, lambda proxies, 
specialization) we've chosen to implement a single "user class" as 
multiple "VM classes", necessitating hacks and backdoors to preserve the 
user's expectations (and sometimes failing.)  It seems reasonable that 
other languages would similarly want to define an encapsulation domain 
that spans multiple VM classes (like, perhaps, C#'s "partial class" 
construct).  The nestmate concept is motivated by the problem we have 
*now*, but we'd like it to be useful beyond the immediate situation.  On 
the other hand, we don't want to extrapolate wildly to what some 
language might want, or define something that is so complicated that we 
can't predict the security consequences.

More inline.

On 1/21/2016 6:10 PM, Vlad Ureche wrote:
> Thank you for sharing these insights Brian!
>
> I think I understand the problem and the solution, but let me ask 
> three questions to make sure I understood well:
>
> 1) The NestTop attribute must contain the child classes (except 
> specializations and lambdas, which are added dynamically), right? Is 
> this for security, so another class could not pose as a NestChild to 
> access private data? What about allowing the NestTop attribute to say 
> "anyone who wants to nest here is welcome to do so"?

Security is indeed the motivation.  It's the VM's job to enforce what 
private means; while there are plenty of ways to dodge this (and we'd 
like to close some of them down), accessibility control is part of the 
integrity model of the JVM. In order to inject yourself into another 
classes' nest, you'd need to modify that classes classfile -- in which 
case you can do anything you want with their data anyway (just make all 
the fields public.)  The two attributes form a handshake.

An easy way to say "anyone who wants to nest here may do so" is to make 
your state public, and possibly enforce language-level access control to 
make sure that they've actually requested such nesting.

> 2) Why did you choose to have symmetry and transitivity? I understand 
> that having an equivalence relation allows partitioning, but it's not 
> clear to me why partitioning is important in this case.

Simplicity.  By having a partitioning, and a distinguished canonical 
member, it is very easy to determine what access control credentials are 
in play.  It also goes back to the user model -- there is some unit of 
encapsulation that the language defines (class) that determines what 
"private" means.

We could do something fancy with friends, and transitive friends, and 
marriage brokers, and itinerant priests selling indulgences, but it 
doesn't seem that the return on complexity there is worth it.

> 3) Why is the NestChild limited to a single top class?

Again, because the encapsulation model is supposed to be simple.  Just 
as each line of code belongs to only one source file, each bytecode 
belongs to one nest.  \\

Perhaps another way to think of it is: its the simplest thing we could 
think of that solves our current problem and yet seems not completely 
ad-hoc :)

> These questions stem from pondering whether we can use the nestmates 
> mechanism to implement Scala's enclosing-entity-private access 
> specifiers (e.g. a variable in class List can be 
> private[scala.collection.List] or private[scala.collection] or 
> private[scala])... Still, I don't think this can be done at the 
> granularity required by Scala, so we'll continue to have name-mangled 
> accessors where necessary :(

Right.  These are overlapping scopes of accessibility, more like 
"virtual packages" than "nestmates".  This seems like a slightly 
different problem.



From peter.levart at gmail.com  Fri Jan 22 09:03:01 2016
From: peter.levart at gmail.com (Peter Levart)
Date: Fri, 22 Jan 2016 10:03:01 +0100
Subject: Nestmates
In-Reply-To: <569FE666.9070103@oracle.com>
References: <569FE666.9070103@oracle.com>
Message-ID: <56A1F045.2010408@gmail.com>

Hi Brian,

If I understand correctly, the "top" class is there just to simplify the 
calculation of whether two classes belong to the same nest. Are there 
any other functions that might be attached to the "top" class? Will the 
top class have to be loaded in order to verify access of one peer to 
another peer? Or will it just have to be parsed to extract the NestTop 
attribute?

An alternative might be a symmetric configuration where each nest-mate 
lists all nest-mates in a single Nest attribute, with possible 
additional bit to flag the "top" member if it is to have a special role. 
In such arrangement the resource (.class file) of the top class need not 
even be opened to verify the access of one peer to another peer. 
Nestmate-ness would still be an equivalence relation and the consistency 
of the common "Nest" attribute would be verified dynamically as each 
member of the nest gets loaded lazily...

Regards, Peter

On 01/20/2016 08:56 PM, Brian Goetz wrote:
> This topic is at the complete opposite end of the spectrum from topics 
> we've been discussing so far.  It's mostly an implementation story, 
> and of particular interest to the compiler and VM implementers here.
>
>
> Background
> ----------
>
> Since Java 1.1, the rules for accessibility when inner classes are 
> involved at the language level are not fully aligned with those at the 
> VM level.  In particular, private and protected access from and to 
> inner classes is stricter in the VM than in the language, meaning that 
> in these cases, the static compiler emits an access bridge 
> (access$000) which effectively downgrades the accessed member's 
> accessibility to package.
>
> Access bridges have some disadvantages.  They're ugly, but that's not 
> a really big deal.  They're imprecise; they allow wider-than-necessary 
> access to the member.  Again, this is not a huge deal on its own.  But 
> the real problem is the complexity of the compiler implementation when 
> we add generic specialization to the story.
>
> Specialization adds a new category of cross-class accesses that are 
> allowed at the language level but not at the VM level, which would 
> dramatically increase the need for, and complexity of, accessibility 
> bridges.  For example:
>
> class Foo<any T> {
>     private T t;
>
>     void m(Foo<int> foo) {
>         int i = foo.t;
>     }
> }
>
> Now we execute:
>
>     Foo<long> fl = ...
>     Foo<int> fi = ...
>     fl.m(fi)
>
> The spirit of the language rules clearly allow the access from 
> Foo<long> to Foo<int>.t -- they are in the "same class".  But at the 
> VM level, Foo<int> and Foo<long> are different classes, so the access 
> from Foo<long> to a private member of Foo<int> is disallowed.
>
> One reason that this increases the complexity, and not just the 
> number, of accessibility bridges is that bridges are (currently) 
> static methods; if they represent instance methods, we pass the 
> receiver as the first argument.  For access between inner classes, 
> this is fine, but when it comes to access between specializations, 
> this breeds new complexity -- because the method signature of the 
> accessor needs to be specialized based on the type parameters of the 
> receiver.  This interaction means the current static-accessor solution 
> would need its own special, ad-hoc treatment in specialization, adding 
> to the complexity of specialization.
>
> More generally, this situation arises in any case where a single 
> logical unit of encapsulation at the source level is split into 
> multiple runtime classes (inner classes, specialization classes, 
> synthetic helper classes.)  We propose to address this problem more 
> generally, by providing a mechanism where language compilers can 
> indicate that multiple runtime classes live in the same unit of 
> encapsulation.  We do so by (a) adding metadata to classes to indicate 
> which classes belong in the same encapsulation unit and (b) relaxing 
> some VM accessibility rules to bring them more in alignment with the 
> language level rules.
>
>
> Overview
> --------
>
> Our proposed strategy is to reify the relationship between classes 
> that are members of the same _nest_.  Nestmate-ness can then be 
> considered in access control decisions (JVMS 5.4.4).
>
> Classes that derive from a common source class form a _nest_, and two 
> classes in the same nest are called _nestmates_. Nestmate-ness is an 
> equivalence relation (reflexive, symmetric, and transitive.)  
> Nestmates of a class C include C's inner classes, synthetic classes 
> generated as part of translating C, and specializations thereof.
>
> Since nestmate-ness is an equivalence relation, it forms a partition 
> over classes, and we can nominate a canonical member for each 
> partition.  We nominate the "top" (outermost lexically enclosing) 
> class in the nest as the canonical member; this is the top-level 
> source class from which all other nestmates derive.
>
> This makes it easy to calculate nestmate-ness for two classes C and D; 
> C and D are nestmates if their "top" class is the same.
>
> Example
> -------
>
> class Top<any T> {
>     class A<any U> { }
>         class B<V> { }
>     }
>
>     <any T> void genericMethod() { }
> }
>
> When we compile this, we get:
>    Top.class                   // Top
>    Top$A.class                 // Inner class Top.A
>    Top$A$B.class               // Inner class Top.A.B
>    Top$Any.class               // Wildcard interface for Top
>    Top$A$Any.class             // Wildcard interface for Top.A
>    Top$genericMethod.class     // Holder class for generic method
>
> The explicit classes Top, Top.A, and Top.A.B, the synthetic $Any 
> classes, and the synthetic holder class for genericMethod, along with 
> all of their specializations, form a nest.  The top member of this 
> nest is Top.
>
> Since nestmates all derive from a common top-level class, they are by 
> definition in the same package and module.  A class can be in only one 
> nest at once.
>
>
> Runtime Representation
> ----------------------
>
> We represent nestmate-ness with two new attributes -- one in the top 
> member, which describes all the members of the nest, and one in each 
> member, which requests access to the nest.
>
>     NestTop {
>         u2 name_index;
>         u4 length;
>         u2 child_count;
>         u2 childClazz[child_count];
>     }
>
>     NestChild {
>         u2 name_index;
>         u4 length;
>         u2 topClazz;
>     }
>
> If a class has a NestTop attribute, its nest top is itself. If a class 
> has a NestChild attribute, its nest top is the class named via 
> topClazz. If a class is a specialization of another class, its nest 
> top is the nest top of the class for which it is a specialization.
>
> When loading a class with a NestChild attribute, the VM can verify 
> that the requested nest permits it as a member, and reject the class 
> if the child and top do not agree.
>
> The NestTop attribute can enumerate all inner classes and synthetic 
> classes, but cannot enumerate all specializations thereof. When 
> creating a specialization of a class, the VM records the 
> specialization as being a member of whatever nest the template class 
> was a member of.
>
>
> Semantics
> ---------
>
> The accessibility rules here are strictly additions; nestmate-ness 
> creates additional accessibility over and above the existing rules.
>
> Informally:
>   - A class can access the private members of its nestmates;
>   - A class can access protected members inherited by its nestmates.
>
> This is slightly broader than the language semantics (but still less 
> broad than what we do today with access bridges.)  The static compiler 
> can continue to enforce the same rules, and the VM will allow these 
> accesses without bridges.  (We could make the proposal match the 
> language semantics more closely at the cost of additional complexity, 
> but its not clear this is worthwhile.)
>
> For private access, we can add the following to 5.4.4:
>   - A class C may access a private member D.R if C and D are nestmates.
>
> The rules for protected members are more complicated.  5.4.3.{2,3} 
> first resolve the true owner of the member, and feed that to 5.4.4; 
> this process throws away some needed information.  We would augment 
> 5.4.3.{2,3} as follows:
>  - When performing member resolution from class C on member D.R, we 
> remember both D (the target class) and E (the resolved class) and make 
> them both available to 5.4.4.
>
> We then adjust 5.4.4 accordingly, by adding:
>  - If R is protected, and C and D are nestmates, and E is accessible 
> to D, then access is allowed.
>
>
> Examples
> --------
>
> For private fields, we generate access bridges whenever an inner class 
> accesses a private member (field or method) of the enclosing class, or 
> of another inner class in the same nest.
>
> In the classes below, the accesses shown are all permitted by the 
> language spec (child to parent, sibling to sibling, sibling to child 
> of sibling, etc), and the ones requiring access bridges are noted.
>
>     class Foo {
>         public static Foo aFoo;
>         public static Inner1 aInner1;
>         public static Inner1.Inner2 aInner2;
>         public static Inner3 aInner3;
>
>         private int foo;
>
>         class Inner1 {
>             private int inner1;
>
>             class Inner2 {
>                 private int inner2;
>             }
>
>             void m() {
>                 int i = aFoo.foo           // bridge
>                       + aInner1.inner1
>                       + aInner2.inner2     // bridge
>                       + aInner3.inner3;    // bridge
>             }
>         }
>
>         class Inner3 {
>             private int inner3;
>
>             void m() {
>                 int i = aFoo.foo           // bridge
>                       + aInner1.inner1     // bridge
>                       + aInner2.inner2     // bridge
>                       + aInner3.inner3;
>             }
>         }
>     }
>
> For protected members, the situation is more subtle.
>
>     /* package p1 */
>     public class Sup {
>         protected int pro;
>     }
>
>     /* package p2 */
>     public class Sub extends p1.Sup {
>         void test() {
>             ... pro ... //no bridge (invokespecial)
>         }
>
>         class Inner {
>             void test() {
>                 ... sub.pro ... // bridge generated in Sub
>             }
>         }
>     }
>
> Here, the VM rules allow Sub to access protected members of Sup, but 
> for accesses from Sub.Inner or Sibling to Sub.pro to succeed, Sub 
> provides an access bridge (which effectively makes Sub.pro 
> package-visible throughout package p2.)
>
> The rules outlined eliminate access bridges in all of these cases.
>
>
> Interaction with defineAnonymousClass
> -------------------------------------
>
> Nestmate-ness also potentially connects nicely with 
> Unsafe.defineAnonymousClass.  The intuitive notion of dAC is, when you 
> load anonymous class C with a host class of H, that C is being 
> "injected into" H -- access control decisions for C are made using H's 
> credentials.  With a formal notion of nestmateness, we can bring 
> additional predictability to dAC by saying that C is injected into H's 
> nest.
>
>


From vlad.ureche at gmail.com  Fri Jan 22 09:35:24 2016
From: vlad.ureche at gmail.com (Vlad Ureche)
Date: Fri, 22 Jan 2016 10:35:24 +0100
Subject: Nestmates
In-Reply-To: <56A1F045.2010408@gmail.com>
References: <569FE666.9070103@oracle.com> <56A1F045.2010408@gmail.com>
Message-ID: <CAKBs1_rnJc0WK4Y_Bij8CR98qRqNLi3SLQXJyTuKV80J3Bs6ew@mail.gmail.com>

Thank you for your answers Brian!
Indeed, the current model is easy to understand and reason about, which is
the most important aspect. And it maps well to Java source code.

Regarding Scala, the main problem is the granularity: setting the
visibility of each method individually goes far beyond the curly brace
encapsulation model. I think the only reasonable way short of having a
Scala VM is to keep generating accessors (or, as you suggested, making
members public and reasoning about the accessibility at language level).

Regarding Peter's question, I assume the reason you want to have a top of
the nesting hierarchy is because you need to dynamically add new nestmates,
which is easier to do in one place than in all nestmates. Right?

Thanks,
Vlad

From brian.goetz at oracle.com  Fri Jan 22 15:04:02 2016
From: brian.goetz at oracle.com (Brian Goetz)
Date: Fri, 22 Jan 2016 10:04:02 -0500
Subject: Nestmates
In-Reply-To: <56A1F045.2010408@gmail.com>
References: <569FE666.9070103@oracle.com> <56A1F045.2010408@gmail.com>
Message-ID: <56A244E2.5070102@oracle.com>


> If I understand correctly, the "top" class is there just to simplify 
> the calculation of whether two classes belong to the same nest. 

It's a little more than that, but this is part of it.  Having a 
canonical member simplifies the calculation of "are A and B members of 
the same nest."  Conveniently, there *is* an obvious canonical member, 
so we might as well not make our life harder.

We need a way to refer to a nest.  Having a canonical member is ideal 
for that.

We need some way of (in)validating the claim "I am a member of the same 
nest as java.lang.String".  Having a canonical member, who can answer 
such questions authoritatively, is ideal for that.

Because the intention is that "nest" corresponds to some logically 
grouped source unit, and therefore a class may be a member of only one 
nest, we need some way of enforcing that a nest forms a partition.  
Having a canonical member makes this far easier.

> Are there any other functions that might be attached to the "top" 
> class? Will the top class have to be loaded in order to verify access 
> of one peer to another peer? Or will it just have to be parsed to 
> extract the NestTop attribute?

The obvious implementation strategies would likely bring us there. It is 
conceivable that we could delay loading the Top until its nest-ness 
credentials are needed, but that seems like it would add a lot of 
implementation complexity (and would keep us from failing fast.)  So its 
likely that this would happen.

However, this seems pretty likely without nestmate-ness too.  How often 
do we instantiate an enclosed class without loading the enclosing 
class?  (Never for instance inner classes.)  It happens, but usually 
only in cases where nesting is used "for organizational purposes only" 
(and in those cases, the enclosing class is usually trivial.)  So this 
doesn't seem a terrible constraint to me.

What other functions did you have in mind for what the Top can do to 
help its children?




From brian.goetz at oracle.com  Fri Jan 22 15:09:30 2016
From: brian.goetz at oracle.com (Brian Goetz)
Date: Fri, 22 Jan 2016 10:09:30 -0500
Subject: Nestmates
In-Reply-To: <CAKBs1_rnJc0WK4Y_Bij8CR98qRqNLi3SLQXJyTuKV80J3Bs6ew@mail.gmail.com>
References: <569FE666.9070103@oracle.com> <56A1F045.2010408@gmail.com>
	<CAKBs1_rnJc0WK4Y_Bij8CR98qRqNLi3SLQXJyTuKV80J3Bs6ew@mail.gmail.com>
Message-ID: <56A2462A.8010907@oracle.com>

>  Regarding Scala, the main problem is the granularity: setting the 
> visibility of each method
> individually goes far beyond the curly brace encapsulation model. I 
> think the only
> reasonable way short of having a Scala VM is to keep generating 
> accessors (or, as you
> suggested, making members public and reasoning about the accessibility 
> at language level).

It seems that Scala's accessibility model is finer-grained than what the 
JVM has in mind.  The JVM envisions three concentric rings -- everyone 
(public), library (package), and source file (private), plus the 
weirdness that is protected. Nestmates doesn't change that basic model, 
it just allows the compiler to have more latitude in clarifying the 
boundaries that determine private-ness (and to some degree protected-ness.)

> Regarding Peter's question, I assume the reason you want to have a top 
> of the nesting hierarchy is because you need to dynamically add new 
> nestmates, which is easier to do in one place than in all nestmates. 
> Right?
>

Dynamism is a part of it, but also (a) minimizing the error modes 
introduced by separate compilation and/or broken compilers (that then 
have to be checked at load time), and (b) minimizing the runtime 
overhead on classloading.  The current proposal is very light in terms 
of the load-time overhead -- for a top class, you just have to record 
the NestTop attribute, and for a child class, you just have to validate 
it (which is a straight lookup.)



From brian.goetz at oracle.com  Fri Jan 22 16:52:17 2016
From: brian.goetz at oracle.com (Brian Goetz)
Date: Fri, 22 Jan 2016 11:52:17 -0500
Subject: Model 3 classfile design document
Message-ID: <56A25E41.6040508@oracle.com>

Please find a document here:

http://cr.openjdk.java.net/~briangoetz/valhalla/eg-attachments/model3-01.html

that describes our current thinking for evolving the classfile format to 
clearly and efficiently represent parametric polymorphism.  The early 
concepts of this approach were outlined in my talk at JVMLS last year; 
this represents a refinement of those ideas, and a reasonable "stake in 
the ground" description of what seems the most sensible way to balance 
preserving parametric information in the classfile without imposing 
excessive runtime costs for loading specializations.

We're working on an updated compiler prototype which people will be able 
to play with soon (along with a formal model.)

Please ask questions!

Some things this document does not address yet:
  - How we deal with types implicit in the bytecodes (aload vs iload) 
and how they get specialized;
  - How we represent restricted methods in the classfile;
  - How we represent the wildcard type Foo<any>



From ali.ebrahimi1781 at gmail.com  Sat Jan 23 10:45:40 2016
From: ali.ebrahimi1781 at gmail.com (Ali Ebrahimi)
Date: Sat, 23 Jan 2016 14:15:40 +0330
Subject: Model 3 classfile design document
In-Reply-To: <56A25E41.6040508@oracle.com>
References: <56A25E41.6040508@oracle.com>
Message-ID: <CAA0cW5AY7_uNgpOhjYNV2uACn0nuVHWxvJyWHCStXe1OWkSwuQ@mail.gmail.com>

Hi,
Excellent work.
I think there is typo in document:
"
 Method descriptor for List<long> m(List<int>):
 MethodDescriptor[1, ParamType['L', "List", "I"], ParamType['L', "List",
"J"]]

"
*Should not be this as:*

 MethodDescriptor[1, ParamType['L', "List", "J"], ParamType['L', "List",
"I"]]



On Fri, Jan 22, 2016 at 8:22 PM, Brian Goetz <brian.goetz at oracle.com> wrote:

> Please find a document here:
>
>
> http://cr.openjdk.java.net/~briangoetz/valhalla/eg-attachments/model3-01.html
>
> that describes our current thinking for evolving the classfile format to
> clearly and efficiently represent parametric polymorphism.  The early
> concepts of this approach were outlined in my talk at JVMLS last year; this
> represents a refinement of those ideas, and a reasonable "stake in the
> ground" description of what seems the most sensible way to balance
> preserving parametric information in the classfile without imposing
> excessive runtime costs for loading specializations.
>
> We're working on an updated compiler prototype which people will be able
> to play with soon (along with a formal model.)
>
> Please ask questions!
>
> Some things this document does not address yet:
>  - How we deal with types implicit in the bytecodes (aload vs iload) and
> how they get specialized;
>  - How we represent restricted methods in the classfile;
>  - How we represent the wildcard type Foo<any>
>
>
>


-- 

Best Regards,
Ali Ebrahimi

From ali.ebrahimi1781 at gmail.com  Sun Jan 24 08:32:17 2016
From: ali.ebrahimi1781 at gmail.com (Ali Ebrahimi)
Date: Sun, 24 Jan 2016 12:02:17 +0330
Subject: Model 3 classfile design document
In-Reply-To: <56A25E41.6040508@oracle.com>
References: <56A25E41.6040508@oracle.com>
Message-ID: <CAA0cW5BGLmToT+FeDCtgj7BOTGwLwEwi-2qu9e-A3UOAH8w+PQ@mail.gmail.com>

Hi,
Another two typo:

1-
"

To illustrate this, consider the class below:

class Bar<any V> { }
class Foo<any T extends Runnable, any U> extends Bar<T> {
    T aT;
    Foo<T,U> aFoo;
}

In the parameterized type Foo<Integer, int>, the first type parameter is
erased, and its erasure is Runnable."

Here is not better we use some subtype of Runnable such as Thread:

Foo<Thread, int>


2)

"
Member References

To illustrate further the complexity of erasure, and why we want to leave
it in the hands of the static compiler, suppose we have a method invocation:

class Foo<X> { void m(X x) { } }

class Bar<any Y extends Bound> {
    Foo<Y> foo = ...
    Y y = ...
    foo.m(y)

How do we describe the invocation foo.m(y)? The invokevirtual instruction
describes both the receiver type and the descriptor. If Y is erased, we
want the descriptor to be List::m(Object); if not, we want it to be
List<Y>::m(Y). "

If Y is erased, we want the descriptor to be Foo::m(Object); if not, we
want it to be Foo<Y>::m(Y).


But, one question:

Does your proposed translation scheme for generic methods only apply to
static ones or both (static & instance)?





-- 

Best Regards,
Ali Ebrahimi

From ali.ebrahimi1781 at gmail.com  Sun Jan 24 09:11:52 2016
From: ali.ebrahimi1781 at gmail.com (Ali Ebrahimi)
Date: Sun, 24 Jan 2016 12:41:52 +0330
Subject: Model 3 classfile design document
In-Reply-To: <CAA0cW5BGLmToT+FeDCtgj7BOTGwLwEwi-2qu9e-A3UOAH8w+PQ@mail.gmail.com>
References: <56A25E41.6040508@oracle.com>
	<CAA0cW5BGLmToT+FeDCtgj7BOTGwLwEwi-2qu9e-A3UOAH8w+PQ@mail.gmail.com>
Message-ID: <CAA0cW5CsQyUdLzEX+qfRDPvZr6eX-Z5NTPPUYHVqy24NapRqig@mail.gmail.com>

Hi,

On Sun, Jan 24, 2016 at 12:02 PM, Ali Ebrahimi <ali.ebrahimi1781 at gmail.com>
wrote:

> Hi,
> Another two typo:
>
> 1-
> "
>
> To illustrate this, consider the class below:
>
> class Bar<any V> { }
> class Foo<any T extends Runnable, any U> extends Bar<T> {
>     T aT;
>     Foo<T,U> aFoo;
> }
>
> In the parameterized type Foo<Integer, int>, the first type parameter is
> erased, and its erasure is Runnable."
>
> Here is not better we use some subtype of Runnable such as Thread:
>
> Foo<Thread, int>
>
>
> 2)
>
> "
> Member References
>
> To illustrate further the complexity of erasure, and why we want to leave
> it in the hands of the static compiler, suppose we have a method invocation:
>
> class Foo<X> { void m(X x) { } }
>
> class Bar<any Y extends Bound> {
>     Foo<Y> foo = ...
>     Y y = ...
>     foo.m(y)
>
> How do we describe the invocation foo.m(y)? The invokevirtual instruction
> describes both the receiver type and the descriptor. If Y is erased, we
> want the descriptor to be List::m(Object); if not, we want it to be
> List<Y>::m(Y). "
>
> If Y is erased, we want the descriptor to be Foo::m(Object); if not, we
> want it to be Foo<Y>::m(Y).
>
>
> But, one question:
>
> Does your proposed translation scheme for generic methods only apply to
> static ones or both (static & instance)?
>
Sorry, I will answer this one. Yes...

-- 

Best Regards,
Ali Ebrahimi

From forax at univ-mlv.fr  Sun Jan 24 09:52:33 2016
From: forax at univ-mlv.fr (=?ISO-8859-1?Q?R=E9mi_Forax?=)
Date: Sun, 24 Jan 2016 09:52:33 +0000
Subject: Model 3 classfile design document
In-Reply-To: <56A25E41.6040508@oracle.com>
References: <56A25E41.6040508@oracle.com>
Message-ID: <A69DB1EA-2E6F-4C30-A752-6899A6D9E2D6@univ-mlv.fr>

Nice write up !

In,
class Bar<any V> { } 
class Foo<any T extends Runnable, any U> extends Bar<T> { }

the bounds of T is not Runnable but any & Runnable, i.e. it's not an object so the specialization has to occur and if run() is called on a variable of type T, the compiler should first cast the value to Runnable and then do an invokeinterface.

This is very similar to the transformation actually done by the compiler when a type variable has several bounds.

R?mi


Le 22 janvier 2016 17:52:17 CET, Brian Goetz <brian.goetz at oracle.com> a ?crit :
>Please find a document here:
>
>http://cr.openjdk.java.net/~briangoetz/valhalla/eg-attachments/model3-01.html
>
>that describes our current thinking for evolving the classfile format
>to 
>clearly and efficiently represent parametric polymorphism.  The early 
>concepts of this approach were outlined in my talk at JVMLS last year; 
>this represents a refinement of those ideas, and a reasonable "stake in
>
>the ground" description of what seems the most sensible way to balance 
>preserving parametric information in the classfile without imposing 
>excessive runtime costs for loading specializations.
>
>We're working on an updated compiler prototype which people will be
>able 
>to play with soon (along with a formal model.)
>
>Please ask questions!
>
>Some things this document does not address yet:
>  - How we deal with types implicit in the bytecodes (aload vs iload) 
>and how they get specialized;
>  - How we represent restricted methods in the classfile;
>  - How we represent the wildcard type Foo<any>


From john.r.rose at oracle.com  Thu Jan 28 01:50:20 2016
From: john.r.rose at oracle.com (John Rose)
Date: Wed, 27 Jan 2016 17:50:20 -0800
Subject: Nestmates
In-Reply-To: <56A1F045.2010408@gmail.com>
References: <569FE666.9070103@oracle.com> <56A1F045.2010408@gmail.com>
Message-ID: <06DFC44C-2684-417E-8625-29D63CE1EAEC@oracle.com>

On Jan 22, 2016, at 1:03 AM, Peter Levart <peter.levart at gmail.com> wrote:
> 
> a symmetric configuration where each nest-mate lists all nest-mates in a single Nest attribute

That uses storage quadratic in the number of nestmates.  A big price for beauty!
Better to go with transitive closure on a graph (linear storage).
That takes us straight to a simple star graph, which is why we have a NestTop.

Fun fact:  If the nestmate relation is an equivalence relation, any class can
be the top.  This means that a compiler could put the bulky nestmate list
into a small synthetic class, at least in principle.  (Might look surprising
under reflection.)

? John

From john.r.rose at oracle.com  Thu Jan 28 01:54:59 2016
From: john.r.rose at oracle.com (John Rose)
Date: Wed, 27 Jan 2016 17:54:59 -0800
Subject: Nestmates
In-Reply-To: <CAKBs1_r1VUh-5-uLHcXQYntdY_6C-QbJW-oh21CSGh2ttnacdw@mail.gmail.com>
References: <569FE666.9070103@oracle.com>
	<CAKBs1_r1VUh-5-uLHcXQYntdY_6C-QbJW-oh21CSGh2ttnacdw@mail.gmail.com>
Message-ID: <6E558648-C0E9-44EC-8E32-6CFA8504CF15@oracle.com>

On Jan 21, 2016, at 3:10 PM, Vlad Ureche <vlad.ureche at gmail.com> wrote:
> 
> Thank you for sharing these insights Brian!
> 
> I think I understand the problem and the solution, but let me ask three questions to make sure I understood well:
> 
> 1) The NestTop attribute must contain the child classes (except specializations and lambdas, which are added dynamically), right? Is this for security, so another class could not pose as a NestChild to access private data? What about allowing the NestTop attribute to say "anyone who wants to nest here is welcome to do so"?

No, the security property comes from the bidirectional links.  The top class could be anywhere.

We are deliberately avoiding linking the semantics of NestTop to the InnerClasses attribute.

Java will of course use them in an aligned manner, but that's just a compiler choice.

The main security property that seems important is that nestmates must all be in the same package, so that package access is a strict subset of nestmate access.  (No new wormholes, please.)

> 2) Why did you choose to have symmetry and transitivity? I understand that having an equivalence relation allows partitioning, but it's not clear to me why partitioning is important in this case.

So the security model can be simple, based on containment:  global < module < package < nest.

> 3) Why is the NestChild limited to a single top class?

See previous message.  It could be a more complex graph, but why bother with it?  Any benefit from having a graph depth greater than one link?  (Not a rhetorical question!)

? John

> These questions stem from pondering whether we can use the nestmates mechanism to implement Scala's enclosing-entity-private access specifiers (e.g. a variable in class List can be private[scala.collection.List] or private[scala.collection] or private[scala])... Still, I don't think this can be done at the granularity required by Scala, so we'll continue to have name-mangled accessors where necessary :(
> 
> Thanks,
> Vlad


From peter.levart at gmail.com  Thu Jan 28 09:12:12 2016
From: peter.levart at gmail.com (Peter Levart)
Date: Thu, 28 Jan 2016 10:12:12 +0100
Subject: Nestmates
In-Reply-To: <06DFC44C-2684-417E-8625-29D63CE1EAEC@oracle.com>
References: <569FE666.9070103@oracle.com> <56A1F045.2010408@gmail.com>
	<06DFC44C-2684-417E-8625-29D63CE1EAEC@oracle.com>
Message-ID: <56A9DB6C.7060703@gmail.com>



On 01/28/2016 02:50 AM, John Rose wrote:
> On Jan 22, 2016, at 1:03 AM, Peter Levart <peter.levart at gmail.com 
> <mailto:peter.levart at gmail.com>> wrote:
>>
>> a symmetric configuration where each nest-mate lists all nest-mates 
>> in a single Nest attribute
>
> That uses storage quadratic in the number of nestmates.  A big price 
> for beauty!
> Better to go with transitive closure on a graph (linear storage).
> That takes us straight to a simple star graph, which is why we have a 
> NestTop.
>
> Fun fact:  If the nestmate relation is an equivalence relation, any 
> class can
> be the top.  This means that a compiler could put the bulky nestmate list
> into a small synthetic class, at least in principle.  (Might look 
> surprising
> under reflection.)
>
> ? John

I was just concerned that a decision to store the information centrally 
in one class would necessitate loading of the top class even when the 
access was checked between two non-top classes and that such loading 
could have an undesirable side-effect to the execution semantics. If the 
top-class attribute extraction can be performed without visible 
site-effects then keeping the information in one place is certainly 
preferable and simpler.

Regards, Peter


From ericlippert at gmail.com  Thu Jan 28 16:06:53 2016
From: ericlippert at gmail.com (Eric Lippert)
Date: Thu, 28 Jan 2016 08:06:53 -0800
Subject: Nestmates
In-Reply-To: <569FE666.9070103@oracle.com>
References: <569FE666.9070103@oracle.com>
Message-ID: <CAOasfz45t7t2MZn+gNS_PohySFwNJ0OMUtsch=2X0COPkKAoFg@mail.gmail.com>

Hello all,

First off, thanks to Brian for inviting me to join this list and for his
earlier introduction.  I am not an expert on Java by any means but I am an
expert on C#; Brian has asked me to comment occasionally on how the issues
you face were resolved (or not resolved!) in C#. I'll pop up now and then.
It's already an interesting discussion!

So, a few comments on how C# deals with these issues, in this email, and
then I'll post a second a little later.

>  In particular, private and protected access from and to inner classes is
stricter in the VM than in the language, meaning that in these cases, the
static compiler emits an access bridge

C# faces a similar, though subtly different problem: the rules in the CLR
for what makes *verifiable* code are in some places stricter than the C#
language would like them to be. The compiler strives to produce verifiable
code whenever possible. The C# compiler solves these problems in the same
way: emitting a helper method to bridge the gap.

C# does not suffer from the *particular* problem you mention; calling a
private method on Foo<int> from inside an activation of Foo<long> is no
problem for either the C# language or the CLR. But let me give you an
example of a mismatch that does occur:

public delegate void Delta();
public class Alpha
{
    public virtual void Echo() { }
}
public class Bravo : Alpha
{
    public override void Echo() { }
    public void Charlie()
    {
        int x = 123;
        Delta foxtrot = () =>
        {
            this.Echo();
            base.Echo();
            System.Console.WriteLine(x);
        };
        foxtrot();
    }
}

Note that the lambda captures both local x and "this". Now, what code
should the compiler emit?  It turns out that the naive way to emit the code
would involve using a feature that the C# language does not have:

    public void Charlie()
    {
        Closure closure = new Closure();
        closure. at this = this;
        closure.x = 123;
        Delta foxtrot = closure.M;
        foxtrot();
    }
    private class Closure
    {
        public Bravo @this;
        public int x;
        public void M()
        {
            @this.Echo(); // Perfectly legal
            // invoke Alpha.Echo() with receiver @this  -- but how?
            System.Console.WriteLine(x);
        }
    }

Now it turns out that the CLR does allow you to generate code that does
non-virtual dispatch to a particular override of a virtual method, but the
*verifier* requires that such code appear in *a class equal to or derived
from the class that declares the method being invoked*.

Our class Closure here is not considered by the verifier to have the right
to call Alpha.Echo non-virtually because Closure is not Bravo, it is merely
*nested* in Bravo. From the point of view of the compiler writer, this is a
crazy restriction; if Bravo has the right to do a call then why shouldn't a
class entirely owned by Bravo? But that's the rule in the verifier.

Early versions of C# simply generated non-verifiable code and produced a
warning that this pattern led to non-verifiable code. Later versions
generated a private bridge method in Bravo that called Alpha.Echo and then
had the generated closure invoke that helper method.

Cheers,
Eric







On Wed, Jan 20, 2016 at 11:56 AM, Brian Goetz <brian.goetz at oracle.com>
wrote:

> This topic is at the complete opposite end of the spectrum from topics
> we've been discussing so far.  It's mostly an implementation story, and of
> particular interest to the compiler and VM implementers here.
>
>
> Background
> ----------
>
> Since Java 1.1, the rules for accessibility when inner classes are
> involved at the language level are not fully aligned with those at the VM
> level.  In particular, private and protected access from and to inner
> classes is stricter in the VM than in the language, meaning that in these
> cases, the static compiler emits an access bridge (access$000) which
> effectively downgrades the accessed member's accessibility to package.
>
> Access bridges have some disadvantages.  They're ugly, but that's not a
> really big deal.  They're imprecise; they allow wider-than-necessary access
> to the member.  Again, this is not a huge deal on its own.  But the real
> problem is the complexity of the compiler implementation when we add
> generic specialization to the story.
>
> Specialization adds a new category of cross-class accesses that are
> allowed at the language level but not at the VM level, which would
> dramatically increase the need for, and complexity of, accessibility
> bridges.  For example:
>
> class Foo<any T> {
>     private T t;
>
>     void m(Foo<int> foo) {
>         int i = foo.t;
>     }
> }
>
> Now we execute:
>
>     Foo<long> fl = ...
>     Foo<int> fi = ...
>     fl.m(fi)
>
> The spirit of the language rules clearly allow the access from Foo<long>
> to Foo<int>.t -- they are in the "same class".  But at the VM level,
> Foo<int> and Foo<long> are different classes, so the access from Foo<long>
> to a private member of Foo<int> is disallowed.
>
> One reason that this increases the complexity, and not just the number, of
> accessibility bridges is that bridges are (currently) static methods; if
> they represent instance methods, we pass the receiver as the first
> argument.  For access between inner classes, this is fine, but when it
> comes to access between specializations, this breeds new complexity --
> because the method signature of the accessor needs to be specialized based
> on the type parameters of the receiver.  This interaction means the current
> static-accessor solution would need its own special, ad-hoc treatment in
> specialization, adding to the complexity of specialization.
>
> More generally, this situation arises in any case where a single logical
> unit of encapsulation at the source level is split into multiple runtime
> classes (inner classes, specialization classes, synthetic helper classes.)
> We propose to address this problem more generally, by providing a mechanism
> where language compilers can indicate that multiple runtime classes live in
> the same unit of encapsulation.  We do so by (a) adding metadata to classes
> to indicate which classes belong in the same encapsulation unit and (b)
> relaxing some VM accessibility rules to bring them more in alignment with
> the language level rules.
>
>
> Overview
> --------
>
> Our proposed strategy is to reify the relationship between classes that
> are members of the same _nest_.  Nestmate-ness can then be considered in
> access control decisions (JVMS 5.4.4).
>
> Classes that derive from a common source class form a _nest_, and two
> classes in the same nest are called _nestmates_.  Nestmate-ness is an
> equivalence relation (reflexive, symmetric, and transitive.)  Nestmates of
> a class C include C's inner classes, synthetic classes generated as part of
> translating C, and specializations thereof.
>
> Since nestmate-ness is an equivalence relation, it forms a partition over
> classes, and we can nominate a canonical member for each partition.  We
> nominate the "top" (outermost lexically enclosing) class in the nest as the
> canonical member; this is the top-level source class from which all other
> nestmates derive.
>
> This makes it easy to calculate nestmate-ness for two classes C and D; C
> and D are nestmates if their "top" class is the same.
>
> Example
> -------
>
> class Top<any T> {
>     class A<any U> { }
>         class B<V> { }
>     }
>
>     <any T> void genericMethod() { }
> }
>
> When we compile this, we get:
>    Top.class                   // Top
>    Top$A.class                 // Inner class Top.A
>    Top$A$B.class               // Inner class Top.A.B
>    Top$Any.class               // Wildcard interface for Top
>    Top$A$Any.class             // Wildcard interface for Top.A
>    Top$genericMethod.class     // Holder class for generic method
>
> The explicit classes Top, Top.A, and Top.A.B, the synthetic $Any classes,
> and the synthetic holder class for genericMethod, along with all of their
> specializations, form a nest.  The top member of this nest is Top.
>
> Since nestmates all derive from a common top-level class, they are by
> definition in the same package and module.  A class can be in only one nest
> at once.
>
>
> Runtime Representation
> ----------------------
>
> We represent nestmate-ness with two new attributes -- one in the top
> member, which describes all the members of the nest, and one in each
> member, which requests access to the nest.
>
>     NestTop {
>         u2 name_index;
>         u4 length;
>         u2 child_count;
>         u2 childClazz[child_count];
>     }
>
>     NestChild {
>         u2 name_index;
>         u4 length;
>         u2 topClazz;
>     }
>
> If a class has a NestTop attribute, its nest top is itself. If a class has
> a NestChild attribute, its nest top is the class named via topClazz. If a
> class is a specialization of another class, its nest top is the nest top of
> the class for which it is a specialization.
>
> When loading a class with a NestChild attribute, the VM can verify that
> the requested nest permits it as a member, and reject the class if the
> child and top do not agree.
>
> The NestTop attribute can enumerate all inner classes and synthetic
> classes, but cannot enumerate all specializations thereof. When creating a
> specialization of a class, the VM records the specialization as being a
> member of whatever nest the template class was a member of.
>
>
> Semantics
> ---------
>
> The accessibility rules here are strictly additions; nestmate-ness creates
> additional accessibility over and above the existing rules.
>
> Informally:
>   - A class can access the private members of its nestmates;
>   - A class can access protected members inherited by its nestmates.
>
> This is slightly broader than the language semantics (but still less broad
> than what we do today with access bridges.)  The static compiler can
> continue to enforce the same rules, and the VM will allow these accesses
> without bridges.  (We could make the proposal match the language semantics
> more closely at the cost of additional complexity, but its not clear this
> is worthwhile.)
>
> For private access, we can add the following to 5.4.4:
>   - A class C may access a private member D.R if C and D are nestmates.
>
> The rules for protected members are more complicated.  5.4.3.{2,3} first
> resolve the true owner of the member, and feed that to 5.4.4; this process
> throws away some needed information.  We would augment 5.4.3.{2,3} as
> follows:
>  - When performing member resolution from class C on member D.R, we
> remember both D (the target class) and E (the resolved class) and make them
> both available to 5.4.4.
>
> We then adjust 5.4.4 accordingly, by adding:
>  - If R is protected, and C and D are nestmates, and E is accessible to D,
> then access is allowed.
>
>
> Examples
> --------
>
> For private fields, we generate access bridges whenever an inner class
> accesses a private member (field or method) of the enclosing class, or of
> another inner class in the same nest.
>
> In the classes below, the accesses shown are all permitted by the language
> spec (child to parent, sibling to sibling, sibling to child of sibling,
> etc), and the ones requiring access bridges are noted.
>
>     class Foo {
>         public static Foo aFoo;
>         public static Inner1 aInner1;
>         public static Inner1.Inner2 aInner2;
>         public static Inner3 aInner3;
>
>         private int foo;
>
>         class Inner1 {
>             private int inner1;
>
>             class Inner2 {
>                 private int inner2;
>             }
>
>             void m() {
>                 int i = aFoo.foo           // bridge
>                       + aInner1.inner1
>                       + aInner2.inner2     // bridge
>                       + aInner3.inner3;    // bridge
>             }
>         }
>
>         class Inner3 {
>             private int inner3;
>
>             void m() {
>                 int i = aFoo.foo           // bridge
>                       + aInner1.inner1     // bridge
>                       + aInner2.inner2     // bridge
>                       + aInner3.inner3;
>             }
>         }
>     }
>
> For protected members, the situation is more subtle.
>
>     /* package p1 */
>     public class Sup {
>         protected int pro;
>     }
>
>     /* package p2 */
>     public class Sub extends p1.Sup {
>         void test() {
>             ... pro ... //no bridge (invokespecial)
>         }
>
>         class Inner {
>             void test() {
>                 ... sub.pro ... // bridge generated in Sub
>             }
>         }
>     }
>
> Here, the VM rules allow Sub to access protected members of Sup, but for
> accesses from Sub.Inner or Sibling to Sub.pro to succeed, Sub provides an
> access bridge (which effectively makes Sub.pro package-visible throughout
> package p2.)
>
> The rules outlined eliminate access bridges in all of these cases.
>
>
> Interaction with defineAnonymousClass
> -------------------------------------
>
> Nestmate-ness also potentially connects nicely with
> Unsafe.defineAnonymousClass.  The intuitive notion of dAC is, when you load
> anonymous class C with a host class of H, that C is being "injected into" H
> -- access control decisions for C are made using H's credentials.  With a
> formal notion of nestmateness, we can bring additional predictability to
> dAC by saying that C is injected into H's nest.
>
>
>

From ericlippert at gmail.com  Thu Jan 28 16:25:46 2016
From: ericlippert at gmail.com (Eric Lippert)
Date: Thu, 28 Jan 2016 08:25:46 -0800
Subject: Nestmates
In-Reply-To: <56A18518.1050204@oracle.com>
References: <569FE666.9070103@oracle.com>
	<CAKBs1_r1VUh-5-uLHcXQYntdY_6C-QbJW-oh21CSGh2ttnacdw@mail.gmail.com>
	<56A18518.1050204@oracle.com>
Message-ID: <CAOasfz7MDhxwM9ZWjZHoehJWQv7svoyaq3SJobe7b_w+w03q=Q@mail.gmail.com>

Good day all,

>  It seems reasonable that other languages would similarly want to define
an encapsulation domain that spans multiple VM classes (like, perhaps, C#'s
"partial class" construct).

I just want to clarify what partial classes do in C#. All portions of a
partial class in C# combine to form a single CLR class *at compile time*.

Partial classes are entirely a syntactic sugar for "I want to express the
logic of this class in two or more physically distinct files". The compiler
simply gathers them up, verifies that they do not grossly contradict each
other (by, say, inheriting from two different base classes), pastes them
together, and compiles them as though they were lexically a single class.

The motivation for partial classes was automatic code generation scenarios
where the IDE generates boilerplate code based on the state of the forms
designer; we do not wish the user to edit this code, but we also want them
to be able to add their own methods to the class without having to create a
derived class.  So the machine-generated code goes in one portion -- with a
big comment saying you can read this code but please don't edit it -- and
the user-generated code goes in another, and the compiler takes care of
putting them together later.

So these are entirely a compile-time construct. There's no way to "add on"
to a partial class after compilation. The accessibility rules in partial
classes are exactly the same as though the class text was lexically all
together, as you'd expect.

There is a feature of C# that does allow extension of an accessibility
domain across compilations in an unusual way though.

In C# the accessibility modifiers are private, public, protected and
internal; internal means accessibility is granted within the same
*assembly*. C# has an "internals become visible" feature that allows you to
mark assembly Alpha as "my internals should be treated as public when the
assembly doing the asking is assembly Bravo".  This is typically used for
testing scenarios where Bravo contains the unit tests for Alpha's internal
implementation details.  It is a feature which gives rise to odd corner
cases, and some members of the C# design team are *not big fans*, to say
the least.

Cheers,
Eric












On Thu, Jan 21, 2016 at 5:25 PM, Brian Goetz <brian.goetz at oracle.com> wrote:

> Certainly we would like it to be useful beyond Java. Extrapolating from
> Java's situation, we've told the users that a "class" is everything between
> the outermost { ... }, and defined encapsulation rules based on this notion
> of class boundary.  For various expedient reasons, at various points in
> Java's evolution (inner classes, lambda proxies, specialization) we've
> chosen to implement a single "user class" as multiple "VM classes",
> necessitating hacks and backdoors to preserve the user's expectations (and
> sometimes failing.)  It seems reasonable that other languages would
> similarly want to define an encapsulation domain that spans multiple VM
> classes (like, perhaps, C#'s "partial class" construct).  The nestmate
> concept is motivated by the problem we have *now*, but we'd like it to be
> useful beyond the immediate situation.  On the other hand, we don't want to
> extrapolate wildly to what some language might want, or define something
> that is so complicated that we can't predict the security consequences.
>
> More inline.
>
> On 1/21/2016 6:10 PM, Vlad Ureche wrote:
>
>> Thank you for sharing these insights Brian!
>>
>> I think I understand the problem and the solution, but let me ask three
>> questions to make sure I understood well:
>>
>> 1) The NestTop attribute must contain the child classes (except
>> specializations and lambdas, which are added dynamically), right? Is this
>> for security, so another class could not pose as a NestChild to access
>> private data? What about allowing the NestTop attribute to say "anyone who
>> wants to nest here is welcome to do so"?
>>
>
> Security is indeed the motivation.  It's the VM's job to enforce what
> private means; while there are plenty of ways to dodge this (and we'd like
> to close some of them down), accessibility control is part of the integrity
> model of the JVM. In order to inject yourself into another classes' nest,
> you'd need to modify that classes classfile -- in which case you can do
> anything you want with their data anyway (just make all the fields
> public.)  The two attributes form a handshake.
>
> An easy way to say "anyone who wants to nest here may do so" is to make
> your state public, and possibly enforce language-level access control to
> make sure that they've actually requested such nesting.
>
> 2) Why did you choose to have symmetry and transitivity? I understand that
>> having an equivalence relation allows partitioning, but it's not clear to
>> me why partitioning is important in this case.
>>
>
> Simplicity.  By having a partitioning, and a distinguished canonical
> member, it is very easy to determine what access control credentials are in
> play.  It also goes back to the user model -- there is some unit of
> encapsulation that the language defines (class) that determines what
> "private" means.
>
> We could do something fancy with friends, and transitive friends, and
> marriage brokers, and itinerant priests selling indulgences, but it doesn't
> seem that the return on complexity there is worth it.
>
> 3) Why is the NestChild limited to a single top class?
>>
>
> Again, because the encapsulation model is supposed to be simple.  Just as
> each line of code belongs to only one source file, each bytecode belongs to
> one nest.  \\
>
> Perhaps another way to think of it is: its the simplest thing we could
> think of that solves our current problem and yet seems not completely
> ad-hoc :)
>
> These questions stem from pondering whether we can use the nestmates
>> mechanism to implement Scala's enclosing-entity-private access specifiers
>> (e.g. a variable in class List can be private[scala.collection.List] or
>> private[scala.collection] or private[scala])... Still, I don't think this
>> can be done at the granularity required by Scala, so we'll continue to have
>> name-mangled accessors where necessary :(
>>
>
> Right.  These are overlapping scopes of accessibility, more like "virtual
> packages" than "nestmates".  This seems like a slightly different problem.
>
>
>

From john.r.rose at oracle.com  Thu Jan 28 18:48:21 2016
From: john.r.rose at oracle.com (John Rose)
Date: Thu, 28 Jan 2016 10:48:21 -0800
Subject: Nestmates
In-Reply-To: <CAOasfz45t7t2MZn+gNS_PohySFwNJ0OMUtsch=2X0COPkKAoFg@mail.gmail.com>
References: <569FE666.9070103@oracle.com>
	<CAOasfz45t7t2MZn+gNS_PohySFwNJ0OMUtsch=2X0COPkKAoFg@mail.gmail.com>
Message-ID: <8842C27F-33C2-4146-ACB7-6E17D2995F4F@oracle.com>

On Jan 28, 2016, at 8:06 AM, Eric Lippert <ericlippert at gmail.com> wrote:
> 
> Later versions generated a private bridge method in Bravo that called Alpha.Echo and then had the generated closure invoke that helper method.

FTR here's the a version of Eric's example, coded with lambdas and then with inner classes.
Some disassembly is included.

In both cases javac emits a non-public helper method in Bravo that is shared with the Delta object.
In the case of lambdas, it fits together securely with a method handle.
In the case of inner classes, there is a package-scope "hole" in the access control.

The IC case would be fixed by nestmate access, if we figure out how to expand "protected"
access across a nest.  Which is tricky.  It might be that protected access must always go
through delegation via private methods with "proprietary" rights to call protected.

? John

http://cr.openjdk.java.net/~jrose/draft/ClosureCallSuper.java <http://cr.openjdk.java.net/~jrose/draft/ClosureCallSuper.java>

class ClosureCallSuper {
    public static void main(String... av) {
        new Bravo().charlie();
        new Bravo_with_IC().charlie();
    }
}

interface Delta { void invoke(); }
class Alpha {
    public void echo() { System.out.println("Alpha.echo"); }
}
class Bravo extends Alpha {
    public @Override void echo() { System.out.println("Bravo.echo"); }
    public void charlie() {
        int x = 123;
        Delta foxtrot = () ->
        {
            this.echo(); //Bravo.echo
            super.echo(); //Alpha.echo
            System.out.println(x); //123
        };
        foxtrot.invoke();
    }
}
class Bravo_with_IC extends Alpha {
    public @Override void echo() { System.out.println("Bravo_with_IC.echo"); }
    public void charlie() {
        int x = 123;
        Delta foxtrot = new Delta() {
        public void invoke() {
            Bravo_with_IC.this.echo(); //Bravo_with_IC.echo
            Bravo_with_IC.super.echo(); //Alpha.echo
            System.out.println(x); //123
        } };
        foxtrot.invoke();
    }
}

/*
$ javac ClosureCallSuper.java && java ClosureCallSuper
Bravo.echo
Alpha.echo
123
Bravo_with_IC.echo
Alpha.echo
123

$ javap -p -c Bravo.class
Compiled from "ClosureCallSuper.java"
class Bravo extends Alpha {
  Bravo();
    Code:
       0: aload_0
       1: invokespecial #1                  // Method Alpha."<init>":()V
       4: return

  public void echo();
    Code:
       0: getstatic     #2                  // Field java/lang/System.out:Ljava/io/PrintStream;
       3: ldc           #3                  // String Bravo.echo
       5: invokevirtual #4                  // Method java/io/PrintStream.println:(Ljava/lang/String;)V
       8: return

  public void charlie();
    Code:
       0: bipush        123
       2: istore_1
       3: aload_0
       4: iload_1
       5: invokedynamic #5,  0              // InvokeDynamic #0:invoke:(LBravo;I)LDelta;
[[ (javap includes this info in -v mode)
BootstrapMethods:
  0: #29 invokestatic java/lang/invoke/LambdaMetafactory.metafactory:(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodHandle;Ljava/lang/invoke/MethodType;)Ljava/lang/invoke/CallSite;
    Method arguments:
      #30 ()V
      #31 invokespecial Bravo.lambda$charlie$0:(I)V
      #30 ()V
]]
      10: astore_2
      11: aload_2
      12: invokeinterface #6,  1            // InterfaceMethod Delta.invoke:()V
      17: return

  private void lambda$charlie$0(int);
    Code:
       0: aload_0
       1: invokevirtual #7                  // Method echo:()V
       4: aload_0
       5: invokespecial #8                  // Method Alpha.echo:()V
       8: getstatic     #2                  // Field java/lang/System.out:Ljava/io/PrintStream;
      11: iload_1
      12: invokevirtual #9                  // Method java/io/PrintStream.println:(I)V
      15: return
}

$ javap -p -c Bravo_with_IC.class
Compiled from "ClosureCallSuper.java"
class Bravo_with_IC extends Alpha {
  Bravo_with_IC();
    Code:
       0: aload_0
       1: invokespecial #2                  // Method Alpha."<init>":()V
       4: return

  public void echo();
    Code:
       0: getstatic     #3                  // Field java/lang/System.out:Ljava/io/PrintStream;
       3: ldc           #4                  // String Bravo_with_IC.echo
       5: invokevirtual #5                  // Method java/io/PrintStream.println:(Ljava/lang/String;)V
       8: return

  public void charlie();
    Code:
       0: bipush        123
       2: istore_1
       3: new           #6                  // class Bravo_with_IC$1
       6: dup
       7: aload_0
       8: iload_1
       9: invokespecial #7                  // Method Bravo_with_IC$1."<init>":(LBravo_with_IC;I)V
      12: astore_2
      13: aload_2
      14: invokeinterface #8,  1            // InterfaceMethod Delta.invoke:()V
      19: return

  static void access$001(Bravo_with_IC);  <<<< THIS SHOULD BE PRIVATE BUT IS PACKAGE
    Code:
       0: aload_0
       1: invokespecial #1                  // Method Alpha.echo:()V
       4: return
}

*/