Nest host validation vs NestHost attribute performed by Lookup::defineHiddenClass

[John Rose]

A few top-level points about the intersection between nests and hidden classes:

1. Nests are *statically* defined in terms of *validated* names of the nestmates (on NestHost and NestMembers attrs).
(Validation is done by means of name resolution.)

2a. Hidden classes do not have names that can be validated, and so cannot participate in NestHost/NestMembers.

2b. Thus, hidden classes, in order to participate in nests, must have a non-symbolic (therefore dynamic) association with nests.

2c. (Therefore, it’s probably a good idea to *forbid* or at least *ignore* NestHost and NestMembers on hidden classes.)

3a. After validation of the asymmetric NH/NM attributes (or after a “bad” NH is thrown away), access control effects of
nesting structure are applied uniformly across the nest.

3b. The NestHost has no dynamic distinction from the other nest members, other than a “ceremonial” role in reflection.

In the MHs.Lookup API, the LC with a full-power access is a proxy for access to the nest, which means it
should (barring good reason to the contrary) should have the *same* capabilities as *any* of its nest mates.
I.e., the “ceremonial” role of NH should not confer special properties in the Lookup API.

I think this simplifies some of the questions below:  If the NH/NMs attrs are “cooked away” at validation time,
all you have left is a dynamic pointer (in the VM) which relates all the NMs together (via some host klass,
which can be chosen independently from the NestHost attribute, as is the case with a “bad” NestHost).

So I’d recommend to ignore the attributes when working with hidden classes, and go straight to the
simpler and clearer host_klass relation, which is a dynamic result from the static precursors.

On Sep 24, 2019, at 5:27 PM, Mandy Chung <mandy.chung at oracle.com> wrote:
> 
> This is regarding an incompatibility concern when LambdaMetaFactory and any other framework library converts the use of `Unsafe::defineAnonymousClass` to `Lookup::defineHiddenClass` when a caller caller `C` whose nest host `H` indicated by `NestHost` attribute is "bad".
> 
> `NestHost` and `NestMembers` attributes are consulted by JVM access control (JVMS 5.4.4).  If `H` cannot be loaded, or is not in the same run-time package as `C`, or does not authorize nest membership (`H` has a `NestMembers attribute but there is no entry named `C`), then a linkage error occurs.
> 
> OTOH, `Class::getNestHost` may return a Class that is not the named class indicated by `NestHost` attribute.  Specifically:
> 
>    If there is a linkage error accessing the nest host, or if this class
>    or interface is not enumerated as a member of the nest by the nest host,
>    then it is considered to belong to its own nest and this is returned
>    as the host.
> 
> If `C` has a bad nest host indicated by `NestHost` attribute, then effectively `NestHost` attribute is ignored and `C` is the host of its own nest as if `C` lacked of `NestHost` attribute.   No error occurs until `C` attempts to access a private member of `H` - consistency for VM and core reflection.
> 
> Now `Lookup::defineHiddenClass` can be used to add a hidden class to a nest of the lookup class.
> 
> If a framework uses C's full-power lookup object to call `Lookup::defineHiddenClass` to add a hidden class `X` into C's nest.  What is the nest host of `X`?
> 
> If C has a good nest host, of course X will have the same nest host as C.  If C has a bad nest host `H`, the lookup object on C is used to call `Lookup::defineHiddenClass` - should it fail with illegal nest host?  Should it succeed to be consistent with core reflection (asC.class::getNestHost returns C - see more)?
> 
> That is the current proposed behavior of `Lookup::defineHiddenClass` to validate that the lookup class must be the host of a nest.  If the lookup class has `NestHost` attribute, it is a member of some other class's nest and so fails.  The design principle is that the lookup class and the proper permission together authorizes the lookup object to add a new member into a nest.  In addition, C's nest host is H and adding a class with C as the nest host should be disallowed as this conceptually creates a "nested nest" (H is C's nest host, C is X's nest host).
> 
> A framework library can't tell whether C is the class indicated by C's NestHost attribute. Take an example in practice.  C uses lambda expression or method reference expression but H cannot be found while C never accesses H's private member (If C accesses H's private member, then IAE must occur since it fails the access check). The first invocation of indy LMF for C class unfortunately throws BootstrapMethodError because LambdaMetaFactory calls`Lookup::defineHiddenClass` to define a lambda proxy class for C and IAE occurs as C is a member of `H` statically.
> 
> It is expected that this error case should rarely occur but it does have an incompatibility risk that impacts existing applications.
> 
> As Class::getNestHost returns C as the nest host, a framework library calls `defineHiddenClass` with the returned nest host but surprisely it throws an exception that nothing it can do.
> 
> If C has a bad nest host `H`, it will not get an error until C attempts to access a private member of H.  I think we should revisit this validation and relax `defineHiddenClass` validation to check that the lookup class is the same class as Class::getNestHost returns.  The hidden class X has C as the nest host (but NOT H).  The VM access check will ensure that X will fail when accessing H's private members.  No nested nest is created since at runtime C is the host of its own nest and X is added to C's nest (where the static nestmate relationship is not in effect).
> 
> Summary:
> 
> `Lookup::defineHiddenClass` performs the nest host validation but won't consultthe `NestHost` attribute of the lookup class.
> 
> Feedback?
> 
> thanks
> Mandy