Nest host validation vs NestHost attribute performed by Lookup::defineHiddenClass

[Mandy Chung]

On 9/24/19 5:50 PM, John Rose wrote:
> A few top-level points about the intersection between nests and hidden classes:
>
> 1. Nests are *statically* defined in terms of *validated* names of the nestmates (on NestHost and NestMembers attrs).
> (Validation is done by means of name resolution.)
>
> 2a. Hidden classes do not have names that can be validated, and so cannot participate in NestHost/NestMembers.
>
> 2b. Thus, hidden classes, in order to participate in nests, must have a non-symbolic (therefore dynamic) association with nests.
>
> 2c. (Therefore, it’s probably a good idea to *forbid* or at least *ignore* NestHost and NestMembers on hidden classes.)
>
> 3a. After validation of the asymmetric NH/NM attributes (or after a “bad” NH is thrown away), access control effects of
> nesting structure are applied uniformly across the nest.
>
> 3b. The NestHost has no dynamic distinction from the other nest members, other than a “ceremonial” role in reflection.
>
> In the MHs.Lookup API, the LC with a full-power access is a proxy for access to the nest, which means it
> should (barring good reason to the contrary) should have the *same* capabilities as *any* of its nest mates.
> I.e., the “ceremonial” role of NH should not confer special properties in the Lookup API.
>
> I think this simplifies some of the questions below:  If the NH/NMs attrs are “cooked away” at validation time,
> all you have left is a dynamic pointer (in the VM) which relates all the NMs together (via some host klass,
> which can be chosen independently from the NestHost attribute, as is the case with a “bad” NestHost).
>
> So I’d recommend to ignore the attributes when working with hidden classes, and go straight to the
> simpler and clearer host_klass relation, which is a dynamic result from the static precursors.

What about one refinement:  since hidden classes cannot participate in 
NestHost/NestMembers, I suggest to a stricter rule to disallow a hidden 
class containing NH/NM attribute (rather than ignoring it) which also 
forbids "nested nest".

Mandy

> On Sep 24, 2019, at 5:27 PM, Mandy Chung <mandy.chung at oracle.com> wrote:
>> This is regarding an incompatibility concern when LambdaMetaFactory and any other framework library converts the use of `Unsafe::defineAnonymousClass` to `Lookup::defineHiddenClass` when a caller caller `C` whose nest host `H` indicated by `NestHost` attribute is "bad".
>>
>> `NestHost` and `NestMembers` attributes are consulted by JVM access control (JVMS 5.4.4).  If `H` cannot be loaded, or is not in the same run-time package as `C`, or does not authorize nest membership (`H` has a `NestMembers attribute but there is no entry named `C`), then a linkage error occurs.
>>
>> OTOH, `Class::getNestHost` may return a Class that is not the named class indicated by `NestHost` attribute.  Specifically:
>>
>>     If there is a linkage error accessing the nest host, or if this class
>>     or interface is not enumerated as a member of the nest by the nest host,
>>     then it is considered to belong to its own nest and this is returned
>>     as the host.
>>
>> If `C` has a bad nest host indicated by `NestHost` attribute, then effectively `NestHost` attribute is ignored and `C` is the host of its own nest as if `C` lacked of `NestHost` attribute.   No error occurs until `C` attempts to access a private member of `H` - consistency for VM and core reflection.
>>
>> Now `Lookup::defineHiddenClass` can be used to add a hidden class to a nest of the lookup class.
>>
>> If a framework uses C's full-power lookup object to call `Lookup::defineHiddenClass` to add a hidden class `X` into C's nest.  What is the nest host of `X`?
>>
>> If C has a good nest host, of course X will have the same nest host as C.  If C has a bad nest host `H`, the lookup object on C is used to call `Lookup::defineHiddenClass` - should it fail with illegal nest host?  Should it succeed to be consistent with core reflection (asC.class::getNestHost returns C - see more)?
>>
>> That is the current proposed behavior of `Lookup::defineHiddenClass` to validate that the lookup class must be the host of a nest.  If the lookup class has `NestHost` attribute, it is a member of some other class's nest and so fails.  The design principle is that the lookup class and the proper permission together authorizes the lookup object to add a new member into a nest.  In addition, C's nest host is H and adding a class with C as the nest host should be disallowed as this conceptually creates a "nested nest" (H is C's nest host, C is X's nest host).
>>
>> A framework library can't tell whether C is the class indicated by C's NestHost attribute. Take an example in practice.  C uses lambda expression or method reference expression but H cannot be found while C never accesses H's private member (If C accesses H's private member, then IAE must occur since it fails the access check). The first invocation of indy LMF for C class unfortunately throws BootstrapMethodError because LambdaMetaFactory calls`Lookup::defineHiddenClass` to define a lambda proxy class for C and IAE occurs as C is a member of `H` statically.
>>
>> It is expected that this error case should rarely occur but it does have an incompatibility risk that impacts existing applications.
>>
>> As Class::getNestHost returns C as the nest host, a framework library calls `defineHiddenClass` with the returned nest host but surprisely it throws an exception that nothing it can do.
>>
>> If C has a bad nest host `H`, it will not get an error until C attempts to access a private member of H.  I think we should revisit this validation and relax `defineHiddenClass` validation to check that the lookup class is the same class as Class::getNestHost returns.  The hidden class X has C as the nest host (but NOT H).  The VM access check will ensure that X will fail when accessing H's private members.  No nested nest is created since at runtime C is the host of its own nest and X is added to C's nest (where the static nestmate relationship is not in effect).
>>
>> Summary:
>>
>> `Lookup::defineHiddenClass` performs the nest host validation but won't consultthe `NestHost` attribute of the lookup class.
>>
>> Feedback?
>>
>> thanks
>> Mandy