Nest host validation vs NestHost attribute performed by Lookup::defineHiddenClass

Mandy Chung mandy.chung at
Wed Sep 25 00:27:21 UTC 2019

This is regarding an incompatibility concern when LambdaMetaFactory and 
any other framework library converts the use of 
`Unsafe::defineAnonymousClass` to `Lookup::defineHiddenClass` when a 
caller caller `C` whose nest host `H` indicated by `NestHost` attribute 
is "bad".

`NestHost` and `NestMembers` attributes are consulted by JVM access 
control (JVMS 5.4.4).  If `H` cannot be loaded, or is not in the same 
run-time package as `C`, or does not authorize nest membership (`H` has 
a `NestMembers attribute but there is no entry named `C`), then a 
linkage error occurs.

OTOH, `Class::getNestHost` may return a Class that is not the named 
class indicated by `NestHost` attribute.  Specifically:

    If there is a linkage error accessing the nest host, or if this class
    or interface is not enumerated as a member of the nest by the nest host,
    then it is considered to belong to its own nest and this is returned
    as the host.

If `C` has a bad nest host indicated by `NestHost` attribute, then 
effectively `NestHost` attribute is ignored and `C` is the host of its 
own nest as if `C` lacked of `NestHost` attribute.   No error occurs 
until `C` attempts to access a private member of `H` - consistency for 
VM and core reflection.

Now `Lookup::defineHiddenClass` can be used to add a hidden class to a 
nest of the lookup class.

If a framework uses C's full-power lookup object to call 
`Lookup::defineHiddenClass` to add a hidden class `X` into C's nest.  
What is the nest host of `X`?

If C has a good nest host, of course X will have the same nest host as 
C.  If C has a bad nest host `H`, the lookup object on C is used to call 
`Lookup::defineHiddenClass` - should it fail with illegal nest host?  
Should it succeed to be consistent with core reflection 
(asC.class::getNestHost returns C - see more)?

That is the current proposed behavior of `Lookup::defineHiddenClass` to 
validate that the lookup class must be the host of a nest.  If the 
lookup class has `NestHost` attribute, it is a member of some other 
class's nest and so fails.  The design principle is that the lookup 
class and the proper permission together authorizes the lookup object to 
add a new member into a nest.  In addition, C's nest host is H and 
adding a class with C as the nest host should be disallowed as this 
conceptually creates a "nested nest" (H is C's nest host, C is X's nest 

A framework library can't tell whether C is the class indicated by C's 
NestHost attribute. Take an example in practice.  C uses lambda 
expression or method reference expression but H cannot be found while C 
never accesses H's private member (If C accesses H's private member, 
then IAE must occur since it fails the access check). The first 
invocation of indy LMF for C class unfortunately throws 
BootstrapMethodError because LambdaMetaFactory 
calls`Lookup::defineHiddenClass` to define a lambda proxy class for C 
and IAE occurs as C is a member of `H` statically.

It is expected that this error case should rarely occur but it does have 
an incompatibility risk that impacts existing applications.

As Class::getNestHost returns C as the nest host, a framework library 
calls `defineHiddenClass` with the returned nest host but surprisely it 
throws an exception that nothing it can do.

If C has a bad nest host `H`, it will not get an error until C attempts 
to access a private member of H.  I think we should revisit this 
validation and relax `defineHiddenClass` validation to check that the 
lookup class is the same class as Class::getNestHost returns.  The 
hidden class X has C as the nest host (but NOT H).  The VM access check 
will ensure that X will fail when accessing H's private members.  No 
nested nest is created since at runtime C is the host of its own nest 
and X is added to C's nest (where the static nestmate relationship is 
not in effect).


`Lookup::defineHiddenClass` performs the nest host validation but won't 
consultthe `NestHost` attribute of the lookup class.



More information about the valhalla-dev mailing list