Draft JVMS changes for Nestmates

[Dan Smith]

Hi, all.

I've uploaded a draft of JVMS changes for JEP 181 "Align JVM Checks with Java Language Rules for Nested Classes" to:
http://cr.openjdk.java.net/~dlsmith/private-access.html

Some comments below on my thinking in drafting the spec text, and on the JEP generally.

JEP name

I don't know how permanent JEP names are supposed to be, but I'd prefer a different name at this point. Something like: "Expanded JVM Access to Private Members"—shorter, focused on the feature itself rather than its relationship to the Java language. Or maybe "Class Nests for Access to Private Members".

Terminology

The term "nest" is nice because it's short and mostly unspoiled by overloading in this context (I think?); it's not great because it's informal and doesn't mean anything the first time you hear it. I thought about something more clinical like "access control context", but I'm not convinced that's an improvement. How do others feel?

The JEP uses "nest top" to describe the class that nest members reference; I prefer "host class", which better describes the class's role and isn't tied to the Java "top level class" concept. I know we use "host class" internally in Hotspot, perhaps when working with anonymous classes (of the JVM flavor), but I think in that context it will ultimately mean the same thing? Are we comfortable repurposing the term in this way?

I follow Brian's model when it comes to nest membership (5.4.4): every class belongs to a nest, possibly (in the absence of MemberOfNest) the nest hosted by itself. Many nests are singletons without any associated explicit attributes.

Verification of MemberOfNest

I include a discussion block about different options of validating MemberOfNest. I think the consensus, and my preference, is to do it during verification of the member class. (NestMembers, on the other hand, is never validated, except as a side-effect of checking MemberOfNest.)

Verification of invokespecial

Allowing invokespecial to refer to classes other than the current class and its supers is a significant change. I noticed and relied heavily on the parallel with invokevirtual making protected method calls. So I tried to make the two as similar as possible. In a few places, the treatment of protected methods doesn't seem ideal, and rather than trying to mirror that with non-private invokespecial, I modified the protected method treatment.

The "protected check" of verification, in particular, was a mess before, and I think I've made it a lot more manageable, and compatible with a parallel rule for invokespecial. I could use some feedback on exactly what Hotspot's verifier has been doing here, though, since I'm pretty sure it didn't match the old specified rules.

MethodHandle resolution

The spec (5.4.3.5) is vague about what errors can occur during MethodHandle resolution. I assume any linkage error specified for the instruction can also occur via MethodHandle resolution, and that will include failures due to invokespecial improperly referencing a class.

Dynamic checking of invokespecial receivers

When invokespecial involves interface types, the verifier can't guarantee that receiver objects actually implement that interface (JDK-8134358). It's an open question whether this is really a problem, but I included a tentative fix in the invokespecial runtime rules.

Compiler changes

The JEP text can't seem to decide if compiler changes are part of it, or a follow-up exercise. I think we're best off explicitly including the compiler changes, which will provide opportunities for design validation and testing.

API changes

I haven't tried to address changes that need to be made to APIs. Somebody will need to. For example, Lookup.findSpecial probably needs to make adjustments to account for private members in nestmates, and to parallel the new verification/linkage rules (e.g., follow Lookup.findVirtual in restricting the receiver type).

Security risk

The JEP text should acknowledge that, while this does allow compilers to grant finer-grained access to members shared by nestmates, it also pushes compilers to grant broader access to members that were previously kept private. It's a trade-off, and presumably a good one because nestmates are completely trusted, while package-mates might sometimes be suspect.

(I guess this argument really ought to be made from the top: declaring things with package access is worse than inventing a new level of access because ________.)

—Dan