From john.r.rose at oracle.com Sat Apr 1 00:00:19 2017 From: john.r.rose at oracle.com (John Rose) Date: Fri, 31 Mar 2017 17:00:19 -0700 Subject: Value Type on stack in the interpreter In-Reply-To: <1248015349.2427147.1491003044116.JavaMail.zimbra@u-pem.fr> References: <715214554.2375004.1490975856821.JavaMail.zimbra@u-pem.fr> <1248015349.2427147.1491003044116.JavaMail.zimbra@u-pem.fr> Message-ID: <983E981F-0790-4894-AA9B-653ECC3E4B30@oracle.com> On Mar 31, 2017, at 4:30 PM, forax at univ-mlv.fr wrote: > > ----- Mail original ----- >> De: "John Rose" >> ?: "R?mi Forax" >> Cc: valhalla-spec-experts at openjdk.java.net >> Envoy?: Samedi 1 Avril 2017 01:06:07 >> Objet: Re: Value Type on stack in the interpreter > >> On Mar 31, 2017, at 8:57 AM, Remi Forax wrote: >>> >>> I've some trouble to see how value types are laid on stack (conceptually, not >>> necessary the real implementation). >>> >>> First, i may be wrong but it seems that vload do not reference the corresponding >>> Q-type, so i wonder how the verifier knows that it can then access to component >>> of the value type. >>> Then, it seems that value type use one slot on stack, so the components (and the >>> boxed reference if it exists) has to be stored somewhere but there is no >>> corresponding max value type buffer size. >> >> In the interpreter value types all share a common _carrier type_ which includes >> metadata about the size and type of the value being stored, plus a pointer to >> the actual storage. The actual storage may be on the heap, or maybe not. >> We are prototyping with thread-local arenas for the non-heap case. >> There are also ways to store value payloads in the stack frame, but >> in that case there is a special handshake when you want to return a >> value from a method call. > > Ok, > why the type of the value is stored in the _carrier_type_ and not in the vload/vstore instruction. There are two choices: 1. store the type metadata in the carrier (on stack and in local), or 2. reaffirm the type metadata at every use site (not just vload/vstore). #1 allows simpler bytecodes which put less load on the verifier. A significant subset of #2 is needed, anyway, for telling the GC where to find object references. So #2 wins. > >> >> From the verifier's point of view, there isn't much difference between >> the various Q-types. We will probably track classes of Q-types, mainly >> to simplify vgetfield operations. > > pre-compute the offset of a component ? Yes, as with objects. The techniques are basically the same. Q-types behave like on-stack objects. (They are also copy-able hence identity-free, and they are also non-nullable.) ? John From john.r.rose at oracle.com Sat Apr 1 00:03:39 2017 From: john.r.rose at oracle.com (John Rose) Date: Fri, 31 Mar 2017 17:03:39 -0700 Subject: Value Type on stack in the interpreter In-Reply-To: <983E981F-0790-4894-AA9B-653ECC3E4B30@oracle.com> References: <715214554.2375004.1490975856821.JavaMail.zimbra@u-pem.fr> <1248015349.2427147.1491003044116.JavaMail.zimbra@u-pem.fr> <983E981F-0790-4894-AA9B-653ECC3E4B30@oracle.com> Message-ID: On Mar 31, 2017, at 5:00 PM, John Rose wrote: > > There are two choices: 1. store the type metadata in the carrier (on stack and in local), > or 2. reaffirm the type metadata at every use site (not just vload/vstore). #1 allows > simpler bytecodes which put less load on the verifier. A significant subset of #2 > is needed, anyway, for telling the GC where to find object references. So #2 wins. P.S. And that's before we get into layout-polymorphic generics and type variables that include Q-values. Those also require the robust carrier type of #2. Option #1 was always a mirage. From karen.kinnear at oracle.com Tue Apr 11 22:38:52 2017 From: karen.kinnear at oracle.com (Karen Kinnear) Date: Tue, 11 Apr 2017 18:38:52 -0400 Subject: Valhalla Minimal Value Types review invitation In-Reply-To: <5CBE6C49-E7DC-418E-A7D0-0595E237D6E0@oracle.com> References: <5CBE6C49-E7DC-418E-A7D0-0595E237D6E0@oracle.com> Message-ID: <06CFD96F-95C3-4BF9-8F1C-7F2918A639E8@oracle.com> Summary of requested modifications from early prototyping experiments. Followed by action items. Terms used below: VCC for Value Capable Class annotated for creating a DVT: Derived Value This is to distinguish from the longer-term VVT, or Valhalla Value Type that will be supported by javac and have more capabilities, such as supporting interfaces and methods. 1. Features: Could you possibly introduce the term "Derived Value Type" for the unboxed form of a Value-capable-class? 2. Features: "Three" bytecode instructions ->"A small set of" 3. Value-capable classes annotation restrictions: (The details are TBD, but will be similar to the restrictions on internal annotations like @Contended or @PolymorphicSignature.) For early access, don't restrict the annotation. Option via command-line switch 4. Remove this entirely: "(Until there are bytecodes for directly accessing members of unboxed values, method handles will be available for the purpose, as they are for members of regular objects. This design endows both boxes and pure values with a common set of methods; it lifts box methods onto the derived values. The reasons for doing this, in the prototype system, are noted below. This somewhat awkward, backward setup is the necessary initial scaffolding, not the final edifice.)" -- we have simplified the implementation by removing methods from the DVT completely -- we have proposed adding new bytecodes for directly accessing members of DVTs vgetfield - access a field of a value type and vwithfield - "wither" - to create a new value type copy-on-write 5. Clarify: DVT will just contain the instance fields. VCC will remain unchanged. To invoke a method or access a static field, must box to VCC. Identity for both the VCC and DVT is "accidental?. 6. Restrictions on the POJO: A. It would help to have a bulleted list of restrictions, including error to throw I know the restrictions are intermixed in the text today. This is what I extracted: (p.3-4, 6) VCC 1. VCC superclass must be Object (and should be omitted) 2. May not inherit from any interfaces 3. the class must be final 4. all instance fields must be final 5. an Interface can not be marked with @DeriveValueType please change p.6 "interfaces (especially with default methods)" to clarify that there are no value type interfaces period. 6. must replace equals, hashcode, toString (with current Object syntax) 7. may not use any methods provided on Object specifically: may not use clone, finalize, wait, notify, notifyAll (directly) 8. May not inherit from any interfaces 9. may use getClass Remove Restrictions: 1. all constructors private *** remove this restriction *** Note: the DVT does not have a constructor - it can be created via a vunbox or a vdefault + vwithfield or as a side effect of newarray/multianewarray of a DVT 2. may contain primitive instance fields, but no reference instance fields *** requested extension by IBM and Oracle: please remove this restriction 3. may not contain generic instance fields *** erased generics are ok. No type variables in instance fields and no "any? generics 7. Continue with "value class may not contain a value class as an instance field". Value classes contained in references will not be flattened. 8. Splitting the value type from the object type Propose not using the nested class approach, to not tie us into this relationship longer term - so remove example and the "looks like an inner class" - leave this as an implementation detail, but do NOT expose DVT as a nested class 9. Splitting the value type from the object type p.5 "The original class is given a new synthetic field of the new value type, to hold the state for the original class". - to simplify implementation, and allow experiments which go beyond the initial MVT plans, we propose * that the VCC is left untouched * the DVT has a copy of the immutable instance fields - We think this qualifies as "any equivalent technique" on p.5 - the quote above would need modifying or removing 10. value bytecodes - the following are useful in the MethodHandle implementation, and likely to be useful for direct bytecode access - we would like to propose the following as the minimal bytecode set: in addition to vload, vstore, vreturn (and slot-specific variants) - vdefault/vwithfield - vbox/vunbox - vaload/vastore - vgetfield (fetch a field from a value type) - NOT vcmp_eq/ne (equality can be implemented as component-wise comparison) 11. clarify that for MVT 1.0, statics are only available through the box. (TODO: where does this go in shady?) 12. vdefault/vwithfield restrictions: vdefault only within constructor, vwithfield only within the class note: temporary issue: since DVT has no constructor or other methods, must allow vdefault also within the DVT and vwithfield across VCC/DVT pairs. note: a value type array allows default value to escape 13. value bytecodes vbytecodes for early access (not typed bytecodes for early access) 14. Value bytecodes use of Qtype as class component: "Initially the only valid use of a Q-type [is] as the class component of a CONSTANT_Methodref or CONSTANT_Fieldref is as a CONSTANT_MethodHandle constant." - if we extend the bytecodes as above, and we disallow anyone (MethodHandles, bytecodes) from invoking methods on Qtypes, we could modify this to disallow Q-types as the class component for CONSTANT_Methodref or CONSTANT_InterfaceMethodRef completely. 15. Q-types and bytecodes We propose modifying anewarray and multianewarray to allow operands that are Q-types. 16. Value Type Reflection With the proposed modifications in #5 above: i.e. leaving the VCC untouched and copying the instance fields to the DVT, the VCC now matches the source file. So Class.forName() would return the VCC which is the original POJO which fits the backward compatibility model. So we don't need a separate SourceClass, but leaving it in the proposal provides implementation flexibility. 17. splitting the value type from the VCC - name of the derived value type Decided that MVT 1.0 should clarify that we do not want reflection or bytecodes to resolve a temporary name to find a DVT. Users can use ValueType.valueClass() to get the class or vunbox or QVCCNAME in a descriptor (i.e. not the temporary name) - so the internal naming does not have to be specified or known. 18. Reflection: There is no contract in MVT for the DVT to support reflection. It is ok to throw an exception - need to investigate what is easiest here. Core reflection works on the VCC, so box to use it. 19. When we update to supporting VVT there is no guarantee that MVT classes will continue to work. This is experimental, the classfile format is experimental, the byte codes are temporary. 20. Vdefault/vwithfield: need to study relative to verification rules to ensure no escape of partially constructed AIs: 1. Comparison of PackedObjects vs. Minimal Value Types: Dan Heidinga/Bjorn Vandal 2. Request for JVMS extensions draft: Dan Smith will be at the April 26th meeting 3. MVT JEP: John 4. MVT update based on feedback: John 5. JEP for how to do experimental JVMS/JVM features: ?? - e.g. command-line flag vs. classfile version? Additional questions: MethodHandle support: Want a way to have MH.Lookup operate on a DVT as if it had methods. Possibly by doing the Lookup on the VCC and doing an asType conversion. Also want a way for MH.arraygetter/arraysetter to pass a DVT. Additional questions welcome. thanks, Karen > On Mar 30, 2017, at 12:35 PM, Karen Kinnear wrote: > > Those interested in the Valhalla Minimal Value Types early adopters proposal are invited > to give feedback on John Rose?s 0.2 version: > http://cr.openjdk.java.net/~jrose/values/shady-values.html > > The goal is to make an early subset of value type behaviors available for experimentation. > Email feedback is welcome. > > In addition, you may join us for interactive discussions: > > Biweekly starting Wednesday April 12th, 9am PT > > URL to join the Oracle Zoom Conference: https://oracle.zoom.us/j/608512921?pwd=tVeSnAlOQYI%3D > > audio only: +1 646 558 8656 or +1 408 638 0968 US Toll > Meeting ID: 608 512 921 > > My sincere apologies for missing you on the original invitation which we issued to IBM so > that we could have a JVM implementors' discussion on questions about the Minimal Value Types > proposal. > > I will send a follow-up email with a summary of the requested modifications so far from the prototyping > experiments. > > thanks, > Karen > > > > > > From john.r.rose at oracle.com Wed Apr 12 10:09:00 2017 From: john.r.rose at oracle.com (John Rose) Date: Wed, 12 Apr 2017 03:09:00 -0700 Subject: Valhalla Minimal Value Types review invitation In-Reply-To: <06CFD96F-95C3-4BF9-8F1C-7F2918A639E8@oracle.com> References: <5CBE6C49-E7DC-418E-A7D0-0595E237D6E0@oracle.com> <06CFD96F-95C3-4BF9-8F1C-7F2918A639E8@oracle.com> Message-ID: <877D2810-68D3-45D0-8B5B-E31581198D9D@oracle.com> On Apr 11, 2017, at 3:38 PM, Karen Kinnear wrote: > > Summary of requested modifications from early prototyping experiments. Followed by action items. Thanks, Karen. It is very, very helpful to have all these points rolled up like this. I just updated the posted proposal to respond to your points 1 to 13. http://cr.openjdk.java.net/~jrose/values/shady-values.html http://cr.openjdk.java.net/~jrose/values/shady-values.md I should get through the rest of the points this week. There are only two places so far where I have had to disagree slightly. > On Apr 11, 2017, at 3:38 PM, Karen Kinnear wrote: > > 6. Restrictions on the POJO: ... > 8. May not inherit from any interfaces It is useful and harmless for the POJO to implement interfaces. In the new section "Boxing, unboxing, and borrowing" I touch on this. ... > 12. vdefault/vwithfield > restrictions: vdefault only within constructor, vwithfield only within the class > note: temporary issue: since DVT has no constructor or other methods, must allow > vdefault also within the DVT and vwithfield across VCC/DVT pairs. > note: a value type array allows default value to escape I added discussion of restrictions on these bytecodes. Bottom line: There is no reason to restrict vdefault. Talk to you soon! ? John P.S. The referenced file and its source is enclosed for archiving and convenient access. From daniel.smith at oracle.com Tue Apr 18 18:42:16 2017 From: daniel.smith at oracle.com (Dan Smith) Date: Tue, 18 Apr 2017 12:42:16 -0600 Subject: Draft JVMS changes for Nestmates Message-ID: Hi, all. I've uploaded a draft of JVMS changes for JEP 181 "Align JVM Checks with Java Language Rules for Nested Classes" to: http://cr.openjdk.java.net/~dlsmith/private-access.html Some comments below on my thinking in drafting the spec text, and on the JEP generally. JEP name I don't know how permanent JEP names are supposed to be, but I'd prefer a different name at this point. Something like: "Expanded JVM Access to Private Members"?shorter, focused on the feature itself rather than its relationship to the Java language. Or maybe "Class Nests for Access to Private Members". Terminology The term "nest" is nice because it's short and mostly unspoiled by overloading in this context (I think?); it's not great because it's informal and doesn't mean anything the first time you hear it. I thought about something more clinical like "access control context", but I'm not convinced that's an improvement. How do others feel? The JEP uses "nest top" to describe the class that nest members reference; I prefer "host class", which better describes the class's role and isn't tied to the Java "top level class" concept. I know we use "host class" internally in Hotspot, perhaps when working with anonymous classes (of the JVM flavor), but I think in that context it will ultimately mean the same thing? Are we comfortable repurposing the term in this way? I follow Brian's model when it comes to nest membership (5.4.4): every class belongs to a nest, possibly (in the absence of MemberOfNest) the nest hosted by itself. Many nests are singletons without any associated explicit attributes. Verification of MemberOfNest I include a discussion block about different options of validating MemberOfNest. I think the consensus, and my preference, is to do it during verification of the member class. (NestMembers, on the other hand, is never validated, except as a side-effect of checking MemberOfNest.) Verification of invokespecial Allowing invokespecial to refer to classes other than the current class and its supers is a significant change. I noticed and relied heavily on the parallel with invokevirtual making protected method calls. So I tried to make the two as similar as possible. In a few places, the treatment of protected methods doesn't seem ideal, and rather than trying to mirror that with non-private invokespecial, I modified the protected method treatment. The "protected check" of verification, in particular, was a mess before, and I think I've made it a lot more manageable, and compatible with a parallel rule for invokespecial. I could use some feedback on exactly what Hotspot's verifier has been doing here, though, since I'm pretty sure it didn't match the old specified rules. MethodHandle resolution The spec (5.4.3.5) is vague about what errors can occur during MethodHandle resolution. I assume any linkage error specified for the instruction can also occur via MethodHandle resolution, and that will include failures due to invokespecial improperly referencing a class. Dynamic checking of invokespecial receivers When invokespecial involves interface types, the verifier can't guarantee that receiver objects actually implement that interface (JDK-8134358). It's an open question whether this is really a problem, but I included a tentative fix in the invokespecial runtime rules. Compiler changes The JEP text can't seem to decide if compiler changes are part of it, or a follow-up exercise. I think we're best off explicitly including the compiler changes, which will provide opportunities for design validation and testing. API changes I haven't tried to address changes that need to be made to APIs. Somebody will need to. For example, Lookup.findSpecial probably needs to make adjustments to account for private members in nestmates, and to parallel the new verification/linkage rules (e.g., follow Lookup.findVirtual in restricting the receiver type). Security risk The JEP text should acknowledge that, while this does allow compilers to grant finer-grained access to members shared by nestmates, it also pushes compilers to grant broader access to members that were previously kept private. It's a trade-off, and presumably a good one because nestmates are completely trusted, while package-mates might sometimes be suspect. (I guess this argument really ought to be made from the top: declaring things with package access is worse than inventing a new level of access because ________.) ?Dan From brian.goetz at oracle.com Tue Apr 18 18:58:22 2017 From: brian.goetz at oracle.com (Brian Goetz) Date: Tue, 18 Apr 2017 14:58:22 -0400 Subject: Draft JVMS changes for Nestmates In-Reply-To: References: Message-ID: <815641f5-e1c9-5b4c-2fd6-d5e1c6610ea7@oracle.com> > JEP name > > I don't know how permanent JEP names are supposed to be, but I'd prefer a different name at this point. Something like: "Expanded JVM Access to Private Members"?shorter, focused on the feature itself rather than its relationship to the Java language. Or maybe "Class Nests for Access to Private Members". I like the latter; it captures the core feature here, which is introducing a new concentric sphere of access control in the VM, which we call "nests". That it was inspired by translation challenges in Java, or useful to addressing those, is secondary. > Terminology > > The term "nest" is nice because it's short and mostly unspoiled by overloading in this context (I think?); it's not great because it's informal and doesn't mean anything the first time you hear it. I thought about something more clinical like "access control context", but I'm not convinced that's an improvement. How do others feel? Nest connotes "things in other things", like Matryoshka dolls, which I think is a positive. > The JEP uses "nest top" to describe the class that nest members reference; I prefer "host class", which better describes the class's role and isn't tied to the Java "top level class" concept. I know we use "host class" internally in Hotspot, perhaps when working with anonymous classes (of the JVM flavor), but I think in that context it will ultimately mean the same thing? Are we comfortable repurposing the term in this way? Or: "nest host" or "nest host class" or "nest parent" ... The primary purpose of having a nest top is so that there is a well-defined canonical member of each nest, to which we can attach all the nest metadata and ensure that computations regarding nest-ness are well-defined. It doesn't have to be the "top" class at the language level. I think the term should connote canonical-ness rather than top-ness. > I follow Brian's model when it comes to nest membership (5.4.4): every class belongs to a nest, possibly (in the absence of MemberOfNest) the nest hosted by itself. Many nests are singletons without any associated explicit attributes. More precisely: nests form a /partition/ over classes. This means every class belongs to exactly one nest, so the function Nest(Class) is total. > Compiler changes > > The JEP text can't seem to decide if compiler changes are part of it, or a follow-up exercise. I think we're best off explicitly including the compiler changes, which will provide opportunities for design validation and testing. That was my bad. I was trying to capture that we didn't require an immediate flag day -- that for any given compiler, dropping access bridges out of the compiler implementation could happen any time after the VM acquired nestmate support -- but for our own implementation, we'd be silly to not get the test and validation benefit of dropping unnecessary bridge generation ASAP. > API changes We need to decide whether we are delivering a reflection component, and if so, what that looks like. Minimal candidate: class Nest { Class hostClass() { ... } boolean isMember(Class clazz) { .... } } class Class { Nest getNest() { ... } } From forax at univ-mlv.fr Tue Apr 18 20:12:16 2017 From: forax at univ-mlv.fr (Remi Forax) Date: Tue, 18 Apr 2017 22:12:16 +0200 (CEST) Subject: Draft JVMS changes for Nestmates In-Reply-To: <0848391A-AF8C-4B4D-8CDA-FFC727E5CF6B@oracle.com> References: <0848391A-AF8C-4B4D-8CDA-FFC727E5CF6B@oracle.com> Message-ID: <1247948811.901298.1492546336021.JavaMail.zimbra@u-pem.fr> ----- Mail original ----- > De: "Dan Smith" > ?: valhalla-dev at openjdk.java.net > Envoy?: Mardi 18 Avril 2017 20:44:11 > Objet: Fwd: Draft JVMS changes for Nestmates > For those not subscribed to (or not paying attention to) valhalla-spec-experts, > I just posted a link to an updated draft of JVMS changes. > >> Begin forwarded message: >> >> From: Dan Smith >> Subject: Draft JVMS changes for Nestmates >> Date: April 18, 2017 at 12:42:16 PM MDT >> To: valhalla-spec-experts at openjdk.java.net >> >> Hi, all. >> >> I've uploaded a draft of JVMS changes for JEP 181 "Align JVM Checks with Java >> Language Rules for Nested Classes" to: >> http://cr.openjdk.java.net/~dlsmith/private-access.html >> >> Some comments below on my thinking in drafting the spec text, and on the JEP >> generally. >> >> JEP name >> >> I don't know how permanent JEP names are supposed to be, but I'd prefer a >> different name at this point. Something like: "Expanded JVM Access to Private >> Members"?shorter, focused on the feature itself rather than its relationship to >> the Java language. Or maybe "Class Nests for Access to Private Members". >> >> Terminology >> >> The term "nest" is nice because it's short and mostly unspoiled by overloading >> in this context (I think?); it's not great because it's informal and doesn't >> mean anything the first time you hear it. I thought about something more >> clinical like "access control context", but I'm not convinced that's an >> improvement. How do others feel? >> >> The JEP uses "nest top" to describe the class that nest members reference; I >> prefer "host class", which better describes the class's role and isn't tied to >> the Java "top level class" concept. I know we use "host class" internally in >> Hotspot, perhaps when working with anonymous classes (of the JVM flavor), but I >> think in that context it will ultimately mean the same thing? Are we >> comfortable repurposing the term in this way? The semantics of an anonynmous host class and a nest host class are different. an anonymous host class is used for every access check while the nest host class is used Anyway, nest host class works for me. >> >> I follow Brian's model when it comes to nest membership (5.4.4): every class >> belongs to a nest, possibly (in the absence of MemberOfNest) the nest hosted by >> itself. Many nests are singletons without any associated explicit attributes. >> >> Verification of MemberOfNest >> >> I include a discussion block about different options of validating MemberOfNest. >> I think the consensus, and my preference, is to do it during verification of >> the member class. (NestMembers, on the other hand, is never validated, except >> as a side-effect of checking MemberOfNest.) It means that a NestMembers can contains Class that are not yet/never defined. It's not a problem for me. >> >> Verification of invokespecial >> >> Allowing invokespecial to refer to classes other than the current class and its >> supers is a significant change. I noticed and relied heavily on the parallel >> with invokevirtual making protected method calls. So I tried to make the two as >> similar as possible. In a few places, the treatment of protected methods >> doesn't seem ideal, and rather than trying to mirror that with non-private >> invokespecial, I modified the protected method treatment. >> >> The "protected check" of verification, in particular, was a mess before, and I >> think I've made it a lot more manageable, and compatible with a parallel rule >> for invokespecial. I could use some feedback on exactly what Hotspot's verifier >> has been doing here, though, since I'm pretty sure it didn't match the old >> specified rules. >> >> MethodHandle resolution >> >> The spec (5.4.3.5) is vague about what errors can occur during MethodHandle >> resolution. I assume any linkage error specified for the instruction can also >> occur via MethodHandle resolution, and that will include failures due to >> invokespecial improperly referencing a class. method handle behavior is the same as their corresponding bytecodes behavior, errors are converted to exceptions. >> >> Dynamic checking of invokespecial receivers >> >> When invokespecial involves interface types, the verifier can't guarantee that >> receiver objects actually implement that interface (JDK-8134358). It's an open >> question whether this is really a problem, but I included a tentative fix in >> the invokespecial runtime rules. I'm not sure it's related to nestmate ? >> >> Compiler changes >> >> The JEP text can't seem to decide if compiler changes are part of it, or a >> follow-up exercise. I think we're best off explicitly including the compiler >> changes, which will provide opportunities for design validation and testing. >> >> API changes >> >> I haven't tried to address changes that need to be made to APIs. Somebody will >> need to. For example, Lookup.findSpecial probably needs to make adjustments to >> account for private members in nestmates, and to parallel the new >> verification/linkage rules (e.g., follow Lookup.findVirtual in restricting the >> receiver type). >> >> Security risk >> >> The JEP text should acknowledge that, while this does allow compilers to grant >> finer-grained access to members shared by nestmates, it also pushes compilers >> to grant broader access to members that were previously kept private. It's a >> trade-off, and presumably a good one because nestmates are completely trusted, >> while package-mates might sometimes be suspect. >> >> (I guess this argument really ought to be made from the top: declaring things >> with package access is worse than inventing a new level of access because >> ________.) >> > > ?Dan R?mi From brian.goetz at oracle.com Tue Apr 18 20:19:40 2017 From: brian.goetz at oracle.com (Brian Goetz) Date: Tue, 18 Apr 2017 16:19:40 -0400 Subject: Draft JVMS changes for Nestmates In-Reply-To: <1247948811.901298.1492546336021.JavaMail.zimbra@u-pem.fr> References: <0848391A-AF8C-4B4D-8CDA-FFC727E5CF6B@oracle.com> <1247948811.901298.1492546336021.JavaMail.zimbra@u-pem.fr> Message-ID: > The semantics of an anonynmous host class and a nest host class are different. > an anonymous host class is used for every access check while the nest host class is used They are *now*. But we would like to converge them. As we bring VMACs into the "supported light", we can adjust the semantics. From forax at univ-mlv.fr Tue Apr 18 20:37:45 2017 From: forax at univ-mlv.fr (forax at univ-mlv.fr) Date: Tue, 18 Apr 2017 22:37:45 +0200 (CEST) Subject: Draft JVMS changes for Nestmates In-Reply-To: References: <0848391A-AF8C-4B4D-8CDA-FFC727E5CF6B@oracle.com> <1247948811.901298.1492546336021.JavaMail.zimbra@u-pem.fr> Message-ID: <336147522.903656.1492547865474.JavaMail.zimbra@u-pem.fr> ----- Mail original ----- > De: "Brian Goetz" > ?: "Remi Forax" , "Dan Smith" > Cc: valhalla-spec-experts at openjdk.java.net > Envoy?: Mardi 18 Avril 2017 22:19:40 > Objet: Re: Draft JVMS changes for Nestmates >> The semantics of an anonynmous host class and a nest host class are different. >> an anonymous host class is used for every access check while the nest host class >> is used > > They are *now*. But we would like to converge them. As we bring VMACs > into the "supported light", we can adjust the semantics. and retired unsafe.defineAnonymousClass which still have some perf issue ... R?mi From daniel.smith at oracle.com Tue Apr 18 20:49:08 2017 From: daniel.smith at oracle.com (Dan Smith) Date: Tue, 18 Apr 2017 14:49:08 -0600 Subject: Draft JVMS changes for Nestmates In-Reply-To: <1247948811.901298.1492546336021.JavaMail.zimbra@u-pem.fr> References: <0848391A-AF8C-4B4D-8CDA-FFC727E5CF6B@oracle.com> <1247948811.901298.1492546336021.JavaMail.zimbra@u-pem.fr> Message-ID: <25912A06-ABE8-4E83-8EC6-9A9CA6888B18@oracle.com> > On Apr 18, 2017, at 2:12 PM, Remi Forax wrote: > >>> Verification of MemberOfNest >>> >>> I include a discussion block about different options of validating MemberOfNest. >>> I think the consensus, and my preference, is to do it during verification of >>> the member class. (NestMembers, on the other hand, is never validated, except >>> as a side-effect of checking MemberOfNest.) > > It means that a NestMembers can contains Class that are not yet/never defined. > It's not a problem for me. Yep. Also, multiple classes can claim the same nest member class in their NestMembers attributes. Not a problem as long as the MemberOfNest attribute (if any) of the member class points to a host class that claims it. ?Dan From john.r.rose at oracle.com Wed Apr 19 05:52:32 2017 From: john.r.rose at oracle.com (John Rose) Date: Tue, 18 Apr 2017 22:52:32 -0700 Subject: Draft JVMS changes for Nestmates In-Reply-To: <815641f5-e1c9-5b4c-2fd6-d5e1c6610ea7@oracle.com> References: <815641f5-e1c9-5b4c-2fd6-d5e1c6610ea7@oracle.com> Message-ID: On Apr 18, 2017, at 11:58 AM, Brian Goetz wrote: > > Minimal candidate: > > class Nest { > Class hostClass() { ... } > boolean isMember(Class clazz) { .... } > } > > class Class { > Nest getNest() { ... } > } Or just: class Class { Class getNestHost(); // = this.MemberOfNest_attribute.host_class || this //and maybe: //boolean isInSameNest(Class that) { return this.getNestHost() == that.getNestHost(); } } From david.holmes at oracle.com Wed Apr 19 06:40:46 2017 From: david.holmes at oracle.com (David Holmes) Date: Wed, 19 Apr 2017 16:40:46 +1000 Subject: Draft JVMS changes for Nestmates In-Reply-To: References: Message-ID: Hi Dan, First, thanks for getting to this so quickly! Much appreciated. On 19/04/2017 4:42 AM, Dan Smith wrote: > Hi, all. > > I've uploaded a draft of JVMS changes for JEP 181 "Align JVM Checks with Java Language Rules for Nested Classes" to: > http://cr.openjdk.java.net/~dlsmith/private-access.html > > Some comments below on my thinking in drafting the spec text, and on the JEP generally. > > JEP name > > I don't know how permanent JEP names are supposed to be, but I'd prefer a different name at this point. Something like: "Expanded JVM Access to Private Members"?shorter, focused on the feature itself rather than its relationship to the Java language. Or maybe "Class Nests for Access to Private Members". The intent of the name is to aid in the contextual positioning of this JEP ie address why do we even need this. The JEP addresses an inconsistency between access controls in the Java programming language and access controls in the JVM. The primary goal here is to align the two, while at the same time allowing for additional usage of nestmates in the future. > Terminology > > The term "nest" is nice because it's short and mostly unspoiled by overloading in this context (I think?); it's not great because it's informal and doesn't mean anything the first time you hear it. I thought about something more clinical like "access control context", but I'm not convinced that's an improvement. How do others feel? I think John has been talking about "nests" for at least 4 years now. :) It fits nicely with the existing terminology of "nested types" - it seems quite natural that "nested types" form a "nest". I have no problem with it being "informal" in some sense, but we are using it to define a specific relationship between certain types. > The JEP uses "nest top" to describe the class that nest members reference; I prefer "host class", which better describes the class's role and isn't tied to the Java "top level class" concept. I know we use "host class" internally in Hotspot, perhaps when working with anonymous classes (of the JVM flavor), but I think in that context it will ultimately mean the same thing? Are we comfortable repurposing the term in this way? Given that a nest-top type must be a top-level type, I think nest-top fits perfectly. I dislike "host class" because that is already used for the quite different case of VMACs. I can live with other names but I think they should incorporate "nest" eg. nest-holder, nest-host. But I think nest-top is an ideal match. (And no I didn't invent this terminology :) ). > I follow Brian's model when it comes to nest membership (5.4.4): every class belongs to a nest, possibly (in the absence of MemberOfNest) the nest hosted by itself. Many nests are singletons without any associated explicit attributes. I have not yet seen a follow up from John as to whether we require or just allow, an empty NestMembers attribute to indicate a singleton nest member. > Verification of MemberOfNest > > I include a discussion block about different options of validating MemberOfNest. I think the consensus, and my preference, is to do it during verification of the member class. (NestMembers, on the other hand, is never validated, except as a side-effect of checking MemberOfNest.) Given verification is optional the current approach is to validate during linking just prior to the verification step. Whether that can conceptually be considered part of verification I am unsure. > Verification of invokespecial > > Allowing invokespecial to refer to classes other than the current class and its supers is a significant change. I noticed and relied heavily on the parallel with invokevirtual making protected method calls. So I tried to make the two as similar as possible. In a few places, the treatment of protected methods doesn't seem ideal, and rather than trying to mirror that with non-private invokespecial, I modified the protected method treatment. This concerns me because protected-access is a somewhat complex/messy issue so aligning private-access with it seems to go in the wrong direction. Cleaning up the protected access rules, while perhaps desirable, is technically out-of-scope for this JEP in my opinion. > The "protected check" of verification, in particular, was a mess before, and I think I've made it a lot more manageable, and compatible with a parallel rule for invokespecial. I could use some feedback on exactly what Hotspot's verifier has been doing here, though, since I'm pretty sure it didn't match the old specified rules. Might be better to take this up separately. > MethodHandle resolution > > The spec (5.4.3.5) is vague about what errors can occur during MethodHandle resolution. I assume any linkage error specified for the instruction can also occur via MethodHandle resolution, and that will include failures due to invokespecial improperly referencing a class. This is an area I have least understanding of, but this seems to cover it: "In each step, any exception that can be thrown as a result of failure of resolution of a class or interface or field or method reference can be thrown as a result of failure of method handle resolution." > Dynamic checking of invokespecial receivers > > When invokespecial involves interface types, the verifier can't guarantee that receiver objects actually implement that interface (JDK-8134358). It's an open question whether this is really a problem, but I included a tentative fix in the invokespecial runtime rules. This seems out of scope in the general case, but there may be nestmate specific actions required. > Compiler changes > > The JEP text can't seem to decide if compiler changes are part of it, or a follow-up exercise. I think we're best off explicitly including the compiler changes, which will provide opportunities for design validation and testing. Not sure what gives you that impression, but javac changes are an essential part of this. Lacking a compiler spec I assume this will be handled informally. But the attribute definitions (see comments below) should dictate what a source compiler is required to do. > API changes > > I haven't tried to address changes that need to be made to APIs. Somebody will need to. For example, Lookup.findSpecial probably needs to make adjustments to account for private members in nestmates, and to parallel the new verification/linkage rules (e.g., follow Lookup.findVirtual in restricting the receiver type). API changes will be addressed as each API needs to be modified. > Security risk > > The JEP text should acknowledge that, while this does allow compilers to grant finer-grained access to members shared by nestmates, it also pushes compilers to grant broader access to members that were previously kept private. It's a trade-off, and presumably a good one because nestmates are completely trusted, while package-mates might sometimes be suspect. I'm not following you here. What broader access is being granted to members that were previously kept private? This effort allows private-only access to members that were previously package-accessible. > (I guess this argument really ought to be made from the top: declaring things with package access is worse than inventing a new level of access because ________.) > > ?Dan Now looking at specific proposed changes to the spec wording ... 4.7 Attributes Regarding the Exceptions attribute, it seems to be used only to store data to be passed back to the libverify and the reflection API. Regarding the definitions of NestMembers and MemberOfNest ... I modelled the definitions in: https://bugs.openjdk.java.net/browse/JDK-8177020 based on those for the innerclass (and related) attributes. I'm not sure why you moved away from those definitions as they pinned down exactly when these attributes are expected to appear. Without those parts the current definitions seem optional - ie they define what the attribute means if present, but they do not require its presence. I do not believe these attributes should be optional for classfile version 54, but required whenever a nest does exist. A java source compiler writer should look at these definitions and know when they must generate these attributes. --- 4.7.26 The MemberOfNest Attribute "the class is implicitly a member of its own nest" I suggest changing "a" to "the sole". Otherwise "a" implies there may be other members. As these are static properties being defined I don't think we need to be accounting for some future dynamic expansion of a nest. --- 4.9.2 Structural Constraints The changes to invokespecial seem okay, but I have a problem with terminology. In: "If invokespecial is used to invoke a non-private, non- method, referenced via a superclass or a direct superinterface, ..." what does "referenced via" mean? In: "If getfield or putfield is used to access a protected field referenced in a superclass ..." and following, you changed "declared" to read "referenced". Again I do not understand what this is supposed to mean. It is the declaration site of the field or method that is needed to determine whether that member is in the same or another package. To me "declared in a superclass ..." was exactly correct. Your note does not make the change any clearer to me. --- 4.10.1 Verification by Type Checking Does the order in the rules mandate the order in which the VM must actually check things? classHasValidNest seems much more complex than the actual runtime process of validation (though it may just be exposing distinct steps that are lumped together at runtime). --- 4.10.1.8 Type Checking for Restricted Member References I'm not very good at reading or understanding these rules, but I'm surprised that the invokespecial rules seem to make no mention of nestmates at all: "A method reference is allowed by an invokespecial instruction if it is not a restricted invokespecial reference, ..." This seems to open things up to all private method references?? --- 5.4.4 Access Control I don't think you need to restate: "A class with a MemberOfNest attribute belongs to the nest hosted by the referenced host class. A class without a MemberOfNest attribute implicitly belongs to a nest hosted by the class itself. (If the class also lacks a NestMembers attribute, then the nest has only one member.)" The rule is simply stated as is: "belonging to the same nest as D" - belonging to the same nest is, or should be, already defined elsewhere. I'm avoiding commenting on the protected access changes but do want to raise a concern that there are no changes to Method resolution (5.4.3.3) yet resolution relies on the definition of access control to prune/discard inaccessible methods. It appears now that we're allowing more potential successful resolutions, then using the additional rules on the invoke* bytecodes to try and discard any undesirable results. --- 6.5 getfield (and others) You note "These rules are redundant: verification already guarantees them " but this brings me back to an area I keep raising concerns about: the split between static verification based checks and dynamic runtime checks. Yes the verifier precludes certain things but if we run without verification the rules expressed for the bytecodes are considered to be still required at runtime. If you delete them because they overlap with verifier rules we have no way to tell which rules must be enforced regardless of verification status. --- 6.5 invokespecial You note regarding the IllegalAccessError: "This replaces a VerifyError previously specified by 4.9.2. The check must be delayed until after resolution in order to determine whether the referenced method is private." We know the access flags for the referenced method at verification time - shouldn't that be the sole basis for the verifier check? Afterall it is verifying the static properties of the classfile and bytecode. If the actual resolved method has different access to the referenced method then that may lead to ICCE (depending on the exact nature of the change - a private method made public may not be an issue for example). --- 5.4.3.3 Method resolution You did not make any changes here, but as per my comment in the bug report we do require, IMHO, further tightening here to ensure nestmate invokespecial invocations do not resolve to method implementations that they should not. The example here is a hierarchy of nestmates (C extends B extends A) where A and C both declare a private method "void m()" and we have an invocation c.m() where c is an instance of class C. We then "separately compile" C to remove the definition of m(). At runtime method resolution will locate A.m, even though the method reference was for C.m. Normally the accessibility rules would exclude A.m from being a viable candidate but here the classes are nestmates so A.m is accessible. But it would be wrong to invoke A.m. This case should throw ICCE or NSME. Thanks, David ----- From john.r.rose at oracle.com Wed Apr 19 06:45:06 2017 From: john.r.rose at oracle.com (John Rose) Date: Tue, 18 Apr 2017 23:45:06 -0700 Subject: Draft JVMS changes for Nestmates In-Reply-To: <25912A06-ABE8-4E83-8EC6-9A9CA6888B18@oracle.com> References: <0848391A-AF8C-4B4D-8CDA-FFC727E5CF6B@oracle.com> <1247948811.901298.1492546336021.JavaMail.zimbra@u-pem.fr> <25912A06-ABE8-4E83-8EC6-9A9CA6888B18@oracle.com> Message-ID: On Apr 18, 2017, at 1:49 PM, Dan Smith wrote: > > >> On Apr 18, 2017, at 2:12 PM, Remi Forax > wrote: >> >>>> Verification of MemberOfNest >>>> >>>> I include a discussion block about different options of validating MemberOfNest. >>>> I think the consensus, and my preference, is to do it during verification of >>>> the member class. (NestMembers, on the other hand, is never validated, except >>>> as a side-effect of checking MemberOfNest.) >> >> It means that a NestMembers can contains Class that are not yet/never defined. >> It's not a problem for me. > > Yep. Also, multiple classes can claim the same nest member class in their NestMembers attributes. Not a problem as long as the MemberOfNest attribute (if any) of the member class points to a host class that claims it. We need to specify more structural constraints on the new attributes. If we make them more strongly normalizing, there will be fewer chances for semantic bugs on unexpected inputs. Also, making them structural constraints means we check them early, during class file loading. These are good, I think: - NMs must be non-empty (degenerate nest is never explicit) - NMs may not contain duplicates (cf. treatment of ClassFile.interfaces) - NMs may not contain the current class (i.e., an index to a class with the same name as this_class) - MoN may not contain the current class (ditto) - MoN may contain only a package sibling (i.e., a referenced class name must have the same package prefix as this_class) - NMs may contain only package siblings (ditto) - NMs and MoN may not refer to array classes (this is probably implied by the package prefix checks) You already covered: - a class may have exactly one or zero of NMs or MoN (no double dipping of any kind) - NMs and MoN may only have CONSTANT_Class attributes (implies no nulls, well-formed names) David's prototype has the duplication check, and some of the other checks happen later. I think they should all happen during class loading. ? John From john.r.rose at oracle.com Wed Apr 19 07:38:36 2017 From: john.r.rose at oracle.com (John Rose) Date: Wed, 19 Apr 2017 00:38:36 -0700 Subject: Draft JVMS changes for Nestmates In-Reply-To: References: Message-ID: <6E328B19-85A5-44AB-A549-06E7BAD4D12C@oracle.com> On Apr 18, 2017, at 11:42 AM, Dan Smith wrote: > > Hi, all. > > I've uploaded a draft of JVMS changes for JEP 181 "Align JVM Checks with Java Language Rules for Nested Classes" to: > http://cr.openjdk.java.net/~dlsmith/private-access.html Nice. More comments on that later. > Some comments below on my thinking in drafting the spec text, and on the JEP generally. > > JEP name > > I don't know how permanent JEP names are supposed to be, but I'd prefer a different name at this point. Something like: "Expanded JVM Access to Private Members"?shorter, focused on the feature itself rather than its relationship to the Java language. Or maybe "Class Nests for Access to Private Members". +1 to Brian's suggestion > Terminology > > The term "nest" is nice because it's short and mostly unspoiled by overloading in this context (I think?); it's not great because it's informal and doesn't mean anything the first time you hear it. I thought about something more clinical like "access control context", but I'm not convinced that's an improvement. How do others feel? The terms for the JVM's circles of trust are "nest", "package", "module". "Access control context" does not belong on that list of terms. "Nest" connotes the motivation to support nested classes. (Could also have said "block" or "scope"?those are also relevant language constructs.) "Nest" also connotes a home-like place and therefore privacy. (Could also have said "home" or "family"?those are also private spaces. Or "crib"?) Only "nest" connotes both AFAIK. > The JEP uses "nest top" to describe the class that nest members reference; I prefer "host class", which better describes the class's role and isn't tied to the Java "top level class" concept. I know we use "host class" internally in Hotspot, perhaps when working with anonymous classes (of the JVM flavor), but I think in that context it will ultimately mean the same thing? Are we comfortable repurposing the term in this way? I am comfortable handing over the term "host" for this purpose. I expect Lookup.defineClass to inject into the nest of the Lookup.lookupClass, if the lookup has private access. That's what we call hosting for VMACs, almost exactly. (+1 to Brian's comments to Remi.) I would prefer "nest host" or "nest host class" or "host class of nest" to plain "host class", because it will be better if we mention "nest" when we mention the term for a nest's host class. Downside is pronunciation of "nest-host" sounds like a sneeze. > I follow Brian's model when it comes to nest membership (5.4.4): every class belongs to a nest, possibly (in the absence of MemberOfNest) the nest hosted by itself. Many nests are singletons without any associated explicit attributes. +1 > Verification of MemberOfNest > > I include a discussion block about different options of validating MemberOfNest. I think the consensus, and my preference, is to do it during verification of the member class. (NestMembers, on the other hand, is never validated, except as a side-effect of checking MemberOfNest.) See my previous message. There are more structural constraints I'd like to see. (But no additional class loading!) I agree that MemberOfNest needs to be checked early. I'd prefer to do it along with the supers but that leads to vicious dependency cycles. So verification time is best. IIRC that's also called "link time", which is intuitive to me: One of the first things you do when you link a class is determine which nest it is in. I totally agree that NestMembers should only be checked as needed. It's asymmetric, but the asymmetry lets us escape a bunch of class file loading. Here's a corner case we need to cover: I have to load my supers before I can load myself. If one (or more) of my supers are in the same nest as me, the checking of MoN must succeed no matter who is the host class of the nest. This could get embarrassing if the host class is the subtype, if the super then needs to recursively load the subtype (which is waiting on the supertype to load) before it can validate its MoN attribute. If verification (linking) is started after all of the classes are loaded, things are OK, since the MoN checks happen only after everything is loaded. N.B. The class file parser code in HotSpot sometimes says "verify" when I think it really means "check constraints". (The code also mentions constraints.) It does not mean verification in the sense that we are talking about here. The extra constraint checking I propose should happen during parsing/loading not linking/"verification". > Verification of invokespecial > > Allowing invokespecial to refer to classes other than the current class and its supers is a significant change. I noticed and relied heavily on the parallel with invokevirtual making protected method calls. So I tried to make the two as similar as possible. Nice work. > In a few places, the treatment of protected methods doesn't seem ideal, and rather than trying to mirror that with non-private invokespecial, I modified the protected method treatment. I wish we could maximize common formulations of the rules for protected references and invokespecial. They are very similar. I was surprised to see invokespecial occasionally mentioned cases where it only works on private members. Is that a current constraint, or a new one? I thought a class could invokespecial any of its methods, willy-nilly, and this permission would seem to extend naturally to nestmates. (I think I'm forgetting something here.) > The "protected check" of verification, in particular, was a mess before, and I think I've made it a lot more manageable, and compatible with a parallel rule for invokespecial. I could use some feedback on exactly what Hotspot's verifier has been doing here, though, since I'm pretty sure it didn't match the old specified rules. (Uh? Paging Dr. Protector. What? He retired when!?) Well, this will take some doing. But I agree strongly that you are making an improvement to the spec. > MethodHandle resolution > > The spec (5.4.3.5) is vague about what errors can occur during MethodHandle resolution. I assume any linkage error specified for the instruction can also occur via MethodHandle resolution, and that will include failures due to invokespecial improperly referencing a class. That is correct. The errors are also morphed to ReflectiveOperationExceptions for the corresponding Lookup API calls. > Dynamic checking of invokespecial receivers > > When invokespecial involves interface types, the verifier can't guarantee that receiver objects actually implement that interface (JDK-8134358). It's an open question whether this is really a problem, but I included a tentative fix in the invokespecial runtime rules. I am leaning towards putting this check in. If we put it in, it should go in everywhere, including invokespecial of default methods, both in and out of the caller's nest. I think your language might have allowed an untyped loophole for nestmates. My goal in this is simplicity of effect: There are no loopholes, no exceptions to typing of the incoming receiver (L0) of a default method. If we are going to check any of these invokespecial receivers, we must check them all. > Compiler changes > > The JEP text can't seem to decide if compiler changes are part of it, or a follow-up exercise. I think we're best off explicitly including the compiler changes, which will provide opportunities for design validation and testing. Good point. > API changes > > I haven't tried to address changes that need to be made to APIs. Somebody will need to. For example, Lookup.findSpecial probably needs to make adjustments to account for private members in nestmates, and to parallel the new verification/linkage rules (e.g., follow Lookup.findVirtual in restricting the receiver type). The spec. of findSpecial mentions the bytecode behavior of the invokespecial instruction. This might mean we only need a few non-normative comments in the javadoc. I can dream. The spec. of Lookup.in is more subtle. As I've said elsewhere, I would like to make a clean break from what we currently do, which is grub through the InnerClasses attributes to emulate nest-detection. We still have to do this for compatibility, but if either class has a non-trivial nest (MoN or NMs attr.) then Lookup.in will *only* consult that nest, and *ignore* InnerClasses. > Security risk > > The JEP text should acknowledge that, while this does allow compilers to grant finer-grained access to members shared by nestmates, it also pushes compilers to grant broader access to members that were previously kept private. It's a trade-off, and presumably a good one because nestmates are completely trusted, while package-mates might sometimes be suspect. > > (I guess this argument really ought to be made from the top: declaring things with package access is worse than inventing a new level of access because ________.) > Because packages are generally unsealed and can accept new members. The new thing about a nest is that its member list is fixed. That's more secure. And barring funny compilation tricks, the nestmates are all created at the same time, and presumably from a consistent configuration of source code. That's even more secure, since you can audit the nest as a whole in source. ? John From john.r.rose at oracle.com Wed Apr 19 07:51:06 2017 From: john.r.rose at oracle.com (John Rose) Date: Wed, 19 Apr 2017 00:51:06 -0700 Subject: Draft JVMS changes for Nestmates In-Reply-To: <90b555a2-1290-e81c-da0c-54aa9bbfdaa7@oracle.com> References: <0848391A-AF8C-4B4D-8CDA-FFC727E5CF6B@oracle.com> <1247948811.901298.1492546336021.JavaMail.zimbra@u-pem.fr> <25912A06-ABE8-4E83-8EC6-9A9CA6888B18@oracle.com> <90b555a2-1290-e81c-da0c-54aa9bbfdaa7@oracle.com> Message-ID: On Apr 19, 2017, at 12:14 AM, David Holmes wrote: > > Some of these are a lot more awkward to do during classfile parsing and will require symbol comparisons. I was wanting to avoid "deep validation" of NMs as it penalizes all the good code. Having a "bad" NM entry seems harmless as these entries are only used to validate the initial claim of nest membership. If an entry is "bad" then by definition it will not match with any claimee. I don't see how that is a significant concern. The symbol bodies are hot in cache at the point we would check prefixes, since they are already being scanned for other purposes, such as initial interning, and also syntax checking. Existing processing is exactly as deep (or shallow) as the checks I want. If we turn off the "verify" flag for class loading, then maybe we can buy something by dropping those (and all the other) constraint checks. By syntax checking, I mean that if I mention a CONSTANT_Class in the CP, and its corresponding CONSTANT_Utf8 has a broken syntax (e.g., "." instead of "/", or two "//" in a row) the JVMS mandates an error. But those are the same bytes I want to look at when they are referenced by a MoN or NMs attribute. It will all be in cache, and the cycles to do the checks will be undetectable. ? John From ali.ebrahimi1781 at gmail.com Wed Apr 19 08:58:24 2017 From: ali.ebrahimi1781 at gmail.com (Ali Ebrahimi) Date: Wed, 19 Apr 2017 13:28:24 +0430 Subject: Draft JVMS changes for Nestmates In-Reply-To: <6E328B19-85A5-44AB-A549-06E7BAD4D12C@oracle.com> References: <6E328B19-85A5-44AB-A549-06E7BAD4D12C@oracle.com> Message-ID: Hi all, On Wed, Apr 19, 2017 at 12:08 PM, John Rose wrote: > On Apr 18, 2017, at 11:42 AM, Dan Smith wrote: > > > > Hi, all. > > > > I've uploaded a draft of JVMS changes for JEP 181 "Align JVM Checks with > Java Language Rules for Nested Classes" to: > > http://cr.openjdk.java.net/~dlsmith/private-access.html > > Nice. More comments on that later. > > > Some comments below on my thinking in drafting the spec text, and on the > JEP generally. > > > > JEP name > > > > I don't know how permanent JEP names are supposed to be, but I'd prefer > a different name at this point. Something like: "Expanded JVM Access to > Private Members"?shorter, focused on the feature itself rather than its > relationship to the Java language. Or maybe "Class Nests for Access to > Private Members". > Or "JVM support for nested classes" OR "[Native] Nested classes in JVM" > I expect Lookup.defineClass to inject into the nest of the > Lookup.lookupClass, > if the lookup has private access. That's what we call hosting for VMACs, > almost exactly. (+1 to Brian's comments to Remi.) > > I would prefer "nest host" or "nest host class" or "host class of nest" to > plain "host class", > because it will be better if we mention "nest" when we mention the term > for a nest's host class. > Downside is pronunciation of "nest-host" sounds like a sneeze. > What about "nest root"? -- Best Regards, Ali Ebrahimi From brian.goetz at oracle.com Wed Apr 19 14:12:54 2017 From: brian.goetz at oracle.com (Brian Goetz) Date: Wed, 19 Apr 2017 10:12:54 -0400 Subject: Draft JVMS changes for Nestmates In-Reply-To: References: <0848391A-AF8C-4B4D-8CDA-FFC727E5CF6B@oracle.com> <1247948811.901298.1492546336021.JavaMail.zimbra@u-pem.fr> <25912A06-ABE8-4E83-8EC6-9A9CA6888B18@oracle.com> Message-ID: <3e91872b-741f-107f-c263-53e5cf75c80f@oracle.com> > Yep. Also, multiple classes can claim the same nest member class in > their NestMembers attributes. Not a problem as long as the > MemberOfNest attribute (if any) of the member class points to a host > class that claims it. At the risk of bikeshedding, I find the names NestMembers and MemberOfNest confusing; I keep having to think about the directionality, since they sound so similar. Would be good to find names that have obviously opposite directionality. A not-very-good suggestion (but which illustrates the directionality thing I'm going for) would be "NestDeclaration" and "NestUse". It's pretty clear which is which without thinking about it. Perhaps: NestHost and NestMember ? From david.holmes at oracle.com Wed Apr 19 07:14:55 2017 From: david.holmes at oracle.com (David Holmes) Date: Wed, 19 Apr 2017 17:14:55 +1000 Subject: Draft JVMS changes for Nestmates In-Reply-To: References: <0848391A-AF8C-4B4D-8CDA-FFC727E5CF6B@oracle.com> <1247948811.901298.1492546336021.JavaMail.zimbra@u-pem.fr> <25912A06-ABE8-4E83-8EC6-9A9CA6888B18@oracle.com> Message-ID: <90b555a2-1290-e81c-da0c-54aa9bbfdaa7@oracle.com> Hi John, Aside: my replies to the experts list are held up for moderation. On 19/04/2017 4:45 PM, John Rose wrote: > On Apr 18, 2017, at 1:49 PM, Dan Smith > wrote: >> >> >>> On Apr 18, 2017, at 2:12 PM, Remi Forax >> > wrote: >>> >>>>> Verification of MemberOfNest >>>>> >>>>> I include a discussion block about different options of validating >>>>> MemberOfNest. >>>>> I think the consensus, and my preference, is to do it during >>>>> verification of >>>>> the member class. (NestMembers, on the other hand, is never >>>>> validated, except >>>>> as a side-effect of checking MemberOfNest.) >>> >>> It means that a NestMembers can contains Class that are not yet/never >>> defined. >>> It's not a problem for me. >> >> Yep. Also, multiple classes can claim the same nest member class in >> their NestMembers attributes. Not a problem as long as the >> MemberOfNest attribute (if any) of the member class points to a host >> class that claims it. > > We need to specify more structural constraints on the new attributes. > If we make them more strongly normalizing, there will be fewer chances > for semantic bugs on unexpected inputs. Also, making them structural > constraints means we check them early, during class file loading. > > These are good, I think: > > - NMs must be non-empty (degenerate nest is never explicit) > - NMs may not contain duplicates (cf. treatment of ClassFile.interfaces) > - NMs may not contain the current class (i.e., an index to a class with > the same name as this_class) > - MoN may not contain the current class (ditto) > - MoN may contain only a package sibling (i.e., a referenced class name > must have the same package prefix as this_class) > - NMs may contain only package siblings (ditto) > - NMs and MoN may not refer to array classes (this is probably implied > by the package prefix checks) > > You already covered: > - a class may have exactly one or zero of NMs or MoN (no double dipping > of any kind) > - NMs and MoN may only have CONSTANT_Class attributes (implies no > nulls, well-formed names) > > David's prototype has the duplication check, and some of the other > checks happen later. I think they should all happen during class loading. Some of these are a lot more awkward to do during classfile parsing and will require symbol comparisons. I was wanting to avoid "deep validation" of NMs as it penalizes all the good code. Having a "bad" NM entry seems harmless as these entries are only used to validate the initial claim of nest membership. If an entry is "bad" then by definition it will not match with any claimee. David ----- > ? John From david.holmes at oracle.com Wed Apr 19 07:56:37 2017 From: david.holmes at oracle.com (David Holmes) Date: Wed, 19 Apr 2017 17:56:37 +1000 Subject: Draft JVMS changes for Nestmates In-Reply-To: References: <0848391A-AF8C-4B4D-8CDA-FFC727E5CF6B@oracle.com> <1247948811.901298.1492546336021.JavaMail.zimbra@u-pem.fr> <25912A06-ABE8-4E83-8EC6-9A9CA6888B18@oracle.com> <90b555a2-1290-e81c-da0c-54aa9bbfdaa7@oracle.com> Message-ID: On 19/04/2017 5:51 PM, John Rose wrote: > On Apr 19, 2017, at 12:14 AM, David Holmes > wrote: >> >> Some of these are a lot more awkward to do during classfile parsing >> and will require symbol comparisons. I was wanting to avoid "deep >> validation" of NMs as it penalizes all the good code. Having a "bad" >> NM entry seems harmless as these entries are only used to validate the >> initial claim of nest membership. If an entry is "bad" then by >> definition it will not match with any claimee. > > I don't see how that is a significant concern. The symbol bodies are > hot in cache > at the point we would check prefixes, since they are already being > scanned for other > purposes, such as initial interning, and also syntax checking. Existing > processing > is exactly as deep (or shallow) as the checks I want. I don't follow that. If I'm loading a nest-top class and validating its NM entries none of those entries need have been loaded yet and so none will be in the cache. David ----- > If we turn off the "verify" flag for class loading, then maybe we can > buy something > by dropping those (and all the other) constraint checks. > > By syntax checking, I mean that if I mention a CONSTANT_Class in the CP, > and its corresponding CONSTANT_Utf8 has a broken syntax (e.g., "." instead > of "/", or two "//" in a row) the JVMS mandates an error. But those are the > same bytes I want to look at when they are referenced by a MoN or NMs > attribute. It will all be in cache, and the cycles to do the checks will be > undetectable. > > ? John From daniel.smith at oracle.com Wed Apr 19 16:06:23 2017 From: daniel.smith at oracle.com (Dan Smith) Date: Wed, 19 Apr 2017 10:06:23 -0600 Subject: Draft JVMS changes for Nestmates In-Reply-To: References: <0848391A-AF8C-4B4D-8CDA-FFC727E5CF6B@oracle.com> <1247948811.901298.1492546336021.JavaMail.zimbra@u-pem.fr> <25912A06-ABE8-4E83-8EC6-9A9CA6888B18@oracle.com> Message-ID: > On Apr 19, 2017, at 12:45 AM, John Rose wrote: > > - NMs must be non-empty (degenerate nest is never explicit) > - NMs may not contain duplicates (cf. treatment of ClassFile.interfaces) > - NMs may not contain the current class (i.e., an index to a class with the same name as this_class) > - NMs may contain only package siblings (ditto) > - NMs and MoN may not refer to array classes (this is probably implied by the package prefix checks) These are all do-able, but I'm not sure they're consistent with the spirit of JVMS in its treatment of attributes. We don't usually assert that lists are non-empty, don't contain duplicates, etc. (Granted, most attributes are not relevant to JVM execution.) You mention `interfaces`, but I don't see any such assertion in 4.1. You can identify some package mismatches by looking at names, but not all (names with the same "package" name may be handled by different class loaders). I worry that making a partial effort here will give a false sense of security and lead someone in the future to see that as a bug and try to fully validate the package restriction. I hate array "class names". Ugh. But, again, we're not generally concerned with that kind of hygiene in attributes. > - MoN may not contain the current class (ditto) > - MoN may contain only a package sibling (i.e., a referenced class name must have the same package prefix as this_class) > - NMs and MoN may not refer to array classes (this is probably implied by the package prefix checks) If MoN names the current class, verification will fail (there's no matching NM attribute). If MoN names a class in a different package, verification will fail (the rule checks "samePackageName"). If MoN names an array type, verification will fail (either due to the "samePackageName" check or because there's no matching NM attribute). I don't see the benefit in making these "syntax" checks in addition to verification checks. (Even a class naming itself as its own superclass isn't a syntax error?it gets checked later.) > David's prototype has the duplication check, and some of the other checks happen later. I think they should all happen during class loading. Something you might like is if we moved the MoN verification check out of verification (which is mainly concerned with Code attributes anyway) and into a late step of the class loading process (5.3.5)? Something like: 1) Get the bits 2) Parse the class 3) Load & validate superclasses 4) Load & validate interfaces 5) Load & validate MemberOfNest ?Dan From daniel.smith at oracle.com Wed Apr 19 16:44:36 2017 From: daniel.smith at oracle.com (Dan Smith) Date: Wed, 19 Apr 2017 10:44:36 -0600 Subject: Draft JVMS changes for Nestmates In-Reply-To: <6E328B19-85A5-44AB-A549-06E7BAD4D12C@oracle.com> References: <6E328B19-85A5-44AB-A549-06E7BAD4D12C@oracle.com> Message-ID: > On Apr 19, 2017, at 1:38 AM, John Rose wrote: > > "Nest" also connotes a home-like place and therefore privacy. > (Could also have said "home" or "family"?those are also private spaces. Or "crib"?) Sigh. Now I will forever wish we had called this feature "cribs". https://www.youtube.com/watch?v=cAb54qbMF_k > Here's a corner case we need to cover: I have to load my supers before I can load myself. > If one (or more) of my supers are in the same nest as me, the checking of MoN must succeed > no matter who is the host class of the nest. > > This could get embarrassing if the host class is the subtype, if the super then needs to > recursively load the subtype (which is waiting on the supertype to load) before it can > validate its MoN attribute. If verification (linking) is started after all of the classes are > loaded, things are OK, since the MoN checks happen only after everything is loaded. Yes, I think the earliest we want to be loading MemberOfNest is after superclasses and superinterfaces are loaded (5.3.5 step 4). That avoids the cycle. > I was surprised to see invokespecial occasionally mentioned cases where it only works on > private members. Is that a current constraint, or a new one? > > I thought a class could invokespecial any of its methods, willy-nilly, and this permission > would seem to extend naturally to nestmates. (I think I'm forgetting something here.) The old constraint is that (non-) invokespecial can only reference the current class, a superclass, or a direct superinterface. The new constraint is that (non-) invokespecial must either reference a private method, or must reference the current class, a superclass, or a direct superinterface. The purpose of this check is to force outsiders to use virtual dispatch, rather than jumping into the middle of an override chain. But private methods can't be overridden, so an "outsider" from the same nest won't be defeating any restrictions. (We could grant all members of a nest the right to call invokespecial on all nest members' non-private members and supers, but that's a new feature. Would be similar to granting access to inherited protected members.) >> When invokespecial involves interface types, the verifier can't guarantee that receiver objects actually implement that interface (JDK-8134358). It's an open question whether this is really a problem, but I included a tentative fix in the invokespecial runtime rules. > > I am leaning towards putting this check in. If we put it in, it should go in everywhere, > including invokespecial of default methods, both in and out of the caller's nest. > I think your language might have allowed an untyped loophole for nestmates. > > My goal in this is simplicity of effect: There are no loopholes, no exceptions > to typing of the incoming receiver (L0) of a default method. If we are going to > check any of these invokespecial receivers, we must check them all. The verifier attempts to ensure that: - For non- invokespecial, the type of the objectref is a subtype of the referenced class/interface - For non-, non-private invokespecial, the type of the objectref is a subtype of the current class/interface The proposed rules repeat these two assertions, but with a dynamic check. Private methods are treated differently because they aren't subject to the same constraints. (It could simplify our lives considerably if we had a dedicated `invokeprivate` method, but alas. That's how you should think about it, anyway.) ?Dan From daniel.smith at oracle.com Wed Apr 19 16:47:44 2017 From: daniel.smith at oracle.com (Dan Smith) Date: Wed, 19 Apr 2017 10:47:44 -0600 Subject: Draft JVMS changes for Nestmates In-Reply-To: <3e91872b-741f-107f-c263-53e5cf75c80f@oracle.com> References: <0848391A-AF8C-4B4D-8CDA-FFC727E5CF6B@oracle.com> <1247948811.901298.1492546336021.JavaMail.zimbra@u-pem.fr> <25912A06-ABE8-4E83-8EC6-9A9CA6888B18@oracle.com> <3e91872b-741f-107f-c263-53e5cf75c80f@oracle.com> Message-ID: <27C41496-0782-4690-9242-A7660736AB0E@oracle.com> > On Apr 19, 2017, at 8:12 AM, Brian Goetz wrote: > >> Yep. Also, multiple classes can claim the same nest member class in their NestMembers attributes. Not a problem as long as the MemberOfNest attribute (if any) of the member class points to a host class that claims it. > > > At the risk of bikeshedding, I find the names NestMembers and MemberOfNest confusing; I keep having to think about the directionality, since they sound so similar. Would be good to find names that have obviously opposite directionality. > > A not-very-good suggestion (but which illustrates the directionality thing I'm going for) would be "NestDeclaration" and "NestUse". It's pretty clear which is which without thinking about it. > > Perhaps: NestHost and NestMember ? I also don't love that "NestMembers" seems more authoritative than it really is. (Doesn't get validated, doesn't suggest that other classes can be dynamically added.) Something like "AllowedInNest" would better convey the actual meaning of that attribute. ?Dan From john.r.rose at oracle.com Wed Apr 19 18:07:55 2017 From: john.r.rose at oracle.com (John Rose) Date: Wed, 19 Apr 2017 11:07:55 -0700 Subject: Draft JVMS changes for Nestmates In-Reply-To: <27C41496-0782-4690-9242-A7660736AB0E@oracle.com> References: <0848391A-AF8C-4B4D-8CDA-FFC727E5CF6B@oracle.com> <1247948811.901298.1492546336021.JavaMail.zimbra@u-pem.fr> <25912A06-ABE8-4E83-8EC6-9A9CA6888B18@oracle.com> <3e91872b-741f-107f-c263-53e5cf75c80f@oracle.com> <27C41496-0782-4690-9242-A7660736AB0E@oracle.com> Message-ID: <75B2F2FC-549C-4D00-8DA3-E0FC43A75653@oracle.com> On Apr 19, 2017, at 9:47 AM, Dan Smith wrote: > > Something like "AllowedInNest" would better convey the actual meaning of that attribute. It's true that this is what the attribute does, which is a plus. But it also is less directional than NMs. (Am I claiming I am allowed in some nest or am I granting nest privileges to others? Can't tell from the name.) To me at least the plural "s" makes it clear who is in charge of the list of "grantees". From daniel.smith at oracle.com Wed Apr 19 19:21:56 2017 From: daniel.smith at oracle.com (Dan Smith) Date: Wed, 19 Apr 2017 13:21:56 -0600 Subject: Draft JVMS changes for Nestmates In-Reply-To: References: Message-ID: <524BFE31-4A61-49B3-9AE9-01B154052F96@oracle.com> Thanks a lot for the feedback! My comments below: > On Apr 19, 2017, at 12:40 AM, David Holmes wrote: > >> JEP name >> >> I don't know how permanent JEP names are supposed to be, but I'd prefer a different name at this point. Something like: "Expanded JVM Access to Private Members"?shorter, focused on the feature itself rather than its relationship to the Java language. Or maybe "Class Nests for Access to Private Members". > > The intent of the name is to aid in the contextual positioning of this JEP ie address why do we even need this. The JEP addresses an inconsistency between access controls in the Java programming language and access controls in the JVM. The primary goal here is to align the two, while at the same time allowing for additional usage of nestmates in the future. Yeah, but I think explaining the motivation in the title makes for a long title and limits the imagination to one particular use case. >> The JEP uses "nest top" to describe the class that nest members reference; I prefer "host class", which better describes the class's role and isn't tied to the Java "top level class" concept. I know we use "host class" internally in Hotspot, perhaps when working with anonymous classes (of the JVM flavor), but I think in that context it will ultimately mean the same thing? Are we comfortable repurposing the term in this way? > > Given that a nest-top type must be a top-level type, I think nest-top fits perfectly. This is a good example of my point about the title: the JVM has no concept* of "top level type", and doesn't need one. We just need a specific class to act as the reference point by which a group of classes can talk about a nest. For example, I can imagine some language introducing a class file?foo.bar.Nest$$001?solely for the purpose of providing a hook for this reference point. javac's choice to designate a top-level class as the nest-top/nest-host is a compilation strategy. (*Yes, there's an InnerClasses attribute, and Signature attributes, and lots of other stuff to facilitate compilation in the Java language. But those represent auxiliary metadata, not core JVM concepts.) > I dislike "host class" because that is already used for the quite different case of VMACs. I can live with other names but I think they should incorporate "nest" eg. nest-holder, nest-host. But I think nest-top is an ideal match. (And no I didn't invent this terminology :) ). Okay. Probably best to sit on this for awhile and circle back to it, see how we're all feeling. >> I follow Brian's model when it comes to nest membership (5.4.4): every class belongs to a nest, possibly (in the absence of MemberOfNest) the nest hosted by itself. Many nests are singletons without any associated explicit attributes. > > I have not yet seen a follow up from John as to whether we require or just allow, an empty NestMembers attribute to indicate a singleton nest member. If in our model *every* class has a nest, then we definitely don't want to require every class to have a NestMembers attribute. In my spec (5.4.4), the presence of a NestMembers attribute is irrelevant. A class without a MemberOfNest attribute belongs to its own nest, and there's no need to validate anything about NestMembers. >> Verification of MemberOfNest >> >> I include a discussion block about different options of validating MemberOfNest. I think the consensus, and my preference, is to do it during verification of the member class. (NestMembers, on the other hand, is never validated, except as a side-effect of checking MemberOfNest.) > > Given verification is optional the current approach is to validate during linking just prior to the verification step. Whether that can conceptually be considered part of verification I am unsure. I think this is the source of a lot of anomalies?spec treats verification as an essential part of linking, while Hotspot treats it as an optional add-on. You should be okay, for some definition of "just prior". These are the constraints, as I read them: - "A class or interface is completely loaded before it is linked." (5.4) - Loading includes structural checks (CFE), version checks (UCVE), and validation and loading of superclasses and superinterfaces (5.3.5) - "Errors detected during linkage are thrown at a point in the program where some action is taken by the program that might, directly or indirectly, require linkage to the class or interface involved in the error." (5.4) That last one is pretty mysterious to me (what does "some action" mean? what about "indirectly"?), but seems to grant pretty wide latitude on the timing. >> Verification of invokespecial >> >> Allowing invokespecial to refer to classes other than the current class and its supers is a significant change. I noticed and relied heavily on the parallel with invokevirtual making protected method calls. So I tried to make the two as similar as possible. In a few places, the treatment of protected methods doesn't seem ideal, and rather than trying to mirror that with non-private invokespecial, I modified the protected method treatment. > > This concerns me because protected-access is a somewhat complex/messy issue so aligning private-access with it seems to go in the wrong direction. Cleaning up the protected access rules, while perhaps desirable, is technically out-of-scope for this JEP in my opinion. > >> The "protected check" of verification, in particular, was a mess before, and I think I've made it a lot more manageable, and compatible with a parallel rule for invokespecial. I could use some feedback on exactly what Hotspot's verifier has been doing here, though, since I'm pretty sure it didn't match the old specified rules. > > Might be better to take this up separately. Yeah, I hear you. But I worry about the technical debt if we just look the other way and specify/implement a fresh invokespecial check separately. The subtle inconsistencies between the two will lead to all sorts of bugs and tweaks in the future; the duplicate code/spec will be an ongoing maintenance burden. From the spec perspective, pulling on the thread looks like this: - The new verification we need on invokespecial looks almost the same as the verification we need on protected methods. - Okay, I'll re-use that rule. But it's a complicated mess, and has bugs, so I've got to clean it up first. - What exactly are these rules trying to achieve? I can see the hard constraints (in configuration X, an error occurs), but there's a soft requirement to minimize class loading. - To figure out when class loading is acceptable, I need to know what the reference implementation actually does. >> MethodHandle resolution >> >> The spec (5.4.3.5) is vague about what errors can occur during MethodHandle resolution. I assume any linkage error specified for the instruction can also occur via MethodHandle resolution, and that will include failures due to invokespecial improperly referencing a class. > > This is an area I have least understanding of, but this seems to cover it: > > "In each step, any exception that can be thrown as a result of failure of resolution of a class or interface or field or method reference can be thrown as a result of failure of method handle resolution." The ambiguity is in the definition of "resolution". The preferred interpretation, which everyone seems to confirm reflects reality, is that it includes all linkage errors specified for the referencing instruction. I.e., errors from both both 5.4.3 and 6.5 can occur. >> Dynamic checking of invokespecial receivers >> >> When invokespecial involves interface types, the verifier can't guarantee that receiver objects actually implement that interface (JDK-8134358). It's an open question whether this is really a problem, but I included a tentative fix in the invokespecial runtime rules. > > This seems out of scope in the general case, but there may be nestmate specific actions required. I touch it here because: - It's a dynamic companion to verification, and we've modified the verification rules - There's an existing rule that may or may not be intended to perform part of this dynamic check; that needs to be addressed somehow - Any rule we introduce is impacted by the new treatment of private methods, so easier to get it right while working on this feature - Easier to address small things like this when it's swapped in already If necessary, we can yank it and address the bug separately in the future, but it seems convenient to tackle it now. >> Compiler changes >> >> The JEP text can't seem to decide if compiler changes are part of it, or a follow-up exercise. I think we're best off explicitly including the compiler changes, which will provide opportunities for design validation and testing. > > Not sure what gives you that impression, but javac changes are an essential part of this. Lacking a compiler spec I assume this will be handled informally. But the attribute definitions (see comments below) should dictate what a source compiler is required to do. - Would be clearest if "Goals" mentioned modifying javac - Language like "javac can" suggests optional followup work; should have a paragraph saying "javac will" instead - Remove the paragraph saying "it is not strictly required to remove these immediately as part of this feature" >> Security risk >> >> The JEP text should acknowledge that, while this does allow compilers to grant finer-grained access to members shared by nestmates, it also pushes compilers to grant broader access to members that were previously kept private. It's a trade-off, and presumably a good one because nestmates are completely trusted, while package-mates might sometimes be suspect. > > I'm not following you here. What broader access is being granted to members that were previously kept private? This effort allows private-only access to members that were previously package-accessible. Old approach is to widen access to private members to the package level on a per-member basis. New approach is to widen access to private members to the nest level on a per-class basis. Nest is narrower than package (good!), but per-class is more permissive than per-member (bad). Some members that are compiled as totally private in JDK 9 will be incidentally shared with all nest members in . > 4.7 Attributes > > Regarding the Exceptions attribute, it seems to be used only to store data to be passed back to the libverify and the reflection API. I'm not familiar with libverify, but I think you're confirming that the attribute is on par with Signature? Something for tools and standard APIs to rely on, but irrelevant to execution? > Regarding the definitions of NestMembers and MemberOfNest ... I modelled the definitions in: > > https://bugs.openjdk.java.net/browse/JDK-8177020 > > based on those for the innerclass (and related) attributes. I'm not sure why you moved away from those definitions as they pinned down exactly when these attributes are expected to appear. Without those parts the current definitions seem optional - ie they define what the attribute means if present, but they do not require its presence. I do not believe these attributes should be optional for classfile version 54, but required whenever a nest does exist. A java source compiler writer should look at these definitions and know when they must generate these attributes. These sorts of assertions belong in the "Java Compiler Specification". (Yes, this document does not exist. Yes, that's kind of a problem, but we muddle through.) In an ideal world, JVMS would not acknowledge that the Java Programming Language exists. It should stand on its own, and Java is merely a client. As is, we have some history that has led to some dependencies on Java in spec, but these are generally auxiliary items: attributes that don't impact JVM behavior (4.7), a tutorial on how to compile Java code (3), occasional explanatory references to Java language concepts (e.g., 2.9). But it would be wrong for the specification of a "critical" attribute (4.7) to be defined in terms of Java language concepts. It should stand on its own, general enough to meet the needs of Java language compilers and others with different use cases. > 4.7.26 The MemberOfNest Attribute > > "the class is implicitly a member of its own nest" > > I suggest changing "a" to "the sole". Otherwise "a" implies there may be other members. As these are static properties being defined I don't think we need to be accounting for some future dynamic expansion of a nest. I intended to include both the cases of a singleton nest and the case of a class that has a NestMembers attribute. But I'll add a clarifying sentence. > 4.9.2 Structural Constraints > > The changes to invokespecial seem okay, but I have a problem with terminology. In: > > "If invokespecial is used to invoke a non-private, non- method, referenced via a superclass or a direct superinterface, ..." > > what does "referenced via" mean? I'll change to "where the referenced class is" > In: > > "If getfield or putfield is used to access a protected field referenced in a superclass ..." > > and following, you changed "declared" to read "referenced". Again I do not understand what this is supposed to mean. It is the declaration site of the field or method that is needed to determine whether that member is in the same or another package. To me "declared in a superclass ..." was exactly correct. Your note does not make the change any clearer to me. Good catch. It's important that the rule be limited to cases that reference a superclass*; but it's also important that the declaring class be used for the package check. (*Here's why: if I reference my subclass, and that resolves to a field declared in my superclass, the verifier will perform no check, because it may not know that the referenced class is a subclass.) Revised: "If `getfield` or `putfield` is used to access a `protected` field by reference to a superclass, and that field is declared in a different run-time package than the current class or interface, then the type of the class instance being accessed must be assignable to the current class or interface." > 4.10.1 Verification by Type Checking > > Does the order in the rules mandate the order in which the VM must actually check things? classHasValidNest seems much more complex than the actual runtime process of validation (though it may just be exposing distinct steps that are lumped together at runtime). Probably not, but dependencies between inputs/outputs do suggest an ordering. (We could probably get into an argument with a logician or a Prolog expert about whether there's such a thing as an "output" at all, and whether it would be acceptable for a JVM to load every class in the universe in order to test certain predicates, which is why it's probably not a great idea to specify the verifier using Prolog rules. Alas.) The important elements are: 1) Resolve the host class using the current class's loader 2) Check that the host class belongs to the same package 3) Check that the current class is named in the host's NestMembers 4) Resolve the current class's name in the host class's loader, and make sure it produces the same class > > --- > > 4.10.1.8 Type Checking for Restricted Member References > > I'm not very good at reading or understanding these rules, but I'm surprised that the invokespecial rules seem to make no mention of nestmates at all: > > "A method reference is allowed by an invokespecial instruction if it is not a restricted invokespecial reference, ..." > > This seems to open things up to all private method references?? There's a careful partitioning of error conditions that has to happen in order to avoid too much class loading. 4.10.1.8 is only concerned with ensuring, in narrow cases, that the type on the stack is assignable to the current class. For invokespecial, the narrow case is a method reference naming a superclass/direct superinterface and a method not named "", where the resolved method is not private. Separately, 5.4.4 ensures that resolving a private method will produce an IllegalAccessError if the method is not in the same nest. Also separately, 6.5 ensures that if the resolved method is not private, the referenced class is the current class, a superclass, or a direct superinterface. And also separately, 4.10.1.9 ensures that the type on the stack is always assignable to the referenced class. > 5.4.4 Access Control > > I don't think you need to restate: > > "A class with a MemberOfNest attribute belongs to the nest hosted by the referenced host class. > > A class without a MemberOfNest attribute implicitly belongs to a nest hosted by the class itself. (If the class also lacks a NestMembers attribute, then the nest has only one member.)" > > The rule is simply stated as is: "belonging to the same nest as D" - belonging to the same nest is, or should be, already defined elsewhere. My intent is that this *is* the definition. But sounds like you're expecting that definition to be in 4.7.25/4.7.26, which is probably more intuitive. I'll change that. > I'm avoiding commenting on the protected access changes but do want to raise a concern that there are no changes to Method resolution (5.4.3.3) yet resolution relies on the definition of access control to prune/discard inaccessible methods. It appears now that we're allowing more potential successful resolutions, then using the additional rules on the invoke* bytecodes to try and discard any undesirable results. Yeah, this motivated my question about MethodHandle resolution: it seems that when we talk about "resolution", we really mean the process defined in 5.4.3, followed by any linkage checks defined in 6.5. If so, this is just a presentational reshuffling (with the minor exceptions I noted in the document). Why bother? Because if the check on the referenced class of a protected invokevirtual/getfield/putfield is considered an access check (5.4.3.3), then the check on the referenced class of a non-private invokespecial probably ought to be considered an access check, too, but that leads to a concept of "accessible" that is really unwieldy and context-dependent. > 6.5 getfield (and others) > > You note "These rules are redundant: verification already guarantees them " but this brings me back to an area I keep raising concerns about: the split between static verification based checks and dynamic runtime checks. Yes the verifier precludes certain things but if we run without verification the rules expressed for the bytecodes are considered to be still required at runtime. If you delete them because they overlap with verifier rules we have no way to tell which rules must be enforced regardless of verification status. I propose removing them because on their face they appear completely redundant, an artifact of a time when we didn't have clear definitions of loading, linking, verification, etc. But if we decide it really is helpful to keep some or all of these assertions, that's fine. It seems to me that attempting to interpret unverified bytecodes is an implementation-specific feature that JVMS knows nothing about. Deciding which rules must be enforced in the absence of verification is an important part of designing that feature, and in an ideal world some comprehensive documentation about which verification assertions you've chosen to enforce dynamically would be maintained somewhere, but not in JVMS. If we want JVMS to maintain that list, then running without verification should be a first-class fully-specified feature. The status quo, I'm guessing, is that a few dynamic checks are sprinkled throughout JVMS, but plenty of other checks are performed without any supporting spec text. Which isn't a great place to be in. > 6.5 invokespecial > > You note regarding the IllegalAccessError: > > "This replaces a VerifyError previously specified by 4.9.2. The check must be delayed until after resolution in order to determine whether the referenced method is private." > > We know the access flags for the referenced method at verification time - shouldn't that be the sole basis for the verifier check? Afterall it is verifying the static properties of the classfile and bytecode. If the actual resolved method has different access to the referenced method then that may lead to ICCE (depending on the exact nature of the change - a private method made public may not be an issue for example). Example: invokespecial SomeOtherClass.foo(I)V Verification relies on the descriptor to tell it that this method expects an int and returns nothing. Those kinds of checks can be performed locally. The descriptor doesn't tell you if the method is private. Only way to find out is to resolve SomeOtherClass, then resolve "foo(I)V". If we wanted to do the check at verification time, we'd have to load every class named by an invokespecial instruction. (I think you might be conflating declaration-site metadata, which we have easy access to, with use-site metadata, which requires resolving references.) > 5.4.3.3 Method resolution > > You did not make any changes here, but as per my comment in the bug report we do require, IMHO, further tightening here to ensure nestmate invokespecial invocations do not resolve to method implementations that they should not. The example here is a hierarchy of nestmates (C extends B extends A) where A and C both declare a private method "void m()" and we have an invocation c.m() where c is an instance of class C. We then "separately compile" C to remove the definition of m(). At runtime method resolution will locate A.m, even though the method reference was for C.m. Normally the accessibility rules would exclude A.m from being a viable candidate but here the classes are nestmates so A.m is accessible. But it would be wrong to invoke A.m. This case should throw ICCE or NSME. You're taking issue with the fact that method resolution will match private methods in superclasses, when our intuition (and the Java language model) is that private methods are not inherited. I agree: https://bugs.openjdk.java.net/browse/JDK-8021581 Is this the right venue to address that issue? As demonstrated above, I'm happy to lump in tangentially-related bug fixes. :-) You could argue that this situation isn't so different from the status quo, but it does seem unique that you can delete 1 private method from a consistently-compiled program and then get a silent behavioral change. In practice, javac won't separately compile nestmates, so this would be hard to reproduce. More generally, the feature is not designed for separate compilation, so real-world scenarios may be hard to come by, even in other languages. Anyway, I'm game to take another look at JDK-8021581 if we want. ?Dan From brian.goetz at oracle.com Wed Apr 19 19:24:54 2017 From: brian.goetz at oracle.com (Brian Goetz) Date: Wed, 19 Apr 2017 15:24:54 -0400 Subject: Draft JVMS changes for Nestmates In-Reply-To: <524BFE31-4A61-49B3-9AE9-01B154052F96@oracle.com> References: <524BFE31-4A61-49B3-9AE9-01B154052F96@oracle.com> Message-ID: <1a5d5db3-3827-b24c-dde0-688f43ca10f1@oracle.com> Nest host Nest master Nest top Nest owner Nest dictator-for-life ... On 4/19/2017 3:21 PM, Dan Smith wrote: >> I dislike "host class" because that is already used for the quite different case of VMACs. I can live with other names but I think they should incorporate "nest" eg. nest-holder, nest-host. But I think nest-top is an ideal match. (And no I didn't invent this terminology:) ). > Okay. Probably best to sit on this for awhile and circle back to it, see how we're all feeling. > From ali.ebrahimi1781 at gmail.com Wed Apr 19 19:40:15 2017 From: ali.ebrahimi1781 at gmail.com (Ali Ebrahimi) Date: Thu, 20 Apr 2017 00:10:15 +0430 Subject: Draft JVMS changes for Nestmates In-Reply-To: <1a5d5db3-3827-b24c-dde0-688f43ca10f1@oracle.com> References: <524BFE31-4A61-49B3-9AE9-01B154052F96@oracle.com> <1a5d5db3-3827-b24c-dde0-688f43ca10f1@oracle.com> Message-ID: Hi, On Wed, Apr 19, 2017 at 11:54 PM, Brian Goetz wrote: > Nest host > Nest master > Nest top > Nest owner > Nest dictator-for-life > add to list : Nest root -- Best Regards, Ali Ebrahimi From david.holmes at oracle.com Wed Apr 19 21:10:42 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 20 Apr 2017 07:10:42 +1000 Subject: Draft JVMS changes for Nestmates In-Reply-To: <3e91872b-741f-107f-c263-53e5cf75c80f@oracle.com> References: <0848391A-AF8C-4B4D-8CDA-FFC727E5CF6B@oracle.com> <1247948811.901298.1492546336021.JavaMail.zimbra@u-pem.fr> <25912A06-ABE8-4E83-8EC6-9A9CA6888B18@oracle.com> <3e91872b-741f-107f-c263-53e5cf75c80f@oracle.com> Message-ID: On 20/04/2017 12:12 AM, Brian Goetz wrote: >> Yep. Also, multiple classes can claim the same nest member class in >> their NestMembers attributes. Not a problem as long as the >> MemberOfNest attribute (if any) of the member class points to a host >> class that claims it. > > > At the risk of bikeshedding, I find the names NestMembers and > MemberOfNest confusing; I keep having to think about the directionality, > since they sound so similar. Would be good to find names that have > obviously opposite directionality. I find these names very good - granted I've been working with them more than anyone else (but didn't define them). The plurality makes it clear to me. David > A not-very-good suggestion (but which illustrates the directionality > thing I'm going for) would be "NestDeclaration" and "NestUse". It's > pretty clear which is which without thinking about it. > > Perhaps: NestHost and NestMember ? > > From daniel.smith at oracle.com Wed Apr 19 21:58:34 2017 From: daniel.smith at oracle.com (Dan Smith) Date: Wed, 19 Apr 2017 15:58:34 -0600 Subject: Draft JVMS changes for Nestmates In-Reply-To: <524BFE31-4A61-49B3-9AE9-01B154052F96@oracle.com> References: <524BFE31-4A61-49B3-9AE9-01B154052F96@oracle.com> Message-ID: <48B779AB-6D9A-464B-B78A-AAFE5B9D8A5C@oracle.com> > On Apr 19, 2017, at 1:21 PM, Dan Smith wrote: > >> >> 5.4.4 Access Control >> >> I don't think you need to restate: >> >> "A class with a MemberOfNest attribute belongs to the nest hosted by the referenced host class. >> >> A class without a MemberOfNest attribute implicitly belongs to a nest hosted by the class itself. (If the class also lacks a NestMembers attribute, then the nest has only one member.)" >> >> The rule is simply stated as is: "belonging to the same nest as D" - belonging to the same nest is, or should be, already defined elsewhere. > > My intent is that this *is* the definition. But sounds like you're expecting that definition to be in 4.7.25/4.7.26, which is probably more intuitive. I'll change that. Okay, here's another attempt at defining the attributes and the "nest" concept. I've been uneasy about `NestMembers` being presented as the "declaration" of the nest, since it's not validated. Instead, I see `MemberOfNest` as really where nests are defined (as a cumulative result of many class declarations?like packages), and `NestMembers` as just a help for validation. With that in mind, `MemberOfNest` ought to come first: ~~~~ **4.7.25 The `MemberOfNest` Attribute** --------------------------------------- **The `MemberOfNest` attribute is a fixed-length attribute in the `attributes` table of a `ClassFile` structure ([4.1]).** **A _nest_ is a set of classes and interfaces that share access to their `private` members ([5.4.4]).** **A class or interface with a `MemberOfNest` attribute belongs to the nest hosted by a designated _host class_. The host class authorizes membership in the nest with a corresponding entry in its `NestMembers` attribute ([4.7.26], [4.10]).** **A class or interface without a `MemberOfNest` attribute belongs to the nest hosted by itself. (Often, this nest is a singleton consisting only of the class itself.)** **There may be at most one `MemberOfNest` attribute in the `attributes` table of a `ClassFile` structure.** **The `MemberOfNest` attribute has the following format:** ... **4.7.26 The `NestMembers` Attribute** -------------------------------------- **The `NestMembers` attribute is a variable-length attribute in the `attributes` table of a `ClassFile` structure ([4.1]). It authorizes an enumerated set classes and interfaces to claim membership in a nest hosted by the current class or interface.** **There may be at most one `NestMembers` attribute in the `attributes` table of a `ClassFile` structure.** **The `attributes` table of a `ClassFile` structure must not contain both a `MemberOfNest` attribute and a `NestMembers` attribute.** > **This rule prevents a host class from claiming membership in a different nest. It is implicitly a member of the nest that it hosts.** **The `NestMembers` attribute has the following format:** ... 5.4.4 Access Control -------------------- A class or interface _C_ is _accessible_ to a class or interface _D_ if and only if either of the following is true: ... - _R_ is `private` and is declared ~~in _D_~~ **by a class belonging to the same nest as _D_ ([4.7.25])**. ~~~~ And now to the bikeshedding game: `MemberOfNest`, as the key thing, could just be called `Nest` (imagine a `Package` attribute?we wouldn't call it `MemberOfPackage`). Or, since it's a reference to a class, `NestHost ` / `NestTop` / `NestMother` / `NestWhatever`. `NestMembers` isn't horrible, but we could highlight its validation role with something like `AuthorizedNestMembers`. ?Dan From daniel.smith at oracle.com Wed Apr 19 22:58:20 2017 From: daniel.smith at oracle.com (Dan Smith) Date: Wed, 19 Apr 2017 16:58:20 -0600 Subject: Draft JVMS changes for Nestmates In-Reply-To: References: Message-ID: One other topic to add to this thread, which I forgot to mention: compatibility. Sometimes JVMS changes need to be guarded by rules that preserve old behavior for old class file version numbers, in order to prevent changes in behavior when an old class is run on a new JVM. My hope is that there's no need for such logic here. Here's my analysis. Let me know if you can think of any other cases, or if you feel that any of these changes are concerning. --- - The MemberOfNest and NestMembers attributes are illegal prior to 54.0. So well-formed older classes always belong to singleton nests?they can't claim membership in a different nest, and can't act as hosts. - Access control for private members declared in or called from old class files then degenerates to the old rule: the private member must be declared by the calling class. - The verification-time checks on the referenced class of an invokespecial have been moved to resolution time. Some cases: - Referencing a public method in a disallowed class used to be a VerifyError*; now it's a IAE at resolution time - Referencing a private method in a disallowed class used to be a VerifyError; now it's an IAE at resolution time - Referencing a private method in a subclass that resolves to a member of the current class used to be a VerifyError; now it's allowed. - Referencing a nonexistent method in a disallowed class used to be a VerifyError; now it's a NSME at resolution time - Referencing a static method in a disallowed class used to be a VerifyError; now it's a ICCE at resolution time (*Confirmed with some testing, although I found that referencing an indirect superinterface caused an ICCE, apparently at resolution time.) - The verification-time restriction on the target reference stack type of an invokespecial is only enforced if the referenced class is a superclass and the resolved method is non-private. Only change here (besides those outlined above due to a reference to a non-superclass) is that a private method referenced in a superclass that used to get a VerifyError (due to a disallowed stack type) will now get an IAE at resolution time. - Verification is more eager to resolve classes and methods referenced by `invokespecial`. The change in class loading only affects programs that would have had VerifyErrors before (for a reference to a disallowed class?superclasses are already loaded). The change in method resolution may be visible, depending on how the implementation chooses to handle errors. But flexibility for these sorts of changes is granted by 5.4. - As described in the comment in 5.4.4, tweaking the applicability of the `protected` access referenced class restriction causes some IAEs to become ICCEs or NSMEs. - The runtime checks added to `invokespecial` introduce some new ICCEs when the current class or referenced class are interfaces. Summary: almost all of these are simply changes in the error produced, and perhaps in the timing of errors being reported, depending on when resolution happens. There's one case where the new rules for private invokespecial allow something that was an error before (fine). There's also one case where new errors are introduced, but that would be a deliberate choice in response to a bug report. ?Dan From john.r.rose at oracle.com Wed Apr 19 23:30:36 2017 From: john.r.rose at oracle.com (John Rose) Date: Wed, 19 Apr 2017 16:30:36 -0700 Subject: Draft JVMS changes for Nestmates In-Reply-To: References: <0848391A-AF8C-4B4D-8CDA-FFC727E5CF6B@oracle.com> <1247948811.901298.1492546336021.JavaMail.zimbra@u-pem.fr> <25912A06-ABE8-4E83-8EC6-9A9CA6888B18@oracle.com> <3e91872b-741f-107f-c263-53e5cf75c80f@oracle.com> Message-ID: <92268D36-B380-4027-B985-5DCF26412A2C@oracle.com> On Apr 19, 2017, at 2:10 PM, David Holmes wrote: > > I find these names very good - granted I've been working with them more than anyone else (but didn't define them). The plurality makes it clear to me. +1 for plurality; that does it for me too, although it is a one-letter hint. I'm happy with the names, though I have a little problem (that Dan is trying to fix) with making clear the distinction between a Nest and the Host (or Top or Root) of a Nest. Things get messy when conflate them, which terminology can help avoid. ? John P.S. Can't resist, so OK, in the spirit of "crib", how about throwing a private "party"? Background: The asymmetry of the relation is essential to the design of the attributes. But it does not come from one class being somehow semantically or syntactically distinguished in any way, other than one: The special "nest host" (or "nest top") is responsible for inviting any number of classes (including zero) to join its nest. (The nest host is like the admin of a skype conversation, in which all conversants are otherwise equal. And like skype conversations, the group has only one purpose. For nestmates, the purpose is to broaden the definition of ACC_PRIVATE.) Under the theory that the nest host invites the other nestmates to the party, we could emphasize that role in the attribute names: NestMembers => NestInvitations (=> NestGrants since it is checked reactively) MemberOfNest => RequestNest (=> Nest, or NestHost?requesting is a given when we do access control) (Hmm? NMs => PartyInvitations, PartyRoster; MoN => RSVPToParty, JoinParty.) So, at this point, the best I change I can suggest to strengthen the asymmetry, but keep the nicest metaphor (nest) is NestGrants / Nest or NestHost. Why isn't "Nest" ambiguous? Doesn't it define a nest? Well, no, if you look at the way attributes are named in the JVMS, they refer to an "attribute" (duh) of the containing object. Thus, if ClassFile has a Nest attribute, it means that the ClassFile *has* an attribute of type Nest, not that the ClassFile is defining another entity of type Nest *outside* of the ClassFile. So I claim that "Nest" is perfectly unambiguous: It means that the ClassFile is asserting the identity of its Nest, using the convention of naming the nest-host. Choosing the more specific term "NestHost" would make it more clear that the Nest is being named by mentioning its host. The word "host" is a good fit (better than top or root) if we are emphasizing the fact that the host's job is to grant entry into the nest. Although reusing "host" is a feature rather than a bug, it could also be NestSecretary, NestDoorman, NestMaitreDeHotel, NestBouncer, etc. OK, I'm stopping. From john.r.rose at oracle.com Thu Apr 20 00:09:44 2017 From: john.r.rose at oracle.com (John Rose) Date: Wed, 19 Apr 2017 17:09:44 -0700 Subject: Draft JVMS changes for Nestmates In-Reply-To: References: <0848391A-AF8C-4B4D-8CDA-FFC727E5CF6B@oracle.com> <1247948811.901298.1492546336021.JavaMail.zimbra@u-pem.fr> <25912A06-ABE8-4E83-8EC6-9A9CA6888B18@oracle.com> <90b555a2-1290-e81c-da0c-54aa9bbfdaa7@oracle.com> Message-ID: <8B895CBF-76E1-466D-AC72-7B23F5350C65@oracle.com> On Apr 19, 2017, at 12:56 AM, David Holmes wrote: > >> I don't see how that is a significant concern. The symbol bodies are >> hot in cache >> at the point we would check prefixes, since they are already being >> scanned for other >> purposes, such as initial interning, and also syntax checking. Existing >> processing >> is exactly as deep (or shallow) as the checks I want. > > I don't follow that. If I'm loading a nest-top class and validating its NM entries none of those entries need have been loaded yet and so none will be in the cache. I see what the problem is: The only checking I am proposing is syntax checking on the names. If by "deep checking" you mean loading classes from the names, I agree, we don't want to do that. The MemberOfNest attribute is passive. It does not need to initiate loads; it just needs to validate claims of nest membership by other classes. (I even doubt Dan's assertion that in some corner cases, when the nest-host is trying out a private access, there must be some loading done to validate the access. Probably we can get away with no loading at all, just name checks against already-loaded classes.) ? John From john.r.rose at oracle.com Thu Apr 20 00:16:44 2017 From: john.r.rose at oracle.com (John Rose) Date: Wed, 19 Apr 2017 17:16:44 -0700 Subject: Draft JVMS changes for Nestmates In-Reply-To: References: Message-ID: On Apr 18, 2017, at 11:40 PM, David Holmes wrote: > > I have not yet seen a follow up from John as to whether we require or just allow, an empty NestMembers attribute to indicate a singleton nest member. I think we should neither require nor allow an empty NestMembers attribute. Dan may prefer to minimize syntax checking, in which case an empty NestMembers attribute would be allowed. Nobody has proposed a reason to require it. A nest of one member is (depending on your POV) either the overwhelmingly common trivial case, or a useless degree of freedom. Taking the POV (from Brian and me) hat every class has a nest (just as every class has a package), an empty NestMembers attribute is just a waste of bits stating the obvious, about a stand-alone class. (I'd prefer to disallow the empty NMs attribute, lest it accidentally acquire some additional meaning. But as long as we specify that an empty one conveys the same condition as no attributes at all, we are safe from accident.) ? John From john.r.rose at oracle.com Thu Apr 20 00:32:06 2017 From: john.r.rose at oracle.com (John Rose) Date: Wed, 19 Apr 2017 17:32:06 -0700 Subject: Draft JVMS changes for Nestmates In-Reply-To: References: Message-ID: <73906CB5-80CF-42B4-B7C8-3F55FE1425F9@oracle.com> On Apr 18, 2017, at 11:40 PM, David Holmes wrote: > > 4.7.26 The MemberOfNest Attribute > > "the class is implicitly a member of its own nest" > > I suggest changing "a" to "the sole". Otherwise "a" implies there may be other members. As these are static properties being defined I don't think we need to be accounting for some future dynamic expansion of a nest. I think you're missing the point of the spec. logic here. If there is a non-empty NestMembers attribute, then in fact there *are* other members. Therefore "a" is correct, and "the sole" would be incorrect. Separately, we do need to leave wiggle room for dynamic injection. Dan, I think that would be done by allowing a dynamic injection operation to implicitly extend the NestMembers list to make a "ticket" for the injected class, and implicitly add (if necessary) a MemberOfNest to the injected class. ... > > 4.10.1 Verification by Type Checking > > Does the order in the rules mandate the order in which the VM must actually check things? classHasValidNest seems much more complex than the actual runtime process of validation (though it may just be exposing distinct steps that are lumped together at runtime). Agree. I think this logic can be simplified. (Will take that to a separate sub-thread.) > --- > > 4.10.1.8 Type Checking for Restricted Member References > > I'm not very good at reading or understanding these rules, but I'm surprised that the invokespecial rules seem to make no mention of nestmates at all: > > "A method reference is allowed by an invokespecial instruction if it is not a restricted invokespecial reference, ..." > > This seems to open things up to all private method references?? Well, the whole point of nestmates is to extend, in a regular way, all access rules that pertain to self-access and privacy. If the extension is truly regular, it is not surprising that the spec. changes are subtle like this. We're not trying to do an ad hoc patch on some set of particular behaviors, but rather extend pre-existing notions so they apply in more conditions, including invokespecial restrictions. > > --- > > 5.4.4 Access Control > > I don't think you need to restate: > > "A class with a MemberOfNest attribute belongs to the nest hosted by the referenced host class. > > A class without a MemberOfNest attribute implicitly belongs to a nest hosted by the class itself. (If the class also lacks a NestMembers attribute, then the nest has only one member.)" > > The rule is simply stated as is: "belonging to the same nest as D" - belonging to the same nest is, or should be, already defined elsewhere. Actually, there is no "elsewhere"; this is where the condition is defined. We don't need a special section for "nests"; we don't have a special section for "packages" either. So that part reads fine to me. > I'm avoiding commenting on the protected access changes but do want to raise a concern that there are no changes to Method resolution (5.4.3.3) yet resolution relies on the definition of access control to prune/discard inaccessible methods. It appears now that we're allowing more potential successful resolutions, then using the additional rules on the invoke* bytecodes to try and discard any undesirable results. > > --- > > 6.5 getfield (and others) > > You note "These rules are redundant: verification already guarantees them " but this brings me back to an area I keep raising concerns about: the split between static verification based checks and dynamic runtime checks. Yes the verifier precludes certain things but if we run without verification the rules expressed for the bytecodes are considered to be still required at runtime. If you delete them because they overlap with verifier rules we have no way to tell which rules must be enforced regardless of verification status. > > --- > > 6.5 invokespecial > > You note regarding the IllegalAccessError: > > "This replaces a VerifyError previously specified by 4.9.2. The check must be delayed until after resolution in order to determine whether the referenced method is private." > > We know the access flags for the referenced method at verification time - shouldn't that be the sole basis for the verifier check? Afterall it is verifying the static properties of the classfile and bytecode. If the actual resolved method has different access to the referenced method then that may lead to ICCE (depending on the exact nature of the change - a private method made public may not be an issue for example). > > --- > > 5.4.3.3 Method resolution > > You did not make any changes here, but as per my comment in the bug report we do require, IMHO, further tightening here to ensure nestmate invokespecial invocations do not resolve to method implementations that they should not. The example here is a hierarchy of nestmates (C extends B extends A) where A and C both declare a private method "void m()" and we have an invocation c.m() where c is an instance of class C. We then "separately compile" C to remove the definition of m(). At runtime method resolution will locate A.m, even though the method reference was for C.m. Normally the accessibility rules would exclude A.m from being a viable candidate but here the classes are nestmates so A.m is accessible. But it would be wrong to invoke A.m. This case should throw ICCE or NSME. > > Thanks, > David > ----- From david.holmes at oracle.com Thu Apr 20 01:11:40 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 20 Apr 2017 11:11:40 +1000 Subject: Draft JVMS changes for Nestmates In-Reply-To: References: <0848391A-AF8C-4B4D-8CDA-FFC727E5CF6B@oracle.com> <1247948811.901298.1492546336021.JavaMail.zimbra@u-pem.fr> <25912A06-ABE8-4E83-8EC6-9A9CA6888B18@oracle.com> <90b555a2-1290-e81c-da0c-54aa9bbfdaa7@oracle.com> Message-ID: <52dd13a7-b57f-7397-7319-f05cb12ddc1e@oracle.com> Correction below ... On 19/04/2017 5:56 PM, David Holmes wrote: > On 19/04/2017 5:51 PM, John Rose wrote: >> On Apr 19, 2017, at 12:14 AM, David Holmes > > wrote: >>> >>> Some of these are a lot more awkward to do during classfile parsing >>> and will require symbol comparisons. I was wanting to avoid "deep >>> validation" of NMs as it penalizes all the good code. Having a "bad" >>> NM entry seems harmless as these entries are only used to validate the >>> initial claim of nest membership. If an entry is "bad" then by >>> definition it will not match with any claimee. >> >> I don't see how that is a significant concern. The symbol bodies are >> hot in cache >> at the point we would check prefixes, since they are already being >> scanned for other >> purposes, such as initial interning, and also syntax checking. Existing >> processing >> is exactly as deep (or shallow) as the checks I want. > > I don't follow that. If I'm loading a nest-top class and validating its > NM entries none of those entries need have been loaded yet and so none > will be in the cache. Sorry John - brain fart on my part. We're talking about the symbols that have just been parsed and loaded into the constant pool, not the classes those symbols name. David ----- > David > ----- > >> If we turn off the "verify" flag for class loading, then maybe we can >> buy something >> by dropping those (and all the other) constraint checks. >> >> By syntax checking, I mean that if I mention a CONSTANT_Class in the CP, >> and its corresponding CONSTANT_Utf8 has a broken syntax (e.g., "." >> instead >> of "/", or two "//" in a row) the JVMS mandates an error. But those >> are the >> same bytes I want to look at when they are referenced by a MoN or NMs >> attribute. It will all be in cache, and the cycles to do the checks >> will be >> undetectable. >> >> ? John From david.holmes at oracle.com Thu Apr 20 01:31:51 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 20 Apr 2017 11:31:51 +1000 Subject: Draft JVMS changes for Nestmates In-Reply-To: References: <0848391A-AF8C-4B4D-8CDA-FFC727E5CF6B@oracle.com> <1247948811.901298.1492546336021.JavaMail.zimbra@u-pem.fr> <25912A06-ABE8-4E83-8EC6-9A9CA6888B18@oracle.com> Message-ID: <190b389f-d63c-4e01-e7b5-9269ce5ddc8c@oracle.com> Hi Dan, On 20/04/2017 2:06 AM, Dan Smith wrote: >> On Apr 19, 2017, at 12:45 AM, John Rose > > wrote: >> >> - NMs must be non-empty (degenerate nest is never explicit) >> - NMs may not contain duplicates (cf. treatment of ClassFile.interfaces) >> - NMs may not contain the current class (i.e., an index to a class >> with the same name as this_class) >> - NMs may contain only package siblings (ditto) >> - NMs and MoN may not refer to array classes (this is probably >> implied by the package prefix checks) > > These are all do-able, but I'm not sure they're consistent with the > spirit of JVMS in its treatment of attributes. We don't usually assert > that lists are non-empty, don't contain duplicates, etc. (Granted, most > attributes are not relevant to JVM execution.) You mention `interfaces`, > but I don't see any such assertion in 4.1. The NestMembers duplicate check exists because I copied the spec and code from the InnerClasses attribute. 4.7.6 states the classes[] "must have exactly one corresponding entry ..." - hence a check for duplicates is done. I used the same language when I defined NestMembers. So the precedent does exist. > You can identify some package mismatches by looking at names, but not > all (names with the same "package" name may be handled by different > class loaders). I worry that making a partial effort here will give a > false sense of security and lead someone in the future to see that as a > bug and try to fully validate the package restriction. I concur. We need the actual package check at nest-top validation time anyway, so checking this at parsing time seems pure overhead. > I hate array "class names". Ugh. But, again, we're not generally > concerned with that kind of hygiene in attributes. > >> - MoN may not contain the current class (ditto) >> - MoN may contain only a package sibling (i.e., a referenced class >> name must have the same package prefix as this_class) >> - NMs and MoN may not refer to array classes (this is probably >> implied by the package prefix checks) > > If MoN names the current class, verification will fail (there's no > matching NM attribute). > > If MoN names a class in a different package, verification will fail (the > rule checks "samePackageName"). > > If MoN names an array type, verification will fail (either due to the > "samePackageName" check or because there's no matching NM attribute). > > I don't see the benefit in making these "syntax" checks in addition to > verification checks. (Even a class naming itself as its own superclass > isn't a syntax error?it gets checked later.) > >> David's prototype has the duplication check, and some of the other >> checks happen later. I think they should all happen during class loading. > > Something you might like is if we moved the MoN verification check out > of verification (which is mainly concerned with Code attributes anyway) > and into a late step of the class loading process (5.3.5)? Something like: > > 1) Get the bits > 2) Parse the class > 3) Load & validate superclasses > 4) Load & validate interfaces > 5) Load & validate MemberOfNest At present I do (5) during "linking" just prior to verification (in case verification is disabled). It happens after the superclass and superinterfaces are themselves linked. But note that nest-top validation only requires the nest-top to be loaded, not linked. David ----- > ?Dan From john.r.rose at oracle.com Thu Apr 20 02:42:23 2017 From: john.r.rose at oracle.com (John Rose) Date: Wed, 19 Apr 2017 19:42:23 -0700 Subject: Nestmate same-package check In-Reply-To: References: Message-ID: <162B4430-FAFC-44DE-ABA4-941B423FEC37@oracle.com> On Apr 18, 2017, at 11:42 AM, Dan Smith wrote: > I've uploaded a draft of JVMS changes for JEP 181 "Align JVM Checks with Java Language Rules for Nested Classes" to: > http://cr.openjdk.java.net/~dlsmith/private-access.html > classHasValidNest(Class) :- > > classMemberOfNestName(Class, HostName), > classDefiningLoader(Class, L1), > loadedClass(HostName, L1, HostClass), > samePackageName(Class, HostClass), > classClassName(Class, Name), > classNestMemberNames(HostClass, MemberNames), > member(Name, MemberNames), > classDefiningLoader(HostClass, L2), > loadedClass(Name, L2, Class). In English: When validating nestmate attributes, if a class C with name "C" has an attribute MemberOfNest("H"), it is valid only if the following are all true: 1. The string "H" resolves from C's loader (L1) as H. 2. The strings "C" and "H" have the same package prefix. 3. H contains an attribute of the form NestMembers(?"C"?). 4. The string "C" resolves from H's loader (L2) as C. Steps 1 and 4 are both CONSTANT_Class resolution operations. Step 3 is a simple search of a symbol list, which is easy. This logic can be simplified by testing that C and H must be in the same runtime package. In that case, L1 and L2 are identical, and step 4 becomes: 4'. C and H have the same defining loader. Then we can refactor 2 and 4' into a more direct check on runtime package (using the functor 'sameRuntimePackage'): 1. The string "H" resolves from C's loader (L1) as H. 2'. C and H have the same runtime package. 3. H contains an attribute of the form NestMembers({... "C", ...}). This is a simpler specification, and it also closes any possible loopholes in package/nest alignment by insisting on equal runtime packages from the start. It also avoids a possibly wasteful class loading step, which is only necessary if we allow L1 and L2 to differ. That would be a pathology IMO, not a use case. The way I visualize this is that each newly arrived class checks in with its host. If the name is on the list, it gets in. Sound good? ? John From brian.goetz at oracle.com Thu Apr 20 02:46:50 2017 From: brian.goetz at oracle.com (Brian Goetz) Date: Wed, 19 Apr 2017 22:46:50 -0400 Subject: Nestmate same-package check In-Reply-To: <162B4430-FAFC-44DE-ABA4-941B423FEC37@oracle.com> References: <162B4430-FAFC-44DE-ABA4-941B423FEC37@oracle.com> Message-ID: <07b988a4-f30c-1f9b-1005-b0f270c4c5bd@oracle.com> > The way I visualize this is that each newly arrived class > checks in with its host. If the name is on the list, it gets in. > Right, and if it's not on the list? Does it get assigned its own nest-of-one, or do we reject the class? I would expect the former? (If you show up at a party and you're not on the invite list, you can still have your own party, just not here.) From john.r.rose at oracle.com Thu Apr 20 03:08:08 2017 From: john.r.rose at oracle.com (John Rose) Date: Wed, 19 Apr 2017 20:08:08 -0700 Subject: Nestmate same-package check In-Reply-To: <07b988a4-f30c-1f9b-1005-b0f270c4c5bd@oracle.com> References: <162B4430-FAFC-44DE-ABA4-941B423FEC37@oracle.com> <07b988a4-f30c-1f9b-1005-b0f270c4c5bd@oracle.com> Message-ID: <42BB2C01-1A21-483E-AE3C-FC057EAF5815@oracle.com> On Apr 19, 2017, at 7:46 PM, Brian Goetz wrote: > > >> The way I visualize this is that each newly arrived class >> checks in with its host. If the name is on the list, it gets in. >> > > Right, and if it's not on the list? Does it get assigned its own nest-of-one, or do we reject the class? I would expect the former? (If you show up at a party and you're not on the invite list, you can still have your own party, just not here.) Hmm? I assumed you would fail the load. Let's see? Failing the load leads to a harsh, "fast-fail" binary compatibility policy. It specifically affects left-over class files from a previous compile. If you load a leftover by accident, it will point to either a non-existent, or a newly-compiled host that won't acknowledge it. Is there a binary compatibility case for loading leftovers like that? I kind of think no: It's always a bug when files from previous compilations stick around. Let's fail them instead of try to load them in some party-of-one recovery mode. Do you have a different use case in mind where a recovery mode is useful? ? John From john.r.rose at oracle.com Thu Apr 20 03:22:45 2017 From: john.r.rose at oracle.com (John Rose) Date: Wed, 19 Apr 2017 20:22:45 -0700 Subject: Valhalla Minimal Value Types review invitation In-Reply-To: References: Message-ID: On Apr 19, 2017, at 8:16 AM, Bjorn B Vardal wrote: > > I'm writing up my notes from the meeting, and I have a question about the box's no-arg constructor. My notes say that this constructor is automatically provided and may not be replaced by the user. My understanding is that this is done in order to force the constructor to have the same behaviour as "vdefault; vunbox;". Correct. As in C#, a value type always has a system-supplied nullary constructor which cannot be replaced by the user. Given the ubiquitous access to "vdefault" (by both the instruction and by sampling unassigned fields and array elements) there is a huge potential performance cost (plus bootstrapping problems) if we allow users to supply computed bit patterns at all variable creation points. It's not just bit-blt, either, since if the computed patterns have non-null managed pointers in them, the JVM is likely to have to issue extra card mark operations. It's a mess, and not worth it, so we just say a default value looks like a fresh object, before any of its fields are initialized. > My question is, did we decide to restrict the accessibility of these constructors, Yes, that's in the draft. > and if so, what would be the reason to do this? As stated above, the default box can already be obtained by unboxing a default value. In fact, all constructors must be private in a VCC. The reason is to prevent client code from calling "new VCC" directly (as "new;dup;invokespecial"). That in turn prevents clients from creating their own fresh VCC identities. It doesn't prevent those identities from coming into being other ways, but makes it clear that there is no first-class API for creating a box that you can view as "all your own". Instead, VCC authors are forced to write factory methods. And those will typically never (well, maybe rarely) make guarantees about creating fresh object identities. ? John From david.holmes at oracle.com Thu Apr 20 03:24:26 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 20 Apr 2017 13:24:26 +1000 Subject: Nestmate same-package check In-Reply-To: <162B4430-FAFC-44DE-ABA4-941B423FEC37@oracle.com> References: <162B4430-FAFC-44DE-ABA4-941B423FEC37@oracle.com> Message-ID: On 20/04/2017 12:42 PM, John Rose wrote: > On Apr 18, 2017, at 11:42 AM, Dan Smith > wrote: >> I've uploaded a draft of JVMS changes for JEP 181 "Align JVM Checks >> with Java Language Rules for Nested Classes" to: >> http://cr.openjdk.java.net/~dlsmith/private-access.html > >> classHasValidNest(Class) :- >> >> classMemberOfNestName(Class, HostName), >> classDefiningLoader(Class, L1), >> loadedClass(HostName, L1, HostClass), >> samePackageName(Class, HostClass), >> classClassName(Class, Name), >> classNestMemberNames(HostClass, MemberNames), >> member(Name, MemberNames), >> classDefiningLoader(HostClass, L2), >> loadedClass(Name, L2, Class). > > In English: > > When validating nestmate attributes, if a class C with name > "C" has an attribute MemberOfNest("H"), it is valid only if the > following are all true: > 1. The string "H" resolves from C's loader (L1) as H. > 2. The strings "C" and "H" have the same package prefix. > 3. H contains an attribute of the form NestMembers(?"C"?). > 4. The string "C" resolves from H's loader (L2) as C. > > Steps 1 and 4 are both CONSTANT_Class resolution operations. > Step 3 is a simple search of a symbol list, which is easy. > > This logic can be simplified by testing that C and H must > be in the same runtime package. In that case, L1 and L2 are > identical, and step 4 becomes: > 4'. C and H have the same defining loader. > > Then we can refactor 2 and 4' into a more direct check on > runtime package (using the functor 'sameRuntimePackage'): > 1. The string "H" resolves from C's loader (L1) as H. > 2'. C and H have the same runtime package. > 3. H contains an attribute of the form NestMembers({... "C", ...}). > > This is a simpler specification, and it also closes any possible > loopholes in package/nest alignment by insisting on equal > runtime packages from the start. > > It also avoids a possibly wasteful class loading step, which > is only necessary if we allow L1 and L2 to differ. That would > be a pathology IMO, not a use case. > > The way I visualize this is that each newly arrived class > checks in with its host. If the name is on the list, it gets in. > > Sound good? This is basically what the existing validation logic does - modulo the order of operations: 1. load "H" using C's classloader 2. H contains an attribute of the form NestMembers({... "C", ...}) 3. Check is_same_package(C, H) The order can be changed if that is significant. David ----- > ? John From david.holmes at oracle.com Thu Apr 20 03:30:15 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 20 Apr 2017 13:30:15 +1000 Subject: Nestmate same-package check In-Reply-To: <07b988a4-f30c-1f9b-1005-b0f270c4c5bd@oracle.com> References: <162B4430-FAFC-44DE-ABA4-941B423FEC37@oracle.com> <07b988a4-f30c-1f9b-1005-b0f270c4c5bd@oracle.com> Message-ID: <5f347a0b-91dc-a9ad-a86c-fa89e86f26dc@oracle.com> On 20/04/2017 12:46 PM, Brian Goetz wrote: > >> The way I visualize this is that each newly arrived class >> checks in with its host. If the name is on the list, it gets in. >> > > Right, and if it's not on the list? Does it get assigned its own > nest-of-one, or do we reject the class? I would expect the former? (If > you show up at a party and you're not on the invite list, you can still > have your own party, just not here.) View this not as "trying to sneak into a party with fraudulent credentials" but "trying to sneak into the secure defense establishment with fraudulent credentials". There is no "nice try, but on your way", you are injected with a lethal dose of IncompatibleClassChangeError and unceremoniously ejected. This, as with a failed same-package check, seems the most reasonable course of action as the user obviously made in mistake in only partially recompiling their sources when the nesting structure changed. Cheers, David > From david.holmes at oracle.com Thu Apr 20 03:38:18 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 20 Apr 2017 13:38:18 +1000 Subject: Draft JVMS changes for Nestmates In-Reply-To: <27C41496-0782-4690-9242-A7660736AB0E@oracle.com> References: <0848391A-AF8C-4B4D-8CDA-FFC727E5CF6B@oracle.com> <1247948811.901298.1492546336021.JavaMail.zimbra@u-pem.fr> <25912A06-ABE8-4E83-8EC6-9A9CA6888B18@oracle.com> <3e91872b-741f-107f-c263-53e5cf75c80f@oracle.com> <27C41496-0782-4690-9242-A7660736AB0E@oracle.com> Message-ID: On 20/04/2017 2:47 AM, Dan Smith wrote: > >> On Apr 19, 2017, at 8:12 AM, Brian Goetz wrote: >> >>> Yep. Also, multiple classes can claim the same nest member class in their NestMembers attributes. Not a problem as long as the MemberOfNest attribute (if any) of the member class points to a host class that claims it. >> >> >> At the risk of bikeshedding, I find the names NestMembers and MemberOfNest confusing; I keep having to think about the directionality, since they sound so similar. Would be good to find names that have obviously opposite directionality. >> >> A not-very-good suggestion (but which illustrates the directionality thing I'm going for) would be "NestDeclaration" and "NestUse". It's pretty clear which is which without thinking about it. >> >> Perhaps: NestHost and NestMember ? > > I also don't love that "NestMembers" seems more authoritative than it really is. (Doesn't get validated, doesn't suggest that other classes can be dynamically added.) > > Something like "AllowedInNest" would better convey the actual meaning of that attribute. I don't see how AllowedInNest is any less authoritative sounding than NestMembers. Neither suggests other classes may be added dynamically - nor should they in my opinion. These are static attributes of a classfile defining a relationship as it was known to exist when the classfile was created. Any dynamic means to add to a nest at runtime would not be modifying the classfile NestMembers attribute, but the runtime representation thereof. I suppose InitialNestMembers may capture this more accurately - though it does then beg the question as to how to add the to set later. Aside: when first discussed in the context of generic specialization I assumed the NestMembers entry would be a "wildcard" which indicated that not only was the named generic type a nestmember, but all specializations thereof (which would presumably have a name derived from the base generic type). But in the context of Lookup.defineClass I have no idea how it is/was expected that nest membership would expand - or how the validity of doing so would be established. David > ?Dan > From david.holmes at oracle.com Thu Apr 20 03:56:51 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 20 Apr 2017 13:56:51 +1000 Subject: Draft JVMS changes for Nestmates In-Reply-To: <48B779AB-6D9A-464B-B78A-AAFE5B9D8A5C@oracle.com> References: <524BFE31-4A61-49B3-9AE9-01B154052F96@oracle.com> <48B779AB-6D9A-464B-B78A-AAFE5B9D8A5C@oracle.com> Message-ID: <8b1d4bc1-bf8f-e9e6-fe6a-b876295b5d3b@oracle.com> On 20/04/2017 7:58 AM, Dan Smith wrote: >> On Apr 19, 2017, at 1:21 PM, Dan Smith > > wrote: >> >>> >>> 5.4.4 Access Control >>> >>> I don't think you need to restate: >>> >>> "A class with a MemberOfNest attribute belongs to the nest hosted by >>> the referenced host class. >>> >>> A class without a MemberOfNest attribute implicitly belongs to a nest >>> hosted by the class itself. (If the class also lacks a NestMembers >>> attribute, then the nest has only one member.)" >>> >>> The rule is simply stated as is: "belonging to the same nest as D" - >>> belonging to the same nest is, or should be, already defined elsewhere. >> >> My intent is that this *is* the definition. But sounds like you're >> expecting that definition to be in 4.7.25/4.7.26, which is probably >> more intuitive. I'll change that. Thanks. Yes I was expecting a definition somewhere else, up front. > Okay, here's another attempt at defining the attributes and the "nest" > concept. > > I've been uneasy about `NestMembers` being presented as the > "declaration" of the nest, since it's not validated. Instead, I see > `MemberOfNest` as really where nests are defined (as a cumulative result > of many class declarations?like packages), and `NestMembers` as just a > help for validation. I can't decide whether that understates the importance of the bi-directionality of membership, or not - both parties must agree on membership for that membership to exist. It may not have any practical significance. > With that in mind, `MemberOfNest` ought to come first: > > ~~~~ > > **4.7.25 The `MemberOfNest` Attribute** > --------------------------------------- > > **The `MemberOfNest` attribute is a fixed-length attribute in the > `attributes` table of a `ClassFile` structure ([4.1]).** > > **A _nest_ is a set of classes and interfaces that share access to their > `private` members ([5.4.4]).** Aside: it is this basic definition of "nest" that I expected to appear somewhere earlier in the spec. JVMS doesn't have any upfront "concepts and definitions" section, so not sure where it would go. > **A class or interface with a `MemberOfNest` attribute belongs to the > nest hosted by a designated _host class_. The host class authorizes > membership in the nest with a corresponding entry in its `NestMembers` > attribute ([4.7.26], [4.10]).** Only nit is I would say "claims membership of" rather than "belongs to". That emphasises that membership must be ratified by the "host class", and matches the following "authorizes membership" text. > **A class or interface without a `MemberOfNest` attribute belongs to the > nest hosted by itself. (Often, this nest is a singleton consisting only > of the class itself.)** > > **There may be at most one `MemberOfNest` attribute in the `attributes` > table of a `ClassFile` structure.** > > **The `MemberOfNest` attribute has the following format:** > > ... > > > **4.7.26 The `NestMembers` Attribute** > -------------------------------------- > > **The `NestMembers` attribute is a variable-length attribute in the > `attributes` table of a `ClassFile` structure ([4.1]). It authorizes an > enumerated set classes and interfaces to claim membership in a nest > hosted by the current class or interface.** > > **There may be at most one `NestMembers` attribute in the `attributes` > table of a `ClassFile` structure.** > > **The `attributes` table of a `ClassFile` structure must not contain > both a `MemberOfNest` attribute and a `NestMembers` attribute.** > >> **This rule prevents a host class from claiming membership in a > different nest. It is implicitly a member of the nest that it hosts.** > > **The `NestMembers` attribute has the following format:** > > ... > > > 5.4.4 Access Control > -------------------- > > A class or interface _C_ is _accessible_ to a class or interface _D_ if > and only if either of the following is true: > > ... > > - _R_ is `private` and is declared ~~in _D_~~ **by a class belonging to > the same nest as _D_ ([4.7.25])**. > > > ~~~~ > > And now to the bikeshedding game: :) > `MemberOfNest`, as the key thing, could just be called `Nest` (imagine a > `Package` attribute?we wouldn't call it `MemberOfPackage`). Or, since > it's a reference to a class, `NestHost ` / `NestTop` / `NestMother` / > `NestWhatever`. I like to distinguish the name of the attribute from the entity the attribute represents. Given the names Maurizio chose are not terribly wrong, and nothing else is significantly better, I'm quite content with them. > `NestMembers` isn't horrible, but we could highlight its validation role > with something like `AuthorizedNestMembers`. To me the "authorized" is implicit. By calling it out explicitly it suggests to me there are also UnauthorizedNestMembers. But to me this is an oxymoron - if you are unauthorized you are not an unauthorized-member, you are not any kind of member at all. Cheers, David > ?Dan From david.holmes at oracle.com Thu Apr 20 04:12:24 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 20 Apr 2017 14:12:24 +1000 Subject: Draft JVMS changes for Nestmates In-Reply-To: <73906CB5-80CF-42B4-B7C8-3F55FE1425F9@oracle.com> References: <73906CB5-80CF-42B4-B7C8-3F55FE1425F9@oracle.com> Message-ID: <74387a51-f811-46a3-ab28-4b3fac97393a@oracle.com> Limited response as issues have been covered elsewhere, and Dan has revisited suggested spec wording. But for completeness ... On 20/04/2017 10:32 AM, John Rose wrote: > On Apr 18, 2017, at 11:40 PM, David Holmes > wrote: >> >> 4.7.26 The MemberOfNest Attribute >> >> "the class is implicitly a member of its own nest" >> >> I suggest changing "a" to "the sole". Otherwise "a" implies there may >> be other members. As these are static properties being defined I don't >> think we need to be accounting for some future dynamic expansion of a >> nest. > > I think you're missing the point of the spec. logic here. If there is a > non-empty NestMembers attribute, then in fact there *are* other members. > Therefore "a" is correct, and "the sole" would be incorrect. Yes another gaseous brain expulsion on my part. I was thinking only of the case where neither attributes are present. > Separately, we do need to leave wiggle room for dynamic injection. > Dan, I think that would be done by allowing a dynamic injection > operation to implicitly extend the NestMembers list to make a > "ticket" for the injected class, and implicitly add (if necessary) > a MemberOfNest to the injected class. > > ... >> >> 4.10.1 Verification by Type Checking >> >> Does the order in the rules mandate the order in which the VM must >> actually check things? classHasValidNest seems much more complex than >> the actual runtime process of validation (though it may just be >> exposing distinct steps that are lumped together at runtime). > > Agree. I think this logic can be simplified. (Will take that to a > separate sub-thread.) > >> --- >> >> 4.10.1.8 Type Checking for Restricted Member References >> >> I'm not very good at reading or understanding these rules, but I'm >> surprised that the invokespecial rules seem to make no mention of >> nestmates at all: >> >> "A method reference is allowed by an invokespecial instruction if it >> is not a restricted invokespecial reference, ..." >> >> This seems to open things up to all private method references?? > > Well, the whole point of nestmates is to extend, in a regular way, > all access rules that pertain to self-access and privacy. > If the extension is truly regular, it is not surprising that the spec. > changes are subtle like this. We're not trying to do an ad hoc > patch on some set of particular behaviors, but rather extend > pre-existing notions so they apply in more conditions, including > invokespecial restrictions. I was expecting to see the restrictions have a dependency on nestmates actually being involved - which is how I've currently implemented it. I can throw away some of the verifier changes if I only need to check for private access. But I'm unclear exactly where (in the spec) we enforce that only nestmates get this right to use invokespecial on private methods? Is it the access check applied during method resolution? Cheers, David ----- >> >> --- >> >> 5.4.4 Access Control >> >> I don't think you need to restate: >> >> "A class with a MemberOfNest attribute belongs to the nest hosted by >> the referenced host class. >> >> A class without a MemberOfNest attribute implicitly belongs to a nest >> hosted by the class itself. (If the class also lacks a NestMembers >> attribute, then the nest has only one member.)" >> >> The rule is simply stated as is: "belonging to the same nest as D" - >> belonging to the same nest is, or should be, already defined elsewhere. > > Actually, there is no "elsewhere"; this is where the condition is defined. > We don't need a special section for "nests"; we don't have a special > section for "packages" either. So that part reads fine to me. > >> I'm avoiding commenting on the protected access changes but do want to >> raise a concern that there are no changes to Method resolution >> (5.4.3.3) yet resolution relies on the definition of access control to >> prune/discard inaccessible methods. It appears now that we're allowing >> more potential successful resolutions, then using the additional rules >> on the invoke* bytecodes to try and discard any undesirable results. >> >> --- >> >> 6.5 getfield (and others) >> >> You note "These rules are redundant: verification already guarantees >> them " but this brings me back to an area I keep raising concerns >> about: the split between static verification based checks and dynamic >> runtime checks. Yes the verifier precludes certain things but if we >> run without verification the rules expressed for the bytecodes are >> considered to be still required at runtime. If you delete them because >> they overlap with verifier rules we have no way to tell which rules >> must be enforced regardless of verification status. >> >> --- >> >> 6.5 invokespecial >> >> You note regarding the IllegalAccessError: >> >> "This replaces a VerifyError previously specified by 4.9.2. The check >> must be delayed until after resolution in order to determine whether >> the referenced method is private." >> >> We know the access flags for the referenced method at verification >> time - shouldn't that be the sole basis for the verifier check? >> Afterall it is verifying the static properties of the classfile and >> bytecode. If the actual resolved method has different access to the >> referenced method then that may lead to ICCE (depending on the exact >> nature of the change - a private method made public may not be an >> issue for example). >> >> --- >> >> 5.4.3.3 Method resolution >> >> You did not make any changes here, but as per my comment in the bug >> report we do require, IMHO, further tightening here to ensure nestmate >> invokespecial invocations do not resolve to method implementations >> that they should not. The example here is a hierarchy of nestmates (C >> extends B extends A) where A and C both declare a private method "void >> m()" and we have an invocation c.m() where c is an instance of class >> C. We then "separately compile" C to remove the definition of m(). At >> runtime method resolution will locate A.m, even though the method >> reference was for C.m. Normally the accessibility rules would exclude >> A.m from being a viable candidate but here the classes are nestmates >> so A.m is accessible. But it would be wrong to invoke A.m. This case >> should throw ICCE or NSME. >> >> Thanks, >> David >> ----- > From david.holmes at oracle.com Thu Apr 20 04:28:13 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 20 Apr 2017 14:28:13 +1000 Subject: Draft JVMS changes for Nestmates In-Reply-To: <8B895CBF-76E1-466D-AC72-7B23F5350C65@oracle.com> References: <0848391A-AF8C-4B4D-8CDA-FFC727E5CF6B@oracle.com> <1247948811.901298.1492546336021.JavaMail.zimbra@u-pem.fr> <25912A06-ABE8-4E83-8EC6-9A9CA6888B18@oracle.com> <90b555a2-1290-e81c-da0c-54aa9bbfdaa7@oracle.com> <8B895CBF-76E1-466D-AC72-7B23F5350C65@oracle.com> Message-ID: One follow up to a new comment John made ... On 20/04/2017 10:09 AM, John Rose wrote: > On Apr 19, 2017, at 12:56 AM, David Holmes > wrote: >> >>> I don't see how that is a significant concern. The symbol bodies are >>> hot in cache >>> at the point we would check prefixes, since they are already being >>> scanned for other >>> purposes, such as initial interning, and also syntax checking. Existing >>> processing >>> is exactly as deep (or shallow) as the checks I want. >> >> I don't follow that. If I'm loading a nest-top class and validating >> its NM entries none of those entries need have been loaded yet and so >> none will be in the cache. > > I see what the problem is: The only checking I am proposing > is syntax checking on the names. If by "deep checking" you > mean loading classes from the names, I agree, we don't want > to do that. The MemberOfNest attribute is passive. It does > not need to initiate loads; it just needs to validate claims > of nest membership by other classes. > > (I even doubt Dan's assertion that in some corner cases, > when the nest-host is trying out a private access, there > must be some loading done to validate the access. Probably > we can get away with no loading at all, just name checks > against already-loaded classes.) If the access is legitimate then no loading is needed as the common nest-host is already loaded. Currently the implementation will load the purported nest-host if needed, so in the case where they are not actually nest-mates this would load the "foreign" nest-host class. But when the initiator is the nest-host the check can be inverted by searching NestMembers for the target class instead. I see one quirky aspect of this in that in the unlikely case that the target class is in NestMembers but does not itself list the nest-host in its MemberOfNest, the access check will succeed, but before the invocation can actually happen the target class must be linked and at that point validation would fail (assuming we adopt the fail-fast model). David > ? John From david.holmes at oracle.com Thu Apr 20 08:21:48 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 20 Apr 2017 18:21:48 +1000 Subject: Draft JVMS changes for Nestmates In-Reply-To: <524BFE31-4A61-49B3-9AE9-01B154052F96@oracle.com> References: <524BFE31-4A61-49B3-9AE9-01B154052F96@oracle.com> Message-ID: <4cc09ba6-a99b-4a4e-455a-b38d3b18ba07@oracle.com> Hi Dan, I'll try not to double up on things already discussed elsewhere. And avoid bikeshedding :) On 20/04/2017 5:21 AM, Dan Smith wrote: > Thanks a lot for the feedback! My comments below: > >> On Apr 19, 2017, at 12:40 AM, David Holmes wrote: >> >>> JEP name >>> >>> I don't know how permanent JEP names are supposed to be, but I'd prefer a different name at this point. Something like: "Expanded JVM Access to Private Members"?shorter, focused on the feature itself rather than its relationship to the Java language. Or maybe "Class Nests for Access to Private Members". >> >> The intent of the name is to aid in the contextual positioning of this JEP ie address why do we even need this. The JEP addresses an inconsistency between access controls in the Java programming language and access controls in the JVM. The primary goal here is to align the two, while at the same time allowing for additional usage of nestmates in the future. > > Yeah, but I think explaining the motivation in the title makes for a long title and limits the imagination to one particular use case. > >>> The JEP uses "nest top" to describe the class that nest members reference; I prefer "host class", which better describes the class's role and isn't tied to the Java "top level class" concept. I know we use "host class" internally in Hotspot, perhaps when working with anonymous classes (of the JVM flavor), but I think in that context it will ultimately mean the same thing? Are we comfortable repurposing the term in this way? >> >> Given that a nest-top type must be a top-level type, I think nest-top fits perfectly. > > This is a good example of my point about the title: the JVM has no concept* of "top level type", and doesn't need one. We just need a specific class to act as the reference point by which a group of classes can talk about a nest. But we are attempting to justify this by relating to the impedance mismatch with the Java language. And we do utilize the notion of "top-level class" (as defined by JLS) elsewhere. > For example, I can imagine some language introducing a class file?foo.bar.Nest$$001?solely for the purpose of providing a hook for this reference point. javac's choice to designate a top-level class as the nest-top/nest-host is a compilation strategy. > > (*Yes, there's an InnerClasses attribute, and Signature attributes, and lots of other stuff to facilitate compilation in the Java language. But those represent auxiliary metadata, not core JVM concepts.) > >> I dislike "host class" because that is already used for the quite different case of VMACs. I can live with other names but I think they should incorporate "nest" eg. nest-holder, nest-host. But I think nest-top is an ideal match. (And no I didn't invent this terminology :) ). > > Okay. Probably best to sit on this for awhile and circle back to it, see how we're all feeling. I think nest-host is starting to take the lead :) >>> I follow Brian's model when it comes to nest membership (5.4.4): every class belongs to a nest, possibly (in the absence of MemberOfNest) the nest hosted by itself. Many nests are singletons without any associated explicit attributes. >> >> I have not yet seen a follow up from John as to whether we require or just allow, an empty NestMembers attribute to indicate a singleton nest member. > > If in our model *every* class has a nest, then we definitely don't want to require every class to have a NestMembers attribute. > > In my spec (5.4.4), the presence of a NestMembers attribute is irrelevant. A class without a MemberOfNest attribute belongs to its own nest, and there's no need to validate anything about NestMembers. > >>> Verification of MemberOfNest >>> >>> I include a discussion block about different options of validating MemberOfNest. I think the consensus, and my preference, is to do it during verification of the member class. (NestMembers, on the other hand, is never validated, except as a side-effect of checking MemberOfNest.) >> >> Given verification is optional the current approach is to validate during linking just prior to the verification step. Whether that can conceptually be considered part of verification I am unsure. > > I think this is the source of a lot of anomalies?spec treats verification as an essential part of linking, while Hotspot treats it as an optional add-on. > > You should be okay, for some definition of "just prior". These are the constraints, as I read them: > - "A class or interface is completely loaded before it is linked." (5.4) > - Loading includes structural checks (CFE), version checks (UCVE), and validation and loading of superclasses and superinterfaces (5.3.5) > - "Errors detected during linkage are thrown at a point in the program where some action is taken by the program that might, directly or indirectly, require linkage to the class or interface involved in the error." (5.4) > > That last one is pretty mysterious to me (what does "some action" mean? what about "indirectly"?), but seems to grant pretty wide latitude on the timing. I believe the intent with that is to preserve the notion of precise exceptions that the language has. Exceptions should be seen to be thrown due to a specific action of the program that required a class to be loaded/linked/initialized and which subsequently threw an exception. Anyway, yes I think I'm okay with validation during linking. That said the reason I placed it before verification was because verification could require the nestmate-access check if it encountered a private-invokespecial. But as per discussion elsewhere (and perhaps below) your spec changes do not require the nestmate check, so the problem I was addressing no longer exists. >>> Verification of invokespecial >>> >>> Allowing invokespecial to refer to classes other than the current class and its supers is a significant change. I noticed and relied heavily on the parallel with invokevirtual making protected method calls. So I tried to make the two as similar as possible. In a few places, the treatment of protected methods doesn't seem ideal, and rather than trying to mirror that with non-private invokespecial, I modified the protected method treatment. >> >> This concerns me because protected-access is a somewhat complex/messy issue so aligning private-access with it seems to go in the wrong direction. Cleaning up the protected access rules, while perhaps desirable, is technically out-of-scope for this JEP in my opinion. >> >>> The "protected check" of verification, in particular, was a mess before, and I think I've made it a lot more manageable, and compatible with a parallel rule for invokespecial. I could use some feedback on exactly what Hotspot's verifier has been doing here, though, since I'm pretty sure it didn't match the old specified rules. >> >> Might be better to take this up separately. > > Yeah, I hear you. But I worry about the technical debt if we just look the other way and specify/implement a fresh invokespecial check separately. The subtle inconsistencies between the two will lead to all sorts of bugs and tweaks in the future; the duplicate code/spec will be an ongoing maintenance burden. In terms of implementation the verification of invokespecial involves a number of checks, including the subtype check. What I have been doing is simply ignoring the result of the subtype check if I am dealing with nestmates. With the proposed spec for private invokespecials I can simply skip the subtype test altogether. It is unclear what the ramifications are of the changes to protected access on the implementation. The mapping from rules to code is not obvious to me. > From the spec perspective, pulling on the thread looks like this: > - The new verification we need on invokespecial looks almost the same as the verification we need on protected methods. > - Okay, I'll re-use that rule. But it's a complicated mess, and has bugs, so I've got to clean it up first. > - What exactly are these rules trying to achieve? I can see the hard constraints (in configuration X, an error occurs), but there's a soft requirement to minimize class loading. > - To figure out when class loading is acceptable, I need to know what the reference implementation actually does. I just don't like the idea of messing with the protected access checks. >>> MethodHandle resolution >>> >>> The spec (5.4.3.5) is vague about what errors can occur during MethodHandle resolution. I assume any linkage error specified for the instruction can also occur via MethodHandle resolution, and that will include failures due to invokespecial improperly referencing a class. >> >> This is an area I have least understanding of, but this seems to cover it: >> >> "In each step, any exception that can be thrown as a result of failure of resolution of a class or interface or field or method reference can be thrown as a result of failure of method handle resolution." > > The ambiguity is in the definition of "resolution". The preferred interpretation, which everyone seems to confirm reflects reality, is that it includes all linkage errors specified for the referencing instruction. I.e., errors from both both 5.4.3 and 6.5 can occur. Yes. 5.4.3 states that "Certain of the instructions above require additional linking checks when resolving symbolic references." And that those linking exceptions are listed with the instructions themselves. So yes resolution errors consist of the union thereof. >>> Dynamic checking of invokespecial receivers >>> >>> When invokespecial involves interface types, the verifier can't guarantee that receiver objects actually implement that interface (JDK-8134358). It's an open question whether this is really a problem, but I included a tentative fix in the invokespecial runtime rules. >> >> This seems out of scope in the general case, but there may be nestmate specific actions required. > > I touch it here because: > - It's a dynamic companion to verification, and we've modified the verification rules > - There's an existing rule that may or may not be intended to perform part of this dynamic check; that needs to be addressed somehow > - Any rule we introduce is impacted by the new treatment of private methods, so easier to get it right while working on this feature > - Easier to address small things like this when it's swapped in already > > If necessary, we can yank it and address the bug separately in the future, but it seems convenient to tackle it now. > >>> Compiler changes >>> >>> The JEP text can't seem to decide if compiler changes are part of it, or a follow-up exercise. I think we're best off explicitly including the compiler changes, which will provide opportunities for design validation and testing. >> >> Not sure what gives you that impression, but javac changes are an essential part of this. Lacking a compiler spec I assume this will be handled informally. But the attribute definitions (see comments below) should dictate what a source compiler is required to do. > > - Would be clearest if "Goals" mentioned modifying javac > - Language like "javac can" suggests optional followup work; should have a paragraph saying "javac will" instead > - Remove the paragraph saying "it is not strictly required to remove these immediately as part of this feature" Agreed - I think I already gave similar feedback but it didn't get taken up. >>> Security risk >>> >>> The JEP text should acknowledge that, while this does allow compilers to grant finer-grained access to members shared by nestmates, it also pushes compilers to grant broader access to members that were previously kept private. It's a trade-off, and presumably a good one because nestmates are completely trusted, while package-mates might sometimes be suspect. >> >> I'm not following you here. What broader access is being granted to members that were previously kept private? This effort allows private-only access to members that were previously package-accessible. > > Old approach is to widen access to private members to the package level on a per-member basis. > > New approach is to widen access to private members to the nest level on a per-class basis. > > Nest is narrower than package (good!), but per-class is more permissive than per-member (bad). Some members that are compiled as totally private in JDK 9 will be incidentally shared with all nest members in . I'm not sure how you are coming at this. For a private member in 9 to be incidentally shared in the class of the member has to be added to a nest. With no context as to how this may happen I can't really comment further. From the Java language perspective if the class is in a nest then such access to private members already exists. >> 4.7 Attributes >> >> Regarding the Exceptions attribute, it seems to be used only to store data to be passed back to the libverify and the reflection API. > > I'm not familiar with libverify, but I think you're confirming that the attribute is on par with Signature? Something for tools and standard APIs to rely on, but irrelevant to execution? Right - sorry that wasn't clear. >> Regarding the definitions of NestMembers and MemberOfNest ... I modelled the definitions in: >> >> https://bugs.openjdk.java.net/browse/JDK-8177020 >> >> based on those for the innerclass (and related) attributes. I'm not sure why you moved away from those definitions as they pinned down exactly when these attributes are expected to appear. Without those parts the current definitions seem optional - ie they define what the attribute means if present, but they do not require its presence. I do not believe these attributes should be optional for classfile version 54, but required whenever a nest does exist. A java source compiler writer should look at these definitions and know when they must generate these attributes. > > These sorts of assertions belong in the "Java Compiler Specification". (Yes, this document does not exist. Yes, that's kind of a problem, but we muddle through.) Part of that muddling through, to me, is the ability for a compiler writer to understand when they must generate certain classfile contents. Hence my approach here. > In an ideal world, JVMS would not acknowledge that the Java Programming Language exists. It should stand on its own, and Java is merely a client. > > As is, we have some history that has led to some dependencies on Java in spec, but these are generally auxiliary items: attributes that don't impact JVM behavior (4.7), a tutorial on how to compile Java code (3), occasional explanatory references to Java language concepts (e.g., 2.9). > > But it would be wrong for the specification of a "critical" attribute (4.7) to be defined in terms of Java language concepts. It should stand on its own, general enough to meet the needs of Java language compilers and others with different use cases. I think you have somewhat rose-coloured glasses on if you think the JVMS is only tainted by the Java language in "auxiliary items". :) The whole set of rules around invokespecial pertain specifically to how it supports the Java language requirements for constructor invocations and super-type invocations. Otherwise invokespecial would have remained as its original invokenonvirtual - which makes more sense to me even now. >> 4.7.26 The MemberOfNest Attribute >> >> "the class is implicitly a member of its own nest" >> >> I suggest changing "a" to "the sole". Otherwise "a" implies there may be other members. As these are static properties being defined I don't think we need to be accounting for some future dynamic expansion of a nest. > > I intended to include both the cases of a singleton nest and the case of a class that has a NestMembers attribute. But I'll add a clarifying sentence. > >> 4.9.2 Structural Constraints >> >> The changes to invokespecial seem okay, but I have a problem with terminology. In: >> >> "If invokespecial is used to invoke a non-private, non- method, referenced via a superclass or a direct superinterface, ..." >> >> what does "referenced via" mean? > > I'll change to "where the referenced class is" Ok. > >> In: >> >> "If getfield or putfield is used to access a protected field referenced in a superclass ..." >> >> and following, you changed "declared" to read "referenced". Again I do not understand what this is supposed to mean. It is the declaration site of the field or method that is needed to determine whether that member is in the same or another package. To me "declared in a superclass ..." was exactly correct. Your note does not make the change any clearer to me. > > Good catch. It's important that the rule be limited to cases that reference a superclass*; but it's also important that the declaring class be used for the package check. > > (*Here's why: if I reference my subclass, and that resolves to a field declared in my superclass, the verifier will perform no check, because it may not know that the referenced class is a subclass.) > > Revised: > > "If `getfield` or `putfield` is used to access a `protected` field by reference to a superclass, and that field is declared in a different run-time package than the current class or interface, then the type of the class instance being accessed must be assignable to the current class or interface." I still think "declared in" is the correct terminology. There's a nuance to how you are expressing this that I'm just not getting. >> 4.10.1 Verification by Type Checking >> >> Does the order in the rules mandate the order in which the VM must actually check things? classHasValidNest seems much more complex than the actual runtime process of validation (though it may just be exposing distinct steps that are lumped together at runtime). > > Probably not, but dependencies between inputs/outputs do suggest an ordering. > > (We could probably get into an argument with a logician or a Prolog expert about whether there's such a thing as an "output" at all, and whether it would be acceptable for a JVM to load every class in the universe in order to test certain predicates, which is why it's probably not a great idea to specify the verifier using Prolog rules. Alas.) > > The important elements are: > 1) Resolve the host class using the current class's loader > 2) Check that the host class belongs to the same package > 3) Check that the current class is named in the host's NestMembers > 4) Resolve the current class's name in the host class's loader, and make sure it produces the same class > >> >> --- >> >> 4.10.1.8 Type Checking for Restricted Member References >> >> I'm not very good at reading or understanding these rules, but I'm surprised that the invokespecial rules seem to make no mention of nestmates at all: >> >> "A method reference is allowed by an invokespecial instruction if it is not a restricted invokespecial reference, ..." >> >> This seems to open things up to all private method references?? > > There's a careful partitioning of error conditions that has to happen in order to avoid too much class loading. > > 4.10.1.8 is only concerned with ensuring, in narrow cases, that the type on the stack is assignable to the current class. For invokespecial, the narrow case is a method reference naming a superclass/direct superinterface and a method not named "", where the resolved method is not private. > > Separately, 5.4.4 ensures that resolving a private method will produce an IllegalAccessError if the method is not in the same nest. > > Also separately, 6.5 ensures that if the resolved method is not private, the referenced class is the current class, a superclass, or a direct superinterface. > > And also separately, 4.10.1.9 ensures that the type on the stack is always assignable to the referenced class. Okay. I have touched on this elsewhere but worth reiterating I think. I had been working on the expectation that an attempt to use private-invokespecial when the particpants were not nestmates would fail verification. I added an explicit nestmate check to the verifier logic. But instead you're loosening the verifier to simply accept any private-invokespecial. Then the actual invocation is access-checked. That's okay - it simplifies things somewhat and avoids the potential need to load a foreign nest-host during verification. I'm just a little surprised. :) >> 5.4.4 Access Control >> >> I don't think you need to restate: >> >> "A class with a MemberOfNest attribute belongs to the nest hosted by the referenced host class. >> >> A class without a MemberOfNest attribute implicitly belongs to a nest hosted by the class itself. (If the class also lacks a NestMembers attribute, then the nest has only one member.)" >> >> The rule is simply stated as is: "belonging to the same nest as D" - belonging to the same nest is, or should be, already defined elsewhere. > > My intent is that this *is* the definition. But sounds like you're expecting that definition to be in 4.7.25/4.7.26, which is probably more intuitive. I'll change that. > >> I'm avoiding commenting on the protected access changes but do want to raise a concern that there are no changes to Method resolution (5.4.3.3) yet resolution relies on the definition of access control to prune/discard inaccessible methods. It appears now that we're allowing more potential successful resolutions, then using the additional rules on the invoke* bytecodes to try and discard any undesirable results. > > Yeah, this motivated my question about MethodHandle resolution: it seems that when we talk about "resolution", we really mean the process defined in 5.4.3, followed by any linkage checks defined in 6.5. If so, this is just a presentational reshuffling (with the minor exceptions I noted in the document). > > Why bother? Because if the check on the referenced class of a protected invokevirtual/getfield/putfield is considered an access check (5.4.3.3), then the check on the referenced class of a non-private invokespecial probably ought to be considered an access check, too, but that leads to a concept of "accessible" that is really unwieldy and context-dependent. > >> 6.5 getfield (and others) >> >> You note "These rules are redundant: verification already guarantees them " but this brings me back to an area I keep raising concerns about: the split between static verification based checks and dynamic runtime checks. Yes the verifier precludes certain things but if we run without verification the rules expressed for the bytecodes are considered to be still required at runtime. If you delete them because they overlap with verifier rules we have no way to tell which rules must be enforced regardless of verification status. > > I propose removing them because on their face they appear completely redundant, an artifact of a time when we didn't have clear definitions of loading, linking, verification, etc. But if we decide it really is helpful to keep some or all of these assertions, that's fine. > > It seems to me that attempting to interpret unverified bytecodes is an implementation-specific feature that JVMS knows nothing about. Deciding which rules must be enforced in the absence of verification is an important part of designing that feature, and in an ideal world some comprehensive documentation about which verification assertions you've chosen to enforce dynamically would be maintained somewhere, but not in JVMS. If we want JVMS to maintain that list, then running without verification should be a first-class fully-specified feature. > > The status quo, I'm guessing, is that a few dynamic checks are sprinkled throughout JVMS, but plenty of other checks are performed without any supporting spec text. Which isn't a great place to be in. Okay this remains a problem area for me as I have flagged elsewhere. I had assumed the constraints in 6.x defined runtime constraints that had to be enforced regardless of whether or not verification had occurred. >> 6.5 invokespecial >> >> You note regarding the IllegalAccessError: >> >> "This replaces a VerifyError previously specified by 4.9.2. The check must be delayed until after resolution in order to determine whether the referenced method is private." >> >> We know the access flags for the referenced method at verification time - shouldn't that be the sole basis for the verifier check? Afterall it is verifying the static properties of the classfile and bytecode. If the actual resolved method has different access to the referenced method then that may lead to ICCE (depending on the exact nature of the change - a private method made public may not be an issue for example). > > Example: > > invokespecial SomeOtherClass.foo(I)V > > Verification relies on the descriptor to tell it that this method expects an int and returns nothing. Those kinds of checks can be performed locally. > > The descriptor doesn't tell you if the method is private. Only way to find out is to resolve SomeOtherClass, then resolve "foo(I)V". > > If we wanted to do the check at verification time, we'd have to load every class named by an invokespecial instruction. Now I am confused. I thought the access modifier of the method described by the Method_Ref was available to us. If it is not then how can all these verification rules be expressed in terms of non-private when we do not have the means at verification time to determine if the access is private or not ??? > (I think you might be conflating declaration-site metadata, which we have easy access to, with use-site metadata, which requires resolving references.) > >> 5.4.3.3 Method resolution >> >> You did not make any changes here, but as per my comment in the bug report we do require, IMHO, further tightening here to ensure nestmate invokespecial invocations do not resolve to method implementations that they should not. The example here is a hierarchy of nestmates (C extends B extends A) where A and C both declare a private method "void m()" and we have an invocation c.m() where c is an instance of class C. We then "separately compile" C to remove the definition of m(). At runtime method resolution will locate A.m, even though the method reference was for C.m. Normally the accessibility rules would exclude A.m from being a viable candidate but here the classes are nestmates so A.m is accessible. But it would be wrong to invoke A.m. This case should throw ICCE or NSME. > > You're taking issue with the fact that method resolution will match private methods in superclasses, when our intuition (and the Java language model) is that private methods are not inherited. > > I agree: https://bugs.openjdk.java.net/browse/JDK-8021581 Interesting - yes this is a similar problem. > Is this the right venue to address that issue? As demonstrated above, I'm happy to lump in tangentially-related bug fixes. :-) > > You could argue that this situation isn't so different from the status quo, but it does seem unique that you can delete 1 private method from a consistently-compiled program and then get a silent behavioral change. Yes. The code was written to expect C.m to execute not A.m - the code doesn't even know what A.m might do! > In practice, javac won't separately compile nestmates, so this would be hard to reproduce. More generally, the feature is not designed for separate compilation, so real-world scenarios may be hard to come by, even in other languages. Right - it is handcrafted example in the current context. But I have no idea how a non-Java language may utilise nestmates or allow separate compilation to occur. I addressed this in the prototype by ensuring that the declared class of the resolved method equals the class of the method from the method descriptor. Thanks, David > Anyway, I'm game to take another look at JDK-8021581 if we want. > > ?Dan > From daniel.smith at oracle.com Thu Apr 20 17:42:09 2017 From: daniel.smith at oracle.com (Dan Smith) Date: Thu, 20 Apr 2017 11:42:09 -0600 Subject: Nestmate same-package check In-Reply-To: <162B4430-FAFC-44DE-ABA4-941B423FEC37@oracle.com> References: <162B4430-FAFC-44DE-ABA4-941B423FEC37@oracle.com> Message-ID: <20272AEF-9478-4918-A946-6D12CFAA31DC@oracle.com> > On Apr 19, 2017, at 8:42 PM, John Rose wrote: > > On Apr 18, 2017, at 11:42 AM, Dan Smith wrote: >> I've uploaded a draft of JVMS changes for JEP 181 "Align JVM Checks with Java Language Rules for Nested Classes" to: >> http://cr.openjdk.java.net/~dlsmith/private-access.html > >> classHasValidNest(Class) :- >> >> classMemberOfNestName(Class, HostName), >> classDefiningLoader(Class, L1), >> loadedClass(HostName, L1, HostClass), >> samePackageName(Class, HostClass), >> classClassName(Class, Name), >> classNestMemberNames(HostClass, MemberNames), >> member(Name, MemberNames), >> classDefiningLoader(HostClass, L2), >> loadedClass(Name, L2, Class). > > In English: > > When validating nestmate attributes, if a class C with name > "C" has an attribute MemberOfNest("H"), it is valid only if the > following are all true: > 1. The string "H" resolves from C's loader (L1) as H. > 2. The strings "C" and "H" have the same package prefix. > 3. H contains an attribute of the form NestMembers(?"C"?). > 4. The string "C" resolves from H's loader (L2) as C. > > Steps 1 and 4 are both CONSTANT_Class resolution operations. > Step 3 is a simple search of a symbol list, which is easy. > > This logic can be simplified by testing that C and H must > be in the same runtime package. In that case, L1 and L2 are > identical, and step 4 becomes: > 4'. C and H have the same defining loader. > > Then we can refactor 2 and 4' into a more direct check on > runtime package (using the functor 'sameRuntimePackage'): > 1. The string "H" resolves from C's loader (L1) as H. > 2'. C and H have the same runtime package. > 3. H contains an attribute of the form NestMembers({... "C", ...}). > > This is a simpler specification, and it also closes any possible > loopholes in package/nest alignment by insisting on equal > runtime packages from the start. > > It also avoids a possibly wasteful class loading step, which > is only necessary if we allow L1 and L2 to differ. That would > be a pathology IMO, not a use case. > > The way I visualize this is that each newly arrived class > checks in with its host. If the name is on the list, it gets in. > > Sound good? Yep, good. I failed to notice that "samePackageName(Class, HostClass)" isn't what I want, but "sameRuntimePackage(Class, HostName)". We want a strict guarantee that these classes are in the same (runtime) package. And if we assume they're in the same package, then, yes, they have the same loaders, and so names in both classes resolve to the same thing. Revised: classHasValidNest(Class) :- classMemberOfNestName(Class, HostName), classDefiningLoader(Class, L), loadedClass(HostName, L, HostClass), classDefiningLoader(HostClass, L), samePackageName(Class, HostClass), classClassName(Class, Name), classNestMemberNames(HostClass, MemberNames), member(Name, MemberNames). ?Dan From daniel.smith at oracle.com Thu Apr 20 18:43:12 2017 From: daniel.smith at oracle.com (Dan Smith) Date: Thu, 20 Apr 2017 12:43:12 -0600 Subject: Draft JVMS changes for Nestmates In-Reply-To: <4cc09ba6-a99b-4a4e-455a-b38d3b18ba07@oracle.com> References: <524BFE31-4A61-49B3-9AE9-01B154052F96@oracle.com> <4cc09ba6-a99b-4a4e-455a-b38d3b18ba07@oracle.com> Message-ID: > On Apr 19, 2017, at 7:31 PM, David Holmes wrote: > > Hi Dan, > > On 20/04/2017 2:06 AM, Dan Smith wrote: >>> On Apr 19, 2017, at 12:45 AM, John Rose >> > wrote: >>> >>> - NMs must be non-empty (degenerate nest is never explicit) >>> - NMs may not contain duplicates (cf. treatment of ClassFile.interfaces) >>> - NMs may not contain the current class (i.e., an index to a class >>> with the same name as this_class) >>> - NMs may contain only package siblings (ditto) >>> - NMs and MoN may not refer to array classes (this is probably >>> implied by the package prefix checks) >> >> These are all do-able, but I'm not sure they're consistent with the >> spirit of JVMS in its treatment of attributes. We don't usually assert >> that lists are non-empty, don't contain duplicates, etc. (Granted, most >> attributes are not relevant to JVM execution.) You mention `interfaces`, >> but I don't see any such assertion in 4.1. > > The NestMembers duplicate check exists because I copied the spec and code from the InnerClasses attribute. 4.7.6 states the classes[] "must have exactly one corresponding entry ..." - hence a check for duplicates is done. I used the same language when I defined NestMembers. So the precedent does exist. I think you mean this? "Every CONSTANT_Class_info entry in the constant_pool table which represents a class or interface C that is not a package member must have exactly one corresponding entry in the classes array." Again, this is compiler spec territory. Which I don't mind tolerating in an attribute definition that is for Java-specific metadata and that is ignored by the JVM. But I'd like to hold critical attributes like NestMembers to a different standard. (That said, if the consensus is to check NestMembers for duplicates, I can live with it.) > On Apr 19, 2017, at 9:56 PM, David Holmes wrote: > > On 20/04/2017 7:58 AM, Dan Smith wrote: >> **4.7.25 The `MemberOfNest` Attribute** >> --------------------------------------- >> >> **The `MemberOfNest` attribute is a fixed-length attribute in the >> `attributes` table of a `ClassFile` structure ([4.1]).** >> >> **A _nest_ is a set of classes and interfaces that share access to their >> `private` members ([5.4.4]).** > > Aside: it is this basic definition of "nest" that I expected to appear somewhere earlier in the spec. JVMS doesn't have any upfront "concepts and definitions" section, so not sure where it would go. Well, there is: Chapter 2. But it doesn't mention accessibility at all, so hard to explain nests. 4.7.25, as the first place where the topic comes up, seems like the next best place. >> **A class or interface with a `MemberOfNest` attribute belongs to the >> nest hosted by a designated _host class_. The host class authorizes >> membership in the nest with a corresponding entry in its `NestMembers` >> attribute ([4.7.26], [4.10]).** > > Only nit is I would say "claims membership of" rather than "belongs to". That emphasises that membership must be ratified by the "host class", and matches the following "authorizes membership" text. Somewhere we need a definitive "class C belongs to the nest N" specification. Hence my strong language here. As you said in the other thread, if I say I belong to a nest, but it turns out I'm not authorized, that's a basic well-formedness error and I get a lethal injection. Similarly, I can claim to be my own superclass, and we might say "the superclass of C is C", but that class immediately gets killed and anybody who depends on the meaning of "superclass" can assume that it has the desired properties. > To me the "authorized" is implicit. By calling it out explicitly it suggests to me there are also UnauthorizedNestMembers. But to me this is an oxymoron - if you are unauthorized you are not an unauthorized-member, you are not any kind of member at all. An unauthorized nest member is one that gets a lethal injection. > On Apr 19, 2017, at 10:12 PM, David Holmes wrote: > >>> 4.10.1.8 Type Checking for Restricted Member References >>> >>> I'm not very good at reading or understanding these rules, but I'm >>> surprised that the invokespecial rules seem to make no mention of >>> nestmates at all: >>> >>> "A method reference is allowed by an invokespecial instruction if it >>> is not a restricted invokespecial reference, ..." >>> >>> This seems to open things up to all private method references?? >> >> Well, the whole point of nestmates is to extend, in a regular way, >> all access rules that pertain to self-access and privacy. >> If the extension is truly regular, it is not surprising that the spec. >> changes are subtle like this. We're not trying to do an ad hoc >> patch on some set of particular behaviors, but rather extend >> pre-existing notions so they apply in more conditions, including >> invokespecial restrictions. > > I was expecting to see the restrictions have a dependency on nestmates actually being involved - which is how I've currently implemented it. I can throw away some of the verifier changes if I only need to check for private access. But I'm unclear exactly where (in the spec) we enforce that only nestmates get this right to use invokespecial on private methods? Is it the access check applied during method resolution? There are two ways to do this, as we've discovered: 1) As specified, require references to non-private methods to follow the old rules, enforced by a combination of verification of the stack type for select cases (4.10.1.8) and a resolution-time check on the referenced class (6.5.invokespecial). Private invocations are exempted from these rules, and only need to satisfy accessibility (5.4.4). 2) As implemented, allow unrestricted use of invokespecial when the referenced class is a nestmate. All other methods are subject to the old rules. There are different implications about rights granted by membership in a nest and classes that need to be loaded at verification time. We'll have to explore these implications further... > On Apr 20, 2017, at 2:21 AM, David Holmes wrote: > >>>> MethodHandle resolution >>>> >>>> The spec (5.4.3.5) is vague about what errors can occur during MethodHandle resolution. I assume any linkage error specified for the instruction can also occur via MethodHandle resolution, and that will include failures due to invokespecial improperly referencing a class. >>> >>> This is an area I have least understanding of, but this seems to cover it: >>> >>> "In each step, any exception that can be thrown as a result of failure of resolution of a class or interface or field or method reference can be thrown as a result of failure of method handle resolution." >> >> The ambiguity is in the definition of "resolution". The preferred interpretation, which everyone seems to confirm reflects reality, is that it includes all linkage errors specified for the referencing instruction. I.e., errors from both both 5.4.3 and 6.5 can occur. > > Yes. 5.4.3 states that "Certain of the instructions above require additional linking checks when resolving symbolic references." And that those linking exceptions are listed with the instructions themselves. So yes resolution errors consist of the union thereof. Ah, great! I had not discovered that. Given the 5.4.3 text, we have an unambiguous definition of resolution that includes the 6.5 content. >>>> Security risk >>>> >>>> The JEP text should acknowledge that, while this does allow compilers to grant finer-grained access to members shared by nestmates, it also pushes compilers to grant broader access to members that were previously kept private. It's a trade-off, and presumably a good one because nestmates are completely trusted, while package-mates might sometimes be suspect. >>> >>> I'm not following you here. What broader access is being granted to members that were previously kept private? This effort allows private-only access to members that were previously package-accessible. >> >> Old approach is to widen access to private members to the package level on a per-member basis. >> >> New approach is to widen access to private members to the nest level on a per-class basis. >> >> Nest is narrower than package (good!), but per-class is more permissive than per-member (bad). Some members that are compiled as totally private in JDK 9 will be incidentally shared with all nest members in . > > I'm not sure how you are coming at this. For a private member in 9 to be incidentally shared in the class of the member has to be added to a nest. With no context as to how this may happen I can't really comment further. From the Java language perspective if the class is in a nest then such access to private members already exists. Example: class Outer { private int x; private int y; class Inner { private int z = x; } } In JDK 9, this compiles to: class Outer { private int x; private int y; Outer() {} static int access$000(Outer arg) { return x; } } class Outer$Inner { private int z; final Outer this$0; Outer$Inner(Outer arg) { this$0 = arg; z = Outer.access$000(this$0); } } In , this compiles to: class Outer [NestMembers:{Outer$Inner}] { private int x; private int y; Outer() {} } class Outer$Inner [MemberOfNest:Outer] { private int z; final Outer this$0; Outer$Inner(Outer arg) { this$0 = arg; z = this$0.x; } } Notice that in JDK 9, Outer shares access to y with nobody, while in , Outer shares access to y with its NestMembers. Can this expanded access to y be abused? Probably not, because Outer trusts its nestmates. But maybe there's a scenario where someone has ironclad control over the bytecode of Outer and thinks this means their 'y' data is safe, but in an attacker is able to modify the bytecode of Outer$Inner and thus reach the 'y' data. >> In an ideal world, JVMS would not acknowledge that the Java Programming Language exists. It should stand on its own, and Java is merely a client. >> >> As is, we have some history that has led to some dependencies on Java in spec, but these are generally auxiliary items: attributes that don't impact JVM behavior (4.7), a tutorial on how to compile Java code (3), occasional explanatory references to Java language concepts (e.g., 2.9). >> >> But it would be wrong for the specification of a "critical" attribute (4.7) to be defined in terms of Java language concepts. It should stand on its own, general enough to meet the needs of Java language compilers and others with different use cases. > > I think you have somewhat rose-coloured glasses on if you think the JVMS is only tainted by the Java language in "auxiliary items". :) The whole set of rules around invokespecial pertain specifically to how it supports the Java language requirements for constructor invocations and super-type invocations. Otherwise invokespecial would have remained as its original invokenonvirtual - which makes more sense to me even now. _Influenced by_ the Java language, yes, of course. But _dependent on_ the Java language, for example on the idea of nested classes, is something to avoid. If there are concepts we depend on, they should be defined in JVMS, not externally. >>> In: >>> >>> "If getfield or putfield is used to access a protected field referenced in a superclass ..." >>> >>> and following, you changed "declared" to read "referenced". Again I do not understand what this is supposed to mean. It is the declaration site of the field or method that is needed to determine whether that member is in the same or another package. To me "declared in a superclass ..." was exactly correct. Your note does not make the change any clearer to me. >> >> Good catch. It's important that the rule be limited to cases that reference a superclass*; but it's also important that the declaring class be used for the package check. >> >> (*Here's why: if I reference my subclass, and that resolves to a field declared in my superclass, the verifier will perform no check, because it may not know that the referenced class is a subclass.) >> >> Revised: >> >> "If `getfield` or `putfield` is used to access a `protected` field by reference to a superclass, and that field is declared in a different run-time package than the current class or interface, then the type of the class instance being accessed must be assignable to the current class or interface." > > I still think "declared in" is the correct terminology. There's a nuance to how you are expressing this that I'm just not getting. Example: public class p1.A { protected int i; } public class p2.B extends p1.A { int test(p2.C arg) { aload_1 getfield p2.C.i:I ireturn } } public class p2.C extends p2.B { } When the verifier checks the getfield, it will apply the rules in 4.10.1.8. The check will trivially pass, because p2.C is not a superclass of the current class (p2.B). There will be no test that p2.C is "assignable to the current class or interface". Thus, p2.C need not be loaded during verification. Strictly reading the old rule, "field declared in a superclass" means that the verifier would have to load p2.C and resolve (or simulate resolution of) p2.C.i:I in order to determine whether the field is declared in a superclass of the current class (p2.B). >>> 6.5 invokespecial >>> >>> You note regarding the IllegalAccessError: >>> >>> "This replaces a VerifyError previously specified by 4.9.2. The check must be delayed until after resolution in order to determine whether the referenced method is private." >>> >>> We know the access flags for the referenced method at verification time - shouldn't that be the sole basis for the verifier check? Afterall it is verifying the static properties of the classfile and bytecode. If the actual resolved method has different access to the referenced method then that may lead to ICCE (depending on the exact nature of the change - a private method made public may not be an issue for example). >> >> Example: >> >> invokespecial SomeOtherClass.foo(I)V >> >> Verification relies on the descriptor to tell it that this method expects an int and returns nothing. Those kinds of checks can be performed locally. >> >> The descriptor doesn't tell you if the method is private. Only way to find out is to resolve SomeOtherClass, then resolve "foo(I)V". >> >> If we wanted to do the check at verification time, we'd have to load every class named by an invokespecial instruction. > > Now I am confused. I thought the access modifier of the method described by the Method_Ref was available to us. If it is not then how can all these verification rules be expressed in terms of non-private when we do not have the means at verification time to determine if the access is private or not ??? The rule is: 1) if the referenced method name is not (cheap), and 2) if the referenced class name is the name of a superclass or direct superinterface (cheap?superclass chain is already loaded), and 3) if the loaded referenced class actually is a superclass or direct superinterface (almost always true and cheap, unless there's a name clash), and 4) if the resolved method is not private (pretty cheap, because referenced class is already loaded) *then* check that the stack type is assignable to the current class type. This is, essentially, the same logic being employed by the old protected check: if the reference is to a superclass, find the field/method and decide if it's protected; if the reference is to some other class, don't worry about it. ?Dan From david.holmes at oracle.com Thu Apr 20 23:36:16 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 21 Apr 2017 09:36:16 +1000 Subject: Draft JVMS changes for Nestmates In-Reply-To: References: <524BFE31-4A61-49B3-9AE9-01B154052F96@oracle.com> <4cc09ba6-a99b-4a4e-455a-b38d3b18ba07@oracle.com> Message-ID: <0ea78e26-a9bc-4444-fab4-8e73af0ccdd1@oracle.com> Hi Dan, On 21/04/2017 4:43 AM, Dan Smith wrote: >>>>> Security risk >>>>> >>>>> The JEP text should acknowledge that, while this does allow compilers to grant finer-grained access to members shared by nestmates, it also pushes compilers to grant broader access to members that were previously kept private. It's a trade-off, and presumably a good one because nestmates are completely trusted, while package-mates might sometimes be suspect. >>>> >>>> I'm not following you here. What broader access is being granted to members that were previously kept private? This effort allows private-only access to members that were previously package-accessible. >>> >>> Old approach is to widen access to private members to the package level on a per-member basis. >>> >>> New approach is to widen access to private members to the nest level on a per-class basis. >>> >>> Nest is narrower than package (good!), but per-class is more permissive than per-member (bad). Some members that are compiled as totally private in JDK 9 will be incidentally shared with all nest members in . >> >> I'm not sure how you are coming at this. For a private member in 9 to be incidentally shared in the class of the member has to be added to a nest. With no context as to how this may happen I can't really comment further. From the Java language perspective if the class is in a nest then such access to private members already exists. > > Example: > > class Outer { > private int x; > private int y; > class Inner { private int z = x; } > } > > In JDK 9, this compiles to: > > class Outer { > private int x; > private int y; > Outer() {} > static int access$000(Outer arg) { return x; } > } > > class Outer$Inner { > private int z; > final Outer this$0; > Outer$Inner(Outer arg) { > this$0 = arg; > z = Outer.access$000(this$0); > } > } > > In , this compiles to: > > class Outer [NestMembers:{Outer$Inner}] { > private int x; > private int y; > Outer() {} > } > > class Outer$Inner [MemberOfNest:Outer] { > private int z; > final Outer this$0; > Outer$Inner(Outer arg) { > this$0 = arg; > z = this$0.x; > } > } > > Notice that in JDK 9, Outer shares access to y with nobody, while in , Outer shares access to y with its NestMembers. Only because in JDK9 javac doesn't generate all potential accessors (which it could). The current language rules say that y is accessible to nestmates. As none try to actually access it javac doesn't bother generating the accessor. >>>> In: >>>> >>>> "If getfield or putfield is used to access a protected field referenced in a superclass ..." >>>> >>>> and following, you changed "declared" to read "referenced". Again I do not understand what this is supposed to mean. It is the declaration site of the field or method that is needed to determine whether that member is in the same or another package. To me "declared in a superclass ..." was exactly correct. Your note does not make the change any clearer to me. >>> >>> Good catch. It's important that the rule be limited to cases that reference a superclass*; but it's also important that the declaring class be used for the package check. >>> >>> (*Here's why: if I reference my subclass, and that resolves to a field declared in my superclass, the verifier will perform no check, because it may not know that the referenced class is a subclass.) >>> >>> Revised: >>> >>> "If `getfield` or `putfield` is used to access a `protected` field by reference to a superclass, and that field is declared in a different run-time package than the current class or interface, then the type of the class instance being accessed must be assignable to the current class or interface." >> >> I still think "declared in" is the correct terminology. There's a nuance to how you are expressing this that I'm just not getting. > > Example: > > public class p1.A { > protected int i; > } > > public class p2.B extends p1.A { > int test(p2.C arg) { > aload_1 > getfield p2.C.i:I > ireturn > } > } > > public class p2.C extends p2.B { > } > > When the verifier checks the getfield, it will apply the rules in 4.10.1.8. The check will trivially pass, because p2.C is not a superclass of the current class (p2.B). There will be no test that p2.C is "assignable to the current class or interface". Thus, p2.C need not be loaded during verification. > > Strictly reading the old rule, "field declared in a superclass" means that the verifier would have to load p2.C and resolve (or simulate resolution of) p2.C.i:I in order to determine whether the field is declared in a superclass of the current class (p2.B). I'm not 100% sure of what needs loading when, but don't know why this is an issue either. The package check is needed to see if the more specific protected-access check is needed. As p2.C.i actually refers to the 'i' declared in p1.A, then we should be checking if B and A are in the same package. >>>> 6.5 invokespecial >>>> >>>> You note regarding the IllegalAccessError: >>>> >>>> "This replaces a VerifyError previously specified by 4.9.2. The check must be delayed until after resolution in order to determine whether the referenced method is private." >>>> >>>> We know the access flags for the referenced method at verification time - shouldn't that be the sole basis for the verifier check? Afterall it is verifying the static properties of the classfile and bytecode. If the actual resolved method has different access to the referenced method then that may lead to ICCE (depending on the exact nature of the change - a private method made public may not be an issue for example). >>> >>> Example: >>> >>> invokespecial SomeOtherClass.foo(I)V >>> >>> Verification relies on the descriptor to tell it that this method expects an int and returns nothing. Those kinds of checks can be performed locally. >>> >>> The descriptor doesn't tell you if the method is private. Only way to find out is to resolve SomeOtherClass, then resolve "foo(I)V". >>> >>> If we wanted to do the check at verification time, we'd have to load every class named by an invokespecial instruction. >> >> Now I am confused. I thought the access modifier of the method described by the Method_Ref was available to us. If it is not then how can all these verification rules be expressed in terms of non-private when we do not have the means at verification time to determine if the access is private or not ??? > > The rule is: > 1) if the referenced method name is not (cheap), and > 2) if the referenced class name is the name of a superclass or direct superinterface (cheap?superclass chain is already loaded), and > 3) if the loaded referenced class actually is a superclass or direct superinterface (almost always true and cheap, unless there's a name clash), and > 4) if the resolved method is not private (pretty cheap, because referenced class is already loaded) > *then* check that the stack type is assignable to the current class type. > > This is, essentially, the same logic being employed by the old protected check: if the reference is to a superclass, find the field/method and decide if it's protected; if the reference is to some other class, don't worry about it. So here's the problem, in the spec 5.4 states: "Linking a class or interface involves verifying and preparing that class or interface, its direct superclass, its direct superinterfaces, and its element type (if it is an array type), if necessary. Resolution of symbolic references in the class or interface is an optional part of linking." So resolution is _optional_ at link time! But your updated spec requires resolution to happen before we can complete verification! When we do the verification of invokespecial in the VM it is before resolution and we do not know if the target method is private or not. David ----- > ?Dan > From daniel.smith at oracle.com Fri Apr 21 16:54:39 2017 From: daniel.smith at oracle.com (Dan Smith) Date: Fri, 21 Apr 2017 10:54:39 -0600 Subject: Draft JVMS changes for Nestmates In-Reply-To: <0ea78e26-a9bc-4444-fab4-8e73af0ccdd1@oracle.com> References: <524BFE31-4A61-49B3-9AE9-01B154052F96@oracle.com> <4cc09ba6-a99b-4a4e-455a-b38d3b18ba07@oracle.com> <0ea78e26-a9bc-4444-fab4-8e73af0ccdd1@oracle.com> Message-ID: <307690E8-D028-421E-B49A-E6521DCAAF96@oracle.com> > On Apr 20, 2017, at 5:36 PM, David Holmes wrote: > >> The rule is: >> 1) if the referenced method name is not (cheap), and >> 2) if the referenced class name is the name of a superclass or direct superinterface (cheap?superclass chain is already loaded), and >> 3) if the loaded referenced class actually is a superclass or direct superinterface (almost always true and cheap, unless there's a name clash), and >> 4) if the resolved method is not private (pretty cheap, because referenced class is already loaded) >> *then* check that the stack type is assignable to the current class type. >> >> This is, essentially, the same logic being employed by the old protected check: if the reference is to a superclass, find the field/method and decide if it's protected; if the reference is to some other class, don't worry about it. > > So here's the problem, in the spec 5.4 states: > > "Linking a class or interface involves verifying and preparing that class or interface, its direct superclass, its direct superinterfaces, and its element type (if it is an array type), if necessary. Resolution of symbolic references in the class or interface is an optional part of linking." > > So resolution is _optional_ at link time! But your updated spec requires resolution to happen before we can complete verification! > > When we do the verification of invokespecial in the VM it is before resolution and we do not know if the target method is private or not. In the narrow case of an invokespecial in which (1), (2), and (3) are true, yes, step (4) requires the verifier to find the declaration that is being referenced and decide if it is 'private'. This is the same as what the old 4.10.1.8 check has always required for every invokevirtual, getfield, and putfield: in narrow circumstances, track down the declared method and decide if it is or not. I've tried to be careful not to claim this process is actually resolution. It could be, but it could also be a simulation of resolution that just tells you what method would be found if resolution occurred. Specific text (tweaked slightly since published version): "Given a symbolic reference to a field or method in class _ReferencedClass_ named _MemberName_ with descriptor _MemberDescriptor_, identifies the field or method _Member_ declared in class _DeclaringClass_ that would be produced by resolution, as specified in [5.4.3]." I am curious about the actual implemented details of this in the protected check. I'm happy to make adjustments to the spec to align with how we actually do this. ?Dan From karen.kinnear at oracle.com Fri Apr 21 19:14:36 2017 From: karen.kinnear at oracle.com (Karen Kinnear) Date: Fri, 21 Apr 2017 15:14:36 -0400 Subject: Valhalla Minimal Value Types review invitation In-Reply-To: References: Message-ID: <5466821F-6EB5-41DA-8AFC-8AD8B563E89E@oracle.com> Next meeting: Wednesday April 26, 9am PT: NEW DIAL-IN: https://oracle.zoom.us/j/251372518 (you don?t need a zoom account - apologies for last meeting) Notes from the April 12, 2017 Minimal Value Types (MVT) review meeting: attendees: Ian Graves (Intel), Bjorn Vardal (IBM), Dan Heidinga (IBM), Doug Lea, Oracle: John Rose, Brian Goetz, Vladimir Ivanov, Karen Kinnear Thank you John for the update from 0.2 to 0.3 Shady values. Highlights: 1. implementation related changes (e.g. # byte codes) 2. References in value capable class (VCC)/derived value type (DVT)- leaving this open to the implementation teams as a potential extension to MVT - both Ian and Doug say they are ok with just primitives at this time 3. Simplified initial Derived Value Type to just contain instance fields, no , , no other methods, no static fields simplifies implementation, access to static fields or any methods requires boxing 4. New section on boxing, unboxing and borrowing AI: John: there were additional notes in the earlier email summary which still need to be incorporated into an update. Brief summary of incubator/experimental approach: Exploring expanding the JEP 11 Incubator Modules to allow for a small number of Incubator VM Features which would require JVM Specification changes. Key is to set expectations both with JVMS implementors of the optionality of specific incubator features, and with users that these are experimental, perhaps via command line switches and warning. Our goal is to improve our early adopter feedback cycle still within a manageable level. Early Access: Goal is to provide an early access binary early summer, so a few early adopters can give us feedback as to usefulness of the model and what is missing. The expectation is that we will have more performance optimizations to add after Early Access, and actually hope to get some use cases to guide our optimizations. Intel?s Ian Graves and Doug Lea have offered to be early adopters. Discussion: 1. Type bytecode prefix plans? Not in the MVT timeframe. AI John: to remove from Shady Supported byte codes will be: vload, vstore, vaload, vastore, vreturn, vdefault, vwithfield, vgetfield (get a field from a value type), vbox, vunbox byte codes that will allow an operand of a value type: anewarray, multianewarray 2. Interface support Yes: VCC may implement interfaces. For MVT, it will be required to box to invoke any interface methods. invoke interface may not take a QType for a receiver 3. Do we need to restrict vdefault to being called from within the owning derived value type? Answer: No. vdefault can be accessed from anywhere. vdefault does not run a user-supplied constructor. vdefault creates a value type which is all 0?s in memory which is treated as containing the default values of 0, 0.0, null, for each of its fields. This is done for both performance and security. We are defining the default value type as ?initialized? by the nullary constructor, so from a verification perspective this is considered initialized. Ed. note: expanding creation of a default value type instance: Creation of a default value type instance: which is all 0?s in memory to represent the 0 or 0.0 or null value for fields of the dvt 1) bytecode vdefault 2) anewarray/multianewarray on a derived value type - creates a value array which is all 0?s in memory representing the flattened elements of the array - which does not entail invoking any constructor on the VCC 3) vaload of an element from a value array (which then copies the contents) 3) getfield of a derived value type field (which then copies the contents) 4) vwithfield (which then copies the contents) 5) internal implementation details such as copying a DVT from registers, stack, off-heap storage, ? 4. , For MVT, the DVT does NOT have any methods including a or an Note that any creation of a default value type instance should ensure that the derived value type is ?initialized?, which implies that the VCC has run . This may be done explicitly or by construction. AI: Karen: send out an initial load/link/init proposal Open Issues: 1. java.lang.Reflect support Initial proposal is to see if we can not support reflection on derived value types, but require boxing. AI: implementors - see if the MethodHandle optimized paths require reflection for internal implementation. Early Access should give us some initial feedback from our users. 2. JVMTI support AI: Implementors - what support can we offer without changing the JVMTI APIs 3. Draft JVMS extensions AI: Dan Smith thanks, Karen p.s. a couple of notes below > On Apr 19, 2017, at 11:22 PM, John Rose wrote: > > On Apr 19, 2017, at 8:16 AM, Bjorn B Vardal > wrote: >> >> I'm writing up my notes from the meeting, and I have a question about the box's no-arg constructor. My notes say that this constructor is automatically provided and may not be replaced by the user. My understanding is that this is done in order to force the constructor to have the same behaviour as "vdefault; vunbox;?. Did you mean "vdefault;vbox;? ? As in vdefault always creates a DVT, not a VCC. > > Correct. As in C#, a value type always has a system-supplied nullary constructor > which cannot be replaced by the user. Given the ubiquitous access to "vdefault" > (by both the instruction and by sampling unassigned fields and array elements) > there is a huge potential performance cost (plus bootstrapping problems) if we > allow users to supply computed bit patterns at all variable creation points. > It's not just bit-blt, either, since if the computed patterns have non-null managed > pointers in them, the JVM is likely to have to issue extra card mark operations. > It's a mess, and not worth it, so we just say a default value looks like a fresh > object, before any of its fields are initialized. > >> My question is, did we decide to restrict the accessibility of these constructors, > > Yes, that's in the draft. > >> and if so, what would be the reason to do this? As stated above, the default box can already be obtained by unboxing a default value. Just checking that I get this - vdefault can be called from anywhere. A VCC constructor must be private - because we want to move to a factory method model. Longer term, post-MVT, valhalla value types for which we will have javac support and will derive the box from the value type - those constructors would also be private. thanks, Karen > > In fact, all constructors must be private in a VCC. The reason is to prevent > client code from calling "new VCC" directly (as "new;dup;invokespecial"). > That in turn prevents clients from creating their own fresh VCC identities. > It doesn't prevent those identities from coming into being other ways, but > makes it clear that there is no first-class API for creating a box that you can > view as "all your own?. > > Instead, VCC authors are forced to write factory methods. And those will > typically never (well, maybe rarely) make guarantees about creating fresh > object identities. > > ? John From david.holmes at oracle.com Mon Apr 24 07:42:38 2017 From: david.holmes at oracle.com (David Holmes) Date: Mon, 24 Apr 2017 17:42:38 +1000 Subject: Draft JVMS changes for Nestmates In-Reply-To: <307690E8-D028-421E-B49A-E6521DCAAF96@oracle.com> References: <524BFE31-4A61-49B3-9AE9-01B154052F96@oracle.com> <4cc09ba6-a99b-4a4e-455a-b38d3b18ba07@oracle.com> <0ea78e26-a9bc-4444-fab4-8e73af0ccdd1@oracle.com> <307690E8-D028-421E-B49A-E6521DCAAF96@oracle.com> Message-ID: <856f1eeb-c43a-2803-df7d-50e475245eb6@oracle.com> Hi Dan, On 22/04/2017 2:54 AM, Dan Smith wrote: > >> On Apr 20, 2017, at 5:36 PM, David Holmes > > wrote: >> >>> The rule is: >>> 1) if the referenced method name is not (cheap), and >>> 2) if the referenced class name is the name of a superclass or direct >>> superinterface (cheap?superclass chain is already loaded), and >>> 3) if the loaded referenced class actually is a superclass or direct >>> superinterface (almost always true and cheap, unless there's a name >>> clash), and >>> 4) if the resolved method is not private (pretty cheap, because >>> referenced class is already loaded) >>> *then* check that the stack type is assignable to the current class type. >>> >>> This is, essentially, the same logic being employed by the old >>> protected check: if the reference is to a superclass, find the >>> field/method and decide if it's protected; if the reference is to >>> some other class, don't worry about it. >> >> So here's the problem, in the spec 5.4 states: >> >> "Linking a class or interface involves verifying and preparing that >> class or interface, its direct superclass, its direct superinterfaces, >> and its element type (if it is an array type), if necessary. >> Resolution of symbolic references in the class or interface is an >> optional part of linking." >> >> So resolution is _optional_ at link time! But your updated spec >> requires resolution to happen before we can complete verification! >> >> When we do the verification of invokespecial in the VM it is before >> resolution and we do not know if the target method is private or not. > > In the narrow case of an invokespecial in which (1), (2), and (3) are > true, yes, step (4) requires the verifier to find the declaration that > is being referenced and decide if it is 'private'. > > This is the same as what the old 4.10.1.8 check has always required for > every invokevirtual, getfield, and putfield: in narrow circumstances, > track down the declared method and decide if it is package> or not. > > I've tried to be careful not to claim this process is actually > resolution. It could be, but it could also be a simulation of resolution > that just tells you what method would be found if resolution occurred. > Specific text (tweaked slightly since published version): "Given a > symbolic reference to a field or method in class _ReferencedClass_ named > _MemberName_ with descriptor _MemberDescriptor_, identifies the field or > method _Member_ declared in class _DeclaringClass_ that would be > produced by resolution, as specified in [5.4.3]." > > I am curious about the actual implemented details of this in the > protected check. I'm happy to make adjustments to the spec to align with > how we actually do this. It looks like we do the "pseudo-resolution" of the method (InstanceKlass::uncached_lookup_method()) or field (InstanceKlass::find_field()). I'm now curious about whether we essentially do this twice! That would be very inefficient. :( That aside, it seems it is okay for the spec to require this kind of "look-ahead". Thanks, David > ?Dan > From karen.kinnear at oracle.com Wed Apr 26 14:47:14 2017 From: karen.kinnear at oracle.com (Karen Kinnear) Date: Wed, 26 Apr 2017 10:47:14 -0400 Subject: Valhalla Minimal Value Types review invitation In-Reply-To: <5466821F-6EB5-41DA-8AFC-8AD8B563E89E@oracle.com> References: <5466821F-6EB5-41DA-8AFC-8AD8B563E89E@oracle.com> Message-ID: <46245130-70A5-421A-A018-39FF788FB500@oracle.com> > Next meeting: Wednesday April 26, 9am PT: > > NEW DIAL-IN: https://oracle.zoom.us/j/251372518 Rough initial cut at load/link/init proposal - one potential topic for today?s agenda. Class Load, Link, Init Behavior Goals for VCC and DVT for Minimal Value Types 1.0: MVT Assumptions: VCC can not have a nullary constructor. DVT does not have a nor an method. Behavior Goals for contained value types for load, link, init: I. Resolution of a VCC or DVT, i.e. classfile contains an LFoo or QFoo: Resolve a VCC: (LFoo) 1. load VCC annotation based: derive DVT class with an internal name - eagerly load DVT 2. link VCC does not trigger linking of DVT 3. initialization of VCC triggered by: new, static bytecodes does not trigger initialization of DVT Resolve a DVT: (QFoo) 1. load DVT first load VCC, which derives and loads DVT 2. link DVT first link VCC 3. initialize DVT first initialize VCC what triggers initialization of DVT? normally: new, static bytecodes - these are invalid for DVT vdefault vunbox anewarray/multianewarray on a DVT element type Open for Discussion: The proposal is that you must not only load a DVT element of an array, you must also link and initialize the DVT element. Otherwise you would need to link and initialize the DVT element on the first vaload, in case you did not perform a prior vastore. The verifier could ensure that you perform a prior vastore, in which case you would only need to load the DVT element of an array, not link and initialize it. II. Instance creation of a DVT, DVT has no Creation of a default value type instance: which is all 0s in memory to represent the 0 or 0.0 or null value for fields of the DVT Triggered by: 1) vdefault 2) anewarray/multianewarray on a derived value type - creates a value array which is all 0s in memory representing the flattened elements of the array - which does not entail invoking any constructor on the VCC 3) vunbox 4) internal implementation details such as copying a DVT - all of which imply that the DVT is already initialized It is required that the DVT is in the initialized state prior to the creation of a default value type instance. III. "uninitialized" value type/ partially initialized value type There is no such thing as an uninitialized value type. vdefault and anewarray/multianewarray can be invoked from anywhere. QUESTION: vwithfield: is this restricted to invocation within the DVT? (For MVT, this would also be within the VCC) That would allow the wither or instance factory to decide whether a partially initialized value type would be returned if there were an exception. IV. DVT in a Container: other object, other DVT Class contains a DVT field - QUESTION: is this supported via bytecodes? For MVT, since we do not flatten DVT fields in objects or in other DVTs, then we do not require preloading of DVT classes used to define fields. thanks, Karen > On Apr 21, 2017, at 3:14 PM, Karen Kinnear wrote: > > Next meeting: Wednesday April 26, 9am PT: > > NEW DIAL-IN: https://oracle.zoom.us/j/251372518 > (you don?t need a zoom account - apologies for last meeting) > > Notes from the April 12, 2017 Minimal Value Types (MVT) review meeting: > attendees: Ian Graves (Intel), Bjorn Vardal (IBM), Dan Heidinga (IBM), Doug Lea, > Oracle: John Rose, Brian Goetz, Vladimir Ivanov, Karen Kinnear > > > Thank you John for the update from 0.2 to 0.3 Shady values. Highlights: > 1. implementation related changes (e.g. # byte codes) > 2. References in value capable class (VCC)/derived value type (DVT)- leaving this open to the implementation teams as a potential extension > to MVT - both Ian and Doug say they are ok with just primitives at this time > 3. Simplified initial Derived Value Type to just contain instance fields, no , , no other methods, no static fields > simplifies implementation, access to static fields or any methods requires boxing > 4. New section on boxing, unboxing and borrowing > > AI: John: there were additional notes in the earlier email summary which still need to be incorporated into an update. > > Brief summary of incubator/experimental approach: > Exploring expanding the JEP 11 Incubator Modules to allow for a small number of Incubator VM Features which would > require JVM Specification changes. Key is to set expectations both with JVMS implementors of the optionality of specific incubator > features, and with users that these are experimental, perhaps via command line switches and warning. > Our goal is to improve our early adopter feedback cycle still within a manageable level. > > Early Access: > Goal is to provide an early access binary early summer, so a few early adopters can give us feedback as to > usefulness of the model and what is missing. The expectation is that we will have more performance optimizations to > add after Early Access, and actually hope to get some use cases to guide our optimizations. > Intel?s Ian Graves and Doug Lea have offered to be early adopters. > > Discussion: > 1. Type bytecode prefix plans? > Not in the MVT timeframe. > AI John: to remove from Shady > Supported byte codes will be: > vload, vstore, vaload, vastore, vreturn, vdefault, vwithfield, vgetfield (get a field from a value type), vbox, vunbox > byte codes that will allow an operand of a value type: anewarray, multianewarray > > 2. Interface support > Yes: VCC may implement interfaces. For MVT, it will be required to box to invoke any interface methods. > invoke interface may not take a QType for a receiver > > 3. Do we need to restrict vdefault to being called from within the owning derived value type? > Answer: No. vdefault can be accessed from anywhere. > > vdefault does not run a user-supplied constructor. vdefault creates a value type which is all 0?s in memory > which is treated as containing the default values of 0, 0.0, null, for each of its fields. This is done for both > performance and security. We are defining the default value type as ?initialized? by the nullary constructor, > so from a verification perspective this is considered initialized. > > Ed. note: expanding creation of a default value type instance: > > Creation of a default value type instance: which is all 0?s in memory to represent the 0 or 0.0 or null value for fields of the dvt > 1) bytecode vdefault > 2) anewarray/multianewarray on a derived value type > - creates a value array which is all 0?s in memory representing the flattened elements of the array > - which does not entail invoking any constructor on the VCC > 3) vaload of an element from a value array (which then copies the contents) > 3) getfield of a derived value type field (which then copies the contents) > 4) vwithfield (which then copies the contents) > 5) internal implementation details such as copying a DVT from registers, stack, off-heap storage, ? > > 4. , > For MVT, the DVT does NOT have any methods including a or an > Note that any creation of a default value type instance should ensure that the derived value type > is ?initialized?, which implies that the VCC has run . This may be done explicitly or by construction. > > AI: Karen: send out an initial load/link/init proposal > > Open Issues: > 1. java.lang.Reflect support > Initial proposal is to see if we can not support reflection on derived value types, but require boxing. > AI: implementors - see if the MethodHandle optimized paths require reflection for internal implementation. > Early Access should give us some initial feedback from our users. > > 2. JVMTI support > AI: Implementors - what support can we offer without changing the JVMTI APIs > > 3. Draft JVMS extensions > AI: Dan Smith > > thanks, > Karen > > p.s. a couple of notes below >> On Apr 19, 2017, at 11:22 PM, John Rose > wrote: >> >> On Apr 19, 2017, at 8:16 AM, Bjorn B Vardal > wrote: >>> >>> I'm writing up my notes from the meeting, and I have a question about the box's no-arg constructor. My notes say that this constructor is automatically provided and may not be replaced by the user. My understanding is that this is done in order to force the constructor to have the same behaviour as "vdefault; vunbox;?. > Did you mean "vdefault;vbox;? ? As in vdefault always creates a DVT, not a VCC. >> >> Correct. As in C#, a value type always has a system-supplied nullary constructor >> which cannot be replaced by the user. Given the ubiquitous access to "vdefault" >> (by both the instruction and by sampling unassigned fields and array elements) >> there is a huge potential performance cost (plus bootstrapping problems) if we >> allow users to supply computed bit patterns at all variable creation points. >> It's not just bit-blt, either, since if the computed patterns have non-null managed >> pointers in them, the JVM is likely to have to issue extra card mark operations. >> It's a mess, and not worth it, so we just say a default value looks like a fresh >> object, before any of its fields are initialized. >> >>> My question is, did we decide to restrict the accessibility of these constructors, >> >> Yes, that's in the draft. >> >>> and if so, what would be the reason to do this? As stated above, the default box can already be obtained by unboxing a default value. > Just checking that I get this - vdefault can be called from anywhere. > A VCC constructor must be private - because we want to move to a factory method > model. > Longer term, post-MVT, valhalla value types for which we will have javac support and > will derive the box from the value type - those constructors would also be private. > > thanks, > Karen >> >> In fact, all constructors must be private in a VCC. The reason is to prevent >> client code from calling "new VCC" directly (as "new;dup;invokespecial"). >> That in turn prevents clients from creating their own fresh VCC identities. >> It doesn't prevent those identities from coming into being other ways, but >> makes it clear that there is no first-class API for creating a box that you can >> view as "all your own?. >> >> Instead, VCC authors are forced to write factory methods. And those will >> typically never (well, maybe rarely) make guarantees about creating fresh >> object identities. > >> >> ? John From karen.kinnear at oracle.com Wed Apr 26 19:49:09 2017 From: karen.kinnear at oracle.com (Karen Kinnear) Date: Wed, 26 Apr 2017 15:49:09 -0400 Subject: MVT April 26th notes In-Reply-To: <46245130-70A5-421A-A018-39FF788FB500@oracle.com> References: <5466821F-6EB5-41DA-8AFC-8AD8B563E89E@oracle.com> <46245130-70A5-421A-A018-39FF788FB500@oracle.com> Message-ID: <4C018C64-E773-499C-831D-804529B8EB88@oracle.com> Next meeting May 10th: dial-in: https://oracle.zoom.us/j/251372518 Notes from April 26: Attendees: Remi, Dan H, Bjorn, Doug, John, Vladimir, Frederic, Karen AIs: 1. John - please update Shady 2. All - please review email rough proposal for load/link/init behavior for MVT 3. Karen: work with Dan Smith to get him initial proposals for JVMS extensions for new bytecodes for chapters 4 and 6 and load/link/init for chapter 5 get early drafts distributed for more general early review 1. Bjorn: Concern about new bytecodes - concern about limited available bytecodes for future - concern about tooling ecosystem - would it be better if we moved to an expected preferred form sooner, i.e. typed bytecodes? Responses: Karen, Bjorn: both VM implementations use some bytecodes for internal quickening Frederic: with typed bytecodes, the constant pool changes become more complex, which also would be seen by the tooling ecosystem. Remi: what if we had an experimental prefix? John: Even if we moved to typed bytecodes today, we would still expect changes note: values are inherently polymorphic post MVT - e.g. interface support looking at a 6th BasicType. Interpreter will need to have full type information with the type, 2. Karen: MVT interface support Last time we agreed that we need to support interfaces for MVT Can we assume VCC will implement the interface and the DVT will NOT implement the interface Response: John: yes. 3. Karen: Do we have a need for a QFoo in a field descriptor? We support QFoo in a method descriptor as well as [QFoo in a field descriptor. Concern: reflection and tooling exposure to value types Lesser concern: implementation complexity Responses: Remi: If play with optional, want to store in a field (note: not supported today) Remi: monad - not want to pay abstraction Frederic: workaround is to use an array of 1 element given we are not flattening, there are no density or lack of indirection benefits of supporting Doug: Ask Scala folks or Martin Hadusky (sp?) John: propose as side project - i.e. target of opportunity, but not planned for MVT Bjorn, Karen: have experimented with it, ok as side project 4. new root type for values We needed this internally for LambdaForms to avoid specializing per value type. Initially we are calling this java/lang/__Value At this point this is not a global root, but a root for value types only Exposing this so that MethodHandles can use this is a big step. Responses: Packed Objects - initially were in a separate hierarchy, this was painful, so later rolled back in Doug: "any" as a top type, folks will want exposure. This will change rules for dispatch and overload - looks inevitable John: plan for longer term - If we were to add a QObject, which is a top type for all QTypes then a UType which is a tagged union of Q and L types (and maybe primitives?) we could perhaps expose LTypes with a different tag bit to the user note: LTypes can be null QTypes - not nullable, no guaranteed identity, immutable UType - might be nullable, or it might have identity - don't count on it, but must preserve it if it carries an LType Doug: could do monads with this scheme, could help switches and pattern matching Remi: with Value Types already have tuples, do you still need a UType? 5. question from last tim about verifier checking for complete instances of values before letting them go. John's answer: No - verification is not needed here. See next update of Shady. [ed note: see load/link/init email proposal - I think we are in agreement that there is no such thing as an uninitialized value type] 6. Packed Objects - See email from Bjorn on Packed Objects vs. Minimal Value Types Bjorn walked us through highlights - email questions welcome minor notes: MVT allows off heap allocation today Keep an eye on Method Builder recipes with bootstramethods to help with issues like generating equals, hashcode etc. boilerplate Questions: How did customers for PackedObjects repond to some of the accidental identity issues? - expectations of Object behaviors were a surprise, e.g. test cases failed due to accidental updates to a PackedObject - in general customers got good results - e.g. power users explicitly controlled layout John: Look into Panama experiments on persistent memory data structures 7. Nestmates Can jvmti retransform change nestmate information? Response: NO Can you grow the membership list? Not via retransform, run into issues if the member has already been loaded. Remi: need to dynamically add to the nest Karen: yes - if the host has defined a rule rather than a list (more advanced) John: yes - see Lookup.defineClass - initially in 9, extending in future to allow temporary classes as well as adding nest members (ed. note - thank you for the reminder John - using the Lookup Private access mode as the author allowing classes as they are defined to be added to a nest) Constant pool (CP) patching plans? John:goal is to not patch the CP, but rather find a way to pass user data Dan: happier with a patch to a single field rather than CP extensions John: head's up on another issue exp[loring - thread confinement - how to ensure that we can protect thread bound data - e.g. ByteBuffers in Panama - want first accessor as owner, other accessors, even readers to get an error rathern than block or race thanks, Karen > On Apr 26, 2017, at 10:47 AM, Karen Kinnear wrote: > >> Next meeting: Wednesday April 26, 9am PT: >> >> NEW DIAL-IN: https://oracle.zoom.us/j/251372518 > Rough initial cut at load/link/init proposal - one potential topic for today?s agenda. > > Class Load, Link, Init Behavior Goals for VCC and DVT for Minimal Value Types 1.0: > > MVT Assumptions: > VCC can not have a nullary constructor. > DVT does not have a nor an method. > > Behavior Goals for contained value types for load, link, init: > > I. Resolution of a VCC or DVT, i.e. classfile contains an LFoo or QFoo: > Resolve a VCC: (LFoo) > 1. load VCC > annotation based: derive DVT class with an internal name - eagerly load DVT > 2. link VCC > does not trigger linking of DVT > 3. initialization of VCC > triggered by: new, static bytecodes > does not trigger initialization of DVT > > Resolve a DVT: (QFoo) > 1. load DVT > first load VCC, which derives and loads DVT > > 2. link DVT > first link VCC > > 3. initialize DVT > first initialize VCC > what triggers initialization of DVT? > normally: new, static bytecodes - these are invalid for DVT > vdefault > vunbox > anewarray/multianewarray on a DVT element type > > Open for Discussion: > The proposal is that you must not only load a DVT element of an array, you must also link and > initialize the DVT element. > Otherwise you would need to link and initialize the DVT element on the first vaload, > in case you did not perform a prior vastore. > The verifier could ensure that you perform a prior vastore, in which case you would only > need to load the DVT element of an array, not link and initialize it. > > II. Instance creation of a DVT, DVT has no > Creation of a default value type instance: which is all 0s in memory to represent > the 0 or 0.0 or null value for fields of the DVT > > Triggered by: > 1) vdefault > 2) anewarray/multianewarray on a derived value type > - creates a value array which is all 0s in memory representing the flattened > elements of the array > - which does not entail invoking any constructor on the VCC > 3) vunbox > 4) internal implementation details such as copying a DVT - all of which imply that > the DVT is already initialized > It is required that the DVT is in the initialized state prior to the creation of a > default value type instance. > > > III. "uninitialized" value type/ partially initialized value type > There is no such thing as an uninitialized value type. > vdefault and anewarray/multianewarray can be invoked from anywhere. > > QUESTION: vwithfield: is this restricted to invocation within the DVT? > (For MVT, this would also be within the VCC) > That would allow the wither or instance factory to decide whether > a partially initialized value type would be returned if there were > an exception. > > > IV. DVT in a Container: other object, other DVT > Class contains a DVT field > - QUESTION: is this supported via bytecodes? > > For MVT, since we do not flatten DVT fields in objects or in other DVTs, then we > do not require preloading of DVT classes used to define fields. > > thanks, > Karen > > >> On Apr 21, 2017, at 3:14 PM, Karen Kinnear > wrote: >> >> Next meeting: Wednesday April 26, 9am PT: >> >> NEW DIAL-IN: https://oracle.zoom.us/j/251372518 >> (you don?t need a zoom account - apologies for last meeting) >> >> Notes from the April 12, 2017 Minimal Value Types (MVT) review meeting: >> attendees: Ian Graves (Intel), Bjorn Vardal (IBM), Dan Heidinga (IBM), Doug Lea, >> Oracle: John Rose, Brian Goetz, Vladimir Ivanov, Karen Kinnear >> >> >> Thank you John for the update from 0.2 to 0.3 Shady values. Highlights: >> 1. implementation related changes (e.g. # byte codes) >> 2. References in value capable class (VCC)/derived value type (DVT)- leaving this open to the implementation teams as a potential extension >> to MVT - both Ian and Doug say they are ok with just primitives at this time >> 3. Simplified initial Derived Value Type to just contain instance fields, no , , no other methods, no static fields >> simplifies implementation, access to static fields or any methods requires boxing >> 4. New section on boxing, unboxing and borrowing >> >> AI: John: there were additional notes in the earlier email summary which still need to be incorporated into an update. >> >> Brief summary of incubator/experimental approach: >> Exploring expanding the JEP 11 Incubator Modules to allow for a small number of Incubator VM Features which would >> require JVM Specification changes. Key is to set expectations both with JVMS implementors of the optionality of specific incubator >> features, and with users that these are experimental, perhaps via command line switches and warning. >> Our goal is to improve our early adopter feedback cycle still within a manageable level. >> >> Early Access: >> Goal is to provide an early access binary early summer, so a few early adopters can give us feedback as to >> usefulness of the model and what is missing. The expectation is that we will have more performance optimizations to >> add after Early Access, and actually hope to get some use cases to guide our optimizations. >> Intel?s Ian Graves and Doug Lea have offered to be early adopters. >> >> Discussion: >> 1. Type bytecode prefix plans? >> Not in the MVT timeframe. >> AI John: to remove from Shady >> Supported byte codes will be: >> vload, vstore, vaload, vastore, vreturn, vdefault, vwithfield, vgetfield (get a field from a value type), vbox, vunbox >> byte codes that will allow an operand of a value type: anewarray, multianewarray >> >> 2. Interface support >> Yes: VCC may implement interfaces. For MVT, it will be required to box to invoke any interface methods. >> invoke interface may not take a QType for a receiver >> >> 3. Do we need to restrict vdefault to being called from within the owning derived value type? >> Answer: No. vdefault can be accessed from anywhere. >> >> vdefault does not run a user-supplied constructor. vdefault creates a value type which is all 0?s in memory >> which is treated as containing the default values of 0, 0.0, null, for each of its fields. This is done for both >> performance and security. We are defining the default value type as ?initialized? by the nullary constructor, >> so from a verification perspective this is considered initialized. >> >> Ed. note: expanding creation of a default value type instance: >> >> Creation of a default value type instance: which is all 0?s in memory to represent the 0 or 0.0 or null value for fields of the dvt >> 1) bytecode vdefault >> 2) anewarray/multianewarray on a derived value type >> - creates a value array which is all 0?s in memory representing the flattened elements of the array >> - which does not entail invoking any constructor on the VCC >> 3) vaload of an element from a value array (which then copies the contents) >> 3) getfield of a derived value type field (which then copies the contents) >> 4) vwithfield (which then copies the contents) >> 5) internal implementation details such as copying a DVT from registers, stack, off-heap storage, ? >> >> 4. , >> For MVT, the DVT does NOT have any methods including a or an >> Note that any creation of a default value type instance should ensure that the derived value type >> is ?initialized?, which implies that the VCC has run . This may be done explicitly or by construction. >> >> AI: Karen: send out an initial load/link/init proposal >> >> Open Issues: >> 1. java.lang.Reflect support >> Initial proposal is to see if we can not support reflection on derived value types, but require boxing. >> AI: implementors - see if the MethodHandle optimized paths require reflection for internal implementation. >> Early Access should give us some initial feedback from our users. >> >> 2. JVMTI support >> AI: Implementors - what support can we offer without changing the JVMTI APIs >> >> 3. Draft JVMS extensions >> AI: Dan Smith >> >> thanks, >> Karen >> >> p.s. a couple of notes below >>> On Apr 19, 2017, at 11:22 PM, John Rose > wrote: >>> >>> On Apr 19, 2017, at 8:16 AM, Bjorn B Vardal > wrote: >>>> >>>> I'm writing up my notes from the meeting, and I have a question about the box's no-arg constructor. My notes say that this constructor is automatically provided and may not be replaced by the user. My understanding is that this is done in order to force the constructor to have the same behaviour as "vdefault; vunbox;?. >> Did you mean "vdefault;vbox;? ? As in vdefault always creates a DVT, not a VCC. >>> >>> Correct. As in C#, a value type always has a system-supplied nullary constructor >>> which cannot be replaced by the user. Given the ubiquitous access to "vdefault" >>> (by both the instruction and by sampling unassigned fields and array elements) >>> there is a huge potential performance cost (plus bootstrapping problems) if we >>> allow users to supply computed bit patterns at all variable creation points. >>> It's not just bit-blt, either, since if the computed patterns have non-null managed >>> pointers in them, the JVM is likely to have to issue extra card mark operations. >>> It's a mess, and not worth it, so we just say a default value looks like a fresh >>> object, before any of its fields are initialized. >>> >>>> My question is, did we decide to restrict the accessibility of these constructors, >>> >>> Yes, that's in the draft. >>> >>>> and if so, what would be the reason to do this? As stated above, the default box can already be obtained by unboxing a default value. >> Just checking that I get this - vdefault can be called from anywhere. >> A VCC constructor must be private - because we want to move to a factory method >> model. >> Longer term, post-MVT, valhalla value types for which we will have javac support and >> will derive the box from the value type - those constructors would also be private. >> >> thanks, >> Karen >>> >>> In fact, all constructors must be private in a VCC. The reason is to prevent >>> client code from calling "new VCC" directly (as "new;dup;invokespecial"). >>> That in turn prevents clients from creating their own fresh VCC identities. >>> It doesn't prevent those identities from coming into being other ways, but >>> makes it clear that there is no first-class API for creating a box that you can >>> view as "all your own?. >>> >>> Instead, VCC authors are forced to write factory methods. And those will >>> typically never (well, maybe rarely) make guarantees about creating fresh >>> object identities. >> >>> >>> ? John > From brian.goetz at oracle.com Wed Apr 26 21:46:24 2017 From: brian.goetz at oracle.com (Brian Goetz) Date: Wed, 26 Apr 2017 17:46:24 -0400 Subject: Fwd: JEP 303: Intrinsics for the LDC and INVOKEDYNAMIC Instructions In-Reply-To: References: Message-ID: The following was received through the valhalla-spec-comments mailbox. -------- Forwarded Message -------- Subject: JEP 303: Intrinsics for the LDC and INVOKEDYNAMIC Instructions Date: Wed, 26 Apr 2017 23:24:40 +0200 From: Jeremy Barrow To: valhalla-spec-comments at openjdk.java.net I'll preface this with: I know that making a language construct for this has already been set as an explicit non-goal. If you're 100% set on that, then you can probably skip this email, but if you're interested in the opinions of some random who had a free afternoon, then read on. For the same reason that collection literals are a bad idea within the language; compiler intrinsics are a bad idea within the library. Assuming a class gets added to the standard library, how would other languages react to that? Are they going to have to implement some support for it, or instead explain to their userbase that some portion of the library is redundant, more so if the language already supports language level intrinsics. This would be the perfect time to finally use "const". const MethodType type = ...; (Side-note: this might start dipping into ConstantDynamic (condyn) territory, but bear with me.) The idea would allow the programmer to declare a field or a local variable with the const modifier. When the programmer does this, it should be treated as a compile time constant. As such, it doesn't make sense to be able to be able to declare a const field, as a static final const field. This might affect readability, and should probably be reviewed, but semantically, it makes sense. If the field/variable is used in a non-const context, then it should be treated as if it was written there. (Excluding some types, such as strings, classes, etc) Because it's a compile time constant, it raises some questions: Should the source code contain something that won't be present in the final runtime class? I think it makes sense, as it's not the only thing that java can omit from the runtime class. Annotations can have a non-runtime retention, so I don't think the concept is a shock to java programmers, and arguably, a field/local variable with const is easier to spot than a non-runtime annotation (because you have to inspect the annotation's class). I was originally going to also suggest maybe having support for const methods as well, and variables and parameters can only be const, etc, but I think that's far too much for this simple API, and would play very well into condyn's implementation. Should the compiler be intrinsically aware of the type, or be able to be made aware? (Intrinsic types or some way to communicate to the compiler how to convert a type into a constant) This is dangerously close to condyn, but I felt the need to bring it up just to show how well the two could go together. With this API, I think, initially, intrinsic types are perfect. Higher level types that can be known to the compiler, such as MethodType, Class, String, are more or less perfect for it. Now, the main problem: You can't declare a method with those constants without an intrinsic type, or if you could, it might not be too readable. That being said, I may have overlooked a nice way to declare methods with constants. The first solution that came to mind was representing MethodHandle as a const. Currently, there's no explicit creation mechanism, outside of the Lookup system itself, for obvious reasons. Should there be a const factory method for MethodHandles, that can only accept const parameters? We have method references, could we use them? I spent some time thinking about this, and it almost always boiled down to an intrinsic class. As even if we can define a MethodHandle in a const manner, we still have to deal with the compiler needing to be aware of the point where it needs to emit the invokedynamic instruction. In effect, const is a way to define the constants, not the instructions. I think that was the shortcoming, and having something at the language level for bytecode specific instructions is a potentially toxic idea. Now, all that being said, and the final result is in more or less the exact same position as before, I have suggested using const. Which means we can declare new semantics. What if we could define classes that _have_ to be entirely constant? What if we could attach a @Retention to it, or define a RetentionPolicy for it? This means the JVM could toss the class, much like it does annotations. This could make the way for a whole section of programmable constants later on with condyn. I will admit, this got much bigger than I originally planned, but hopefully, I got across my point, if I failed, I at least hope that I gave someone a good idea to work it. Thanks for reading this wall of text.