From daniel.smith at oracle.com Sat Dec 2 00:41:28 2017 From: daniel.smith at oracle.com (Dan Smith) Date: Fri, 1 Dec 2017 17:41:28 -0700 Subject: Final nestmates spec Message-ID: <808AE338-72C4-49DA-A4A1-3441BFB8AB83@oracle.com> I've uploaded what I believe could be a final spec for nestmates, here: http://cr.openjdk.java.net/~dlsmith/nestmates.html Since October, changes include: - Tweaks to the explanatory material in 2.11.8 and 3.7 - Some restructuring of the new 5.4.4 text to be a little less disruptive and more clearly define "same nest" and "nest host". There are some subtle behavioral differences compared to the previous iteration when errors occur. - Added an example to 5.4.5 to illustrate transitive overriding ?Dan From john.r.rose at oracle.com Sat Dec 2 00:52:59 2017 From: john.r.rose at oracle.com (John Rose) Date: Fri, 1 Dec 2017 16:52:59 -0800 Subject: Final nestmates spec In-Reply-To: <808AE338-72C4-49DA-A4A1-3441BFB8AB83@oracle.com> References: <808AE338-72C4-49DA-A4A1-3441BFB8AB83@oracle.com> Message-ID: <99864909-94DC-4792-B09B-954B7FB0E073@oracle.com> On Dec 1, 2017, at 4:41 PM, Dan Smith wrote: > > I've uploaded what I believe could be a final spec for nestmates, here: > http://cr.openjdk.java.net/~dlsmith/nestmates.html > > Since October, changes include: > > - Tweaks to the explanatory material in 2.11.8 and 3.7 > > - Some restructuring of the new 5.4.4 text to be a little less disruptive and more clearly define "same nest" and "nest host". There are some subtle behavioral differences compared to the previous iteration when errors occur. > > - Added an example to 5.4.5 to illustrate transitive overriding > > ?Dan I suggest changing the references to 18.3 to 10, based on today's information copied below. ? John http://mail.openjdk.java.net/pipermail/jdk-dev/2017-December/000301.html Begin forwarded message: > > From: mark.reinhold at oracle.com > Subject: Re: Draft JEP: Time-Based Release Versioning > Date: December 1, 2017 at 7:59:47 AM PST > To: volker.simonis at gmail.com > Cc: jdk-dev at openjdk.java.net > > 2017/12/1 3:11:18 -0800, volker.simonis at gmail.com : >> thanks for publishing the draft. Overall it looks good! >> >> I have just a few comment :) >> >> 1. Is thisJEP (i.e. the new version scheme) intended to be targeted >> for Java 10? > > It's intended to be targeted when it's ready, just like any other JEP. > It's obviously desirable to have it in 10, but if it doesn't make it > then it'll be in 11. > >> I would appreciate to have it in ten but aren't we already quite late? >> This is specification relevant (i.e. has to go into JSR 383) because >> it changes java.lang.Runtime.Version and various standard system >> properties which refer to it. JSR 383 has to be renamed from "Java SE >> 18.3 Platform JSR" to "Java SE 10 Platform JSR" afterwards, right? > > That should be done anyway, regardless of whether this JEP makes it into > JDK 10. If we change nothing in JDK 10, as it stands today, then the > JDK 10 GA release will identify itself as 10, not 18.3. > > From john.r.rose at oracle.com Sat Dec 2 02:48:02 2017 From: john.r.rose at oracle.com (John Rose) Date: Fri, 1 Dec 2017 18:48:02 -0800 Subject: stricter symbolic references to protected methods In-Reply-To: <808AE338-72C4-49DA-A4A1-3441BFB8AB83@oracle.com> References: <808AE338-72C4-49DA-A4A1-3441BFB8AB83@oracle.com> Message-ID: <132AB27A-6B2B-452A-9D2A-14250FE31F06@oracle.com> (This came up while reviewing the nestmates stuff. I'm sending it in its own thread, FTR.) Here's a question to mull over for the future. It belongs in the "strict VM" bucket of ideas, which I have occasionally spoken about. There is a lot of (IMO needless) complexity in the JVM dealing with a symbolic reference from a class D to a protected member C.m/I.m of a super C/I outside of D's package. I think such references can and should be normalized specifically to D's immediate super, whether that is C or not. Could we do this? I think it would simplify JVMS 5.4.4, at a point where it is rather opaque. This is the language I'm talking about: > ? R is protected and is declared in a class C, and D is either a subclass of C or C itself. Furthermore, if R is not static, then the symbolic reference to R must contain a symbolic reference to a class T, such that T is either a subclass of D, a superclass of D, or D itself. > > During verification, it was also required that, even if T is a superclass of D, the target reference of a protected instance field access or method invocation must be an instance of D or a subclass of D (4.10.1.8). We could replace it by this: > ? R is protected and is declared in a class C *not in the same run-time package as D*, and D is *a subclass of C*. Furthermore the symbolic reference to R must contain a symbolic reference to *the immediate superclass of D*. > > During verification, it was also required that the target reference of a protected instance field access or method invocation must be an instance of D or a subclass of D (4.10.1.8). The existing complexities are due IMO to historical mis-steps (as recognized in hindsight) in the way symbolic references were formulated. This is the ancient 1.0 time frame, before we invented ACC_SUPER and when miranda methods were a new thing. We've come a ways since then; perhaps we can make use of our knowledge. ? John From john.r.rose at oracle.com Sat Dec 2 03:21:58 2017 From: john.r.rose at oracle.com (John Rose) Date: Fri, 1 Dec 2017 19:21:58 -0800 Subject: Final nestmates spec In-Reply-To: <99864909-94DC-4792-B09B-954B7FB0E073@oracle.com> References: <808AE338-72C4-49DA-A4A1-3441BFB8AB83@oracle.com> <99864909-94DC-4792-B09B-954B7FB0E073@oracle.com> Message-ID: <688F8EEE-CEBB-40DC-A725-BA147EB6338E@oracle.com> I read the whole thing with care. It's great; ship it. I do have some editorial comments and nitpicks. There's one substantive change, suggested tentatively. s/18.3/10/ +"It authorizes an enumerated set classes and interfaces to claim membership" s/ set / set of / I notice that we have minimized the structural requirements on nestmate attributes are minimized. That's good, but it means there is a certain practical uncertainty about what can be in them. For example, the NestHost attribute can point to a crazy class like java.lang.Object, and this only gets diagnosed during access control (5.4.4). We've agreed this is the right thing to do. Likewise, there can be garbage in the NestMembers list, and nothing bad will happen, at least until reflection tries to reify it. I suggest a non-normative comment guiding users about the intended contents of these attributes: ++"The NestHost and NestMembers attributes of a class D are used to make symbolic references to classes and interfaces in the same run-time package as D. References to other types are ineffective under the rules of 5.4.1.1 and should not be introduced. Similarly, a nest host H should not list itself in its NestMembers attribute, although this will have no effect under the rules of 5.4.1.1." (The rules call for loading the NestHost H even if it is not in the same package as the referring class M. This is the only substantive thing I feel slightly uncomfortable about in the spec., and it's not enough to demand a change. But it's there.) "? R is protected and is declared in a class C, and D is either a subclass of C or C itself. Furthermore, if R is not static, then the symbolic reference to R must contain a symbolic reference to a class T, such that T is either a subclass of D, a superclass of D, or D itself." + "During verification, it was also required that, even if T is a superclass of D, the target reference of a protectedinstance field access or method invocation must be an instance of D or a subclass of D (4.10.1.8)." I'm glad to see this addition, even if it's non-normative. The web of checks that collectively enforce protected access is not well explained in the JVMS, because they are scattered in several places. (See also my separate message on stricter symbolic references.) If new subsections are desirable, one could be introduced here, for ease of cross referencing: +"If a referenced field or method is not accessible, access checking throws an IllegalAccessError. If an exception is thrown while attempting to determine the nest host of a class or interface, access checking fails for the same reason." ++"5.4.4.1 Nest Membership Checking" +"To determine whether a class or interface C belongs to the same nest as D, the following steps are performed:" This feels slightly ambiguous: "2. Otherwise, where i is the host_class_index item of M's NestHost attribute, the symbolic reference at index i of M's run-time constant pool is resolved to a class or interface H" Often when describing resolution it is obvious from context what is the resolving class, but sometimes it needs to be explicitly stated. Is this a place that needs to be more explicit? Suggest: s/ is resolved / is resolved from M / Or is there a place elsewhere that makes it really clear that a symbolic reference in a CP of M is always resolved relative to M? (Such questions might become more urgent when we have disembodied symbolic references created by the Constable API.) At this point comes the only substantive change I am tempted to suggest: +"2. ? Otherwise, where i is the host_class_index item of M's NestHost attribute, the symbolic reference at index i of M's run-time constant pool is resolved to a class or interface H (5.4.3.1). Any of the exceptions pertaining to class or interface resolution can be thrown." s/H/H in the same runtime package as M/ s/Any/Resolution is suppressed if the symbolic reference is to a package other than that of M. Any/ +"3. ? If resolution of H succeeds, but H is not declared in the same run-time package as M, an IncompatibleClassChangeErroris thrown." s/as M/as M, or if resolution was suppressed in step 2/ I'm pretty sure we've discussed this before, and although I am not in favor of deep checks on the new attributes, this one would have the most benefit, IMO, since it localizes the side effects from a bad class file to resolutions on that class file itself. Here's another possible subsection could be introduced for cross references: ++"5.4.5.1 Method Selection" +"Where an invokevirtual or an invokeinterface invocation refers to a resolved method mR, the selected method of the invocation for an instance of a class or interface C is determined as follows:" (Alternatively, we could just remember, or even point out, that overriding prepares for selection, and is useless without it.) Later on, there are several references to "step 3 of the lookup procedure" which might read better in the context of a sharper reference (5.4.5.1) to that lookup procedure. /Adresses JDK-8098577/s/Ad/Add/ +"3. Otherwise, if there is exactly one maximally-specific method (5.4.3.3) in the superinterfaces of C that matches the resolved method's name and descriptor and is not abstract, then it is the selected method." There's no use of the concept "can override" in point 3, which is the only place where selection of super-interface methods occurs. I assume that this is because the can-override relation is not relevant at this point, but it feels like a narrative loose end. If I read the logic right, the only way to select a method here is if the symbolic reference already resolved to a public interface method, so the can-override relation is actually true, regardless of what the 5.4.3.3 process produces. But if it's true, the proof is subtle and non-local. I would like to be assured of this fact in the spec. itself. Failing that, I think there is room for future improvement at this point, by saying that the can-override relation *is enforced* at step 3, and allow JVMs to prove that the enforcement is a nop. Thanks for the outstanding spec. work. ? John From daniel.smith at oracle.com Tue Dec 5 00:32:49 2017 From: daniel.smith at oracle.com (Dan Smith) Date: Mon, 4 Dec 2017 17:32:49 -0700 Subject: Final nestmates spec In-Reply-To: <688F8EEE-CEBB-40DC-A725-BA147EB6338E@oracle.com> References: <808AE338-72C4-49DA-A4A1-3441BFB8AB83@oracle.com> <99864909-94DC-4792-B09B-954B7FB0E073@oracle.com> <688F8EEE-CEBB-40DC-A725-BA147EB6338E@oracle.com> Message-ID: <25A5B940-49CB-4083-960C-DE097D1C053A@oracle.com> Actioned. Notes on nontrivial suggestions below: > On Dec 1, 2017, at 8:21 PM, John Rose wrote: > > ++"The NestHost and NestMembers attributes of a class D are used > to make symbolic references to classes and interfaces in the same > run-time package as D. References to other types are ineffective under > the rules of 5.4.1.1 and should not be introduced. Similarly, a nest host > H should not list itself in its NestMembers attribute, although this > will have no effect under the rules of 5.4.1.1." Good suggestion, but where to put it? I ended up with this: Under `NestHost`: **`host_class_index`** : **The value of the `host_class_index` item must be a valid index into the `constant_pool` table. The `constant_pool` entry at that index must be a `CONSTANT_Class_info` structure ([4.4.1]) representing a class or interface.** > **If the referenced class cannot be loaded, does not belong to the same run-time package, or does not authorize nest membership, an error may occur during access checking ([5.4.4]).** And under `NestMembers`: **`classes[]`** : **Each value in the `classes` array must be a valid index into the `constant_pool` table. The `constant_pool` entry at that index must be a `CONSTANT_Class_info` structure ([4.4.1]) representing a class or interface.** > **This array is consulted during access checking ([5.4.4]). It should consist of references to other classes and interfaces that belong to the same run-time package and have `NestHost` attributes referencing this class or interface. Items that do not meet this description are discouraged and will be ignored by access checking.** > (The rules call for loading the NestHost H even if it is not in the same > package as the referring class M. This is the only substantive thing > I feel slightly uncomfortable about in the spec., and it's not enough > to demand a change. But it's there.) Yes. Discussed this with David and Alex, and we tried a few different iterations before settling on this. A key observation is that, if we eagerly detect that the class can't possibly be a packagemate, we can quickly return "false", but then an IAE will occur. It's important to avoid loading in the fast/common path, but when you're throwing errors you're in the slow/exceptional path. So the extra complexity of a special-case rule doesn't seem justified. Stepping back, I asked myself the question, "what good would an extra pre-check do?", and couldn't come up with anything. It's not like the ability to ask to load classes is a closely-guarded right. > "? R is protected and is declared in a class C, and D is either a subclass of C or C itself. Furthermore, if R is not static, then the symbolic reference to R must contain a symbolic reference to a class T, such that T is either a subclass of D, a superclass of D, or D itself." > + "During verification, it was also required that, even if T is a superclass of D, the target reference of a protectedinstance field access or method invocation must be an instance of D or a subclass of D (4.10.1.8)." > > I'm glad to see this addition, even if it's non-normative. The web of > checks that collectively enforce protected access is not well explained > in the JVMS, because they are scattered in several places. > (See also my separate message on stricter symbolic references.) It's not new, but I rearranged and rephrased it in a way that is hopefully clearer. :-) > ++"5.4.4.1 Nest Membership Checking" I'll add a note and leave it to Alex to decide. I think he may not like singleton subsections... > Or is there a place elsewhere that makes it really clear that a symbolic > reference in a CP of M is always resolved relative to M? (Such questions > might become more urgent when we have disembodied symbolic > references created by the Constable API.) It is always the case, yes, and I don't think the spec does anything to suggest that this is even a well-formed question. That is, symbolic references can only be resolved one way, and the class in which they occur seems to be an implicit part of them. For example, 'checkcast' invokes resolve(ref), which immediately starts talking about 'D', the class in which the reference occurs (5.4.3.1). > ++"5.4.5.1 Method Selection" Noted. > +"3. Otherwise, if there is exactly one maximally-specific method (5.4.3.3) in the superinterfaces of C that matches the resolved method's name and descriptor and is not abstract, then it is the selected method." > There's no use of the concept "can override" in point 3, which is the only place where selection > of super-interface methods occurs. I assume that this is because the can-override > relation is not relevant at this point, but it feels like a narrative loose end. > > If I read the logic right, the only way to select a method here is > if the symbolic reference already resolved to a public interface > method, so the can-override relation is actually true, regardless > of what the 5.4.3.3 process produces. But if it's true, the proof > is subtle and non-local. I would like to be assured of this fact > in the spec. itself. Failing that, I think there is room for future > improvement at this point, by saying that the can-override > relation *is enforced* at step 3, and allow JVMs to prove that > the enforcement is a nop. We have the following two functions: canOverride(m1, m2) maximallySpecific(C, name, desc) The latter, by design, produces a set of methods that can override m2 (the resolved method). Once you have that set, there's no need to assert canOverride on its elements. How does 'maximallySpecific' have this property? Because m2 is not private (that's an earlier case in selection), meaning it's public, and we search for non-private methods with the same name and descriptor. If we had protected or package interface methods, then we'd need something to consider accessibility, like maximallySpecificOverrides(C, m2) But, for now, it works to re-use the 'maximallySpecific' used by resolution. ?Dan From karen.kinnear at oracle.com Tue Dec 5 23:01:44 2017 From: karen.kinnear at oracle.com (Karen Kinnear) Date: Tue, 5 Dec 2017 18:01:44 -0500 Subject: Final nestmates spec In-Reply-To: <99864909-94DC-4792-B09B-954B7FB0E073@oracle.com> References: <808AE338-72C4-49DA-A4A1-3441BFB8AB83@oracle.com> <99864909-94DC-4792-B09B-954B7FB0E073@oracle.com> Message-ID: Dan, Many thanks for the careful updates and working so closely with all of us with each iteration. Couple of minor comments: 1. Since we postponed this, and each release will change the ClassFile Version: - the new attributes will be in ClassFile Version 55 - If we want to use the new versioning, that would be 18.9 (rather than 11) 2. 3.7 Code example update - there is an invokespecial #4 // Method Near.getItNear()I I don?t think getItNear exists in the green example. I assume you meant Near.getIt(). 3. 5.4.2. Preparation: 2nd changed paragraph - The loader constraint equation below compares types relative to L2 (which is the class loader for the selected method) vs. L3. I do not see L3 defined here. I believe that is intended to be the super interface which declares the method in the first part of the sentence. 4. 5.4.4 I like the improvements suggested in this email thread clarifying that a host_class_index refers to a Constant_Class. And thank you for the future reminder of the circularity issue if we allow classes to be private. 5.5.4.5 Thank you for the example and for sweating the details. As an implementor, I very much like having all of this information in one place in the JVMS. 6. 5.4.5 ?the selected method of the invocation for an instance of a class or interface C? For invokeinterface and invoke virtual ?C? is always an object ref which must always be a class not an interface. Assuming you might be looking ahead to having this also work for invoke special when we get rid of ACC_SUPER? Otherwise I think this might work better with ?class C?. thanks, Karen > On Dec 1, 2017, at 7:52 PM, John Rose wrote: > > On Dec 1, 2017, at 4:41 PM, Dan Smith > wrote: >> >> I've uploaded what I believe could be a final spec for nestmates, here: >> http://cr.openjdk.java.net/~dlsmith/nestmates.html >> >> Since October, changes include: >> >> - Tweaks to the explanatory material in 2.11.8 and 3.7 >> >> - Some restructuring of the new 5.4.4 text to be a little less disruptive and more clearly define "same nest" and "nest host". There are some subtle behavioral differences compared to the previous iteration when errors occur. >> >> - Added an example to 5.4.5 to illustrate transitive overriding >> >> ?Dan > > I suggest changing the references to 18.3 to 10, based on today's information copied below. > > ? John > > > http://mail.openjdk.java.net/pipermail/jdk-dev/2017-December/000301.html > > Begin forwarded message: >> >> From: mark.reinhold at oracle.com >> Subject: Re: Draft JEP: Time-Based Release Versioning >> Date: December 1, 2017 at 7:59:47 AM PST >> To: volker.simonis at gmail.com >> Cc: jdk-dev at openjdk.java.net >> >> 2017/12/1 3:11:18 -0800, volker.simonis at gmail.com : >>> thanks for publishing the draft. Overall it looks good! >>> >>> I have just a few comment :) >>> >>> 1. Is thisJEP (i.e. the new version scheme) intended to be targeted >>> for Java 10? >> >> It's intended to be targeted when it's ready, just like any other JEP. >> It's obviously desirable to have it in 10, but if it doesn't make it >> then it'll be in 11. >> >>> I would appreciate to have it in ten but aren't we already quite late? >>> This is specification relevant (i.e. has to go into JSR 383) because >>> it changes java.lang.Runtime.Version and various standard system >>> properties which refer to it. JSR 383 has to be renamed from "Java SE >>> 18.3 Platform JSR" to "Java SE 10 Platform JSR" afterwards, right? >> >> That should be done anyway, regardless of whether this JEP makes it into >> JDK 10. If we change nothing in JDK 10, as it stands today, then the >> JDK 10 GA release will identify itself as 10, not 18.3. >> >> From john.r.rose at oracle.com Wed Dec 6 00:07:53 2017 From: john.r.rose at oracle.com (John Rose) Date: Tue, 5 Dec 2017 16:07:53 -0800 Subject: Final nestmates spec In-Reply-To: <25A5B940-49CB-4083-960C-DE097D1C053A@oracle.com> References: <808AE338-72C4-49DA-A4A1-3441BFB8AB83@oracle.com> <99864909-94DC-4792-B09B-954B7FB0E073@oracle.com> <688F8EEE-CEBB-40DC-A725-BA147EB6338E@oracle.com> <25A5B940-49CB-4083-960C-DE097D1C053A@oracle.com> Message-ID: On Dec 4, 2017, at 4:32 PM, Dan Smith wrote: > > Actioned. Notes on nontrivial suggestions below: > >> On Dec 1, 2017, at 8:21 PM, John Rose wrote: >> >> ++"The NestHost and NestMembers attributes of a class D are used >> to make symbolic references to classes and interfaces in the same >> run-time package as D. References to other types are ineffective under >> the rules of 5.4.1.1 and should not be introduced. Similarly, a nest host >> H should not list itself in its NestMembers attribute, although this >> will have no effect under the rules of 5.4.1.1." > > Good suggestion, but where to put it? I ended up with this: > ? That's great, much better than my suggested wording; thanks! >> (The rules call for loading the NestHost H even if it is not in the same >> package as the referring class M. This is the only substantive thing >> I feel slightly uncomfortable about in the spec., and it's not enough >> to demand a change. But it's there.) > > Yes. Discussed this with David and Alex, and we tried a few different iterations before settling on this. A key observation is that, if we eagerly detect that the class can't possibly be a packagemate, we can quickly return "false", but then an IAE will occur. It's important to avoid loading in the fast/common path, but when you're throwing errors you're in the slow/exceptional path. So the extra complexity of a special-case rule doesn't seem justified. > > Stepping back, I asked myself the question, "what good would an extra pre-check do?", and couldn't come up with anything. It's not like the ability to ask to load classes is a closely-guarded right. Yes. If I load a class Foo and it has a NestHost of Bar, then I might have to load Bar, but there are many other ways that loading Foo might trigger a load of Bar. And, loading Foo and then Foo's NestHost can only be triggered in two circumstances: Foo is originating a symbolic reference to a private thing (maybe in Bar or maybe not), or a private member in Foo is the target of a symbolic reference (maybe in Bar's nest or maybe not). In both cases, both origin and target classes are already loaded and are fully competent to authorize loading of their respective NestHosts. The loading of the NestHost is a new thing, but not any newer than other "on the fly" loading of related classes. So I don't see any opportunity for trickery here. > ... >> ++"5.4.4.1 Nest Membership Checking" > > I'll add a note and leave it to Alex to decide. I think he may not like singleton subsections? Either way is fine with me. >> ... >> If I read the logic right, the only way to select a method here is >> if the symbolic reference already resolved to a public interface >> method, so the can-override relation is actually true, regardless >> of what the 5.4.3.3 process produces. But if it's true, the proof >> is subtle and non-local. I would like to be assured of this fact >> in the spec. itself. Failing that, I think there is room for future >> improvement at this point, by saying that the can-override >> relation *is enforced* at step 3, and allow JVMs to prove that >> the enforcement is a nop. > > We have the following two functions: > canOverride(m1, m2) > maximallySpecific(C, name, desc) > > The latter, by design, produces a set of methods that can override m2 (the resolved method). Once you have that set, there's no need to assert canOverride on its elements. > > How does 'maximallySpecific' have this property? Because m2 is not private (that's an earlier case in selection), meaning it's public, and we search for non-private methods with the same name and descriptor. Good, that's the way I read the logic. (I'm relieved; I still understand the spec!) > If we had protected or package interface methods, then we'd need something to consider accessibility, like > maximallySpecificOverrides(C, m2) And that's the "non-local" part I referred to, that made me a little uncomfortable. MaximallySpecific happens to include canOverride as a corollary, but only indirectly. So if we make a non-local change to interface methods, then this logic may grow a bug. > But, for now, it works to re-use the 'maximallySpecific' used by resolution. IMO a non-normative observation of this subtle corollary would be helpful. The spec. is full of such subtleties, but sometimes?as with the cross references regarding protected access checks?it's good to shine a little light on them. ? John From forax at univ-mlv.fr Wed Dec 6 14:47:41 2017 From: forax at univ-mlv.fr (Remi Forax) Date: Wed, 6 Dec 2017 15:47:41 +0100 (CET) Subject: Doubling down on the erasure vs constant pool specialization Message-ID: <1668458067.1387621.1512571661150.JavaMail.zimbra@u-pem.fr> Hi John, hi all, i've read the doc you send us during our last meeting on the classfile specialization, while it's powerful i do not think we need to do constant pool specialization at least to implements generics over primitives and value types. One of the results of the MVT prototype is that we need Q types only for the layout, you do not need Q types to represent value types on the stack, so we should not need to specialize the signature of a method if the class is generified, we can use Object as carrier type, an Object can represent reference type by subtyping and primitive and value type by buffering. If we do not need method descriptor specialization, we do not need vtable specialization, etc. We have 3 places where we can specialize the Java code, at bytecode level, at constant pool level, at JIT level, i believe we should investigate trying to do the specialization at JIT level only, mostly because it doesn't have the complexity of doing the specialization at constant pool level and it is the same level of bet as saying that we can use Object as a carrier type for all Q types. So here is my counter proposal: doubling down on the erasure, let method descriptor to use Object, introduce class species and method species, make the few bytecodes that deals with the layout species aware. for a field, like for Q-type field, the field need to be annotated with the type variable of the species, so when the layout is created, the VM can decide or not to specialize the layout for array, again, array are species so the component type is reified (again, the VM can decide to not specialize the array) and tested when storing an element. in term of bytecode, we have two kind of bytecodes that need to have there semantics changed, - new/anewarray, and all invoke* need to be able to send (out of band) argument types to create a species-class or a species-method, this can be done by using a typevar bytecode as prefix, the typevar being able to specify either a type argument, or a type variable from the current species-class or species-method - getfield/aaload doesn't need to be changed, they will both put an Object on stack - for putfield, a supplementary check to type argument of the species is needed before storing the value. - for aastore, the component check already exist. regards, R?mi From karen.kinnear at oracle.com Wed Dec 6 15:47:48 2017 From: karen.kinnear at oracle.com (Karen Kinnear) Date: Wed, 6 Dec 2017 10:47:48 -0500 Subject: Valhalla EG minutes Nov 22 Message-ID: <4CB62F85-1E84-4241-8059-06C416D12F48@oracle.com> Attendees: Remi, Tobi, Dan H, Ron, Frederic, John, Karen Welcome Tobi Ajila from IBM! AIs: Nestmates overriding review - Graham and Dan H Reflection API review for nestmates - email thread: http://mail.openjdk.java.net/pipermail/valhalla-spec-experts/2017-November/000411.html Feedback on new LWorld email thread (?abandon all U-types?): http://mail.openjdk.java.net/pipermail/valhalla-spec-experts/2017-November/000432.html Feedback on Template Classes in the VM: http://cr.openjdk.java.net/~jrose/values/template-classes.html 1. ConstantDynamic Moved from JDK 18.3 to JDK 18.9 No one voice concerns. In fact since Remi is restructuring ASM this makes the timing easier. 2. ClassFileVersion: Plan: this will change for every JDK release regardless of whether there were ClassFile changes in the jVMS. Goal: mid December change for JDK 18.3, then in future make the change very early in the release cycle. Remi: thumbs up 3. Nestmates: JEP 181 Note that the JEP covers both the JVM changes and javac changes to add nestmate information for inner classes. The inner class information will not go away - it contains additional information. A set of currently generated package-private bridges will no longer be generated. If you have reviews for the nestmates JVMS or reflection API please respond in email. 4. Minimal Value Types Oracle status - bug fixing and release engineering/licensing - working through logistics of getting an Early Access open project out. ASM state - refactoring and branching - ~ 1 month behind would very much like binaries to remove the overhead of downloading and building the repository ====== 5. Value Types next step John described an overview of L-world - see email ?abandon all U-types? : http://mail.openjdk.java.net/pipermail/valhalla-spec-experts/2017-November/000432.html A few folks from Oracle?s VM team, Dan Smith, Rolan Westrelin and John met for a week to discuss the next steps in Value Types - post MVT. Initially we were brainstorming a proposal for universal types - U-type - which could be a disjoint union of an L-type (reference) or a Q-type (value type) so that we could solve problems such as creating default methods in interfaces that could support subtypes that were either value types or references) To fill out the potential kind space we introduced an R-type - which is a ?legacy? L-type. This allowed us to clearly break out migration issues for existing code that currently expects a reference when it for instance has a parameter of java.lang.Object or an interface. John highlighted a few of the things we learned during MVT prototyping: - interpreter buffering mechanism for Q-Types - which can be buffered either off heap (potentially thread local) or on heap and the benefits of using the same header layout in both cases - JIT handling of Q-types which can have a 1+N representation - i.e. in addition to scalarizing the needed fields, might keep a pointer as well - between these we realized that a 64 bit pointer can represent object references, nulls or ValueTypes - looks like a 128 bit pointer is not needed - key is - if we keep the indirection and unpack opportunistically, can we get the performance we need? - from our prototyping so far we think we can - more prototyping needed Question arose: do we need new carrier types - e.g. a U-type? Or do we even need a Q-type, and if so where? One option that we explored is to re-use the existing L-Type to represent either a reference or a value-type + bridging goes away, default boxes go away + legacy code operates on values risk: some legacy code will not work and we will need to defend against values - e.g. synchronization risk: performance of if_acmp_eq/ne - and some other byte codes Dan H: what about identity? John: NO identity for value types, they can be flattened or buffered, but there is no accidental identity, no heisenboxes e.g. acmp: John will send a follow-up email: choices (discussed with Guy Steele): an acmp in which both operands are not references could always return false or could throw an exception - model is that people have been trained to use acmp or .equals, so the false would allow the .equals to work Remi: likes John: note: this was a vm level discussion, still need to discuss with language folks Karen: brief summary of proposal: L-Type as reference or value type Q-Type - as value type - AI: where do we need this? R-Type - as reference only - open question - do we need this at all? Root: LObject - keep the top type, have it be a top type for ALL types: references, value types, more like an interface Value types are: immutable (not all the way down) not nullable no identity flattenable (not guaranteed flattened) no default box - if you want identity, create a reference storing a value type field - we buffer value types, we do not box them support interfaces author must opt-in for a class to become a value type Discussion of when we might want to explicitly use a Q-Type? - propose - for a field or array declaration - for a field declaration goal is to only preload value types not all references - want faster startup and reduced circularity issues Remi: a Q descriptor might cause incompatibility C# objects ? (I missed the rest of the sentence - something about Dynamic and must map as same type) Dan H: packed objects used annotations (list of packed classes) if Packed - and threw errors if mismatched. This was not well loved. If you call a Q-type - if it is actually not a value type, could throw an error Could have a modifier bit in the field vs. a QDescriptor? Remi: flattening is dynamic For an array - at array creation we know the element so we might not need a Type Frederic: for fields - for performance we precompute the layout of fields Remi: agrees needed for fields Method arguments? Do we need a Q Descriptor John: for arrays - does the verifier need to check element and therefore need to load the element class? Or would this be a runtime failure if there were a mismatch? John : silver lining: Q means - can not observe object identity, null is not allowed, value-based, immutable Note: talked about an Ljava/util/List could not pass through a Qjava/util/List descriptor unless you ran List.copyOf first (editor?s note - I don?t get this - is this a ?freeze? operation? or dynamic conversion to a value type? We don?t have a default box ?) Remi: If we have a Q in a method descriptor - do we need a bridge? John: overloads? handle covariant return types? Remi: QType <= LType? John :yes If you have a QType as a return type and it calls a method that returns an LType, then you need a covariant bridge and current code does NOT work John: recompile to make a bridge John: method signature might need to use Q or R - if it only works for one or the other C# solves this: Tony Hoare?s null problem Remi: if you store null in a QType field - throw an exception Remi with generics - added bridges to fix JLS/JVMS overriding rules mismatch looks like we would need bridges in current code Issue with dynamic languages if interact with Java John: ?BSM? everywhere bridge-o-matic concept - ?near miss? linkage error -> call BSM will need to explore what ?near miss? means Karen: what if we had no Q method descriptors and all byte codes worked for both Q and L types? - performance risk - need to find out - still able to do a string search on method descriptor match Remi: Do we need Q-types at all? Real constraint is the object layout which we need to know before we finish loading the class containing the field John JIT wants to know as well what about a modifier on a field? flattenable? Note verifier can not check nullability if we don?t have Types Note: verifier will check differently based on class file version Remi: what about frozen arrays rather than declaring in the type system? What if we start trying to not ever use Q, and see where we need to add it John: same for R - think about Karen: performance of byte codes is the other key requirement for the L-world John: Dan H & Tobi: did you ever add tag bits to managed references? Dan H: not now - ask the GC guys John: acmp - pointers with tag bits could allow fast check for is this a reference rather than a value type ? John: We also discussed potential JVM exploration for Generics with Value Types and References Starting with templates from C++ issues: heap layouts of parameterized classes - different sizes - different optionality (i.e. potential to be null) Proposal: (JVM perspective) Create constant pools with holes for: classes, method handles, fields, method, ? Track dependencies of constant pool entries on parameters Segment constant pool - associate a set of constant pool entries with a segment have a common/root segment Rule: If holes are not filled, there is very little you can do (e.g. wildcards) If hole is not filled: ?Riddled?, so you can?t resolve Class concept species as instantiation of a template new relationship between the template class and the species Dan H: Descriptors? John only unriddled descriptors can be resolved structured descriptors which are not the same as flattened strings, however if the constant pool Riddle is Solved, then a descriptor could be flattened into a string caveat: if a template client creates a species by requesting holes filled - it fills them with live types which are resolved by the template client - and must be visible to the template client?s class loader, module, etc. they do NOT have to be resolvable from the template class? class loader ed. note: all the species have to be created in the same runtime package/module/ class loader/ protection domain Remi: MethodTypes will have the same issue John: will need some extensions on class loader constraints Note: can only verify after filling the holes Challenges: 1. wildcare types 2. generic methods - need a carrier for genericity 3. generic virtual method in generic class (infinite table size?) (stretchable table? or change to non-viable?) Template invariant parts are shared variant parts are not shared John: if no global dictionary of keys? Need constant pool links thanks, Karen From daniel.smith at oracle.com Fri Dec 8 18:30:46 2017 From: daniel.smith at oracle.com (Dan Smith) Date: Fri, 8 Dec 2017 11:30:46 -0700 Subject: stricter symbolic references to protected methods In-Reply-To: <132AB27A-6B2B-452A-9D2A-14250FE31F06@oracle.com> References: <808AE338-72C4-49DA-A4A1-3441BFB8AB83@oracle.com> <132AB27A-6B2B-452A-9D2A-14250FE31F06@oracle.com> Message-ID: > On Dec 1, 2017, at 7:48 PM, John Rose wrote: > > This is the language I'm talking about: > >> ? R is protected and is declared in a class C, and D is either a subclass of C or C itself. Furthermore, if R is not static, then the symbolic reference to R must contain a symbolic reference to a class T, such that T is either a subclass of D, a superclass of D, or D itself. >> >> During verification, it was also required that, even if T is a superclass of D, the target reference of a protected instance field access or method invocation must be an instance of D or a subclass of D (4.10.1.8). > > > We could replace it by this: > >> ? R is protected and is declared in a class C *not in the same run-time package as D*, and D is *a subclass of C*. This is fine?just defers the "or C itself" part to the next bullet ("C itself" implies same package). >> Furthermore the symbolic reference to R must contain a symbolic reference to *the immediate superclass of D*. So, here's a class hierarchy: package p1; public class A { protected static void m() {} protected void n() {} } package p2; public class B extends A { } package p2; public class C extends B { } package p2; public class D extends C { } From the body of C, it's legal in Java to say: A.m(); // invokestatic A.m()V B.m(); // invokestatic B.m()V C.m(); // invokestatic C.m()V C.m(); // invokestatic D.m()V So your proposed change is bad for static members. For *instance* members, from the body of C: void test(A a, B b, C c, D d) { a.n(); // error b.n(); // error c.n(); // ok: invokevirtual C.n()V d.n(); // ok: invokevirtual D.n()V super.n(); // ok: invokespecial B.n()V } The one thing that I think we can safely assert will never be referenced by Java is 'A.n'. One exception, though: for some reason, methods of Object ('clone') are referenced directly via Object, even when, say, the receiver has type C. So we need to keep the "subclass of D" and "D itself" parts of the rule about T. From javac's perspective, we could restrict the "superclass of D" part of the rule to mean "direct superclass", as long as we do something about the Object case. At the same time, we would probably want to limit invokespecial to a direct superclass. >> During verification, it was also required that the target reference of a protected instance field access or method invocation must be an instance of D or a subclass of D (4.10.1.8). FYI, I believe the only time Java will have a mismatch between the target reference type and the referenced class type is when 'super' is used. Elsewhere, T= and this verification rule isn't needed. (Except for the 'Object' special case, noted above.) ?Dan From daniel.smith at oracle.com Fri Dec 8 19:48:12 2017 From: daniel.smith at oracle.com (Dan Smith) Date: Fri, 8 Dec 2017 12:48:12 -0700 Subject: Final nestmates spec In-Reply-To: References: <808AE338-72C4-49DA-A4A1-3441BFB8AB83@oracle.com> <99864909-94DC-4792-B09B-954B7FB0E073@oracle.com> Message-ID: All fixed, thanks. > On Dec 5, 2017, at 4:01 PM, Karen Kinnear wrote: > > 1. Since we postponed this, and each release will change the ClassFile Version: > - the new attributes will be in ClassFile Version 55 > - If we want to use the new versioning, that would be 18.9 (rather than 11) I think version name 11 is the consensus now? > 3. 5.4.2. Preparation: > 2nd changed paragraph - > The loader constraint equation below compares types relative to L2 (which is the class loader for the selected method) vs. L3. I do not see L3 defined here. > I believe that is intended to be the super interface which declares the method in the first part of the sentence. Markdown anomaly. Forgot to escape the '<'. > 6. 5.4.5 ?the selected method of the invocation for an instance of a class or interface C? > For invokeinterface and invoke virtual ?C? is always an object ref which must always be a class > not an interface. See the last sentence in this section: "While C will typically be a class, it may be an interface when these rules are applied during preparation" ?Dan From david.holmes at oracle.com Tue Dec 12 07:52:49 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 12 Dec 2017 17:52:49 +1000 Subject: Final nestmates spec In-Reply-To: <808AE338-72C4-49DA-A4A1-3441BFB8AB83@oracle.com> References: <808AE338-72C4-49DA-A4A1-3441BFB8AB83@oracle.com> Message-ID: <10b09128-1251-b8bf-8f5e-68c06eee6151@oracle.com> Hi Dan, I have one query with regards to the processing of the NestMembers attribute. Originally when I wrote the draft I copied the InnerClasses text and that included prohibiting duplicate entries in NestMembers. You dropped that restriction from the spec (though I haven't yet removed it from the VM). How does that interact with the Class.getNestMembers() API? Should it return the raw NestMembers contents (which may include duplicates - including additional references to "self") or should it return the set of members (ie no duplicates, including self)? Thanks, David On 2/12/2017 10:41 AM, Dan Smith wrote: > I've uploaded what I believe could be a final spec for nestmates, here: > http://cr.openjdk.java.net/~dlsmith/nestmates.html > > Since October, changes include: > > - Tweaks to the explanatory material in 2.11.8 and 3.7 > > - Some restructuring of the new 5.4.4 text to be a little less disruptive and more clearly define "same nest" and "nest host". There are some subtle behavioral differences compared to the previous iteration when errors occur. > > - Added an example to 5.4.5 to illustrate transitive overriding > > ?Dan > From john.r.rose at oracle.com Tue Dec 12 21:22:29 2017 From: john.r.rose at oracle.com (John Rose) Date: Tue, 12 Dec 2017 13:22:29 -0800 Subject: Final nestmates spec In-Reply-To: <10b09128-1251-b8bf-8f5e-68c06eee6151@oracle.com> References: <808AE338-72C4-49DA-A4A1-3441BFB8AB83@oracle.com> <10b09128-1251-b8bf-8f5e-68c06eee6151@oracle.com> Message-ID: <34E1593E-1AA9-4EBE-B070-0E9C71C21666@oracle.com> On Dec 11, 2017, at 11:52 PM, David Holmes wrote: > > I have one query with regards to the processing of the NestMembers attribute. Originally when I wrote the draft I copied the InnerClasses text and that included prohibiting duplicate entries in NestMembers. You dropped that restriction from the spec (though I haven't yet removed it from the VM). How does that interact with the Class.getNestMembers() API? Should it return the raw NestMembers contents (which may include duplicates - including additional references to "self") or should it return the set of members (ie no duplicates, including self)? My take: We have landed on laissez-faire for this kind of format check at class load time. The access checking rules are cleverly written so that garbage in the list is ignored. This is noted by the non-normative language in 4.7.29: > This array is consulted during access checking (5.4.4). It should consist of references to other classes and interfaces that belong to the same run-time package and have NestHost attributes referencing this class or interface. Items that do not meet this description are discouraged and will be ignored by access checking. There are four kinds of garbage: 1. [self] the nest host class itself (the one containing NestMs) 2. [outsider] valid classes whose package prefix differs form that of the nest host class 3. [duplicate] duplicate entries (either the same CONSTANT_Class or a homonym) 4. [not-found] references to classes which will not exist when resolved The spec. says these are *allowed* and *ignored*. (A fifth kind of garbage, syntax errors in class names, is caught by routine processing of CONSTANT_Class items. Maybe this is worth a unit test. The other four cases are worth a unit test, for that matter.) Each of these four cases poses a question for reflection: Do we hide it or expose it as an anomalous entry in the reflected list? In an earlier conversation, I think we agree that case 4 should just throw an error when reflection is performed, following a similar precedent with Class.getDefinedClasses. I don't remember our resolution of what to do about 1/2/3. The overall choice is to sanitize on reflection or just pass through the garbage. Given the action on 4 (just let it hit the fan), I suggest that we should also let self, outsider, and duplicate entries show through on reflection. Counterargument: We could sanitize the reflected data, and also sort or randomize its order for good measure, to prevent buggy reflection users from accidentally depending on irrelevant specifics of classfile input. Remember how we got into trouble with reflected method lists, which when they were reordered caused some buggy reflectors to break. See also iteration order of maps and ImmutableCollections.SALT. My recommendation: Just let the bad data through reflection, and rely on javac to avoid putting bad data there in the first place. The javadoc could be adorned with weasel words saying that response to bad data (of forms 1..4 above) and sequence order are undefined and may change without notice. ? John From david.holmes at oracle.com Tue Dec 12 21:55:55 2017 From: david.holmes at oracle.com (David Holmes) Date: Wed, 13 Dec 2017 07:55:55 +1000 Subject: Final nestmates spec In-Reply-To: <34E1593E-1AA9-4EBE-B070-0E9C71C21666@oracle.com> References: <808AE338-72C4-49DA-A4A1-3441BFB8AB83@oracle.com> <10b09128-1251-b8bf-8f5e-68c06eee6151@oracle.com> <34E1593E-1AA9-4EBE-B070-0E9C71C21666@oracle.com> Message-ID: <5533938e-352f-0711-9052-75242b4ff402@oracle.com> Hi John, On 13/12/2017 7:22 AM, John Rose wrote: > On Dec 11, 2017, at 11:52 PM, David Holmes > wrote: >> >> I have one query with regards to the processing of the NestMembers >> attribute. Originally when I wrote the draft I copied the InnerClasses >> text and that included prohibiting duplicate entries in NestMembers. >> You dropped that restriction from the spec (though I haven't yet >> removed it from the VM). How does that interact with the >> Class.getNestMembers() API? Should it return the raw NestMembers >> contents (which may include duplicates - including additional >> references to "self") or should it return the set of members (ie no >> duplicates, including self)? > > My take: > > We have landed on laissez-faire for this kind of format check > at class load time. ?The access checking rules are cleverly > written so that garbage in the list is ignored. ?This is noted > by the non-normative language in 4.7.29: > >> This array is consulted during access checking (5.4.4). It should >> consist of references to other classes and interfaces that?belong to >> the same run-time package and have?NestHost?attributes referencing >> this class or interface. Items that do not?meet this description are >> discouraged and will be ignored by access checking. > > There are four kinds of garbage: > > 1. [self] the nest host class itself (the one containing NestMs) > 2. [outsider] valid classes whose package prefix differs form that of > the nest host class > 3. [duplicate] duplicate entries (either the same CONSTANT_Class or a > homonym) > 4. [not-found] references to classes which will not exist when resolved > > The spec. says these are *allowed* and *ignored*. > > (A fifth kind of garbage, syntax errors in class names, is caught > by routine processing of CONSTANT_Class items. ?Maybe this is worth > a unit test. ?The other four cases are worth a unit test, for that matter.) A sixth kind of garbage, or an expansion of #2, are classes that don't list the current class as their nest host. > Each of these four cases poses a question for reflection: ?Do we hide > it or expose it as an anomalous entry in the reflected list? > > In an earlier conversation, I think we agree that case 4 should just > throw an error when reflection is performed, following a similar > precedent with Class.getDefinedClasses. > > I don't remember our resolution of what to do about 1/2/3. ?The overall I recall no discussion of #1 and #3. For #2, #4 and #6 - ie any listed member that is not validated as being a member - we throw exceptions. *

Each listed nest member must be validated by checking its own * declared {@linkplain #getNestHost() nest host}. Any exceptions that occur * as part of this process will be thrown. * @throws LinkageError if there is any problem loading or validating * a nest member or its nest host In practice these will be IncompatibleClassChangeError for #2 and #6, and NoClassDefFoundError for #4. For #5, depending on exactly what you mean by "syntax error", it could be rejected at classfile parsing time; or it simply forms the wrong name and will then come under #2, #4 or #6. > choice is to sanitize on reflection or just pass through the garbage. > Given the action on 4 (just let it hit the fan), I suggest that we should > also let self, outsider, and duplicate entries show through on reflection. > > Counterargument: ?We could sanitize the reflected data, and also sort > or randomize its order for good measure, to prevent buggy reflection users > from accidentally depending on irrelevant specifics of classfile input. > Remember how we got into trouble with reflected method lists, which > when they were reordered caused some buggy reflectors to break. > See also iteration order of maps and ImmutableCollections.SALT. > > My recommendation: ?Just let the bad data through reflection, and > rely on javac to avoid putting bad data there in the first place. > > The javadoc could be adorned with weasel words saying that response > to bad data (of forms 1..4 above) and sequence order are undefined > and may change without notice. Recommendation noted. :) Personally I'd prefer it if the array elements did form a set, but I don't want every correct use of this API to pay the cost of ensuring that is the case. So some additional words will be needed I think. I'll file a RFE to track that change and propose it here. Thanks, David > ? John From john.r.rose at oracle.com Wed Dec 13 01:17:44 2017 From: john.r.rose at oracle.com (John Rose) Date: Tue, 12 Dec 2017 17:17:44 -0800 Subject: Final nestmates spec In-Reply-To: <5533938e-352f-0711-9052-75242b4ff402@oracle.com> References: <808AE338-72C4-49DA-A4A1-3441BFB8AB83@oracle.com> <10b09128-1251-b8bf-8f5e-68c06eee6151@oracle.com> <34E1593E-1AA9-4EBE-B070-0E9C71C21666@oracle.com> <5533938e-352f-0711-9052-75242b4ff402@oracle.com> Message-ID: On Dec 12, 2017, at 1:55 PM, David Holmes wrote: > > Hi John, > > On 13/12/2017 7:22 AM, John Rose wrote: >> On Dec 11, 2017, at 11:52 PM, David Holmes > wrote: >>> >>> I have one query with regards to the processing of the NestMembers attribute. Originally when I wrote the draft I copied the InnerClasses text and that included prohibiting duplicate entries in NestMembers. You dropped that restriction from the spec (though I haven't yet removed it from the VM). How does that interact with the Class.getNestMembers() API? Should it return the raw NestMembers contents (which may include duplicates - including additional references to "self") or should it return the set of members (ie no duplicates, including self)? >> My take: >> We have landed on laissez-faire for this kind of format check >> at class load time. The access checking rules are cleverly >> written so that garbage in the list is ignored. This is noted >> by the non-normative language in 4.7.29: >>> This array is consulted during access checking (5.4.4). It should consist of references to other classes and interfaces that belong to the same run-time package and have NestHost attributes referencing this class or interface. Items that do not meet this description are discouraged and will be ignored by access checking. >> There are four kinds of garbage: >> 1. [self] the nest host class itself (the one containing NestMs) >> 2. [outsider] valid classes whose package prefix differs form that of the nest host class >> 3. [duplicate] duplicate entries (either the same CONSTANT_Class or a homonym) >> 4. [not-found] references to classes which will not exist when resolved >> The spec. says these are *allowed* and *ignored*. >> (A fifth kind of garbage, syntax errors in class names, is caught >> by routine processing of CONSTANT_Class items. Maybe this is worth >> a unit test. The other four cases are worth a unit test, for that matter.) > > A sixth kind of garbage, or an expansion of #2, are classes that don't list the current class as their nest host. > >> Each of these four cases poses a question for reflection: Do we hide >> it or expose it as an anomalous entry in the reflected list? >> In an earlier conversation, I think we agree that case 4 should just >> throw an error when reflection is performed, following a similar >> precedent with Class.getDefinedClasses. >> I don't remember our resolution of what to do about 1/2/3. The overall > > I recall no discussion of #1 and #3. For #2, #4 and #6 - ie any listed member that is not validated as being a member - we throw exceptions. Ah, yes; I remember that part now. That makes sense: 2/4/6 are a more dangerous kind of garbage, potentially misleading the reader of the reflected data, so it needs to be blocked, and we have agreed to throw exceptions (LEs of various sorts). That's good. > *

Each listed nest member must be validated by checking its own > * declared {@linkplain #getNestHost() nest host}. Any exceptions that occur > * as part of this process will be thrown. > > * @throws LinkageError if there is any problem loading or validating > * a nest member or its nest host > > In practice these will be IncompatibleClassChangeError for #2 and #6, and NoClassDefFoundError for #4. > > For #5, depending on exactly what you mean by "syntax error", it could be rejected at classfile parsing time; or it simply forms the wrong name and will then come under #2, #4 or #6. By #5 I mean stuff that's locally malformed at class file parsing time, such as CONSTANT_Class["foo///bar"], or even CONSTANT_String["Foo"]. That's correctly checked by parse_classfile_nest_members_attribute. > choice is to sanitize on reflection or just pass through the garbage. >> Given the action on 4 (just let it hit the fan), I suggest that we should >> also let self, outsider, and duplicate entries show through on reflection. >> Counterargument: We could sanitize the reflected data, and also sort >> or randomize its order for good measure, to prevent buggy reflection users >> from accidentally depending on irrelevant specifics of classfile input. >> Remember how we got into trouble with reflected method lists, which >> when they were reordered caused some buggy reflectors to break. >> See also iteration order of maps and ImmutableCollections.SALT. >> My recommendation: Just let the bad data through reflection, and >> rely on javac to avoid putting bad data there in the first place. >> The javadoc could be adorned with weasel words saying that response >> to bad data (of forms 1..4 above) and sequence order are undefined >> and may change without notice. > > Recommendation noted. :) > > Personally I'd prefer it if the array elements did form a set, but I don't want every correct use of this API to pay the cost of ensuring that is the case. So some additional words will be needed I think. I'll file a RFE to track that change and propose it here. Good. ? John From karen.kinnear at oracle.com Wed Dec 13 19:58:01 2017 From: karen.kinnear at oracle.com (Karen Kinnear) Date: Wed, 13 Dec 2017 14:58:01 -0500 Subject: LWorld1 initial phase proposal Message-ID: Folks, Based on Dan Smith?s write-up: http://cr.openjdk.java.net/~dlsmith/values-notes.html of a possible LWorld direction, we?ve written a proposal for a potential initial LWorld1 prototype and would appreciate review. We can walk through at our next meeting Dec 20th. JVM Lworld Value Types Phase I http://cr.openjdk.java.net/~dlsmith/values-notes.html I. New terminology: for LWorld L-Type as reference or value type Q-Type as value type R-Type as reference only - open question - do we need this? II. Assumptions: 1. New root: LObject - for all references and value types 2. Value Type characteristics: immutable (not all the way down) not nullable no identity flattenable when contained in reference, value type or array no default box (buffer, not box) - if you want identity, create a reference storing a value type field support interfaces III. goals: existing interfaces should be implementable by both references and value types - without requiring recompilation existing code should be able to handle both references and value type arguments - without requiring recompilation Migration: LType -> QType migration author must opt-in: by declaring in source (language policy) - requires recompilation Any existing class that meets the requirements could become a value type ValueBasedClasses are candidates LWorld Value Type Static restrictions (for javac) j.l.Object as the only superclass no use of identity: no use of Object methods invalid for value types: wait/notify*/finalize no assumptions of nullability no non-private constructors no use of reference-only bytecodes: new, putfield, monitorenter/exit, aconst_null LWorld Static restrictions (for verifier) TBD no use of reference-only bytecodes: new, putfield, monitorenter/exit, aconst_null LWorld new Dynamic restrictions (based on dynamic type) for Value Types: no use of reference-only bytecodes: new, putfield, monitorenter/exit, aconst_null for Reference Types: no use of value-type-only bytecodes: vdefault, withfield no use of if_acmpeq/ne without following .equals call IV: Non-goals: Primitives as value types - future phase V. Expected Behaviors: 1. Object methods wait/notify/etc: if QType: throw ICCE (IncompatibleClassChangeError) getClass: normal behavior (no ambiguity with no default box) toString: nothing special clone: nothing special (must implement cloneable) finalize: ICCE equals: if QType - component-wise equals (ucmp API? ucmp bytecode?) (overridable) hashcode: TBD - needs to be based on equals 2. Java level APIs isValue isFlattened (for a reflection Field or array) isElementValue (for an array - not the same as is the array a Value) ucmp - substitutability check (not overridable) 3. Support beyond MVT Allow Value Types in static fields (not flattened) Method and method invocation support Interface support: default methods - must handle both L-types and Q-types 4. LWorld bytecodes vs. JVMS 9 Key Challenge: can we apply the same bytecodes to QTypes and RTypes? can they check dynamically without loss of performance? special handling: if_acmpeq/if_acmpne: false/true if either is a Qtype. They should fall back to .equals ifnull/ifnonnull: as if acmp vs. Null: false/true if QType needs dynamic different handling: aaload/aastore: handle LType or QType dynamically aload/astore: handle LType or QType dynamically areturn: handle LType or QType dynamically exception if wrong kind: ICCE putfield: QType exception: ICCE monitorenter/exit: exception for QType: ICCE new: exception for QType: ICCE (expects uninitialized state) aconst_null: exception for QType: ICCE vdefault: exception for LType: ICCE withfield: exception for LType: ICCE unchanged or already implemented (in MVT) or should fall out: getfield: handle LType or QType dynamically (already implemented) newarray/etc.: handle LType or QType dynamically (already implemented) athrow: always LType (subtype of Error) - unchanged invoke*: handle LType or QType dynamically (should fall out) checkcast/instanceof: should fall out ldc: should fall out V. Implementation use of explicit QType 1. Field descriptors Goal: not require verifier or class file parser to load all fields. ICCE if misclaimed, at first runtime mismatch (kind constraints) To allow flattening, want field and arrays to explicitly use QTypes at language level 2. Array descriptors: propose - yes Remi: not needed - at array creation you know the element type Propsal: uniformity - confusing to explain inconsistency - javac already knows the information and has done the work, why slow down? - safety - kind constraints - could be checked by verifier 3. Method descriptors : propose - no 1) No indication of receiver type 2) method descriptor parameters/return type - all use LTypes Propose: do NOT support QTypes in Method Descriptors challenge: descriptor mismatches based on migration - for client inheritance/overriding, caller/callee matching Resolved Questions for LWorld1 1. Q: support other superclasses? A: QType has no subclasses A. for now - QType has only jlO as superclass, may be extended in future (see if any that would break any optimizations) 2. Q:acmp behavior options: a) failing: return false <- propose for try 1 b) throw exception c) field-equality using ucmp as "substitutable" - field-wise comparison general bit equality including floating point may need to recurse on values buffereed A: LWorld1: if >= one operand is a QType: if_acmpeq -> false, if_acmpne -> true A: null handling: as if acmp vs. NULL if operand is a QType: ifnull -> false, ifnonnull->true 3. Q: Do we need to know if an LType is an old L-Type or a new LType? A: be on the lookout - we have not yet identified any cases - note: If we do, we have ClassFileVersion 4. Q: Any issues with argument passing/argument return handling if we have runtime mismatches of kind? A: If we have kind constraints then we should not get runtime mismatches A: advanced error handling - check implications if verification is skipped 5. Q: Do we need a new carrier type? A: TBD - so far requirement not identified. 6. Q: What does it mean for LObject to be more like an interface? A: TBD (TODO - ask Dan?) A1. disallow adding fields A2. superclass of references and value types (with modified methods) 7. Q: Do we need a fast way to distinguish an LType from a QType? in the JVM itself? A. Initial prototype can work with instancKlass vs. valueKlass Later - look at potential optimizations 8. Q: What does updated Object.hashcode do? - field equality based hashcode - assume cache in header optimization ? A: Dan: call hashcode or identity hashcode (throw) - performance tradeoff 9. Q: Do we need a fast way for Java to determine ValueType (isValue call?) A: Frederic proposed: e.g. give all value types a common super interface e.g. ValueMarker - verifier or class file format checking at class loading - ensure that this can't be a superinterface if not a value type - this is probably temporary, but useful 10. Q: Migration QType->LType support? Customers will try this A: Need to ensure we catch failures challenges: instance creation: value type - must have a private constructor - so new will fail IAE - except for nestmates (dynamically added which are not same compilation) - except Reflection.setAccessible 11. Q: Circularity handling for Field types? A: today we get StackOverflowError - hotspot folks propose we leave that alone to avoid complexity and costly testing 12. Q: Class.forName() and internal loaded class cache A: NO L vs. Q type naming here, there is no ambiguity Unresolved Questions for LWorld1 1. Q: Reflection requirements/plans? 2. Q. What can the verifier check, what do we want to check later to avoid early class loading? A. Expect to create kind constraints, if we have bytecodes that require a value type which would be checked when we load that value type and throw ICCE A. Verifier can still check primitive vs. LType - leave those checks in A. note: for loaded classes such as supertypes, value type fields or isAssignable checks, some classes are already loaded, so may not need a kind constraint Q: Do we also need to do this for references - e.g. "new" bytecode? Q: Is there a concern that by loading a class that uses the wrong bytecodes for someone else's kind - that could prevent a class from successfully loading? =================== Early experiment: Add to JDK 11 (not MVT specific) add dynamic checks for ValueBasedClasses follow LWorld dynamic restrictions add checks for ifacmp_eq/ne NOT followed by a call to .equals? (todo: find Dan's corpus search results email) todo: consider using JFR? This will be open in jdk11 Experimental steps: 1. new repo - remove MVT parts, just include -XX:+EnableValhalla tests (already split) 2. LangTools: 2.1 test bytecode generator - generate LWorld bytecodes (from simp 2.2 Javac: __ByValue for class declaration (for simplified JVMS bytecode changes document) request: super as java.lang.Object generate LWorld bytecodes LWorld restrictions checking - e.g. allow superinterfaces , allow value types in static fields 2.2 new static utility class for new bytecodes isValue isElementValue (for array) isFlattened (for field or array) ucmp 2.3 java.lang.Object methods Using isValue - rewrite 2.4 real JVMS LWorld1 proposal - Dan 3. Runtime a. write up simplified JVMS bytecode changes document (not verifier yet) b. interpreter bytecode changes - see list above c. migrate VVT tests to LVT1 d. verifier - propose JVMS verifier changes propose kind constraint handling - to not add any additional class loading - note: value type fields, supertypes and isAssignable checks already perform loading e. method handle support 4. JIT: a. bytecodes b. migrate VVT tests to LVT1 c. adaptor generation d. optimizations 5. Core libraries: a. methodhandle support - e.g. changes to LambdaForms - (simplified from MVT with no __Value) - no boxing b. Reflection/new reflection? TODO - what are the requirements here? 6. Migration testing: Try VBC -> value type -- run tests and see what breaks From john.r.rose at oracle.com Thu Dec 14 01:29:31 2017 From: john.r.rose at oracle.com (John Rose) Date: Wed, 13 Dec 2017 17:29:31 -0800 Subject: LWorld1 initial phase proposal In-Reply-To: References: Message-ID: Nice!! I'm going to tear through this with a bunch of comments. (But anything having to do with optimizations specific to HotSpot will go to valhalla-dev.) On Dec 13, 2017, at 11:58 AM, Karen Kinnear wrote: > > Folks, > > Based on Dan Smith?s write-up: http://cr.openjdk.java.net/~dlsmith/values-notes.html of a possible > LWorld direction, we?ve written a proposal for a potential initial LWorld1 prototype and would appreciate review. > We can walk through at our next meeting Dec 20th. > > JVM Lworld Value Types Phase I > > http://cr.openjdk.java.net/~dlsmith/values-notes.html > > I. New terminology: for LWorld > L-Type as reference or value type > Q-Type as value type > R-Type as reference only - open question - do we need this? For the JVM's descriptor language, we need the legacy syntax for L-types and maybe a new syntax for Q-types: TBD. But for syntax it's conceivable that we only need this: L-Descriptor: the descriptor of any reference type or value type. For talking about the values and object references flowing around inside the JVM and through its data structures, we need terminology to distinguish values from objects. For that, we are currently leaning heavily on the word "value" and the opposed word "object": value instance = entity in the JVM w/o object identity, and not a reference value class = class whose instances are value instances (disregarding possibility of boxes) object instance = entity in the JVM w/ object identity, manipulated by reference object class = class whose instances are object instances Also: null = the unique reference which does not point to any object instance reference = the identity of an object instance, or else null value = ambiguous for (a) a value instance, or (b) any entity the JVM can store in some variable (So use the unqualified term "value" with caution; context must determine.) Then: value type = the type of a variable which can hold some value instances but not any references reference type = the type of a variable which can hold some references but not any value instances open/general/union type = the type of a variable which can hold some mix of value instances and references (incl. null) I don't think we have settled on a term for open/general/union types. The JVM has nothing called an "object type" because any JVM variable that can refer to an object is in fact a nullable reference. An "object type", if it were a useful thing, would be the type of a non-null reference variable. We can agree to use it as a synonym to "reference type" or as a non-nullable reference, when being careful about nulls. Context will determine, as with the term "value". Given those distinctions, we can then use the following shorthand terms in our conversations, shorthands which are *inspired* by the JVM descriptor syntax, but are not necessarily identical to it: R-type: a reference type Q-type: a value type U-type: an open/general/union type L-type: same thing as a U-type, as long as we are in the context of L-world We can say "U-type" or "open/general/union type" when we want to be clear that some entity in the JVM can be either a value instance or reference. We can say "R-type" when we want to be clear that value instances are excluded. In the simplest version of L-world, R-types don't appear to play a large role; if a variable can carry an object reference, it can also carry a value. Anyway, that gives us the following additional options for descriptors: Q-descriptor: the descriptor of a Q-type, when context could interpret an L-descriptor differently (TBD) R-descriptor: the descriptor of an R-type, when context could interpret an L-descriptor differently (TBD) U-descriptor: the descriptor of an L-type, when context could interpret an L-descriptor differently (TBD) We only need these options if there are contexts that require them. It is probably better to try to avoid such contexts in the first place. The way to do that is to do everything with L-descriptors, and supply contextual bits when those L-descriptors should be interpreted as Q-types or R-types. Example: A field of value type either needs an explicit flatten flag, or a Q-descriptor. The flatten flag (ACC_VALUE) makes an unambiguous context for the L-descriptor of the field type, so that it means a Q-type rather than an L-type. > II. Assumptions: > 1. New root: LObject - for all references and value types java.lang.Object is a U-type. Object classes (other than java.lang.Object) define R-types. Value classes define Q-types. Interfaces define U-types. Abstract classes define R-types. The following are possibilities but I hope most of them can be false: Perhaps some object classes will *also* imply frozen versions which are Q-types; TBD. Perhaps some value classes will *also* imply heavy-boxed versions which are R-types; TBD. Perhaps some interfaces will be *restricted* to Q-types or R-types; TBD. Perhaps some abstract classes can help define Q-types or U-types; TBD. > 2. Value Type characteristics: > immutable (not all the way down) Exposition: In terms of its component fields, a value instance is shallowly immutable, not deeply immutable. The JDK term is "unmodifiable". In terms of "codes like a class", the value class is a value-based class. In terms of its containing variable, a value instance may be replaced by another value instance, unless (of course) the containing variable is final. In terms of "works like an int", you have to overwrite the whole value in its variable. You cannot individually address and update fields within the value, just as you cannot address and update individual bits within an int. This is the JVM level view; at the language level you might have operators like "&=" or "++" which might *seem* to address and update part or all of the value itself, independently of its containing variable. At the JVM level, there must always be a write-back of the new value to the old varabiel. > not nullable (Thus, null is never a value instance, only a reference.) > no identity (Comment: Value types enforce a superset of the constraints on value-based classes. In that way, VBC's anticipate VTs.) > flattenable when contained in reference, value type or array "contained in reference" is ambiguous. I suggest this language: flattenable when contained in a variable (instance field, static field, array element, or local) > no default box (buffer, not box) > - if you want identity, create a reference storing a value type field I think "create an object instance" is less ambiguous than "create a reference". There are use cases for boxes, and the troubling legacy types like java.lang.Integer, but I agree that we should push back on them and try to get away without them. See above where I put a TBD on value classes defining secondary R-types. > support interfaces Yes. Interfaces define U-types. (TBD whether there are any variations on this theme.) And we can say java.lang.Object is an "honorary interface". It is a U-type which all classes implement. Like interfaces, it is a valid bound for all types and all generic variables. > III. goals: > existing interfaces should be implementable by both references and value types > - without requiring recompilation > existing code should be able to handle both references and value type arguments > - without requiring recompilation (This is a big task. If we pull it off, we enjoy the simplicity of L-world relative to U-world. If not, we have to deal with the complexity of defining Q/U/L-types and the bridges between them.) > Migration: > LType -> QType migration > author must opt-in: by declaring in source (language policy) > - requires recompilation Specifically, there is object class to value class migration. Or, emphasizing the object-ness of the original class, R-type to Q-type class migration. For descriptors (as opposed to classes), there might be no migration, if there are no new descriptor spellings. But if we do any (non-empty) subset of Q/R/U-descriptors, there are associated migration stories. (Note that R-type to U-type migration is probably cheap or a no-op in L-world, since R-types in the old world and U-types in the new world are both spelled with L-descriptors. Indeed, that's the point of L-world.) > Any existing class that meets the requirements could become a value type > ValueBasedClasses are candidates Yes. (Spelling: "value-based classes".) See: https://docs.oracle.com/javase/8/docs/api/java/lang/doc-files/ValueBased.html > > LWorld Value Type Static restrictions (for javac) > j.l.Object as the only superclass (Yes; bookmark a later question of value classes which are also abstract.) Value instances must be protected from identity operations, and generic code which uses identity operations must be protected from ill effects of value instances. This involves a web of restrictions and compromises in both the JLS, JVMS, and JDK. > no use of identity: > no use of Object methods invalid for value types: wait/notify*/finalize This is a JDK restriction, in the relevant native methods. The restriction on finalize is a JVM restriction. > no assumptions of nullability This turns into a restriction where U-types are narrowed to Q-types. Any such narrowing operation must throw NPE for nulls. > no non-private constructors This is a JLS restriction. The JVM should simply outlaw methods altogether. "I don't think that word means what you think it means." > no use of reference-only bytecodes: new, putfield, monitorenter/exit, aconst_null These don't fit together easily. new and aconst_null produce R-types. That doesn't seem to need a restriction. putfield and monitore/e consume R-types. That needs a restriction against Q-types. It must be dynamically applied, at least in the case of L-Object and L-interface. Perhaps we need a new exception, NotAnObjectException. (Cue bikeshed team. Don't wanna call it Value*Exception, please.) Actually, putfield is linked against a class which (at link time, no earlier) is detected to be either an object class or a value class. If the latter, a LinkageError needs to get thrown, not NotAnObjectException. That might be NotAnObjectError <: LE. (If we do Q-types derived from object classes, then putfield *will* need a runtime check *also*, to detect frozen Q-versions of R-objects. Bookmark that for later.) > LWorld Static restrictions (for verifier) TBD > no use of reference-only bytecodes: new, putfield, monitorenter/exit, aconst_null No, these are link-time, not verify-time restrictions. The verifier must not be required to load classes just to determine the value-vs-object distinction. If we want/need to verify value-vs-object, we need new instruction bits or new descriptors (Q-descriptors) to express that to the verifier w/o class loading. Conversely, we can avoid Q-descriptors if we don't require the verifier to make those distinctions. I would prefer the latter. Is it possible? > > LWorld new Dynamic restrictions (based on dynamic type) > for Value Types: no use of reference-only bytecodes: new, putfield, monitorenter/exit, aconst_null > for Reference Types: no use of value-type-only bytecodes: vdefault, withfield If a class can be only object or instance (and never both), then a bytecode that links to that class can know, at link time, whether it is operating on objects or instances. That is (probably) the case with vdefault and withfield. (It might get more complex in the future if we fool around with deriving Q-types from R-classes, aka object classes.) > no use of if_acmpeq/ne without following .equals call That's a JLS restriction, I think. At the JVMS level we simply say that acmp treats values like op==(float,float) treats NaNs. Or maybe say it rolls a die if it can predict that the two value operands are "the same" in some internal sense of substitutability, else returns false. I prefer the first, even though the second is a little easier to implement. > IV: Non-goals: > Primitives as value types - future phase (Yes, another bookmark.) > > V. Expected Behaviors: > > 1. Object methods > wait/notify/etc: if QType: throw ICCE (IncompatibleClassChangeError) > getClass: normal behavior (no ambiguity with no default box) > toString: nothing special > clone: nothing special (must implement cloneable) You are mostly about that one. The value might choose to clone object substructure, using a copy method which might be named "clone". But implementing Cloneable is not the same as implementing clone. They are different requests. And "Cloneable" gates "do a shallow copy of my fields in the default Object.clone method". That is the object version of simple value instance assignment. In other words, all values are Cloneable for free. So it doesn't make any sense to require a VT to implement Cloneable, unless that is just a gating factor for allowing Object.clone to return the same value. Arguably value types should not be allowed to implement Cloneable, unless there is some use case for that (awful thing) as a marker of some sort. Maybe it's a laissez-faire thing here; allow it. > finalize: ICCE As with clone, maybe we can be laissez-faire. "Sure, define that method, but don't expect me to call it ever, because you don't have an identity which can become unreachable." > equals: if QType - component-wise equals (ucmp API? ucmp bytecode?) (overridable) IMO the component-wise default should be a JDK feature, but not a JVM feature. Maybe we want System.isSubstitutable(x,y) and System.substitutabilityHashCode(x) as JDK functions. I don't think we really need a ucmp bytecode, just an acmp bytecode which treats values like NaNs. So, bookmark ucmp for later. > hashcode: TBD - needs to be based on equals Ditto. (More later.) ? John From ali.ebrahimi1781 at gmail.com Thu Dec 14 09:29:05 2017 From: ali.ebrahimi1781 at gmail.com (Ali Ebrahimi) Date: Thu, 14 Dec 2017 12:59:05 +0330 Subject: LWorld1 initial phase proposal In-Reply-To: References: Message-ID: Hi, This post is actually my question from John, but per Brain request I can not add John mail address to receiver list, but I hope John hear me. On Thu, Dec 14, 2017 at 4:59 AM, John Rose wrote: > Nice!! I'm going to tear through this with a bunch of comments. > .... > > > > > LWorld new Dynamic restrictions (based on dynamic type) > > for Value Types: no use of reference-only bytecodes: new, > putfield, monitorenter/exit, aconst_null > > for Reference Types: no use of value-type-only bytecodes: > vdefault, withfield > > If a class can be only object or instance (and never both), then a > bytecode that > links to that class can know, at link time, whether it is operating on > objects or > instances. That is (probably) the case with vdefault and withfield. (It > might > get more complex in the future if we fool around with deriving Q-types from > R-classes, aka object classes.) > What do you mean by object or instance hear? Is not object or value? And I don't have good sense from using Object as root by already existing background of being root as Ref types. What is interaction with generics? ex. List vs List -- Best Regards, Ali Ebrahimi From david.holmes at oracle.com Fri Dec 15 02:50:18 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 15 Dec 2017 12:50:18 +1000 Subject: Final nestmates spec In-Reply-To: <808AE338-72C4-49DA-A4A1-3441BFB8AB83@oracle.com> References: <808AE338-72C4-49DA-A4A1-3441BFB8AB83@oracle.com> Message-ID: Hi Dan, Sorry, one further clarification. If we encounter nestmate attributes in a classfile with a version that pre-dates the Nestmate release, do we ignore them completely (as an unknown attribute) or assume they are in fact our nestmate attributes? Thanks, David On 2/12/2017 10:41 AM, Dan Smith wrote: > I've uploaded what I believe could be a final spec for nestmates, here: > http://cr.openjdk.java.net/~dlsmith/nestmates.html > > Since October, changes include: > > - Tweaks to the explanatory material in 2.11.8 and 3.7 > > - Some restructuring of the new 5.4.4 text to be a little less disruptive and more clearly define "same nest" and "nest host". There are some subtle behavioral differences compared to the previous iteration when errors occur. > > - Added an example to 5.4.5 to illustrate transitive overriding > > ?Dan > From john.r.rose at oracle.com Fri Dec 15 03:53:19 2017 From: john.r.rose at oracle.com (John Rose) Date: Thu, 14 Dec 2017 19:53:19 -0800 Subject: Final nestmates spec In-Reply-To: References: <808AE338-72C4-49DA-A4A1-3441BFB8AB83@oracle.com> Message-ID: <47AF7133-BE2A-4D43-8A7C-6FFCCF11C3BC@oracle.com> On Dec 14, 2017, at 6:50 PM, David Holmes wrote: > > If we encounter nestmate attributes in a classfile with a version that pre-dates the Nestmate release, do we ignore them completely (as an unknown attribute) or assume they are in fact our nestmate attributes? I think we need to ignore them for down-rev class files. That's what we do currently do for attributes like annotations or generic signatures or bootstrap specifiers. From david.holmes at oracle.com Fri Dec 15 06:45:55 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 15 Dec 2017 16:45:55 +1000 Subject: Final nestmates spec In-Reply-To: <47AF7133-BE2A-4D43-8A7C-6FFCCF11C3BC@oracle.com> References: <808AE338-72C4-49DA-A4A1-3441BFB8AB83@oracle.com> <47AF7133-BE2A-4D43-8A7C-6FFCCF11C3BC@oracle.com> Message-ID: <516c9411-4411-0d25-1f9a-247a2b9b2dff@oracle.com> On 15/12/2017 1:53 PM, John Rose wrote: > On Dec 14, 2017, at 6:50 PM, David Holmes > wrote: >> >> If we encounter nestmate attributes in a classfile with a version that >> pre-dates the Nestmate release, do we ignore them completely (as an >> unknown attribute) or assume they are in fact our nestmate attributes? > > I think we need to ignore them for down-rev class files. > That's what we do currently do for attributes like annotations > or generic signatures or bootstrap specifiers. Thanks John. I went looking for existing examples but for some reason my brain kept reading the JAVA_1_5_VERSION check as if it were JAVA_1_2. TGIF. :) Cheers, David From daniel.smith at oracle.com Fri Dec 15 18:44:42 2017 From: daniel.smith at oracle.com (Dan Smith) Date: Fri, 15 Dec 2017 11:44:42 -0700 Subject: Final nestmates spec In-Reply-To: <10b09128-1251-b8bf-8f5e-68c06eee6151@oracle.com> References: <808AE338-72C4-49DA-A4A1-3441BFB8AB83@oracle.com> <10b09128-1251-b8bf-8f5e-68c06eee6151@oracle.com> Message-ID: > On Dec 12, 2017, at 12:52 AM, David Holmes wrote: > > Hi Dan, > > I have one query with regards to the processing of the NestMembers attribute. Originally when I wrote the draft I copied the InnerClasses text and that included prohibiting duplicate entries in NestMembers. You dropped that restriction from the spec (though I haven't yet removed it from the VM). How does that interact with the Class.getNestMembers() API? Should it return the raw NestMembers contents (which may include duplicates - including additional references to "self") or should it return the set of members (ie no duplicates, including self)? Just one thing I'll add: is 'getNestMembers' meant to give clients a raw view the NestMembers attribute, or to tell them what is in the nest? My sense is that the latter is the goal, and so the method should use the attribute as a tool to inform its result, but it *shouldn't* just regurgitate what it reads. A good litmus test: should the result contain the current class or not? If it does, you're giving clients a polished list of actual nest members. If it does not, you're just repeating to clients what the attribute says. ?Dan From daniel.smith at oracle.com Fri Dec 15 18:37:05 2017 From: daniel.smith at oracle.com (Dan Smith) Date: Fri, 15 Dec 2017 11:37:05 -0700 Subject: JLS spec changes for nestmates Message-ID: <86C4EB0E-6B50-44EB-BBBF-736911BB6027@oracle.com> FYI, here are the spec changes for JLS related to nestmates. http://cr.openjdk.java.net/~dlsmith/nestmates-jls.html I've also attached them to the JBS subtask: https://bugs.openjdk.java.net/browse/JDK-8177019 This is just some cleanup to more closely align the JLS description of method invocation with the actual (updated) JVMS rules. I looked over 13.1, and think it does a fine job of allowing binaries to use the NestHost and NestMembers attributes without any changes (see items 9 and 10 in the second list). ?Dan From forax at univ-mlv.fr Fri Dec 15 22:41:07 2017 From: forax at univ-mlv.fr (Remi Forax) Date: Fri, 15 Dec 2017 23:41:07 +0100 (CET) Subject: Doubling down on the erasure vs constant pool specialization In-Reply-To: <1668458067.1387621.1512571661150.JavaMail.zimbra@u-pem.fr> References: <1668458067.1387621.1512571661150.JavaMail.zimbra@u-pem.fr> Message-ID: <1675569533.2498297.1513377667528.JavaMail.zimbra@u-pem.fr> Hi all, There is a nice talk about implementing generics with separate compilation in Swift: https://youtu.be/ctS8FzqcRug I really think we should follow a similar road, their "witness table" is very similar to what the specialization part of a species can be, obviously all the data structure used for representing generics should be created by the VM not by the compiler. They are only able to specialize the code if the source is available, but we have the VM JITs for that. During the question at the end, there is an interesting remarks that separate compilation of value types and generics specialization are parts of the same mechanism. R?mi ----- Mail original ----- > De: "Remi Forax" > ?: "valhalla-spec-experts" > Envoy?: Mercredi 6 D?cembre 2017 15:47:41 > Objet: Doubling down on the erasure vs constant pool specialization > Hi John, hi all, > i've read the doc you send us during our last meeting on the classfile > specialization, > while it's powerful i do not think we need to do constant pool specialization at > least to implements generics over primitives and value types. > > One of the results of the MVT prototype is that we need Q types only for the > layout, you do not need Q types to represent value types on the stack, so we > should not need to specialize the signature of a method if the class is > generified, we can use Object as carrier type, an Object can represent > reference type by subtyping and primitive and value type by buffering. If we do > not need method descriptor specialization, we do not need vtable > specialization, etc. > > We have 3 places where we can specialize the Java code, at bytecode level, at > constant pool level, at JIT level, i believe we should investigate trying to do > the specialization at JIT level only, mostly because it doesn't have the > complexity of doing the specialization at constant pool level and it is the > same level of bet as saying that we can use Object as a carrier type for all Q > types. > > So here is my counter proposal: doubling down on the erasure, let method > descriptor to use Object, introduce class species and method species, make the > few bytecodes that deals with the layout species aware. > > for a field, like for Q-type field, the field need to be annotated with the type > variable of the species, so when the layout is created, the VM can decide or > not to specialize the layout > for array, again, array are species so the component type is reified (again, the > VM can decide to not specialize the array) and tested when storing an element. > > in term of bytecode, we have two kind of bytecodes that need to have there > semantics changed, > - new/anewarray, and all invoke* need to be able to send (out of band) argument > types to create a species-class or a species-method, > this can be done by using a typevar bytecode as prefix, the typevar being able > to specify either a type argument, or a type variable from the current > species-class or species-method > - getfield/aaload doesn't need to be changed, they will both put an Object on > stack > - for putfield, a supplementary check to type argument of the species is needed > before storing the value. > - for aastore, the component check already exist. > > regards, > R?mi From david.holmes at oracle.com Sat Dec 16 04:41:52 2017 From: david.holmes at oracle.com (David Holmes) Date: Sat, 16 Dec 2017 14:41:52 +1000 Subject: Final nestmates spec In-Reply-To: References: <808AE338-72C4-49DA-A4A1-3441BFB8AB83@oracle.com> <10b09128-1251-b8bf-8f5e-68c06eee6151@oracle.com> Message-ID: Hi Dan, On 16/12/2017 4:44 AM, Dan Smith wrote: >> On Dec 12, 2017, at 12:52 AM, David Holmes wrote: >> >> Hi Dan, >> >> I have one query with regards to the processing of the NestMembers attribute. Originally when I wrote the draft I copied the InnerClasses text and that included prohibiting duplicate entries in NestMembers. You dropped that restriction from the spec (though I haven't yet removed it from the VM). How does that interact with the Class.getNestMembers() API? Should it return the raw NestMembers contents (which may include duplicates - including additional references to "self") or should it return the set of members (ie no duplicates, including self)? > > Just one thing I'll add: is 'getNestMembers' meant to give clients a raw view the NestMembers attribute, or to tell them what is in the nest? My sense is that the latter is the goal, and so the method should use the attribute as a tool to inform its result, but it *shouldn't* just regurgitate what it reads. That conflicts with what John suggested. > A good litmus test: should the result contain the current class or not? If it does, you're giving clients a polished list of actual nest members. If it does not, you're just repeating to clients what the attribute says. The result should of course contain the current class - if it is the nest host it is always returned as the zeroeth element. I don't see how that relates to "polished" versus "regurgitated? getNetsMembers() doesn't mean "read the Nestmember attribute of this class", it means " get all the nest members of the nest in which this class is a member". Thanks, David > ?Dan > From david.holmes at oracle.com Mon Dec 18 01:15:07 2017 From: david.holmes at oracle.com (David Holmes) Date: Mon, 18 Dec 2017 11:15:07 +1000 Subject: JLS spec changes for nestmates In-Reply-To: <86C4EB0E-6B50-44EB-BBBF-736911BB6027@oracle.com> References: <86C4EB0E-6B50-44EB-BBBF-736911BB6027@oracle.com> Message-ID: Hi Dan, On 16/12/2017 4:37 AM, Dan Smith wrote: > FYI, here are the spec changes for JLS related to nestmates. > > http://cr.openjdk.java.net/~dlsmith/nestmates-jls.html > > I've also attached them to the JBS subtask: > https://bugs.openjdk.java.net/browse/JDK-8177019 > > This is just some cleanup to more closely align the JLS description of method invocation with the actual (updated) JVMS rules. That all seems fine to me. > I looked over 13.1, and think it does a fine job of allowing binaries to use the NestHost and NestMembers attributes without any changes (see items 9 and 10 in the second list). I don't follow. Items 9 and 10 seem to pertain to the InnerClasses attribute. NestHost and NestMembers can not, in general, be used to convey the information required by items 9 and 10. The NestHost attribute need not refer to the immediately enclosing class. The NestMembers attribute only appears in the top-level NestHost class. Thanks, David > ?Dan > From daniel.smith at oracle.com Mon Dec 18 20:53:41 2017 From: daniel.smith at oracle.com (Dan Smith) Date: Mon, 18 Dec 2017 13:53:41 -0700 Subject: JLS spec changes for nestmates In-Reply-To: References: <86C4EB0E-6B50-44EB-BBBF-736911BB6027@oracle.com> Message-ID: <29131B95-F935-4BF0-8241-CF0B67B03523@oracle.com> > On Dec 17, 2017, at 6:15 PM, David Holmes wrote: > >> I looked over 13.1, and think it does a fine job of allowing binaries to use the NestHost and NestMembers attributes without any changes (see items 9 and 10 in the second list). > > I don't follow. Items 9 and 10 seem to pertain to the InnerClasses attribute. NestHost and NestMembers can not, in general, be used to convey the information required by items 9 and 10. The NestHost attribute need not refer to the immediately enclosing class. The NestMembers attribute only appears in the top-level NestHost class. Yeah, you're right. This section is very hand-wavy, so I tend to give it some latitude to say things that aren't exactly right. And there's a difference between _allowing_ a reference to something and _requiring_ it. But we could probably do a better job here and require the references we expect to appear in NestHost/NestMembers. 9. Every nested class and nested interface must have a symbolic reference to its immediately enclosing class (?8.1.3) **and, if different, the top-level type that encloses the class or interface**. 10. Every class must contain symbolic references to all of its member types (?8.5), and to all local and anonymous classes that appear in its methods, constructors, static initializers, instance initializers, and field initializers. Every interface must contain symbolic references to all of its member types (?9.5), and to all local and anonymous classes that appear in its default methods and field initializers. **If the class or interface is a top-level type declaration, it must contain symbolic references to all other classes or interfaces declared in its body and nested within other type declarations.** ?Dan From daniel.smith at oracle.com Mon Dec 18 21:28:11 2017 From: daniel.smith at oracle.com (Dan Smith) Date: Mon, 18 Dec 2017 14:28:11 -0700 Subject: Final nestmates spec In-Reply-To: References: <808AE338-72C4-49DA-A4A1-3441BFB8AB83@oracle.com> <10b09128-1251-b8bf-8f5e-68c06eee6151@oracle.com> Message-ID: > On Dec 15, 2017, at 9:41 PM, David Holmes wrote: > >> A good litmus test: should the result contain the current class or not? If it does, you're giving clients a polished list of actual nest members. If it does not, you're just repeating to clients what the attribute says. > > The result should of course contain the current class - if it is the nest host it is always returned as the zeroeth element. > getNetsMembers() doesn't mean "read the Nestmember attribute of this class", it means " get all the nest members of the nest in which this class is a member". Okay, then my argument is that the result of 'getNestMembers' is a semantically meaningful set of classes, not just a view of the attribute contents. And under that interpretation, anomalous entries should either be silently filtered out or trigger an error. As a user, I think I'd view the method as "go find me all the classes in this nest; you may use 'NestMembers' as a hint for your search". > That conflicts with what John suggested. Sorry, John, I'd prefer different behavior. :-) Specifically: > Given the action on 4 (just let it hit the fan), I suggest that we should > also let self, outsider, and duplicate entries show through on reflection. The proposed treatment of a nonexistent class (4) is to report an error. That's fine, and maybe it makes sense to report an error in some or all of the cases. (I wouldn't object to silently ignoring the nonexistent class, though. Depends on what we think users want when a nest member class is missing?which, as we've discovered, is not uncommon.) But letting things "show through" is, I think, never what the end user will want. They will believe that anything in the array returned by 'getNestMembers' actually *is* a member of the nest. ?Dan From david.holmes at oracle.com Mon Dec 18 22:03:03 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 19 Dec 2017 08:03:03 +1000 Subject: Final nestmates spec In-Reply-To: References: <808AE338-72C4-49DA-A4A1-3441BFB8AB83@oracle.com> <10b09128-1251-b8bf-8f5e-68c06eee6151@oracle.com> Message-ID: <3716b390-204c-2cbc-17a1-502f57b488de@oracle.com> On 19/12/2017 7:28 AM, Dan Smith wrote: >> On Dec 15, 2017, at 9:41 PM, David Holmes wrote: >> >>> A good litmus test: should the result contain the current class or not? If it does, you're giving clients a polished list of actual nest members. If it does not, you're just repeating to clients what the attribute says. >> >> The result should of course contain the current class - if it is the nest host it is always returned as the zeroeth element. > >> getNetsMembers() doesn't mean "read the Nestmember attribute of this class", it means " get all the nest members of the nest in which this class is a member". > > Okay, then my argument is that the result of 'getNestMembers' is a semantically meaningful set of classes, not just a view of the attribute contents. And under that interpretation, anomalous entries should either be silently filtered out or trigger an error. > > As a user, I think I'd view the method as "go find me all the classes in this nest; you may use 'NestMembers' as a hint for your search". > >> That conflicts with what John suggested. > > Sorry, John, I'd prefer different behavior. :-) > > Specifically: >> Given the action on 4 (just let it hit the fan), I suggest that we should >> also let self, outsider, and duplicate entries show through on reflection. > > The proposed treatment of a nonexistent class (4) is to report an error. That's fine, and maybe it makes sense to report an error in some or all of the cases. (I wouldn't object to silently ignoring the nonexistent class, though. Depends on what we think users want when a nest member class is missing?which, as we've discovered, is not uncommon.) > > But letting things "show through" is, I think, never what the end user will want. They will believe that anything in the array returned by 'getNestMembers' actually *is* a member of the nest. Given we're only talking about the case where the nesthost is explicitly listed in NestMembers, or where there are duplicate (correct) entries in NestMembers, then everything in the returned array _is_ a member of the nest. The issue is whether we need to condense the array so that it holds the set of members. From a purist perspective I'd like it to return the set of members in the array. But pragmatically I hate to add overhead to every single correct use of the attribute just to account for the allowed-for-but-not-expected corner case of there being duplicates. Thanks, David > ?Dan > From daniel.smith at oracle.com Mon Dec 18 23:01:11 2017 From: daniel.smith at oracle.com (Dan Smith) Date: Mon, 18 Dec 2017 16:01:11 -0700 Subject: Final nestmates spec In-Reply-To: <3716b390-204c-2cbc-17a1-502f57b488de@oracle.com> References: <808AE338-72C4-49DA-A4A1-3441BFB8AB83@oracle.com> <10b09128-1251-b8bf-8f5e-68c06eee6151@oracle.com> <3716b390-204c-2cbc-17a1-502f57b488de@oracle.com> Message-ID: <8EF9210A-2CC6-4FCE-8F49-1EE22598C4CA@oracle.com> > On Dec 18, 2017, at 3:03 PM, David Holmes wrote: > > On 19/12/2017 7:28 AM, Dan Smith wrote: >> But letting things "show through" is, I think, never what the end user will want. They will believe that anything in the array returned by 'getNestMembers' actually *is* a member of the nest. > > Given we're only talking about the case where the nesthost is explicitly listed in NestMembers, or where there are duplicate (correct) entries in NestMembers, then everything in the returned array _is_ a member of the nest. The issue is whether we need to condense the array so that it holds the set of members. Yeah, I guess I'm mostly just concerned about #2/#6?non-nest members appearing in the attribute. You said this: > I recall no discussion of #1 and #3. For #2, #4 and #6 - ie any listed member that is not validated as being a member - we throw exceptions. > > *

Each listed nest member must be validated by checking its own > * declared {@linkplain #getNestHost() nest host}. Any exceptions that occur > * as part of this process will be thrown. > > * @throws LinkageError if there is any problem loading or validating > * a nest member or its nest host > > In practice these will be IncompatibleClassChangeError for #2 and #6, and NoClassDefFoundError for #4. Not clear to me that an error will occur in all cases?if I put java.lang.String in my NestMembers attribute, validating java.lang.String by calling its getNestHost isn't going to prompt any error. I agree that redundant elements are not ideal but tolerable, depending on engineering considerations. ?Dan From david.holmes at oracle.com Mon Dec 18 23:49:15 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 19 Dec 2017 09:49:15 +1000 Subject: Final nestmates spec In-Reply-To: <8EF9210A-2CC6-4FCE-8F49-1EE22598C4CA@oracle.com> References: <808AE338-72C4-49DA-A4A1-3441BFB8AB83@oracle.com> <10b09128-1251-b8bf-8f5e-68c06eee6151@oracle.com> <3716b390-204c-2cbc-17a1-502f57b488de@oracle.com> <8EF9210A-2CC6-4FCE-8F49-1EE22598C4CA@oracle.com> Message-ID: On 19/12/2017 9:01 AM, Dan Smith wrote: >> On Dec 18, 2017, at 3:03 PM, David Holmes > > wrote: >> >> On 19/12/2017 7:28 AM, Dan Smith wrote: >>> But letting things "show through" is, I think, never what the end >>> user will want. They will believe that anything in the array returned >>> by 'getNestMembers' actually *is* a member of the nest. >> >> Given we're only talking about the case where the nesthost is >> explicitly listed in NestMembers, or where there are duplicate >> (correct) entries in NestMembers, then everything in the returned >> array _is_ a member of the nest. The issue is whether we need to >> condense the array so that it holds the set of members. > > Yeah, I guess I'm mostly just concerned about #2/#6?non-nest members > appearing in the attribute. > > You said this: > >> I recall no discussion of #1 and #3. For #2, #4 and #6 - ie any listed >> member that is not validated as being a member - we throw exceptions. >> >> ?*

Each listed nest member must be validated by checking its own >> ?* declared {@linkplain #getNestHost() nest host}. Any exceptions that >> occur >> ?* as part of this process will be thrown. >> >> ??* @throws LinkageError if there is any problem loading or validating >> ??* a nest member or its nest host >> >> In practice these will be IncompatibleClassChangeError for #2 and #6, >> and NoClassDefFoundError for #4. > > Not clear to me that an error will occur in all cases?if I put > java.lang.String in my NestMembers attribute, validating > java.lang.String by calling its getNestHost isn't going to prompt any error. Right I need to clarify that the validation performed in getNestMembers is _not_ done by calling Class.getNestHost on it - as that never throws any exceptions. The validation is done at the VM level. The doc quoted should not be linked to #getNestHost. I have an open issue to update getNestMembers to clarify what's been discussed. > I agree that redundant elements are not ideal but tolerable, depending > on engineering considerations. Okay. Thanks, David > ?Dan From david.holmes at oracle.com Tue Dec 19 20:50:32 2017 From: david.holmes at oracle.com (David Holmes) Date: Wed, 20 Dec 2017 06:50:32 +1000 Subject: JLS spec changes for nestmates In-Reply-To: <29131B95-F935-4BF0-8241-CF0B67B03523@oracle.com> References: <86C4EB0E-6B50-44EB-BBBF-736911BB6027@oracle.com> <29131B95-F935-4BF0-8241-CF0B67B03523@oracle.com> Message-ID: On 19/12/2017 6:53 AM, Dan Smith wrote: >> On Dec 17, 2017, at 6:15 PM, David Holmes wrote: >> >>> I looked over 13.1, and think it does a fine job of allowing binaries to use the NestHost and NestMembers attributes without any changes (see items 9 and 10 in the second list). >> >> I don't follow. Items 9 and 10 seem to pertain to the InnerClasses attribute. NestHost and NestMembers can not, in general, be used to convey the information required by items 9 and 10. The NestHost attribute need not refer to the immediately enclosing class. The NestMembers attribute only appears in the top-level NestHost class. > > Yeah, you're right. This section is very hand-wavy, so I tend to give it some latitude to say things that aren't exactly right. And there's a difference between _allowing_ a reference to something and _requiring_ it. But we could probably do a better job here and require the references we expect to appear in NestHost/NestMembers. But I didn't think it was a requirement to use the new attributes. I thought a source compiler could choose to continue to generate bridge accessors and not need the new attributes? (That's what the JEP implies.) David ----- > > 9. Every nested class and nested interface must have a symbolic reference to its immediately enclosing class (?8.1.3) **and, if different, the top-level type that encloses the class or interface**. > > 10. Every class must contain symbolic references to all of its member types (?8.5), and to all local and anonymous classes that appear in its methods, constructors, static initializers, instance initializers, and field initializers. > > Every interface must contain symbolic references to all of its member types (?9.5), and to all local and anonymous classes that appear in its default methods and field initializers. > > **If the class or interface is a top-level type declaration, it must contain symbolic references to all other classes or interfaces declared in its body and nested within other type declarations.** > > ?Dan > From karen.kinnear at oracle.com Tue Dec 19 21:46:06 2017 From: karen.kinnear at oracle.com (Karen Kinnear) Date: Tue, 19 Dec 2017 16:46:06 -0500 Subject: Valhalla EG minutes Dec 6 2017 Message-ID: Reminder - we will have our usual meeting Wednesday Dec 20. attendees: Remi, Tobi, Dan H, Mr Simms, Lois, Frederic, Dan S, John, Karen Corrections/clarification welcome I. Condy - targeted for JDK 18.9/jdk 11 for current JVMS version - note: next phase of Condy - with lazy resolution for BootstrapMethod static arguments - will be LATER than jdk11 II. Nestmates - futures phase 2: proposal for sealed classes - Class or Interface - can only be extended by nest members - JDK 11 also looking to add Lookup.defineClass with private mode that allows dynamically extending the nest - based on having been given a Lookup. III. Nestmates JVMS review: 1. selection and overriding rule changes - are they class file version dependent? Dan S: NO - the changes are universal editor?s note - let?s discuss again - just to clarify - when the JVMS added support for transitive overriding - that was based on the class file version of the class containing the overriding method 2. selection changes - Dan H and Karen believe this is correct as written, covers the issue well and thank you for the examples 3. Dan H: access control error handling clarification - if access control check causes e.g. resolution which can throw NCDFE, then we throw that error, we do not wrap in an IAE with cause, right? Dan S: correct John - no subtype of Error covered all 4. Nestmates JVMTI spec update Believe we need a change here regarding the new nest mate attributes Karen: propose we do not allow NestHost or NestMembers to change choice: 1. disallow change 2. allow adding to NestMembers AI: Hotspot & J9 - double-check how complex this would be to implement, note requirement that re-resolution needs to return the same result, so if we allowed adding to NestMembers, attempts to access a private member of a non-nestmember should fail resolution with an IAE - and needs to continue to fail even if we change it to become a nest member 5. Nestmate reflection requirements Remi: does reflection reflect the static or dynamic nestmates? If not the dynamic ones - how else could you discover them? IV. Class Specialization - Template proposal Remi: propose that rather than early template copying and optimization per specialization, move the specialization to the JIT - do not specify Q-Types lots of places - try to have the same byte code, so do not specialize byte codes or constant pool - let JIT optimize John: agree with not specializing byte code (model 1 & 2 explorations) - template class reduces the number of classes loaded - depends on riddled constant pool segment - any method only depends on root constant pool segment, does not need specialization Remi: why holes in the constant pool? John: how else do you express the variants? Remi: too powerful? John: want to do other things. Goal: formalism to allow expressing what C++ templates can do with more runtime safety Let the JIT optimize specialization - including methodHandles Remi: stage 1 of generics - only need Q-Type information is for layout stage 2 generics: template class Karen: If in future primitives can be represented as value types (hard to do/get performance) - with L-World - there will be a reduced need for generics John: still need wildcard support, e.g. need holes in the constant pool to inject field descriptor Remi: How do you inject a type argument? choices: 1) inject type argument - push type vs. 2) BootstrapMethod - pull type John: document allows condy and indy with holes for each specialization, condy or indy would need separate resolution Remi: could we get stage 1 specialization in Valhalla VT? John: cruel to do specialization without language support Remi: With Value Types - see a path for class species - method species - not so clear John: calculus of template classes can solve reified generics on methods - need a method specializer -> gives you a MethodHandle - named linkage - to match against a species (editor?s note: I didn?t follow how you pick a species to then search for the method?) - issue: expand vtable slots - not want infinite - e.g. vtable slot for each set of type variables on generic method - we need a translation strategy not vm magic Remi: concern about memory for infrequently used John: holey constant pool - only use where it adds value - strategy includes lots of erasure, inlining, loop optimization, coroutines - note profiling fails with generics - hole based templates give us some expressiveness - goal: flattened data structures, hints to JIT for profiling Remi: drawback - size of metadata - would like generic methods approach without hole based templates John: like Haskell Type Classes? - like Brian?s shadow instances? can often statically link at a given use point - holey template - new hooks to express variants/invariants, at execution becomes invariant - generic methods - confine variance e.g. to private inner helper class - which author could do explicitly Remi: what about based on type argument on stack? John: concern performance cost of non-generic code - which needs to do the checking - we started with extra args on stack model, bound to constants -> led to constant pool - if you do the same specialization twice - reuse - do not need to make a new one - channel info mostly through constant resolution Remi: what if you had a type arg or species that then was treated specially John: e.g. 2 entry points - 1) general 2) need specialization - want only 1 method, with multiple ?displays? of bound constants - this gives you a single pointer (e.g. register ) for piece of constant pool, rather than one per variable Remi: what if instead of segment of constant pool - varied type argument - since lifetime is stack based Karen: if this is a field descriptor - lifetime is not stack based Remi: what if introduce a T-type in descriptor? Karen: we need to explore where we need type descriptors - e.g. Q-Type or possibly T-Type thanks, Karen From karen.kinnear at oracle.com Tue Dec 19 22:47:29 2017 From: karen.kinnear at oracle.com (Karen Kinnear) Date: Tue, 19 Dec 2017 17:47:29 -0500 Subject: nestmates JVMTI spec proposal changes Message-ID: <1B2ECA5E-402C-41F6-954E-4987BF7199A2@oracle.com> I believe we are all in agreement that: 1. Redefinition should NOT be allowed to change the NestHost attribute and 2. Redefinition should NOT be allowed to remove NestMembers (equivalent to reducing access controls) The question was - should we allow Redefinition to add NestMembers or not? Hotspot team would like to propose that at least for the initial nestmates release - we do NOT allow Redefinition to add NestMembers. In all cases, NOT allow would mean a failure to redefine a class. Open to suggestions on error that should be returned. This is the least risky approach and allows future reducing restrictions if we find we need to. It is extremely difficult to increase restrictions. Discussion: JVMTI redefinition restrictions today: The redefinition may change method bodies, the constant pool and attributes. The redefinition must not add, remove or rename fields or methods, change the signatures of methods, change modifiers, or change inheritance. These restrictions may be lifted in future versions. See the error return description below for information on error codes returned if an unsupported redefinition is attempted. The class file bytes are not verified or installed until they have passed through the chain of ClassFileLoadHook events, thus the returned error code reflects the result of the transformations applied to the bytes passed into class_definitions . If any error code is returned other than JVMTI_ERROR_NONE, none of the classes to be redefined will have a new definition installed. When this function returns (with the error code of JVMTI_ERROR_NONE) all of the classes to be redefined will have their new definitions installed. Note that redefinition today is not allowed to change modifiers or change inheritance, so no changes that could change access control behavior. The jigsaw AddExports allows dynamic increasing of access to types. To ensure that resolution of a constant pool type entries that fails always fails in the same way, the implementation must cache those failures. If we were to allow dynamic increases in member access, to ensure that resolution of content pool member entries (fields, methods) would fail in the same way, the implementation would need to cache those resolution failures, which is non-trivial, and would need to ensure that the redefinition implementation retained errors for resolution from other types. Note that Redefinition is allowed to change inner class attributes today; however those do not have any affect on the JVM, they just change what reflection returns. Redefinition does not allow adding trampoline methods. thanks, Karen From karen.kinnear at oracle.com Wed Dec 20 15:45:17 2017 From: karen.kinnear at oracle.com (Karen Kinnear) Date: Wed, 20 Dec 2017 10:45:17 -0500 Subject: Nestmates JVMS changes to selection and behavior implications Message-ID: <82E5D22B-C4DE-4C60-9AAA-62291CD9FDC2@oracle.com> David Holmes and I were studying the JVMS changes for nestmates - in particular the selection and preparation changes which are designed to make invoke interface more like invoke virtual (JDK-8024806). I wanted to ensure that the behavior changes were all either error -> success, or error 1 -> error 2. As I went through this again, I had a couple more questions on the JVMS changes for selection preparation to sanity check. thanks, Karen -------------- next part -------------- From forax at univ-mlv.fr Wed Dec 20 16:00:42 2017 From: forax at univ-mlv.fr (Remi Forax) Date: Wed, 20 Dec 2017 17:00:42 +0100 (CET) Subject: nestmates JVMTI spec proposal changes In-Reply-To: References: <1B2ECA5E-402C-41F6-954E-4987BF7199A2@oracle.com> Message-ID: <1947897958.1334237.1513785642920.JavaMail.zimbra@u-pem.fr> Given that we will have a NestMate phase 2, i agree with Dan, we can revisit that point later. R?mi > De: "Daniel Heidinga" > ?: "Karen Kinnear" > Cc: "valhalla-spec-experts" > Envoy?: Mercredi 20 D?cembre 2017 16:58:03 > Objet: Re: nestmates JVMTI spec proposal changes > We're on board with this proposal. Any change to the NestMembers, adding or > removing members, should fail redefinition. As should changes to NestHost. > Taking this approach now eases the implementation costs and allows us to > re-examine this in the future. > --Dan >> ----- Original message ----- >> From: Karen Kinnear >> Sent by: "valhalla-spec-experts" >> >> To: valhalla-spec-experts >> Cc: >> Subject: nestmates JVMTI spec proposal changes >> Date: Tue, Dec 19, 2017 5:47 PM >> I believe we are all in agreement that: >> 1. Redefinition should NOT be allowed to change the NestHost attribute and >> 2. Redefinition should NOT be allowed to remove NestMembers (equivalent to >> reducing access controls) >> The question was - should we allow Redefinition to add NestMembers or not? >> Hotspot team would like to propose that at least for the initial nestmates >> release - we do NOT allow >> Redefinition to add NestMembers. >> In all cases, NOT allow would mean a failure to redefine a class. >> Open to suggestions on error that should be returned. >> This is the least risky approach and allows future reducing restrictions if we >> find we need to. It is >> extremely difficult to increase restrictions. >> Discussion: >> JVMTI redefinition restrictions today: >> The redefinition may change method bodies, the constant pool and attributes. The >> redefinition must not add, remove or rename fields or methods, change the >> signatures of methods, change modifiers, or change inheritance. These >> restrictions may be lifted in future versions. See the error return description >> below for information on error codes returned if an unsupported redefinition is >> attempted. The class file bytes are not verified or installed until they have >> passed through the chain of [ >> https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.oracle.com_javase_8_docs_platform_jvmti_jvmti.html-23ClassFileLoadHook&d=DwMFAg&c=jf_iaSHvJObTbx-siA1ZOg&r=LBQnmyrHQkEBElM8bAxhzfwLG2HbsYDdzEznFrQoob4&m=o8e88cUxfs0Xn3ibrFYJf9-m6XEnAau1Hr_tRPsFNqE&s=J0Lf6yC70kW4MaNf8VLwXaPcEPV4wj0IAIH3yWGvI7Q&e= >> | ClassFileLoadHook ] events, thus the returned error code reflects the result >> of the transformations applied to the bytes passed into [ >> https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.oracle.com_javase_8_docs_platform_jvmti_jvmti.html-23RedefineClasses.class-5Fdefinitions&d=DwMFAg&c=jf_iaSHvJObTbx-siA1ZOg&r=LBQnmyrHQkEBElM8bAxhzfwLG2HbsYDdzEznFrQoob4&m=o8e88cUxfs0Xn3ibrFYJf9-m6XEnAau1Hr_tRPsFNqE&s=F0smHtdTXv7JiNt8sNItG9TAJfAT4UHZdHEId0HXWpQ&e= >> | class_definitions ] . If any error code is returned other than >> JVMTI_ERROR_NONE , none of the classes to be redefined will have a new >> definition installed. When this function returns (with the error code of >> JVMTI_ERROR_NONE ) all of the classes to be redefined will have their new >> definitions installed. >> Note that redefinition today is not allowed to change modifiers or change >> inheritance, so no changes that could change access control >> behavior. >> The jigsaw AddExports allows dynamic increasing of access to types. To ensure >> that resolution >> of a constant pool type entries that fails always fails in the same way, the >> implementation must cache those >> failures. >> If we were to allow dynamic increases in member access, to ensure that >> resolution of content pool member >> entries (fields, methods) would fail in the same way, the implementation would >> need to cache those >> resolution failures, which is non-trivial, and would need to ensure that the >> redefinition implementation >> retained errors for resolution from other types. >> Note that Redefinition is allowed to change inner class attributes today; >> however those do not >> have any affect on the JVM, they just change what reflection returns. >> Redefinition does not >> allow adding trampoline methods. >> thanks, >> Karen From david.holmes at oracle.com Wed Dec 20 20:53:50 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 21 Dec 2017 06:53:50 +1000 Subject: Nestmates JVMS changes to selection and behavior implications In-Reply-To: <82E5D22B-C4DE-4C60-9AAA-62291CD9FDC2@oracle.com> References: <82E5D22B-C4DE-4C60-9AAA-62291CD9FDC2@oracle.com> Message-ID: <23a8ac5c-b777-b939-ad0e-ee36f6e979d6@oracle.com> For the benefit of those on the observer list Karen sent an attachment which has been stripped. David On 21/12/2017 1:45 AM, Karen Kinnear wrote: > David Holmes and I were studying the JVMS changes for nestmates - in particular the selection and preparation changes which > are designed to make invoke interface more like invoke virtual (JDK-8024806). > > I wanted to ensure that the behavior changes were all either error -> success, or error 1 -> error 2. > > As I went through this again, I had a couple more questions on the JVMS changes for selection preparation to sanity check. > > thanks, > Karen > > > > > > > From daniel.smith at oracle.com Wed Dec 20 23:49:37 2017 From: daniel.smith at oracle.com (Dan Smith) Date: Wed, 20 Dec 2017 16:49:37 -0700 Subject: Structural descriptors in the constant pool Message-ID: As discussed at the meeting today, here's an outline of what a structural descriptor (pointer-based rather than string-based) in the constant pool might look like. New constants representing types: CONSTANT_PrimitiveType_info { u1 tag; u1 kind; // any valid 'atype' as specified for 'newarray', or T_VOID } CONSTANT_ArrayType_info { u1 tag; u2 component_type_index; // a type constant } (An alternative encoding of ArrayType would consist of an element type and a dimensions count. Putting each lower-dimension array type in its own constant potentially improves sharing, but adds overhead where the component types would otherwise go unused.) A "type constant" is one of: - Class - ArrayType - PrimitiveType For historical reasons, a CONSTANT_Class may also refer to an array type, but this usage is discouraged. New constant and attribute for method descriptors (in the spirit of NameAndType and BootstrapMethods, these are supplementary substructures of a MethodType): CONSTANT_ParametersAndReturn_info { u1 tag; u2 return_type_index; // a type constant u2 parameter_types_attr_index; // an entry in TypeLists } TypeLists_attribute { u2 attribute_name_index; // Utf8 "TypeLists" u4 attribute_length; u2 num_type_lists; { u2 num_types; u2 types[num_types]; // type constants } } (Two design goals constraining this solution: 1) re-use MethodType as the preferred representation of method descriptors, and 2) avoid introducing a new variable-length constant pool entry. Thus, two levels of indirection, one to provide a non-Utf8 constant for MethodType to reference, and another to offload the variable-length list to an attribute. We can drop these goals to reduce indirections.) Changes in usage of constants: - A MethodType can refer to a ParametersAndReturn (preferred) or a Utf8 method descriptor (legacy) - The descriptor_index of a field may refer to a type constant (preferred) or a Utf8 field descriptor (legacy) - The descriptor_index of a method may refer to a MethodType (preferred) or a Utf8 method descriptor (legacy) - The descriptor of a NameAndType may refer to a type constant or MethodType (preferred), or a Utf8 field/method descriptor (legacy) - ldc can refer any type constant - checkcast/instanceof can refer to a Class or an ArrayType - anewarray/multianewarray can refer to any type constant (and arguably the opcode names should be changed; newarray can be viewed as a compact shortcut, like iconst_0) Changes in interpreting descriptors: - A MethodType that refers to a Utf8 treats the descriptor as if it were expressed with a fresh ParametersAndReturn referencing fresh type constants - A field or NameAndType that refers to a Utf8 field descriptor treats the descriptor as if it were expressed with a fresh type constant - A method or NameAndType that refers to a Utf8 method descriptor treats the descriptor as if it were expressed with a fresh MethodType referencing fresh type constants - Descriptor comparisons (during field/method resolution and method selection) continue to occur *without resolving* the referenced type constants. They are simply tree equality tests. - Class loader constraints are generated for matching Class constants, identified by recurring through the trees. (Note that implementations can continue to work with strings, if desired.) From daniel.smith at oracle.com Thu Dec 21 20:18:04 2017 From: daniel.smith at oracle.com (Dan Smith) Date: Thu, 21 Dec 2017 13:18:04 -0700 Subject: JLS spec changes for nestmates In-Reply-To: References: <86C4EB0E-6B50-44EB-BBBF-736911BB6027@oracle.com> <29131B95-F935-4BF0-8241-CF0B67B03523@oracle.com> Message-ID: <6E5A0F76-D5F8-4FDE-9ED9-90409A20C147@oracle.com> > On Dec 19, 2017, at 1:50 PM, David Holmes wrote: > > On 19/12/2017 6:53 AM, Dan Smith wrote: >>> On Dec 17, 2017, at 6:15 PM, David Holmes wrote: >>> >>>> I looked over 13.1, and think it does a fine job of allowing binaries to use the NestHost and NestMembers attributes without any changes (see items 9 and 10 in the second list). >>> >>> I don't follow. Items 9 and 10 seem to pertain to the InnerClasses attribute. NestHost and NestMembers can not, in general, be used to convey the information required by items 9 and 10. The NestHost attribute need not refer to the immediately enclosing class. The NestMembers attribute only appears in the top-level NestHost class. >> Yeah, you're right. This section is very hand-wavy, so I tend to give it some latitude to say things that aren't exactly right. And there's a difference between _allowing_ a reference to something and _requiring_ it. But we could probably do a better job here and require the references we expect to appear in NestHost/NestMembers. > > But I didn't think it was a requirement to use the new attributes. I thought a source compiler could choose to continue to generate bridge accessors and not need the new attributes? (That's what the JEP implies.) Yep, ran this by Maurizio, he agrees it's optional. So it seems like the right thing to do is leave 13.1 unchanged. As far as I can tell, an implementation can choose to add references to classes that aren't enumerated in 13.1, and we don't need to say anything about it. ?Dan