From zoltan.majo at oracle.com Mon Apr 3 09:16:49 2017 From: zoltan.majo at oracle.com (=?UTF-8?B?Wm9sdMOhbiBNYWrDsw==?=) Date: Mon, 3 Apr 2017 11:16:49 +0200 Subject: RFR (M): Generate checks for vbox/vunbox; extend CI to include VCC-DVT mapping In-Reply-To: <79c73089-912f-fd6d-66b1-e1b0671c7bfa@oracle.com> References: <79c73089-912f-fd6d-66b1-e1b0671c7bfa@oracle.com> Message-ID: <2291f0b4-453f-063a-b170-b15d5cd04571@oracle.com> Hi Tobias, thank you for the feedback. Please see below my replies to the issues you've raised. On 03/30/2017 10:26 AM, Tobias Hartmann wrote: > Hi Zoltan, > > On 28.03.2017 13:20, Zolt?n Maj? wrote: >> http://cr.openjdk.java.net/~zmajo/valhalla/03.vbox-vunbox-checks/webrev.00/ >> >> The patch adds checks to vbox/vunbox that are missing in the current implementation. The patch enables C2 to generate code that includes null checks and type checks (where required). Other checks (e.g., checking if the source/target class of the bytecode instruction is loaded and if it is a value-capable-class/derived value type) are performed at compile-time. The patch also extends the CI to mirror the VCC-DVT mapping to the compiler. > Here are some comments/questions: > - I think it's sufficient to have the "// TODO: Check if updating flag works properly" in one place OK, I changed to code to have the comment only at one place. > - In graphKit.cpp, why do you set 'treat_throw_as_hot' to always true? Shouldn't the checks in line 549 be sufficient to set it (also for the vbox/vunbox bytecodes)? I was curious how the code with pre-generated looks like. But you are right, we can use profiling information about traps and use pre-generated exceptions only if we hit the trap limit (just as in the "standard" VM). > - Since we don't support subtyping for value types, shouldn't we check for type equality instead of a subtype relation? > - In vbox(), you can get the exact type of the ValueTypeNode via ValueTypeNode::value_klass() and you can check that for equality with the target_dvt_klass I agree -- let's do that. > - In vunbox(), I would check for equality of the source_type and the target_vcc_klass In some cases (e.g., when the declared type source_type is java.lang.Object) the check you mention is not sufficient. We need to generate a dynamic type check in that case -- test_64 covers that scenario. Here is the updated webrev: http://cr.openjdk.java.net/~zmajo/valhalla/03.vbox-vunbox-checks/webrev.01/ I ran all hotspot_valhalla tests, all pass. If the changes look good, I will run the patch through JPRT and with Octane before pushing. Thank you! Best regards, Zoltan > > Thanks, > Tobias From tobias.hartmann at oracle.com Mon Apr 3 13:12:01 2017 From: tobias.hartmann at oracle.com (Tobias Hartmann) Date: Mon, 3 Apr 2017 15:12:01 +0200 Subject: RFR (M): Generate checks for vbox/vunbox; extend CI to include VCC-DVT mapping In-Reply-To: <2291f0b4-453f-063a-b170-b15d5cd04571@oracle.com> References: <79c73089-912f-fd6d-66b1-e1b0671c7bfa@oracle.com> <2291f0b4-453f-063a-b170-b15d5cd04571@oracle.com> Message-ID: Hi Zoltan, On 03.04.2017 11:16, Zolt?n Maj? wrote: >> - In graphKit.cpp, why do you set 'treat_throw_as_hot' to always true? Shouldn't the checks in line 549 be sufficient to set it (also for the vbox/vunbox bytecodes)? > > I was curious how the code with pre-generated looks like. But you are right, we can use profiling information about traps and use pre-generated exceptions only if we hit the trap limit (just as in the "standard" VM). Okay, thanks for checking! >> - Since we don't support subtyping for value types, shouldn't we check for type equality instead of a subtype relation? >> - In vbox(), you can get the exact type of the ValueTypeNode via ValueTypeNode::value_klass() and you can check that for equality with the target_dvt_klass > > I agree -- let's do that. Looks good. I just wonder if we really need 'guarantee' for the static checks. Existing code uses 'assert' for similar checks but it doesn't matter too much for now. >> - In vunbox(), I would check for equality of the source_type and the target_vcc_klass > > In some cases (e.g., when the declared type source_type is java.lang.Object) the check you mention is not sufficient. We need to generate a dynamic type check in that case -- test_64 covers that scenario. Sure but what I meant is a dynamic check for type equality of the VCC (derived from the target value type) and the source type instead of a subtype check. Because we don't support subtyping, right? Test64 should only ever pass if 'vcc' is ValueCapableClass1. For example, with VCC A and class B which is a subtype of A, vunbox of B to A$Value should fail, right? I think we should also add a test for this case. Thanks, Tobias From zoltan.majo at oracle.com Mon Apr 3 14:08:02 2017 From: zoltan.majo at oracle.com (=?UTF-8?B?Wm9sdMOhbiBNYWrDsw==?=) Date: Mon, 3 Apr 2017 16:08:02 +0200 Subject: RFR (M): Generate checks for vbox/vunbox; extend CI to include VCC-DVT mapping In-Reply-To: References: <79c73089-912f-fd6d-66b1-e1b0671c7bfa@oracle.com> <2291f0b4-453f-063a-b170-b15d5cd04571@oracle.com> Message-ID: Hi Tobias, thank you for the feedback. On 04/03/2017 03:12 PM, Tobias Hartmann wrote: > [...] > >>> - Since we don't support subtyping for value types, shouldn't we check for type equality instead of a subtype relation? >>> - In vbox(), you can get the exact type of the ValueTypeNode via ValueTypeNode::value_klass() and you can check that for equality with the target_dvt_klass >> I agree -- let's do that. > Looks good. I just wonder if we really need 'guarantee' for the static checks. Existing code uses 'assert' for similar checks but it doesn't matter too much for now. OK, let's keep the guarantees for now and change them to asserts later on if needed. > >>> - In vunbox(), I would check for equality of the source_type and the target_vcc_klass >> In some cases (e.g., when the declared type source_type is java.lang.Object) the check you mention is not sufficient. We need to generate a dynamic type check in that case -- test_64 covers that scenario. > Sure but what I meant is a dynamic check for type equality of the VCC (derived from the target value type) and the source type instead of a subtype check. Oh, I see. The code in webrev.01 generates code to perform a type equality check (and does not generate subtype checks). > Because we don't support subtyping, right? That is right. > Test64 should only ever pass if 'vcc' is ValueCapableClass1. That is correct and that's what also happens. > > For example, with VCC A and class B which is a subtype of A, vunbox of B to A$Value should fail, right? I think we should also add a test for this case. That is also correct. Unfortunately, we cannot add a test like that because VCCs must be final (I tried making ValueCapableClass1 but that causes an java.lang.InternalError to be thrown "DeriveValueType class 'compiler.valhalla.valuetypes.ValueCapableClass1' is not a final class"). Thank you! Best regards, Zoltan > Thanks, > Tobias From tobias.hartmann at oracle.com Mon Apr 3 14:23:20 2017 From: tobias.hartmann at oracle.com (Tobias Hartmann) Date: Mon, 3 Apr 2017 16:23:20 +0200 Subject: RFR (M): Generate checks for vbox/vunbox; extend CI to include VCC-DVT mapping In-Reply-To: References: <79c73089-912f-fd6d-66b1-e1b0671c7bfa@oracle.com> <2291f0b4-453f-063a-b170-b15d5cd04571@oracle.com> Message-ID: <5d97e87d-0c34-434d-0d8b-00ee9bc8e9c9@oracle.com> Hi Zoltan, On 03.04.2017 16:08, Zolt?n Maj? wrote: >> Looks good. I just wonder if we really need 'guarantee' for the static checks. Existing code uses 'assert' for similar checks but it doesn't matter too much for now. > > OK, let's keep the guarantees for now and change them to asserts later on if needed. Okay, sounds good. >>>> - In vunbox(), I would check for equality of the source_type and the target_vcc_klass >>> In some cases (e.g., when the declared type source_type is java.lang.Object) the check you mention is not sufficient. We need to generate a dynamic type check in that case -- test_64 covers that scenario. >> Sure but what I meant is a dynamic check for type equality of the VCC (derived from the target value type) and the source type instead of a subtype check. > > Oh, I see. The code in webrev.01 generates code to perform a type equality check (and does not generate subtype checks). Right, I got confused by the 'target_vcc_klass->is_subclass_of(source_type->klass()' check. >> Because we don't support subtyping, right? > > That is right. > >> Test64 should only ever pass if 'vcc' is ValueCapableClass1. > > That is correct and that's what also happens. Okay, thanks for verifying. >> For example, with VCC A and class B which is a subtype of A, vunbox of B to A$Value should fail, right? I think we should also add a test for this case. > > That is also correct. Unfortunately, we cannot add a test like that because VCCs must be final (I tried making ValueCapableClass1 but that causes an java.lang.InternalError to be thrown "DeriveValueType class 'compiler.valhalla.valuetypes.ValueCapableClass1' is not a final class"). Okay, if this ever changes we can still add a test. Your changes look good to me! Thanks, Tobias From zoltan.majo at oracle.com Mon Apr 3 14:50:41 2017 From: zoltan.majo at oracle.com (=?UTF-8?B?Wm9sdMOhbiBNYWrDsw==?=) Date: Mon, 3 Apr 2017 16:50:41 +0200 Subject: RFR (M): Generate checks for vbox/vunbox; extend CI to include VCC-DVT mapping In-Reply-To: <5d97e87d-0c34-434d-0d8b-00ee9bc8e9c9@oracle.com> References: <79c73089-912f-fd6d-66b1-e1b0671c7bfa@oracle.com> <2291f0b4-453f-063a-b170-b15d5cd04571@oracle.com> <5d97e87d-0c34-434d-0d8b-00ee9bc8e9c9@oracle.com> Message-ID: Hi Tobias, On 04/03/2017 04:23 PM, Tobias Hartmann wrote: > [...] > > Your changes look good to me! Thank you for the review! Best regards, Zoltan > > Thanks, > Tobias From maurizio.cimadamore at oracle.com Mon Apr 3 16:56:18 2017 From: maurizio.cimadamore at oracle.com (maurizio.cimadamore at oracle.com) Date: Mon, 03 Apr 2017 16:56:18 +0000 Subject: hg: valhalla/valhalla/nashorn: Fix: add support for nestmate attributes to nasgen Message-ID: <201704031656.v33GuI3G002845@aojmv0008.oracle.com> Changeset: c336be86b44d Author: mcimadamore Date: 2017-04-03 17:55 +0100 URL: http://hg.openjdk.java.net/valhalla/valhalla/nashorn/rev/c336be86b44d Fix: add support for nestmate attributes to nasgen ! buildtools/nasgen/src/jdk/nashorn/internal/tools/nasgen/Main.java From zoltan.majo at oracle.com Tue Apr 4 10:52:04 2017 From: zoltan.majo at oracle.com (zoltan.majo at oracle.com) Date: Tue, 04 Apr 2017 10:52:04 +0000 Subject: hg: valhalla/valhalla/hotspot: Generate checks for vbox/vunbox; extend CI to include VCC-DVT mapping Message-ID: <201704041052.v34Aq4Vd025275@aojmv0008.oracle.com> Changeset: 382b4bedf348 Author: zmajo Date: 2017-04-04 12:50 +0200 URL: http://hg.openjdk.java.net/valhalla/valhalla/hotspot/rev/382b4bedf348 Generate checks for vbox/vunbox; extend CI to include VCC-DVT mapping Reviewed-by: thartmann ! src/share/vm/ci/ciInstanceKlass.cpp ! src/share/vm/ci/ciInstanceKlass.hpp ! src/share/vm/oops/instanceKlass.hpp ! src/share/vm/opto/parse3.cpp ! test/compiler/valhalla/valuetypes/ValueCapableClass1.java + test/compiler/valhalla/valuetypes/ValueCapableClass2.java ! test/compiler/valhalla/valuetypes/ValueTypeTestBench.java From david.simms at oracle.com Tue Apr 4 11:58:11 2017 From: david.simms at oracle.com (david.simms at oracle.com) Date: Tue, 04 Apr 2017 11:58:11 +0000 Subject: hg: valhalla/valhalla/jdk: Correct args to hashCode Message-ID: <201704041158.v34BwBWF012761@aojmv0008.oracle.com> Changeset: 6e66a983e953 Author: dsimms Date: 2017-04-04 13:57 +0200 URL: http://hg.openjdk.java.net/valhalla/valhalla/jdk/rev/6e66a983e953 Correct args to hashCode ! src/java.base/share/classes/jdk/experimental/value/ValueType.java From david.simms at oracle.com Wed Apr 5 08:54:26 2017 From: david.simms at oracle.com (david.simms at oracle.com) Date: Wed, 05 Apr 2017 08:54:26 +0000 Subject: hg: valhalla/valhalla/hotspot: Minor fix for removing __Value from array class hierarchy Message-ID: <201704050854.v358sQuQ012618@aojmv0008.oracle.com> Changeset: 3dccbd2a8cd0 Author: dsimms Date: 2017-04-05 10:53 +0200 URL: http://hg.openjdk.java.net/valhalla/valhalla/hotspot/rev/3dccbd2a8cd0 Minor fix for removing __Value from array class hierarchy ! src/share/vm/oops/objArrayKlass.cpp From brian.goetz at oracle.com Tue Apr 11 13:57:15 2017 From: brian.goetz at oracle.com (Brian Goetz) Date: Tue, 11 Apr 2017 09:57:15 -0400 Subject: New JEP: Javac intrinsics for LDC and INVOKEDYNAMIC Message-ID: Please review the this JEP draft: https://bugs.openjdk.java.net/browse/JDK-8178320 This JEP creates (a) an official set of purely-nominal wrapper classes for linkable constants (Class, MethodType, MethodHandle, VarHandle), provides constant-propagation (not folding) support in javac for these types, and (b) intrinsification for LDC and INVOKEDYNAMIC so that they can be predictably translated into bytecode. This allows Java source code to be a first-class client of invokedynamic, including using Java to test indy bootstraps. From forax at univ-mlv.fr Tue Apr 11 15:29:27 2017 From: forax at univ-mlv.fr (Remi Forax) Date: Tue, 11 Apr 2017 17:29:27 +0200 (CEST) Subject: New JEP: Javac intrinsics for LDC and INVOKEDYNAMIC In-Reply-To: References: Message-ID: <252682735.840434.1491924567159.JavaMail.zimbra@u-pem.fr> I like this JEP :) constant propagation: the rule about arrays should be added, an array of constants is a constant. ldc: i don't like having to use another constant hierarchy, i would prefer to have static factories that act as compiler intrinsics from String/int/... representation and avoid an intermediary representation, (the intermediary representation should be inside the compiler). class CompilerIntrinsics { public static Class constClass(String className) { return null; } public static MethodType constMethodType(String descriptor) { return null; } public static MethodHandle constMethodHandle(int tag, Class type, String name, MethodType type) { return null; } } by example for constMethodHandle, the compiler - requires all arguments to be constants - emits an error if the underlying method do not exist or the tag is not valid, etc at compile time, - replace the call with a ldc with the right constant pool constant if the constant is not constant-folded because it's part of another constant so MethodHandle mh = constMethodHandle(6, constClass("java/lang/Integer"), "parseInt", constMethodType("(Ljava/lang/String)I")); creates a method handle (with ldc) on Integer.parseInt() without creating the intermediary Class and MethodType because they are both constant folded. apart if mh is itself part of a bigger constant. invokedynamic: - the proposed encoding does not allow to encode an invokedynamic with 255 arguments (useful for testing) - the BootstrapSpecifier should only contain a constant MethodHandle and an array of constant and a name, the descriptor should be computed by the compiler from the callsite, what @PolymorphicSignature already does. a counter-proposal, let say we have an empty class class InvokedynamicCompilerIntrinsics { // empty } introduce a new syntax, InvokedynamicCompilerIntrinsics.[bsm, arg1, arg2]foo(regular_arguments) that cleanly separate the bootstrap method, the BSM arguments and the regular arguments. The descriptor is computed as is the there is a method "foo" in InvokedynamicCompilerIntrinsics declared @PolymorphicSignature. The BSM and the BSM arguments has to be constant (CE). regards, R?mi ----- Mail original ----- > De: "Brian Goetz" > ?: valhalla-dev at openjdk.java.net > Envoy?: Mardi 11 Avril 2017 15:57:15 > Objet: New JEP: Javac intrinsics for LDC and INVOKEDYNAMIC > Please review the this JEP draft: > > https://bugs.openjdk.java.net/browse/JDK-8178320 > > This JEP creates (a) an official set of purely-nominal wrapper classes for > linkable constants (Class, MethodType, MethodHandle, VarHandle), provides > constant-propagation (not folding) support in javac for these types, and (b) > intrinsification for LDC and INVOKEDYNAMIC so that they can be predictably > translated into bytecode. > > This allows Java source code to be a first-class client of invokedynamic, > including using Java to test indy bootstraps. From brian.goetz at oracle.com Tue Apr 11 16:50:56 2017 From: brian.goetz at oracle.com (Brian Goetz) Date: Tue, 11 Apr 2017 12:50:56 -0400 Subject: New JEP: Javac intrinsics for LDC and INVOKEDYNAMIC In-Reply-To: <252682735.840434.1491924567159.JavaMail.zimbra@u-pem.fr> References: <252682735.840434.1491924567159.JavaMail.zimbra@u-pem.fr> Message-ID: <0218aaf7-e08c-afce-5ab6-ef854d814d71@oracle.com> > I like this JEP :) I thought you might :) It took us a long time to find a way of surfacing this feature that we like. This is at least our fourth design iteration over several years; the others weren't worth sharing :) > ldc: > i don't like having to use another constant hierarchy, > i would prefer to have static factories Let's save the API design discussion for later -- right now, review comments should focus on the scope of the JEP. But as a general goal, we're trying to keep the surface area of the intrinsified portion small -- right now, it's two methods (ldc and indy). Remember, this is an advanced tool for low-level programming; convenience methods may not be as needed as they might be with other APIs. > by example for constMethodHandle, the compiler > - requires all arguments to be constants > - emits an error if the underlying method do not exist or the tag is not valid, etc at compile time, > - replace the call with a ldc with the right constant pool constant if the constant is not constant-folded because it's part of another constant The LDC intrinsification is the point at which we must enforce that we have a constant in hand. The XxxConstant classes can freely tolerate non-constant content (perhaps they should be renamed), it's only when we want to intrinsify that a constant is truly needed. > - the BootstrapSpecifier should only contain a constant MethodHandle and an array of constant and a name, the descriptor should be computed by the compiler from the callsite, what @PolymorphicSignature already does. That's a valid alternative (which we considered.) Again, if the goal is to allow explicit control over indy, it seems preferable to put the MethodType in the indy descriptor -- as this more closely models the BootstrapMethods entry -- and have that take precedence -- and then we can type-check the call site (and adapt args/returns appropriately) against the "official" descriptor. > a counter-proposal, let say we have an empty class > class InvokedynamicCompilerIntrinsics { > // empty > } > > introduce a new syntax, I understand why you want this, but .... no :) It is an explicit anti-goal to have language syntax for this feature. (Finding an acceptable way to do this is part of why we had to wait this long.) This is a feature for a few hundred people, at most. Adding language surface for this would merely encourage the book authors / training providers of the world to include it (either because it seems new and cool, or out of a desire to be comprehensive) -- thereby exposing the other 99.9999% of the Java ecosystem to a feature they will never -- and should never -- use. (Balancing the needs of advanced developers while not polluting "everyone's Java" is a very difficult balance.) From forax at univ-mlv.fr Tue Apr 11 18:11:17 2017 From: forax at univ-mlv.fr (forax at univ-mlv.fr) Date: Tue, 11 Apr 2017 20:11:17 +0200 (CEST) Subject: New JEP: Javac intrinsics for LDC and INVOKEDYNAMIC In-Reply-To: <0218aaf7-e08c-afce-5ab6-ef854d814d71@oracle.com> References: <252682735.840434.1491924567159.JavaMail.zimbra@u-pem.fr> <0218aaf7-e08c-afce-5ab6-ef854d814d71@oracle.com> Message-ID: <126500153.890641.1491934277483.JavaMail.zimbra@u-pem.fr> ----- Mail original ----- > De: "Brian Goetz" > ?: "Remi Forax" > Cc: valhalla-dev at openjdk.java.net > Envoy?: Mardi 11 Avril 2017 18:50:56 > Objet: Re: New JEP: Javac intrinsics for LDC and INVOKEDYNAMIC >> I like this JEP :) > > I thought you might :) > > It took us a long time to find a way of surfacing this feature that we > like. This is at least our fourth design iteration over several years; > the others weren't worth sharing :) > >> ldc: >> i don't like having to use another constant hierarchy, >> i would prefer to have static factories > > Let's save the API design discussion for later -- right now, review > comments should focus on the scope of the JEP. But as a general goal, > we're trying to keep the surface area of the intrinsified portion small > -- right now, it's two methods (ldc and indy). Remember, this is an > advanced tool for low-level programming; convenience methods may not be > as needed as they might be with other APIs. The compiler also have to know all implementations of Constable and all methods that override Constable.resolveConstant, so your API have more entry points. What i propose decouple how the user specify a constant from how its represented in the constant pool. The API i propose - may not support the 'ConstantDynamic' depending on what exactly a ConstantDynamic is. - do not support unknown constant pool constant but that's an explicit no goal. The main difference is that you propose an API based on a tree of object while i think that a constant can be represented as a tree of method call. > >> by example for constMethodHandle, the compiler >> - requires all arguments to be constants >> - emits an error if the underlying method do not exist or the tag is not valid, >> etc at compile time, >> - replace the call with a ldc with the right constant pool constant if the >> constant is not constant-folded because it's part of another constant > > The LDC intrinsification is the point at which we must enforce that we > have a constant in hand. The XxxConstant classes can freely tolerate > non-constant content (perhaps they should be renamed), it's only when we > want to intrinsify that a constant is truly needed. yes, so you do not need XxxConstant per se but just a method that takes parameters and returns a constant pool constant, hence the API i propose. The API you propose is more powerful because it allows to create objects that are considered as constant by the compiler but given that at the end you can only emit ldc of the existing constants. If think that we can have a simpler API using method call instead of object from the user POV and you are free to inside the compiler uses an API like the one you propose to implement the simpler API. > >> - the BootstrapSpecifier should only contain a constant MethodHandle and an >> array of constant and a name, the descriptor should be computed by the compiler >> from the callsite, what @PolymorphicSignature already does. > > That's a valid alternative (which we considered.) Again, if the goal is > to allow explicit control over indy, it seems preferable to put the > MethodType in the indy descriptor -- as this more closely models the > BootstrapMethods entry -- and have that take precedence -- and then we > can type-check the call site (and adapt args/returns appropriately) > against the "official" descriptor. @PolymorphicSignature let you add casts to get the 'official' descriptor you want. Here, you have a redundant info so the compiler - can use the descriptor encoded in the bootstrap specifier to do inference (right ?) - has to verify that the signature computed by the algo behind @PolymorphicSignature is a subsignature of the one of the bootstrap specifier. So this is a new specific way to transfer signature info to the compiler, not @PolymorphicSignature per se. I do not see the point to add another algorithm to do @PolymorphicSignature but not exactly in the compiler and the Java spec. > >> a counter-proposal, let say we have an empty class >> class InvokedynamicCompilerIntrinsics { >> // empty >> } >> >> introduce a new syntax, > > I understand why you want this, but .... no :) > > It is an explicit anti-goal to have language syntax for this feature. > (Finding an acceptable way to do this is part of why we had to wait this > long.) This is a feature for a few hundred people, at most. Adding > language surface for this would merely encourage the book authors / > training providers of the world to include it (either because it seems > new and cool, or out of a desire to be comprehensive) -- thereby > exposing the other 99.9999% of the Java ecosystem to a feature they will > never -- and should never -- use. (Balancing the needs of advanced > developers while not polluting "everyone's Java" is a very difficult > balance.) If it's only a few hundred, i know all of them :) I teach the java.lang.invoke API since 2011 [1], so a few hundred at my uni have all used ASM to encode enough invokedynamics to implement a small dynamic language :) And what about a Java compatible syntax that still separate the bootstrap arguments from the regular arguments ? class InvokedynamicCompilerIntrinsics { // empty } MethodHanlde bsm = ... // a constant method handle to a bootstrap method Xxx bsmargs = ... // some constants InvokedynamicCompilerIntrinsics.bsm(bsm, bsmargs, ...).foo(args, ...) regards, R?mi [1] http://monge.univ-mlv.fr/ens/Master/M2/2016-2017/VM/index.php the main page is in french but not the different labs. From brian.goetz at oracle.com Tue Apr 11 18:27:56 2017 From: brian.goetz at oracle.com (Brian Goetz) Date: Tue, 11 Apr 2017 14:27:56 -0400 Subject: New JEP: Javac intrinsics for LDC and INVOKEDYNAMIC In-Reply-To: <126500153.890641.1491934277483.JavaMail.zimbra@u-pem.fr> References: <252682735.840434.1491924567159.JavaMail.zimbra@u-pem.fr> <0218aaf7-e08c-afce-5ab6-ef854d814d71@oracle.com> <126500153.890641.1491934277483.JavaMail.zimbra@u-pem.fr> Message-ID: <9130658f-f125-80ff-8a78-3a27f8668ec0@oracle.com> You don't need a separate intrinsic for that. Intrinsics.indy(BootstrapSpecifier.of(bsm, bsmArgs), Object... dynArgs); On 4/11/2017 2:11 PM, forax at univ-mlv.fr wrote: > And what about a Java compatible syntax that still separate the bootstrap arguments from the regular arguments ? > > class InvokedynamicCompilerIntrinsics { > // empty > } > > MethodHanlde bsm = ... // a constant method handle to a bootstrap method > Xxx bsmargs = ... // some constants > InvokedynamicCompilerIntrinsics.bsm(bsm, bsmargs, ...).foo(args, ...) From forax at univ-mlv.fr Tue Apr 11 18:32:04 2017 From: forax at univ-mlv.fr (forax at univ-mlv.fr) Date: Tue, 11 Apr 2017 20:32:04 +0200 (CEST) Subject: New JEP: Javac intrinsics for LDC and INVOKEDYNAMIC In-Reply-To: <9130658f-f125-80ff-8a78-3a27f8668ec0@oracle.com> References: <252682735.840434.1491924567159.JavaMail.zimbra@u-pem.fr> <0218aaf7-e08c-afce-5ab6-ef854d814d71@oracle.com> <126500153.890641.1491934277483.JavaMail.zimbra@u-pem.fr> <9130658f-f125-80ff-8a78-3a27f8668ec0@oracle.com> Message-ID: <982352343.892448.1491935524284.JavaMail.zimbra@u-pem.fr> ----- Mail original ----- > De: "Brian Goetz" > ?: forax at univ-mlv.fr > Cc: valhalla-dev at openjdk.java.net > Envoy?: Mardi 11 Avril 2017 20:27:56 > Objet: Re: New JEP: Javac intrinsics for LDC and INVOKEDYNAMIC > You don't need a separate intrinsic for that. > > Intrinsics.indy(BootstrapSpecifier.of(bsm, bsmArgs), > Object... dynArgs); yes, i was trying to avoid the class 'BootstrapSpecifier', but it's perhaps better to use it and teach the compiler about it. R?mi > > On 4/11/2017 2:11 PM, forax at univ-mlv.fr wrote: >> And what about a Java compatible syntax that still separate the bootstrap >> arguments from the regular arguments ? >> >> class InvokedynamicCompilerIntrinsics { >> // empty >> } >> >> MethodHanlde bsm = ... // a constant method handle to a bootstrap method >> Xxx bsmargs = ... // some constants > > InvokedynamicCompilerIntrinsics.bsm(bsm, bsmargs, ...).foo(args, ...) From paul.sandoz at oracle.com Tue Apr 11 18:38:05 2017 From: paul.sandoz at oracle.com (Paul Sandoz) Date: Tue, 11 Apr 2017 11:38:05 -0700 Subject: New JEP: Javac intrinsics for LDC and INVOKEDYNAMIC In-Reply-To: References: Message-ID: > (This means that one can create a description for a class that is not loaded, and that creating a descriptor does not trigger any class loading.) Nor any class initialization (thereby in some cases, at least in JDK allowing us the possibility circumvent bootstrap issues, that?s my hope...). Perhaps mention that it will not be possible to reflectively invoke (via MethodHandles or Method) the compiler intrinsic methods? and the sig-poly is not a runtime sig-poly (as in the internal @MethodHandle.PolymorphicSignature), just the compile time behaviour bit. There is definitely some scope to improving the ease of use for those that want to refer explicit to say Class rather than ClassConstant. Might be worth a quick mention without getting the paint and brushes out. I suspect it in some cases a CE could be extended to that returned from a Intrinics.lcd call (if javac can track it within the compilation unit), although that probably has less value. I would be inclined to make a stronger statement about a new language syntax (as in your reply to Remi), if we ain?t gonna do it, let?s say so, and if someone else comes up with a brilliant idea we can pivot. Also i think it?s worth stating in the summary that the focus of this library is as you say for the advanced developer without being an attractive nuisance to the majority of Java developers. Paul. > On 11 Apr 2017, at 06:57, Brian Goetz wrote: > > Please review the this JEP draft: > > https://bugs.openjdk.java.net/browse/JDK-8178320 > > This JEP creates (a) an official set of purely-nominal wrapper classes for linkable constants (Class, MethodType, MethodHandle, VarHandle), provides constant-propagation (not folding) support in javac for these types, and (b) intrinsification for LDC and INVOKEDYNAMIC so that they can be predictably translated into bytecode. > > This allows Java source code to be a first-class client of invokedynamic, including using Java to test indy bootstraps. > From brian.goetz at oracle.com Tue Apr 11 18:57:18 2017 From: brian.goetz at oracle.com (Brian Goetz) Date: Tue, 11 Apr 2017 14:57:18 -0400 Subject: New JEP: Javac intrinsics for LDC and INVOKEDYNAMIC In-Reply-To: References: Message-ID: > Nor any class initialization (thereby in some cases, at least in JDK allowing us the possibility circumvent bootstrap issues, that?s my hope...). Indeed. We hope this will take some pressure off of bootstrapping. > Perhaps mention that it will not be possible to reflectively invoke (via MethodHandles or Method) the compiler intrinsic methods? and the sig-poly is not a runtime sig-poly (as in the internal @MethodHandle.PolymorphicSignature), just the compile time behaviour bit. Yes. Added to JEP. > There is definitely some scope to improving the ease of use for those that want to refer explicit to say Class rather than ClassConstant. Might be worth a quick mention without getting the paint and brushes out. I suspect it in some cases a CE could be extended to that returned from a Intrinics.lcd call (if javac can track it within the compilation unit), although that probably has less value. There's room to use target-typing on Class literals to allow them to both have a standalone type of Class but target-typing could allow them to represent a ClassConstant. This has the potential to streamline the use a lot, but we haven't worked out all the details. > I would be inclined to make a stronger statement about a new language syntax (as in your reply to Remi), if we ain?t gonna do it, let?s say so, and if someone else comes up with a brilliant idea we can pivot. Also i think it?s worth stating in the summary that the focus of this library is as you say for the advanced developer without being an attractive nuisance to the majority of Java developers. Added to non-goals. From brian.goetz at oracle.com Wed Apr 12 21:28:17 2017 From: brian.goetz at oracle.com (Brian Goetz) Date: Wed, 12 Apr 2017 17:28:17 -0400 Subject: Updated JEP: nestmates Message-ID: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> The "Nestmates" JEP has been updated: https://bugs.openjdk.java.net/browse/JDK-8046171 From forax at univ-mlv.fr Wed Apr 12 22:00:42 2017 From: forax at univ-mlv.fr (Remi Forax) Date: Thu, 13 Apr 2017 00:00:42 +0200 (CEST) Subject: Updated JEP: nestmates In-Reply-To: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> Message-ID: <1709982294.1362541.1492034442839.JavaMail.zimbra@u-pem.fr> Open Issue; "The method MethodHandles.Lookup.in exists primarily to provide reflective access to nestmate names. This method could perhaps be deprecated or more strongly restricted if the JVM provided systematic access to nestmate names." This JEP is not about the JVM providing a way to add a new member to the nest, so if a language (stupidly you may say) allow to dynamically add a new member to a nest, the runtime of that language still needs lookup.in(). R?mi ----- Mail original ----- > De: "Brian Goetz" > ?: valhalla-dev at openjdk.java.net > Envoy?: Mercredi 12 Avril 2017 23:28:17 > Objet: Updated JEP: nestmates > The "Nestmates" JEP has been updated: > > https://bugs.openjdk.java.net/browse/JDK-8046171 From john.r.rose at oracle.com Wed Apr 12 23:16:35 2017 From: john.r.rose at oracle.com (John Rose) Date: Wed, 12 Apr 2017 16:16:35 -0700 Subject: Updated JEP: nestmates In-Reply-To: <1709982294.1362541.1492034442839.JavaMail.zimbra@u-pem.fr> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <1709982294.1362541.1492034442839.JavaMail.zimbra@u-pem.fr> Message-ID: On Apr 12, 2017, at 3:00 PM, Remi Forax wrote: > > This JEP is not about the JVM providing a way to add a new member to the nest, > so if a language (stupidly you may say) allow to dynamically add a new member to a nest, the runtime of that language still needs lookup.in(). Good point, thanks. I removed that bullet item. (Also tidied a few other bits.) ? John From david.holmes at oracle.com Thu Apr 13 00:36:59 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 13 Apr 2017 10:36:59 +1000 Subject: Updated JEP: nestmates In-Reply-To: References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <1709982294.1362541.1492034442839.JavaMail.zimbra@u-pem.fr> Message-ID: On 13/04/2017 9:16 AM, John Rose wrote: > On Apr 12, 2017, at 3:00 PM, Remi Forax wrote: >> >> This JEP is not about the JVM providing a way to add a new member to the nest, >> so if a language (stupidly you may say) allow to dynamically add a new member to a nest, the runtime of that language still needs lookup.in(). > > Good point, thanks. I removed that bullet item. (Also tidied a few other bits.) ? John If lookup() now provides access to nest members (which it does) then I would expect it to provide access to the newly added member as well. David From forax at univ-mlv.fr Thu Apr 13 10:37:31 2017 From: forax at univ-mlv.fr (Remi Forax) Date: Thu, 13 Apr 2017 12:37:31 +0200 (CEST) Subject: Updated JEP: nestmates In-Reply-To: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> Message-ID: <679559779.1604218.1492079851801.JavaMail.zimbra@u-pem.fr> Re-reading the spec again, i think it would be clearer if you introduce T the nest-top, and modifying the classloading order has to be listed in the "Risks and Assumptions" section. Description: "The Java compiler compiles a group of nested types into a corresponding group of class files; it uses the InnerClasses and EnclosingMethod attributes to reify the nesting relationship (JVMS 4.7.6 and 4.7.7)." We can not re-use these attributes to describe nesting relationship because being part of a nest has to be verified, triggering classloading of the nest-top, thus changing the semantics of the existing attributes. ... "For types C and D to be nestmates they must either have the same nest-top T. A type C claims to be a member of the nest of D, if both C and D list T in their MemberOfNest attribute. The membership is validated during class loading if T also lists C and D in its NestMembers attribute. This may require loading of nest-top types earlier than might otherwise occur." A type classfile has zero or one MemberOfNest attribute and zero or one NestMembers. So a type can only be part of zero or one nest. Risks and Assumptions This proposal modifies the classloading order because a nest-top of a type needs to be loaded either eagerly when a member of a nest is loaded or more lazily when a member of a nest try to access to another member of a nest. This is not a problem for an inner-class because it also has a reference to the enclosing class, so the nest-top class is already loaded, for non inner class (static enclosed class), it will depend on the java compiler implementation, i.e. if the compiler eagerly add nestmates that mirror the InnerClasses attribute even if the enclosed class/interface do not access to a nestmate member or if the compiler add nestmates to the nest only when necessary (like currently it only generates bridges when necessary). cheers, R?mi ----- Mail original ----- > De: "Brian Goetz" > ?: valhalla-dev at openjdk.java.net > Envoy?: Mercredi 12 Avril 2017 23:28:17 > Objet: Updated JEP: nestmates > The "Nestmates" JEP has been updated: > > https://bugs.openjdk.java.net/browse/JDK-8046171 From forax at univ-mlv.fr Thu Apr 13 10:51:00 2017 From: forax at univ-mlv.fr (forax at univ-mlv.fr) Date: Thu, 13 Apr 2017 12:51:00 +0200 (CEST) Subject: Updated JEP: nestmates In-Reply-To: References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <1709982294.1362541.1492034442839.JavaMail.zimbra@u-pem.fr> Message-ID: <1972301568.1606896.1492080660322.JavaMail.zimbra@u-pem.fr> ----- Mail original ----- > De: "David Holmes" > ?: "John Rose" , "R?mi Forax" > Cc: valhalla-dev at openjdk.java.net > Envoy?: Jeudi 13 Avril 2017 02:36:59 > Objet: Re: Updated JEP: nestmates > On 13/04/2017 9:16 AM, John Rose wrote: >> On Apr 12, 2017, at 3:00 PM, Remi Forax wrote: >>> >>> This JEP is not about the JVM providing a way to add a new member to the nest, >>> so if a language (stupidly you may say) allow to dynamically add a new member to >>> a nest, the runtime of that language still needs lookup.in(). >> >> Good point, thanks. I removed that bullet item. (Also tidied a few other >> bits.) ? John > > If lookup() now provides access to nest members (which it does) then I > would expect it to provide access to the newly added member as well. Lookup will be upgraded to provides 'nest access' to members defined as nestmates. But as specified by this JEP, the nestmate relationship has to be set up before classloading time: - once a nest-top is loaded you can not add another nest member, - once a type is loaded, it can not be part of a nest afterward. So nestmates is not a dynamic relationship. If i want to have dynamic nests at runtime, i can not use nestmates, but i can *simulate* them, by creating a Lookup on a nest-top (or any other nest members) and using Lookup.in() to see it as a Lookup of a nest member. So Lookup.in() do not provide exactly the same feature as nestmates, thus it should not be deprecated/demoted. > > David R?mi From forax at univ-mlv.fr Thu Apr 13 10:54:27 2017 From: forax at univ-mlv.fr (Remi Forax) Date: Thu, 13 Apr 2017 12:54:27 +0200 (CEST) Subject: Updated JEP: nestmates In-Reply-To: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> Message-ID: <1076292152.1607809.1492080867818.JavaMail.zimbra@u-pem.fr> Answering to David make me think about class transformation at runtime, this JEP should say that you can not modify nest definition attributes at runtime using java.lang.instrument. R?mi ----- Mail original ----- > De: "Brian Goetz" > ?: valhalla-dev at openjdk.java.net > Envoy?: Mercredi 12 Avril 2017 23:28:17 > Objet: Updated JEP: nestmates > The "Nestmates" JEP has been updated: > > https://bugs.openjdk.java.net/browse/JDK-8046171 From david.holmes at oracle.com Thu Apr 13 11:33:56 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 13 Apr 2017 21:33:56 +1000 Subject: Updated JEP: nestmates In-Reply-To: <1972301568.1606896.1492080660322.JavaMail.zimbra@u-pem.fr> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <1709982294.1362541.1492034442839.JavaMail.zimbra@u-pem.fr> <1972301568.1606896.1492080660322.JavaMail.zimbra@u-pem.fr> Message-ID: <6fe49c17-829f-70e6-ba95-7f7a24c79d36@oracle.com> On 13/04/2017 8:51 PM, forax at univ-mlv.fr wrote: > ----- Mail original ----- >> De: "David Holmes" >> ?: "John Rose" , "R?mi Forax" >> Cc: valhalla-dev at openjdk.java.net >> Envoy?: Jeudi 13 Avril 2017 02:36:59 >> Objet: Re: Updated JEP: nestmates > >> On 13/04/2017 9:16 AM, John Rose wrote: >>> On Apr 12, 2017, at 3:00 PM, Remi Forax wrote: >>>> >>>> This JEP is not about the JVM providing a way to add a new member to the nest, >>>> so if a language (stupidly you may say) allow to dynamically add a new member to >>>> a nest, the runtime of that language still needs lookup.in(). >>> >>> Good point, thanks. I removed that bullet item. (Also tidied a few other >>> bits.) ? John >> >> If lookup() now provides access to nest members (which it does) then I >> would expect it to provide access to the newly added member as well. > > Lookup will be upgraded to provides 'nest access' to members defined as nestmates. > But as specified by this JEP, the nestmate relationship has to be set up before classloading time: > - once a nest-top is loaded you can not add another nest member, > - once a type is loaded, it can not be part of a nest afterward. > > So nestmates is not a dynamic relationship. > > If i want to have dynamic nests at runtime, i can not use nestmates, > but i can *simulate* them, by creating a Lookup on a nest-top (or any other nest members) and using Lookup.in() to see it as a Lookup of a nest member. > So Lookup.in() do not provide exactly the same feature as nestmates, thus it should not be deprecated/demoted. Sorry there are too many missing pieces here for me. What does a "simulated" nest-mate even mean? What did you introduce, where and how? And what access are you expecting to have to what, and how is that access being allowed ??? David > > R?mi > From david.holmes at oracle.com Thu Apr 13 11:36:22 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 13 Apr 2017 21:36:22 +1000 Subject: Updated JEP: nestmates In-Reply-To: <1076292152.1607809.1492080867818.JavaMail.zimbra@u-pem.fr> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <1076292152.1607809.1492080867818.JavaMail.zimbra@u-pem.fr> Message-ID: <2030928f-4f84-fd14-f3d9-2f18efe00605@oracle.com> On 13/04/2017 8:54 PM, Remi Forax wrote: > Answering to David make me think about class transformation at runtime, > this JEP should say that you can not modify nest definition attributes at runtime using java.lang.instrument. Yes. David ----- > R?mi > > ----- Mail original ----- >> De: "Brian Goetz" >> ?: valhalla-dev at openjdk.java.net >> Envoy?: Mercredi 12 Avril 2017 23:28:17 >> Objet: Updated JEP: nestmates > >> The "Nestmates" JEP has been updated: >> >> https://bugs.openjdk.java.net/browse/JDK-8046171 From david.holmes at oracle.com Thu Apr 13 11:59:17 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 13 Apr 2017 21:59:17 +1000 Subject: Updated JEP: nestmates In-Reply-To: <679559779.1604218.1492079851801.JavaMail.zimbra@u-pem.fr> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <679559779.1604218.1492079851801.JavaMail.zimbra@u-pem.fr> Message-ID: On 13/04/2017 8:37 PM, Remi Forax wrote: > Re-reading the spec again, > i think it would be clearer if you introduce T the nest-top, and modifying the classloading order has to be listed in the "Risks and Assumptions" section. > > Description: > "The Java compiler compiles a group of nested types into a corresponding group of class files; it uses the InnerClasses and EnclosingMethod attributes to reify the nesting relationship (JVMS 4.7.6 and 4.7.7)." > We can not re-use these attributes to describe nesting relationship because being part of a nest has to be verified, triggering classloading of the nest-top, thus changing the semantics of the existing attributes. > ... > > "For types C and D to be nestmates they must either have the same nest-top T. A type C claims to be a member of the nest of D, if both C and D list T in their MemberOfNest attribute. The membership is validated during class loading if T also lists C and D in its NestMembers attribute. This may require loading of nest-top types earlier than might otherwise occur." That paragraph in the JEP got modified slightly from my original. There was no intent at that point to introduce a third type. My original is: "For types C and D to be nestmates they must either have the same nest-top, or else one must be the nest-top of the other. A type C claims to be a member of the nest of D, if it lists D in it's `MemberOfNest` attribute. The membership is validated during class loading if D also lists C in its `NestMembers` attribute. This may require loading of nest-top types earlier than might otherwise occur." Also note that "loading" is being used to allow some flexibility. What I currently do is validate during class linking, prior to verification (if enabled). But specifying it to happen at that exact point may be too constraining. > A type classfile has zero or one MemberOfNest attribute and zero or one NestMembers. So a type can only be part of zero or one nest. But it can not have both a MemberOfNest and a NestMembers attribute. > Risks and Assumptions > This proposal modifies the classloading order because a nest-top of a type needs to be loaded either eagerly when a member of a nest is loaded or more lazily when a member of a nest try to access to another member of a nest. The interesting case is where the nest-top class never actually gets used in the execution of the application, and so presently is not loaded. I've encountered several tests that fail because they are now forced to load the nest-top class but can't locate it due to the way the test has arranged classes on boot/classpath or via classloaders. In those cases it makes no difference how eagerly or lazily you load the nest-top class. > This is not a problem for an inner-class because it also has a reference to the enclosing class, so the nest-top class is already loaded, for non inner class (static enclosed class), it will depend on the java compiler implementation, i.e. if the compiler eagerly add nestmates that mirror the InnerClasses attribute even if the enclosed class/interface do not access to a nestmate member or if the compiler add nestmates to the nest only when necessary (like currently it only generates bridges when necessary). Yes a compiler could elide the nest attributes if being nestmates is not relied upon by the code. But to the VM they would not be nestmates so any dynamic lookup that required nestmate access would fail. Cheers, David > cheers, > R?mi > > ----- Mail original ----- >> De: "Brian Goetz" >> ?: valhalla-dev at openjdk.java.net >> Envoy?: Mercredi 12 Avril 2017 23:28:17 >> Objet: Updated JEP: nestmates > >> The "Nestmates" JEP has been updated: >> >> https://bugs.openjdk.java.net/browse/JDK-8046171 From forax at univ-mlv.fr Thu Apr 13 16:25:20 2017 From: forax at univ-mlv.fr (forax at univ-mlv.fr) Date: Thu, 13 Apr 2017 18:25:20 +0200 (CEST) Subject: Updated JEP: nestmates In-Reply-To: References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <679559779.1604218.1492079851801.JavaMail.zimbra@u-pem.fr> Message-ID: <1316798436.1782207.1492100720790.JavaMail.zimbra@u-pem.fr> ----- Mail original ----- > De: "David Holmes" > ?: "Remi Forax" , "Brian Goetz" , "John Rose" > Cc: valhalla-dev at openjdk.java.net > Envoy?: Jeudi 13 Avril 2017 13:59:17 > Objet: Re: Updated JEP: nestmates > On 13/04/2017 8:37 PM, Remi Forax wrote: >> Re-reading the spec again, >> i think it would be clearer if you introduce T the nest-top, and modifying the >> classloading order has to be listed in the "Risks and Assumptions" section. >> >> Description: >> "The Java compiler compiles a group of nested types into a corresponding group >> of class files; it uses the InnerClasses and EnclosingMethod attributes to >> reify the nesting relationship (JVMS 4.7.6 and 4.7.7)." >> We can not re-use these attributes to describe nesting relationship because >> being part of a nest has to be verified, triggering classloading of the >> nest-top, thus changing the semantics of the existing attributes. >> ... >> >> "For types C and D to be nestmates they must either have the same nest-top T. A >> type C claims to be a member of the nest of D, if both C and D list T in their >> MemberOfNest attribute. The membership is validated during class loading if T >> also lists C and D in its NestMembers attribute. This may require loading of >> nest-top types earlier than might otherwise occur." > > That paragraph in the JEP got modified slightly from my original. There > was no intent at that point to introduce a third type. My original is: > > "For types C and D to be nestmates they must either have > the same nest-top, or else one must be the nest-top of the other. ok, agree with this one, even if the second part of this sentence is not necessary. > A type C claims to be a member of the nest of D, if it > lists D in it's `MemberOfNest` attribute. while i understand this sentence, i'm not sure everybody will be able to parse it, apart from the 'it's' -> its, "member of the nest of D" is ambiguous, at least for me, it can be read as "member of the same nest as D" or as "member of the nest named D". Perhaps removing the "of" is enough. A type C claims to be a member of the nest D, if it lists D in it's `MemberOfNest` attribute. > The membership is validated > during class loading if D also lists C in its > `NestMembers` attribute. This may require loading of nest-top types > earlier than might otherwise occur." > > Also note that "loading" is being used to allow some flexibility. What I > currently do is validate during class linking, prior to verification (if > enabled). But specifying it to happen at that exact point may be too > constraining. yes agree about what "loading" means here, in term of implementation, i think it should be validated the first time you need the nestmate relationship, it will be less disruptive (see below) > >> A type classfile has zero or one MemberOfNest attribute and zero or one >> NestMembers. So a type can only be part of zero or one nest. > > But it can not have both a MemberOfNest and a NestMembers attribute. The fact that you can not have a MemberOfNest attribute if you have a NestMembers attribute is an optimization, if you have a NestMembers attribute, you are a nest-top thus you do not need a MemberOfNest on yourself. It's mandatory by the spec, but i think it should also be explained. > >> Risks and Assumptions >> This proposal modifies the classloading order because a nest-top of a type needs >> to be loaded either eagerly when a member of a nest is loaded or more lazily >> when a member of a nest try to access to another member of a nest. > > The interesting case is where the nest-top class never actually gets > used in the execution of the application, and so presently is not > loaded. I've encountered several tests that fail because they are now > forced to load the nest-top class but can't locate it due to the way the > test has arranged classes on boot/classpath or via classloaders. In > those cases it makes no difference how eagerly or lazily you load the > nest-top class. Yes and No. Currently in Java 9 a code that's work do not requires any access check that requires to check if a class is a nest mate of another class. So if lazy means validating the nest-top/nest-members relationship only the first time you need to check a nest-mate access, every Java 9 applications should not load nest-top classes. (obviously, here i do not consider Java 9 tests that check that an access check exception is raised, those will load more classes). > >> This is not a problem for an inner-class because it also has a reference to the >> enclosing class, so the nest-top class is already loaded, for non inner class >> (static enclosed class), it will depend on the java compiler implementation, >> i.e. if the compiler eagerly add nestmates that mirror the InnerClasses >> attribute even if the enclosed class/interface do not access to a nestmate >> member or if the compiler add nestmates to the nest only when necessary (like >> currently it only generates bridges when necessary). > > Yes a compiler could elide the nest attributes if being nestmates is not > relied upon by the code. But to the VM they would not be nestmates so > any dynamic lookup that required nestmate access would fail. yes, i've forgotten the reflection. So the Java compiler can not be smart, so the VM has to be smart. The validation of the nest-top/nest members relationship has to be as late as possible to avoid loading classes that will break "compatibility". > > Cheers, > David R?mi > >> cheers, >> R?mi >> >> ----- Mail original ----- >>> De: "Brian Goetz" >>> ?: valhalla-dev at openjdk.java.net >>> Envoy?: Mercredi 12 Avril 2017 23:28:17 >>> Objet: Updated JEP: nestmates >> >>> The "Nestmates" JEP has been updated: >>> > >> https://bugs.openjdk.java.net/browse/JDK-8046171 From forax at univ-mlv.fr Thu Apr 13 16:41:11 2017 From: forax at univ-mlv.fr (forax at univ-mlv.fr) Date: Thu, 13 Apr 2017 18:41:11 +0200 (CEST) Subject: Updated JEP: nestmates In-Reply-To: <6fe49c17-829f-70e6-ba95-7f7a24c79d36@oracle.com> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <1709982294.1362541.1492034442839.JavaMail.zimbra@u-pem.fr> <1972301568.1606896.1492080660322.JavaMail.zimbra@u-pem.fr> <6fe49c17-829f-70e6-ba95-7f7a24c79d36@oracle.com> Message-ID: <1354135365.1784542.1492101671883.JavaMail.zimbra@u-pem.fr> ----- Mail original ----- > De: "David Holmes" > ?: forax at univ-mlv.fr > Cc: "John Rose" , valhalla-dev at openjdk.java.net > Envoy?: Jeudi 13 Avril 2017 13:33:56 > Objet: Re: Updated JEP: nestmates > On 13/04/2017 8:51 PM, forax at univ-mlv.fr wrote: >> ----- Mail original ----- >>> De: "David Holmes" >>> ?: "John Rose" , "R?mi Forax" >>> Cc: valhalla-dev at openjdk.java.net >>> Envoy?: Jeudi 13 Avril 2017 02:36:59 >>> Objet: Re: Updated JEP: nestmates >> >>> On 13/04/2017 9:16 AM, John Rose wrote: >>>> On Apr 12, 2017, at 3:00 PM, Remi Forax wrote: >>>>> >>>>> This JEP is not about the JVM providing a way to add a new member to the nest, >>>>> so if a language (stupidly you may say) allow to dynamically add a new member to >>>>> a nest, the runtime of that language still needs lookup.in(). >>>> >>>> Good point, thanks. I removed that bullet item. (Also tidied a few other >>>> bits.) ? John >>> >>> If lookup() now provides access to nest members (which it does) then I >>> would expect it to provide access to the newly added member as well. >> >> Lookup will be upgraded to provides 'nest access' to members defined as >> nestmates. >> But as specified by this JEP, the nestmate relationship has to be set up before >> classloading time: >> - once a nest-top is loaded you can not add another nest member, >> - once a type is loaded, it can not be part of a nest afterward. >> >> So nestmates is not a dynamic relationship. >> >> If i want to have dynamic nests at runtime, i can not use nestmates, >> but i can *simulate* them, by creating a Lookup on a nest-top (or any other nest >> members) and using Lookup.in() to see it as a Lookup of a nest member. >> So Lookup.in() do not provide exactly the same feature as nestmates, thus it >> should not be deprecated/demoted. > > Sorry there are too many missing pieces here for me. What does a > "simulated" nest-mate even mean? What did you introduce, where and how? > And what access are you expecting to have to what, and how is that > access being allowed ??? Imagine a scripting language that compiles several classes as part of the same script, the semantics of the language allow code of the same script to see private members of the classes if there are in the same script. That scripting language can leverage the nestmate attributes for that. Now imagine that this scripting language allows to dynamically add some newly created classes as if they were part of an existing script, here you can not use the nestmate attributes for that because you can not re-open a nest-top at runtime to add newly created types as part of a NestMembers attribute. For that, you can use Lookup.in(). My first email in that thread was about saying to John that nestmate attributes are a compile-time/loading time mechanism so it can not replace all the use cases of Lookup.in(). > > David R?mi From john.r.rose at oracle.com Thu Apr 13 17:09:32 2017 From: john.r.rose at oracle.com (John Rose) Date: Thu, 13 Apr 2017 10:09:32 -0700 Subject: Updated JEP: nestmates In-Reply-To: <2030928f-4f84-fd14-f3d9-2f18efe00605@oracle.com> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <1076292152.1607809.1492080867818.JavaMail.zimbra@u-pem.fr> <2030928f-4f84-fd14-f3d9-2f18efe00605@oracle.com> Message-ID: <1E11AE50-D130-4618-B8E0-FEEDC3CBA5E2@oracle.com> On Apr 13, 2017, at 4:36 AM, David Holmes wrote: >> you can not modify nest definition attributes at runtime using java.lang.instrument. > > Yes. I'm missing something here. Why must nestmate attributes be specifically hardened against instrumentation changes? Can't instrumentation, in general, change any aspect of a classfile? From john.r.rose at oracle.com Thu Apr 13 17:25:50 2017 From: john.r.rose at oracle.com (John Rose) Date: Thu, 13 Apr 2017 10:25:50 -0700 Subject: Updated JEP: nestmates In-Reply-To: <1354135365.1784542.1492101671883.JavaMail.zimbra@u-pem.fr> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <1709982294.1362541.1492034442839.JavaMail.zimbra@u-pem.fr> <1972301568.1606896.1492080660322.JavaMail.zimbra@u-pem.fr> <6fe49c17-829f-70e6-ba95-7f7a24c79d36@oracle.com> <1354135365.1784542.1492101671883.JavaMail.zimbra@u-pem.fr> Message-ID: <14DE1477-1850-4BC3-8482-D96E94EE6437@oracle.com> On Apr 13, 2017, at 9:41 AM, forax at univ-mlv.fr wrote: > > My first email in that thread was about saying to John that nestmate attributes are a compile-time/loading time mechanism so it can not replace all the use cases of Lookup.in(). Once loaded, the nest is a "live" JVM entity, and as such could be modified by an appropriate API. In fact, we are reserving the private mode from Lookup.defineClass to mean exactly this: Load a dynamically defined class into a pre-existing nest. Even more, a class which is not part of a nest can become one (the top, I suppose), when Lookup.defineClass injects a nestmate into it. Lookup.in must track all this, and so will the native access logic of the JVM. In fact, for compatibility, it must also track InnerClasses attributes. (Yes, yuck.) (David, dynamic nest extension is a likely feature of Lookup.defineClass; it is not part of the base nestmate logic.) ? John From karen.kinnear at oracle.com Thu Apr 13 17:27:51 2017 From: karen.kinnear at oracle.com (Karen Kinnear) Date: Thu, 13 Apr 2017 13:27:51 -0400 Subject: Updated JEP: nestmates In-Reply-To: <1E11AE50-D130-4618-B8E0-FEEDC3CBA5E2@oracle.com> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <1076292152.1607809.1492080867818.JavaMail.zimbra@u-pem.fr> <2030928f-4f84-fd14-f3d9-2f18efe00605@oracle.com> <1E11AE50-D130-4618-B8E0-FEEDC3CBA5E2@oracle.com> Message-ID: No. Class and member access flags can NOT be changed by retransform/redefine classes. thanks, Karen > On Apr 13, 2017, at 1:09 PM, John Rose wrote: > > On Apr 13, 2017, at 4:36 AM, David Holmes wrote: > >>> you can not modify nest definition attributes at runtime using java.lang.instrument. >> >> Yes. > > I'm missing something here. Why must nestmate attributes be specifically hardened against instrumentation changes? Can't instrumentation, in general, change any aspect of a classfile? From forax at univ-mlv.fr Thu Apr 13 17:58:06 2017 From: forax at univ-mlv.fr (forax at univ-mlv.fr) Date: Thu, 13 Apr 2017 19:58:06 +0200 (CEST) Subject: Updated JEP: nestmates In-Reply-To: <14DE1477-1850-4BC3-8482-D96E94EE6437@oracle.com> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <1709982294.1362541.1492034442839.JavaMail.zimbra@u-pem.fr> <1972301568.1606896.1492080660322.JavaMail.zimbra@u-pem.fr> <6fe49c17-829f-70e6-ba95-7f7a24c79d36@oracle.com> <1354135365.1784542.1492101671883.JavaMail.zimbra@u-pem.fr> <14DE1477-1850-4BC3-8482-D96E94EE6437@oracle.com> Message-ID: <2145723641.1794072.1492106286649.JavaMail.zimbra@u-pem.fr> > De: "John Rose" > ?: "R?mi Forax" > Cc: "David Holmes" , valhalla-dev at openjdk.java.net > Envoy?: Jeudi 13 Avril 2017 19:25:50 > Objet: Re: Updated JEP: nestmates > On Apr 13, 2017, at 9:41 AM, forax at univ-mlv.fr wrote: >> My first email in that thread was about saying to John that nestmate attributes >> are a compile-time/loading time mechanism so it can not replace all the use >> cases of Lookup.in(). > Once loaded, the nest is a "live" JVM entity, and as such could be modified by > an appropriate API. > In fact, we are reserving the private mode from Lookup.defineClass to mean > exactly this: Load a dynamically defined class into a pre-existing nest. i've forgotten that. > Even more, a class which is not part of a nest can become one (the top, I > suppose), when Lookup.defineClass injects a nestmate into it. so lookup.defineClass can ask the VM to create a new nest if none exists or add a member to an already existing next. > Lookup.in must track all this, and so will the native access logic of the JVM. Only adding things will work i suppose. Perhaps java.lang.instrument can be updated to allow the same kind of operations. > In fact, for compatibility, it must also track InnerClasses attributes. (Yes, > yuck.) I do not think it's a good idea. I suppose it depends if nestmates have their own reflect API or not. If you can access to nestmates using reflection, i suppose code that uses InnerClasses (Class.getClasses()) can be updated to take care about nestmates. > (David, dynamic nest extension is a likely feature of Lookup.defineClass; it is > not part of the base nestmate logic.) > ? John R?mi From david.holmes at oracle.com Thu Apr 13 21:44:55 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 14 Apr 2017 07:44:55 +1000 Subject: Updated JEP: nestmates In-Reply-To: <1316798436.1782207.1492100720790.JavaMail.zimbra@u-pem.fr> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <679559779.1604218.1492079851801.JavaMail.zimbra@u-pem.fr> <1316798436.1782207.1492100720790.JavaMail.zimbra@u-pem.fr> Message-ID: <3ea33091-e200-4780-365f-7efaa55b14d3@oracle.com> On 14/04/2017 2:25 AM, forax at univ-mlv.fr wrote: > ----- Mail original ----- >> De: "David Holmes" >> ?: "Remi Forax" , "Brian Goetz" , "John Rose" >> Cc: valhalla-dev at openjdk.java.net >> Envoy?: Jeudi 13 Avril 2017 13:59:17 >> Objet: Re: Updated JEP: nestmates > >> On 13/04/2017 8:37 PM, Remi Forax wrote: >>> Re-reading the spec again, >>> i think it would be clearer if you introduce T the nest-top, and modifying the >>> classloading order has to be listed in the "Risks and Assumptions" section. >>> >>> Description: >>> "The Java compiler compiles a group of nested types into a corresponding group >>> of class files; it uses the InnerClasses and EnclosingMethod attributes to >>> reify the nesting relationship (JVMS 4.7.6 and 4.7.7)." >>> We can not re-use these attributes to describe nesting relationship because >>> being part of a nest has to be verified, triggering classloading of the >>> nest-top, thus changing the semantics of the existing attributes. >>> ... >>> >>> "For types C and D to be nestmates they must either have the same nest-top T. A >>> type C claims to be a member of the nest of D, if both C and D list T in their >>> MemberOfNest attribute. The membership is validated during class loading if T >>> also lists C and D in its NestMembers attribute. This may require loading of >>> nest-top types earlier than might otherwise occur." >> >> That paragraph in the JEP got modified slightly from my original. There >> was no intent at that point to introduce a third type. My original is: >> >> "For types C and D to be nestmates they must either have >> the same nest-top, or else one must be the nest-top of the other. > > ok, agree with this one, even if the second part of this sentence is not necessary. The second part most certainly is necessary as a nest-top does not itself have a nest-top! >> A type C claims to be a member of the nest of D, if it >> lists D in it's `MemberOfNest` attribute. > > while i understand this sentence, i'm not sure everybody will be able to parse it, > apart from the 'it's' -> its, "member of the nest of D" is ambiguous, at least for me, it can be read as "member of the same nest as D" or as "member of the nest named D". > Perhaps removing the "of" is enough. > A type C claims to be a member of the nest D, if it lists D in it's `MemberOfNest` attribute. D is not a nest! D is a type that may be in a nest. "nest of D" is perfectly correct. We could rephrase to "of the same nest as D". >> The membership is validated >> during class loading if D also lists C in its >> `NestMembers` attribute. This may require loading of nest-top types >> earlier than might otherwise occur." >> >> Also note that "loading" is being used to allow some flexibility. What I >> currently do is validate during class linking, prior to verification (if >> enabled). But specifying it to happen at that exact point may be too >> constraining. > > yes agree about what "loading" means here, > in term of implementation, i think it should be validated the first time you need the nestmate relationship, it will be less disruptive (see below) Note that may be during verification of another class. Any way the details here can be further explored. I don't think lazier than link-time buys you anything when you may need it earlier anyway. >> >>> A type classfile has zero or one MemberOfNest attribute and zero or one >>> NestMembers. So a type can only be part of zero or one nest. >> >> But it can not have both a MemberOfNest and a NestMembers attribute. > > The fact that you can not have a MemberOfNest attribute if you have a NestMembers attribute is an optimization, if you have a NestMembers attribute, you are a nest-top thus you do not need a MemberOfNest on yourself. > It's mandatory by the spec, but i think it should also be explained. It is not an optimization! You are either a nest-top (and so have a NestMembers attribute) or you are a nested type and so have a MemberOfNest attribute. (Of course you can be neither.) >> >>> Risks and Assumptions >>> This proposal modifies the classloading order because a nest-top of a type needs >>> to be loaded either eagerly when a member of a nest is loaded or more lazily >>> when a member of a nest try to access to another member of a nest. >> >> The interesting case is where the nest-top class never actually gets >> used in the execution of the application, and so presently is not >> loaded. I've encountered several tests that fail because they are now >> forced to load the nest-top class but can't locate it due to the way the >> test has arranged classes on boot/classpath or via classloaders. In >> those cases it makes no difference how eagerly or lazily you load the >> nest-top class. > > Yes and No. > Currently in Java 9 a code that's work do not requires any access check that requires to check if a class is a nest mate of another class. > So if lazy means validating the nest-top/nest-members relationship only the first time you need to check a nest-mate access, every Java 9 applications should not load nest-top classes. > (obviously, here i do not consider Java 9 tests that check that an access check exception is raised, those will load more classes). In Java 9 there is never a nestmate-access-check because of the accessors that are generated. So I'm not sure what you are trying to say. My point was that there are cases where the application logic never requires the nest-top to be loaded today, but with nest-mates it would be loaded. In that context it makes no difference whether the class is lazily or eagerly loaded, given it was previously never loaded at all. >> >>> This is not a problem for an inner-class because it also has a reference to the >>> enclosing class, so the nest-top class is already loaded, for non inner class >>> (static enclosed class), it will depend on the java compiler implementation, >>> i.e. if the compiler eagerly add nestmates that mirror the InnerClasses >>> attribute even if the enclosed class/interface do not access to a nestmate >>> member or if the compiler add nestmates to the nest only when necessary (like >>> currently it only generates bridges when necessary). >> >> Yes a compiler could elide the nest attributes if being nestmates is not >> relied upon by the code. But to the VM they would not be nestmates so >> any dynamic lookup that required nestmate access would fail. > > yes, i've forgotten the reflection. > So the Java compiler can not be smart, so the VM has to be smart. > The validation of the nest-top/nest members relationship has to be as late as possible to avoid loading classes that will break "compatibility". Not sure what you mean here with "compatability". Cheers, David >> >> Cheers, >> David > > R?mi > >> >>> cheers, >>> R?mi >>> >>> ----- Mail original ----- >>>> De: "Brian Goetz" >>>> ?: valhalla-dev at openjdk.java.net >>>> Envoy?: Mercredi 12 Avril 2017 23:28:17 >>>> Objet: Updated JEP: nestmates >>> >>>> The "Nestmates" JEP has been updated: >>>> >>>> https://bugs.openjdk.java.net/browse/JDK-8046171 From brian.goetz at oracle.com Thu Apr 13 21:50:21 2017 From: brian.goetz at oracle.com (Brian Goetz) Date: Thu, 13 Apr 2017 17:50:21 -0400 Subject: Updated JEP: nestmates In-Reply-To: <3ea33091-e200-4780-365f-7efaa55b14d3@oracle.com> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <679559779.1604218.1492079851801.JavaMail.zimbra@u-pem.fr> <1316798436.1782207.1492100720790.JavaMail.zimbra@u-pem.fr> <3ea33091-e200-4780-365f-7efaa55b14d3@oracle.com> Message-ID: <7b5f9bb1-45e1-0c31-e0f4-c2f185f1af7e@oracle.com> > The second part most certainly is necessary as a nest-top does not > itself have a nest-top! I would think we'd consider a nest-top as its own nest-top. That way, every nest has a top, which acts as a canonical element for that nest. > It is not an optimization! You are either a nest-top (and so have a > NestMembers attribute) or you are a nested type and so have a > MemberOfNest attribute. (Of course you can be neither.) Similarly, I'd think that every class that has neither a NestMembers or MemberOfNest attribute constitutes a nest of one. Taken together, now: - Every class belongs to exactly one nest; - Every nest has a canonical, consistent nest-top (every member of the nest agrees on the nest top) This will become relevant when we extend this to support Lookup.defineClass with a private lookup, which has the effect of injecting a new class into an existing nest. If every class is already in a nest, this injection is simplified. From david.holmes at oracle.com Thu Apr 13 21:51:28 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 14 Apr 2017 07:51:28 +1000 Subject: Updated JEP: nestmates In-Reply-To: <1E11AE50-D130-4618-B8E0-FEEDC3CBA5E2@oracle.com> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <1076292152.1607809.1492080867818.JavaMail.zimbra@u-pem.fr> <2030928f-4f84-fd14-f3d9-2f18efe00605@oracle.com> <1E11AE50-D130-4618-B8E0-FEEDC3CBA5E2@oracle.com> Message-ID: On 14/04/2017 3:09 AM, John Rose wrote: > On Apr 13, 2017, at 4:36 AM, David Holmes wrote: > >>> you can not modify nest definition attributes at runtime using java.lang.instrument. >> >> Yes. > > I'm missing something here. Why must nestmate attributes be specifically hardened against instrumentation changes? Can't instrumentation, in general, change any aspect of a classfile? Instrumentation is primarily about adding bytecodes to methods. There are strict constraints on what instrumentation can change with respect to the "shape" of a type (members, inheritance hierarchy). "The retransformation may change method bodies, the constant pool and attributes. The retransformation must not add, remove or rename fields or methods, change the signatures of methods, change modifiers, or change inheritance.". [ JVM TI spec.] Similarly for redefinition, which also adds " or change inheritance". So no changes to super-types. And now no changes to nest membership. David From david.holmes at oracle.com Thu Apr 13 21:57:59 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 14 Apr 2017 07:57:59 +1000 Subject: Updated JEP: nestmates In-Reply-To: <7b5f9bb1-45e1-0c31-e0f4-c2f185f1af7e@oracle.com> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <679559779.1604218.1492079851801.JavaMail.zimbra@u-pem.fr> <1316798436.1782207.1492100720790.JavaMail.zimbra@u-pem.fr> <3ea33091-e200-4780-365f-7efaa55b14d3@oracle.com> <7b5f9bb1-45e1-0c31-e0f4-c2f185f1af7e@oracle.com> Message-ID: <7dee140d-6a49-a7de-0c2e-dfcfbd0d3962@oracle.com> On 14/04/2017 7:50 AM, Brian Goetz wrote: >> The second part most certainly is necessary as a nest-top does not >> itself have a nest-top! > > I would think we'd consider a nest-top as its own nest-top. That way, > every nest has a top, which acts as a canonical element for that nest. From an implementation perspective certainly that simplifies nest-membership checks. but do you want that in the spec? Do you want javac to generate a self-referential MemberOfNest attribute? If you want this then the definitions need a rewrite. This is not where we are today. >> It is not an optimization! You are either a nest-top (and so have a >> NestMembers attribute) or you are a nested type and so have a >> MemberOfNest attribute. (Of course you can be neither.) > > Similarly, I'd think that every class that has neither a NestMembers or > MemberOfNest attribute constitutes a nest of one. Sure logically you can consider it that way. Not sure it has any real affect on anything. > Taken together, now: > - Every class belongs to exactly one nest; > - Every nest has a canonical, consistent nest-top (every member of the > nest agrees on the nest top) Yes. > This will become relevant when we extend this to support > Lookup.defineClass with a private lookup, which has the effect of > injecting a new class into an existing nest. If every class is already > in a nest, this injection is simplified. I don't really see how it makes a difference, but you presumably have something in mind here. Thanks, David From forax at univ-mlv.fr Thu Apr 13 22:05:48 2017 From: forax at univ-mlv.fr (forax at univ-mlv.fr) Date: Fri, 14 Apr 2017 00:05:48 +0200 (CEST) Subject: Updated JEP: nestmates In-Reply-To: <7dee140d-6a49-a7de-0c2e-dfcfbd0d3962@oracle.com> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <679559779.1604218.1492079851801.JavaMail.zimbra@u-pem.fr> <1316798436.1782207.1492100720790.JavaMail.zimbra@u-pem.fr> <3ea33091-e200-4780-365f-7efaa55b14d3@oracle.com> <7b5f9bb1-45e1-0c31-e0f4-c2f185f1af7e@oracle.com> <7dee140d-6a49-a7de-0c2e-dfcfbd0d3962@oracle.com> Message-ID: <1902427860.1821668.1492121148906.JavaMail.zimbra@u-pem.fr> It seems we are all on the same page but some of us talk about the spec and other about the implementation. >From the spec point of view, i agree with Brian, "Every class belongs to exactly one nest", - if there is a MemberOfNest attribute, it defines the nest, - if there is a NestMembers attribute, it defines the nest-top, - if there is no attribute, it defines its own nest. and "Every nest has a canonical, consistent nest-top". R?mi ----- Mail original ----- > De: "David Holmes" > ?: "Brian Goetz" , forax at univ-mlv.fr > Cc: "John Rose" , valhalla-dev at openjdk.java.net > Envoy?: Jeudi 13 Avril 2017 23:57:59 > Objet: Re: Updated JEP: nestmates > On 14/04/2017 7:50 AM, Brian Goetz wrote: >>> The second part most certainly is necessary as a nest-top does not >>> itself have a nest-top! >> >> I would think we'd consider a nest-top as its own nest-top. That way, >> every nest has a top, which acts as a canonical element for that nest. > > From an implementation perspective certainly that simplifies > nest-membership checks. but do you want that in the spec? Do you want > javac to generate a self-referential MemberOfNest attribute? If you want > this then the definitions need a rewrite. This is not where we are today. > >>> It is not an optimization! You are either a nest-top (and so have a >>> NestMembers attribute) or you are a nested type and so have a >>> MemberOfNest attribute. (Of course you can be neither.) >> >> Similarly, I'd think that every class that has neither a NestMembers or >> MemberOfNest attribute constitutes a nest of one. > > Sure logically you can consider it that way. Not sure it has any real > affect on anything. > >> Taken together, now: >> - Every class belongs to exactly one nest; >> - Every nest has a canonical, consistent nest-top (every member of the >> nest agrees on the nest top) > > Yes. > >> This will become relevant when we extend this to support >> Lookup.defineClass with a private lookup, which has the effect of >> injecting a new class into an existing nest. If every class is already >> in a nest, this injection is simplified. > > I don't really see how it makes a difference, but you presumably have > something in mind here. > > Thanks, > David From forax at univ-mlv.fr Thu Apr 13 22:10:54 2017 From: forax at univ-mlv.fr (forax at univ-mlv.fr) Date: Fri, 14 Apr 2017 00:10:54 +0200 (CEST) Subject: Updated JEP: nestmates In-Reply-To: <3ea33091-e200-4780-365f-7efaa55b14d3@oracle.com> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <679559779.1604218.1492079851801.JavaMail.zimbra@u-pem.fr> <1316798436.1782207.1492100720790.JavaMail.zimbra@u-pem.fr> <3ea33091-e200-4780-365f-7efaa55b14d3@oracle.com> Message-ID: <2125517239.1824274.1492121454846.JavaMail.zimbra@u-pem.fr> ----- Mail original ----- > De: "David Holmes" > ?: forax at univ-mlv.fr > Cc: "Brian Goetz" , "John Rose" , valhalla-dev at openjdk.java.net > Envoy?: Jeudi 13 Avril 2017 23:44:55 > Objet: Re: Updated JEP: nestmates >> >>> On 13/04/2017 8:37 PM, Remi Forax wrote: >>>> Re-reading the spec again, >>>> i think it would be clearer if you introduce T the nest-top, and modifying the >>>> classloading order has to be listed in the "Risks and Assumptions" section. >>>> >>>> Description: >>>> "The Java compiler compiles a group of nested types into a corresponding group >>>> of class files; it uses the InnerClasses and EnclosingMethod attributes to >>>> reify the nesting relationship (JVMS 4.7.6 and 4.7.7)." >>>> We can not re-use these attributes to describe nesting relationship because >>>> being part of a nest has to be verified, triggering classloading of the >>>> nest-top, thus changing the semantics of the existing attributes. >>>> ... >>>> >>>> "For types C and D to be nestmates they must either have the same nest-top T. A >>>> type C claims to be a member of the nest of D, if both C and D list T in their >>>> MemberOfNest attribute. The membership is validated during class loading if T >>>> also lists C and D in its NestMembers attribute. This may require loading of >>>> nest-top types earlier than might otherwise occur." >>> >>> That paragraph in the JEP got modified slightly from my original. There >>> was no intent at that point to introduce a third type. My original is: >>> >>> "For types C and D to be nestmates they must either have >>> the same nest-top, or else one must be the nest-top of the other. >> >> ok, agree with this one, even if the second part of this sentence is not >> necessary. > > The second part most certainly is necessary as a nest-top does not > itself have a nest-top! Conceptually a nest-top is its own nest-top, but in term of attribute, so yes, you're right. > >>> A type C claims to be a member of the nest of D, if it >>> lists D in it's `MemberOfNest` attribute. >> >> while i understand this sentence, i'm not sure everybody will be able to parse >> it, >> apart from the 'it's' -> its, "member of the nest of D" is ambiguous, at least >> for me, it can be read as "member of the same nest as D" or as "member of the >> nest named D". >> Perhaps removing the "of" is enough. >> A type C claims to be a member of the nest D, if it lists D >> in it's `MemberOfNest` attribute. > > D is not a nest! D is a type that may be in a nest. "nest of D" is > perfectly correct. We could rephrase to "of the same nest as D". if D is not the nest, then MemberOfNest attribute of C is not D but the nest-top of D. R?mi From brian.goetz at oracle.com Thu Apr 13 22:12:34 2017 From: brian.goetz at oracle.com (Brian Goetz) Date: Thu, 13 Apr 2017 18:12:34 -0400 Subject: Updated JEP: nestmates In-Reply-To: <7dee140d-6a49-a7de-0c2e-dfcfbd0d3962@oracle.com> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <679559779.1604218.1492079851801.JavaMail.zimbra@u-pem.fr> <1316798436.1782207.1492100720790.JavaMail.zimbra@u-pem.fr> <3ea33091-e200-4780-365f-7efaa55b14d3@oracle.com> <7b5f9bb1-45e1-0c31-e0f4-c2f185f1af7e@oracle.com> <7dee140d-6a49-a7de-0c2e-dfcfbd0d3962@oracle.com> Message-ID: <6824725d-962e-0ac9-49ed-44742ece1449@oracle.com> I was trying to make the model simpler to reason about. My mathematical model is that nests form a partition over classes, meaning: - every class belongs to a nest - no two nests have any members in common - every nest has a canonical member We can't rely on javac (or other compilers) to always generate "nest of one" attributes, so the VM would have to, on seeing a class with no nest attributes, set "nestTop = me". But I think that would be the only change in the implementation? And then nestTop(C) becomes a total function, which simplifies things. I'm fine if the spec says this; in fact, I like it. On 4/13/2017 5:57 PM, David Holmes wrote: > On 14/04/2017 7:50 AM, Brian Goetz wrote: >>> The second part most certainly is necessary as a nest-top does not >>> itself have a nest-top! >> >> I would think we'd consider a nest-top as its own nest-top. That way, >> every nest has a top, which acts as a canonical element for that nest. > > From an implementation perspective certainly that simplifies > nest-membership checks. but do you want that in the spec? Do you want > javac to generate a self-referential MemberOfNest attribute? If you > want this then the definitions need a rewrite. This is not where we > are today. > >>> It is not an optimization! You are either a nest-top (and so have a >>> NestMembers attribute) or you are a nested type and so have a >>> MemberOfNest attribute. (Of course you can be neither.) >> >> Similarly, I'd think that every class that has neither a NestMembers or >> MemberOfNest attribute constitutes a nest of one. > > Sure logically you can consider it that way. Not sure it has any real > affect on anything. > >> Taken together, now: >> - Every class belongs to exactly one nest; >> - Every nest has a canonical, consistent nest-top (every member of the >> nest agrees on the nest top) > > Yes. > >> This will become relevant when we extend this to support >> Lookup.defineClass with a private lookup, which has the effect of >> injecting a new class into an existing nest. If every class is already >> in a nest, this injection is simplified. > > I don't really see how it makes a difference, but you presumably have > something in mind here. > > Thanks, > David > From forax at univ-mlv.fr Thu Apr 13 22:21:48 2017 From: forax at univ-mlv.fr (forax at univ-mlv.fr) Date: Fri, 14 Apr 2017 00:21:48 +0200 (CEST) Subject: Updated JEP: nestmates In-Reply-To: <3ea33091-e200-4780-365f-7efaa55b14d3@oracle.com> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <679559779.1604218.1492079851801.JavaMail.zimbra@u-pem.fr> <1316798436.1782207.1492100720790.JavaMail.zimbra@u-pem.fr> <3ea33091-e200-4780-365f-7efaa55b14d3@oracle.com> Message-ID: <1907964258.1828186.1492122108606.JavaMail.zimbra@u-pem.fr> [...] >>> The membership is validated >>> during class loading if D also lists C in its >>> `NestMembers` attribute. This may require loading of nest-top types >>> earlier than might otherwise occur." >>> >>> Also note that "loading" is being used to allow some flexibility. What I >>> currently do is validate during class linking, prior to verification (if >>> enabled). But specifying it to happen at that exact point may be too >>> constraining. >> >> yes agree about what "loading" means here, >> in term of implementation, i think it should be validated the first time you >> need the nestmate relationship, it will be less disruptive (see below) > > Note that may be during verification of another class. Any way the > details here can be further explored. I don't think lazier than > link-time buys you anything when you may need it earlier anyway. > sorry, i was not clear enough here. Being lazier than linking time in my mind means doing the check at runtime the first time you access to the member. i.e. during link time (at any phase) you mark that you can not do the full access check but because you have a nest-top, and you delay a part of the access check at runtime. that is what i was thinking when i write 'the first time you need'. regards, R?mi From david.holmes at oracle.com Fri Apr 14 03:16:48 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 14 Apr 2017 13:16:48 +1000 Subject: Updated JEP: nestmates In-Reply-To: <6824725d-962e-0ac9-49ed-44742ece1449@oracle.com> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <679559779.1604218.1492079851801.JavaMail.zimbra@u-pem.fr> <1316798436.1782207.1492100720790.JavaMail.zimbra@u-pem.fr> <3ea33091-e200-4780-365f-7efaa55b14d3@oracle.com> <7b5f9bb1-45e1-0c31-e0f4-c2f185f1af7e@oracle.com> <7dee140d-6a49-a7de-0c2e-dfcfbd0d3962@oracle.com> <6824725d-962e-0ac9-49ed-44742ece1449@oracle.com> Message-ID: On 14/04/2017 8:12 AM, Brian Goetz wrote: > I was trying to make the model simpler to reason about. My mathematical > model is that nests form a partition over classes, meaning: > - every class belongs to a nest > - no two nests have any members in common > - every nest has a canonical member > > We can't rely on javac (or other compilers) to always generate "nest of > one" attributes, so the VM would have to, on seeing a class with no nest > attributes, set "nestTop = me". But I think that would be the only > change in the implementation? And then nestTop(C) becomes a total > function, which simplifies things. The implementation already sets nestTop='me'. But it never expects to see a MemberOfNest attribute already set to 'me'. It complicates things if the specification both allows no nestmate attributes and allows MemberOfNest='me' (at parsing time there is not yet a 'me' so this would based on the name in the constant pool.) David > I'm fine if the spec says this; in fact, I like it. > > On 4/13/2017 5:57 PM, David Holmes wrote: >> On 14/04/2017 7:50 AM, Brian Goetz wrote: >>>> The second part most certainly is necessary as a nest-top does not >>>> itself have a nest-top! >>> >>> I would think we'd consider a nest-top as its own nest-top. That way, >>> every nest has a top, which acts as a canonical element for that nest. >> >> From an implementation perspective certainly that simplifies >> nest-membership checks. but do you want that in the spec? Do you want >> javac to generate a self-referential MemberOfNest attribute? If you >> want this then the definitions need a rewrite. This is not where we >> are today. >> >>>> It is not an optimization! You are either a nest-top (and so have a >>>> NestMembers attribute) or you are a nested type and so have a >>>> MemberOfNest attribute. (Of course you can be neither.) >>> >>> Similarly, I'd think that every class that has neither a NestMembers or >>> MemberOfNest attribute constitutes a nest of one. >> >> Sure logically you can consider it that way. Not sure it has any real >> affect on anything. >> >>> Taken together, now: >>> - Every class belongs to exactly one nest; >>> - Every nest has a canonical, consistent nest-top (every member of the >>> nest agrees on the nest top) >> >> Yes. >> >>> This will become relevant when we extend this to support >>> Lookup.defineClass with a private lookup, which has the effect of >>> injecting a new class into an existing nest. If every class is already >>> in a nest, this injection is simplified. >> >> I don't really see how it makes a difference, but you presumably have >> something in mind here. >> >> Thanks, >> David >> > From david.holmes at oracle.com Fri Apr 14 03:20:49 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 14 Apr 2017 13:20:49 +1000 Subject: Updated JEP: nestmates In-Reply-To: <2125517239.1824274.1492121454846.JavaMail.zimbra@u-pem.fr> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <679559779.1604218.1492079851801.JavaMail.zimbra@u-pem.fr> <1316798436.1782207.1492100720790.JavaMail.zimbra@u-pem.fr> <3ea33091-e200-4780-365f-7efaa55b14d3@oracle.com> <2125517239.1824274.1492121454846.JavaMail.zimbra@u-pem.fr> Message-ID: On 14/04/2017 8:10 AM, forax at univ-mlv.fr wrote: > ----- Mail original ----- >> De: "David Holmes" >> ?: forax at univ-mlv.fr >> Cc: "Brian Goetz" , "John Rose" , valhalla-dev at openjdk.java.net >> Envoy?: Jeudi 13 Avril 2017 23:44:55 >> Objet: Re: Updated JEP: nestmates > > >>> >>>> On 13/04/2017 8:37 PM, Remi Forax wrote: >>>>> Re-reading the spec again, >>>>> i think it would be clearer if you introduce T the nest-top, and modifying the >>>>> classloading order has to be listed in the "Risks and Assumptions" section. >>>>> >>>>> Description: >>>>> "The Java compiler compiles a group of nested types into a corresponding group >>>>> of class files; it uses the InnerClasses and EnclosingMethod attributes to >>>>> reify the nesting relationship (JVMS 4.7.6 and 4.7.7)." >>>>> We can not re-use these attributes to describe nesting relationship because >>>>> being part of a nest has to be verified, triggering classloading of the >>>>> nest-top, thus changing the semantics of the existing attributes. >>>>> ... >>>>> >>>>> "For types C and D to be nestmates they must either have the same nest-top T. A >>>>> type C claims to be a member of the nest of D, if both C and D list T in their >>>>> MemberOfNest attribute. The membership is validated during class loading if T >>>>> also lists C and D in its NestMembers attribute. This may require loading of >>>>> nest-top types earlier than might otherwise occur." >>>> >>>> That paragraph in the JEP got modified slightly from my original. There >>>> was no intent at that point to introduce a third type. My original is: >>>> >>>> "For types C and D to be nestmates they must either have >>>> the same nest-top, or else one must be the nest-top of the other. >>> >>> ok, agree with this one, even if the second part of this sentence is not >>> necessary. >> >> The second part most certainly is necessary as a nest-top does not >> itself have a nest-top! > > Conceptually a nest-top is its own nest-top, but in term of attribute, so yes, you're right. > >> >>>> A type C claims to be a member of the nest of D, if it >>>> lists D in it's `MemberOfNest` attribute. >>> >>> while i understand this sentence, i'm not sure everybody will be able to parse >>> it, >>> apart from the 'it's' -> its, "member of the nest of D" is ambiguous, at least >>> for me, it can be read as "member of the same nest as D" or as "member of the >>> nest named D". >>> Perhaps removing the "of" is enough. >>> A type C claims to be a member of the nest D, if it lists D >>> in it's `MemberOfNest` attribute. >> >> D is not a nest! D is a type that may be in a nest. "nest of D" is >> perfectly correct. We could rephrase to "of the same nest as D". > > if D is not the nest, then MemberOfNest attribute of C is not D but the nest-top of D. Sorry I did not write what I intended, but I also think you are using "nest" to refer to the nest-top ? What I intended to say was D is not a nest! D is a type that may be the nest-top of C. David > R?mi > From david.holmes at oracle.com Fri Apr 14 03:24:49 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 14 Apr 2017 13:24:49 +1000 Subject: Updated JEP: nestmates In-Reply-To: <1907964258.1828186.1492122108606.JavaMail.zimbra@u-pem.fr> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <679559779.1604218.1492079851801.JavaMail.zimbra@u-pem.fr> <1316798436.1782207.1492100720790.JavaMail.zimbra@u-pem.fr> <3ea33091-e200-4780-365f-7efaa55b14d3@oracle.com> <1907964258.1828186.1492122108606.JavaMail.zimbra@u-pem.fr> Message-ID: <966b8eb6-f0b9-82b4-7f07-190e64d08722@oracle.com> On 14/04/2017 8:21 AM, forax at univ-mlv.fr wrote: > [...] > >>>> The membership is validated >>>> during class loading if D also lists C in its >>>> `NestMembers` attribute. This may require loading of nest-top types >>>> earlier than might otherwise occur." >>>> >>>> Also note that "loading" is being used to allow some flexibility. What I >>>> currently do is validate during class linking, prior to verification (if >>>> enabled). But specifying it to happen at that exact point may be too >>>> constraining. >>> >>> yes agree about what "loading" means here, >>> in term of implementation, i think it should be validated the first time you >>> need the nestmate relationship, it will be less disruptive (see below) >> >> Note that may be during verification of another class. Any way the >> details here can be further explored. I don't think lazier than >> link-time buys you anything when you may need it earlier anyway. >> > > sorry, i was not clear enough here. > Being lazier than linking time in my mind means doing the check at runtime the first time you access to the member. > i.e. during link time (at any phase) you mark that you can not do the full access check but because you have a nest-top, and you delay a part of the access check at runtime. > > that is what i was thinking when i write 'the first time you need'. You can't do that if verification is involved. As it stands the use of invokespecial to access a private nestmate method will not pass the verifier. So we have to relax the rule in the verifier to allow for nestmate access - which means we need to resolve (and validate) the nest-top when verification occurs. Otherwise you've opened up a hole in the verification process. David > regards, > R?mi > From david.holmes at oracle.com Fri Apr 14 03:28:32 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 14 Apr 2017 13:28:32 +1000 Subject: Updated JEP: nestmates In-Reply-To: <1902427860.1821668.1492121148906.JavaMail.zimbra@u-pem.fr> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <679559779.1604218.1492079851801.JavaMail.zimbra@u-pem.fr> <1316798436.1782207.1492100720790.JavaMail.zimbra@u-pem.fr> <3ea33091-e200-4780-365f-7efaa55b14d3@oracle.com> <7b5f9bb1-45e1-0c31-e0f4-c2f185f1af7e@oracle.com> <7dee140d-6a49-a7de-0c2e-dfcfbd0d3962@oracle.com> <1902427860.1821668.1492121148906.JavaMail.zimbra@u-pem.fr> Message-ID: <9d1c7c50-4161-8427-af23-899744dc2c2b@oracle.com> On 14/04/2017 8:05 AM, forax at univ-mlv.fr wrote: > It seems we are all on the same page but some of us talk about the spec and other about the implementation. Not sure which of "us" you are referring to for what discussion. :) I was trying to stick with the spec but the definition of the attributes and when they are allowed and not allowed, is part of the spec. David > From the spec point of view, i agree with Brian, > "Every class belongs to exactly one nest", > - if there is a MemberOfNest attribute, it defines the nest, > - if there is a NestMembers attribute, it defines the nest-top, > - if there is no attribute, it defines its own nest. > and "Every nest has a canonical, consistent nest-top". > > R?mi > > ----- Mail original ----- >> De: "David Holmes" >> ?: "Brian Goetz" , forax at univ-mlv.fr >> Cc: "John Rose" , valhalla-dev at openjdk.java.net >> Envoy?: Jeudi 13 Avril 2017 23:57:59 >> Objet: Re: Updated JEP: nestmates > >> On 14/04/2017 7:50 AM, Brian Goetz wrote: >>>> The second part most certainly is necessary as a nest-top does not >>>> itself have a nest-top! >>> >>> I would think we'd consider a nest-top as its own nest-top. That way, >>> every nest has a top, which acts as a canonical element for that nest. >> >> From an implementation perspective certainly that simplifies >> nest-membership checks. but do you want that in the spec? Do you want >> javac to generate a self-referential MemberOfNest attribute? If you want >> this then the definitions need a rewrite. This is not where we are today. >> >>>> It is not an optimization! You are either a nest-top (and so have a >>>> NestMembers attribute) or you are a nested type and so have a >>>> MemberOfNest attribute. (Of course you can be neither.) >>> >>> Similarly, I'd think that every class that has neither a NestMembers or >>> MemberOfNest attribute constitutes a nest of one. >> >> Sure logically you can consider it that way. Not sure it has any real >> affect on anything. >> >>> Taken together, now: >>> - Every class belongs to exactly one nest; >>> - Every nest has a canonical, consistent nest-top (every member of the >>> nest agrees on the nest top) >> >> Yes. >> >>> This will become relevant when we extend this to support >>> Lookup.defineClass with a private lookup, which has the effect of >>> injecting a new class into an existing nest. If every class is already >>> in a nest, this injection is simplified. >> >> I don't really see how it makes a difference, but you presumably have >> something in mind here. >> >> Thanks, >> David From forax at univ-mlv.fr Fri Apr 14 10:14:40 2017 From: forax at univ-mlv.fr (forax at univ-mlv.fr) Date: Fri, 14 Apr 2017 12:14:40 +0200 (CEST) Subject: Updated JEP: nestmates In-Reply-To: <966b8eb6-f0b9-82b4-7f07-190e64d08722@oracle.com> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <679559779.1604218.1492079851801.JavaMail.zimbra@u-pem.fr> <1316798436.1782207.1492100720790.JavaMail.zimbra@u-pem.fr> <3ea33091-e200-4780-365f-7efaa55b14d3@oracle.com> <1907964258.1828186.1492122108606.JavaMail.zimbra@u-pem.fr> <966b8eb6-f0b9-82b4-7f07-190e64d08722@oracle.com> Message-ID: <434246662.1982115.1492164880780.JavaMail.zimbra@u-pem.fr> ----- Mail original ----- > De: "David Holmes" > ?: forax at univ-mlv.fr > Cc: "Brian Goetz" , "John Rose" , valhalla-dev at openjdk.java.net > Envoy?: Vendredi 14 Avril 2017 05:24:49 > Objet: Re: Updated JEP: nestmates > On 14/04/2017 8:21 AM, forax at univ-mlv.fr wrote: >> [...] >> >>>>> The membership is validated >>>>> during class loading if D also lists C in its >>>>> `NestMembers` attribute. This may require loading of nest-top types >>>>> earlier than might otherwise occur." >>>>> >>>>> Also note that "loading" is being used to allow some flexibility. What I >>>>> currently do is validate during class linking, prior to verification (if >>>>> enabled). But specifying it to happen at that exact point may be too >>>>> constraining. >>>> >>>> yes agree about what "loading" means here, >>>> in term of implementation, i think it should be validated the first time you >>>> need the nestmate relationship, it will be less disruptive (see below) >>> >>> Note that may be during verification of another class. Any way the >>> details here can be further explored. I don't think lazier than >>> link-time buys you anything when you may need it earlier anyway. >>> >> >> sorry, i was not clear enough here. >> Being lazier than linking time in my mind means doing the check at runtime the >> first time you access to the member. >> i.e. during link time (at any phase) you mark that you can not do the full >> access check but because you have a nest-top, and you delay a part of the >> access check at runtime. >> >> that is what i was thinking when i write 'the first time you need'. > > You can't do that if verification is involved. As it stands the use of > invokespecial to access a private nestmate method will not pass the > verifier. So we have to relax the rule in the verifier to allow for > nestmate access - which means we need to resolve (and validate) the > nest-top when verification occurs. Otherwise you've opened up a hole in > the verification process. > yes, there is a hole in the verification if you do not add a check at runtime. First, i'm not advocating that i should be implemented that way, i'm just saying that if you delay the class loading, you will have less spurious classloading from the user POV. When doing the verification of invokespecial: - if the verification fail if you do not consider nestmate - if invokespecial is a call to a class with an attribute MemberOfNest and the callsite class also an attribute MemberOfNest, and both attributes have the same value then instead of loading the nest-top at that point, the verifier can mark that invokespecial and delay the loading of the nest-top to the first time the invokespecial is executed. This is very similar to what the VM currently does when checking a cast to an interface. And again, i may significantly reduce the number of nest-tops loaded or do not worth the trouble to implement that. > David > regards, R?mi From john.r.rose at oracle.com Fri Apr 14 20:00:01 2017 From: john.r.rose at oracle.com (John Rose) Date: Fri, 14 Apr 2017 13:00:01 -0700 Subject: Updated JEP: nestmates In-Reply-To: <2145723641.1794072.1492106286649.JavaMail.zimbra@u-pem.fr> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <1709982294.1362541.1492034442839.JavaMail.zimbra@u-pem.fr> <1972301568.1606896.1492080660322.JavaMail.zimbra@u-pem.fr> <6fe49c17-829f-70e6-ba95-7f7a24c79d36@oracle.com> <1354135365.1784542.1492101671883.JavaMail.zimbra@u-pem.fr> <14DE1477-1850-4BC3-8482-D96E94EE6437@oracle.com> <2145723641.1794072.1492106286649.JavaMail.zimbra@u-pem.fr> Message-ID: On Apr 13, 2017, at 10:58 AM, forax at univ-mlv.fr wrote: >> In fact, for compatibility, it must also track InnerClasses attributes. (Yes, yuck.) > > I do not think it's a good idea. > I suppose it depends if nestmates have their own reflect API or not. > If you can access to nestmates using reflection, i suppose code that uses InnerClasses (Class.getClasses()) can be updated to take care about nestmates. To be clear, I am not proposing that innerclasses and nestmates are to be confused by reflection or the JVM or JLI. (Such confusion would be yuck^4.) They are different things altogether, at the JVM level, different expressions of source-level structure. The reflection API for nestmates needs to be new. (Something like Class.getNestTop, Class.getNestMembers.) The existing reflective machinery for Class.getDeclaredClasses, Class.getDeclaringClass, etc., needs to stay exactly the same, for compatibility. It provides much more structural information than nestmates ever could. Lookup.in uses the InnerClasses attribute today, for purposes of deciding private access within a source-level nest of classes (and injected code). But if the new kind of information is present in the future, it should ignore the old information it uses today. If the new nestmate information is not present, it needs to continue doing what it is doing today, for compatibility. (I suppose it could also sniff the class file version and mask out InnerClasses for up-rev classfiles, but that's not necessary.) For class-files compiled with nestmates turned on, either *both* or *neither* of the old InnerClasses and the new nestmate information will be stored in the classfile. (Right?) And Lookup.in needs to ignore the InnerClasses in that case. Meanwhile, the JVM's internal access control logic will also ignore the InnerClasses, as it has always done and always will do. The net result is that Lookup.in will continue to reflectively model the source-level access control rules for private members of nested classes, but use the upgraded information source from the classfile when it is available. Make sense? ? John From john.r.rose at oracle.com Fri Apr 14 20:17:08 2017 From: john.r.rose at oracle.com (John Rose) Date: Fri, 14 Apr 2017 13:17:08 -0700 Subject: Updated JEP: nestmates In-Reply-To: <3ea33091-e200-4780-365f-7efaa55b14d3@oracle.com> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <679559779.1604218.1492079851801.JavaMail.zimbra@u-pem.fr> <1316798436.1782207.1492100720790.JavaMail.zimbra@u-pem.fr> <3ea33091-e200-4780-365f-7efaa55b14d3@oracle.com> Message-ID: <7147186B-A8C9-41C8-825A-2C32BB590BE6@oracle.com> On Apr 13, 2017, at 2:44 PM, David Holmes wrote: > > D is not a nest! D is a type that may be in a nest. "nest of D" is perfectly correct. We could rephrase to "of the same nest as D". Hmm, this is tricky to express unambiguously in English. There are several things going on: A nest (viewed as a single "thing"), a set of all classes "in the nest", the reverse relation of each class "in the nest" to the nest it is in, and finally the precursor relations, defined exactly by the attributes, from which the nest emerges: nest-top, nest-members. Conflating the precursor relations with the main relation will cause endless confusion if we are not careful. As you point out, David, it helps to distinguish the "nest" from the "nest top". In pseudocode: // main bidirectional relation: datatype Nest: SetOf(Class) NestMembers: (Nest) -> SetOf(Class) IsInNest: (Class, Nest) -> boolean // utility: IsInSameNestAs: (Class, Class) -> boolean // precursor relations carried directly by attributes: NestTop: Class -> Optional(Class) NestMembers: Class -> Optional(ListOf(Class)) ? John P.S. This relates back to reflection: To me it is a coin-flip whether the JDK Core Reflection API should reify the derived relation (with a Class.Nest object) or the precursor relations. The "style" of Class APIs is to postprocess the internal precursor data into a franken-something that looks like source structure. But I'm leaning towards reifying only the simpler precursor relations, in this case. From john.r.rose at oracle.com Fri Apr 14 20:18:31 2017 From: john.r.rose at oracle.com (John Rose) Date: Fri, 14 Apr 2017 13:18:31 -0700 Subject: Updated JEP: nestmates In-Reply-To: References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <1076292152.1607809.1492080867818.JavaMail.zimbra@u-pem.fr> <2030928f-4f84-fd14-f3d9-2f18efe00605@oracle.com> <1E11AE50-D130-4618-B8E0-FEEDC3CBA5E2@oracle.com> Message-ID: On Apr 13, 2017, at 2:51 PM, David Holmes wrote: > > On 14/04/2017 3:09 AM, John Rose wrote: >> On Apr 13, 2017, at 4:36 AM, David Holmes wrote: >> >>>> you can not modify nest definition attributes at runtime using java.lang.instrument. >>> >>> Yes. >> >> I'm missing something here. Why must nestmate attributes be specifically hardened against instrumentation changes? Can't instrumentation, in general, change any aspect of a classfile? > > Instrumentation is primarily about adding bytecodes to methods. There are strict constraints on what instrumentation can change with respect to the "shape" of a type (members, inheritance hierarchy). > > "The retransformation may change method bodies, the constant pool and attributes. The retransformation must not add, remove or rename fields or methods, change the signatures of methods, change modifiers, or change inheritance.". [ JVM TI spec.] > > Similarly for redefinition, which also adds " or change inheritance". > > So no changes to super-types. And now no changes to nest membership. Thanks; that what I was missing. From john.r.rose at oracle.com Fri Apr 14 20:26:37 2017 From: john.r.rose at oracle.com (John Rose) Date: Fri, 14 Apr 2017 13:26:37 -0700 Subject: Updated JEP: nestmates In-Reply-To: <1902427860.1821668.1492121148906.JavaMail.zimbra@u-pem.fr> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <679559779.1604218.1492079851801.JavaMail.zimbra@u-pem.fr> <1316798436.1782207.1492100720790.JavaMail.zimbra@u-pem.fr> <3ea33091-e200-4780-365f-7efaa55b14d3@oracle.com> <7b5f9bb1-45e1-0c31-e0f4-c2f185f1af7e@oracle.com> <7dee140d-6a49-a7de-0c2e-dfcfbd0d3962@oracle.com> <1902427860.1821668.1492121148906.JavaMail.zimbra@u-pem.fr> Message-ID: On Apr 13, 2017, at 3:05 PM, forax at univ-mlv.fr wrote: > > It seems we are all on the same page but some of us talk about the spec and other about the implementation. > > From the spec point of view, i agree with Brian, > "Every class belongs to exactly one nest", > - if there is a MemberOfNest attribute, it defines the nest, > - if there is a NestMembers attribute, it defines the nest-top, > - if there is no attribute, it defines its own nest. > and "Every nest has a canonical, consistent nest-top". Confirmed. So the attributes are precursor relations which should not be confused with the final product entity, a nest. By "implementation" you mean the same thing as I mean when I say "precursor relations". (The product entity can be represented unambiguously by a Class, the nest-top of the nest. This is what leads to confusion, because that's not exactly the same thing as the class which contains the NestMembers attribute, nor does it *ever* contain the MemberOfNest attribute, despite being a member of the nest of which it is the top, nor is it *ever* in the NestMembers list, despite being a member of that same nest.) This tricky language is another reason to settle on a reflective API sooner rather than later, so we have a rigorous notation for this stuff. Should it be the precursors or the emergent Nest entity? If the latter, should there be a Class.Nest type or should we take a gamble and use a Class object to represent the Nest of which it is the nest-top? ? John From john.r.rose at oracle.com Fri Apr 14 20:34:42 2017 From: john.r.rose at oracle.com (John Rose) Date: Fri, 14 Apr 2017 13:34:42 -0700 Subject: Updated JEP: nestmates In-Reply-To: <6824725d-962e-0ac9-49ed-44742ece1449@oracle.com> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <679559779.1604218.1492079851801.JavaMail.zimbra@u-pem.fr> <1316798436.1782207.1492100720790.JavaMail.zimbra@u-pem.fr> <3ea33091-e200-4780-365f-7efaa55b14d3@oracle.com> <7b5f9bb1-45e1-0c31-e0f4-c2f185f1af7e@oracle.com> <7dee140d-6a49-a7de-0c2e-dfcfbd0d3962@oracle.com> <6824725d-962e-0ac9-49ed-44742ece1449@oracle.com> Message-ID: <4174162E-256B-4E75-BD3B-5E6997DFF137@oracle.com> On Apr 13, 2017, at 3:12 PM, Brian Goetz wrote: > > We can't rely on javac (or other compilers) to always generate "nest of one" attributes, so the VM would have to, on seeing a class with no nest attributes, set "nestTop = me". But I think that would be the only change in the implementation? And then nestTop(C) becomes a total function, which simplifies things. Confirmed. Regarding javac: It would be polite if javac were to agree that if the output classfile has an InnerClasses attribute, and the current class either records an outer-class or the current class *is* an outer-class, then the classfile will also have one of the nestmate attributes. But the system has to be robust if javac drops the ball once in a while. And it certainly has to read down-rev class files correctly. On Apr 13, 2017, at 2:57 PM, David Holmes wrote: > > Do you want javac to generate a self-referential MemberOfNest attribute? If you want this then the definitions need a rewrite. This is not where we are today. FTR, the form of a degenerate nest is not a self-referential MON attribute, but an empty NMs attribute. That's allowed by the spec. ? John From john.r.rose at oracle.com Fri Apr 14 20:40:43 2017 From: john.r.rose at oracle.com (John Rose) Date: Fri, 14 Apr 2017 13:40:43 -0700 Subject: Updated JEP: nestmates In-Reply-To: <1907964258.1828186.1492122108606.JavaMail.zimbra@u-pem.fr> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <679559779.1604218.1492079851801.JavaMail.zimbra@u-pem.fr> <1316798436.1782207.1492100720790.JavaMail.zimbra@u-pem.fr> <3ea33091-e200-4780-365f-7efaa55b14d3@oracle.com> <1907964258.1828186.1492122108606.JavaMail.zimbra@u-pem.fr> Message-ID: On Apr 13, 2017, at 3:21 PM, forax at univ-mlv.fr wrote: > > Being lazier than linking time in my mind means doing the check at runtime the first time you access to the member. That's an option. I kind of like that, but there's a cost: You can get strange failures as a side effect of access checking. I ran into this when prototyping nest access checking on top of InnerClasses. The internal JVM paths, in HotSpot, don't provide for such errors bubbling up from access checking, nor do they provide for logic that says "drop what you are doing and try to load a random class, right now". In pathological cases you might start to see deadlocks. So I'm actually more comfortable with loading the nest-top no later than linking (i.e., verifying or preparing) any of its nestmates. Can we get away with that? ? John From peter.levart at gmail.com Fri Apr 14 22:00:55 2017 From: peter.levart at gmail.com (Peter Levart) Date: Sat, 15 Apr 2017 00:00:55 +0200 Subject: Updated JEP: nestmates In-Reply-To: <4174162E-256B-4E75-BD3B-5E6997DFF137@oracle.com> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <679559779.1604218.1492079851801.JavaMail.zimbra@u-pem.fr> <1316798436.1782207.1492100720790.JavaMail.zimbra@u-pem.fr> <3ea33091-e200-4780-365f-7efaa55b14d3@oracle.com> <7b5f9bb1-45e1-0c31-e0f4-c2f185f1af7e@oracle.com> <7dee140d-6a49-a7de-0c2e-dfcfbd0d3962@oracle.com> <6824725d-962e-0ac9-49ed-44742ece1449@oracle.com> <4174162E-256B-4E75-BD3B-5E6997DFF137@oracle.com> Message-ID: <77ee11f1-8d73-706e-d53b-0dec37f836d7@gmail.com> On 04/14/2017 10:34 PM, John Rose wrote: > Regarding javac: It would be polite if javac were to agree that if the output classfile > has an InnerClasses attribute, and the current class either records an outer-class > or the current class*is* an outer-class, then the classfile will also have one of the > nestmate attributes. > > But the system has to be robust if javac drops the ball once in a while. > And it certainly has to read down-rev class files correctly. ...at first it appears that down-rev files might not need to be treated as nest-mates on the basis of their InnerClasses attributes. They have access bridges to communicate. But someone might want to "inject" a dynamically generated nestmate into a self-nest of an old-rev top-level class then which communicates with nested classes via bridges and expect from this dynamically generated class to have access to private members of nested classes too. So treating down-rev files' InnerClasses attributes as nestmate metadata might be necessary. Regards, Peter From forax at univ-mlv.fr Fri Apr 14 22:28:09 2017 From: forax at univ-mlv.fr (forax at univ-mlv.fr) Date: Sat, 15 Apr 2017 00:28:09 +0200 (CEST) Subject: Updated JEP: nestmates In-Reply-To: References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <1316798436.1782207.1492100720790.JavaMail.zimbra@u-pem.fr> <3ea33091-e200-4780-365f-7efaa55b14d3@oracle.com> <7b5f9bb1-45e1-0c31-e0f4-c2f185f1af7e@oracle.com> <7dee140d-6a49-a7de-0c2e-dfcfbd0d3962@oracle.com> <1902427860.1821668.1492121148906.JavaMail.zimbra@u-pem.fr> Message-ID: <1356223704.2142537.1492208889840.JavaMail.zimbra@u-pem.fr> > De: "John Rose" > ?: "R?mi Forax" > Cc: "David Holmes" , "Brian Goetz" > , valhalla-dev at openjdk.java.net > Envoy?: Vendredi 14 Avril 2017 22:26:37 > Objet: Re: Updated JEP: nestmates > On Apr 13, 2017, at 3:05 PM, forax at univ-mlv.fr wrote: >> It seems we are all on the same page but some of us talk about the spec and >> other about the implementation. >> From the spec point of view, i agree with Brian, >> "Every class belongs to exactly one nest", >> - if there is a MemberOfNest attribute, it defines the nest, >> - if there is a NestMembers attribute, it defines the nest-top, >> - if there is no attribute, it defines its own nest. >> and "Every nest has a canonical, consistent nest-top". > Confirmed. So the attributes are precursor relations which > should not be confused with the final product entity, a nest. > By "implementation" you mean the same thing as I mean > when I say "precursor relations". yes ! > (The product entity can be represented unambiguously > by a Class, the nest-top of the nest. This is what leads > to confusion, because that's not exactly the same thing > as the class which contains the NestMembers attribute, > nor does it *ever* contain the MemberOfNest attribute, > despite being a member of the nest of which it is the top, > nor is it *ever* in the NestMembers list, despite being > a member of that same nest.) > This tricky language is another reason to settle on a reflective > API sooner rather than later, so we have a rigorous notation > for this stuff. Should it be the precursors or the emergent > Nest entity? If the latter, should there be a Class.Nest > type or should we take a gamble and use a Class object > to represent the Nest of which it is the nest-top? The "precursor relation" is class based, so it's not a gamble to use the java.lang.Class object, the nest-top, to represent a nest in the reflection API. > ? John R?mi From forax at univ-mlv.fr Fri Apr 14 22:33:23 2017 From: forax at univ-mlv.fr (forax at univ-mlv.fr) Date: Sat, 15 Apr 2017 00:33:23 +0200 (CEST) Subject: Updated JEP: nestmates In-Reply-To: References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <679559779.1604218.1492079851801.JavaMail.zimbra@u-pem.fr> <1316798436.1782207.1492100720790.JavaMail.zimbra@u-pem.fr> <3ea33091-e200-4780-365f-7efaa55b14d3@oracle.com> <1907964258.1828186.1492122108606.JavaMail.zimbra@u-pem.fr> Message-ID: <395298965.2142639.1492209203829.JavaMail.zimbra@u-pem.fr> I'm fine with that, we can still comeback to that option latter if loading the nest-top during the parsing cause too much breakage with the existing application. R?mi > De: "John Rose" > ?: "R?mi Forax" > Cc: "David Holmes" , "Brian Goetz" > , valhalla-dev at openjdk.java.net > Envoy?: Vendredi 14 Avril 2017 22:40:43 > Objet: Re: Updated JEP: nestmates > On Apr 13, 2017, at 3:21 PM, forax at univ-mlv.fr wrote: >> Being lazier than linking time in my mind means doing the check at runtime the >> first time you access to the member. > That's an option. I kind of like that, but there's a cost: You can get strange > failures > as a side effect of access checking. I ran into this when prototyping nest > access > checking on top of InnerClasses. The internal JVM paths, in HotSpot, don't > provide > for such errors bubbling up from access checking, nor do they provide for logic > that says "drop what you are doing and try to load a random class, right now". > In pathological cases you might start to see deadlocks. > So I'm actually more comfortable with loading the nest-top no later than linking > (i.e., verifying or preparing) any of its nestmates. Can we get away with that? > ? John From david.holmes at oracle.com Sat Apr 15 00:18:59 2017 From: david.holmes at oracle.com (David Holmes) Date: Sat, 15 Apr 2017 10:18:59 +1000 Subject: Updated JEP: nestmates In-Reply-To: <77ee11f1-8d73-706e-d53b-0dec37f836d7@gmail.com> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <679559779.1604218.1492079851801.JavaMail.zimbra@u-pem.fr> <1316798436.1782207.1492100720790.JavaMail.zimbra@u-pem.fr> <3ea33091-e200-4780-365f-7efaa55b14d3@oracle.com> <7b5f9bb1-45e1-0c31-e0f4-c2f185f1af7e@oracle.com> <7dee140d-6a49-a7de-0c2e-dfcfbd0d3962@oracle.com> <6824725d-962e-0ac9-49ed-44742ece1449@oracle.com> <4174162E-256B-4E75-BD3B-5E6997DFF137@oracle.com> <77ee11f1-8d73-706e-d53b-0dec37f836d7@gmail.com> Message-ID: <27e22d6c-1745-3421-60bb-541916de73c5@oracle.com> On 15/04/2017 8:00 AM, Peter Levart wrote: > > > On 04/14/2017 10:34 PM, John Rose wrote: >> Regarding javac: It would be polite if javac were to agree that if >> the output classfile >> has an InnerClasses attribute, and the current class either records an >> outer-class >> or the current class*is* an outer-class, then the classfile will also >> have one of the >> nestmate attributes. >> >> But the system has to be robust if javac drops the ball once in a while. >> And it certainly has to read down-rev class files correctly. > > ...at first it appears that down-rev files might not need to be treated > as nest-mates on the basis of their InnerClasses attributes. They have > access bridges to communicate. > > But someone might want to "inject" a dynamically generated nestmate into > a self-nest of an old-rev top-level class then which communicates with > nested classes via bridges and expect from this dynamically generated > class to have access to private members of nested classes too. So > treating down-rev files' InnerClasses attributes as nestmate metadata > might be necessary. There is no scope for "injection of dynamically generated nestmates" in the current JEP. I'm not even sure what that means in terms of classfiles and bytecode. There would be no bridges in the old-rev top-level class to allow access from the generated "nestmate". My expectation is that classfile version 53- treats nested types exactly as today. Classfile version 54+ uses the new nestmate attributes and does need to generate private accessors. The two worlds should be quite distinct. Regards, David > Regards, Peter > From mark.reinhold at oracle.com Mon Apr 17 18:31:45 2017 From: mark.reinhold at oracle.com (mark.reinhold at oracle.com) Date: Mon, 17 Apr 2017 11:31:45 -0700 (PDT) Subject: JEP 303: Intrinsics for the LDC and INVOKEDYNAMIC Instructions Message-ID: <20170417183145.72208B08DD@eggemoggin.niobe.net> New JEP Candidate: http://openjdk.java.net/jeps/303 - Mark From david.holmes at oracle.com Tue Apr 18 04:41:27 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 18 Apr 2017 14:41:27 +1000 Subject: Updated JEP: nestmates In-Reply-To: <1354135365.1784542.1492101671883.JavaMail.zimbra@u-pem.fr> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <1709982294.1362541.1492034442839.JavaMail.zimbra@u-pem.fr> <1972301568.1606896.1492080660322.JavaMail.zimbra@u-pem.fr> <6fe49c17-829f-70e6-ba95-7f7a24c79d36@oracle.com> <1354135365.1784542.1492101671883.JavaMail.zimbra@u-pem.fr> Message-ID: <1966bc85-b90d-7634-e96f-452fa584a4f6@oracle.com> On 14/04/2017 2:41 AM, forax at univ-mlv.fr wrote: > ----- Mail original ----- >> De: "David Holmes" >> ?: forax at univ-mlv.fr >> Cc: "John Rose" , valhalla-dev at openjdk.java.net >> Envoy?: Jeudi 13 Avril 2017 13:33:56 >> Objet: Re: Updated JEP: nestmates > >> On 13/04/2017 8:51 PM, forax at univ-mlv.fr wrote: >>> ----- Mail original ----- >>>> De: "David Holmes" >>>> ?: "John Rose" , "R?mi Forax" >>>> Cc: valhalla-dev at openjdk.java.net >>>> Envoy?: Jeudi 13 Avril 2017 02:36:59 >>>> Objet: Re: Updated JEP: nestmates >>> >>>> On 13/04/2017 9:16 AM, John Rose wrote: >>>>> On Apr 12, 2017, at 3:00 PM, Remi Forax wrote: >>>>>> >>>>>> This JEP is not about the JVM providing a way to add a new member to the nest, >>>>>> so if a language (stupidly you may say) allow to dynamically add a new member to >>>>>> a nest, the runtime of that language still needs lookup.in(). >>>>> >>>>> Good point, thanks. I removed that bullet item. (Also tidied a few other >>>>> bits.) ? John >>>> >>>> If lookup() now provides access to nest members (which it does) then I >>>> would expect it to provide access to the newly added member as well. >>> >>> Lookup will be upgraded to provides 'nest access' to members defined as >>> nestmates. >>> But as specified by this JEP, the nestmate relationship has to be set up before >>> classloading time: >>> - once a nest-top is loaded you can not add another nest member, >>> - once a type is loaded, it can not be part of a nest afterward. >>> >>> So nestmates is not a dynamic relationship. >>> >>> If i want to have dynamic nests at runtime, i can not use nestmates, >>> but i can *simulate* them, by creating a Lookup on a nest-top (or any other nest >>> members) and using Lookup.in() to see it as a Lookup of a nest member. >>> So Lookup.in() do not provide exactly the same feature as nestmates, thus it >>> should not be deprecated/demoted. >> >> Sorry there are too many missing pieces here for me. What does a >> "simulated" nest-mate even mean? What did you introduce, where and how? >> And what access are you expecting to have to what, and how is that >> access being allowed ??? > > Imagine a scripting language that compiles several classes as part of the same script, > the semantics of the language allow code of the same script to see private members of the classes if there are in the same script. > That scripting language can leverage the nestmate attributes for that. > > Now imagine that this scripting language allows to dynamically add some newly created classes as if they were part of an existing script, here you can not use the nestmate attributes for that because you can not re-open a nest-top at runtime to add newly created types as part of a NestMembers attribute. For that, you can use Lookup.in(). Sorry I don't follow. Who has written this Lookup.in() such that it knows how to locate/identify these dynamically added pseudo-nestmates? Thanks, David ----- > My first email in that thread was about saying to John that nestmate attributes are a compile-time/loading time mechanism so it can not replace all the use cases of Lookup.in(). > >> >> David > > R?mi > From david.holmes at oracle.com Tue Apr 18 04:48:03 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 18 Apr 2017 14:48:03 +1000 Subject: Updated JEP: nestmates In-Reply-To: <14DE1477-1850-4BC3-8482-D96E94EE6437@oracle.com> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <1709982294.1362541.1492034442839.JavaMail.zimbra@u-pem.fr> <1972301568.1606896.1492080660322.JavaMail.zimbra@u-pem.fr> <6fe49c17-829f-70e6-ba95-7f7a24c79d36@oracle.com> <1354135365.1784542.1492101671883.JavaMail.zimbra@u-pem.fr> <14DE1477-1850-4BC3-8482-D96E94EE6437@oracle.com> Message-ID: <2ad6b04a-ed83-b3bd-c3e3-2ba35dcf557a@oracle.com> On 14/04/2017 3:25 AM, John Rose wrote: > On Apr 13, 2017, at 9:41 AM, forax at univ-mlv.fr > wrote: >> >> My first email in that thread was about saying to John that nestmate >> attributes are a compile-time/loading time mechanism so it can not >> replace all the use cases of Lookup.in(). > > Once loaded, the nest is a "live" JVM entity, and as such could be > modified by an appropriate API. > > In fact, we are reserving the private mode from Lookup.defineClass to > mean exactly this: Load a dynamically defined class into a pre-existing > nest. > > Even more, a class which is not part of a nest can become one (the top, > I suppose), when Lookup.defineClass injects a nestmate into it. > > Lookup.in must track all this, and so will the native access logic of > the JVM. In fact, for compatibility, it must also track InnerClasses > attributes. (Yes, yuck.) > > (David, dynamic nest extension is a likely feature of > Lookup.defineClass; it is not part of the base nestmate logic.) Understood - conceptually at least. I was under the perhaps misguided assumption that there was still a static link between the "anonymous class" generated and the nest-top to which it would be added, such that the addition could be validated somewhat (and similarly for the generic specialization case). I don't know how Lookup.defineClass would validate a request to inject a nestmate. Thanks, David > ? John From david.holmes at oracle.com Tue Apr 18 05:15:50 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 18 Apr 2017 15:15:50 +1000 Subject: Updated JEP: nestmates In-Reply-To: <434246662.1982115.1492164880780.JavaMail.zimbra@u-pem.fr> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <679559779.1604218.1492079851801.JavaMail.zimbra@u-pem.fr> <1316798436.1782207.1492100720790.JavaMail.zimbra@u-pem.fr> <3ea33091-e200-4780-365f-7efaa55b14d3@oracle.com> <1907964258.1828186.1492122108606.JavaMail.zimbra@u-pem.fr> <966b8eb6-f0b9-82b4-7f07-190e64d08722@oracle.com> <434246662.1982115.1492164880780.JavaMail.zimbra@u-pem.fr> Message-ID: Hi Remi, On 14/04/2017 8:14 PM, forax at univ-mlv.fr wrote: > ----- Mail original ----- >> De: "David Holmes" >> ?: forax at univ-mlv.fr >> Cc: "Brian Goetz" , "John Rose" , valhalla-dev at openjdk.java.net >> Envoy?: Vendredi 14 Avril 2017 05:24:49 >> Objet: Re: Updated JEP: nestmates > >> On 14/04/2017 8:21 AM, forax at univ-mlv.fr wrote: >>> [...] >>> >>>>>> The membership is validated >>>>>> during class loading if D also lists C in its >>>>>> `NestMembers` attribute. This may require loading of nest-top types >>>>>> earlier than might otherwise occur." >>>>>> >>>>>> Also note that "loading" is being used to allow some flexibility. What I >>>>>> currently do is validate during class linking, prior to verification (if >>>>>> enabled). But specifying it to happen at that exact point may be too >>>>>> constraining. >>>>> >>>>> yes agree about what "loading" means here, >>>>> in term of implementation, i think it should be validated the first time you >>>>> need the nestmate relationship, it will be less disruptive (see below) >>>> >>>> Note that may be during verification of another class. Any way the >>>> details here can be further explored. I don't think lazier than >>>> link-time buys you anything when you may need it earlier anyway. >>>> >>> >>> sorry, i was not clear enough here. >>> Being lazier than linking time in my mind means doing the check at runtime the >>> first time you access to the member. >>> i.e. during link time (at any phase) you mark that you can not do the full >>> access check but because you have a nest-top, and you delay a part of the >>> access check at runtime. >>> >>> that is what i was thinking when i write 'the first time you need'. >> >> You can't do that if verification is involved. As it stands the use of >> invokespecial to access a private nestmate method will not pass the >> verifier. So we have to relax the rule in the verifier to allow for >> nestmate access - which means we need to resolve (and validate) the >> nest-top when verification occurs. Otherwise you've opened up a hole in >> the verification process. >> > > yes, there is a hole in the verification if you do not add a check at runtime. Philosophically I have problems with very loose verification rules - their laxity seems to me to defeat the purpose of verification ("thou shalt not pass!"). In this case though we only get into the verifier due to the existing knot of rules around invokespecial that we need to circumvent. > First, i'm not advocating that i should be implemented that way, i'm just saying that if you delay the class loading, you will have less spurious classloading from the user POV. > > When doing the verification of invokespecial: > - if the verification fail if you do not consider nestmate > - if invokespecial is a call to a class with an attribute MemberOfNest and the callsite class also an attribute MemberOfNest, > and both attributes have the same value > then instead of loading the nest-top at that point, the verifier can mark that invokespecial and delay the loading of the nest-top to the first time the invokespecial is executed. This seems quite reasonable - reduce from a strict "are nestmates" to "may possibly be nestmates" (or more strictly "are not definitely not nestmates). Though I'm not certain exactly where the full check would have to be introduced. Thanks, David ----- > This is very similar to what the VM currently does when checking a cast to an interface. > > And again, i may significantly reduce the number of nest-tops loaded or do not worth the trouble to implement that. > >> David >> > > regards, > R?mi > From david.holmes at oracle.com Tue Apr 18 05:27:26 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 18 Apr 2017 15:27:26 +1000 Subject: Updated JEP: nestmates In-Reply-To: References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <1709982294.1362541.1492034442839.JavaMail.zimbra@u-pem.fr> <1972301568.1606896.1492080660322.JavaMail.zimbra@u-pem.fr> <6fe49c17-829f-70e6-ba95-7f7a24c79d36@oracle.com> <1354135365.1784542.1492101671883.JavaMail.zimbra@u-pem.fr> <14DE1477-1850-4BC3-8482-D96E94EE6437@oracle.com> <2145723641.1794072.1492106286649.JavaMail.zimbra@u-pem.fr> Message-ID: <42d284bc-df14-20b2-256b-6aeb4daddbc2@oracle.com> On 15/04/2017 6:00 AM, John Rose wrote: > On Apr 13, 2017, at 10:58 AM, forax at univ-mlv.fr > wrote: >>> In fact, for compatibility, it must also track InnerClasses >>> attributes. (Yes, yuck.) >> >> I do not think it's a good idea. >> I suppose it depends if nestmates have their own reflect API or not. >> If you can access to nestmates using reflection, i suppose code that >> uses InnerClasses (Class.getClasses()) can be updated to take care >> about nestmates. > > To be clear, I am not proposing that innerclasses and nestmates are > to be confused by reflection or the JVM or JLI. (Such confusion > would beyuck^4.) > They are different things altogether, at the JVM level, different > expressions of source-level structure. > > The reflection API for nestmates needs to be new. (Something like > Class.getNestTop, Class.getNestMembers.) The existing reflective > machinery for Class.getDeclaredClasses, Class.getDeclaringClass, > etc., needs to stay exactly the same, for compatibility. It provides > much more structural information than nestmates ever could. A reflection API for nestmates has not formed part of the investigation, or implementation, to date. We should add this to the JEP if we do intend to add one. > Lookup.in uses the InnerClasses attribute today, for purposes of > deciding private access within a source-level nest of classes (and > injected code). But if the new kind of information is present in the > future, it should ignore the old information it uses today. FTR currently I've modified things such that lookup() will find nestmate private members. I have not intentionally done anything so far with Lookup.in() and I don't know what it currently will provide. > If the new nestmate information is not present, it needs to continue > doing what it is doing today, for compatibility. (I suppose it could > also sniff the class file version and mask out InnerClasses for > up-rev classfiles, but that's not necessary.) I was planning to make everything related to the use of nestmate attributes classfile version 54+ dependent. > For class-files compiled with nestmates turned on, either *both* or > *neither* of the old InnerClasses and the new nestmate information > will be stored in the classfile. (Right?) And Lookup.in needs to > ignore the InnerClasses in that case. Meanwhile, the JVM's internal > access control logic will also ignore the InnerClasses, as it has > always done and always will do. Currently the nestmate work is completely independent-of/ignorant-of/agnostic-to anything related to innerclass attributes. The two universes are disjoint. > The net result is that Lookup.in will continue to reflectively model the > source-level access > control rules for private members of nested classes, but use the > upgraded information > source from the classfile when it is available. I had thought from previous comments that Lookup.in was effectively deprecated wrt. nestmates and that lookup() would provide the needed access. I was expecting Lookup.in() to continue to do whatever it does today, the way it does it today. Note I'm not opposing this, just noting it would need to be added to the to-do list. Cheers, David ----- > Make sense? > > ? John From david.holmes at oracle.com Tue Apr 18 05:35:32 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 18 Apr 2017 15:35:32 +1000 Subject: Updated JEP: nestmates In-Reply-To: <4174162E-256B-4E75-BD3B-5E6997DFF137@oracle.com> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <679559779.1604218.1492079851801.JavaMail.zimbra@u-pem.fr> <1316798436.1782207.1492100720790.JavaMail.zimbra@u-pem.fr> <3ea33091-e200-4780-365f-7efaa55b14d3@oracle.com> <7b5f9bb1-45e1-0c31-e0f4-c2f185f1af7e@oracle.com> <7dee140d-6a49-a7de-0c2e-dfcfbd0d3962@oracle.com> <6824725d-962e-0ac9-49ed-44742ece1449@oracle.com> <4174162E-256B-4E75-BD3B-5E6997DFF137@oracle.com> Message-ID: <62174450-9415-fc1d-7da8-2f65c75e0b34@oracle.com> Just to reiterate comments I made in other responses ... On 15/04/2017 6:34 AM, John Rose wrote: > On Apr 13, 2017, at 3:12 PM, Brian Goetz > wrote: >> >> We can't rely on javac (or other compilers) to always generate "nest >> of one" attributes, so the VM would have to, on seeing a class with no >> nest attributes, set "nestTop = me". But I think that would be the >> only change in the implementation? And then nestTop(C) becomes a >> total function, which simplifies things. > > Confirmed. > > Regarding javac: It would be polite if javac were to agree that if > the output classfile has an InnerClasses attribute, and the current > class either records an outer-class or the current class *is* an > outer-class, then the classfile will also have one of the nestmate > attributes. Currently nestmate attribute generation is completely independent of innerclass atrributes and only enabled for "-release 10" or later. > But the system has to be robust if javac drops the ball once in a while. Not sure what that means. The spec will define legal use of the new attributes and the VM will reject any non-conformant use. > And it certainly has to read down-rev class files correctly. Yes, as I said, pre 54 classfiles are expected to continue to use innerclass attributes (and only those - though IIUC we can't reject unexpected attributes, only ignore them). > On Apr 13, 2017, at 2:57 PM, David Holmes > wrote: >> >> Do you want javac to generate a self-referential MemberOfNest >> attribute? If you want this then the definitions need a rewrite. This >> is not where we are today. > > FTR, the form of a degenerate nest is not a self-referential MON > attribute, but an empty NMs attribute. That's allowed by the spec. Are you saying that is what you want javac to generate? The current proposed definition of the NM attribute prohibits it: https://bugs.openjdk.java.net/browse/JDK-8177020 Add: 4.7.26 The `NestMembers` Attribute The `NestMembers` attribute is a variable-length attribute in the attributes table of a `ClassFile` structure (?4.1). If the constant pool of a class or interface `C` contains at least one `CONSTANT_Class_info` entry (?4.4.1) which represents a class or interface that is not a member of a package, then there must be exactly one `NestMembers` attribute in the attributes table of the `ClassFile` structure for `C`. If `C` itself is not a member of a package, then there must not be a `NestMembers` attribute in the attributes table of the `ClassFile` structure for `C`. --- Cheers, David > ? John From david.holmes at oracle.com Tue Apr 18 05:48:45 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 18 Apr 2017 15:48:45 +1000 Subject: Updated JEP: nestmates In-Reply-To: <62174450-9415-fc1d-7da8-2f65c75e0b34@oracle.com> References: <9a30448d-6293-a0a6-3889-0fa84697072b@oracle.com> <679559779.1604218.1492079851801.JavaMail.zimbra@u-pem.fr> <1316798436.1782207.1492100720790.JavaMail.zimbra@u-pem.fr> <3ea33091-e200-4780-365f-7efaa55b14d3@oracle.com> <7b5f9bb1-45e1-0c31-e0f4-c2f185f1af7e@oracle.com> <7dee140d-6a49-a7de-0c2e-dfcfbd0d3962@oracle.com> <6824725d-962e-0ac9-49ed-44742ece1449@oracle.com> <4174162E-256B-4E75-BD3B-5E6997DFF137@oracle.com> <62174450-9415-fc1d-7da8-2f65c75e0b34@oracle.com> Message-ID: <4390673a-d5d0-b9d0-8639-132f71a78414@oracle.com> Correction ... On 18/04/2017 3:35 PM, David Holmes wrote: > Just to reiterate comments I made in other responses ... > > On 15/04/2017 6:34 AM, John Rose wrote: >> On Apr 13, 2017, at 3:12 PM, Brian Goetz > > wrote: >>> >>> We can't rely on javac (or other compilers) to always generate "nest >>> of one" attributes, so the VM would have to, on seeing a class with no >>> nest attributes, set "nestTop = me". But I think that would be the >>> only change in the implementation? And then nestTop(C) becomes a >>> total function, which simplifies things. >> >> Confirmed. >> >> Regarding javac: It would be polite if javac were to agree that if >> the output classfile has an InnerClasses attribute, and the current >> class either records an outer-class or the current class *is* an >> outer-class, then the classfile will also have one of the nestmate >> attributes. > > Currently nestmate attribute generation is completely independent of > innerclass atrributes and only enabled for "-release 10" or later. > >> But the system has to be robust if javac drops the ball once in a while. > > Not sure what that means. The spec will define legal use of the new > attributes and the VM will reject any non-conformant use. > >> And it certainly has to read down-rev class files correctly. > > Yes, as I said, pre 54 classfiles are expected to continue to use > innerclass attributes (and only those - though IIUC we can't reject > unexpected attributes, only ignore them). > >> On Apr 13, 2017, at 2:57 PM, David Holmes > > wrote: >>> >>> Do you want javac to generate a self-referential MemberOfNest >>> attribute? If you want this then the definitions need a rewrite. This >>> is not where we are today. >> >> FTR, the form of a degenerate nest is not a self-referential MON >> attribute, but an empty NMs attribute. That's allowed by the spec. > > Are you saying that is what you want javac to generate? The current > proposed definition of the NM attribute prohibits it: > > https://bugs.openjdk.java.net/browse/JDK-8177020 > > Add: 4.7.26 The `NestMembers` Attribute > > The `NestMembers` attribute is a variable-length attribute in the > attributes table of a `ClassFile` structure (?4.1). > > If the constant pool of a class or interface `C` contains at least one > `CONSTANT_Class_info` entry (?4.4.1) which represents a class or > interface that is not a member of a package, then there must be exactly > one `NestMembers` attribute in the attributes table of the `ClassFile` > structure for `C`. If `C` itself is not a member of a package, then > there must not be a `NestMembers` attribute in the attributes table of > the `ClassFile` structure for `C`. > --- Sorry - the above doesn't actually prohibit putting a NM attribute with an empty array. David > Cheers, > David > >> ? John From daniel.smith at oracle.com Tue Apr 18 18:44:11 2017 From: daniel.smith at oracle.com (Dan Smith) Date: Tue, 18 Apr 2017 12:44:11 -0600 Subject: Fwd: Draft JVMS changes for Nestmates References: Message-ID: <0848391A-AF8C-4B4D-8CDA-FFC727E5CF6B@oracle.com> For those not subscribed to (or not paying attention to) valhalla-spec-experts, I just posted a link to an updated draft of JVMS changes. > Begin forwarded message: > > From: Dan Smith > Subject: Draft JVMS changes for Nestmates > Date: April 18, 2017 at 12:42:16 PM MDT > To: valhalla-spec-experts at openjdk.java.net > > Hi, all. > > I've uploaded a draft of JVMS changes for JEP 181 "Align JVM Checks with Java Language Rules for Nested Classes" to: > http://cr.openjdk.java.net/~dlsmith/private-access.html > > Some comments below on my thinking in drafting the spec text, and on the JEP generally. > > JEP name > > I don't know how permanent JEP names are supposed to be, but I'd prefer a different name at this point. Something like: "Expanded JVM Access to Private Members"?shorter, focused on the feature itself rather than its relationship to the Java language. Or maybe "Class Nests for Access to Private Members". > > Terminology > > The term "nest" is nice because it's short and mostly unspoiled by overloading in this context (I think?); it's not great because it's informal and doesn't mean anything the first time you hear it. I thought about something more clinical like "access control context", but I'm not convinced that's an improvement. How do others feel? > > The JEP uses "nest top" to describe the class that nest members reference; I prefer "host class", which better describes the class's role and isn't tied to the Java "top level class" concept. I know we use "host class" internally in Hotspot, perhaps when working with anonymous classes (of the JVM flavor), but I think in that context it will ultimately mean the same thing? Are we comfortable repurposing the term in this way? > > I follow Brian's model when it comes to nest membership (5.4.4): every class belongs to a nest, possibly (in the absence of MemberOfNest) the nest hosted by itself. Many nests are singletons without any associated explicit attributes. > > Verification of MemberOfNest > > I include a discussion block about different options of validating MemberOfNest. I think the consensus, and my preference, is to do it during verification of the member class. (NestMembers, on the other hand, is never validated, except as a side-effect of checking MemberOfNest.) > > Verification of invokespecial > > Allowing invokespecial to refer to classes other than the current class and its supers is a significant change. I noticed and relied heavily on the parallel with invokevirtual making protected method calls. So I tried to make the two as similar as possible. In a few places, the treatment of protected methods doesn't seem ideal, and rather than trying to mirror that with non-private invokespecial, I modified the protected method treatment. > > The "protected check" of verification, in particular, was a mess before, and I think I've made it a lot more manageable, and compatible with a parallel rule for invokespecial. I could use some feedback on exactly what Hotspot's verifier has been doing here, though, since I'm pretty sure it didn't match the old specified rules. > > MethodHandle resolution > > The spec (5.4.3.5) is vague about what errors can occur during MethodHandle resolution. I assume any linkage error specified for the instruction can also occur via MethodHandle resolution, and that will include failures due to invokespecial improperly referencing a class. > > Dynamic checking of invokespecial receivers > > When invokespecial involves interface types, the verifier can't guarantee that receiver objects actually implement that interface (JDK-8134358). It's an open question whether this is really a problem, but I included a tentative fix in the invokespecial runtime rules. > > Compiler changes > > The JEP text can't seem to decide if compiler changes are part of it, or a follow-up exercise. I think we're best off explicitly including the compiler changes, which will provide opportunities for design validation and testing. > > API changes > > I haven't tried to address changes that need to be made to APIs. Somebody will need to. For example, Lookup.findSpecial probably needs to make adjustments to account for private members in nestmates, and to parallel the new verification/linkage rules (e.g., follow Lookup.findVirtual in restricting the receiver type). > > Security risk > > The JEP text should acknowledge that, while this does allow compilers to grant finer-grained access to members shared by nestmates, it also pushes compilers to grant broader access to members that were previously kept private. It's a trade-off, and presumably a good one because nestmates are completely trusted, while package-mates might sometimes be suspect. > > (I guess this argument really ought to be made from the top: declaring things with package access is worse than inventing a new level of access because ________.) > > ?Dan From tobias.hartmann at oracle.com Tue Apr 18 19:35:06 2017 From: tobias.hartmann at oracle.com (Tobias Hartmann) Date: Tue, 18 Apr 2017 21:35:06 +0200 Subject: C2 support for reference fields in value types Message-ID: <7cc8c827-b7de-2397-c33c-f5a6ca6f84ee@oracle.com> Hi, here's my prototype for C2 support for reference fields in value types: http://cr.openjdk.java.net/~thartmann/valhalla/vt_prototype/webrev.14/ Changes: - Changed the C2 code and the C2I/I2C adapters to correctly read/write oop fields of value types. After initializing fields of newly allocated values types in the C2I adapter, we call into the runtime to apply the GC post barriers to oop fields. - Adapted escape analysis and C2's type system to correctly handle oop fields. - Extended nmethod::preserve_callee_argument_oops() to correctly iterate over reference fields that are passed as arguments if ValueTypePassFieldsAsArgs is enabled - I had to change the match rule for ALLOC to distinguish between value type allocations and the new Integer() allocations. - Added tests Open issues: - Disabling the call to SharedRuntime::apply_post_barriers() does not trigger any problems. Either post-barriers are not required for some reason or we are missing the corresponding test case. I'm also seeing some crashes in apply_post_barriers() on JPRT but I was not yet able to reproduce those locally. - I had to disable the assert in ConstantPool::klass_at_impl() if ValueTypePassFieldsAsArgs is enabled because we need to resolve call sites from the GC thread. We should find a better solution and re-enable the assert. Also, the resolve_invoke() call in nmethod.cpp very rarely fails with "Should not have any exceptions pending". - The ValueOops.java test fails with "RuntimeException: Number of frame oops incorrect" if executed with -Xcomp. I'm not sure if it is a test bug or an issue with C2. - We need tests for value types with array fields I'll be very busy with preparing a presentation for Oracle Code until mid next week. Roland, Zoltan, feel free to have a look! Thanks, Tobias From rwestrel at redhat.com Thu Apr 27 15:57:32 2017 From: rwestrel at redhat.com (Roland Westrelin) Date: Thu, 27 Apr 2017 17:57:32 +0200 Subject: C2 support for reference fields in value types In-Reply-To: <7cc8c827-b7de-2397-c33c-f5a6ca6f84ee@oracle.com> References: <7cc8c827-b7de-2397-c33c-f5a6ca6f84ee@oracle.com> Message-ID: Hi Tobias, > - Extended nmethod::preserve_callee_argument_oops() to correctly > iterate over reference fields that are passed as arguments if > ValueTypePassFieldsAsArgs is enabled Instead of indirecting through the adapter to get the extended signature, could you instead: - in gen_c2i_adapter() rather than build an OopMap with all registers, construct one that only contains live oops - then make the gc code use that oop map, maybe by not going through nmethod::preserve_callee_argument_oops() at all? Roland.