From karen.kinnear at oracle.com Wed Nov 1 15:01:02 2017 From: karen.kinnear at oracle.com (Karen Kinnear) Date: Wed, 1 Nov 2017 11:01:02 -0400 Subject: Valhalla EG minutes October 25, 2017 Message-ID: Attendees: Remi, Dan H, John, Mr Simms, Frederic, Lois, Karen AIs: 1. All - review updated ConDy spec 2. Dan H, Dan S - review Karen?s summary of transitive overriding implementation - once we agree on that, we can sanity check if the specification update matches (the intention was not to change the meaning, but to improve consistency of the overall selection and overriding specifications) 3. Dan S - respond to concerns below 4. Remi - send link to latest ASM - with fixes for stackmap generation for wide prefixes? I. ConDy spec: latest copies linked off of: https://bugs.openjdk.java.net/browse/JDK-8189199 1. Issue - why do we have to rename Constant_InvokeDynamic to Constant_DynamicCallSite? Remi brought up a serious concern here Summary: there are three levels of impact here: 1. JDK tools and JVM implementations e.g. IBM ROM classes today convert back to using constant pool names 2. All byte code tools - e.g. ASM, e.g. IDEs, debuggers, profilers a) would need to change to accept either one *** see level 3 for how long this must be supported b) when you print - must choose which one - so you there is not simple ?support both? - so user would need to use flags to specify which one they want - goal is to reduce flags, this goes in the wrong direction 3. users of byte code tools - e.g. tests - (e.g. IBM and Oracle both raised concerns here) - those who use ASM and other bytecode generation/modification APIs for byte code generation *** If the byte code tools do not maintain both names forever, then the level 3: users of byte code tools will have to change all their code Concern is - this is a significant change for the ecosystem and it does not appear to be justified given a cost/benefit analysis ? perhaps - this name?s ship has sailed? 2. ConDy spec update Dan S and John have been working on this - John will send an update (see above link) Open Issue: nested conDy handling - risk of infinite recursion - please feel free to make suggestions - in future we hope to clarify this using the lazy resolution/BootstrapCallInfo, so a bit vague now III. Futures - ConDy futures and some Amber projects John: planning to improve BSM handling in future, with a bit more clarification of VM <-> library interaction Karen: we want to allow flexibility of implementation John: under Amber - more coming - - specification on reflecting ConstantPool entries coming and ways to express unresolved constants, e.g. lambda metafactory only needs symbolic form, not live types Remi: OK if the API does not see if we have a a resolved or not resolved constant John: ok 95% of the time Remi: please send a use case for a edge case John: e.g. might be where we care about the order of exceptions Remi: concern about transition from live types to symbolic forms Remi: how do we decide what projects are discussed under Amber vs. Valhalla so we don?t miss knowing what is coming? Amber covers language features, if they require nontrivial JVM changes, they move to Valhalla for discussion - e.g. ConDy, nestmates, expect future sealed interfaces and sealed fields and frozen arrays Folks here are not yet on Amber mailing lists, so useful to bring periodic Amber updates to this discussion John: Sealed classes, sealed interfaces: asymmetric access controls for an interface - today we use the same access flags for ability to call and ability to subtype for a field - today we use the same access flags for ability to read and ability to write - e.g. Builders may have special permission to write, as an extended constructor and then publish as immutable fields Remi: interface can be hidden via module export for calling, but not for sub typing Dan H: field - limited form of properties John: very very limited - we not stepping in to properties, use methods to simulate properties Remi: like frozen objects? John: can simulate that via publishing Karen: concern vdefault/vwithfield - could expose partially/inconsistenty formed value types Remi: plans for named parameters? John: not planning IV Nestmates: summary of spec issues: 1. Reflection APIs - being discussed in an email with David Holmes 2. additional exception information - David Holmes proposed adding a cause to IllegalAccessError, e.g. as a way to report class not found or verifier error or ... Dan H ok with that approach 3. transitive overriding - Karen sent an email Dan H will check out - then we can re-review the JVMS overriding description with Dan S V. MVT binary timing? 1-2 weeks We are racing stability bug fixing and binary release process Remi: The build takes 40 minutes, Running a test takes 4-5 seconds - so looking forward to pre-built binary Good luck to Bjorn in his new job - we will miss working with you . Tobi Ajila will be attending these meetings thanks, Karen From daniel.smith at oracle.com Wed Nov 1 23:17:38 2017 From: daniel.smith at oracle.com (Dan Smith) Date: Wed, 1 Nov 2017 17:17:38 -0600 Subject: Valhalla EG minutes October 25, 2017 In-Reply-To: References: Message-ID: <9FB9DDCF-D487-4ED4-AC4A-550093DBB082@oracle.com> > On Nov 1, 2017, at 9:01 AM, Karen Kinnear wrote: > > 3. Dan S - respond to concerns below > 1. Issue - why do we have to rename Constant_InvokeDynamic to Constant_DynamicCallSite? > Remi brought up a serious concern here > > Summary: there are three levels of impact here: > 1. JDK tools and JVM implementations > e.g. IBM ROM classes today convert back to using constant pool names > > 2. All byte code tools - e.g. ASM, e.g. IDEs, debuggers, profilers > a) would need to change to accept either one *** see level 3 for how long this must be supported > b) when you print - must choose which one - so you there is not simple ?support both? > - so user would need to use flags to specify which one they want > - goal is to reduce flags, this goes in the wrong direction > > 3. users of byte code tools > - e.g. tests - (e.g. IBM and Oracle both raised concerns here) > - those who use ASM and other bytecode generation/modification APIs for byte code generation > > *** If the byte code tools do not maintain both names forever, then the level 3: users of byte code tools > will have to change all their code > > Concern is - this is a significant change for the ecosystem and it does not appear to be justified given a cost/benefit analysis > > ? perhaps - this name?s ship has sailed? I can respond, in the sense that I can provide some thoughts on why this is being proposed and how important it is. If you're looking for someone to weigh the considerations and make a decision, that's not me. :-) So, some things to consider: - We can survive without every changing names used as aliases for integer values in JVMS, but over time, especially as we take on some ambitious projects like overhauling the bytecodes, this will accrue into unnecessary complexity/obfuscation. - It is not strictly necessary for anybody else to use the names in JVMS. But over time the mismatch will accrue into unnecessary complexity/obfuscation on their part. So the decision to change JVMS should be viewed as a commitment to use the new names in tools, etc., after a reasonable amount of time. - It is good to exercise these "rename a JVM constant" muscles so that they don't atrophy and we find ourselves trapped, unable to ever rename anything. - This change is not particularly important or urgent, although it makes more sense to change it now with the introduction of CONSTANT_Dynamic than in some arbitrary future release. ?Dan From david.holmes at oracle.com Thu Nov 2 01:10:52 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 2 Nov 2017 11:10:52 +1000 Subject: Valhalla EG minutes October 25, 2017 In-Reply-To: References: Message-ID: On 2/11/2017 1:01 AM, Karen Kinnear wrote: > IV Nestmates: > summary of spec issues: > 1. Reflection APIs - being discussed in an email with David Holmes > 2. additional exception information - David Holmes proposed adding a cause to IllegalAccessError, > e.g. as a way to report class not found or verifier error or ... > Dan H ok with that approach This was superceded by Dan S.' JVMS changes to access checking. Resolution errors are simply rethrown (though I do wrap NCDFE in a new NCDFE so that I can tell the user it came from nest-host resolution). Nest membership validation failures result in IllegalAccessErrors (and I add a suitable message explaining why). David ----- > 3. transitive overriding - Karen sent an email > Dan H will check out - then we can re-review the JVMS overriding description with Dan S > > V. MVT binary timing? > 1-2 weeks > We are racing stability bug fixing and binary release process > > Remi: The build takes 40 minutes, Running a test takes 4-5 seconds - so looking forward to pre-built binary > > Good luck to Bjorn in his new job - we will miss working with you . > Tobi Ajila will be attending these meetings > > thanks, > Karen > > > > From karen.kinnear at oracle.com Thu Nov 2 21:35:59 2017 From: karen.kinnear at oracle.com (Karen Kinnear) Date: Thu, 2 Nov 2017 17:35:59 -0400 Subject: Valhalla EG minutes October 25, 2017 In-Reply-To: References: Message-ID: <9B914773-4C75-4C13-A2D6-341B00F0AAEC@oracle.com> Thank you for the clarification - I had misunderstood that. So if you ask for access and return false, with an error due to resolution - you just re throw the resolution failure. If there was no resolution error, then IllegalAccessError is returned with no cause. thank you! Karen > On Nov 1, 2017, at 9:10 PM, David Holmes wrote: > > > > On 2/11/2017 1:01 AM, Karen Kinnear wrote: >> IV Nestmates: >> summary of spec issues: >> 1. Reflection APIs - being discussed in an email with David Holmes >> 2. additional exception information - David Holmes proposed adding a cause to IllegalAccessError, >> e.g. as a way to report class not found or verifier error or ... >> Dan H ok with that approach > > This was superceded by Dan S.' JVMS changes to access checking. Resolution errors are simply rethrown (though I do wrap NCDFE in a new NCDFE so that I can tell the user it came from nest-host resolution). Nest membership validation failures result in IllegalAccessErrors (and I add a suitable message explaining why). > > David > ----- > >> 3. transitive overriding - Karen sent an email >> Dan H will check out - then we can re-review the JVMS overriding description with Dan S >> V. MVT binary timing? >> 1-2 weeks >> We are racing stability bug fixing and binary release process >> Remi: The build takes 40 minutes, Running a test takes 4-5 seconds - so looking forward to pre-built binary >> Good luck to Bjorn in his new job - we will miss working with you . >> Tobi Ajila will be attending these meetings >> thanks, >> Karen >> From david.holmes at oracle.com Thu Nov 2 21:42:55 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 3 Nov 2017 07:42:55 +1000 Subject: Valhalla EG minutes October 25, 2017 In-Reply-To: <9B914773-4C75-4C13-A2D6-341B00F0AAEC@oracle.com> References: <9B914773-4C75-4C13-A2D6-341B00F0AAEC@oracle.com> Message-ID: <08edb359-aee8-9b3d-d2c6-259f17b51ad6@oracle.com> On 3/11/2017 7:35 AM, Karen Kinnear wrote: > Thank you for the clarification - I had misunderstood that. > > So if you ask for access and return false, with an error due to resolution - you just > re throw the resolution failure. If there was no resolution error, then IllegalAccessError is > returned with no cause. Yes. The message gives more detail when the problem was nest membership validation. David > thank you! > Karen > >> On Nov 1, 2017, at 9:10 PM, David Holmes wrote: >> >> >> >> On 2/11/2017 1:01 AM, Karen Kinnear wrote: >>> IV Nestmates: >>> summary of spec issues: >>> 1. Reflection APIs - being discussed in an email with David Holmes >>> 2. additional exception information - David Holmes proposed adding a cause to IllegalAccessError, >>> e.g. as a way to report class not found or verifier error or ... >>> Dan H ok with that approach >> >> This was superceded by Dan S.' JVMS changes to access checking. Resolution errors are simply rethrown (though I do wrap NCDFE in a new NCDFE so that I can tell the user it came from nest-host resolution). Nest membership validation failures result in IllegalAccessErrors (and I add a suitable message explaining why). >> >> David >> ----- >> >>> 3. transitive overriding - Karen sent an email >>> Dan H will check out - then we can re-review the JVMS overriding description with Dan S >>> V. MVT binary timing? >>> 1-2 weeks >>> We are racing stability bug fixing and binary release process >>> Remi: The build takes 40 minutes, Running a test takes 4-5 seconds - so looking forward to pre-built binary >>> Good luck to Bjorn in his new job - we will miss working with you . >>> Tobi Ajila will be attending these meetings >>> thanks, >>> Karen >>> > From david.holmes at oracle.com Tue Nov 7 07:26:47 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 7 Nov 2017 17:26:47 +1000 Subject: [Nestmates] Draft core reflection API Message-ID: <9f36321d-ab54-9a61-9d15-ef11c6156f2b@oracle.com> For comment: http://cr.openjdk.java.net/~dholmes/8188075/webrev/raw_files/new/java/lang/Class.html#getNestHost() Three functions added: - getNestHost - isNestmateOf - getNestMembers The first two never throw exceptions related to nest hosts or nest membership. The third does. Thanks, David From brian.goetz at oracle.com Tue Nov 7 08:41:01 2017 From: brian.goetz at oracle.com (Brian Goetz) Date: Tue, 7 Nov 2017 08:41:01 +0000 Subject: [Nestmates] Draft core reflection API In-Reply-To: <9f36321d-ab54-9a61-9d15-ef11c6156f2b@oracle.com> References: <9f36321d-ab54-9a61-9d15-ef11c6156f2b@oracle.com> Message-ID: <7EC16151-1307-447E-891F-C6944CA38F64@oracle.com> The specs are fine. But a reader who doesn't know what a nest is will never figure it out from the docs. Would be good if nest host had an API note that explained how classes become nestmantes. Sent from my MacBook Wheel > On Nov 7, 2017, at 7:26 AM, David Holmes wrote: > > For comment: > > http://cr.openjdk.java.net/~dholmes/8188075/webrev/raw_files/new/java/lang/Class.html#getNestHost() > > Three functions added: > - getNestHost > - isNestmateOf > - getNestMembers > > The first two never throw exceptions related to nest hosts or nest membership. The third does. > > Thanks, > David From david.holmes at oracle.com Tue Nov 7 10:31:39 2017 From: david.holmes at oracle.com (David Holmes) Date: Tue, 7 Nov 2017 20:31:39 +1000 Subject: [Nestmates] Draft core reflection API In-Reply-To: <7EC16151-1307-447E-891F-C6944CA38F64@oracle.com> References: <9f36321d-ab54-9a61-9d15-ef11c6156f2b@oracle.com> <7EC16151-1307-447E-891F-C6944CA38F64@oracle.com> Message-ID: <0113856c-245d-2048-0949-673da515a7bc@oracle.com> Hi Brian, Thanks for the prompt review! On 7/11/2017 6:41 PM, Brian Goetz wrote: > The specs are fine. But a reader who doesn't know what a nest is will never figure it out from the docs. Would be good if nest host had an API note that explained how classes become nestmantes. True - though I would prefer it if there were some definitive source we could refer to that explains this rather than putting an ad-hoc definition into the API docs. Perhaps we need something in the JLS (as API docs tend not to reference the JVMS). ?? Otherwise ... "A nest is a set of classes (nest mates) that form an access control context in which each class has access to the private members of the other classes in the nest. The set of classes consisting of a top-level class plus all of its nested classes, is an example of a nest. The nest host is the class designated to hold the list of classes that make up the nest, and to which each of the other nest mates refer - a top-level class is always a nest host." But it gets messy if you then have to explain that unless compiled for nest mates, every class is considered its own nest and nest host. Thanks, David > Sent from my MacBook Wheel > >> On Nov 7, 2017, at 7:26 AM, David Holmes wrote: >> >> For comment: >> >> http://cr.openjdk.java.net/~dholmes/8188075/webrev/raw_files/new/java/lang/Class.html#getNestHost() >> >> Three functions added: >> - getNestHost >> - isNestmateOf >> - getNestMembers >> >> The first two never throw exceptions related to nest hosts or nest membership. The third does. >> >> Thanks, >> David > From brian.goetz at oracle.com Tue Nov 7 12:25:06 2017 From: brian.goetz at oracle.com (Brian Goetz) Date: Tue, 7 Nov 2017 13:25:06 +0100 Subject: [Nestmates] Draft core reflection API In-Reply-To: <0113856c-245d-2048-0949-673da515a7bc@oracle.com> References: <9f36321d-ab54-9a61-9d15-ef11c6156f2b@oracle.com> <7EC16151-1307-447E-891F-C6944CA38F64@oracle.com> <0113856c-245d-2048-0949-673da515a7bc@oracle.com> Message-ID: That's a good description of what a nest mate is. What could be added is a reference the the jvms section that defines it, and a comment that says it is up to the source language compiler to decide what classes are nest mates , and that java puts inner classes in the nest of their enclosing class (w jlink ref). This could be moved to a section in the package javadoc and referenced from the Class doc. Sent from my MacBook Wheel > On Nov 7, 2017, at 11:31 AM, David Holmes wrote: > > Hi Brian, > > Thanks for the prompt review! > >> On 7/11/2017 6:41 PM, Brian Goetz wrote: >> The specs are fine. But a reader who doesn't know what a nest is will never figure it out from the docs. Would be good if nest host had an API note that explained how classes become nestmantes. > > True - though I would prefer it if there were some definitive source we could refer to that explains this rather than putting an ad-hoc definition into the API docs. Perhaps we need something in the JLS (as API docs tend not to reference the JVMS). ?? > > Otherwise ... > > "A nest is a set of classes (nest mates) that form an access control context in which each class has access to the private members of the other classes in the nest. The set of classes consisting of a top-level class plus all of its nested classes, is an example of a nest. The nest host is the class designated to hold the list of classes that make up the nest, and to which each of the other nest mates refer - a top-level class is always a nest host." > > But it gets messy if you then have to explain that unless compiled for nest mates, every class is considered its own nest and nest host. > > Thanks, > David > > > >> Sent from my MacBook Wheel >>> On Nov 7, 2017, at 7:26 AM, David Holmes wrote: >>> >>> For comment: >>> >>> http://cr.openjdk.java.net/~dholmes/8188075/webrev/raw_files/new/java/lang/Class.html#getNestHost() >>> >>> Three functions added: >>> - getNestHost >>> - isNestmateOf >>> - getNestMembers >>> >>> The first two never throw exceptions related to nest hosts or nest membership. The third does. >>> >>> Thanks, >>> David From michael.rasmussen at zeroturnaround.com Tue Nov 7 14:03:21 2017 From: michael.rasmussen at zeroturnaround.com (Michael Rasmussen) Date: Tue, 7 Nov 2017 16:03:21 +0200 Subject: [Nestmates] Draft core reflection API In-Reply-To: References: <9f36321d-ab54-9a61-9d15-ef11c6156f2b@oracle.com> <7EC16151-1307-447E-891F-C6944CA38F64@oracle.com> <0113856c-245d-2048-0949-673da515a7bc@oracle.com> Message-ID: Just a stray observation (from an observer). Isn't using two different terms in the API bound to cause confusion for users? getNestMembers vs isNestmateOf Wouldn't it be more intuitive if they used the same term, getNestmates & isNestmateOf, or partly removed the term, something like getNestMembers & isNestedWith Kind regards Michael Rasmussen On 7 November 2017 at 14:25, Brian Goetz wrote: > That's a good description of what a nest mate is. What could be added is > a reference the the jvms section that defines it, and a comment that says > it is up to the source language compiler to decide what classes are nest > mates , and that java puts inner classes in the nest of their enclosing > class (w jlink ref). This could be moved to a section in the package > javadoc and referenced from the Class doc. > > Sent from my MacBook Wheel > > > On Nov 7, 2017, at 11:31 AM, David Holmes > wrote: > > > > Hi Brian, > > > > Thanks for the prompt review! > > > >> On 7/11/2017 6:41 PM, Brian Goetz wrote: > >> The specs are fine. But a reader who doesn't know what a nest is will > never figure it out from the docs. Would be good if nest host had an API > note that explained how classes become nestmantes. > > > > True - though I would prefer it if there were some definitive source we > could refer to that explains this rather than putting an ad-hoc definition > into the API docs. Perhaps we need something in the JLS (as API docs tend > not to reference the JVMS). ?? > > > > Otherwise ... > > > > "A nest is a set of classes (nest mates) that form an access control > context in which each class has access to the private members of the other > classes in the nest. The set of classes consisting of a top-level class > plus all of its nested classes, is an example of a nest. The nest host is > the class designated to hold the list of classes that make up the nest, and > to which each of the other nest mates refer - a top-level class is always a > nest host." > > > > But it gets messy if you then have to explain that unless compiled for > nest mates, every class is considered its own nest and nest host. > > > > Thanks, > > David > > > > > > > >> Sent from my MacBook Wheel > >>> On Nov 7, 2017, at 7:26 AM, David Holmes > wrote: > >>> > >>> For comment: > >>> > >>> http://cr.openjdk.java.net/~dholmes/8188075/webrev/raw_ > files/new/java/lang/Class.html#getNestHost() > >>> > >>> Three functions added: > >>> - getNestHost > >>> - isNestmateOf > >>> - getNestMembers > >>> > >>> The first two never throw exceptions related to nest hosts or nest > membership. The third does. > >>> > >>> Thanks, > >>> David > > From david.holmes at oracle.com Tue Nov 7 22:09:16 2017 From: david.holmes at oracle.com (David Holmes) Date: Wed, 8 Nov 2017 08:09:16 +1000 Subject: [Nestmates] Draft core reflection API In-Reply-To: References: <9f36321d-ab54-9a61-9d15-ef11c6156f2b@oracle.com> <7EC16151-1307-447E-891F-C6944CA38F64@oracle.com> <0113856c-245d-2048-0949-673da515a7bc@oracle.com> Message-ID: <8db1e467-979c-1d1d-b50d-6bd2c8ea0cb2@oracle.com> Hi Michael, On 8/11/2017 12:03 AM, Michael Rasmussen wrote: > Just a stray observation (from an observer). > > Isn't using two different terms in the API bound to cause confusion for > users? > getNestMembers vs isNestmateOf Maybe it's just familiarity but I don't have an issue with saying that two members of the same nest are nest mates (like having a ships crew where two members are ship-mates; or being a member of a team and referring to team-mates :) ). Whether I then refer to "members" or "mates" depends on the context - the set as a whole relates to "members", while relationships between members is "mates". Happy to hear further opinions though. Thanks, David > Wouldn't it be more intuitive if they used the same term, getNestmates & > isNestmateOf, or partly removed the term, something like getNestMembers > & isNestedWith > > Kind regards > Michael Rasmussen > > > On 7 November 2017 at 14:25, Brian Goetz > wrote: > > That's a good description of what a nest mate is.? What could be > added is a reference the the jvms section that defines it, and a > comment that says it is up to the source language compiler to decide > what classes are nest mates , and that java puts inner classes in > the nest of their enclosing class (w jlink ref).? This could be > moved to a section in the package javadoc and referenced from the > Class doc. > > Sent from my MacBook Wheel > > > On Nov 7, 2017, at 11:31 AM, David Holmes > > wrote: > > > > Hi Brian, > > > > Thanks for the prompt review! > > > >> On 7/11/2017 6:41 PM, Brian Goetz wrote: > >> The specs are fine. But a reader who doesn't know what a nest is > will never figure it out from the docs.? Would be good if nest host > had an API note that explained how classes become nestmantes. > > > > True - though I would prefer it if there were some definitive > source we could refer to that explains this rather than putting an > ad-hoc definition into the API docs. Perhaps we need something in > the JLS (as API docs tend not to reference the JVMS). ?? > > > > Otherwise ... > > > > "A nest is a set of classes (nest mates) that form an access > control context in which each class has access to the private > members of the other classes in the nest. The set of classes > consisting of a top-level class plus all of its nested classes, is > an example of a nest. The nest host is the class designated to hold > the list of classes that make up the nest, and to which each of the > other nest mates refer - a top-level class is always a nest host." > > > > But it gets messy if you then have to explain that unless > compiled for nest mates, every class is considered its own nest and > nest host. > > > > Thanks, > > David > > > > > > > >> Sent from my MacBook Wheel > >>> On Nov 7, 2017, at 7:26 AM, David Holmes > > wrote: > >>> > >>> For comment: > >>> > >>> > http://cr.openjdk.java.net/~dholmes/8188075/webrev/raw_files/new/java/lang/Class.html#getNestHost() > > >>> > >>> Three functions added: > >>> - getNestHost > >>> - isNestmateOf > >>> - getNestMembers > >>> > >>> The first two never throw exceptions related to nest hosts or > nest membership. The third does. > >>> > >>> Thanks, > >>> David > > From daniel.smith at oracle.com Tue Nov 7 23:21:12 2017 From: daniel.smith at oracle.com (Dan Smith) Date: Tue, 7 Nov 2017 16:21:12 -0700 Subject: [Nestmates] Draft core reflection API In-Reply-To: <0113856c-245d-2048-0949-673da515a7bc@oracle.com> References: <9f36321d-ab54-9a61-9d15-ef11c6156f2b@oracle.com> <7EC16151-1307-447E-891F-C6944CA38F64@oracle.com> <0113856c-245d-2048-0949-673da515a7bc@oracle.com> Message-ID: <3BE8FCA8-E36D-448A-8C3C-F388C56C79E6@oracle.com> I would be careful to match the behavior of 'isNestmateOf' to JVMS 5.4.4, rather than define it in terms of 'getNestHost'. In particular, 'c.isNestmateOf(c)' shouldn't need to perform any class loading. For other cases, if class loading errors occur, is the proposal to swallow them and return 'false'? That seems okay, I guess, but is a different conclusion than what we came up with in JVMS. The other operations are not directly defined by JVMS and so have more flexibility to decide for themselves what is sensible behavior. > On Nov 7, 2017, at 3:31 AM, David Holmes wrote: > > I would prefer it if there were some definitive source we could refer to that explains this rather than putting an ad-hoc definition into the API docs. Perhaps we need something in the JLS (as API docs tend not to reference the JVMS). ?? I don't think there's any good reason for this. The reflection API deals in class file artifacts, so naturally should sometimes refer to their specification. It also deals in a Java-language-based view of class file artifacts, and in those cases it makes sense to refer to JLS. ?Dan From david.holmes at oracle.com Wed Nov 8 00:19:03 2017 From: david.holmes at oracle.com (David Holmes) Date: Wed, 8 Nov 2017 10:19:03 +1000 Subject: [Nestmates] Draft core reflection API In-Reply-To: <3BE8FCA8-E36D-448A-8C3C-F388C56C79E6@oracle.com> References: <9f36321d-ab54-9a61-9d15-ef11c6156f2b@oracle.com> <7EC16151-1307-447E-891F-C6944CA38F64@oracle.com> <0113856c-245d-2048-0949-673da515a7bc@oracle.com> <3BE8FCA8-E36D-448A-8C3C-F388C56C79E6@oracle.com> Message-ID: <395d90ef-5976-c348-b8d6-af4307cd9c01@oracle.com> Hi Dan, Thanks for taking a look at this. On 8/11/2017 9:21 AM, Dan Smith wrote: > I would be careful to match the behavior of 'isNestmateOf' to JVMS 5.4.4, rather than define it in terms of 'getNestHost'. In particular, 'c.isNestmateOf(c)' shouldn't need to perform any class loading. I could short-circuit that case, but why special case this instead of just retrieving the nest host (which may need to be loaded)? This is not an access-check (for which we bail out very early for the same class) but a simple query, so the access-check process per JVMS 5.4.4 don't directly need to apply. > For other cases, if class loading errors occur, is the proposal to swallow them and return 'false'? That seems okay, I guess, but is a different conclusion than what we came up with in JVMS. This was recently discussed on this list under "nestmates spec open issues". http://mail.openjdk.java.net/pipermail/valhalla-spec-experts/2017-October/000386.html John prefers to minimize exceptions for the Java API. :) > The other operations are not directly defined by JVMS and so have more flexibility to decide for themselves what is sensible behavior. > >> On Nov 7, 2017, at 3:31 AM, David Holmes wrote: >> >> I would prefer it if there were some definitive source we could refer to that explains this rather than putting an ad-hoc definition into the API docs. Perhaps we need something in the JLS (as API docs tend not to reference the JVMS). ?? > > I don't think there's any good reason for this. The reflection API deals in class file artifacts, so naturally should sometimes refer to their specification. It also deals in a Java-language-based view of class file artifacts, and in those cases it makes sense to refer to JLS. Okay so no JLS reference but potentially a JVMS reference, and/or some more general class or package javadoc as Brian suggested. Thanks, David ----- > ?Dan > From john.r.rose at oracle.com Wed Nov 8 00:24:44 2017 From: john.r.rose at oracle.com (John Rose) Date: Tue, 7 Nov 2017 16:24:44 -0800 Subject: [Nestmates] Draft core reflection API In-Reply-To: <9f36321d-ab54-9a61-9d15-ef11c6156f2b@oracle.com> References: <9f36321d-ab54-9a61-9d15-ef11c6156f2b@oracle.com> Message-ID: <754F7CC3-D546-4964-9802-830A02AC0484@oracle.com> On Nov 6, 2017, at 11:26 PM, David Holmes wrote: > > For comment: > > http://cr.openjdk.java.net/~dholmes/8188075/webrev/raw_files/new/java/lang/Class.html#getNestHost() > Looks good. A very small comment: Two links with text "nest host" appear to be in code font, from a {@link } construct. It should be {@linkplain } instead. From john.r.rose at oracle.com Wed Nov 8 00:27:17 2017 From: john.r.rose at oracle.com (John Rose) Date: Tue, 7 Nov 2017 16:27:17 -0800 Subject: [Nestmates] Draft core reflection API In-Reply-To: <0113856c-245d-2048-0949-673da515a7bc@oracle.com> References: <9f36321d-ab54-9a61-9d15-ef11c6156f2b@oracle.com> <7EC16151-1307-447E-891F-C6944CA38F64@oracle.com> <0113856c-245d-2048-0949-673da515a7bc@oracle.com> Message-ID: <83C91142-51AE-4F00-B763-3ADF05B011AD@oracle.com> On Nov 7, 2017, at 2:31 AM, David Holmes wrote: > > a top-level class is always a nest host The ad hoc explanation you suggest is OK with me but this last statement is TMI. There's no physical reason that must be true; it is an arbitrary convention which will probably go away as we use nestmates for more complex purposes. From david.holmes at oracle.com Wed Nov 8 00:28:06 2017 From: david.holmes at oracle.com (David Holmes) Date: Wed, 8 Nov 2017 10:28:06 +1000 Subject: [Nestmates] Draft core reflection API In-Reply-To: <754F7CC3-D546-4964-9802-830A02AC0484@oracle.com> References: <9f36321d-ab54-9a61-9d15-ef11c6156f2b@oracle.com> <754F7CC3-D546-4964-9802-830A02AC0484@oracle.com> Message-ID: <5bd4b628-5892-5b69-46ac-6dc1f918896a@oracle.com> On 8/11/2017 10:24 AM, John Rose wrote: > On Nov 6, 2017, at 11:26 PM, David Holmes > wrote: >> >> For comment: >> >> http://cr.openjdk.java.net/~dholmes/8188075/webrev/raw_files/new/java/lang/Class.html#getNestHost() >> > > Looks good. ?A very small comment: ?Two links with text "nest host" > appear to be in code font, from a {@link } construct. ?It should be > {@linkplain } instead. Thanks John. Will fix. David From david.holmes at oracle.com Wed Nov 8 00:32:29 2017 From: david.holmes at oracle.com (David Holmes) Date: Wed, 8 Nov 2017 10:32:29 +1000 Subject: [Nestmates] Draft core reflection API In-Reply-To: <83C91142-51AE-4F00-B763-3ADF05B011AD@oracle.com> References: <9f36321d-ab54-9a61-9d15-ef11c6156f2b@oracle.com> <7EC16151-1307-447E-891F-C6944CA38F64@oracle.com> <0113856c-245d-2048-0949-673da515a7bc@oracle.com> <83C91142-51AE-4F00-B763-3ADF05B011AD@oracle.com> Message-ID: On 8/11/2017 10:27 AM, John Rose wrote: > On Nov 7, 2017, at 2:31 AM, David Holmes > wrote: >> >> a top-level class is always a nest host > > The ad hoc explanation you suggest is OK with me but > this last statement is TMI. ?There's no physical reason > that must be true; it is an arbitrary convention which > will probably go away as we use nestmates for more > complex purposes. Ok. Thanks, David From karen.kinnear at oracle.com Wed Nov 8 20:12:36 2017 From: karen.kinnear at oracle.com (Karen Kinnear) Date: Wed, 8 Nov 2017 15:12:36 -0500 Subject: Valhalla EG minutes Nov 8 2017 Message-ID: <76FA28F5-8E7A-46BA-9090-95B77AED2568@oracle.com> attendees: Remi, Dan H, John, Frederic, Lois, Mr Simms, Karen AI: Dan H/Karen - meet offline to discuss expected behavior today due to JVMS overriding rules AI: all - feedback on Nestmate Reflection API email thread I. Nestmates specifications 1. JVMS overriding rule checking: - Dan H reported that IBM has different results from the tests Oracle provided. Will follow up to understand differences 2. Reflection proposed APIs: http://cr.openjdk.java.net/~dholmes/8188075/webrev/raw_files/new/java/lang/Class.html#getNestHost() Summary: isNestMateOf: returns true if succeeds (even for self), false if any error getNestHost: returns Host if succeeds, otherwise returns ?this? if none or resolution error getNestMembers: gets statically defied members, including host first, will throw LinkageError if any fail to resolve Works for Remi. Others - feedback on email thread please 3. JVMTI for Nestmates Karen proposed concern that today?s JVMTI specification which allows changing attributes could cause problems if we were to decrease access. Remi pointed out that it could also cause problems if we were to increase access. Propose: modify JVMTI rdedefineclasses restrictions to not allow changing NestHost or NestMembers attributes. Editor?s note: Dan S is working on a minor update to the NestMates JVMS II. Constant Dynamic 1. valid REF_kind for BootstrapMethod specification change in 4.7.23: JDK 9 JVMS: Accordingly, the reference_kind item of the CONSTANT_MethodHandle_info structure should have the value 6 or 8 (?5.4.3.5 ) Condy JVMS: The method handle typically has reference_kind REF_invokeStatic, pointing to a static method designed specifically to act as a bootstrap method. Offline Paul Sandoz pointed out that this is non-normative text so not validated. The question was what values are valid: answer: REF_invokeStatic, REF_NewInvokeSpecial and Remi came up with a REF_InvokeVirtual if you had a method in the Lookup class that took a String and MethodType, i.e. the first static argument would therefore be a Lookup. Thanks! Testing otherwise throws Summary - ok to leave the specification as is, we just wanted to make sure we covered all bases in our understanding and therefore testing. 2. circularity handling: JVMS for Condy: 4.4.13 The referenced entry in the bootstrap_methods table must not refer, directly or indirectly, to this CONSTANT_Dynamic_info structure. JVMS for Condy: 5.4.3.6 As a special condition, if resolution is attempted of a dynamically-computed constant which is already in this process of resolution in the same thread, the attempted re-resolution may terminate immediately with a BootstrapMethodError, and it must eventually throw an exception instead of looping infinitely. Syntactic - within the same class file circular references e.g. ConDy -> BSM with a static arg of the same ConDy Semantic - this includes both circularity due to BSM code or due to cross class file link Agreed semantic circularity could only be detected at runtime. John proposed alternatives for syntactic checking: 0. do nothing - no checks at load/verify time 1. check loops at class load/verify time 2. precedence order on BSM records - BSM can not have reference to static argument which is a constantDynamic which refers to an earlier BSM Discussion: Remi prefers option 0. ASM builds constants based on order of instructions. If you add instructions it changes ordering. Dan H: make sure we support use cases in which we add changes to code and constant pool. John: make sure this works with Pack200 [ed note: make sure this will work in future with specialization and potential sharing of partial class files] Dan H: we have to detect the dynamic case anyway, and that is a superset check for the static case Karen: Agreed - from a runtime perspective there is no benefit in an additional check. Question is - is this easier on tools? John: Is there a concern that tools which process a class file with a recursive syntax and re-write could run into recursion doing the rewrite? Remi: ok if you read garbage and get a StackOverFlow when rewriting consensus of the meeting - option 0. Wording of 5.4.3.6: John - worded to allow StackOverFlow today. Future explicit check could throw BootStrapMethodError Dan H - SOE is natural - what is the benefit of the explicit overhead to generate BSME? Is there a risk of forbidding cases people want to use? John: in far future - may want to remove cycle constraint and have tools allow it. Note that BsCI with lazy resolution is tolerant of cycles that are not executed Summarize: - leave 5.4.3.6 which describes throwing exception on cycle. VM throw SOE - no significant value in early detection - tool vendors can refuse cycles as they see fit [ed. note - would this remove the line from 4.4.13? Or is that non-normative text so we are disallowing without static enforcement?] 3. Renaming of CONSTANT_InvokeDynamic to CONSTANT_DynamicCallSite John personally recants this proposal. Special reason - in future may want ldc of CONSTANT_InvokeDynamic and want the BSM return to convert a CallSite to a MethodHandle for ldc, or a MethodHandle to a ConstantCallSite for an indy One use case would be for static types in the constant pool, i.e. lazily initialized ConstantValue static finals Note ldc of CONSTANT_InvokeDynamic would SHARE the result III. Class File Versions for experimentation John reminded us of the model he has proposed of exploring a private use area for major versions for experimentation. Proposed 0XE000 and above. Folks were ok with the proposed range. Remi pointed out the need for minor versions - so for instance you could test with MVT and Condy by setting a single major version and multiple minor versions. JVMS will add a paragraph that allows this - such that standard versions will not accept these classifies. In practice a flag could be added to enable an experimental version. No registration planned, squatters rights. thanks, Karen From forax at univ-mlv.fr Wed Nov 8 20:47:04 2017 From: forax at univ-mlv.fr (Remi Forax) Date: Wed, 8 Nov 2017 21:47:04 +0100 (CET) Subject: [Nestmates] Draft core reflection API In-Reply-To: <9f36321d-ab54-9a61-9d15-ef11c6156f2b@oracle.com> References: <9f36321d-ab54-9a61-9d15-ef11c6156f2b@oracle.com> Message-ID: <955153318.1556033.1510174024463.JavaMail.zimbra@u-pem.fr> Apart for the corrections already mentioned, this is perfect. R?mi ----- Mail original ----- > De: "David Holmes" > ?: "valhalla-spec-experts" > Envoy?: Mardi 7 Novembre 2017 08:26:47 > Objet: [Nestmates] Draft core reflection API > For comment: > > http://cr.openjdk.java.net/~dholmes/8188075/webrev/raw_files/new/java/lang/Class.html#getNestHost() > > Three functions added: > - getNestHost > - isNestmateOf > - getNestMembers > > The first two never throw exceptions related to nest hosts or nest > membership. The third does. > > Thanks, > David From david.holmes at oracle.com Wed Nov 8 22:26:51 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 9 Nov 2017 08:26:51 +1000 Subject: [Nestmates] Draft core reflection API In-Reply-To: <955153318.1556033.1510174024463.JavaMail.zimbra@u-pem.fr> References: <9f36321d-ab54-9a61-9d15-ef11c6156f2b@oracle.com> <955153318.1556033.1510174024463.JavaMail.zimbra@u-pem.fr> Message-ID: <5e4eef00-cbd2-5eb7-e4b1-ac68976d5ee2@oracle.com> Thanks for the review Remi. David On 9/11/2017 6:47 AM, Remi Forax wrote: > Apart for the corrections already mentioned, > this is perfect. > > R?mi > > ----- Mail original ----- >> De: "David Holmes" >> ?: "valhalla-spec-experts" >> Envoy?: Mardi 7 Novembre 2017 08:26:47 >> Objet: [Nestmates] Draft core reflection API > >> For comment: >> >> http://cr.openjdk.java.net/~dholmes/8188075/webrev/raw_files/new/java/lang/Class.html#getNestHost() >> >> Three functions added: >> - getNestHost >> - isNestmateOf >> - getNestMembers >> >> The first two never throw exceptions related to nest hosts or nest >> membership. The third does. >> >> Thanks, >> David From david.holmes at oracle.com Thu Nov 9 07:25:11 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 9 Nov 2017 17:25:11 +1000 Subject: [Nestmates] Draft core reflection API In-Reply-To: References: <9f36321d-ab54-9a61-9d15-ef11c6156f2b@oracle.com> <7EC16151-1307-447E-891F-C6944CA38F64@oracle.com> <0113856c-245d-2048-0949-673da515a7bc@oracle.com> Message-ID: On 7/11/2017 10:25 PM, Brian Goetz wrote: > That's a good description of what a nest mate is. What could be added is a reference the the jvms section that defines it, and a comment that says it is up to the source language compiler to decide what classes are nest mates , and that java puts inner classes in the nest of their enclosing class (w jlink ref). This could be moved to a section in the package javadoc and referenced from the Class doc. Okay I've incorporated the suggested changes and updated in place: http://cr.openjdk.java.net/~dholmes/8188075/webrev/raw_files/new/java/lang/Class.html#getNestHost() I hope it isn't TMI for John. I looked at the package javadoc for java.lang, and java.lang.reflect, and I looked at the class javadoc for Class, and I couldn't see anywhere I could put this that would not look completely out of place. So I've left it as extended commentary (apiNote) in getNestHost(). Thanks, David > Sent from my MacBook Wheel > >> On Nov 7, 2017, at 11:31 AM, David Holmes wrote: >> >> Hi Brian, >> >> Thanks for the prompt review! >> >>> On 7/11/2017 6:41 PM, Brian Goetz wrote: >>> The specs are fine. But a reader who doesn't know what a nest is will never figure it out from the docs. Would be good if nest host had an API note that explained how classes become nestmantes. >> >> True - though I would prefer it if there were some definitive source we could refer to that explains this rather than putting an ad-hoc definition into the API docs. Perhaps we need something in the JLS (as API docs tend not to reference the JVMS). ?? >> >> Otherwise ... >> >> "A nest is a set of classes (nest mates) that form an access control context in which each class has access to the private members of the other classes in the nest. The set of classes consisting of a top-level class plus all of its nested classes, is an example of a nest. The nest host is the class designated to hold the list of classes that make up the nest, and to which each of the other nest mates refer - a top-level class is always a nest host." >> >> But it gets messy if you then have to explain that unless compiled for nest mates, every class is considered its own nest and nest host. >> >> Thanks, >> David >> >> >> >>> Sent from my MacBook Wheel >>>> On Nov 7, 2017, at 7:26 AM, David Holmes wrote: >>>> >>>> For comment: >>>> >>>> http://cr.openjdk.java.net/~dholmes/8188075/webrev/raw_files/new/java/lang/Class.html#getNestHost() >>>> >>>> Three functions added: >>>> - getNestHost >>>> - isNestmateOf >>>> - getNestMembers >>>> >>>> The first two never throw exceptions related to nest hosts or nest membership. The third does. >>>> >>>> Thanks, >>>> David > From brian.goetz at oracle.com Thu Nov 9 07:39:17 2017 From: brian.goetz at oracle.com (Brian Goetz) Date: Thu, 9 Nov 2017 08:39:17 +0100 Subject: [Nestmates] Draft core reflection API In-Reply-To: References: <9f36321d-ab54-9a61-9d15-ef11c6156f2b@oracle.com> <7EC16151-1307-447E-891F-C6944CA38F64@oracle.com> <0113856c-245d-2048-0949-673da515a7bc@oracle.com> Message-ID: <3A32DFA2-65CE-4EFF-B91C-1A05B027888F@oracle.com> looks good to me. Might even work in the word ?partition? or mention a class belongs to exactly one nest. Sent from my iPad > On Nov 9, 2017, at 8:25 AM, David Holmes wrote: > >> On 7/11/2017 10:25 PM, Brian Goetz wrote: >> That's a good description of what a nest mate is. What could be added is a reference the the jvms section that defines it, and a comment that says it is up to the source language compiler to decide what classes are nest mates , and that java puts inner classes in the nest of their enclosing class (w jlink ref). This could be moved to a section in the package javadoc and referenced from the Class doc. > > Okay I've incorporated the suggested changes and updated in place: > > http://cr.openjdk.java.net/~dholmes/8188075/webrev/raw_files/new/java/lang/Class.html#getNestHost() > > I hope it isn't TMI for John. > > I looked at the package javadoc for java.lang, and java.lang.reflect, and I looked at the class javadoc for Class, and I couldn't see anywhere I could put this that would not look completely out of place. So I've left it as extended commentary (apiNote) in getNestHost(). > > Thanks, > David > > >> Sent from my MacBook Wheel >>> On Nov 7, 2017, at 11:31 AM, David Holmes wrote: >>> >>> Hi Brian, >>> >>> Thanks for the prompt review! >>> >>>> On 7/11/2017 6:41 PM, Brian Goetz wrote: >>>> The specs are fine. But a reader who doesn't know what a nest is will never figure it out from the docs. Would be good if nest host had an API note that explained how classes become nestmantes. >>> >>> True - though I would prefer it if there were some definitive source we could refer to that explains this rather than putting an ad-hoc definition into the API docs. Perhaps we need something in the JLS (as API docs tend not to reference the JVMS). ?? >>> >>> Otherwise ... >>> >>> "A nest is a set of classes (nest mates) that form an access control context in which each class has access to the private members of the other classes in the nest. The set of classes consisting of a top-level class plus all of its nested classes, is an example of a nest. The nest host is the class designated to hold the list of classes that make up the nest, and to which each of the other nest mates refer - a top-level class is always a nest host." >>> >>> But it gets messy if you then have to explain that unless compiled for nest mates, every class is considered its own nest and nest host. >>> >>> Thanks, >>> David >>> >>> >>> >>>> Sent from my MacBook Wheel >>>>> On Nov 7, 2017, at 7:26 AM, David Holmes wrote: >>>>> >>>>> For comment: >>>>> >>>>> http://cr.openjdk.java.net/~dholmes/8188075/webrev/raw_files/new/java/lang/Class.html#getNestHost() >>>>> >>>>> Three functions added: >>>>> - getNestHost >>>>> - isNestmateOf >>>>> - getNestMembers >>>>> >>>>> The first two never throw exceptions related to nest hosts or nest membership. The third does. >>>>> >>>>> Thanks, >>>>> David From david.holmes at oracle.com Thu Nov 9 08:01:24 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 9 Nov 2017 18:01:24 +1000 Subject: [Nestmates] Draft core reflection API In-Reply-To: <3A32DFA2-65CE-4EFF-B91C-1A05B027888F@oracle.com> References: <9f36321d-ab54-9a61-9d15-ef11c6156f2b@oracle.com> <7EC16151-1307-447E-891F-C6944CA38F64@oracle.com> <0113856c-245d-2048-0949-673da515a7bc@oracle.com> <3A32DFA2-65CE-4EFF-B91C-1A05B027888F@oracle.com> Message-ID: On 9/11/2017 5:39 PM, Brian Goetz wrote: > looks good to me. Might even work in the word ?partition? or mention a class belongs to exactly one nest. Added. "A class or interface that is not explicitly a member of a nest, is a member of the nest consisting only of itself, and is the nest host. _ Every class and interface is a member of exactly one nest. _ " Thanks, David > > Sent from my iPad > >> On Nov 9, 2017, at 8:25 AM, David Holmes wrote: >> >>> On 7/11/2017 10:25 PM, Brian Goetz wrote: >>> That's a good description of what a nest mate is. What could be added is a reference the the jvms section that defines it, and a comment that says it is up to the source language compiler to decide what classes are nest mates , and that java puts inner classes in the nest of their enclosing class (w jlink ref). This could be moved to a section in the package javadoc and referenced from the Class doc. >> >> Okay I've incorporated the suggested changes and updated in place: >> >> http://cr.openjdk.java.net/~dholmes/8188075/webrev/raw_files/new/java/lang/Class.html#getNestHost() >> >> I hope it isn't TMI for John. >> >> I looked at the package javadoc for java.lang, and java.lang.reflect, and I looked at the class javadoc for Class, and I couldn't see anywhere I could put this that would not look completely out of place. So I've left it as extended commentary (apiNote) in getNestHost(). >> >> Thanks, >> David >> >> >>> Sent from my MacBook Wheel >>>> On Nov 7, 2017, at 11:31 AM, David Holmes wrote: >>>> >>>> Hi Brian, >>>> >>>> Thanks for the prompt review! >>>> >>>>> On 7/11/2017 6:41 PM, Brian Goetz wrote: >>>>> The specs are fine. But a reader who doesn't know what a nest is will never figure it out from the docs. Would be good if nest host had an API note that explained how classes become nestmantes. >>>> >>>> True - though I would prefer it if there were some definitive source we could refer to that explains this rather than putting an ad-hoc definition into the API docs. Perhaps we need something in the JLS (as API docs tend not to reference the JVMS). ?? >>>> >>>> Otherwise ... >>>> >>>> "A nest is a set of classes (nest mates) that form an access control context in which each class has access to the private members of the other classes in the nest. The set of classes consisting of a top-level class plus all of its nested classes, is an example of a nest. The nest host is the class designated to hold the list of classes that make up the nest, and to which each of the other nest mates refer - a top-level class is always a nest host." >>>> >>>> But it gets messy if you then have to explain that unless compiled for nest mates, every class is considered its own nest and nest host. >>>> >>>> Thanks, >>>> David >>>> >>>> >>>> >>>>> Sent from my MacBook Wheel >>>>>> On Nov 7, 2017, at 7:26 AM, David Holmes wrote: >>>>>> >>>>>> For comment: >>>>>> >>>>>> http://cr.openjdk.java.net/~dholmes/8188075/webrev/raw_files/new/java/lang/Class.html#getNestHost() >>>>>> >>>>>> Three functions added: >>>>>> - getNestHost >>>>>> - isNestmateOf >>>>>> - getNestMembers >>>>>> >>>>>> The first two never throw exceptions related to nest hosts or nest membership. The third does. >>>>>> >>>>>> Thanks, >>>>>> David > From john.r.rose at oracle.com Thu Nov 9 08:08:11 2017 From: john.r.rose at oracle.com (John Rose) Date: Thu, 9 Nov 2017 00:08:11 -0800 Subject: [Nestmates] Draft core reflection API In-Reply-To: References: <9f36321d-ab54-9a61-9d15-ef11c6156f2b@oracle.com> <7EC16151-1307-447E-891F-C6944CA38F64@oracle.com> <0113856c-245d-2048-0949-673da515a7bc@oracle.com> Message-ID: > On Nov 8, 2017, at 11:25 PM, David Holmes wrote: > > http://cr.openjdk.java.net/~dholmes/8188075/webrev/raw_files/new/java/lang/Class.html#getNestHost() > > I hope it isn't TMI for John. > Looks good to me. I don?t want to rule out other uses of nest mates, and that language is loose enough not to do so. Example: multiple top-level classes in the same source file *might* someday be put into a common nest. From david.holmes at oracle.com Thu Nov 9 10:22:37 2017 From: david.holmes at oracle.com (David Holmes) Date: Thu, 9 Nov 2017 20:22:37 +1000 Subject: [Nestmates] Draft core reflection API In-Reply-To: References: <9f36321d-ab54-9a61-9d15-ef11c6156f2b@oracle.com> <7EC16151-1307-447E-891F-C6944CA38F64@oracle.com> <0113856c-245d-2048-0949-673da515a7bc@oracle.com> Message-ID: Thanks John! David On 9/11/2017 6:08 PM, John Rose wrote: > > >> On Nov 8, 2017, at 11:25 PM, David Holmes wrote: >> >> http://cr.openjdk.java.net/~dholmes/8188075/webrev/raw_files/new/java/lang/Class.html#getNestHost() >> >> I hope it isn't TMI for John. >> > > Looks good to me. I don?t want to rule out other uses of nest mates, and that language is loose enough not to do so. > > Example: multiple top-level classes in the same source file *might* someday be put into a common nest. > From karen.kinnear at oracle.com Fri Nov 10 17:48:54 2017 From: karen.kinnear at oracle.com (Karen Kinnear) Date: Fri, 10 Nov 2017 12:48:54 -0500 Subject: Nestmates JVMS Overriding 5.4.5 request Message-ID: <24DD2E51-80CD-43F4-8E30-95043F016691@oracle.com> Dan Smith - this is a request for a change to one line of the Nestmates JVMS 5.4.5 Overriding. The rest of the wording for 5.4.5 is an improvement, so thank you. Dan Heidinga, Graham Chapman (IBM) and I met to discuss the JVM implementations of JVMS 5.4.5 Overriding, specifically to discuss the expectations relative to the bullet describing transitive overriding that was added for CFV 51 in JDK7. The JVMS 9 bullet says: mC overrides a method m' (m' distinct from mC and mA) such that m' overrides mA. The proposed NestMates bullet says: mC overrides a method m' (m' distinct from mC and mA) such that m' overrides mA mA is marked neither ACC_PUBLIC nor ACC_PROTECTED nor ACC_PRIVATE, and, where mC is declared in a class C and mA is declared in a class A, there exists a method mB declared in a class B, such that C is a subclass of B, B is a subclass of A, mC can override mB, and mB can override mA. From examining a number of test cases, we would like to request that the updated version be modified to remove the restriction on mA as package private. We need to perform what I call transitive overriding checks as long as mA is not ACC_PRIVATE. Dan H also suggested it would be very helpful to all of us if you could add some non-normative examples in the spec here. Examples that might be useful would be: 1. reason we added transitive overriding to start with - see postscript 2. example of why we need to do the transitive overriding even if mA is public/protected as well as package private: (test 841 from our internal test matrix for invoke virtual) > 841| A PUB a.B PP b.C PUB > > call A with B - B.m (B overrides A) > call A with C - C.m (C overrides A) > call B with C - B.m (C does not override B) 3. a fix we had to make in 2014: P1.A: public m() P2.B extends A, PP m() P1.C extends B, public m() P1.C.m() needs to override P1.A.m() and can not override P2.B.m() thanks, Karen p.s. extract from JVMS Clarifications and Addenda to 2nd edition (since the original link is gone): With classfile version 51, which is introduced in JDK7, the invokevirtual behavior will be fixed to match the clarifications in the JVMS which were made to handle the following test case: Description of the original problem from: Why the Definition of Method Override was Revised: package P1; class A { void m(){System.out.println("A.m");} public void callM(){ m();} } public class B extends A { protected void m() { System.out.println("B.m");} } package P2; class C extends P1.B { protected void m(){ System.out.println("C.m");} public static void main(String[] args) { C c = new C(); c.callM(); } Given the original definition of override, we find that the method m() declared in C overrides the method m() declared in B(), since it has the same name and signature and is accessible from class C. However, we a lso find that the method m() declared in C does not override the method m() declared in A(), since it is private to package P1 and therefore not accessible from class C. In this case, invoking callM() on an instance of C causes a call of m() where the target of the method invocation is an instance of C as well. However, this call is in the class P1.A, and is attempting to invoke a package private method of A. Since this method is not overridden by C, the code will not print "C.m" as expected. Instead, it prints "B.m". This is quite unintuitive, and also incompatible with the behavior of the classic VM. We believe that this interpretation, while literally accurate, is undesirable. It prevents a package private method from ever being overridden outside its package, even if a subclass within the package has increased the method's accessibility. Therefore invokevirtual was modified in JVMS Clarifications and Addenda to 2nd edition to: Let C be the class of objectref. The actual method to be invoked is selected by the following lookup procedure: If C contains a declaration for an instance method M with the same name and descriptor as the resolved method, and M overrides the resolved method, then M is the method to be invoked, and the lookup procedure terminates. Otherwise, if C has a superclass, this same lookup procedure is performed recursively using the direct superclass of C ; the method to be invoked is the result of the recursive invocation of this lookup procedure. Otherwise, an AbstractMethodError is raised. From the JVMS 2nd edition invokevirtual: Let C be the class of objectref. The actual method to be invoked is selected by the following lookup procedure: If C contains a declaration for an instance method with the same name and descriptor as the resolved method, and the resolved method is accessible from C, then this is the method to be invoked, and the lookup procedure terminates. Otherwise, if C has a superclass, this same lookup procedure is performed recursively using the direct superclass of C ; the method to be invoked is the result of the recursive invocation of this lookup procedure. Otherwise, an AbstractMethodError is raised. Note: the key difference here is the phrase: "the resolved method is accessible from C" which in the Clarifications and Addenda has been replaced with "M overrides the resolved method" From peter.levart at gmail.com Thu Nov 16 11:37:44 2017 From: peter.levart at gmail.com (Peter Levart) Date: Thu, 16 Nov 2017 12:37:44 +0100 Subject: [Nestmates] Add a core reflection API to get nestmate information Message-ID: <3309a72e-196a-6ea2-c744-bdd503b7cc0f@gmail.com> Hi, I saw the proposal for a core reflection API to get nestmate information and here are my thoughts about the decision to filter out class resolution errors in some methods, discussed below... On 11/08/2017 01:19 AM, David Holmes wrote: > Hi Dan, > > Thanks for taking a look at this. > > On 8/11/2017 9:21 AM, Dan Smith wrote: >> I would be careful to match the behavior of 'isNestmateOf' to JVMS >> 5.4.4, rather than define it in terms of 'getNestHost'. In >> particular, 'c.isNestmateOf(c)' shouldn't need to perform any class >> loading. > > I could short-circuit that case, but why special case this instead of > just retrieving the nest host (which may need to be loaded)? This is > not an access-check (for which we bail out very early for the same > class) but a simple query, so the access-check process per JVMS 5.4.4 > don't directly need to apply. > >> For other cases, if class loading errors occur, is the proposal to >> swallow them and return 'false'? That seems okay, I guess, but is a >> different conclusion than what we came up with in JVMS. > > This was recently discussed on this list under "nestmates spec open > issues". > > http://mail.openjdk.java.net/pipermail/valhalla-spec-experts/2017-October/000386.html > > > John prefers to minimize exceptions for the Java API. :) Suppose that this is basic reflection API to get nestmate information and that it will also be used for reflective access checks. For example in jdk.internal.reflect.Reflection#verifyMemberAccess, which is used from Method.invoke, Fileld.[get|set], Constructor.newInstance. The most appropriate method for this check is Class.isNestmateOf(Class). For example: ??? /** ???? * Verify access to a member and return {@code true} if it is granted. ???? * ???? * @param currentClass the class performing the access ???? * @param memberClass the declaring class of the member being accessed ???? * @param targetClass the class of target object if accessing instance ???? *??????????????????? field or method; ???? *??????????????????? or the declaring class if accessing constructor; ???? *??????????????????? or null if accessing static field or method ???? * @param modifiers the member's access modifiers ???? * @return {@code true} if access to member is granted ???? */ ??? public static boolean verifyMemberAccess(Class currentClass, ???????????????????????????????????????????? Class memberClass, ???????????????????????????????????????????? Class targetClass, ???????????????????????????????????????????? int modifiers) ??? { ??????? if (currentClass.isNestmateOf(memberClass)) { ??????????? // Always succeeds ??????????? return true; ??????? } ??? ??? ... If there is a nest-host class resolution error and isNestmateOf() returns false for two actual nest-mates that have lost their host, reflective access is going to throw IllegalAccessException(s) and the real cause of failure will be hidden. That doesn't help to diagnose the problem. If the reasoning behind the decision to hide resolution exceptions is the desire to mimic OLD behavior, it is not successful, because the OLD behavior allows private access between two nested classes when "top" class is missing while then NEW behavior disallows it. Enter symmetric scheme: instead of one nest-host with a list of nets-matest + N-1 nest-mates that reference just the nest-host, there would be N nest-mates with equal status where each lists all others. With symmetric scheme there would be no need to hide resolution errors that happen now when access checks are performed between two nest-mates where neither of them is a nest-host. If I remember correctly, the symmetric scheme was not considered, because it would be complicated to resolve inconsistencies and spoofing. Is it really that important to have a notion of nest-host? Is it really that important to be able to (consistently) enumerate members of nest? What problem are nest-mates solving? If the answer is just "access checking", then the notion of nest-host and nest-mates enumeration are only artificial implementation details. If those two concepts are really not necessary ingredients, then a symmetric scheme is a simple alternative for access checking. The only method needed is this: Class.isNestmateOf(Class c); A method that returns true if (this == c || ). Spoofing is not possible, because both parties must agree and resolution errors don't need to be hidden, because only resolution of involved parties is necessary. You still have reflexive and symmetric, but you loose transitive property: if A may access B.private and B may access C.private that doesn't imply that A may access C.private, because the nest-lists of classes may be: A: [B] B: [A, C] C: [B] But such cases may only arise as a consequence of faulty assembly. Compilation (even partial/incremental) can always verify the consistency of the whole nest and bail-out if needed. If you really needed to enumerate the members of the nest, the result would be dependent on the member that was given as an anchor point. A good-enough algorithm is this: x.getNestMembers() returns a list composed of x and all classes listed in x's nest-list that also have x listed in their nest-list(s). For above inconsistent example that would mean: A.class.getNestMembers() = [A, B]; B.class.getNestMembers() = [A, B, C]; C.class.getNestMembers() = [B, C]; But then again, if you are asking a trusted class for the list of nest members, no-one can spoof and add additional members to the list that was established when that trusted class was compiled. To wrap-up: symmetric scheme has pros and cons. The question is what is more important - the consistent view (no matter who you ask, you always get the same answer) but with potential resolution troubles when nest-host is not resolvable, or a semi-consistent view (depends on who you ask, but if you ask a trusted class, you get a trusted answer) but never need for resolving any additional classes. Regards, Peter From daniel.smith at oracle.com Thu Nov 16 15:40:22 2017 From: daniel.smith at oracle.com (Dan Smith) Date: Thu, 16 Nov 2017 10:40:22 -0500 Subject: Nestmates JVMS Overriding 5.4.5 request In-Reply-To: <24DD2E51-80CD-43F4-8E30-95043F016691@oracle.com> References: <24DD2E51-80CD-43F4-8E30-95043F016691@oracle.com> Message-ID: <4612B6B3-995C-47C7-87F4-A4B402919FC7@oracle.com> > On Nov 10, 2017, at 12:48 PM, Karen Kinnear wrote: > > Dan Smith - this is a request for a change to one line of the Nestmates JVMS 5.4.5 Overriding. > The rest of the wording for 5.4.5 is an improvement, so thank you. > > Dan Heidinga, Graham Chapman (IBM) and I met to discuss the JVM implementations of > JVMS 5.4.5 Overriding, specifically to discuss the expectations relative to the > bullet describing transitive overriding that was added for CFV 51 in JDK7. > > The JVMS 9 bullet says: > mC overrides a method m' (m' distinct from mC and mA) such that m' overrides mA. > > The proposed NestMates bullet says: > mC overrides a method m' (m' distinct from mC and mA) such that m' overrides mA mA is marked neither ACC_PUBLIC nor ACC_PROTECTED nor ACC_PRIVATE, and, where mC is declared in a class C and mA is declared in a class A, there exists a method mB declared in a class B, such that C is a subclass of B, B is a subclass of A, mC can override mB, and mB can override mA. > From examining a number of test cases, we would like to request that the updated version be modified to remove the restriction on mA as package private. We need to perform what I call transitive overriding checks as long as mA is not ACC_PRIVATE. The intent is not to change the meaning, but to clarify that the check only needs to be performed when mA is package-access. Notice that this appears in a list of or'ed conditions, and the first two are "mA is marked ACC_PUBLIC" and "mA is marked ACC_PROTECTED". So in those cases, the enclosing "One of the following is true" is already true. If, in fact, some sort of transitive analysis is needed, then the problem is deeper than just the access flags checked in this bullet. Could you supply an example where the current rules define unwanted/unimplemented behavior? > Dan H also suggested it would be very helpful to all of us if you could add some non-normative examples in the spec here. Examples that might be useful would be: Sure, some discussion about package access would be helpful. I can do so. ?Dan From david.holmes at oracle.com Thu Nov 16 21:52:51 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 17 Nov 2017 07:52:51 +1000 Subject: [Nestmates] Add a core reflection API to get nestmate information In-Reply-To: <3309a72e-196a-6ea2-c744-bdd503b7cc0f@gmail.com> References: <3309a72e-196a-6ea2-c744-bdd503b7cc0f@gmail.com> Message-ID: Hi Peter, On 16/11/2017 9:37 PM, Peter Levart wrote: > Hi, > > I saw the proposal for a core reflection API to get nestmate information > and here are my thoughts about the decision to filter out class > resolution errors in some methods, discussed below... > > On 11/08/2017 01:19 AM, David Holmes wrote: >> Hi Dan, >> >> Thanks for taking a look at this. >> >> On 8/11/2017 9:21 AM, Dan Smith wrote: >>> I would be careful to match the behavior of 'isNestmateOf' to JVMS >>> 5.4.4, rather than define it in terms of 'getNestHost'. In >>> particular, 'c.isNestmateOf(c)' shouldn't need to perform any class >>> loading. >> >> I could short-circuit that case, but why special case this instead of >> just retrieving the nest host (which may need to be loaded)? This is >> not an access-check (for which we bail out very early for the same >> class) but a simple query, so the access-check process per JVMS 5.4.4 >> don't directly need to apply. >> >>> For other cases, if class loading errors occur, is the proposal to >>> swallow them and return 'false'? That seems okay, I guess, but is a >>> different conclusion than what we came up with in JVMS. >> >> This was recently discussed on this list under "nestmates spec open >> issues". >> >> http://mail.openjdk.java.net/pipermail/valhalla-spec-experts/2017-October/000386.html >> >> >> John prefers to minimize exceptions for the Java API. :) > > Suppose that this is basic reflection API to get nestmate information > and that it will also be used for reflective access checks. For example > in jdk.internal.reflect.Reflection#verifyMemberAccess, which is used It isn't. The reflection API is purely for "external" use. Real access checks are performed in the same way as the VM access checks - as required - and will report the same exceptions in the same way. David ----- > from Method.invoke, Fileld.[get|set], Constructor.newInstance. The most > appropriate method for this check is Class.isNestmateOf(Class). For > example: > > ??? /** > ???? * Verify access to a member and return {@code true} if it is granted. > ???? * > ???? * @param currentClass the class performing the access > ???? * @param memberClass the declaring class of the member being accessed > ???? * @param targetClass the class of target object if accessing instance > ???? *??????????????????? field or method; > ???? *??????????????????? or the declaring class if accessing constructor; > ???? *??????????????????? or null if accessing static field or method > ???? * @param modifiers the member's access modifiers > ???? * @return {@code true} if access to member is granted > ???? */ > ??? public static boolean verifyMemberAccess(Class currentClass, > ???????????????????????????????????????????? Class memberClass, > ???????????????????????????????????????????? Class targetClass, > ???????????????????????????????????????????? int modifiers) > ??? { > ??????? if (currentClass.isNestmateOf(memberClass)) { > ??????????? // Always succeeds > ??????????? return true; > ??????? } > ??? ??? ... > > If there is a nest-host class resolution error and isNestmateOf() > returns false for two actual nest-mates that have lost their host, > reflective access is going to throw IllegalAccessException(s) and the > real cause of failure will be hidden. That doesn't help to diagnose the > problem. > > If the reasoning behind the decision to hide resolution exceptions is > the desire to mimic OLD behavior, it is not successful, because the OLD > behavior allows private access between two nested classes when "top" > class is missing while then NEW behavior disallows it. > > Enter symmetric scheme: instead of one nest-host with a list of > nets-matest + N-1 nest-mates that reference just the nest-host, there > would be N nest-mates with equal status where each lists all others. > With symmetric scheme there would be no need to hide resolution errors > that happen now when access checks are performed between two nest-mates > where neither of them is a nest-host. > > If I remember correctly, the symmetric scheme was not considered, > because it would be complicated to resolve inconsistencies and spoofing. > > Is it really that important to have a notion of nest-host? Is it really > that important to be able to (consistently) enumerate members of nest? > What problem are nest-mates solving? If the answer is just "access > checking", then the notion of nest-host and nest-mates enumeration are > only artificial implementation details. If those two concepts are really > not necessary ingredients, then a symmetric scheme is a simple > alternative for access checking. The only method needed is this: > > Class.isNestmateOf(Class c); > > A method that returns true if (this == c || in their lists of nest-mates>). > > Spoofing is not possible, because both parties must agree and resolution > errors don't need to be hidden, because only resolution of involved > parties is necessary. > > You still have reflexive and symmetric, but you loose transitive > property: if A may access B.private and B may access C.private that > doesn't imply that A may access C.private, because the nest-lists of > classes may be: > > A: [B] > B: [A, C] > C: [B] > > But such cases may only arise as a consequence of faulty assembly. > Compilation (even partial/incremental) can always verify the consistency > of the whole nest and bail-out if needed. > > If you really needed to enumerate the members of the nest, the result > would be dependent on the member that was given as an anchor point. A > good-enough algorithm is this: > > x.getNestMembers() returns a list composed of x and all classes listed > in x's nest-list that also have x listed in their nest-list(s). For > above inconsistent example that would mean: > > A.class.getNestMembers() = [A, B]; > B.class.getNestMembers() = [A, B, C]; > C.class.getNestMembers() = [B, C]; > > But then again, if you are asking a trusted class for the list of nest > members, no-one can spoof and add additional members to the list that > was established when that trusted class was compiled. > > To wrap-up: symmetric scheme has pros and cons. The question is what is > more important - the consistent view (no matter who you ask, you always > get the same answer) but with potential resolution troubles when > nest-host is not resolvable, or a semi-consistent view (depends on who > you ask, but if you ask a trusted class, you get a trusted answer) but > never need for resolving any additional classes. > > Regards, Peter > From peter.levart at gmail.com Fri Nov 17 09:45:48 2017 From: peter.levart at gmail.com (Peter Levart) Date: Fri, 17 Nov 2017 10:45:48 +0100 Subject: [Nestmates] Add a core reflection API to get nestmate information In-Reply-To: References: <3309a72e-196a-6ea2-c744-bdd503b7cc0f@gmail.com> Message-ID: <4b98c3d4-d40b-87da-702b-c20ca9ac7bf9@gmail.com> Hi David, On 11/16/2017 10:52 PM, David Holmes wrote: >>> John prefers to minimize exceptions for the Java API. :) >> >> Suppose that this is basic reflection API to get nestmate information >> and that it will also be used for reflective access checks. For >> example in jdk.internal.reflect.Reflection#verifyMemberAccess, which >> is used > > It isn't. The reflection API is purely for "external" use. Real access > checks are performed in the same way as the VM access checks - as > required - and will report the same exceptions in the same way. > > David Real access checks performed by bytecode(s), yes. But what about reflective access checks - those performed by Method.invoke, Field.[get|set], Constructor.newInstance? They will have to be revised some day to include the nestmates. What facility should those methods use to implement the checks (the guts of those checks is in jdk.internal.reflect.Reflection#verifyMemberAccess). Is the plan to have special internal native methods to facilitate that? I don't quite understand why should a Class.isNestmateOf(Class) behave any differently than real VM access check. Regards, Peter From david.holmes at oracle.com Fri Nov 17 10:44:54 2017 From: david.holmes at oracle.com (David Holmes) Date: Fri, 17 Nov 2017 20:44:54 +1000 Subject: [Nestmates] Add a core reflection API to get nestmate information In-Reply-To: <4b98c3d4-d40b-87da-702b-c20ca9ac7bf9@gmail.com> References: <3309a72e-196a-6ea2-c744-bdd503b7cc0f@gmail.com> <4b98c3d4-d40b-87da-702b-c20ca9ac7bf9@gmail.com> Message-ID: Hi Peter, On 17/11/2017 7:45 PM, Peter Levart wrote: > Hi David, > > On 11/16/2017 10:52 PM, David Holmes wrote: >>>> John prefers to minimize exceptions for the Java API. :) >>> >>> Suppose that this is basic reflection API to get nestmate information >>> and that it will also be used for reflective access checks. For >>> example in jdk.internal.reflect.Reflection#verifyMemberAccess, which >>> is used >> >> It isn't. The reflection API is purely for "external" use. Real access >> checks are performed in the same way as the VM access checks - as >> required - and will report the same exceptions in the same way. >> >> David > > Real access checks performed by bytecode(s), yes. But what about > reflective access checks - those performed by Method.invoke, > Field.[get|set], Constructor.newInstance? They will have to be revised > some day to include the nestmates. What facility should those methods > use to implement the checks (the guts of those checks is in > jdk.internal.reflect.Reflection#verifyMemberAccess). Yes and the real nestmate access check for use by invoke etc has already been implemented in there. http://hg.openjdk.java.net/valhalla/valhalla/file/df423aeaa8d1/src/java.base/share/classes/jdk/internal/reflect/Reflection.java > Is the plan to have special internal native methods to facilitate that? > I don't quite understand why should a Class.isNestmateOf(Class) behave > any differently than real VM access check. Purely because of the difference in exception handling. The view was that the reflection API should mostly "absorb" exceptions. David > Regards, Peter > From john.r.rose at oracle.com Sun Nov 19 19:47:34 2017 From: john.r.rose at oracle.com (John Rose) Date: Sun, 19 Nov 2017 11:47:34 -0800 Subject: class, type, instance, object, value In-Reply-To: <2FD8448E-CA9E-412C-9185-DF847F8FA666@oracle.com> References: <1F516BE7-30D2-47AF-9B83-3F8AA0A00FEE@oracle.com> <2FD8448E-CA9E-412C-9185-DF847F8FA666@oracle.com> Message-ID: <420F3521-1199-4CE0-935B-698415F3FC0B@oracle.com> On Jul 12, 2017, at 6:20 AM, Karen Kinnear wrote: > > John, > > Thank you for writing this up. Couple of questions/comments. > > - The class of a boxed value is the value class. (Thus each value class derives at least two types.) > This one confuses me. I am not sure what you are referring to. > 1) source: declaration - declared as a value class *If* the JLS user model of Valhalla values associates a derived "box" with a value, something like Integer is associated with int, *then* a value class derives both the plain value (the primary thing) and the box (a secondary thing). We are *not* certain that boxes are necessary in L-world. I think they are forced on us in U-world. This is an advantage of L-world, where L-types are at the top and Q-types just "sneak in". > 2) classfile - marked via ACC_VALUE, from could derive a value class and a boxed class Yes, I meant that. > 3) constant_pool entry - at which we want a way to distinguish the two types Yes. A common class with different kind/mode markings, Q vs. L vs. maybe U, but all the same class. > 4) vm internal metadata - again distinguish two types Implementor's choice on that one. > 5) java level use - at which I am assuming we will have a way to distinguish a value class and a boxed value OK, I thought that's what you were asking in 1. Key principle: One source construct = one class. (Inner classes are nested source constructs which allow several classes to be defined by one file; that's something the JVM can usually ignore, or model via nestmates.) > I believe Dan?s use of the term value class is for the unboxed value. There's one class, so if the box is defined from the same one central class, then I'm OK with Dan's term "value class" for the *one class* that defines *both* values and boxes. (I'm discouraging us from using the word "class" to refer to individual values of the object. We don't say "print me a string class", we say "print me a string value" or (maybe) "print me an string instance". I don't think that's how Dan was using his terms, but just in case?) > I think we want to explicitly call out boxed value. Indeed. But there is no need to have multiple classes to describe both values and their boxes; for any given value class it all flows from that one single value class. (I suspect we are in violent agreement here.) > - A "java.lang.Class" is usually a reference to class metadata, but not necessarily unique. > There's wiggle room here for class-for-the-box vs. class-for-the-value, and int.class. > We don't allow java.lang.Class to constrain other uses of the term "class". > - When clarity is at risk, we can say "class mirror" rather than just "Class". > I?m not sure what you mean here by ?not necessarily unique?. Perhaps that is because for me a java.lang.Class > is always a mirror, i.e. a reflection of the vm?s internal representation for the unique class. Not if we do what Brian calls "crasses". For example, if we make a "crass" of java.util.List, then that reflects a "something" (which I call "species") that combines the *class* java.util.List with the template argument "int". The result of that *should not* be called a "class", because that will cause confusion as folks scurry around trying to find the source code and/or classfile for that "something", and find only plain old java.util.List. So the term "class" should refer to "what comes out of a classfile", and/or an occurrence in Java code of a class/interface/enum definition. A "crass" is a java.lang.Class which refers not just to java.util.List (which is a perfectly normal class mirror like today's) but to the *species* List. If we do "crasses" it will be a carefully weighed decision to overload the legacy type java.lang.Class to mirror not just true Java classes (and interfaces and enums) but *also* it will mirror species which are derived from a template. In this framework, a *box* looks like another variation stamped out of the original class. If we don't do "crasses" (the "r" stands for "runtime quasi-class type", approximately), then we will need new kinds of mirrors, to mirror not the proper classes (and interfaces and enums) but to mirror the species which are created by specializing templates, and/or the boxes (if we do them) which represent the identity-laden variants of identity-free values. > Even if we use a name&mode to find a class-for-the-box vs. class-for-the-value, the java.lang.Class itself > would be unique, wouldn?t it? Hmmm? If we do both "crasses" and boxes then there could be a proper class mirror for a real value class like java.math.FloatComplex, and then a "crass" mirror for its identity-laden box, which is not Q-FloatComplex but L-FloatComplex. If we do L-world (see forthcoming message) we can avoid the box, but then we might template the complex numbers, and thus have many mirror "crasses" for the various species Complex, Complex, etc. > > thanks, > Karen > > >> On Jun 25, 2017, at 10:17 PM, John Rose > wrote: >> >> So, I'm writing more and more documentation that discusses >> objects and primitives while bringing values into the mix. >> >> What seems right to me is that we allow the terms "class", "type", >> and "instance" to symmetrically cover both legacy object types >> and new value types. We should continue to use the word "value" >> but be careful about distinguishing its overloadings, especially >> its role as an absolute noun vs. its role as an adjective. >> >> We should tolerate asymmetries that arise from the reference vs. >> value distinction, and from box types which arise from value classes. >> >> Summary: >> >> Classes = Object Classes <+> Value Classes >> Instances = Object Instances <+> Value Instances >> Object Instances = instances of Object Classes <+> boxes of Value Instances >> Reference Values = Object Instances <+> null >> Values (noun) = Reference Values <+> Value Instances <+> Primitives >> >> (?where <+> denotes disjoint union) >> >> Details: >> >> - A "class" is at root metadata describing a type or implementation. >> (It has API surface and/or implementation: super types, methods, fields, etc.) >> - An "instance" is derived from a class and/or conforms to that class's API. >> - An instance of an "object class" (or "object type") is an "object instance" (or just "object"). >> - An instance of an "value class" (or "object value") is a "value instance" (or just "value" if context allows). >> - When clarity is at risk, we can call a value class or value instance a "non-object class" or "non-object value". >> - Because object instances are referred to by reference, a variable bound to one is a "reference". >> (References can be to object instances, to boxes of value instances, or to the unique reference null.) >> - A reference can also take "null" ("the null reference") as a value. >> - References, primitives, and value instances are all "values" since they are passed by value. >> - An instance of a "value class" is a "value instance" or (when clarity is not at risk) just a "value". >> - Because value instances are referred to "by value", a variable bound to one is just a "value". >> (When clarity is at risk, such a variable can be called a "pure value" or "non-reference".) >> >> Ambiguity: >> >> - The term "value" used as a noun can refer to the contents of a variable: reference, primitive, or value instance. >> - The term "value" used as an adjective distinguishes a class, type, or instance from the "object" version. >> - The term "value" can abbreviate "value instance"; context must clear this usage from ambiguity. >> - Thus, the term "value" must always be used in a context which resolves it ambiguity. >> (We could coin a new term to avoid ambiguity, but the meaning of "value" perfect, so let's keep it.) >> - Sometimes when we say "value type" we really mean "non-object type", and expect primitives to be included. >> - As part of fit-and-finish of Value Types we will give the primitives a comfortable seat at the table. >> (Perhaps we can cleverly ret-con primitive types as value types, and their wrappers as boxes thereof.) >> >> Boxing, buffering, identity: >> >> - A value can be "boxed" into an object. Such an object can be "unboxed" back into its value. >> - Boxed values are true objects, with object type. >> - The class of a boxed value is the value class. (Thus each value class derives at least two types.) >> - A value (of any sort) does not have identity, only the object instance under a non-null reference does. >> - A "boxed value" (or "value is an object has identity, since it is an object instance. >> - If an implementation uses pointer indirection to access a value, we say it is stored in a "buffer". >> (This avoids confusion, since boxes are objects but buffers are not necessarily objects.) >> - Buffers are invisible to the user, except perhaps via performance effects, or trusted APIs like Unsafe. >> - Buffers can be on the stack, the heap, or anywhere else in memory. >> - Boxes can secretly serve as buffers. >> >> False friends: >> >> - A "java.lang.Class" is usually a reference to class metadata, but not necessarily unique. >> There's wiggle room here for class-for-the-box vs. class-for-the-value, and int.class. >> We don't allow java.lang.Class to constrain other uses of the term "class". >> - When clarity is at risk, we can say "class mirror" rather than just "Class". >> - Similar point for "CONSTANT_Class" in the constant pool schema. Relatively few folks >> are conscious of this term anyway. >> - "Object-oriented" programming usually refers to some combination of >> classes with reference-based polymorphism. Value types are object >> oriented, even though their reference-based polymorphism is limited to >> interfaces. Also they are squarely based on classes. >> >> Glossary: >> >> value type: a type which may be used without an accompanying reference (i.e., no intrinsic reference identity or aliasing) >> value class: a code entity which defines a value type >> value instance: a possible value (at runtime) of a variable of value type, derived from a value class >> value field: (ambig.) field whose type is a value type (in any kind of class) OR a field in a value class (of any type) >> value parameter: (ambig.) a parameter whose type is a value type OR a parameter, with emphasis on by-value transmission >> value: (ambig.) a reference, value instance, or primitive OR context-dependent ellipsis for value type/class/instance >> >> object type: a type without references (i.e., no reference identity, no aliasing) >> object class: a code entity which defines an object type >> object instance: a possible value (at runtime) of a non-null variable of reference type >> object field/parameter: (ambig., see above) >> object: (ambig.) a reference to an object instance OR context-dependent ellipsis for value type/class/instance >> >> box type: an object type derived from a value class >> box instance: a possible value (at runtime) of a non-null variable of box type >> box: (ambig., see above) >> >> instance/class/type: (ambig., see above) >> > From john.r.rose at oracle.com Sun Nov 19 21:40:33 2017 From: john.r.rose at oracle.com (John Rose) Date: Sun, 19 Nov 2017 13:40:33 -0800 Subject: abandon all U-types, welcome to L-world (or, what I learned in Burlington) Message-ID: <82299AAC-260E-416D-8618-A22C257A1C89@oracle.com> We just had a 50-hour week of face-to-face meetings by the Valhalla VM team. We learned a lot and surprised ourselves by coming to a consensus that a promising design for value types uses mainly the same legacy L-type descriptors, makes relatively little use of Q-type descriptors, and does not appear to need a third descriptor "kind" or "mode", such as U for universal, or R for reference-only. First a few highlights out of many. Fred Parain explained to us how he has prototyped a thread-local analog of Java heaps to store value structs in a form convenient to the interpreter. Tobias Hartmann and Roland Westrelin (of Red Hat) explained what the compiler prefers to see, which is (obviously) the scalarized components of each value. The three of them have worked out detailed rules for calling between interpreted and compiled code. It seems to me that other implementations of the JVM (looking at you, IBM) will tend in similar directions, so although our results are strongly informed by our own prototyping, we think it is likely that they will apply to other, independent JVM implementations. (Or are there platforms where the interpreter will scalarize aggressively and the optimizer will prefer to keep everything in structs? Not.) Karen Kinnear and the Oracle Valhalla lead, David Simms, were there to make sure we solved the important problems and asked the hard questions. As a special appearance, one of our spec. gurus, Dan Smith, was there to help us make rigorous sense out of our intuitions and hacks. Since we were short on language experts, we just worked in the mode (my personal favorite) of pretending that the JVM is the most important thing, and the Java language designers will just have to figure out how to use it. Of course, that's an oversimplification; the JLS and JVMS inform each other very strongly, but it was freeing to temporarily take current thoughts about JLS extensions as a given and vary the JVM to find the sweet spot that would be simple to implement and supportive to what we think we know about the Valhalla Java of the future. We had some long conversations about carrier types: L, Q, U, and more, and that's what I want to write about here. We also make significant progress in the design of crackable lambdas, template classes, and current and future versions of condy. We talked to Ron Pressler about kick starting Loom fibers. But it is L-types I want to talk about here; the above is just a sketch of the past week's environment. Logically speaking, we have two things we want to do, and that unfolds to a choice between three "worlds" of up to four distinct kinds: L/Q/U/R. L is always present because it is a legacy model for reference types. Q is always present because we know we need (at least sometimes) to make a syntactic distinction between flattened values and legacy objects. (Why not just always look inside the classfile? Because the verifier cannot be expected to load a class for every type it sees, so needs a descriptor kind character from time to time.) The U kind came a year or two ago when we realized that any-generics (and/or templates) and interfaces both require a disjoint-union type that is neither Q nor L, but can keep track of Q payloads (value instances) and L payloads (nullable references to object instances), without mixing them up. In other words, neither Q-types nor legacy L-types are parallel class-based constructs, and neither conveniently "sits on top" of the other; they need a common supertype to carry them without confusion. Before I describe the three logically possible "worlds", I'll add one more letter, R. An R-type is exactly a legacy L-type, a nullable reference. Why use a separate letter? Answer: For the same reason we introduced the other kind letters, to preserve all the necessary distinctions among different kinds of payloads and carrier types, and also to talk about the explicit encoding of descriptors. There are three worlds we could design to hold both legacy R-types (today's L-types) and Q-types: U-world, L-world, and R-world. They might be notated respectively as U/QL, L/Q, and U/QR. The "U-world" is what I have been mentally preparing for for many months. It is the design where L-types, marked as such in bytecode type descriptors, are always legacy object references or null, and Q-types, also marked as such in bytecodes, are always new value types. To carry runtime payloads which may dynamically vary between the two modes, we need a third mode, U-types, which carry the two kinds of payloads (I hesitate to say "values" because I want to include reference values also). A U-type is a disjoint union between corresponding, similarly named Q-types and L-types. (Mathematically, a _disjoint union_ of C = A |_| B is no more and no less than the sum of all elements or points comprised by the two constituent sets A and B. The disjoint union has nothing more: no points not in A or B. It has nothing less: every point of C is from either A and B, but never both. If A and B somehow look like they have a non-empty intersection, then C is adjusted so as to keep straight which elements are from A and which are from B.) The "R-world" is a copy of the "U-world", except that the new world has no L-types at all, or rather they are renamed as R-types. In this world, bridges would be required between legacy bytecodes (which use L's) and Valhalla bytecodes (which use R's for the same concept). We are pretty sure we don't want to live in R-world, but it helps to think about it, since it makes the maximum distinctions between legacy APIs and upgraded Valhalla APIs. Any bridge from R-world to legacy code will presumably come after a clear decision has been made to allow the legacy code to see, under the name of L-types, the R's from the new world, plus whatever Q's are also allowed over the bridge to interoperate wit the old code. The U-world has similar need for bridges, but less extreme. We know we will need some bridges to upgrade legacy classes like List to use U-types (List, List References: <82299AAC-260E-416D-8618-A22C257A1C89@oracle.com> Message-ID: <2113509537.7960.1511131636643.JavaMail.zimbra@u-pem.fr> To summarize for myself, we already know that we only need only one U, java.lang.__Value, let try to make it java.lang.Object (with no boxing). The claim is that Object is used more as the root of any types like in collections than as the root of all references like in System.out.println(). Ok, i need to think more about that. regards, R?mi ----- Mail original ----- > De: "John Rose" > ?: "valhalla-spec-experts" > Envoy?: Dimanche 19 Novembre 2017 22:40:33 > Objet: abandon all U-types, welcome to L-world (or, what I learned in Burlington) > We just had a 50-hour week of face-to-face meetings by the > Valhalla VM team. We learned a lot and surprised ourselves > by coming to a consensus that a promising design for value > types uses mainly the same legacy L-type descriptors, makes > relatively little use of Q-type descriptors, and does not appear > to need a third descriptor "kind" or "mode", such as U for > universal, or R for reference-only. > > First a few highlights out of many. Fred Parain explained to us how > he has prototyped a thread-local analog of Java heaps to store value > structs in a form convenient to the interpreter. Tobias Hartmann > and Roland Westrelin (of Red Hat) explained what the compiler > prefers to see, which is (obviously) the scalarized components > of each value. The three of them have worked out detailed > rules for calling between interpreted and compiled code. > > It seems to me that other implementations of the JVM (looking > at you, IBM) will tend in similar directions, so although our > results are strongly informed by our own prototyping, we think > it is likely that they will apply to other, independent JVM > implementations. (Or are there platforms where the interpreter > will scalarize aggressively and the optimizer will prefer to > keep everything in structs? Not.) > > Karen Kinnear and the Oracle Valhalla lead, David Simms, were > there to make sure we solved the important problems and asked > the hard questions. As a special appearance, one of our spec. > gurus, Dan Smith, was there to help us make rigorous sense out > of our intuitions and hacks. > > Since we were short on language experts, we just worked in > the mode (my personal favorite) of pretending that the JVM > is the most important thing, and the Java language designers > will just have to figure out how to use it. Of course, that's an > oversimplification; the JLS and JVMS inform each other very > strongly, but it was freeing to temporarily take current thoughts > about JLS extensions as a given and vary the JVM to find > the sweet spot that would be simple to implement and supportive > to what we think we know about the Valhalla Java of the future. > > We had some long conversations about carrier types: L, Q, U, > and more, and that's what I want to write about here. We also > make significant progress in the design of crackable lambdas, > template classes, and current and future versions of condy. > We talked to Ron Pressler about kick starting Loom fibers. > But it is L-types I want to talk about here; the above is just a > sketch of the past week's environment. > > Logically speaking, we have two things we want to do, and > that unfolds to a choice between three "worlds" of up to four > distinct kinds: L/Q/U/R. L is always present because it is > a legacy model for reference types. Q is always present > because we know we need (at least sometimes) to make > a syntactic distinction between flattened values and legacy > objects. > > (Why not just always look inside the classfile? Because > the verifier cannot be expected to load a class for every > type it sees, so needs a descriptor kind character from > time to time.) > > The U kind came a year or two ago when we realized > that any-generics (and/or templates) and interfaces both > require a disjoint-union type that is neither Q nor L, but > can keep track of Q payloads (value instances) and L > payloads (nullable references to object instances), > without mixing them up. In other words, neither Q-types > nor legacy L-types are parallel class-based constructs, > and neither conveniently "sits on top" of the other; they > need a common supertype to carry them without confusion. > > Before I describe the three logically possible "worlds", > I'll add one more letter, R. An R-type is exactly a legacy > L-type, a nullable reference. Why use a separate letter? > Answer: For the same reason we introduced the other > kind letters, to preserve all the necessary distinctions > among different kinds of payloads and carrier types, > and also to talk about the explicit encoding of descriptors. > > There are three worlds we could design to hold both legacy > R-types (today's L-types) and Q-types: U-world, L-world, > and R-world. They might be notated respectively as U/QL, > L/Q, and U/QR. > > The "U-world" is what I have been mentally preparing for > for many months. It is the design where L-types, marked > as such in bytecode type descriptors, are always legacy > object references or null, and Q-types, also marked as > such in bytecodes, are always new value types. To > carry runtime payloads which may dynamically vary > between the two modes, we need a third mode, U-types, > which carry the two kinds of payloads (I hesitate to say > "values" because I want to include reference values also). > > A U-type is a disjoint union between corresponding, > similarly named Q-types and L-types. > > (Mathematically, a _disjoint union_ of C = A |_| B is no more > and no less than the sum of all elements or points comprised > by the two constituent sets A and B. The disjoint union has > nothing more: no points not in A or B. It has nothing less: > every point of C is from either A and B, but never both. > If A and B somehow look like they have a non-empty > intersection, then C is adjusted so as to keep straight > which elements are from A and which are from B.) > > The "R-world" is a copy of the "U-world", except that the > new world has no L-types at all, or rather they are renamed > as R-types. In this world, bridges would be required > between legacy bytecodes (which use L's) and Valhalla > bytecodes (which use R's for the same concept). > > We are pretty sure we don't want to live in R-world, but > it helps to think about it, since it makes the maximum > distinctions between legacy APIs and upgraded Valhalla > APIs. Any bridge from R-world to legacy code will > presumably come after a clear decision has been made > to allow the legacy code to see, under the name of L-types, > the R's from the new world, plus whatever Q's are also > allowed over the bridge to interoperate wit the old code. > > The U-world has similar need for bridges, but less extreme. > We know we will need some bridges to upgrade legacy > classes like List to use U-types (List, List The L-types of U-world just mix without effort into the legacy > L-types of legacy classes, since the same letter is used. > > The third logical choice, and the one we are now looking > at very seriously, is "L-world". (Break out the "abandon > all hope" and "Niflheim" jokes!) In L-world, we identify > (some would say conflate or confuse) the necessary > U-type which unites R-types and Q-types with the legacy > syntax "L". The Q-type syntax is *maybe* needed, but > in any case does not appear in a parallel position of > importance with the dominant L-type syntax. The R-type > syntax seems even less important; we haven't thought > of a use for it. But it is in reserve, in case we need > R-type descriptors for some corner case. > > The distinction between value types and object types > is still fundamental, as is the distinction between flat > and non-flat data. The classfile which defines any > given type unambiguously declares whether it is an > object or value type. But in L-world, the L-type > descriptors can carry both payloads. That's the > key decision before us. > > (For brevity I'll say R-type/R-value when I mean a > legacy nullable reference type/value, and Q-type/Q-value > for value type/instance. This doesn't mean that we > will need Q's and R's in the final bytecode syntax. > But they are useful concepts.) > > There are many implications from the decision to > put L-types at the top: > > * The type L-Object ("Ljava/lang/Object;") carries both > . Thus, we don't need a > new top-type. (There are objectionable properties of > L-Object which need remediation, but this was always > true, and is not a showstopper for L-world.) > > * Likewise, legacy interface types like L-Comparable > are immediately useful (without bridges) for carrying > value instances as well as object instances (and null). > > * It is possible, in some cases, that standard and user-written > collection classes will work correctly, without recompilation, > with value types. (This is a big claim, and valuable if true. > Read on.) > > * All basic operations that the JVM applies to R-types must > extend immediately and pervasively to Q-types, since it > applies them to L-type values (which may be either, > dynamically). > > * Today, simple movement of R-types is really cheap, just > a machine pointer move. That needs to be true for L-types > in L-world, or else we will get systematic performance hits > for legacy code, and new code will go slow too. > > * There are a number of object-specific operations which > the JVM applies to L-types. The most common is "acmp" > (the "==" operator for references). Those operations must > be enhanced to do something useful with values, with a > possible runtime cost to detect the distinction between > an L-type carrying a Q-value and an L-type carrying a > legacy R-value. The performance and usable semantics > of these object operations will make L-world either > a programmer's paradise or a? well you know. > > * There is no need for boxes, and they turn out to be > undesirable. Legacy types like java.lang.Integer must > be given a golden watch and a pension, somehow. > That's easy for the JVM but hard for the language, > which mandates that "(Object)(int)x" produces an > Integer rather than an "int". It seemed a good idea > at the time. > > * There is no need for a new "universal" carrier type, > since L-types do the whole job. Before the L-world > discussion, my thought has been that we want a 128-bit > U-type and a 64-bit legacy L/R-type. Somebody burst > my bubble this week, by saying that if we do that, > we may find that interpreter speeds for U-type generics > will risk a built-in performance barrier just from the > larger standard carrier type. If we JVM folks can agree > that U-types should be 64-bits (by all available means) > then it is just a simple step to rename U to L. This is > the rabbit hole that took our conversation down to L-world. > > * In L-world, the "acmp" instruction needs a very fast way > to detect Q-values. This *may* require a tag bit on the 64-bit > root value. That in turn will affect GC dynamics. There is > a delicate balance here?but we think there is a way through. > > * We probably need extra interpreter profiling to track whether > a given L-value has ever been a Q-value or an R-value, > dynamically. Today we do null tracking on some instructions. > This probably needs to be upgraded to null/Q/R tracking, > and perhaps on additional instructions such as "acmp". > > * There are a number of ways to assign semantics to > an object-like L operation when it encounters a Q-value. > This will require additional mails, but I think we have > identified about a half dozen models, of which one or two > seem to be very promising: Providing both useful semantics > and amenable to optimization. > > * One residual use for Q-types is in the declaration of > instance fields. In order to avoid loading *all* classfiles > of types mentioned in field declarations, a classfile which > declares a flattened field will need to include enough > information to allow the classfile loader to load *only* > those fields marked as requiring flattening. There are > at least two ways to do this: Use a Q-type descriptor > syntax *only* for field declarations, as today. Or, > require the ACC_VALUE bit on field declarations which > are supposed to be flattened. > > * As we were able to dispense with boxes, we may also > dispense with non-flattened value types. In that case, > the translation strategy might emit an ACC_VALUE bit > or Q-type on a field if and only if the classfile for the > field's type defines it with ACC_VALUE. The JVM will > have to support non-flattened values in L-Object fields, > of course. > > * If the system uses a thread-local store for value structures > (to avoid heap traffic), a store barrier will have to quickly > detect Q-types that are inside the thread and reallocate > them to the heap, when they are first stored to the heap > (e.g., as an element of an L-Object array). > > * The Q-type modifier *might* be useful in some settings > to guarantee, in a verifiable way, that a given value is > *not* an R-type, *not* null, and *not* modifiable; TBD. > > * The R-type modifier *might* be useful in some settings > to guarantee, in a verifiable way, that a given value is > *not* a Q-type, and *does* have an object identity or > is null. This is also TBD. > > * For best compatibility with legacy code, combined with > diagnosability of anti-value algorithms like IdentityHashMap, > the "acmp" instruction should return false unconditionally > if either operand is a Q-value (punting to the following > Object.equals call), and other object-like operations > such as identityHashCode and monitorenter must throw > errors in the JVM. (In the language errors and warnings > will be appropriate.) > > * New operations are needed for substitutability checks > which generalize reference equality and hashcode. > These can be system methods, and do not need to be > loaded onto either new or old bytecodes. > > * We will almost certainly need to make primitives > retroactively values. This means "int" all along has > really been Q-int (in the JVM) and is a real subtype > of L-Object. > > * Covariant array subtyping only works for R-types. > So both int[] and DoubleComplex[] are *not* subtypes > of Object[], even though int and DoubleComplex *are* > subtypes of Object. > > * From some points of view (legacy code), Q-values > are masked invaders coming into the home of code > which expected to work only on R-values. Changing > L-descriptors to encompass Q-values opens such > code to potentially risky new behaviors. Is it safe? > Shouldn't we just have boxes to mediate values > in such settings? It depends on the code, really. > > There's more, but this is enough for one message. > > The L-world is very attractive: No bridges or boxes, > legacy code is value-enabled, and we get all the > flattening we need. > > We need to do some experiments: Can we afford > the extra Q-checks on acmp and storage to the heap? > Will legacy algorithms really work on masked but not > boxed values? Do other JVM implementations experience > similar trade-offs, or is this only a HotSpot-centric set > of compromises? Can we really avoid all those new > descriptors and bridges!!?? > > Let's talk! > > ? John > > P.S. Dan, you should send out your notes on U-types. From john.r.rose at oracle.com Sun Nov 19 23:06:20 2017 From: john.r.rose at oracle.com (John Rose) Date: Sun, 19 Nov 2017 15:06:20 -0800 Subject: abandon all U-types, welcome to L-world (or, what I learned in Burlington) In-Reply-To: <2113509537.7960.1511131636643.JavaMail.zimbra@u-pem.fr> References: <82299AAC-260E-416D-8618-A22C257A1C89@oracle.com> <2113509537.7960.1511131636643.JavaMail.zimbra@u-pem.fr> Message-ID: <339034F4-E369-4A1B-A119-C05A2E115F6D@oracle.com> On Nov 19, 2017, at 2:47 PM, Remi Forax wrote: > > The claim is that Object is used more as the root of any types like in collections than as the root of all references like in System.out.println(). Object and interfaces play the role of top types. One view is that we are making object act more like an interface. Also we don?t add any new carrier types to the interpreter. From forax at univ-mlv.fr Sun Nov 19 23:33:10 2017 From: forax at univ-mlv.fr (forax at univ-mlv.fr) Date: Mon, 20 Nov 2017 00:33:10 +0100 (CET) Subject: abandon all U-types, welcome to L-world (or, what I learned in Burlington) In-Reply-To: <339034F4-E369-4A1B-A119-C05A2E115F6D@oracle.com> References: <82299AAC-260E-416D-8618-A22C257A1C89@oracle.com> <2113509537.7960.1511131636643.JavaMail.zimbra@u-pem.fr> <339034F4-E369-4A1B-A119-C05A2E115F6D@oracle.com> Message-ID: <1078854640.9034.1511134390234.JavaMail.zimbra@u-pem.fr> ----- Mail original ----- > De: "John Rose" > ?: "Remi Forax" > Cc: "valhalla-spec-experts" > Envoy?: Lundi 20 Novembre 2017 00:06:20 > Objet: Re: abandon all U-types, welcome to L-world (or, what I learned in Burlington) > On Nov 19, 2017, at 2:47 PM, Remi Forax wrote: >> >> The claim is that Object is used more as the root of any types like in >> collections than as the root of all references like in System.out.println(). > > > Object and interfaces play the role of top types. One view is that we are making > object act more like an interface. ah, yes, it makes the whole model far simpler. > > Also we don?t add any new carrier types to the interpreter. but you need a way disambiguate a reference type from a value type at runtime in the interpreter. You also nedd to teach JITs to propagate L vs which Q info on local variables for generics specialization (and it works even if the inlining fails because the boxing/wrapping in the thread local storage is done by the adapters so the JITed code doesn't have to be conservative). R?mi From john.r.rose at oracle.com Mon Nov 20 00:15:02 2017 From: john.r.rose at oracle.com (John Rose) Date: Sun, 19 Nov 2017 16:15:02 -0800 Subject: abandon all U-types, welcome to L-world (or, what I learned in Burlington) In-Reply-To: <1078854640.9034.1511134390234.JavaMail.zimbra@u-pem.fr> References: <82299AAC-260E-416D-8618-A22C257A1C89@oracle.com> <2113509537.7960.1511131636643.JavaMail.zimbra@u-pem.fr> <339034F4-E369-4A1B-A119-C05A2E115F6D@oracle.com> <1078854640.9034.1511134390234.JavaMail.zimbra@u-pem.fr> Message-ID: <96802560-C3E1-4E03-AB1F-AF5C09D40875@oracle.com> On Nov 19, 2017, at 3:33 PM, forax at univ-mlv.fr wrote: > > but you need a way disambiguate a reference type from a value type at runtime in the interpreter. > You also nedd to teach JITs to propagate L vs which Q info on local variables for generics specialization (and it works even if the inlining fails because the boxing/wrapping in the thread local storage is done by the adapters so the JITed code doesn't have to be conservative). Yes, that leads to the sort of tag bot schemes I alluded to. I didn?t want to go into detail because it is a spec. list not a dev. list. On the dev. list we will say more about it. The problems seem solvable especially after the MVT prototyping experience on HotSpot, and I hope the IBM experience corroborates. From peter.levart at gmail.com Mon Nov 20 06:31:09 2017 From: peter.levart at gmail.com (Peter Levart) Date: Mon, 20 Nov 2017 07:31:09 +0100 Subject: [Nestmates] Add a core reflection API to get nestmate information In-Reply-To: References: <3309a72e-196a-6ea2-c744-bdd503b7cc0f@gmail.com> <4b98c3d4-d40b-87da-702b-c20ca9ac7bf9@gmail.com> Message-ID: <2c000cb9-5a94-373d-6bb1-ab3fb31b6a07@gmail.com> Hi David, On 11/17/2017 11:44 AM, David Holmes wrote: > Hi Peter, > > On 17/11/2017 7:45 PM, Peter Levart wrote: >> Hi David, >> >> On 11/16/2017 10:52 PM, David Holmes wrote: >>>>> John prefers to minimize exceptions for the Java API. :) >>>> >>>> Suppose that this is basic reflection API to get nestmate >>>> information and that it will also be used for reflective access >>>> checks. For example in >>>> jdk.internal.reflect.Reflection#verifyMemberAccess, which is used >>> >>> It isn't. The reflection API is purely for "external" use. Real >>> access checks are performed in the same way as the VM access checks >>> - as required - and will report the same exceptions in the same way. >>> >>> David >> >> Real access checks performed by bytecode(s), yes. But what about >> reflective access checks - those performed by Method.invoke, >> Field.[get|set], Constructor.newInstance? They will have to be >> revised some day to include the nestmates. What facility should those >> methods use to implement the checks (the guts of those checks is in >> jdk.internal.reflect.Reflection#verifyMemberAccess). > > Yes and the real nestmate access check for use by invoke etc has > already been implemented in there. > > http://hg.openjdk.java.net/valhalla/valhalla/file/df423aeaa8d1/src/java.base/share/classes/jdk/internal/reflect/Reflection.java > Oh, sorry. I have been looking at wrong repository (jdk master) and thought the proposed core reflection nestmate API was the 1st change to reflection. I see you have already updated invoke/get/set/newInstance. > >> Is the plan to have special internal native methods to facilitate >> that? I don't quite understand why should a Class.isNestmateOf(Class) >> behave any differently than real VM access check. > > Purely because of the difference in exception handling. The view was > that the reflection API should mostly "absorb" exceptions. Absorbing exceptions is generally a bad idea. In this particular case, you are trying to hide an implementation detail and instead lie to the user about the nest membership. Some information is lost on the way and that's not helping the user at all. Not being in the same nest and not being able to evaluate the answer are two different things and as much as someone might think the user shouldn't care, the need to discriminate them is probably going to arise and users will have to do sub-optimal things to accomplish that. Why not give them the precise tool to start with? Regards, Peter > > David From david.holmes at oracle.com Mon Nov 20 06:36:48 2017 From: david.holmes at oracle.com (David Holmes) Date: Mon, 20 Nov 2017 16:36:48 +1000 Subject: [Nestmates] Add a core reflection API to get nestmate information In-Reply-To: <2c000cb9-5a94-373d-6bb1-ab3fb31b6a07@gmail.com> References: <3309a72e-196a-6ea2-c744-bdd503b7cc0f@gmail.com> <4b98c3d4-d40b-87da-702b-c20ca9ac7bf9@gmail.com> <2c000cb9-5a94-373d-6bb1-ab3fb31b6a07@gmail.com> Message-ID: <9952db0e-6a16-dcee-5975-292d1f83088e@oracle.com> Hi Peter, On 20/11/2017 4:31 PM, Peter Levart wrote: > Hi David, > > On 11/17/2017 11:44 AM, David Holmes wrote: >> Hi Peter, >> >> On 17/11/2017 7:45 PM, Peter Levart wrote: >>> Hi David, >>> >>> On 11/16/2017 10:52 PM, David Holmes wrote: >>>>>> John prefers to minimize exceptions for the Java API. :) >>>>> >>>>> Suppose that this is basic reflection API to get nestmate >>>>> information and that it will also be used for reflective access >>>>> checks. For example in >>>>> jdk.internal.reflect.Reflection#verifyMemberAccess, which is used >>>> >>>> It isn't. The reflection API is purely for "external" use. Real >>>> access checks are performed in the same way as the VM access checks >>>> - as required - and will report the same exceptions in the same way. >>>> >>>> David >>> >>> Real access checks performed by bytecode(s), yes. But what about >>> reflective access checks - those performed by Method.invoke, >>> Field.[get|set], Constructor.newInstance? They will have to be >>> revised some day to include the nestmates. What facility should those >>> methods use to implement the checks (the guts of those checks is in >>> jdk.internal.reflect.Reflection#verifyMemberAccess). >> >> Yes and the real nestmate access check for use by invoke etc has >> already been implemented in there. >> >> http://hg.openjdk.java.net/valhalla/valhalla/file/df423aeaa8d1/src/java.base/share/classes/jdk/internal/reflect/Reflection.java >> > > Oh, sorry. I have been looking at wrong repository (jdk master) and > thought the proposed core reflection nestmate API was the 1st change to > reflection. I see you have already updated invoke/get/set/newInstance. > >> >>> Is the plan to have special internal native methods to facilitate >>> that? I don't quite understand why should a Class.isNestmateOf(Class) >>> behave any differently than real VM access check. >> >> Purely because of the difference in exception handling. The view was >> that the reflection API should mostly "absorb" exceptions. > > Absorbing exceptions is generally a bad idea. In this particular case, > you are trying to hide an implementation detail and instead lie to the > user about the nest membership. Some information is lost on the way and > that's not helping the user at all. Not being in the same nest and not > being able to evaluate the answer are two different things and as much > as someone might think the user shouldn't care, the need to discriminate > them is probably going to arise and users will have to do sub-optimal > things to accomplish that. Why not give them the precise tool to start > with? That was my personal initial position too. But the EG discussions went elsewhere: http://mail.openjdk.java.net/pipermail/valhalla-spec-experts/2017-October/000386.html Cheers, David > > Regards, Peter > >> >> David > From peter.levart at gmail.com Mon Nov 20 09:20:24 2017 From: peter.levart at gmail.com (Peter Levart) Date: Mon, 20 Nov 2017 10:20:24 +0100 Subject: [Nestmates] Add a core reflection API to get nestmate information In-Reply-To: <9952db0e-6a16-dcee-5975-292d1f83088e@oracle.com> References: <3309a72e-196a-6ea2-c744-bdd503b7cc0f@gmail.com> <4b98c3d4-d40b-87da-702b-c20ca9ac7bf9@gmail.com> <2c000cb9-5a94-373d-6bb1-ab3fb31b6a07@gmail.com> <9952db0e-6a16-dcee-5975-292d1f83088e@oracle.com> Message-ID: <74516177-1820-df3d-cb2f-7f6807483c8e@gmail.com> Hi David, On 11/20/2017 07:36 AM, David Holmes wrote: >>>> I don't quite understand why should a Class.isNestmateOf(Class) >>>> behave any differently than real VM access check. >>> >>> Purely because of the difference in exception handling. The view was >>> that the reflection API should mostly "absorb" exceptions. >> >> Absorbing exceptions is generally a bad idea. In this particular >> case, you are trying to hide an implementation detail and instead lie >> to the user about the nest membership. Some information is lost on >> the way and that's not helping the user at all. Not being in the same >> nest and not being able to evaluate the answer are two different >> things and as much as someone might think the user shouldn't care, >> the need to discriminate them is probably going to arise and users >> will have to do sub-optimal things to accomplish that. Why not give >> them the precise tool to start with? > > That was my personal initial position too. But the EG discussions went > elsewhere: > > http://mail.openjdk.java.net/pipermail/valhalla-spec-experts/2017-October/000386.html > > > Cheers, > David Your comment in the issue: https://bugs.openjdk.java.net/browse/JDK-8187768?focusedCommentId=14123862&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14123862 ...describes how access checking for private members behaves in VM and what are the outcomes when class attributes are missing or not consistent (access is denied - IllegalAccessError is thrown) vs. when nest-host class can not be resolved (NoClassDefFoundError is thrown). I think this is sound. In reflective API, it is common practice for IllegalAccessException to be thrown where VM would throw IllegalAccessError, but NoClassDefFoundError is thrown as is when various reflective methods are invoked. For example, Class.getDeclaredField(fieldName) throws NoClassDefFoundError when the field's type can not be resolved. Likewise for method's and constructor's parameter types. Class.getDeclaredField() is not trying to hide the return type resolution error by lying about the existence of the field and throwing NoSuchFieldException instead. So why should Class.isNestmateOf(Class) hide the NoClassDefFoundError and lie about nest membership instead? I agree that Class.getNestMembers() is a special case, but I think it should nevertheless propagate NoClassDefFoundError when nest-host can not be resolved, while resolution errors for other nest-members listed in nest-host could just skip them from the result (to allow for optimized jars). Although maybe, jar optimizers should rather "patch" the nest-host attribute and remove members that are not included. They would have to be aware of nestmate attributes to not accidentally remove the nest-host class from the optimized jar anyway, or rearrange attributes to designate one of remaining classes to be new nest-host in optimized jar. Regards, Peter From peter.levart at gmail.com Mon Nov 20 09:39:19 2017 From: peter.levart at gmail.com (Peter Levart) Date: Mon, 20 Nov 2017 10:39:19 +0100 Subject: [Nestmates] Add a core reflection API to get nestmate information In-Reply-To: <74516177-1820-df3d-cb2f-7f6807483c8e@gmail.com> References: <3309a72e-196a-6ea2-c744-bdd503b7cc0f@gmail.com> <4b98c3d4-d40b-87da-702b-c20ca9ac7bf9@gmail.com> <2c000cb9-5a94-373d-6bb1-ab3fb31b6a07@gmail.com> <9952db0e-6a16-dcee-5975-292d1f83088e@oracle.com> <74516177-1820-df3d-cb2f-7f6807483c8e@gmail.com> Message-ID: <122b6d11-0e35-bcb4-164d-1775658b6751@gmail.com> Hi David, On 11/20/2017 10:20 AM, Peter Levart wrote: > Your comment in the issue: > > https://bugs.openjdk.java.net/browse/JDK-8187768?focusedCommentId=14123862&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14123862 > > ...describes how access checking for private members behaves in VM and > what are the outcomes when class attributes are missing or not > consistent (access is denied - IllegalAccessError is thrown) vs. when > nest-host class can not be resolved (NoClassDefFoundError is thrown). > I think this is sound. I found a wording in the spec update that maybe requires some explanation. This is the relevant part: /A class or interface H grants nest membership to a class or interface D if H is declared in the same run-time package as D, H has a NestMembers attribute (4.7.29), and there is some value j in the classes array of the NestMembers attribute of H, such that item j of the run-time constant pool of H is a symbolic reference to D. // //// //????????Because H belongs to the same run-time package as D, it is not necessary to resolve a symbolic reference in H in order to determine whether it references D. It is only necessary to check the unresolved name./ What does it mean that "H belongs to the same run-time package as D"? Does it mean that package names of H and D are equal, or does it also imply that both H and D are loaded by the same class loader? While jigsaw makes the class loader check unnecessary (either H and D are in the same module and consequently loaded by same class loader or H is not resolvable from D when they share package name), java still has plain ClassLoader(s) and does not always guarantee the absence of split packages. Regards, Peter From david.holmes at oracle.com Mon Nov 20 11:43:51 2017 From: david.holmes at oracle.com (David Holmes) Date: Mon, 20 Nov 2017 21:43:51 +1000 Subject: [Nestmates] Add a core reflection API to get nestmate information In-Reply-To: <122b6d11-0e35-bcb4-164d-1775658b6751@gmail.com> References: <3309a72e-196a-6ea2-c744-bdd503b7cc0f@gmail.com> <4b98c3d4-d40b-87da-702b-c20ca9ac7bf9@gmail.com> <2c000cb9-5a94-373d-6bb1-ab3fb31b6a07@gmail.com> <9952db0e-6a16-dcee-5975-292d1f83088e@oracle.com> <74516177-1820-df3d-cb2f-7f6807483c8e@gmail.com> <122b6d11-0e35-bcb4-164d-1775658b6751@gmail.com> Message-ID: <25e46958-75af-148b-c0bd-25bb718c7482@oracle.com> Hi Peter, On 20/11/2017 7:39 PM, Peter Levart wrote: > Hi David, > > On 11/20/2017 10:20 AM, Peter Levart wrote: >> Your comment in the issue: >> >> https://bugs.openjdk.java.net/browse/JDK-8187768?focusedCommentId=14123862&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14123862 >> >> ...describes how access checking for private members behaves in VM and >> what are the outcomes when class attributes are missing or not >> consistent (access is denied - IllegalAccessError is thrown) vs. when >> nest-host class can not be resolved (NoClassDefFoundError is thrown). >> I think this is sound. > > I found a wording in the spec update that maybe requires some > explanation. This is the relevant part: Latest wording is here: http://cr.openjdk.java.net/~dlsmith/nestmates.html > /A class or interface H grants nest membership to a class or interface D > if H is declared in the same run-time package as D, H has a NestMembers > attribute (4.7.29), and there is some value j in the classes array of > the NestMembers attribute of H, such that item j of the run-time > constant pool of H is a symbolic reference to D. // > //// > //????????Because H belongs to the same run-time package as D, it is not > necessary to resolve a symbolic reference in H in order to determine > whether it references D. It is only necessary to check the unresolved name./ > > What does it mean that "H belongs to the same run-time package as D"? > Does it mean that package names of H and D are equal, or does it also > imply that both H and D are loaded by the same class loader? While > jigsaw makes the class loader check unnecessary (either H and D are in > the same module and consequently loaded by same class loader or H is not > resolvable from D when they share package name), java still has plain > ClassLoader(s) and does not always guarantee the absence of split packages. Same runtime package == same package name + same defining classloader. This is as defined for verification (JVMS 4.10) and class creation and loading (JVMS 5.3). David ----- > Regards, Peter > From daniel.smith at oracle.com Wed Nov 22 05:54:55 2017 From: daniel.smith at oracle.com (Dan Smith) Date: Tue, 21 Nov 2017 22:54:55 -0700 Subject: Design notes for next values iteration Message-ID: <6FF0E36D-92F1-4B44-B778-A3B0EE4C4A93@oracle.com> Following up on John's mail, here are some detailed notes about the design choices for "U types", and the concrete proposal we settled on last week. http://cr.openjdk.java.net/~dlsmith/values-notes.html John may have raised additional points I didn't cover in this document. I'll do another review of his mail and make any needed updates; or feel free to call them out. :-) ?Dan From forax at univ-mlv.fr Wed Nov 22 09:42:36 2017 From: forax at univ-mlv.fr (Remi Forax) Date: Wed, 22 Nov 2017 10:42:36 +0100 (CET) Subject: Design notes for next values iteration In-Reply-To: <6FF0E36D-92F1-4B44-B778-A3B0EE4C4A93@oracle.com> References: <6FF0E36D-92F1-4B44-B778-A3B0EE4C4A93@oracle.com> Message-ID: <1970438356.1129084.1511343756393.JavaMail.zimbra@u-pem.fr> Interesting ! here is my current view of the problem. Lets say we are in the R/Q/U world, we want U interface to abstract over a reference type or a value type, we want UObject because object is basically an interface, we also want U object like UOptional to be able to painlessly move a reference type to a value type and vice-versa (no bridge-o-matic). But why we want Q type BTW, we want Q type because 1/ you can not apply the same opcodes on Q types and R types (more or less, some ops are not defined on Q types) 2/ we want the VM to flatten array and not box Q type. 3/ the VM need a clean separation between Q type and R type For 1/ we can extends the semantics of the opcodes that only works on R types to work on Q types, we need that anyway for U types and at worst a valid semantics is to throw an exception For 2/ even if we sak for flatten array or no box, there are case where the VM will not flatten the array or the q type, so it's like a hint than a strict requirement. For 3/ the current prototype has shown that the VM is able to box/buffer value types where needed, so no ! I think we do not need Q types, Q types are use site annotations, and here we want declaration site annotations (let say that this class is a value class, the ACC_VALUE). If we have no Q type, it means that R types and U types are the same thing, everything is a U type. In term of migration, it means that L types need to have their semantics extended to work as U types. Obviously, the devil is in the details, but i think from the VM POV, we should extends L types to behave like U types. >From Java the language POV, we may want to not compile when a variable is typed with value type is assigned to null, same with generics, you may want to declare a type variable any but from the VM POV, this should be possible and either throw an exception, return false, etc. Let's make the VM powerful and let's Java the language helps users to not have too many exceptions. regards, R?mi ----- Mail original ----- > De: "daniel smith" > ?: "valhalla-spec-experts" > Envoy?: Mercredi 22 Novembre 2017 06:54:55 > Objet: Design notes for next values iteration > Following up on John's mail, here are some detailed notes about the design > choices for "U types", and the concrete proposal we settled on last week. > > http://cr.openjdk.java.net/~dlsmith/values-notes.html > > John may have raised additional points I didn't cover in this document. I'll do > another review of his mail and make any needed updates; or feel free to call > them out. :-) > > ?Dan From forax at univ-mlv.fr Wed Nov 22 11:05:28 2017 From: forax at univ-mlv.fr (Remi Forax) Date: Wed, 22 Nov 2017 12:05:28 +0100 (CET) Subject: More on seeing the L world as a U world In-Reply-To: <1970438356.1129084.1511343756393.JavaMail.zimbra@u-pem.fr> References: <6FF0E36D-92F1-4B44-B778-A3B0EE4C4A93@oracle.com> <1970438356.1129084.1511343756393.JavaMail.zimbra@u-pem.fr> Message-ID: <1060469278.1179775.1511348728201.JavaMail.zimbra@u-pem.fr> Operations: as in Dan's proposal (lets call it that way), checkcast, putfield, monitorenter/monitorexit, ic_acmpeq, if_acmpne, aload, astore, areturn, anewarray, multianewarray, aaload and aastore have their semantics enhanced. unlike the Dan proposal, newly introduced opcodes, adefault and withfield have their semantics defined on value type and also on reference type, adefault throws an exception on a reference value (to not allow to bypass the constructors) and withfield works the same way for value type and reference type. note that isValue is not necessary because it's getClass() + isValueType(). Primitive types: primitive types are seen by the VM as value types, int.class.isValueType() returns true. there is no need to see those class in the JDK, so like currently there are synthetize by the VM. So storing an int as an Object is equivalent to box/buffer the int like any value type. Performance: If we have U types everywhere, doesn't it introduce perf issues, by example, what about acmp now having to test if it operates on value types or reference types. There are two facts that save us here, with acmp as example, - first for one execution, a type is either a value type or a reference type, so it enables the kind of optimization we do with quick opcodes, i.e. the slow path is if you do not know if the type is a reference type or a value type, but once you have do the test for a type, you can install a fast path for all acmp on the same type. - if the cost is too important we can have say that acmp will always do a pointer check (here we are saved by the fact that in hotspot a value type is also a machine word pointer) and introduces a new opcode, ucmp that have a bigger overhead. The semantics of acmp on value type in that case is garbage, but not less garbage that the current boxing semantics specified by the JLS. So in my opinion, we are no in a better position in term of performance compared to Dan's proposal but we are not in a worst position too. I think the weakest point of the Dan's proposal is that the model is really complex, which means that it will impose a real burden when we will want to maintain or tweak it again in the future. The model i propose is far simpler. R?mi ----- Mail original ----- > De: "Remi Forax" > ?: "daniel smith" > Cc: "valhalla-spec-experts" > Envoy?: Mercredi 22 Novembre 2017 10:42:36 > Objet: Re: Design notes for next values iteration > Interesting ! > here is my current view of the problem. > > Lets say we are in the R/Q/U world, we want U interface to abstract over a > reference type or a value type, we want UObject because object is basically an > interface, we also want U object like UOptional to be able to painlessly move a > reference type to a value type and vice-versa (no bridge-o-matic). > > But why we want Q type BTW, we want Q type because > 1/ you can not apply the same opcodes on Q types and R types (more or less, some > ops are not defined on Q types) > 2/ we want the VM to flatten array and not box Q type. > 3/ the VM need a clean separation between Q type and R type > > For 1/ we can extends the semantics of the opcodes that only works on R types to > work on Q types, we need that anyway for U types and at worst a valid semantics > is to throw an exception > For 2/ even if we sak for flatten array or no box, there are case where the VM > will not flatten the array or the q type, so it's like a hint than a strict > requirement. > For 3/ the current prototype has shown that the VM is able to box/buffer value > types where needed, so no ! > > I think we do not need Q types, Q types are use site annotations, and here we > want declaration site annotations (let say that this class is a value class, > the ACC_VALUE). > > If we have no Q type, it means that R types and U types are the same thing, > everything is a U type. In term of migration, it means that L types need to > have their semantics extended to work as U types. > > Obviously, the devil is in the details, but i think from the VM POV, we should > extends L types to behave like U types. > From Java the language POV, we may want to not compile when a variable is typed > with value type is assigned to null, same with generics, you may want to > declare a type variable any but from the VM POV, this should be possible and > either throw an exception, return false, etc. > > Let's make the VM powerful and let's Java the language helps users to not have > too many exceptions. > > regards, > R?mi > > ----- Mail original ----- >> De: "daniel smith" >> ?: "valhalla-spec-experts" >> Envoy?: Mercredi 22 Novembre 2017 06:54:55 >> Objet: Design notes for next values iteration > >> Following up on John's mail, here are some detailed notes about the design >> choices for "U types", and the concrete proposal we settled on last week. >> >> http://cr.openjdk.java.net/~dlsmith/values-notes.html >> >> John may have raised additional points I didn't cover in this document. I'll do >> another review of his mail and make any needed updates; or feel free to call >> them out. :-) >> > > ?Dan From maurizio.cimadamore at oracle.com Wed Nov 22 11:46:17 2017 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Wed, 22 Nov 2017 11:46:17 +0000 Subject: Design notes for next values iteration In-Reply-To: <6FF0E36D-92F1-4B44-B778-A3B0EE4C4A93@oracle.com> References: <6FF0E36D-92F1-4B44-B778-A3B0EE4C4A93@oracle.com> Message-ID: <44618968-5aec-e24c-8b23-4d47cd33258f@oracle.com> great writeup - I like where this is going (although I'm a but afraid of "value invasions", as we're essentially redefining semantics of existing classfiles). One thing that struck me as inconsistent is that we are enhancing almost all a- bytecodes to work with value types as well as reference types. But acmp is weird, as it always return false for values. I think I get where that is coming from (e.g. one possible way would be to do a bit-by-bit comparisons of the two values on the stack but, in doing so, we would tell as equal values that come from different classes but happen to have the same layout). The thing that concerns me is that this model really really seems to want to tell the classfile users that there's no distinction between R and Q, and your old good L opcodes can be used freely in the new world too. But acmp is a place where the bytecode writer has to stop and think: am I operating on values? If so I need something else. This seems a sticking point in the design IMHO. Maurizio On 22/11/17 05:54, Dan Smith wrote: > Following up on John's mail, here are some detailed notes about the design choices for "U types", and the concrete proposal we settled on last week. > > http://cr.openjdk.java.net/~dlsmith/values-notes.html > > John may have raised additional points I didn't cover in this document. I'll do another review of his mail and make any needed updates; or feel free to call them out. :-) > > ?Dan From brian.goetz at oracle.com Wed Nov 22 13:48:24 2017 From: brian.goetz at oracle.com (Brian Goetz) Date: Wed, 22 Nov 2017 08:48:24 -0500 Subject: abandon all U-types, welcome to L-world (or, what I learned in Burlington) In-Reply-To: <82299AAC-260E-416D-8618-A22C257A1C89@oracle.com> References: <82299AAC-260E-416D-8618-A22C257A1C89@oracle.com> Message-ID: <7f0fee0a-ae5c-e374-9f5d-dea5649dcd01@oracle.com> What's the L-world story for array subtyping?? For any R-type, R[] <: Object[].? If everything is an L type and everything is <: Object, are arrays of Q-types/primitives also subtypes of Object[]? We didn't have a story for this in QU-world either, but at least in QU-world it was believable that QFoo[] We just had a 50-hour week of face-to-face meetings by the > Valhalla VM team. We learned a lot and surprised ourselves > by coming to a consensus that a promising design for value > types uses mainly the same legacy L-type descriptors, makes > relatively little use of Q-type descriptors, and does not appear > to need a third descriptor "kind" or "mode", such as U for > universal, or R for reference-only. > > First a few highlights out of many. Fred Parain explained to us how > he has prototyped a thread-local analog of Java heaps to store value > structs in a form convenient to the interpreter. Tobias Hartmann > and Roland Westrelin (of Red Hat) explained what the compiler > prefers to see, which is (obviously) the scalarized components > of each value. The three of them have worked out detailed > rules for calling between interpreted and compiled code. > > It seems to me that other implementations of the JVM (looking > at you, IBM) will tend in similar directions, so although our > results are strongly informed by our own prototyping, we think > it is likely that they will apply to other, independent JVM > implementations. (Or are there platforms where the interpreter > will scalarize aggressively and the optimizer will prefer to > keep everything in structs? Not.) > > Karen Kinnear and the Oracle Valhalla lead, David Simms, were > there to make sure we solved the important problems and asked > the hard questions. As a special appearance, one of our spec. > gurus, Dan Smith, was there to help us make rigorous sense out > of our intuitions and hacks. > > Since we were short on language experts, we just worked in > the mode (my personal favorite) of pretending that the JVM > is the most important thing, and the Java language designers > will just have to figure out how to use it. Of course, that's an > oversimplification; the JLS and JVMS inform each other very > strongly, but it was freeing to temporarily take current thoughts > about JLS extensions as a given and vary the JVM to find > the sweet spot that would be simple to implement and supportive > to what we think we know about the Valhalla Java of the future. > > We had some long conversations about carrier types: L, Q, U, > and more, and that's what I want to write about here. We also > make significant progress in the design of crackable lambdas, > template classes, and current and future versions of condy. > We talked to Ron Pressler about kick starting Loom fibers. > But it is L-types I want to talk about here; the above is just a > sketch of the past week's environment. > > Logically speaking, we have two things we want to do, and > that unfolds to a choice between three "worlds" of up to four > distinct kinds: L/Q/U/R. L is always present because it is > a legacy model for reference types. Q is always present > because we know we need (at least sometimes) to make > a syntactic distinction between flattened values and legacy > objects. > > (Why not just always look inside the classfile? Because > the verifier cannot be expected to load a class for every > type it sees, so needs a descriptor kind character from > time to time.) > > The U kind came a year or two ago when we realized > that any-generics (and/or templates) and interfaces both > require a disjoint-union type that is neither Q nor L, but > can keep track of Q payloads (value instances) and L > payloads (nullable references to object instances), > without mixing them up. In other words, neither Q-types > nor legacy L-types are parallel class-based constructs, > and neither conveniently "sits on top" of the other; they > need a common supertype to carry them without confusion. > > Before I describe the three logically possible "worlds", > I'll add one more letter, R. An R-type is exactly a legacy > L-type, a nullable reference. Why use a separate letter? > Answer: For the same reason we introduced the other > kind letters, to preserve all the necessary distinctions > among different kinds of payloads and carrier types, > and also to talk about the explicit encoding of descriptors. > > There are three worlds we could design to hold both legacy > R-types (today's L-types) and Q-types: U-world, L-world, > and R-world. They might be notated respectively as U/QL, > L/Q, and U/QR. > > The "U-world" is what I have been mentally preparing for > for many months. It is the design where L-types, marked > as such in bytecode type descriptors, are always legacy > object references or null, and Q-types, also marked as > such in bytecodes, are always new value types. To > carry runtime payloads which may dynamically vary > between the two modes, we need a third mode, U-types, > which carry the two kinds of payloads (I hesitate to say > "values" because I want to include reference values also). > > A U-type is a disjoint union between corresponding, > similarly named Q-types and L-types. > > (Mathematically, a _disjoint union_ of C = A |_| B is no more > and no less than the sum of all elements or points comprised > by the two constituent sets A and B. The disjoint union has > nothing more: no points not in A or B. It has nothing less: > every point of C is from either A and B, but never both. > If A and B somehow look like they have a non-empty > intersection, then C is adjusted so as to keep straight > which elements are from A and which are from B.) > > The "R-world" is a copy of the "U-world", except that the > new world has no L-types at all, or rather they are renamed > as R-types. In this world, bridges would be required > between legacy bytecodes (which use L's) and Valhalla > bytecodes (which use R's for the same concept). > > We are pretty sure we don't want to live in R-world, but > it helps to think about it, since it makes the maximum > distinctions between legacy APIs and upgraded Valhalla > APIs. Any bridge from R-world to legacy code will > presumably come after a clear decision has been made > to allow the legacy code to see, under the name of L-types, > the R's from the new world, plus whatever Q's are also > allowed over the bridge to interoperate wit the old code. > > The U-world has similar need for bridges, but less extreme. > We know we will need some bridges to upgrade legacy > classes like List to use U-types (List, List The L-types of U-world just mix without effort into the legacy > L-types of legacy classes, since the same letter is used. > > The third logical choice, and the one we are now looking > at very seriously, is "L-world". (Break out the "abandon > all hope" and "Niflheim" jokes!) In L-world, we identify > (some would say conflate or confuse) the necessary > U-type which unites R-types and Q-types with the legacy > syntax "L". The Q-type syntax is *maybe* needed, but > in any case does not appear in a parallel position of > importance with the dominant L-type syntax. The R-type > syntax seems even less important; we haven't thought > of a use for it. But it is in reserve, in case we need > R-type descriptors for some corner case. > > The distinction between value types and object types > is still fundamental, as is the distinction between flat > and non-flat data. The classfile which defines any > given type unambiguously declares whether it is an > object or value type. But in L-world, the L-type > descriptors can carry both payloads. That's the > key decision before us. > > (For brevity I'll say R-type/R-value when I mean a > legacy nullable reference type/value, and Q-type/Q-value > for value type/instance. This doesn't mean that we > will need Q's and R's in the final bytecode syntax. > But they are useful concepts.) > > There are many implications from the decision to > put L-types at the top: > > * The type L-Object ("Ljava/lang/Object;") carries both > . Thus, we don't need a > new top-type. (There are objectionable properties of > L-Object which need remediation, but this was always > true, and is not a showstopper for L-world.) > > * Likewise, legacy interface types like L-Comparable > are immediately useful (without bridges) for carrying > value instances as well as object instances (and null). > > * It is possible, in some cases, that standard and user-written > collection classes will work correctly, without recompilation, > with value types. (This is a big claim, and valuable if true. > Read on.) > > * All basic operations that the JVM applies to R-types must > extend immediately and pervasively to Q-types, since it > applies them to L-type values (which may be either, > dynamically). > > * Today, simple movement of R-types is really cheap, just > a machine pointer move. That needs to be true for L-types > in L-world, or else we will get systematic performance hits > for legacy code, and new code will go slow too. > > * There are a number of object-specific operations which > the JVM applies to L-types. The most common is "acmp" > (the "==" operator for references). Those operations must > be enhanced to do something useful with values, with a > possible runtime cost to detect the distinction between > an L-type carrying a Q-value and an L-type carrying a > legacy R-value. The performance and usable semantics > of these object operations will make L-world either > a programmer's paradise or a? well you know. > > * There is no need for boxes, and they turn out to be > undesirable. Legacy types like java.lang.Integer must > be given a golden watch and a pension, somehow. > That's easy for the JVM but hard for the language, > which mandates that "(Object)(int)x" produces an > Integer rather than an "int". It seemed a good idea > at the time. > > * There is no need for a new "universal" carrier type, > since L-types do the whole job. Before the L-world > discussion, my thought has been that we want a 128-bit > U-type and a 64-bit legacy L/R-type. Somebody burst > my bubble this week, by saying that if we do that, > we may find that interpreter speeds for U-type generics > will risk a built-in performance barrier just from the > larger standard carrier type. If we JVM folks can agree > that U-types should be 64-bits (by all available means) > then it is just a simple step to rename U to L. This is > the rabbit hole that took our conversation down to L-world. > > * In L-world, the "acmp" instruction needs a very fast way > to detect Q-values. This *may* require a tag bit on the 64-bit > root value. That in turn will affect GC dynamics. There is > a delicate balance here?but we think there is a way through. > > * We probably need extra interpreter profiling to track whether > a given L-value has ever been a Q-value or an R-value, > dynamically. Today we do null tracking on some instructions. > This probably needs to be upgraded to null/Q/R tracking, > and perhaps on additional instructions such as "acmp". > > * There are a number of ways to assign semantics to > an object-like L operation when it encounters a Q-value. > This will require additional mails, but I think we have > identified about a half dozen models, of which one or two > seem to be very promising: Providing both useful semantics > and amenable to optimization. > > * One residual use for Q-types is in the declaration of > instance fields. In order to avoid loading *all* classfiles > of types mentioned in field declarations, a classfile which > declares a flattened field will need to include enough > information to allow the classfile loader to load *only* > those fields marked as requiring flattening. There are > at least two ways to do this: Use a Q-type descriptor > syntax *only* for field declarations, as today. Or, > require the ACC_VALUE bit on field declarations which > are supposed to be flattened. > > * As we were able to dispense with boxes, we may also > dispense with non-flattened value types. In that case, > the translation strategy might emit an ACC_VALUE bit > or Q-type on a field if and only if the classfile for the > field's type defines it with ACC_VALUE. The JVM will > have to support non-flattened values in L-Object fields, > of course. > > * If the system uses a thread-local store for value structures > (to avoid heap traffic), a store barrier will have to quickly > detect Q-types that are inside the thread and reallocate > them to the heap, when they are first stored to the heap > (e.g., as an element of an L-Object array). > > * The Q-type modifier *might* be useful in some settings > to guarantee, in a verifiable way, that a given value is > *not* an R-type, *not* null, and *not* modifiable; TBD. > > * The R-type modifier *might* be useful in some settings > to guarantee, in a verifiable way, that a given value is > *not* a Q-type, and *does* have an object identity or > is null. This is also TBD. > > * For best compatibility with legacy code, combined with > diagnosability of anti-value algorithms like IdentityHashMap, > the "acmp" instruction should return false unconditionally > if either operand is a Q-value (punting to the following > Object.equals call), and other object-like operations > such as identityHashCode and monitorenter must throw > errors in the JVM. (In the language errors and warnings > will be appropriate.) > > * New operations are needed for substitutability checks > which generalize reference equality and hashcode. > These can be system methods, and do not need to be > loaded onto either new or old bytecodes. > > * We will almost certainly need to make primitives > retroactively values. This means "int" all along has > really been Q-int (in the JVM) and is a real subtype > of L-Object. > > * Covariant array subtyping only works for R-types. > So both int[] and DoubleComplex[] are *not* subtypes > of Object[], even though int and DoubleComplex *are* > subtypes of Object. > > * From some points of view (legacy code), Q-values > are masked invaders coming into the home of code > which expected to work only on R-values. Changing > L-descriptors to encompass Q-values opens such > code to potentially risky new behaviors. Is it safe? > Shouldn't we just have boxes to mediate values > in such settings? It depends on the code, really. > > There's more, but this is enough for one message. > > The L-world is very attractive: No bridges or boxes, > legacy code is value-enabled, and we get all the > flattening we need. > > We need to do some experiments: Can we afford > the extra Q-checks on acmp and storage to the heap? > Will legacy algorithms really work on masked but not > boxed values? Do other JVM implementations experience > similar trade-offs, or is this only a HotSpot-centric set > of compromises? Can we really avoid all those new > descriptors and bridges!!?? > > Let's talk! > > ? John > > P.S. Dan, you should send out your notes on U-types. From karen.kinnear at oracle.com Wed Nov 22 15:00:29 2017 From: karen.kinnear at oracle.com (Karen Kinnear) Date: Wed, 22 Nov 2017 10:00:29 -0500 Subject: Nestmates JVMS Overriding 5.4.5 request In-Reply-To: <4612B6B3-995C-47C7-87F4-A4B402919FC7@oracle.com> References: <24DD2E51-80CD-43F4-8E30-95043F016691@oracle.com> <4612B6B3-995C-47C7-87F4-A4B402919FC7@oracle.com> Message-ID: <17F00A9D-CB6B-4112-A689-46E11823F08E@oracle.com> Dan S, You are correct .The JVMS wording is accurate in terms of behavior, since the bullets represent the resolved method and you already have bullets for public and protected, so this bullet does only need to cover package private. So I withdraw the request for a change. From an implementation standpoint, when we pre-cache the selection information hotspot at least does not track the resolved method information, so we always perform the transitive overriding check. I think Dan H and Graham?s request to add some examples would be valuable. Specifically they asked me a couple of questions that could use clarifying: 1) is it sufficient to check that you override the resolved method, or that you override the resolved method or direct superclass? - No - see the example below. 2) Does each package private method need its own vtable slot? Potentially yes. Since we pre-calculate the size it is much cleaner that way. I could imagine cases in which the only methods in the overriding chain so far were all package private in the same package - and I don?t have a counter-example of that in which you would have to have a separate entry. We are still going to use a separate entry since the complexity budget is already high. Here is an example that might be useful: P1.A: PP m() P1.B extends A, Pub m() P1.C extends B, PP m() P2.D extends C, PP m() P2.D.m should override P1.A.m P2.D.m should not override P1.C.m Correct me folks if this is inaccurate, thanks, Karen > On Nov 16, 2017, at 10:40 AM, Dan Smith wrote: > >> On Nov 10, 2017, at 12:48 PM, Karen Kinnear > wrote: >> >> Dan Smith - this is a request for a change to one line of the Nestmates JVMS 5.4.5 Overriding. >> The rest of the wording for 5.4.5 is an improvement, so thank you. >> >> Dan Heidinga, Graham Chapman (IBM) and I met to discuss the JVM implementations of >> JVMS 5.4.5 Overriding, specifically to discuss the expectations relative to the >> bullet describing transitive overriding that was added for CFV 51 in JDK7. >> >> The JVMS 9 bullet says: >> mC overrides a method m' (m' distinct from mC and mA) such that m' overrides mA. >> >> The proposed NestMates bullet says: >> mC overrides a method m' (m' distinct from mC and mA) such that m' overrides mA mA is marked neither ACC_PUBLIC nor ACC_PROTECTED nor ACC_PRIVATE, and, where mC is declared in a class C and mA is declared in a class A, there exists a method mB declared in a class B, such that C is a subclass of B, B is a subclass of A, mC can override mB, and mB can override mA. >> From examining a number of test cases, we would like to request that the updated version be modified to remove the restriction on mA as package private. We need to perform what I call transitive overriding checks as long as mA is not ACC_PRIVATE. > > The intent is not to change the meaning, but to clarify that the check only needs to be performed when mA is package-access. Notice that this appears in a list of or'ed conditions, and the first two are "mA is marked ACC_PUBLIC" and "mA is marked ACC_PROTECTED". So in those cases, the enclosing "One of the following is true" is already true. > > If, in fact, some sort of transitive analysis is needed, then the problem is deeper than just the access flags checked in this bullet. > > Could you supply an example where the current rules define unwanted/unimplemented behavior? > >> Dan H also suggested it would be very helpful to all of us if you could add some non-normative examples in the spec here. Examples that might be useful would be: > > Sure, some discussion about package access would be helpful. I can do so. > > ?Dan From daniel.smith at oracle.com Wed Nov 22 15:32:36 2017 From: daniel.smith at oracle.com (Dan Smith) Date: Wed, 22 Nov 2017 08:32:36 -0700 Subject: Design notes for next values iteration In-Reply-To: <1970438356.1129084.1511343756393.JavaMail.zimbra@u-pem.fr> References: <6FF0E36D-92F1-4B44-B778-A3B0EE4C4A93@oracle.com> <1970438356.1129084.1511343756393.JavaMail.zimbra@u-pem.fr> Message-ID: <77AF18B7-866F-4411-97CD-33009F937EEC@oracle.com> > On Nov 22, 2017, at 2:42 AM, Remi Forax wrote: > > I think we do not need Q types, Q types are use site annotations, and here we want declaration site annotations (let say that this class is a value class, the ACC_VALUE). > > If we have no Q type, it means that R types and U types are the same thing, everything is a U type. In term of migration, it means that L types need to have their semantics extended to work as U types. Yeah. We discussed this and I mentioned it in the notes, but it could use a deeper exploration at some point. As a minimum, there probably needs to be *some* way to indicate that a field can be flattened and may not be null. If a field has a U type, the referenced class would have to be loaded before we could tell if it's referencing a value class or not, and that's costly. ?Dan From daniel.smith at oracle.com Wed Nov 22 15:46:58 2017 From: daniel.smith at oracle.com (Dan Smith) Date: Wed, 22 Nov 2017 08:46:58 -0700 Subject: Design notes for next values iteration In-Reply-To: <44618968-5aec-e24c-8b23-4d47cd33258f@oracle.com> References: <6FF0E36D-92F1-4B44-B778-A3B0EE4C4A93@oracle.com> <44618968-5aec-e24c-8b23-4d47cd33258f@oracle.com> Message-ID: > On Nov 22, 2017, at 4:46 AM, Maurizio Cimadamore wrote: > > One thing that struck me as inconsistent is that we are enhancing almost all a- bytecodes to work with value types as well as reference types. But acmp is weird, as it always return false for values. I think I get where that is coming from (e.g. one possible way would be to do a bit-by-bit comparisons of the two values on the stack but, in doing so, we would tell as equal values that come from different classes but happen to have the same layout). > > The thing that concerns me is that this model really really seems to want to tell the classfile users that there's no distinction between R and Q, and your old good L opcodes can be used freely in the new world too. But acmp is a place where the bytecode writer has to stop and think: am I operating on values? If so I need something else. This seems a sticking point in the design IMHO. I tend to agree. Bit-by-bit comparison is not quite the right thing for fieldwise equality, because non-flattened fields may have different pointers but be equal values. So it's a little more complex. (FWIW, we did conclude that floating-point values should be tested with bit equality, even though that doesn't match 'dcmp', because the bits are directly observable.) I think the motivation for avoiding this in 'acmp' is to keep it as fast as possible, even in the value cases. There's a hope that a significant number of use cases, including in legacy code, have the form "x == y || (x != null && x.equals(y))", and in that case, the left test is purely an optimization (assuming a well-behaved 'equals'). One purpose of the prototype is to explore this hypothesis. Where you need the full field-equality test, 'ucmp' (prototyped as the 'substitutable' method) is the operation you want. That's probably how I'd interpret '==' in Java. ?Dan From forax at univ-mlv.fr Wed Nov 22 16:42:04 2017 From: forax at univ-mlv.fr (forax at univ-mlv.fr) Date: Wed, 22 Nov 2017 17:42:04 +0100 (CET) Subject: Design notes for next values iteration In-Reply-To: <77AF18B7-866F-4411-97CD-33009F937EEC@oracle.com> References: <6FF0E36D-92F1-4B44-B778-A3B0EE4C4A93@oracle.com> <1970438356.1129084.1511343756393.JavaMail.zimbra@u-pem.fr> <77AF18B7-866F-4411-97CD-33009F937EEC@oracle.com> Message-ID: <58998477.1445927.1511368924905.JavaMail.zimbra@u-pem.fr> ----- Mail original ----- > De: "daniel smith" > ?: "Remi Forax" > Cc: "valhalla-spec-experts" > Envoy?: Mercredi 22 Novembre 2017 16:32:36 > Objet: Re: Design notes for next values iteration >> On Nov 22, 2017, at 2:42 AM, Remi Forax wrote: >> >> I think we do not need Q types, Q types are use site annotations, and here we >> want declaration site annotations (let say that this class is a value class, >> the ACC_VALUE). >> >> If we have no Q type, it means that R types and U types are the same thing, >> everything is a U type. In term of migration, it means that L types need to >> have their semantics extended to work as U types. > > Yeah. We discussed this and I mentioned it in the notes, but it could use a > deeper exploration at some point. > > As a minimum, there probably needs to be *some* way to indicate that a field can > be flattened and may not be null. If a field has a U type, the referenced class > would have to be loaded before we could tell if it's referencing a value class > or not, and that's costly. The L type reference a class that is tagged with ACC_VALUE, so the VM knows if a type is a value type or not. BTW, an array of a class which is tagged with ACC_VALUE is not necessary flatten, again the array might be too big, so it means that being flatten or not is a dynamic property, thus it should not be encoded in the VM type system as a Q-type. > > ?Dan R?mi From forax at univ-mlv.fr Wed Nov 22 16:52:37 2017 From: forax at univ-mlv.fr (Remi Forax) Date: Wed, 22 Nov 2017 17:52:37 +0100 (CET) Subject: Design notes for next values iteration In-Reply-To: <1970438356.1129084.1511343756393.JavaMail.zimbra@u-pem.fr> References: <6FF0E36D-92F1-4B44-B778-A3B0EE4C4A93@oracle.com> <1970438356.1129084.1511343756393.JavaMail.zimbra@u-pem.fr> Message-ID: <590729318.1451835.1511369557400.JavaMail.zimbra@u-pem.fr> Hum, may i should have start with that, the part that is weak in the spec IMO is the relation between a L type and its corresponding Q type. If a Q type is a subtype of the corresponding L type, it means that one can call a method that takes a L type in parameter with a Q type as argument, but it doesn't work for the return type, if a return type is a L type, it can not be assigned to a Q type (BTW, for the same reason why in the JLS we have auto-boxing/unboxing an not only one side). So converting a class like Optional to a value type is not backward compatible. The only possible way in my opinion is if the R Optional and Q Optional occupy the same lattice, i.e. are a L Optional. This is what i have wanted to say with my first paragraph below. R?mi ----- Mail original ----- > De: "Remi Forax" > ?: "daniel smith" > Cc: "valhalla-spec-experts" > Envoy?: Mercredi 22 Novembre 2017 10:42:36 > Objet: Re: Design notes for next values iteration > Interesting ! > here is my current view of the problem. > > Lets say we are in the R/Q/U world, we want U interface to abstract over a > reference type or a value type, we want UObject because object is basically an > interface, we also want U object like UOptional to be able to painlessly move a > reference type to a value type and vice-versa (no bridge-o-matic). > > But why we want Q type BTW, we want Q type because > 1/ you can not apply the same opcodes on Q types and R types (more or less, some > ops are not defined on Q types) > 2/ we want the VM to flatten array and not box Q type. > 3/ the VM need a clean separation between Q type and R type > > For 1/ we can extends the semantics of the opcodes that only works on R types to > work on Q types, we need that anyway for U types and at worst a valid semantics > is to throw an exception > For 2/ even if we sak for flatten array or no box, there are case where the VM > will not flatten the array or the q type, so it's like a hint than a strict > requirement. > For 3/ the current prototype has shown that the VM is able to box/buffer value > types where needed, so no ! > > I think we do not need Q types, Q types are use site annotations, and here we > want declaration site annotations (let say that this class is a value class, > the ACC_VALUE). > > If we have no Q type, it means that R types and U types are the same thing, > everything is a U type. In term of migration, it means that L types need to > have their semantics extended to work as U types. > > Obviously, the devil is in the details, but i think from the VM POV, we should > extends L types to behave like U types. > From Java the language POV, we may want to not compile when a variable is typed > with value type is assigned to null, same with generics, you may want to > declare a type variable any but from the VM POV, this should be possible and > either throw an exception, return false, etc. > > Let's make the VM powerful and let's Java the language helps users to not have > too many exceptions. > > regards, > R?mi > > ----- Mail original ----- >> De: "daniel smith" >> ?: "valhalla-spec-experts" >> Envoy?: Mercredi 22 Novembre 2017 06:54:55 >> Objet: Design notes for next values iteration > >> Following up on John's mail, here are some detailed notes about the design >> choices for "U types", and the concrete proposal we settled on last week. >> >> http://cr.openjdk.java.net/~dlsmith/values-notes.html >> >> John may have raised additional points I didn't cover in this document. I'll do >> another review of his mail and make any needed updates; or feel free to call >> them out. :-) >> > > ?Dan From daniel.smith at oracle.com Wed Nov 22 17:24:22 2017 From: daniel.smith at oracle.com (Dan Smith) Date: Wed, 22 Nov 2017 10:24:22 -0700 Subject: Design notes for next values iteration In-Reply-To: <58998477.1445927.1511368924905.JavaMail.zimbra@u-pem.fr> References: <6FF0E36D-92F1-4B44-B778-A3B0EE4C4A93@oracle.com> <1970438356.1129084.1511343756393.JavaMail.zimbra@u-pem.fr> <77AF18B7-866F-4411-97CD-33009F937EEC@oracle.com> <58998477.1445927.1511368924905.JavaMail.zimbra@u-pem.fr> Message-ID: <5C618C1E-AF2E-4C4C-B374-51FCFBEEB786@oracle.com> > On Nov 22, 2017, at 9:42 AM, forax at univ-mlv.fr wrote: > > > > ----- Mail original ----- >> De: "daniel smith" >> ?: "Remi Forax" >> Cc: "valhalla-spec-experts" >> Envoy?: Mercredi 22 Novembre 2017 16:32:36 >> Objet: Re: Design notes for next values iteration > >>> On Nov 22, 2017, at 2:42 AM, Remi Forax wrote: >>> >>> I think we do not need Q types, Q types are use site annotations, and here we >>> want declaration site annotations (let say that this class is a value class, >>> the ACC_VALUE). >>> >>> If we have no Q type, it means that R types and U types are the same thing, >>> everything is a U type. In term of migration, it means that L types need to >>> have their semantics extended to work as U types. >> >> Yeah. We discussed this and I mentioned it in the notes, but it could use a >> deeper exploration at some point. >> >> As a minimum, there probably needs to be *some* way to indicate that a field can >> be flattened and may not be null. If a field has a U type, the referenced class >> would have to be loaded before we could tell if it's referencing a value class >> or not, and that's costly. > > The L type reference a class that is tagged with ACC_VALUE, so the VM knows if a type is a value type or not. The VM knows *after the class is loaded*. Like I said, that's costly, and forces class loading where it didn't used to happen, because every L type *might* refer to an ACC_VALUE class. In the proposed design, 'Q' types may be flattened, and 'L' types never are. > BTW, an array of a class which is tagged with ACC_VALUE is not necessary flatten, again the array might be too big, so it means that being flatten or not is a dynamic property, thus it should not be encoded in the VM type system as a Q-type. 'Q' doesn't mean "flattened", it means "could be flattened if the VM chooses". It's not necessarily a dynamic property?if you've loaded the class, you know whether it will be flattened or not?but it's determined statically fairly late, probably not until the interpret has run. ?Dan From john.r.rose at oracle.com Wed Nov 22 18:48:18 2017 From: john.r.rose at oracle.com (John Rose) Date: Wed, 22 Nov 2017 10:48:18 -0800 Subject: abandon all U-types, welcome to L-world (or, what I learned in Burlington) In-Reply-To: <7f0fee0a-ae5c-e374-9f5d-dea5649dcd01@oracle.com> References: <82299AAC-260E-416D-8618-A22C257A1C89@oracle.com> <7f0fee0a-ae5c-e374-9f5d-dea5649dcd01@oracle.com> Message-ID: <77053228-4E2D-4AF4-A3BA-31EA83E06BD1@oracle.com> On Nov 22, 2017, at 5:48 AM, Brian Goetz wrote: > > What's the L-world story for array subtyping? For any R-type, R[] <: Object[]. If everything is an L type and everything is <: Object, are arrays of Q-types/primitives also subtypes of Object[]? > > We didn't have a story for this in QU-world either, but at least in QU-world it was believable that QFoo[] References: <6FF0E36D-92F1-4B44-B778-A3B0EE4C4A93@oracle.com> <1970438356.1129084.1511343756393.JavaMail.zimbra@u-pem.fr> <1060469278.1179775.1511348728201.JavaMail.zimbra@u-pem.fr> Message-ID: <3B03E950-70A9-424E-AB51-92B9C76FD8E5@oracle.com> Good comments. On Nov 22, 2017, at 3:05 AM, Remi Forax wrote: > > Primitive types: > primitive types are seen by the VM as value types, int.class.isValueType() returns true. > there is no need to see those class in the JDK, so like currently there are synthetize by the VM. So storing an int as an Object is equivalent to box/buffer the int like any value type. Yes, I think this will work pretty easily in the JVM. The harder adjustment is in the language, where the cast (Object)(int)x produces an (Integer), where the JVM is now willing to produce a buffered (int). > Performance: > If we have U types everywhere, doesn't it introduce perf issues, by example, what about acmp now having to test if it operates on value types or reference types. > > There are two facts that save us here, with acmp as example, > - first for one execution, a type is either a value type or a reference type, so it enables the kind of optimization we do with quick opcodes, i.e. the slow path is if you do not know if the type is a reference type or a value type, but once you have do the test for a type, you can install a fast path for all acmp on the same type. acmp is so fast that even detecting values will cause noticeable overheads. I think the JIT will end up hoisting value detection as much as possible. In some systems, we might consider hoisting the distinction into object references, all the way through the heap and GC. > - if the cost is too important we can have say that acmp will always do a pointer check (here we are saved by the fact that in hotspot a value type is also a machine word pointer) and introduces a new opcode, ucmp that have a bigger overhead. That acmp is exactly what I mean by a Heisenbox comparison. If we could get comfortable with Heisenboxes, the acmp problem would go away. But the cost is great: the expression "a==a" yields an essentially indeterminate result. I don't know how to make that acceptable as a JVM behavior. So "a==a" yielding always false for values, or (more expensively) "a==a" triggering a fieldwise substitutability check for values, seems like the place we have to land on. > The semantics of acmp on value type in that case is garbage, but not less garbage that the current boxing semantics specified by the JLS. I wish that were true, but there's a bit more garbage with Heisenboxes. This contrived but legal program does not always return true for value types: boolean paradox(DoubleComplex x) { DoubleComplex y = x; boolean p1 = (x == y); // might be true or false doSomething(); // might deopt or JIT boolean p2 = (x == y); // independently might be true or false return p1 == p2; } As an old Lisper, I am inclined to turn a blind eye to that kind of thing; that's what EQ does and you use EQV if you need a value-sensitive test. (EQV is your ucmp, EQ is acmp.) But the JVM has a higher standard for reproducibility. ? John From forax at univ-mlv.fr Wed Nov 22 19:53:51 2017 From: forax at univ-mlv.fr (forax at univ-mlv.fr) Date: Wed, 22 Nov 2017 20:53:51 +0100 (CET) Subject: Design notes for next values iteration In-Reply-To: <5C618C1E-AF2E-4C4C-B374-51FCFBEEB786@oracle.com> References: <6FF0E36D-92F1-4B44-B778-A3B0EE4C4A93@oracle.com> <1970438356.1129084.1511343756393.JavaMail.zimbra@u-pem.fr> <77AF18B7-866F-4411-97CD-33009F937EEC@oracle.com> <58998477.1445927.1511368924905.JavaMail.zimbra@u-pem.fr> <5C618C1E-AF2E-4C4C-B374-51FCFBEEB786@oracle.com> Message-ID: <1616145580.1508820.1511380431084.JavaMail.zimbra@u-pem.fr> Hi Dan, > De: "daniel smith" > ?: "Remi Forax" > Cc: "valhalla-spec-experts" > Envoy?: Mercredi 22 Novembre 2017 18:24:22 > Objet: Re: Design notes for next values iteration >> On Nov 22, 2017, at 9:42 AM, [ mailto:forax at univ-mlv.fr | forax at univ-mlv.fr ] >> wrote: >> ----- Mail original ----- >>> De: "daniel smith" < [ mailto:daniel.smith at oracle.com | daniel.smith at oracle.com >>> ] > >>> ?: "Remi Forax" < [ mailto:forax at univ-mlv.fr | forax at univ-mlv.fr ] > >>> Cc: "valhalla-spec-experts" < [ mailto:valhalla-spec-experts at openjdk.java.net | >>> valhalla-spec-experts at openjdk.java.net ] > >>> Envoy?: Mercredi 22 Novembre 2017 16:32:36 >>> Objet: Re: Design notes for next values iteration >>>> On Nov 22, 2017, at 2:42 AM, Remi Forax < [ mailto:forax at univ-mlv.fr | >>>> forax at univ-mlv.fr ] > wrote: >>>> I think we do not need Q types, Q types are use site annotations, and here we >>>> want declaration site annotations (let say that this class is a value class, >>>> the ACC_VALUE). >>>> If we have no Q type, it means that R types and U types are the same thing, >>>> everything is a U type. In term of migration, it means that L types need to >>>> have their semantics extended to work as U types. >>> Yeah. We discussed this and I mentioned it in the notes, but it could use a >>> deeper exploration at some point. >>> As a minimum, there probably needs to be *some* way to indicate that a field can >>> be flattened and may not be null. If a field has a U type, the referenced class >>> would have to be loaded before we could tell if it's referencing a value class >>> or not, and that's costly. >> The L type reference a class that is tagged with ACC_VALUE, so the VM knows if a >> type is a value type or not. > The VM knows *after the class is loaded*. Like I said, that's costly, and forces > class loading where it didn't used to happen, because every L type *might* > refer to an ACC_VALUE class. > In the proposed design, 'Q' types may be flattened, and 'L' types never are. John and Frederic said the same thing during the valhalla meeting, 3 against one, ok, i get it. >> BTW, an array of a class which is tagged with ACC_VALUE is not necessary >> flatten, again the array might be too big, so it means that being flatten or >> not is a dynamic property, thus it should not be encoded in the VM type system >> as a Q-type. > 'Q' doesn't mean "flattened", it means "could be flattened if the VM chooses". > It's not necessarily a dynamic property?if you've loaded the class, you know > whether it will be flattened or not?but it's determined statically fairly late, > probably not until the interpret has run. My point is that for array (not fields), you can delay the decision until you create the array, so very late when at that point you have to load the class anyway. John said this may cause an issue with the verifier. For fields, i like what's you propose i.e tag the field with a modifier saying "flatennable", so the verifier and the interpreter can take the decision earlier without loading the class. Decoupling the type from the property, it's not a Q type anymore, it's a flattenable field that will check that the type when loaded is a value type, avoid to have to bridge between R types and Q types, so it seems the right way to deal with this issue. For array, if there is an issue with the verifier, we may still need Q type for that case, to encode the fact that the component of an array make the array "flatenable". > ?Dan R?mi From forax at univ-mlv.fr Wed Nov 22 20:06:01 2017 From: forax at univ-mlv.fr (Remi Forax) Date: Wed, 22 Nov 2017 21:06:01 +0100 (CET) Subject: abandon all U-types, welcome to L-world (or, what I learned in Burlington) In-Reply-To: <77053228-4E2D-4AF4-A3BA-31EA83E06BD1@oracle.com> References: <82299AAC-260E-416D-8618-A22C257A1C89@oracle.com> <7f0fee0a-ae5c-e374-9f5d-dea5649dcd01@oracle.com> <77053228-4E2D-4AF4-A3BA-31EA83E06BD1@oracle.com> Message-ID: <1856147797.1540399.1511381161572.JavaMail.zimbra@u-pem.fr> > De: "John Rose" > ?: "Brian Goetz" > Cc: "valhalla-spec-experts" > Envoy?: Mercredi 22 Novembre 2017 19:48:18 > Objet: Re: abandon all U-types, welcome to L-world (or, what I learned in > Burlington) > On Nov 22, 2017, at 5:48 AM, Brian Goetz < [ mailto:Brian.Goetz at Oracle.COM | > Brian.Goetz at Oracle.COM ] > wrote: >> What's the L-world story for array subtyping? For any R-type, R[] <: Object[]. >> If everything is an L type and everything is <: Object, are arrays of >> Q-types/primitives also subtypes of Object[]? >> We didn't have a story for this in QU-world either, but at least in QU-world it >> was believable that QFoo[] > there's no syntactic difference between L-uses and Q-uses. (And even less so >> when we might migrate code from L to Q.) > We were just talking about this in the concall with IBM. > Field and array-element flattening are the two places where > the Q/R distinction provides a crucial hint that flattening > *may* (not always *must*) occur. This hint is crucial because > it is statically visible *before* all class files are loaded. > The instance layout algorithm and the verifier both need > to run before all class files are loaded (because of > circularities, also performance). In U-world, the > letter 'Q' in a static descriptor tells the instance layout > generator to load the field class and extract sub-layout. > It also tells the verifier not to assume a covariantly > compatible layout relative to the type Object[]. > If we don't keep a few Q's around for old times' > sake, we will need to signal these subtle difference > some other way, with an ACC_FLATTENABLE > bit on fields and a special "[@" syntax variant > (pick your letter, maybe "Q") for array descriptors. > I think of this as the "residual Q problem", of > finding offices for the few remaining occurrences > of Q that do real work in L-world. > (The user-visible distinction of flat vs. legacy > arrays was one influence that led us towards > user-visible box types. I'd like to resist that this > time around. Perhaps "[@" is a syntax that is > mutually exclusive with plain "[". And there is > a showdown when such an array is created, > so that the descriptor has a "@" if and only if > the loaded element type is in fact a Q-type. > It's a move that of class loader constraints.) in that case, i prefer '{' instead of '@', the angle square bracket is the classical bracket and the curly brace is the fancy bracket. and with my ASM hat, introducing '{' in the ASM code is far easier than introducing 'Q'. > I am not opposed to allowing the existing Q-syntax > for descriptors, in a limited number of places, to > solve those residual problems. That thought leads > me to try on the idea (which Remi discourages) > that perhaps Q-narrowings of some method > descriptors are useful (requiring bridges just > like today's generics). A Q-narrowing of a > class means: No nulls here, identity is not > observable, and value-based semantics are > in force, including unmodifiability. Not a bad > set of guarantees; maybe that's a job for Q's > rather than an invisible type profile. for the record, i'm against because unlike generics where you have to introduce bridge when you actually use generics, here you have to introduce bridges in an already existing code. > BTW, an Ljava/util/List; would not pass through > a Qjava/util/List; descriptor, unless List.copyOf > were applied to it first. The VM and java.base > can conspire to provide a curated set of Q-able > types with enforced value or value-based behavior. > ? John R?mi From daniel.smith at oracle.com Thu Nov 23 06:20:03 2017 From: daniel.smith at oracle.com (Dan Smith) Date: Wed, 22 Nov 2017 23:20:03 -0700 Subject: abandon all U-types, welcome to L-world (or, what I learned in Burlington) In-Reply-To: <7f0fee0a-ae5c-e374-9f5d-dea5649dcd01@oracle.com> References: <82299AAC-260E-416D-8618-A22C257A1C89@oracle.com> <7f0fee0a-ae5c-e374-9f5d-dea5649dcd01@oracle.com> Message-ID: <7BEB0496-537B-4B69-91D2-673CEFA07DF5@oracle.com> > On Nov 22, 2017, at 6:48 AM, Brian Goetz wrote: > > What's the L-world story for array subtyping? For any R-type, R[] <: Object[]. If everything is an L type and everything is <: Object, are arrays of Q-types/primitives also subtypes of Object[]? > > We didn't have a story for this in QU-world either, but at least in QU-world it was believable that QFoo[]