From kevinb at google.com Mon May 2 23:34:42 2016 From: kevinb at google.com (Kevin Bourrillion) Date: Mon, 2 May 2016 16:34:42 -0700 Subject: Value types questions & comments In-Reply-To: <2127B453-99E7-4879-9097-F46BE46DB0C3@oracle.com> References: <2127B453-99E7-4879-9097-F46BE46DB0C3@oracle.com> Message-ID: Still thinking On Mon, Apr 11, 2016 at 4:13 PM, Brian Goetz wrote: For example, I see no reason why `int` can?t implement Comparable or > Serializable ... > > We might even take this further ? by actually describing `int` with a > source file (public native class int implements Comparable { ? }) which > might try and smooth out some of the differences, but I wouldn?t hold out a > lot of hope for this being super successful. > I now think that neither of these bits will accomplish much. Implemented types only mean anything to the *boxed* form, but int doesn't even use "that" kind of boxing; it must continue to be boxed to java.lang.Integer as always, and nor is there a way to make Integer "extend Box", which would have bought us a few things, because it has to extend Number (which tragically is not an interface). So, I think you're looking at a taxonomy something like this: Types +-- Reference types +-- Value types +-- Primitives / "builtin value types" +-- "Custom value types" ... and basically we need to diligently use the term "custom value types" instead of "value types" whenever talking about information that doesn't apply to primitives. But, I think that this will also not work out well, because it belies the many commonalities between kinds 1 and 3 that primitives don't share (they have a source file, they get loaded and initialized, they have methods....). I'm back once again to the idea that there are just plain *three different kinds of types*, where value types are largely but not entirely a hybrid of the other two. Now, sure, the more *areas* (such as specialization) in which we can make primitives and value types work identically the better. -- Kevin Bourrillion | Java Librarian | Google, Inc. | kevinb at google.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.r.rose at oracle.com Mon May 16 18:16:26 2016 From: john.r.rose at oracle.com (John Rose) Date: Mon, 16 May 2016 11:16:26 -0700 Subject: Value types questions & comments In-Reply-To: References: <2127B453-99E7-4879-9097-F46BE46DB0C3@oracle.com> Message-ID: <9A1704E9-8FA9-43D2-9238-A3DAACC40641@oracle.com> On May 2, 2016, at 4:34 PM, Kevin Bourrillion wrote: > ? > On Mon, Apr 11, 2016 at 4:13 PM, Brian Goetz > wrote: > > For example, I see no reason why `int` can?t implement Comparable or Serializable ... > > We might even take this further ? by actually describing `int` with a source file (public native class int implements Comparable { ? }) which might try and smooth out some of the differences, but I wouldn?t hold out a lot of hope for this being super successful. > > I now think that neither of these bits will accomplish much. Implemented types only mean anything to the *boxed* form, Matters look better to me on these points than you think. An interface is a promise about a set of methods in the concrete class that implements that interface. This all about the concrete class, and nothing to do with any future use of invokeinterface. This part of interfaces makes them useful even for ret-conning "int" (if we choose to endow "int" with methods, for better regularity). To take it from the other end, suppose interfaces were only about invokeinterface behavior. Even then, behavioral parity between values and their boxes would require *some* sort of access to interface methods on unboxed values. (Maybe it's *implemented* by boxing under the hood?but maybe not.) Finally, since interfaces can carry behavior in the form of default methods (as well as contracts with other types), we can use interface subtyping on values to express polymorphic algorithms that operate (using bounds-polymorphism) on those values. Depending on how we formulate things, such algorithms need not commit the JVM or JLS to perform the algorithms on boxed values. I think the above would accomplish quite a lot; don't you? > but int doesn't even use "that" kind of boxing; it must continue to be boxed to java.lang.Integer as always, and nor is there a way to make Integer "extend Box", which would have bought us a few things, because it has to extend Number (which tragically is not an interface). Parity of Integer with new boxes is a thorny problem. But I don't think we are out of options yet. We might choose to make Integer boxes look more like new new boxes, or vice versa. We might choose to distinguish between user-created Integer instances and system-created ones (reserving better rules for the latter). And there are several other things we might try. > So, I think you're looking at a taxonomy something like this: > > Types > +-- Reference types > +-- Value types > +-- Primitives / "builtin value types" > +-- "Custom value types" > > ... and basically we need to diligently use the term "custom value types" instead of "value types" whenever talking about information that doesn't apply to primitives. It's true that there will be small observable differences between primitives ("legacy value types"!) and "custom value types". So there will be such a taxonomy, but it will be relevant only to the extent that the "small" differences will cost users some of their attention. We hope it will be negligible for most purposes. > But, I think that this will also not work out well, because it belies the many commonalities between kinds 1 and 3 that primitives don't share (they have a source file, they get loaded and initialized, they have methods....). For some users, it won't work out well. I can say from experience that every change ruins the language for someone (at least until they learn to live with it). Our task is to make it work, for most users, better than the previous version of the language. I'm hopeful we can do this. > I'm back once again to the idea that there are just plain three different kinds of types, where value types are largely but not entirely a hybrid of the other two. Now, sure, the more areas (such as specialization) in which we can make primitives and value types work identically the better. The taxonomy which is most interesting to me is the disjoint union between "val" and "ref", which may be called "any". That is, some things are vals, other things are refs, nothing is both a val and a ref, unless it is an "any" which accepts both. The difference between classic type parameters and new ones is one is limited to refs, while the other accepts both vals and refs. Under this fundamental distinction, we will sometimes observe an additional distinction between legacy value types and custom value types. How often that happens depends on how well we do the job at hand. So, whenever we notice a potential difference between legacy value types and custom value types, we should (a) describe it carefully, and (b) consider how to mitigate it in the user experience. E.g., custom types have methods and other named API points. We can mitigate the difference with "int" by imputing API points to primitives. I think the right way to do this is to give primitives some interfaces to implement. This will allow generic algorithms to operate on both legacy and custom value types, an obvious win for numerical or other algebraic algorithms. And so on? ? John Independently of all of the above, and responding to a previous point you made, we are moving parametric polymorphism from the source type system to the runtime type system, by reifying all val bindings of type variables at runtime. This seems necessary in order to extend genericity uniformly to vals, starting with primitives. That introduces a different set of cross-cutting complexities, because now there are runtime subtypes which are not subclasses, just like in the static types of the language. (See JLS 4.10.2, generic type subtyping as affected by type variable containment, 4.5.1.) This is being surfaced reflectively as species, a distinct and necessary refinement of class. Given that a similar kind of subtype relation was already in the language, we can expect users can learn to know what to do when they see one at runtime. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.goetz at oracle.com Mon May 16 18:20:38 2016 From: brian.goetz at oracle.com (Brian Goetz) Date: Mon, 16 May 2016 14:20:38 -0400 Subject: Value equality (was: Value types questions and comments) In-Reply-To: References: <2127B453-99E7-4879-9097-F46BE46DB0C3@oracle.com> Message-ID: Kevin say: > However, it would be very sad if it does not solve a second problem at > the same time, because it can. Even Java developers who are content > with the performance foibles of their existing "value-based classes" > are constantly irritated by the burden and bug-prone nature of > writing/maintaining such classes. > We have two choices in front of us regarding equality on value types: 1. What does `==` on values do? Choices include a bitwise comparison, a componentwise comparision (identical to bitwise unless there are float members), or invoking the `equals()` method. 2. What is the default for the `equals()` method? The obvious choice is a componentwise comparision, but Kevin has (strongly) suggested we consider a deep comparison, calling .equals() on any reference members. (Note that the two choices collapse to the same thing when there are no reference members.) For values whose members are value-ish objects themselves (e.g., String, Optional, LocalDateTime), the deep-equals is obviously appealing -- a tuple of (int, date) makes total sense to compare using .equals() on the date. On the other hand, for values whose members are references to mutable things (like lists), it's not as obvious what the right default is. So, Kevin -- please make your case! Please share your experience with tools like AutoValue, and if you can, data on how frequently (and why) equals() is underridden on auto values in the Google codebase? From john.r.rose at oracle.com Mon May 16 19:41:12 2016 From: john.r.rose at oracle.com (John Rose) Date: Mon, 16 May 2016 12:41:12 -0700 Subject: Value types questions & comments In-Reply-To: References: <2127B453-99E7-4879-9097-F46BE46DB0C3@oracle.com> Message-ID: On Apr 12, 2016, at 1:51 PM, Brian Goetz wrote: > >> >> Using a value type for something that isn't a value raises alarm bells for me. At the minimum I would expect this user to have to implement eq/hc by hand, because the default behavior users want 99% of the time is (deep) content-based equality. > > This may be the reality-distortion field speaking, but in my view a reference *is* a kind of value ? albeit a very special kind. They?re immutable, like other values. To be more precise: Values have no mutable subfields. References and primitives have no mutable subfields. (References, when not null, point to various objects, some of which themselves have mutable subfields.) In this way, they are all similar. Also, *some* value or references (but not any primitives or some references like String) *may* refer to objects which have mutable subfields or some other kind of mutable state. Mainly because of this property, values, primitives, and references are at least partially referentially transparent under copy operations. When you copy a value (of any sort, ref or val), you capture all of its substructure, including any possible future value of its substructure. But the copy is not infinitely deep. (That would be an additional commitment.) The copy only copies the immediate subfields of the value: The whole ref, both halves of the long, all immediate fields of a custom value type. So far a reference is indistinguishable from a one-field value wrapping a reference. The next question is whether a value-wrapped reference should be restricted in its mutability (or in its type, etc.). I think we have to give a firm *no* here; there's too much to lose from making restrictions that are not natural to the computational machinery of the JVM. For example, if (following some well-intentioned commentators) we require that the transitive closure of a value be fully immutable ("like an int, right?") we give up the ability to use values as cursors and tuples. Q: "Why can't I return two values from my method without boxing to the heap?" A: "Well, one of your values was really a reference to mutable state, don't you see." That's not a conversation I am willing to have. Some languages (even on the JVM) might not allow partial immutability, but Java must, IMO. > Almost all their state is encapsulated (they can be compared by identity, that?s it). They can only be constructed by privileged factories (we call these constructors.) But, ultimately, they behave like values ? they are passed by value, they have no identity of their own. ?In any case, values with immutable immediate components lead us to questions about their equivalence relations (equality). There is more than one natural-looking equality predicate that can be assigned to a tuple (and therefore to any value). This means any choice of default has a ready-made rebuttal. 1. Component-wise boxed normal. Equivalent to boxing the tuple into a List, and running List.equals. 2. Component-wise op==. Equivalent to using the Java op== on each component, and-ing the bits together, or (if possible) packing into an array and using Arrays.equals. 3. Copy-wise equality. Component-wise op==, except that floats and doubles are converted to their corresponding "raw bits". And, perhaps a surprise: 0. Component-wise normal. Equivalent to boxing the tuple into a List, and running List.equals, except that floats and doubles are boxed to a non-standard container that lifts floating-point equality op==(float,float). (Non-tuple values can be thought of as those with private state, as hashcode-savers, or with value-specific symmetries, as rationals. These guys should define their own equality relations explicitly.) The basic choice (for tuples) is whether to recurse into "equals" on references (choices 0, 1). A secondary choice is whether to treat floats and doubles copy-wise (choices 0, 2). If not copy-wise, there is a further choice between Float.equals and op==(float,float). (Sadly, they are not exactly the same; see the javadoc!) The numbered cases 0..3 are in order of increasing refinement. Actually, 2 does not exactly refine 1, because of the difference between boxed and unboxed float equality. But #3 distinguishes the logical maximum number of values. (I.e., there is no stronger equality relation, where the "weakest" relation would make everything appear equal.) #0 captures the common practice of using op==(val,val) for values and Object.equals for refs. I'd like to call this "normal equality", and it is may be what we want for any-variables. ( T t1, t2; t1==t2 could be normal in this way.) For floats there's a second choice that may be called "boxed normal equality", where Object.equal is used uniformly after boxing non-refs. (You can also invent more: What if the refs are box types? What about arrays? Etc.) Both kinds of "normal" of equality (actually, all kinds that resort to Object.equals) are in tension with what I want to call "copy-wise equality". Two values are copy-wise equal if and only if there is no way to prove that one is not an copy (by assignment) of the other. This is basically bitwise equality, but don't tell implementors that, because they will be required to skip padding bits, and to traverse internal indirections (if the implementation uses them), to prevent spurious inequality results. Copy-wise equality is sometimes desirable, in order to prove that an operand has been seen before. (IdentityHashMap is used today for this purpose.) I think it should get a separate name, "System.isEqualCopy(T,T)" or some such. It is possible to imagine value types (such as IHM entries or proxies) which would use copy-wise equality on some of their fields. Here's my evaluation of the four choices: 0. Component-wise normal. This is equivalent to writing the sort of method that people already write, which performs .equals on refs and op== on vals. Probably the least surprising. It is also generally useful, having worked well for collections. Use it if you can afford it. But beware: Cursors require identity comparison on any backing store component instead of Object.equals. 1. Component-wise boxed normal. Easy to describe, but has an unfortunate dependency on the surprising Float.equals, which makes it unsuitable for numerically-oriented values. Avoid for that reason in favor of #0. 2. Component-wise op==. A bad candidate for V.equals, but sometimes a useful analog to op==(ref,ref). However, because of float oddities, consider favoring #3 instead. 3. Copy-wise equality. Component-wise op==, except that floats and doubles are converted to their corresponding "raw bits". Distinguishes the logical maximum number of values (there is no stronger equality relation). Should not be the default, but give it a name and allow users and value class writers to apply it when needed. Based on this analysis, if we were to assign a default equality predicate to value types, I think the weakest (#0) would be best. This is the one which works most like collections, but which "respects" primitives more by not boxing them. I think the current translation strategy uses normal equality for op==(T,T) where T is an any-type variable. This is consistent with imagining the default equality predicate (#0 as recommended) being expanded from any-generic code that gives each field a separate any-type. (IMO, this is a desirable incompatibility with legacy generics, where op==(T,T) means op==(ref,ref), and usually requires an explicit backup call to Object.equals.) Note that copy-wise equality breaks abstraction boundaries in a way analogous to its version applied to refs, which is op==(ref,ref). It can be used to prove that two apparently equivalent values (or refs) are in fact different in some way (perhaps an invisible way). Open question: Are there some values that would want to specialize the isEqualCopy method, perhaps by making it behave more like the equals method? Is that worth the cost of virtualizing this operation? The costs (in user model and implementation complexity) of isEqualCopy are (in my mind) the exact counterpart of today's costs of reference equality. (They add complexity to the user model which is not always needed, as for Strings, and they prevent some important unbox/box optimizations when EA fails.) ? John P.S. Exercise for the reader: For a value type V which needs deep equality through many layers of indirect structure, it might be desirable to have factory methods (V constructors) perform interning (invisibly) on indirect portions of the value, using something like an IdentityHashTable (private to the implementation). How could such a value avoid a full recursion on every call to V.equals? What would the implementation look like? Does this technique require a cost increase somewhere other than V.equals? Under what circumstances will the cost savings in V.equals pay for any other costs? (Hint: Consider using System.isEqualCopy.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From kevinb at google.com Tue May 17 03:05:15 2016 From: kevinb at google.com (Kevin Bourrillion) Date: Mon, 16 May 2016 20:05:15 -0700 Subject: Value equality (was: Value types questions and comments) In-Reply-To: References: <2127B453-99E7-4879-9097-F46BE46DB0C3@oracle.com> Message-ID: On Mon, May 16, 2016 at 11:20 AM, Brian Goetz wrote: Kevin say: > > However, it would be very sad if it does not solve a second problem at the >> same time, because it can. Even Java developers who are content with the >> performance foibles of their existing "value-based classes" are constantly >> irritated by the burden and bug-prone nature of writing/maintaining such >> classes. >> > > We have two choices in front of us regarding equality on value types: > > 1. What does `==` on values do? Choices include a bitwise comparison, a > componentwise comparision (identical to bitwise unless there are float > members), or invoking the `equals()` method. > I'd recommend that this fail to compile. When I come across this code: a == b I want to know what that code is doing, and I want to know that it's correct. So I jump to the definitions of a and b. If I see that they're ints, I'm done. If, however, they are SomeType, today I know that I am looking at reference equality. I'd like that to continue to be the case. People like to know what they're looking at. I wouldn't like to have to dig into the definition of *SomeType* (which may be "far away") just to know what sort of equality I'm looking at. (Note that even in the case of enums, where reference equality and value equality are identical, our internal coding guidelines still recommend always using .equals() for this reason. Then you have code that is not only correct, but is *obviously* correct. Not code that you have to look up a foreign definition to know whether it's correct or not. That's useful, and what's the cost? Just a bit of extra text on the screen. This is Java.) If we insist that == do something, then I'd recommend it be identical to equals(). We should assume that the person who wrote equals() made their choices for a reason. It's impossible to know whether exposing componentwise equality would make sense for a type or not. Not that I love this option; it makes me a little squeamish for an operator to invoke user code. My preference is to not compile. 2. What is the default for the `equals()` method? The obvious choice is a > componentwise comparision, but Kevin has (strongly) suggested we consider a > deep comparison, calling .equals() on any reference members. (Note that > the two choices collapse to the same thing when there are no reference > members.) > Not just equals(), but deepEquals(), as proposed by Bill Pugh. > For values whose members are value-ish objects themselves (e.g., String, > Optional, LocalDateTime), the deep-equals is obviously appealing -- a tuple > of (int, date) makes total sense to compare using .equals() on the date. > On the other hand, for values whose members are references to mutable > things (like lists), it's not as obvious what the right default is. > When you put it this way it seems even more clear that the "value-ish" case is the one to design toward; the one that should not require the user to add custom code. Of course we know that many different people implement equals() according to many different goals. But I've become convinced there is only *one* way to look at equals() that *actually makes sense*, and around which APIs can and have been and should be designed. And that is to say that *equals means interchangeable*. 99% of the time, you should never care which of two equal instances you have. If you do, either you are doing some low-level nuts-and-bolts far away from business logic (and this should apply to extremely few people), or you implemented equals() wrong. Lots of equals() methods *are* implemented wrong and many of those can never be fixed. However, I think it would be very questionable to design toward such cases. *Those* should be the cases where you have to write custom code. The default should be "equals means interchangeable". Mutable lists are a fuzzy case, but they tend to get away with their "wrong" behavior because of how often we *treat* them as immutable (er, meaning, we *don't mutate them*). Still, here we strongly encourage using ImmutableList as much as possible for reasons like this and all the well-known others. So, Kevin -- please make your case! Please share your experience with > tools like AutoValue, and if you can, data on how frequently (and why) > equals() is underridden on auto values in the Google codebase? > Okay. To refresh, we have a code generator for "value-based classes" because they're that much of a pain. As of today, it's used 17,400 times in our codebase. Its equals() behavior is not configurable; it always uses deep value equality unless the user replaces it with their own custom equals() implementation. I've researched how often Google users are doing this and why. I decided to manually review *all* instances of this, but at 70% of the way through I'm getting hungry, so I'll just crudely extrapolate from what I did get to: - About 30 classes customize equals so as to ignore some fields. (These are a little bit suspicious according to the "equals means interchangeable" maxim; I'd probably suggest they redesign a bit and keep their proper values separate from their other stuff. In some cases the ignored fields are derived, so it's fine, although in some of *those* I consider going out of their way to ignore the field in equals as just a hyperoptimization.) - About 25 classes do it so that they can substitute some alternate determination of equivalence for at least one field. Most of these are working around a field's what-I'd-call-broken equals() behavior. None were doing this to switch to identity comparison. - About 15 are doing very strange and suspicious things and *deserve whatever they get*. - About 10 seem to be just plain misunderstandings. - 1 or 2 customize it so that they can also be equal to other sibling types implementing a common interface. Okay. In all, fully 99.5% of usages accept the default behavior. So, for evidence that reference equality or even shallow value equality is what anyone wants, out of 17,400 usages I've come up empty. (Granted, there may be users who want it and therefore just don't use AutoValue at all. But Googlers are very good at complaining when a tool doesn't do quite what they want, and we're not getting that either.) HTH -- Kevin Bourrillion | Java Librarian | Google, Inc. | kevinb at google.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From kevinb at google.com Tue May 17 03:29:37 2016 From: kevinb at google.com (Kevin Bourrillion) Date: Mon, 16 May 2016 20:29:37 -0700 Subject: Value equality (was: Value types questions and comments) In-Reply-To: References: <2127B453-99E7-4879-9097-F46BE46DB0C3@oracle.com> Message-ID: And by "deep equals" I mean equivalent behavior to https://docs.oracle.com/javase/8/docs/api/java/util/Arrays.html#deepEquals-java.lang.Object:A-java.lang.Object:A- at least that is what AutoValue has been doing. On Mon, May 16, 2016 at 8:05 PM, Kevin Bourrillion wrote: > On Mon, May 16, 2016 at 11:20 AM, Brian Goetz > wrote: > > Kevin say: >> >> However, it would be very sad if it does not solve a second problem at >>> the same time, because it can. Even Java developers who are content with >>> the performance foibles of their existing "value-based classes" are >>> constantly irritated by the burden and bug-prone nature of >>> writing/maintaining such classes. >>> >> >> We have two choices in front of us regarding equality on value types: >> >> 1. What does `==` on values do? Choices include a bitwise comparison, a >> componentwise comparision (identical to bitwise unless there are float >> members), or invoking the `equals()` method. >> > > I'd recommend that this fail to compile. > > When I come across this code: > > a == b > > I want to know what that code is doing, and I want to know that it's > correct. So I jump to the definitions of a and b. If I see that they're > ints, I'm done. > > If, however, they are SomeType, today I know that I am looking at > reference equality. I'd like that to continue to be the case. People like > to know what they're looking at. I wouldn't like to have to dig into the > definition of *SomeType* (which may be "far away") just to know what sort > of equality I'm looking at. > > (Note that even in the case of enums, where reference equality and value > equality are identical, our internal coding guidelines still recommend > always using .equals() for this reason. Then you have code that is not only > correct, but is *obviously* correct. Not code that you have to look up a > foreign definition to know whether it's correct or not. That's useful, and > what's the cost? Just a bit of extra text on the screen. This is Java.) > > If we insist that == do something, then I'd recommend it be identical to > equals(). We should assume that the person who wrote equals() made their > choices for a reason. It's impossible to know whether exposing > componentwise equality would make sense for a type or not. Not that I love > this option; it makes me a little squeamish for an operator to invoke user > code. My preference is to not compile. > > > 2. What is the default for the `equals()` method? The obvious choice is >> a componentwise comparision, but Kevin has (strongly) suggested we consider >> a deep comparison, calling .equals() on any reference members. (Note that >> the two choices collapse to the same thing when there are no reference >> members.) >> > > Not just equals(), but deepEquals(), as proposed by Bill Pugh. > > > >> For values whose members are value-ish objects themselves (e.g., String, >> Optional, LocalDateTime), the deep-equals is obviously appealing -- a tuple >> of (int, date) makes total sense to compare using .equals() on the date. >> On the other hand, for values whose members are references to mutable >> things (like lists), it's not as obvious what the right default is. >> > > When you put it this way it seems even more clear that the "value-ish" > case is the one to design toward; the one that should not require the user > to add custom code. > > Of course we know that many different people implement equals() according > to many different goals. But I've become convinced there is only *one* > way to look at equals() that *actually makes sense*, and around which > APIs can and have been and should be designed. And that is to say that *equals > means interchangeable*. 99% of the time, you should never care which of > two equal instances you have. If you do, either you are doing some > low-level nuts-and-bolts far away from business logic (and this should > apply to extremely few people), or you implemented equals() wrong. > > Lots of equals() methods *are* implemented wrong and many of those can > never be fixed. However, I think it would be very questionable to design > toward such cases. *Those* should be the cases where you have to write > custom code. The default should be "equals means interchangeable". > > Mutable lists are a fuzzy case, but they tend to get away with their > "wrong" behavior because of how often we *treat* them as immutable (er, > meaning, we *don't mutate them*). Still, here we strongly encourage using > ImmutableList as much as possible for reasons like this and all the > well-known others. > > > So, Kevin -- please make your case! Please share your experience with >> tools like AutoValue, and if you can, data on how frequently (and why) >> equals() is underridden on auto values in the Google codebase? >> > > Okay. To refresh, we have a code generator for "value-based classes" > because they're that much of a pain. As of today, it's used 17,400 times in > our codebase. Its equals() behavior is not configurable; it always uses > deep value equality unless the user replaces it with their own custom > equals() implementation. > > I've researched how often Google users are doing this and why. I decided > to manually review *all* instances of this, but at 70% of the way through > I'm getting hungry, so I'll just crudely extrapolate from what I did get to: > > - About 30 classes customize equals so as to ignore some fields. > (These are a little bit suspicious according to the "equals means > interchangeable" maxim; I'd probably suggest they redesign a bit and keep > their proper values separate from their other stuff. In some cases the > ignored fields are derived, so it's fine, although in some of *those* I > consider going out of their way to ignore the field in equals as just a > hyperoptimization.) > - About 25 classes do it so that they can substitute some alternate > determination of equivalence for at least one field. Most of these are > working around a field's what-I'd-call-broken equals() behavior. None were > doing this to switch to identity comparison. > - About 15 are doing very strange and suspicious things and *deserve > whatever they get*. > - About 10 seem to be just plain misunderstandings. > - 1 or 2 customize it so that they can also be equal to other sibling > types implementing a common interface. Okay. > > In all, fully 99.5% of usages accept the default behavior. > > So, for evidence that reference equality or even shallow value equality is > what anyone wants, out of 17,400 usages I've come up empty. (Granted, > there may be users who want it and therefore just don't use AutoValue at > all. But Googlers are very good at complaining when a tool doesn't do quite > what they want, and we're not getting that either.) > > HTH > > > -- > Kevin Bourrillion | Java Librarian | Google, Inc. | kevinb at google.com > -- Kevin Bourrillion | Java Librarian | Google, Inc. | kevinb at google.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.r.rose at oracle.com Tue May 17 05:06:08 2016 From: john.r.rose at oracle.com (John Rose) Date: Mon, 16 May 2016 22:06:08 -0700 Subject: Value equality (was: Value types questions and comments) In-Reply-To: References: <2127B453-99E7-4879-9097-F46BE46DB0C3@oracle.com> Message-ID: <581B80CA-9E69-408C-800D-EFAAA987FD32@oracle.com> On May 16, 2016, at 8:29 PM, Kevin Bourrillion wrote: > > And by "deep equals" I mean equivalent behavior to > > https://docs.oracle.com/javase/8/docs/api/java/util/Arrays.html#deepEquals-java.lang.Object:A-java.lang.Object:A- > > at least that is what AutoValue has been doing. To the extent I understand this, it seems wrong: deepEquals treats arrays as if they are lists, which is an abstraction shift. Surely you aren't suggesting cracking a component reference in a subfield and treating its object as a List of its components, and so on recursively? Because that's what deepEquals does. Or do you mean that, just as deepEquals avoids using op==(ref,ref) on array components, so "deep equals" should avoid using op==(ref,ref) on value components? You can avoid op==(ref,ref) by replacing it by a call to ref.equals(ref), or you can avoid op==(ref,ref) by cracking the refs and recursing. That latter breaks an abstraction, so I think we agree it's bad, but *that* is a more "equivalent behavior" to Arrays.deepEquals. ? John -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.goetz at oracle.com Tue May 17 14:56:40 2016 From: brian.goetz at oracle.com (Brian Goetz) Date: Tue, 17 May 2016 10:56:40 -0400 Subject: Value equality In-Reply-To: <581B80CA-9E69-408C-800D-EFAAA987FD32@oracle.com> References: <2127B453-99E7-4879-9097-F46BE46DB0C3@oracle.com> <581B80CA-9E69-408C-800D-EFAAA987FD32@oracle.com> Message-ID: So, its worth pointing out here that the only disagreement is on a pretty cornery (and fundamentally messy) case -- arrays as components of a value. I think there's no dispute over: - primitives components compared with ==(p,p) - value components compared recursively with ==(v,v) - non-array reference components compared recursively with .equals() So the only remaining question is what to do with arrays (and, for equals vs deepEquals, these differ only when arrays are nested within arrays within values -- very much a corner case). Options include: - reference equality / Object.equals() - Arrays.equals on primitive / value arrays, but reference equality on object arrays - Arrays.equals - Arrays.deepEquals The first says "are they the same array". The second lifts equality onto components known to be immutable, but punts on mutable array components. The third lifts .equals() equality onto potentially-mutable array components; the third goes farther and recursively lifts this definition of equality onto elements that are actually arrays (which must be done reflectively -- see the nasty code below which is called on each element.) In what cases will you have an array nested in an array as a value component? I'm having a hard time thinking of any -- can you dig up examples? The cases where I'd have an ordinary array in a value fall into two (opposite) categories: - Where the array is effectively immutable and holds the state of the value, such as a string-like value class (holding a char[]) or a BigDecimal-like value class (holding a long[]); - Where the value is behavior like a cursor into an array. In the first case, two BigDecimal are equals() if they have the same array contents, so at first we might think Arrays.equals() is what we want (and same for the String case.) But if I were coding a BigDecimal value, I wouldn't want this default anyway -- since I want to treat 1.0 and 1.00 as equal, which the default won't do. So while the default is OK, I'm still going to override it (and similarly for String), because Arrays.equals still produces false negatives. In the cursor case, we definitely want to compare the arrays by reference -- two cursors represent the same thing if they point to the same element in the SAME array, not equivalent arrays. And, same with an Optional. So, grading the possibilities: - reference equality -- produces right answer for cursor, produces wrong answer for BigDecimal/String cases. - Arrays.equals -- produces wrong answer for cursor, produces tempting-seeming but still wrong answer for BigDecimal/String. So both cases produce the wrong answer for the value-like uses of arrays -- and the user has to override the default anyway. The only one that produces the right answer in any case is reference equality (which is .equals()). Maybe there's more cases, but these are the ones that come immediately to mind. For arrays, it seems that trying to read the user's mind here is a losing (and at the same time, expensive) game. Arrays are what they are; trying to turn them into acts-like-lists when you don't know the semantics of the array in their enclosing value seems pretty questionable. Which leads me to prefer the following simpler default for equals() on values: - compare primitive and components with == - compare reference components with .equals() Do you have data on what percentage of your AutoValue objects contains arrays? Secondary question: to what degree is the "if you don't like the default, write your own" mitigated by helpers like Guava's Objects.equals? (It does kind of suck that if you disagree with the default treatment of one field, you have to rewrite the whole equals/hashCode methods.) FYI, nasty and expensive method called on every element by deepEquals: static boolean deepEquals0(Object e1, Object e2) { assert e1 !=null; boolean eq; if (e1instanceof Object[] && e2instanceof Object[]) eq =deepEquals ((Object[]) e1, (Object[]) e2); else if (e1instanceof byte[] && e2instanceof byte[]) eq =equals((byte[]) e1, (byte[]) e2); else if (e1instanceof short[] && e2instanceof short[]) eq =equals((short[]) e1, (short[]) e2); else if (e1instanceof int[] && e2instanceof int[]) eq =equals((int[]) e1, (int[]) e2); else if (e1instanceof long[] && e2instanceof long[]) eq =equals((long[]) e1, (long[]) e2); else if (e1instanceof char[] && e2instanceof char[]) eq =equals((char[]) e1, (char[]) e2); else if (e1instanceof float[] && e2instanceof float[]) eq =equals((float[]) e1, (float[]) e2); else if (e1instanceof double[] && e2instanceof double[]) eq =equals((double[]) e1, (double[]) e2); else if (e1instanceof boolean[] && e2instanceof boolean[]) eq =equals((boolean[]) e1, (boolean[]) e2); else eq = e1.equals(e2); return eq; } On 5/17/2016 1:06 AM, John Rose wrote: > On May 16, 2016, at 8:29 PM, Kevin Bourrillion > wrote: >> >> And by "deep equals" I mean equivalent behavior to >> >> https://docs.oracle.com/javase/8/docs/api/java/util/Arrays.html#deepEquals-java.lang.Object:A-java.lang.Object:A- >> >> at least that is what AutoValue has been doing. > > To the extent I understand this, it seems wrong: deepEquals treats > arrays as if they are lists, which is an abstraction shift. Surely > you aren't suggesting cracking a component reference in a subfield and > treating its object as a List of its components, and so on > recursively? Because that's what deepEquals does. > > Or do you mean that, just as deepEquals avoids using op==(ref,ref) on > array components, so "deep equals" should avoid using op==(ref,ref) on > value components? > > You can avoid op==(ref,ref) by replacing it by a call to > ref.equals(ref), or you can avoid op==(ref,ref) by cracking the refs > and recursing. That latter breaks an abstraction, so I think we agree > it's bad, but *that* is a more "equivalent behavior" to Arrays.deepEquals. > > ? John -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.levart at gmail.com Tue May 17 19:20:21 2016 From: peter.levart at gmail.com (Peter Levart) Date: Tue, 17 May 2016 21:20:21 +0200 Subject: Value equality In-Reply-To: References: <2127B453-99E7-4879-9097-F46BE46DB0C3@oracle.com> <581B80CA-9E69-408C-800D-EFAAA987FD32@oracle.com> Message-ID: <0984b7d7-b248-b39e-4d02-d24b7160a57d@gmail.com> Hi, I think that, to avoid confusion, there should be no special treatment of arrays. They should be treated as any other reference. Why? Reason #1: Suppose we have: value U { int[] array; U(int[] array) { this.array = array; } } value V { Object obj; V(Object obj) { this.obj = obj; } } int[] a1 = new int[] { ... }; int[] a2 = new int[] { ... }; U u1 = U(a1); U u2 = U(a2); V v1 = V(a1); V v2 = V(a2); I think that it would be very confusing for the following to no hold for any a1 and a2: assert (u1 == u2) == (v1 == v2); assert u1.equals(u2) == v1.equals(v2); If you want to support above assertions and still treat arrays differently from non-array references, then a runtime dispatch dependent on a runtime reference type will be needed. If you do that then you have specified an operation performed on components of a value that is not expressible in language with two locals holding the values of the those component(s) short of a library function call. Reason #2: It's not difficult to create a standard reusable value type wrapping a single array with suitable equals() override: value Array { private T[] array; Array(int length) { array = new T[length]; } @Override public boolean equals() { ... } } ... and use it instead of plain array(s) as components of other value or reference types, with no runtime overhead compared to plain arrays. For this to work then default .equals() for value types should: - compare primitive components with ==(p, p) (modulo float/double variations for which I don't have an opinion yet) - compare reference components with .equals() - compare value components with .equals() - yes, this gives the most intuitive result and encapsulates equality into the classes of the components. For == operator on value types then I don't know. Perhaps it should be the "bit-wise" comparison of components or, a John would say, copy-equivalence. That's simple to grasp and explain and is also usable. Regards, Peter On 05/17/2016 04:56 PM, Brian Goetz wrote: > So, its worth pointing out here that the only disagreement is on a > pretty cornery (and fundamentally messy) case -- arrays as components > of a value. > > I think there's no dispute over: > - primitives components compared with ==(p,p) > - value components compared recursively with ==(v,v) > - non-array reference components compared recursively with .equals() > > So the only remaining question is what to do with arrays (and, for > equals vs deepEquals, these differ only when arrays are nested within > arrays within values -- very much a corner case). Options include: > - reference equality / Object.equals() > - Arrays.equals on primitive / value arrays, but reference equality > on object arrays > - Arrays.equals > - Arrays.deepEquals > > The first says "are they the same array". The second lifts equality > onto components known to be immutable, but punts on mutable array > components. The third lifts .equals() equality onto > potentially-mutable array components; the third goes farther and > recursively lifts this definition of equality onto elements that are > actually arrays (which must be done reflectively -- see the nasty code > below which is called on each element.) > > In what cases will you have an array nested in an array as a value > component? I'm having a hard time thinking of any -- can you dig up > examples? > > The cases where I'd have an ordinary array in a value fall into two > (opposite) categories: > - Where the array is effectively immutable and holds the state of the > value, such as a string-like value class (holding a char[]) or a > BigDecimal-like value class (holding a long[]); > - Where the value is behavior like a cursor into an array. > > In the first case, two BigDecimal are equals() if they have the same > array contents, so at first we might think Arrays.equals() is what we > want (and same for the String case.) But if I were coding a > BigDecimal value, I wouldn't want this default anyway -- since I want > to treat 1.0 and 1.00 as equal, which the default won't do. So while > the default is OK, I'm still going to override it (and similarly for > String), because Arrays.equals still produces false negatives. > > In the cursor case, we definitely want to compare the arrays by > reference -- two cursors represent the same thing if they point to the > same element in the SAME array, not equivalent arrays. And, same with > an Optional. > > So, grading the possibilities: > - reference equality -- produces right answer for cursor, produces > wrong answer for BigDecimal/String cases. > - Arrays.equals -- produces wrong answer for cursor, produces > tempting-seeming but still wrong answer for BigDecimal/String. > > So both cases produce the wrong answer for the value-like uses of > arrays -- and the user has to override the default anyway. The only > one that produces the right answer in any case is reference equality > (which is .equals()). Maybe there's more cases, but these are the > ones that come immediately to mind. > > For arrays, it seems that trying to read the user's mind here is a > losing (and at the same time, expensive) game. Arrays are what they > are; trying to turn them into acts-like-lists when you don't know the > semantics of the array in their enclosing value seems pretty > questionable. Which leads me to prefer the following simpler default > for equals() on values: > > - compare primitive and components with == > - compare reference components with .equals() > > Do you have data on what percentage of your AutoValue objects contains > arrays? > > Secondary question: to what degree is the "if you don't like the > default, write your own" mitigated by helpers like Guava's > Objects.equals? (It does kind of suck that if you disagree with the > default treatment of one field, you have to rewrite the whole > equals/hashCode methods.) > > > > > FYI, nasty and expensive method called on every element by deepEquals: > static boolean deepEquals0(Object e1, Object e2) { > assert e1 !=null; > boolean eq; > if (e1instanceof Object[] && e2instanceof Object[]) > eq =deepEquals ((Object[]) e1, (Object[]) e2); > else if (e1instanceof byte[] && e2instanceof byte[]) > eq =equals((byte[]) e1, (byte[]) e2); > else if (e1instanceof short[] && e2instanceof short[]) > eq =equals((short[]) e1, (short[]) e2); > else if (e1instanceof int[] && e2instanceof int[]) > eq =equals((int[]) e1, (int[]) e2); > else if (e1instanceof long[] && e2instanceof long[]) > eq =equals((long[]) e1, (long[]) e2); > else if (e1instanceof char[] && e2instanceof char[]) > eq =equals((char[]) e1, (char[]) e2); > else if (e1instanceof float[] && e2instanceof float[]) > eq =equals((float[]) e1, (float[]) e2); > else if (e1instanceof double[] && e2instanceof double[]) > eq =equals((double[]) e1, (double[]) e2); > else if (e1instanceof boolean[] && e2instanceof boolean[]) > eq =equals((boolean[]) e1, (boolean[]) e2); > else eq = e1.equals(e2); > return eq; > } > > > > On 5/17/2016 1:06 AM, John Rose wrote: >> On May 16, 2016, at 8:29 PM, Kevin Bourrillion wrote: >>> >>> And by "deep equals" I mean equivalent behavior to >>> >>> https://docs.oracle.com/javase/8/docs/api/java/util/Arrays.html#deepEquals-java.lang.Object:A-java.lang.Object:A- >>> >>> at least that is what AutoValue has been doing. >> >> To the extent I understand this, it seems wrong: deepEquals treats >> arrays as if they are lists, which is an abstraction shift. Surely >> you aren't suggesting cracking a component reference in a subfield >> and treating its object as a List of its components, and so on >> recursively? Because that's what deepEquals does. >> >> Or do you mean that, just as deepEquals avoids using op==(ref,ref) on >> array components, so "deep equals" should avoid using op==(ref,ref) >> on value components? >> >> You can avoid op==(ref,ref) by replacing it by a call to >> ref.equals(ref), or you can avoid op==(ref,ref) by cracking the refs >> and recursing. That latter breaks an abstraction, so I think we >> agree it's bad, but *that* is a more "equivalent behavior" to >> Arrays.deepEquals. >> >> ? John > -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.r.rose at oracle.com Tue May 17 20:02:42 2016 From: john.r.rose at oracle.com (John Rose) Date: Tue, 17 May 2016 13:02:42 -0700 Subject: Value equality In-Reply-To: <0984b7d7-b248-b39e-4d02-d24b7160a57d@gmail.com> References: <2127B453-99E7-4879-9097-F46BE46DB0C3@oracle.com> <581B80CA-9E69-408C-800D-EFAAA987FD32@oracle.com> <0984b7d7-b248-b39e-4d02-d24b7160a57d@gmail.com> Message-ID: <7E67C573-3A81-4746-A50B-7D1FF3BF6A6D@oracle.com> On May 17, 2016, at 12:20 PM, Peter Levart wrote: > > Hi, > > I think that, to avoid confusion, there should be no special treatment of arrays. They should be treated as any other reference. Why? > > Reason #1: To me, your reason #1 boils down to saying that special treatment of arrays requires a surprising break to a certain property of reference subtyping, that an operation a.m(b) means the same thing regardless of the static type of a. And I concur. That's a valuable property. Generally, static subtypes add new methods (already latent under the supertype) but do not change methods. > Reason #2: > > It's not difficult to create a standard reusable value type wrapping a single array with suitable equals() override: Yes. There is a choice of simple workarounds: override equals or use a ref-to-val semantic wrapper. As I implied in my earlier note, there are many candidates for equivalence relations. But the basic system hangs together best if we make use of as few of them as possible, by default at least, and allow users to import other equivalences, ad hoc, using either overrides or wrappers. > ... and use it instead of plain array(s) as components of other value or reference types, with no runtime overhead compared to plain arrays. For this to work then default .equals() for value types should: > > - compare primitive components with ==(p, p) (modulo float/double variations for which I don't have an opinion yet) > - compare reference components with .equals() > - compare value components with .equals() - yes, this gives the most intuitive result and encapsulates equality into the classes of the components. This is where things get interesting! The .equals method has to tie everything together nicely. > For == operator on value types then I don't know. Perhaps it should be the "bit-wise" comparison of components or, a John would say, copy-equivalence. > > That's simple to grasp and explain and is also usable. I'd like to rework an earlier message on the Big Picture of equality semantics (with steps 0..3), after discussing it locally with some colleagues. I'll do that in a followup. ? John From john.r.rose at oracle.com Tue May 17 22:48:24 2016 From: john.r.rose at oracle.com (John Rose) Date: Tue, 17 May 2016 15:48:24 -0700 Subject: Value equality In-Reply-To: References: <2127B453-99E7-4879-9097-F46BE46DB0C3@oracle.com> Message-ID: <2E984345-E7DC-4FA7-ADED-BE9C10626F29@oracle.com> On May 16, 2016, at 11:20 AM, Brian Goetz wrote: > > We have two choices in front of us regarding equality on value types: > > 1. What does `==` on values do? Choices include a bitwise comparison, a componentwise comparision (identical to bitwise unless there are float members), or invoking the `equals()` method. I.e., what is the semantic binding of the op== syntax? > 2. What is the default for the `equals()` method? The obvious choice is a componentwise comparision, but Kevin has (strongly) suggested we consider a deep comparison, calling .equals() on any reference members. (Note that the two choices collapse to the same thing when there are no reference members.) I.e., what are the global expectations of the .equals method, and how can we embody them in a suitable default? > For values whose members are value-ish objects themselves (e.g., String, Optional, LocalDateTime), the deep-equals is obviously appealing -- a tuple of (int, date) makes total sense to compare using .equals() on the date. On the other hand, for values whose members are references to mutable things (like lists), it's not as obvious what the right default is. I can't vote for "deep" vs. "shallow", or any syntax binding, without setting up a bunch of semantic background. So here goes? Let's start with semantics with no syntax. I would like to single out two especially useful equivalence relations, to be called normal equals ("NE") and copy-wise equals ("CE"). ("Normal" is a shameless play for pre-eminence of a particular theory. But we have to define a normal, because we have to say what we expect of .equals. We might have to tweak my notion of "normal", but here it is.) For exposition, we need also names for legacy comparison operations for primitives and references, to be called primitive equals ("PE") and object equals ("OE"). CE is the most refined possible equality. If two values (reference or primitive) are CE, then you can *never* execute code that detects a difference between them. If two values are CE then each is fully substitutable for the other, in all contexts. This is sometimes a useful property to observe in generic code. CE(a,b) := "a is indistinguishable from an assignment-copy of b" Implementation note: CE is op== today, except for floats. For floats, CE must consult raw bits; cf. Double.doubleToRawLongBits. CE for values has to be a component-wise distribution of CE, or else it wouldn't be CE. But after this point we get less certain. In most cases, CE is exactly the legacy op==, but not for floats. So let's define a slightly coarser equivalence relation for primitive equality: PE(a,b) := legacy(a==b) || (floating(a) && CE(a,b)) Here the op== is the legacy primitive equality appropriate to both a and b. (Note: These formulas will assume that a and b are either both refs, both of the same primitive type, or (in some cases) both of the same value type.) Implementation note: Both PE(+0.0,-0.0) and PE(Float.NaN, Float.NaN) are true, while CE(+0.0,-0.0) is true but Float.NaN==Float.NaN is false. These are the only differences, so we can treat PE as a delta from CE: PE(a,b) == CE(a,b) || (floating(a) && legacy(a==0 && b==0)). Also, since the box types Float and Double use CE internally, we can also boxed comparison to obtain CE: PE(a,b) == legacy(a==b) || ((Float)a).equals((Float)b) Implementation note: By appealing to boxes for a semantic notion like PE, we *do not* require implementations to perform boxing. We simply require that any computation of the semantics produces the same result *as if* the formula were directly evaluated, boxes and all. So PE is neither CE nor the legacy op==, but is an equivalence relation that closely approximates both. It is presented here to show the interactions of general equality with primitive equality, but (spoiler alert) it probably won't make the final cut. OE is simply an appeal to the Object.equals method, guarded first by CE. OE(a,b) := CE(a,b) || (reference(a) && legacy(a != null && a.equals(b))) Implementation note: The CE guard takes care of null==null. For references, this relation is embodied in Objects.equals(a,b). Implementation note: For primitive wrappers, OE is compatible with CE (but not PE or op==) on the wrapped primitive value. This is true even for floats (the .equals methods consult the raw bits). Therefore we can use these relations, if we want: CE(a,b) == OE((Object)a,(Object)b) where primitive(a) PE(a,b) == legacy(a==b) || (primitive(a) && OE((Object)a,(Object)b)) For some objects, OE is the same as CE; this includes arrays. For objects which favor structural equality, including most collection classes, OE recurses into substructure. One might say that two objects which are OE but not CE are not fully substitutable, but "structurally substitutable". The Arrays.equals method family offers an alternative form of structural substitutability by treating arrays as lists. Structural substitutability is a time-varying property, if an object or any subpart is mutable. Still, many objects are never-mutated, so structural substitutability can be viewed as time-invariant in many cases (though this is a fruitful source of bugs and TOCTOU exploits). Now it's time to define "normal", as a concept that works equally well with both objects and primitives, and can extend to the new value types. Here is a symmetric possibility: NE'(a,b) := PE(a,b) || OE(a,b) In other words, a "normal equality" test compares primitives with op== (except it uses CE for NaNs), and compares objects with .equals. We can also rewrite this formula to omit the non-legacy notion of PE, and appeal to boxes instead: NE'(a,b) := legacy(a==b) || OE((Object)a,(Object)b) For reference types, this is obviously our old friend Objects.equals in another guise. For primitive types it is something new, but perhaps desirable. (So I'm putting it on the table, though I don't prefer it.) It is a "naively generified" version of "a == b || a != null && a.equals(b)". Since it includes PE, it compares +0.0 as equal to -0.0. But I prefer a simpler version of NE: NE(a,b) := CE(a,b) || (reference(a) && OE(a,b)) This version drops PE from the equation. Float zeroes are not respected; primitives comparisons are uniformly bitwise (CE). Because primitive boxes perform CE for .equals, we can alternatively formulate NE in terms of boxes: NE(a,b) := OE((Object)a,(Object)b) This finally takes me to why I want to privilege this comparison with the name "normal": It is in use every nanosecond already, in our collection types. When you auto-box a primitive into a collection, it participates in the form of NE. A third candidate for NE would just use legacy comparisons: NE''(a,b) := legacy(a == b) || (reference(a) && OE(a,b)) But this is not an equivalence relation, in the case of floating point NaNs. Nor is it compatible with current computations involving boxed NaNs (which *do* compare as equal if they are CE). I believe these reasons, combined, take NE'' out of the running. Still staying in semantics, let's consider extending these relations to values. As noted above, CE *must* be "componentwise CE", unless we have a way to guarantee that some field will *never* be detectible by user code, in which case we should just drop the field. PE *could* be extended to values componentwise also. This would give a slightly weakened version of CE for values which contain floats. I don't think that move carries its weight. To refer to the equals method for values, we can make a simple partial definition: VE(a,b) := a.equals(b) where value(a) and typeof(a)==typeof(b) Alternatively, OE is easily extended to values, either by boxing, or directly: OE'(a,b) := OE(a,b) || (value(a) && OE((Object)a, (Object)b)) OE'(a,b) := CE(a,b) || (!primitive(a) && legacy(a != null && a.equals(b))) (To make notation easier, we can assume a!=null is identically true for any value a.) Those two definitions will remain in alignment as long as .equals behavior commutes with boxing a value to Object. Then we get an compatibly extended version of NE (with no prime this time) as: NE(a,b) := CE(a,b) || OE(a,b) || VE(a,b) We can fold together the equals calls instead: NE(a,b) := (!primitive(a) && OE'(a,b)) Or, recognizing that PE is not in play, simply fold everything into boxes: NE(a,b) := OE((Object)a,(Object)b) By dropping CE from the formula, we are relying on the fact that all .equals methods are reflexive, even for values. Any of these formulas are plausible starting points for efficient implementations of NE. An NE with values is a highly plausible extension for places in our system which execute the logic of Objects.equals. For places in our system which execute non-CE primitive equality predicates (such as op== or PE), NE is an approximate replacement, and may need special handling. But the differences are small, affecting only to the treatment of NaN dis-equality and signed-zero equality. And today's normative treatment (after boxing) is simply CE, which is compatible with NE for boxed primitives. For the record, we could also define array-traversing predicates which coarsen OE, such as: AE(a,b) := /*OE(a,b) ||*/ Arrays.deepEquals(new Object[]{a},new Object[]{b}) AE'(a,b) := OE(a,b) || (array(statictypeof(a)) && array(statictypeof(b)) && Arrays.equals(a,b)) AE''(a,b) := OE(a,b) || (array(statictypeof(a)) && array(statictypeof(b)) && Arrays.deepEquals(a,b)) I hope that is helpful to others besides me. Now, as for syntax and bindings. CE is obviously useful; when you need it you shouldn't have to approximate it. So it needs its own API point, something like System.isEqualCopy. One reason for this, rather than rely on a legacy operator, is to give programmers a definitive way to avoid binding to the floating op== which has a surprise inside. Generic code, if it specializes to floating op==, will break substitutability in just that one place; surely this will be unwelcome sometimes. Even if generic code uses CE uniformly for op==, System.isEqualCopy can have a pedagogical function, as well as being handy in numeric code. PE is not obviously useful, so unless it is incorporated in NE (as NE') it does not need an API point. It can be coded (in numeric code) as a==b || CE(a,b). It is plausible to bind op== to CE almost everywhere. There is one caveat (besides floating point): For values, there will be a performance cost, because many value comparisons will *not* welcome a pre-pass with CE as an optimization to VE. In other words, if op== is bound to CE, the NE formula for values will be the usual: v==w || v.equals(w) This is nice and symmetric with current practice, *but* it is slow. The big user-written pattern puts a burden on the JIT. The JIT *should* be able to optimize away the various field references in "v==w", by recognizing that the same fields are being compared in the .equals method. *But* if the equals method does anything other than CE to those fields (e.g., NE or AE or a user-defined check), then the JIT might not be able to infer that the original field comparison is subsumed by the later test. There may be control flow in the way which the JIT does not understand. Thus, "v==w||v.equals(w)" may optimize only partially, including machine code artifacts like "v.f==w.f; ?; v.f.equals(w.f)". Finally, even if the JIT is really smart, users are even smarter, and they endlessly produce new variations on the theme of a==b||a.equals(b). The two comparisons, which we want to merge into each other, can be separated by arbitrary user code. This is a classic problem with verbose languages: The compiler has to recognize any code shape the user throws at it. For this reason one might prefer that op== for values (and any-types) be bound to NE, not CE. (This is P2 below.) An underlying assumption here would be that CE is much less useful than NE, so NE should have preferential access. Data is mixed on whether NE is more important than CE. Object reference identity would seem to be the wrong question to answer in many cases, yet some codes use CE to detect full substitutability. Some uses of op== are errors, where the user mean to reach for NE but forgot how to spell it (or was too lazy?). Sometimes CE seems like an expert-only tool, a sharp needle to be kept on a high shelf. Deciding the true utility of CE requires corpus studies. Considered by itself, NE is very useful; we call it Objects.equals(a,b) nowadays, so it has an API point waiting for it already. What do today's classes do when implementing .equals? This is an empirical question, but to me it seems that they typically consult most or all of their fields, comparing each field value with the field value in the comparand. The tendency is to use recursive structural comparison on fields. In short, it is rare to find CE applied to fields in an .equals method, and common to find NE applied. (Sometimes an AE or another thing is used.) (Note: When we say that code executes NE or CE, we really mean that it executes other code whose result is equivalent to NE or CE, as restricted to the types and inputs actually present.) If a class or value models a *cursor* into some mutable state, it may be undesirable to traverse the state. A cursor comparison operation should use CE on the state reference, while is can use NE (or op==) on other parts such as indexes. Java does not have many such data types, but the option must exist for some occasions, to use CE instead of NE for a field comparison. (C++ STL has many such types. Project Panama defines proxies for C pointers which behave like this.) It is simple to use NE to specify a useful default .equals method for values: Simply distribute NE across the fields, in the same way that CE *must* distribute itself across the fields. If a value type wants to use floating op== on a floating point field, it can specify this "manually". Likewise for AE (arrays treated structurally). The manual specialization can be carried out either by hand-writing the .equals method, or else giving the field a type for which NE delegates to the correct equality computation. As Peter Levart points out, this field type can itself be a value type which wraps the "real" field value, and such a value type is probably of negligible cost. So it appears that having NE-across-components be the default equals method is customary and reasonable. Given wrapper types for tweaked fields, it is even (arguably) universal. Going back to op==, there are two plausible options for binding it to new types: (P1) Syntax of op==(val,val) and op==(any,any) binds to CE as with op==(ref,ref). Therefore, NE is uniformly reached by today's idiom, which traverses value fields twice. (P2) Syntax of op==(val,val) and op==(any,any) is direct access to NE. CE is reachable by experts at System.isEqualCopy. The old idiom for NE works also calls equals twice. (P3) Same as P1, op== is uniform access to CE. New op (spelled "===", ".==", "=~", etc.) is uniform, optimizable access to NE, attracting users away from legacy idiom for NE. In all cases, floats have irregularities on NaNs and zeroes. Basically, generic code needs to access CE, and wants to use op== for it, but floating-point code has a slightly different meaning for op==. You can't have two meanings for one operator, so something has to give. I recommend segregating the floating-point meaning of op== to monomorphic code, so polymorphic code can be reasoned about in terms of the clean CE. In P1 and P2, the legacy comparison idiom (a==b || a.equals(b)) reaches NE, but also iterates over value fields twice, with unpredictable optimization results. In P2, both passes are done by V.equals, so if there is a guaranteed that V.equals is idempotent, it can be directly optimized. If P1, if there is a guarantee that op== is subsumed by .equals, the leading op== can be optimized. In any case, P2 allows code to be improved to reach NE by a single expression a==b, allowing full optimization in all cases. With P1, we can use op== for CE substitutability checks (except for non-specialized floats), and use Objects.equals for NE. There is no need to directly call .equals, but users will do this because it is easier than calling Objects.equals. With P2, we can use op== as abbreviation for Objects.equals, and use, System.isEqualCopy for substitutability checks. Users will be attracted away from direct calls to .equals. With P3, we can use op== for CE as in P1, and the new op for NE. Users will be attracted away from direct calls to .equals. Sadly, legacy use of op==(ref,ref) makes P2 difficult to adopt. We could make an agreement, as with float, that op==(ref,ref) means one thing in legacy code, while op==(any,any) means something different (and better) in any-generic code. Note that legacy code includes generics, in which op== is CE not NE. There are good arguments that CE is so useful that we *should* allocate an operator to it, anyway. But if this argument is accepted, it provides even more motivation for an operator for NE, since NE is extremely common. For example, many uses of op== (CE) are written simply to support the idiom for NE (Objects.equals). And if we default field comparison to NE not CE, how is it that users are given the less-friendly notation for NE and the more friendly one for NE? At the cost of another bikeshed and some sugar, P3 would allow users and optimizers smoother access to NE. Too much sugar is bad for the diet, but this sugar (op===) might just be the right way to lead Hansel and Gretel to the correct house, instead of stumbling through the woods with the Objects.equals incantation. ? John From brian.goetz at oracle.com Wed May 18 14:57:24 2016 From: brian.goetz at oracle.com (Brian Goetz) Date: Wed, 18 May 2016 10:57:24 -0400 Subject: Value equality In-Reply-To: <2E984345-E7DC-4FA7-ADED-BE9C10626F29@oracle.com> References: <2127B453-99E7-4879-9097-F46BE46DB0C3@oracle.com> <2E984345-E7DC-4FA7-ADED-BE9C10626F29@oracle.com> Message-ID: Great summary of the options. For those who didn't read the whole thing: - CE is bitwise equality -- "are these two things identical copies" - OE is calling Object.equals() - NE (for values) is the synthetic "recurse with == on primitive components, NE on value components, and OE on reference components" If it were 1995, and we were inventing Java (and we didn't have our heads addled with an interpreter-based cost model), what would we do? I think we'd bind ==(ref,ref) to OE, with an (uglier-named) API point for CE (e.g., Objects.isSameReference) which would be used (a) for known-interned things, (b) for IdentityHashMap, (c) as a default implementation of Object.equals(), and (d) possibly as a short-circuiting optimization *inside* overrides of equals(). This hypothetical world (call it J') still gives users the choice of CE vs OE whenever they want, while nudging users towards OE (by giving it the prime syntactic real estate) which is probably what they want most of the time. Why didn't we do this in 1995? Hard to know (I'll ask James next time I see him), but I'd posit two main forces: - C bias. Since C has *only* CE (and it was desirable to make Java feel like "a safer C") it probably seemed like a big improvement already to offer programmers both CE and OE on all references, and binding == to OE probably seemed too radical at the time. - Cost-model bias. In the Java 1.0 days, pointer comparison was probably 100x faster in the interpreter than a virtual call to Object.equals(). If binding == to OE was even considered, it was probably deemed implausible. Of course, both of these feel a bit silly 20 years later, but here we are. So, in a J' world, what would we do with ==(val,val)? I think it would be a no-brainer -- bind it to NE, since Java developers would already associate == with a deeper comparison. Then we'd just have to adjust whatever the API point for CE is to also accomodate CE on values, and we'd be done. But, we don't live in J' world. So our choices become: P1: Bind ==(val,val) to CE, as we do with refs. Optimization challenges with the usual (a==b || a.equals(b)) idiom [1], but the rules work the same for values and refs. P2: Bind ==(val,val) to NE. This is J' world for values and J world for refs. (With even bigger optimization challenges for the (a==b || a.equals(b)) idiom.) Rules are different for values and refs, meaning (a) users will have to keep in mind which world they're in, (b) when migrating a class from ref to value they'll have to find and update all equality comparisons (!), (c) writing code that's generic over values and refs has to use an idiom that works on both, (d) when migrating code from ref-generic to any-generic, inspect every equality comparison to make sure it's still what was intended. P3: Add a new equality operator. I've already been laughed at enough, thank you. P4: Ban ==(val,val). This might be fine in value-only code, but it complicates writing generic code, especially migrating generic code. [1] John points out that if == is CE, then (a==b||a.equals(b)) will redundantly load the fields on failed ==. But, many equals implementations start with "a==b" as a short-circuiting optimization, which means "a==b" will be a common (pure) subexpression in the resulting expansion (and for values, methods are monomorphic and will get inlined more frequently), so the two checks can be collapsed. > Going back to op==, there are two plausible options for binding it to > new types: > > (P1) Syntax of op==(val,val) and op==(any,any) binds to CE as with > op==(ref,ref). Therefore, NE is uniformly reached by today's idiom, > which traverses value fields twice. > > (P2) Syntax of op==(val,val) and op==(any,any) is direct access to > NE. CE is reachable by experts at System.isEqualCopy. The old idiom > for NE works also calls equals twice. > > (P3) Same as P1, op== is uniform access to CE. New op (spelled > "===", ".==", "=~", etc.) is uniform, optimizable access to NE, > attracting users away from legacy idiom for NE. From john.r.rose at oracle.com Wed May 18 22:11:12 2016 From: john.r.rose at oracle.com (John Rose) Date: Wed, 18 May 2016 15:11:12 -0700 Subject: Value equality In-Reply-To: References: <2127B453-99E7-4879-9097-F46BE46DB0C3@oracle.com> <2E984345-E7DC-4FA7-ADED-BE9C10626F29@oracle.com> Message-ID: <21CE18EF-68C7-48ED-8781-046B61C429BD@oracle.com> On May 18, 2016, at 7:57 AM, Brian Goetz wrote: > > [1] John points out that if == is CE, then (a==b||a.equals(b)) will redundantly load the fields on failed ==. But, many equals implementations start with "a==b" as a short-circuiting optimization, which means "a==b" will be a common (pure) subexpression in the resulting expansion (and for values, methods are monomorphic and will get inlined more frequently), so the two checks can be collapsed. Proposed acronym: Call the explicit legacy idiom for equality "LIFE". This is "a==b||a.equals(b)", maybe with a null (or NaN?) check. Semantically it is CE||OE. The question of optimization is worth expanding on. In a nutshell, pure control flow test like a.f=ACMP=b.f are often collapsible. (The dominating test makes the later test go away, or in other words, the dominated test can rise upward to merge into the dominating test.) But, even pure tests cannot be sunk past exception-producing code. If a sub-operation of OE can produce an exception, it cannot be reordered with a previous CE sub-operation. Unfortunately, exceptions provide a semantic "window" into order of operations; this is one of the major difficulties with Java optimization. For example: if (a.someTag != b.someTag) return NOT_EQUAL; //CE, first pass if (!a.equals(b)) return NOT_EQUAL; //OE, second pass return IS_EQUAL; If the heavy comparison somehow includes a repetition of the "someTag" comparison, that latter comparison can be removed as a dominating test. But, if the heavy comparison can throw an exception (or make any side effect), then the "someTag" comparison cannot be moved *down* into or past the equals method. The net of this is that the writer of the equals method has, in general, no control over the first pass of the equals algorithm, even after optimization. The code might get some spot benefits from removal of explicit tests in the OE pass if they are already covered (dominated) by the CE pass. If the code *omits* the equivalent of a CE test (e.g., an ACMP comparison) for some reason, the CE test is extra work. Not much extra work, but it could cause extra fills or spills as the JIT works to provide a first-pass access to a component that will be needed, again, in a second pass after arbitrary intermediate labor. So a mandatory LIFE sentence (couldn't resist that), if op== is CE (that's P1), means some loss of performance control around equals methods. By comparison, if op== is OE (that's P2), there is no need to explicitly call CE; the .equals method does the exact job, and the coder has control. In legacy code where op== is OE, the LIFE expression produces back-to-back calls to .equals, semantically OE||OE. That would be even worse than CE||OE, unless we have a way to optimize OE||OE to OE, or until the users pick the LIFE (or the Legacy Idiom Connoting Equality) out of their code. But, if it is allowed, collapsing two adjacent calls to .equals is a much simpler JIT transform than merging an adjacent CE and NE. It's a macro-level CSE on partially lowered IR. As noted above, merging a CE into an NE requires that all the logic of the CE be performed before the first possible exception produced by NE. Let's try a more complete example: class MyVal { TypeDesc type; long payload1; long[] payload2; boolean equals(MyVal that) { return this.type.equals(that.type) && this.payload1 == that.payload1 && Arrays.equals(this.payload1, that.payload2); } } class User { boolean foo(MyVal a, MyVal b) { return a==b || a.equals(b); // CE || OE == LIFE } } In P1, the intermediate representation for the legacy idiom will contain these operations: PASS1: // start CE if (a.type !ACMP= b.type) goto PASS2; if (a.payload1 !LCMP= b.payload1) goto PASS2; if (a.payload2 !ACMP= b.payload2) goto PASS2; return IS_EQUAL; PASS2: // start OE z = a.type.INVOKEVIRTUAL[TypeDesc, equals, (TypeDesc)Z](b.type); //[DT a.type?] ? // random logic here; can throw exceptions in general if (!z) return NOT_EQUAL; if (a.payload1 !LCMP= b.payload1) return NOT_EQUAL; //[DT a.payload1] z = INVOKESTATIC[Arrays, equals, ([J [J)Z](a.payload2, b.payload2); //[DT a.payload2] ? // random logic here; can throw exceptions in general if (!z) return NOT_EQUAL; return IS_EQUAL; What's wrong with this? Well, from the JIT POV it is micromanagement, by the CE code, of the order of operations. The JIT has to reverse any unfortunate decisions in the CE code if it wants to respect the order of operations in the OE code. From the POV of the author of MyVal.equals. the CE enforces a policy that it is more profitable to compare the bits of payload1 and the object identity of payload2 before the structure of TypeDesc is consulted. Sometimes this is right, sometimes it is wrong, but the CE+OE idiom fits everybody into a one-size-fits all policy, and takes away ordering control from the .equals method. Could the JIT reorder stuff any way it wants, and make the code more like what the author intended? After all, CE is provably a subset of OE. There are two problems with that. First, the subset relation is subtle and not visible (to the JIT) at the bytecode level. (After all, CE and OE are two completely different bytecode shapes.) Second, as noted above, the possibility of exceptions places barriers to reordering. You cannot reorder exceptions with normal exits. In the above IR, if the payload1 fields differ, TypeDesc.equals is never called, so even if it were going to throw an exception (due to an assert or a bug or a malformed MyVal or TypeDesc, say) the IR says it can't, and the JIT can't reorder the CE code to merge it into the OE code. In this particular example, the possible optimizations are removal of the dominated tests marked "[DT ?]" above. The DT on payload2 is inside the implementation of Arrays.equals. The DT on a.type may or may not be present in the TypeDesc.equals method; it's a programmer choice. More perturbations are possible with larger value objects: If you add more 64-bit payload fields, the bitwise comparison clearly needs to load the entire value object (from whatever buffer or box it is in) before the first call to TypeDesc.equals. If the programmer knows that TypeDesc.equals method is likely to prove inequality, without examination of the payload, then the first pass can be wasted bandwidth, in cases of degenerate values with (say) all zeroes payloads. Is this a problem? Depends on the data structure. In this case, the dominated tests (e.g., DT of payload1) can usually be elided, so at worst it is a reordering of programmer intentions. On the other hand, if you add more reference fields (like type or payload2), the first pass amounts to a reordering of (likely) ACMP operations inside the .equals method of each successive component. Is that bad? Well, as noted above, it forces the JIT to load each reference field twice, once to do ACMP, and then (much later) to call .equals. Are there enough registers to avoid double loads? Often, yes, until the computation gets register bound, and then you simply have multiple loads, just to ensure the semantics of exception ordering. In summary, optimizing CE || OE (the legacy semantics of LIFE) is not horrible but will remove some programmer control from .equals ordering, and will cause more data motion in the presence of multiple reference components. LIFE with P1 will also make the JIT slightly slower and less robust, as it pattern matches on more ad hoc and verbose bytecode shapes. (Complex pattern matching is difficult; that's why we moved the string concatenation idioms into invokedynamic in JDK 9.) Let's go to P2 for a moment. In the presence of LIFE, the IR will first contain unexpanded intrinsics for a repeated OE || OE: // a==b z = INTRINSIC[a.equals(b)]; if (!z) goto NOT_EQUAL; // a.equals(b) z = INTRINSIC[a.equals(b)]; if (!z) goto NOT_EQUAL; goto IS_EQUAL; We need a special permission, an above-the-bytecode guarantee that any normal or abnormal result produced by the second call will be produced by the first call. In that case, we can elide the IR for the second call (as provably redundant): z = INTRINSIC[a.equals(b)]; if (!z) goto NOT_EQUAL; goto IS_EQUAL; Expansion then continues as above, but starting with PASS2. Code needs less optimization and is more under the control of the author of .equals. HTH ? John -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.goetz at oracle.com Thu May 19 14:36:13 2016 From: brian.goetz at oracle.com (Brian Goetz) Date: Thu, 19 May 2016 10:36:13 -0400 Subject: Species-static members vs singletons Message-ID: <4219406f-d842-97f4-7206-3a91ffe1e75c@oracle.com> We discussed two primary means to surface species-specific members in the language: a "species" placement (name TBD) as distinct from static and instance, or a "singleton" abstraction (a la Scala's "object" abstraction, as Peter L suggested). We've done some experiments comparing the two approaches. Separately, we discussed two strategies for handling this at the VM level: having three separate placements (ACC_STATIC, ACC_SPECIES, and instance) or retconning ACC_STATIC to mean "species" and using compiler trickery to simulate traditional statics. In recent discussions with Oracle and IBM VM folks, they seemed happy enough with having a new placement (and possibly new bytecodes, {get,put,invoke}species, or overloading these onto *static with ParamTypes in the owner field of the various XxxRef constants.) There are several places where the language itself can take advantage of species members: 1. Reifying type variables. For an any-generic class Foo, the compiler can generate public static final reflection-thingie-valued fields called "T" and "U", which means that "aFoo.T" (as an ordinary field ref!) would evaluate to the reflective mirror for the reified T -- if present, otherwise it would evaluate to the reflective mirror for 'erased'. 2. Representation of generic methods. The current translation strategy has us translating any-generic methods to classes; a static method static void foo(T t) { } translates to a class (plus an erased bridge): bridge static foo(Object o) { ... invoke erased specialization ... } static class Xxx$foo { void foo(T t) { ... } } This means that an instance of Xxx$foo is needed to invoke the method -- but serves solely to carry the type variables -- which is unfortunate. If instead we translate as: static class Xxx$foo { *species-static *void foo(T t) { ... } } then we can invoke this method via invokespecies: invokespecies ParamType[Xxx$foo, T_inf].foo(T_inf) where T_inf is the erasure-normalized type inferred for T (reified if value, `erased` reference.) No fake receiver required. The translation for generic instance methods is still somewhat messier (will post separately), but still less messy than if we also had to manage / cache a receiver. We also drafted some examples of how such a facility would be used, writing them both with species-static and with singleton. Examples and notes below; the summary is that in all cases, the species-static version is either better or about as good. 1. The old favorite, caching an instantiated instance. Species Singleton class Collections { private static class Holder { private species List empty = new EmptyList(); } static List emptyList() { return Holder.empty; } } class Collections { private singleton Holder { private empty = new EmptyList(); } static List emptyList() { return Holder.empty; } } Note that in this case, species by itself isn't enough -- we still need a holder class, and its a bit ugly. Arguably we could merge Holder into EmptyList (if that's under our control) but because Collections is an old-style "static bag" class (aka "sin bin"), we would still need a holder class for state. (Collections could share a single holder for multiple things; empty list, empty set, etc.) Neither the left nor the right seems particularly better than the other here. (If we were putting this method on Collection, where it would likely go in new code since now interfaces can have statics, the species approach would win, since we'd not need the holder class any more.) 2. Instantiation tracking. Species Singleton class Foo { private species int count; private species List> foos; public Foo() { ++count; foos.add(this); } } class Foo { private singleton FooStuff { private int count; private List> foos; } public Foo() { ++Foo.count; Foo.foos.add(this); } } Because the state is directly tied to the instantiation, the left seems more attractive -- doesn't require an extra artifact, and the constructor body seems more straightforward. 3. Implicit-like associations. Here, we're caching type associations. For example, suppose we have a Box, and we want to cache the associated class for List. Species Singleton class Box { private species Class> listClass = Class.forSpecialization(List, T.crass); } class Box { private singleton ListBuddy { Class> clazz = Class.forSpecialization(List, T.crass); } } The extra singleton declaration feels like "noise" here, because again the association is with the full set of type args for the class. 4. Static factories. Arguably, it makes sense to move factories to the types they describe. Species Singleton interface List { private species List empty = new EmptyList<>(); species List emptyList() { return empty; } } interface List { private singleton Stuff { List empty = new EmptyList<>(); } species List emptyList() { return Stuff.empty; } } In this model, you'd get an empty list with List aList = List.empty() rather than List aList = Collections.empty(); In the latter, the type witnesses can be omitted; in the former they probably can be as well but that's something new. 5. Typevar shredding. Here, we have separate state for different subsets of variables. This should be the place where the singleton approach shines. Species Singleton class HashMap { private static class Keys { species Set allKeys = ... } private static class Vals { species Set allVals = ... } void put(K k, V v) { Keys.allKeys.add(k); Vals.allVals.add(v); } } class HashMap { private singleton Keys { Set allKeys = ... } private singleton Vals { Set allVals = ... } void put(K k, V v) { Keys.allKeys.add(k); Vals.allVals.add(v); } } But, it doesn't really shine that much; the left is not really much worse than the right, just a little more fussy. In cases where the singleton approach is more natural, the corresponding "species in static class" idiom isn't so bad either. But in cases where the species approach is more natural, there's something unappealing about creating classes (both in source and runtime footprint) in cases 2/3/4 when we don't need one. The only place where the singleton approach seems to win big is when there are multiple variables in the same scope bound by invariants -- here, the singleton having a ctor is a big win -- but how often does this happen? So our conclusion is that the species-placement is as good or better for the identified use cases -- and it also fits cleanly into the existing model for member placement. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.goetz at oracle.com Fri May 20 18:33:00 2016 From: brian.goetz at oracle.com (Brian Goetz) Date: Fri, 20 May 2016 14:33:00 -0400 Subject: Wildcards -- Models 4 and 5 Message-ID: <2421796e-cb8e-6241-9f80-5bb761673f2e@oracle.com> In the 4/20 mail ?Wildcards and raw types: story so far?, we outlined our explorations for fitting wildcard types into the first several prototypes. The summary was: * Model 1: no wildcards at all * Model 2: A pale implementation of wildcards, with lots of problems that stem from trying to fake wildcards via interfaces * Model 3: basically the same as Model 2, except members are accessed via indy (which mitigated some of the problems but not all) The conclusion was: compiler-driven translation tricks are not going to cut it (as we suspected all along). We?ve since explored two other models (call them 4 and 5) which explore a range of options for VM support for wildcards. The below is a preliminary analysis of these options. Reflection, classes, and runtime types While it may not be immediately obvious that this subject is deeply connected to reflection, consider a typical implementation of |equals()|: |class Box { T t; public boolean equals(Object o) { if (!(o instanceof Box)) return false; Box other = (Box) o; return (t == null && other.t == null) || t.equals(other.t); } } | Some implementations use raw types (|Box|) for the |instanceof| and cast target; others use wildcards (|Box|). While the latter is recommended, both are widely used in circulation. In any case, as observed in the last mail, were we to interpret |Box| or |Box| as only including erased boxes, then this code would silently break. The term ?class? is horribly overloaded, used to describe the source class (|class Foo { ... }|), the binary classfile, the runtime type derived from the classfile, and the reflective mirror for that runtime type. In the past these existed in 1:1 correspondence, but no more ? a single source class now gives rise to a number of runtime types. Having poor terminology causes confusion, so let?s refine these terms: * /class/ refers to a source-level class declaration * /classfile/ refers to the binary classfile * /template/ refers to the runtime representation of a classfile * /runtime type/ refers to a primitive, value, class, or interface type managed by the VM So historically, all objects had a class, which equally described the source class, the classfile, and the runtime type. Going forward, the class and the runtime type of an object are distinct concepts. So an |ArrayList| has a /class/ of |ArrayList|, but a /runtime type/ of |ArrayList|. Our code name for runtime type is /crass/ (obviously a better name is needed, but we?ll paint that bikeshed later.) This allows us to untangle a question that?s been bugging us: what should |Object.getClass()| return on an |ArrayList|? If we return |ArrayList|, then we can?t distinguish between an erased and a specialized object (bad); if we return |ArrayList|, then existing code that depends on |(x.getClass() == List.class)| may break (bad). The answer is, of course, that there are two questions the user can ask an object: what is your /class/, and what is your /crass/, and they need to be detangled. The existing method |getClass()| will continue to return the class mirror; a new method (|getCrass()|) will return a runtime type mirror of some form for the runtime type. Similarly, a class literal will evaluate to a class, and some other form of literal / reflective lookup will be needed for crass. The reflective features built into the language (|instanceof|, casting, class literals, |getClass()|) are mostly tilted towards classes, not types. (Some exceptions: you can use a wildcard type in an |instanceof|, and you can do unchecked static casts to generic types, which are erased.) We need to extend these to deal in both classes /and/ crasses. For |getClass()| and literals, there?s an obvious path: have two forms. For casting, we are mostly there (except for the treatment of raw types for any-generic classes ? which we need to work out separately.) For instanceof, it seems a forced move that |instanceof Foo| is interpreted as ?an instance of any runtime type projected from class Foo?, but we also would want to apply it to any reifiable type as well. Wildcard types In Model 3, we express a parameterized type with a |ParamType| constant, which names a template class and a set of type parameters, which include both valid runtime types as well as the special type parameter token |erased|. One natural way to express a wildcard type is to introduce a new special type parameter token, |wild|, so we?d translate |Foo| as |ParamType[Foo,wild]|. In order for wildcard types to work seamlessly, the minimum functionality we?d need from the VM is to manage subtyping (which is used by the VM for |instanceof|, |checkcast|, verification, array store checks, and array covariance.) The wildcard must be seen to be a ?top? type for all parameterizations: |ParamType[Foo,T] <: ParamType[Foo,wild] // for all valid T | And, wildcard parameterizations must be seen to be subtypes of of their wildcard-parameterized supertypes. If we have |class Foo extends Bar implements I { ... } class Moo extends Goo { } | then we expect |ParamType[Foo,wild] <: ParamType[Bar,wild] ParamType[Foo,wild] <: ParamType[I,wild] ParamType[Moo,wild] <: Goo | Wildcards must also support method invocation and field access to the members that are in the intersection of the members of all parameterizations (these are the total members (those not restricted to particular instantiations) whose member descriptors do not contain any type variables.) We can continue to implement member access via invokedynamic (as we do in Model 3, or alternately, the VM can support |invoke*| bytecodes on wildcard receivers.) We can apply these wildcard behaviors to any of the wildcard models (i.e., retrofit them onto Model 2/3.) Partial wildcards With multiple type variables, the rules for wildcards generalize cleanly, but the number of wildcard types that are a supertype of any given parameterized type grows exponentially in the number of type variables. We are considering adopting the simplification of erasing all partial wildcards in the source type system to a total wildcard in the runtime type system (the costs of this are: some additional boxing on access paths where boxing might not be necessary, and unchecked casts when casting a broader wildcard to a narrower one.) Model 4 A constraint we are under is: existing binaries translate the types |Foo| (raw type), |Foo| (erased parameterization), and |Foo| all as |LFoo;| (or its equivalent, |CONSTANT_Class[Foo]|); since existing code treats this as meaning an erased class, the natural path would be to continue to interpret |LFoo;| as an erased class. Model 4 asks the question: ?can we reinterpret legacy |LFoo;| in classfiles, and |Foo| in source files, as |any Foo|? (restoring the interpretation of |Foo| to be more in line with user intuition.) Not surprisingly, the cost of reinterpreting the binaries is extensive. Many bytecodes would have to be reinterpreted, including |new|, |{get,put}field|, |invoke*|, to make up the difference between the legacy meaning of these constructs and the desired new meaning. Worse, while boxing provides us a means to have a common representation of signatures involving |T| (T?s bound), in order to get to a common representation for signatures involving |T[]|, we?d need to either (a) make |int[]| a subtype of |Object[]| or (b) have a ?boxing conversion? from |int[]| to |Object[]| (which would be a proxy box; the data would still live in the original |int[]|.) Both are intrusive into the |aaload| and |aastore| bytecodes and still are not anomaly-free. So, overall, while this seems possible, the implementation cost is very high, all of which is for the sake of migration, which will remain as legacy constraints long after the old code has been migrated. Model 5 Model 5 asks the simpler question: can we continue to interpret |LFoo;| as erased in legacy classfiles, but upgrade to treating |Foo| as is expected in source code? This entails changing the compilation translation of |Foo| from ?erased foo? to |ParamType[Foo,wild]|. This is far less intrusive into the bytecode behavior ? legacy code would continue to mean what it did at compile time. It does require some migration support for handling the fact that field and method descriptors have changed (but this is a problem we?re already working on for managing the migration of reference classes to value classes.) There are also some possible source incompatibilities in the face of separate compilation (to be quantified separately). Model 5 allows users to keep their |Foo| and have it mean what they think it should mean. So we don?t need to introduce a confusing |Foo| wildcard, but we will need a way of saying ?erased Foo?, which might be |Foo| or might be something more compact like |Foo|. Comparison Comparing the three models for wildcards (2, 4, 5): * Model 2 defines the source construct |Foo| to permanently mean |Foo|, even when |Foo| is anyfied, and introduces a new wildcard |Foo| ? but maintains source and binary compatibility. * Model 4 let?s us keep |Foo|, and retroactively redefines bytecode behavior ? so an old binary can still interoperate with a reified generic instance, and will think a |Foo| is really a |Foo|. * Model 5 redefines the /source/ meaning of |Foo| to be what users expect, but because we don?t reinterpret old binaries, allows some source incompatibility during migration. I think this pretty much explores the solution space. Our choices are: break the user model of what |Foo| means, take a probably prohibitive hit to distort the VM to apply new semantics to old bytecode, or accept some limited source incompatibility under separate compilation but rescue the source form that users want. In my opinion, the Model 5 direction offers the best balance of costs and benefits ? while there is some short-term migration pain (in relatively limited cases, and can be mitigated with compiler help), in the long run, it gets us to the world we want without permanently burdening either the language (creating confusion between |Foo| and |Foo|) or the VM implementation. In all these cases, we still haven?t defined the semantics of /raw types/. Raw types existed for migration between pre-generic and generic code; we still have that migration problem, plus the new migration problems of generic to any-generic, and of pre-generic to any-generic. So in any case, we?re going to need to define suitable semantics for raw types corresponding to any-generic classes. ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.levart at gmail.com Sun May 22 22:58:23 2016 From: peter.levart at gmail.com (Peter Levart) Date: Mon, 23 May 2016 00:58:23 +0200 Subject: Species-static members vs singletons In-Reply-To: <4219406f-d842-97f4-7206-3a91ffe1e75c@oracle.com> References: <4219406f-d842-97f4-7206-3a91ffe1e75c@oracle.com> Message-ID: <8abbeaf3-6c46-01cf-8e18-94f40299e9cb@gmail.com> Hi Brian, I agree that "species" placement is a better, less verbose option. But how to solve the language problem of having "species" and "instance" members of the same "type-variable" type be assignable to one-another? For example: class Foo { species T st; T it; void m() { it = st; // this can not be allowed st = it; // this can be allowed // maybe this could be allowed? @SuppressWarnings("unchecked") it = (T) st; } Singleton abstraction has the same problem. So while technically possible, it would be weird to have 'T' sometimes not be assignable to 'T'. Can we live with that? Regards, Peter On 05/19/2016 04:36 PM, Brian Goetz wrote: > We discussed two primary means to surface species-specific members in > the language: a "species" placement (name TBD) as distinct from static > and instance, or a "singleton" abstraction (a la Scala's "object" > abstraction, as Peter L suggested). We've done some experiments > comparing the two approaches. > > Separately, we discussed two strategies for handling this at the VM > level: having three separate placements (ACC_STATIC, ACC_SPECIES, and > instance) or retconning ACC_STATIC to mean "species" and using > compiler trickery to simulate traditional statics. In recent > discussions with Oracle and IBM VM folks, they seemed happy enough > with having a new placement (and possibly new bytecodes, > {get,put,invoke}species, or overloading these onto *static with > ParamTypes in the owner field of the various XxxRef constants.) > > > There are several places where the language itself can take advantage > of species members: > > 1. Reifying type variables. For an any-generic class Foo, the > compiler can generate public static final reflection-thingie-valued > fields called "T" and "U", which means that "aFoo.T" (as an ordinary > field ref!) would evaluate to the reflective mirror for the reified T > -- if present, otherwise it would evaluate to the reflective mirror > for 'erased'. > > 2. Representation of generic methods. The current translation > strategy has us translating any-generic methods to classes; a static > method > > static void foo(T t) { } > > translates to a class (plus an erased bridge): > > bridge static foo(Object o) { ... invoke erased specialization ... } > > static class Xxx$foo { > void foo(T t) { ... } > } > > This means that an instance of Xxx$foo is needed to invoke the method > -- but serves solely to carry the type variables -- which is > unfortunate. If instead we translate as: > > static class Xxx$foo { > *species-static *void foo(T t) { ... } > } > > then we can invoke this method via invokespecies: > > invokespecies ParamType[Xxx$foo, T_inf].foo(T_inf) > > where T_inf is the erasure-normalized type inferred for T (reified if > value, `erased` reference.) No fake receiver required. > > The translation for generic instance methods is still somewhat messier > (will post separately), but still less messy than if we also had to > manage / cache a receiver. > > > We also drafted some examples of how such a facility would be used, > writing them both with species-static and with singleton. Examples > and notes below; the summary is that in all cases, the species-static > version is either better or about as good. > > > > 1. The old favorite, caching an instantiated instance. > > Species > Singleton > class Collections { > private static class Holder { > private species List empty = new EmptyList(); > } > > static List emptyList() { return Holder.empty; } > } > class Collections { > private singleton Holder { > private empty = new EmptyList(); > } > > static List emptyList() { return Holder.empty; } > } > > > Note that in this case, species by itself isn't enough -- we still > need a holder class, and its a bit ugly. Arguably we could merge > Holder into EmptyList (if that's under our control) but because > Collections is an old-style "static bag" class (aka "sin bin"), we > would still need a holder class for state. (Collections could share a > single holder for multiple things; empty list, empty set, etc.) > > Neither the left nor the right seems particularly better than the > other here. (If we were putting this method on Collection, where it > would likely go in new code since now interfaces can have statics, the > species approach would win, since we'd not need the holder class any > more.) > > > 2. Instantiation tracking. > > Species > Singleton > class Foo { > private species int count; > private species List> foos; > > public Foo() { > ++count; > foos.add(this); > } > } > class Foo { > private singleton FooStuff { > private int count; > private List> foos; > } > > public Foo() { > ++Foo.count; > Foo.foos.add(this); > } > } > > > Because the state is directly tied to the instantiation, the left > seems more attractive -- doesn't require an extra artifact, and the > constructor body seems more straightforward. > > > 3. Implicit-like associations. Here, we're caching type > associations. For example, suppose we have a Box, and we want to > cache the associated class for List. > > > Species > Singleton > class Box { > private species Class> listClass > = Class.forSpecialization(List, T.crass); > } > class Box { > private singleton ListBuddy { > Class> clazz > = Class.forSpecialization(List, T.crass); > } > } > > > The extra singleton declaration feels like "noise" here, because again > the association is with the full set of type args for the class. > > > 4. Static factories. Arguably, it makes sense to move factories to > the types they describe. > > Species > Singleton > interface List { > private species List empty = new EmptyList<>(); > species List emptyList() { return empty; } > } > interface List { > private singleton Stuff { > List empty = new EmptyList<>(); > } > species List emptyList() { return Stuff.empty; } > } > > > In this model, you'd get an empty list with > > List aList = List.empty() > rather than > List aList = Collections.empty(); > > In the latter, the type witnesses can be omitted; in the former they > probably can be as well but that's something new. > > > 5. Typevar shredding. Here, we have separate state for different > subsets of variables. This should be the place where the singleton > approach shines. > > > Species > Singleton > class HashMap { > private static class Keys { > species Set allKeys = ... > } > > private static class Vals { > species Set allVals = ... > } > > void put(K k, V v) { > Keys.allKeys.add(k); > Vals.allVals.add(v); > } > } > class HashMap { > private singleton Keys { > Set allKeys = ... > } > > private singleton Vals { > Set allVals = ... > } > > void put(K k, V v) { > Keys.allKeys.add(k); > Vals.allVals.add(v); > } > } > > > > But, it doesn't really shine that much; the left is not really much > worse than the right, just a little more fussy. > > In cases where the singleton approach is more natural, the > corresponding "species in static class" idiom isn't so bad either. > But in cases where the species approach is more natural, there's > something unappealing about creating classes (both in source and > runtime footprint) in cases 2/3/4 when we don't need one. The only > place where the singleton approach seems to win big is when there are > multiple variables in the same scope bound by invariants -- here, the > singleton having a ctor is a big win -- but how often does this happen? > > > So our conclusion is that the species-placement is as good or better > for the identified use cases -- and it also fits cleanly into the > existing model for member placement. -------------- next part -------------- An HTML attachment was scrubbed... URL: From maurizio.cimadamore at oracle.com Mon May 23 10:05:32 2016 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Mon, 23 May 2016 11:05:32 +0100 Subject: Species-static members vs singletons In-Reply-To: <8abbeaf3-6c46-01cf-8e18-94f40299e9cb@gmail.com> References: <4219406f-d842-97f4-7206-3a91ffe1e75c@oracle.com> <8abbeaf3-6c46-01cf-8e18-94f40299e9cb@gmail.com> Message-ID: <5742D5EC.70608@oracle.com> Hi Peter, are you sure we need special treatment for 'it = st' ? After all, the compiler will issue unchecked warnings every time you'll try to access a species static from a non-reifiable type i.e. Foo.st = ""; //warn Foo.st = 42; //no warn In other words, can we put the burden of heap pollution-ness on the client and be happy? Maurizio On 22/05/16 23:58, Peter Levart wrote: > Hi Brian, > > I agree that "species" placement is a better, less verbose option. But > how to solve the language problem of having "species" and "instance" > members of the same "type-variable" type be assignable to one-another? > For example: > > class Foo { > species T st; > T it; > > void m() { > it = st; // this can not be allowed > st = it; // this can be allowed > > // maybe this could be allowed? > @SuppressWarnings("unchecked") > it = (T) st; > } > > > Singleton abstraction has the same problem. > > So while technically possible, it would be weird to have 'T' sometimes > not be assignable to 'T'. Can we live with that? > > Regards, Peter > > On 05/19/2016 04:36 PM, Brian Goetz wrote: >> We discussed two primary means to surface species-specific members in >> the language: a "species" placement (name TBD) as distinct from >> static and instance, or a "singleton" abstraction (a la Scala's >> "object" abstraction, as Peter L suggested). We've done some >> experiments comparing the two approaches. >> >> Separately, we discussed two strategies for handling this at the VM >> level: having three separate placements (ACC_STATIC, ACC_SPECIES, and >> instance) or retconning ACC_STATIC to mean "species" and using >> compiler trickery to simulate traditional statics. In recent >> discussions with Oracle and IBM VM folks, they seemed happy enough >> with having a new placement (and possibly new bytecodes, >> {get,put,invoke}species, or overloading these onto *static with >> ParamTypes in the owner field of the various XxxRef constants.) >> >> >> There are several places where the language itself can take advantage >> of species members: >> >> 1. Reifying type variables. For an any-generic class Foo, the >> compiler can generate public static final reflection-thingie-valued >> fields called "T" and "U", which means that "aFoo.T" (as an ordinary >> field ref!) would evaluate to the reflective mirror for the reified T >> -- if present, otherwise it would evaluate to the reflective mirror >> for 'erased'. >> >> 2. Representation of generic methods. The current translation >> strategy has us translating any-generic methods to classes; a static >> method >> >> static void foo(T t) { } >> >> translates to a class (plus an erased bridge): >> >> bridge static foo(Object o) { ... invoke erased specialization ... } >> >> static class Xxx$foo { >> void foo(T t) { ... } >> } >> >> This means that an instance of Xxx$foo is needed to invoke the method >> -- but serves solely to carry the type variables -- which is >> unfortunate. If instead we translate as: >> >> static class Xxx$foo { >> *species-static *void foo(T t) { ... } >> } >> >> then we can invoke this method via invokespecies: >> >> invokespecies ParamType[Xxx$foo, T_inf].foo(T_inf) >> >> where T_inf is the erasure-normalized type inferred for T (reified if >> value, `erased` reference.) No fake receiver required. >> >> The translation for generic instance methods is still somewhat >> messier (will post separately), but still less messy than if we also >> had to manage / cache a receiver. >> >> >> We also drafted some examples of how such a facility would be used, >> writing them both with species-static and with singleton. Examples >> and notes below; the summary is that in all cases, the species-static >> version is either better or about as good. >> >> >> >> 1. The old favorite, caching an instantiated instance. >> >> Species >> Singleton >> class Collections { >> private static class Holder { >> private species List empty = new EmptyList(); >> } >> >> static List emptyList() { return Holder.empty; } >> } >> class Collections { >> private singleton Holder { >> private empty = new EmptyList(); >> } >> >> static List emptyList() { return Holder.empty; } >> } >> >> >> Note that in this case, species by itself isn't enough -- we still >> need a holder class, and its a bit ugly. Arguably we could merge >> Holder into EmptyList (if that's under our control) but because >> Collections is an old-style "static bag" class (aka "sin bin"), we >> would still need a holder class for state. (Collections could share >> a single holder for multiple things; empty list, empty set, etc.) >> >> Neither the left nor the right seems particularly better than the >> other here. (If we were putting this method on Collection, where it >> would likely go in new code since now interfaces can have statics, >> the species approach would win, since we'd not need the holder class >> any more.) >> >> >> 2. Instantiation tracking. >> >> Species >> Singleton >> class Foo { >> private species int count; >> private species List> foos; >> >> public Foo() { >> ++count; >> foos.add(this); >> } >> } >> class Foo { >> private singleton FooStuff { >> private int count; >> private List> foos; >> } >> >> public Foo() { >> ++Foo.count; >> Foo.foos.add(this); >> } >> } >> >> >> Because the state is directly tied to the instantiation, the left >> seems more attractive -- doesn't require an extra artifact, and the >> constructor body seems more straightforward. >> >> >> 3. Implicit-like associations. Here, we're caching type >> associations. For example, suppose we have a Box, and we want to >> cache the associated class for List. >> >> >> Species >> Singleton >> class Box { >> private species Class> listClass >> = Class.forSpecialization(List, T.crass); >> } >> class Box { >> private singleton ListBuddy { >> Class> clazz >> = Class.forSpecialization(List, T.crass); >> } >> } >> >> >> The extra singleton declaration feels like "noise" here, because >> again the association is with the full set of type args for the class. >> >> >> 4. Static factories. Arguably, it makes sense to move factories to >> the types they describe. >> >> Species >> Singleton >> interface List { >> private species List empty = new EmptyList<>(); >> species List emptyList() { return empty; } >> } >> interface List { >> private singleton Stuff { >> List empty = new EmptyList<>(); >> } >> species List emptyList() { return Stuff.empty; } >> } >> >> >> In this model, you'd get an empty list with >> >> List aList = List.empty() >> rather than >> List aList = Collections.empty(); >> >> In the latter, the type witnesses can be omitted; in the former they >> probably can be as well but that's something new. >> >> >> 5. Typevar shredding. Here, we have separate state for different >> subsets of variables. This should be the place where the singleton >> approach shines. >> >> >> Species >> Singleton >> class HashMap { >> private static class Keys { >> species Set allKeys = ... >> } >> >> private static class Vals { >> species Set allVals = ... >> } >> >> void put(K k, V v) { >> Keys.allKeys.add(k); >> Vals.allVals.add(v); >> } >> } >> class HashMap { >> private singleton Keys { >> Set allKeys = ... >> } >> >> private singleton Vals { >> Set allVals = ... >> } >> >> void put(K k, V v) { >> Keys.allKeys.add(k); >> Vals.allVals.add(v); >> } >> } >> >> >> >> But, it doesn't really shine that much; the left is not really much >> worse than the right, just a little more fussy. >> >> In cases where the singleton approach is more natural, the >> corresponding "species in static class" idiom isn't so bad either. >> But in cases where the species approach is more natural, there's >> something unappealing about creating classes (both in source and >> runtime footprint) in cases 2/3/4 when we don't need one. The only >> place where the singleton approach seems to win big is when there are >> multiple variables in the same scope bound by invariants -- here, the >> singleton having a ctor is a big win -- but how often does this happen? >> >> >> So our conclusion is that the species-placement is as good or better >> for the identified use cases -- and it also fits cleanly into the >> existing model for member placement. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.goetz at oracle.com Mon May 23 13:56:23 2016 From: brian.goetz at oracle.com (Brian Goetz) Date: Mon, 23 May 2016 06:56:23 -0700 Subject: Species-static members vs singletons In-Reply-To: <5742D5EC.70608@oracle.com> References: <4219406f-d842-97f4-7206-3a91ffe1e75c@oracle.com> <8abbeaf3-6c46-01cf-8e18-94f40299e9cb@gmail.com> <5742D5EC.70608@oracle.com> Message-ID: Note that we have this same problem with unchecked warnings today in many of the use cases. For example, in the ?cached empty list? case, we always have to use an unchecked cast to cast the cached list to the desired type. When we use species-static to do the same, and it is possible that the species could correspond to more than one T, we still have to do the same unchecked warning (and as you mention, the singleton form has the same problem.) I think its an unescapable consequence of erasure, but one we?re already sort of comfortable with. If you use a more constrained type selector (e.g., List), you won?t get a warning, as the compiler will know that st is exactly int. > On May 23, 2016, at 3:05 AM, Maurizio Cimadamore wrote: > > Hi Peter, > are you sure we need special treatment for 'it = st' ? After all, the compiler will issue unchecked warnings every time you'll try to access a species static from a non-reifiable type i.e. > > Foo.st = ""; //warn > Foo.st = 42; //no warn > > In other words, can we put the burden of heap pollution-ness on the client and be happy? > > Maurizio > > On 22/05/16 23:58, Peter Levart wrote: >> Hi Brian, >> >> I agree that "species" placement is a better, less verbose option. But how to solve the language problem of having "species" and "instance" members of the same "type-variable" type be assignable to one-another? For example: >> >> class Foo { >> species T st; >> T it; >> >> void m() { >> it = st; // this can not be allowed >> st = it; // this can be allowed >> >> // maybe this could be allowed? >> @SuppressWarnings("unchecked") >> it = (T) st; >> } >> >> >> Singleton abstraction has the same problem. >> >> So while technically possible, it would be weird to have 'T' sometimes not be assignable to 'T'. Can we live with that? >> >> Regards, Peter >> >> On 05/19/2016 04:36 PM, Brian Goetz wrote: >>> We discussed two primary means to surface species-specific members in the language: a "species" placement (name TBD) as distinct from static and instance, or a "singleton" abstraction (a la Scala's "object" abstraction, as Peter L suggested). We've done some experiments comparing the two approaches. >>> >>> Separately, we discussed two strategies for handling this at the VM level: having three separate placements (ACC_STATIC, ACC_SPECIES, and instance) or retconning ACC_STATIC to mean "species" and using compiler trickery to simulate traditional statics. In recent discussions with Oracle and IBM VM folks, they seemed happy enough with having a new placement (and possibly new bytecodes, {get,put,invoke}species, or overloading these onto *static with ParamTypes in the owner field of the various XxxRef constants.) >>> >>> >>> There are several places where the language itself can take advantage of species members: >>> >>> 1. Reifying type variables. For an any-generic class Foo, the compiler can generate public static final reflection-thingie-valued fields called "T" and "U", which means that "aFoo.T" (as an ordinary field ref!) would evaluate to the reflective mirror for the reified T -- if present, otherwise it would evaluate to the reflective mirror for 'erased'. >>> >>> 2. Representation of generic methods. The current translation strategy has us translating any-generic methods to classes; a static method >>> >>> static void foo(T t) { } >>> >>> translates to a class (plus an erased bridge): >>> >>> bridge static foo(Object o) { ... invoke erased specialization ... } >>> >>> static class Xxx$foo { >>> void foo(T t) { ... } >>> } >>> >>> This means that an instance of Xxx$foo is needed to invoke the method -- but serves solely to carry the type variables -- which is unfortunate. If instead we translate as: >>> >>> static class Xxx$foo { >>> species-static void foo(T t) { ... } >>> } >>> >>> then we can invoke this method via invokespecies: >>> >>> invokespecies ParamType[Xxx$foo, T_inf].foo(T_inf) >>> >>> where T_inf is the erasure-normalized type inferred for T (reified if value, `erased` reference.) No fake receiver required. >>> >>> The translation for generic instance methods is still somewhat messier (will post separately), but still less messy than if we also had to manage / cache a receiver. >>> >>> >>> We also drafted some examples of how such a facility would be used, writing them both with species-static and with singleton. Examples and notes below; the summary is that in all cases, the species-static version is either better or about as good. >>> >>> >>> >>> 1. The old favorite, caching an instantiated instance. >>> >>> Species >>> Singleton >>> class Collections { >>> private static class Holder { >>> private species List empty = new EmptyList(); >>> } >>> >>> static List emptyList() { return Holder.empty; } >>> } >>> class Collections { >>> private singleton Holder { >>> private empty = new EmptyList(); >>> } >>> >>> static List emptyList() { return Holder.empty; } >>> } >>> Note that in this case, species by itself isn't enough -- we still need a holder class, and its a bit ugly. Arguably we could merge Holder into EmptyList (if that's under our control) but because Collections is an old-style "static bag" class (aka "sin bin"), we would still need a holder class for state. (Collections could share a single holder for multiple things; empty list, empty set, etc.) >>> >>> Neither the left nor the right seems particularly better than the other here. (If we were putting this method on Collection, where it would likely go in new code since now interfaces can have statics, the species approach would win, since we'd not need the holder class any more.) >>> >>> >>> 2. Instantiation tracking. >>> >>> Species >>> Singleton >>> class Foo { >>> private species int count; >>> private species List> foos; >>> >>> public Foo() { >>> ++count; >>> foos.add(this); >>> } >>> } >>> class Foo { >>> private singleton FooStuff { >>> private int count; >>> private List> foos; >>> } >>> >>> public Foo() { >>> ++Foo.count; >>> Foo.foos.add(this); >>> } >>> } >>> Because the state is directly tied to the instantiation, the left seems more attractive -- doesn't require an extra artifact, and the constructor body seems more straightforward. >>> >>> >>> 3. Implicit-like associations. Here, we're caching type associations. For example, suppose we have a Box, and we want to cache the associated class for List. >>> >>> >>> Species >>> Singleton >>> class Box { >>> private species Class> listClass >>> = Class.forSpecialization(List, T.crass); >>> } >>> class Box { >>> private singleton ListBuddy { >>> Class> clazz >>> = Class.forSpecialization(List, T.crass); >>> } >>> } >>> The extra singleton declaration feels like "noise" here, because again the association is with the full set of type args for the class. >>> >>> >>> 4. Static factories. Arguably, it makes sense to move factories to the types they describe. >>> >>> Species >>> Singleton >>> interface List { >>> private species List empty = new EmptyList<>(); >>> species List emptyList() { return empty; } >>> } >>> interface List { >>> private singleton Stuff { >>> List empty = new EmptyList<>(); >>> } >>> species List emptyList() { return Stuff.empty; } >>> } >>> In this model, you'd get an empty list with >>> >>> List aList = List.empty() >>> rather than >>> List aList = Collections.empty(); >>> >>> In the latter, the type witnesses can be omitted; in the former they probably can be as well but that's something new. >>> >>> >>> 5. Typevar shredding. Here, we have separate state for different subsets of variables. This should be the place where the singleton approach shines. >>> >>> >>> Species >>> Singleton >>> class HashMap { >>> private static class Keys { >>> species Set allKeys = ... >>> } >>> >>> private static class Vals { >>> species Set allVals = ... >>> } >>> >>> void put(K k, V v) { >>> Keys.allKeys.add(k); >>> Vals.allVals.add(v); >>> } >>> } >>> class HashMap { >>> private singleton Keys { >>> Set allKeys = ... >>> } >>> >>> private singleton Vals { >>> Set allVals = ... >>> } >>> >>> void put(K k, V v) { >>> Keys.allKeys.add(k); >>> Vals.allVals.add(v); >>> } >>> } >>> >>> But, it doesn't really shine that much; the left is not really much worse than the right, just a little more fussy. >>> >>> In cases where the singleton approach is more natural, the corresponding "species in static class" idiom isn't so bad either. But in cases where the species approach is more natural, there's something unappealing about creating classes (both in source and runtime footprint) in cases 2/3/4 when we don't need one. The only place where the singleton approach seems to win big is when there are multiple variables in the same scope bound by invariants -- here, the singleton having a ctor is a big win -- but how often does this happen? >>> >>> >>> So our conclusion is that the species-placement is as good or better for the identified use cases -- and it also fits cleanly into the existing model for member placement. >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From maurizio.cimadamore at oracle.com Mon May 23 14:18:33 2016 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Mon, 23 May 2016 15:18:33 +0100 Subject: Species-static members vs singletons In-Reply-To: References: <4219406f-d842-97f4-7206-3a91ffe1e75c@oracle.com> <8abbeaf3-6c46-01cf-8e18-94f40299e9cb@gmail.com> <5742D5EC.70608@oracle.com> Message-ID: <57431139.5050401@oracle.com> Sorry - I now realize that the point I made in my earlier email was unclear. What I'm suggesting is to have a single rule for generating unchecked warnings that goes like this: "If the qualifier of a species static access is not reifiable, an unchecked warning should occur". In the example Peter sent, the only thing worth mentioning is that the qualifier is 'implicit' (i.e. can be omitted and be assumed to be the current class Foo); now since Foo is not reifiable, every unqualified access to 'st' from Foo will get a warning - excluding, of course, accesses occurring in a context where T is restricted (i.e. __WhereVal(T)). Maurizio On 23/05/16 14:56, Brian Goetz wrote: > Note that we have this same problem with unchecked warnings today in > many of the use cases. For example, in the ?cached empty list? case, > we always have to use an unchecked cast to cast the cached list to the > desired type. When we use species-static to do the same, and it is > possible that the species could correspond to more than one T, we > still have to do the same unchecked warning (and as you mention, the > singleton form has the same problem.) I think its an unescapable > consequence of erasure, but one we?re already sort of comfortable with. > > If you use a more constrained type selector (e.g., List), you > won?t get a warning, as the compiler will know that st is exactly int. > >> On May 23, 2016, at 3:05 AM, Maurizio Cimadamore >> > > wrote: >> >> Hi Peter, >> are you sure we need special treatment for 'it = st' ? After all, the >> compiler will issue unchecked warnings every time you'll try to >> access a species static from a non-reifiable type i.e. >> >> Foo.st = ""; //warn >> Foo.st = 42; //no warn >> >> In other words, can we put the burden of heap pollution-ness on the >> client and be happy? >> >> Maurizio >> >> On 22/05/16 23:58, Peter Levart wrote: >>> Hi Brian, >>> >>> I agree that "species" placement is a better, less verbose option. >>> But how to solve the language problem of having "species" and >>> "instance" members of the same "type-variable" type be assignable to >>> one-another? For example: >>> >>> class Foo { >>> species T st; >>> T it; >>> >>> void m() { >>> it = st; // this can not be allowed >>> st = it; // this can be allowed >>> >>> // maybe this could be allowed? >>> @SuppressWarnings("unchecked") >>> it = (T) st; >>> } >>> >>> >>> Singleton abstraction has the same problem. >>> >>> So while technically possible, it would be weird to have 'T' >>> sometimes not be assignable to 'T'. Can we live with that? >>> >>> Regards, Peter >>> >>> On 05/19/2016 04:36 PM, Brian Goetz wrote: >>>> We discussed two primary means to surface species-specific members >>>> in the language: a "species" placement (name TBD) as distinct from >>>> static and instance, or a "singleton" abstraction (a la Scala's >>>> "object" abstraction, as Peter L suggested). We've done some >>>> experiments comparing the two approaches. >>>> >>>> Separately, we discussed two strategies for handling this at the VM >>>> level: having three separate placements (ACC_STATIC, ACC_SPECIES, >>>> and instance) or retconning ACC_STATIC to mean "species" and using >>>> compiler trickery to simulate traditional statics. In recent >>>> discussions with Oracle and IBM VM folks, they seemed happy enough >>>> with having a new placement (and possibly new bytecodes, >>>> {get,put,invoke}species, or overloading these onto *static with >>>> ParamTypes in the owner field of the various XxxRef constants.) >>>> >>>> >>>> There are several places where the language itself can take >>>> advantage of species members: >>>> >>>> 1. Reifying type variables. For an any-generic class Foo, >>>> the compiler can generate public static final >>>> reflection-thingie-valued fields called "T" and "U", which means >>>> that "aFoo.T" (as an ordinary field ref!) would evaluate to the >>>> reflective mirror for the reified T -- if present, otherwise it >>>> would evaluate to the reflective mirror for 'erased'. >>>> >>>> 2. Representation of generic methods. The current translation >>>> strategy has us translating any-generic methods to classes; a >>>> static method >>>> >>>> static void foo(T t) { } >>>> >>>> translates to a class (plus an erased bridge): >>>> >>>> bridge static foo(Object o) { ... invoke erased specialization >>>> ... } >>>> >>>> static class Xxx$foo { >>>> void foo(T t) { ... } >>>> } >>>> >>>> This means that an instance of Xxx$foo is needed to invoke the >>>> method -- but serves solely to carry the type variables -- which is >>>> unfortunate. If instead we translate as: >>>> >>>> static class Xxx$foo { >>>> *species-static *void foo(T t) { ... } >>>> } >>>> >>>> then we can invoke this method via invokespecies: >>>> >>>> invokespecies ParamType[Xxx$foo, T_inf].foo(T_inf) >>>> >>>> where T_inf is the erasure-normalized type inferred for T (reified >>>> if value, `erased` reference.) No fake receiver required. >>>> >>>> The translation for generic instance methods is still somewhat >>>> messier (will post separately), but still less messy than if we >>>> also had to manage / cache a receiver. >>>> >>>> >>>> We also drafted some examples of how such a facility would be used, >>>> writing them both with species-static and with singleton. Examples >>>> and notes below; the summary is that in all cases, the >>>> species-static version is either better or about as good. >>>> >>>> >>>> >>>> 1. The old favorite, caching an instantiated instance. >>>> >>>> Species >>>> Singleton >>>> class Collections { >>>> private static class Holder { >>>> private species List empty = new EmptyList(); >>>> } >>>> >>>> static List emptyList() { return Holder.empty; } >>>> } >>>> class Collections { >>>> private singleton Holder { >>>> private empty = new EmptyList(); >>>> } >>>> >>>> static List emptyList() { return Holder.empty; } >>>> } >>>> >>>> >>>> Note that in this case, species by itself isn't enough -- we still >>>> need a holder class, and its a bit ugly. Arguably we could merge >>>> Holder into EmptyList (if that's under our control) but because >>>> Collections is an old-style "static bag" class (aka "sin bin"), we >>>> would still need a holder class for state. (Collections could share >>>> a single holder for multiple things; empty list, empty set, etc.) >>>> >>>> Neither the left nor the right seems particularly better than the >>>> other here. (If we were putting this method on Collection, where >>>> it would likely go in new code since now interfaces can have >>>> statics, the species approach would win, since we'd not need the >>>> holder class any more.) >>>> >>>> >>>> 2. Instantiation tracking. >>>> >>>> Species >>>> Singleton >>>> class Foo { >>>> private species int count; >>>> private species List> foos; >>>> >>>> public Foo() { >>>> ++count; >>>> foos.add(this); >>>> } >>>> } >>>> class Foo { >>>> private singleton FooStuff { >>>> private int count; >>>> private List> foos; >>>> } >>>> >>>> public Foo() { >>>> ++Foo.count; >>>> Foo.foos.add(this); >>>> } >>>> } >>>> >>>> >>>> Because the state is directly tied to the instantiation, the left >>>> seems more attractive -- doesn't require an extra artifact, and the >>>> constructor body seems more straightforward. >>>> >>>> >>>> 3. Implicit-like associations. Here, we're caching type >>>> associations. For example, suppose we have a Box, and we want >>>> to cache the associated class for List. >>>> >>>> >>>> Species >>>> Singleton >>>> class Box { >>>> private species Class> listClass >>>> = Class.forSpecialization(List, T.crass); >>>> } >>>> class Box { >>>> private singleton ListBuddy { >>>> Class> clazz >>>> = Class.forSpecialization(List, T.crass); >>>> } >>>> } >>>> >>>> >>>> The extra singleton declaration feels like "noise" here, because >>>> again the association is with the full set of type args for the class. >>>> >>>> >>>> 4. Static factories. Arguably, it makes sense to move factories >>>> to the types they describe. >>>> >>>> Species >>>> Singleton >>>> interface List { >>>> private species List empty = new EmptyList<>(); >>>> species List emptyList() { return empty; } >>>> } >>>> interface List { >>>> private singleton Stuff { >>>> List empty = new EmptyList<>(); >>>> } >>>> species List emptyList() { return Stuff.empty; } >>>> } >>>> >>>> >>>> In this model, you'd get an empty list with >>>> >>>> List aList = List.empty() >>>> rather than >>>> List aList = Collections.empty(); >>>> >>>> In the latter, the type witnesses can be omitted; in the former >>>> they probably can be as well but that's something new. >>>> >>>> >>>> 5. Typevar shredding. Here, we have separate state for different >>>> subsets of variables. This should be the place where the singleton >>>> approach shines. >>>> >>>> >>>> Species >>>> Singleton >>>> class HashMap { >>>> private static class Keys { >>>> species Set allKeys = ... >>>> } >>>> >>>> private static class Vals { >>>> species Set allVals = ... >>>> } >>>> >>>> void put(K k, V v) { >>>> Keys.allKeys.add(k); >>>> Vals.allVals.add(v); >>>> } >>>> } >>>> class HashMap { >>>> private singleton Keys { >>>> Set allKeys = ... >>>> } >>>> >>>> private singleton Vals { >>>> Set allVals = ... >>>> } >>>> >>>> void put(K k, V v) { >>>> Keys.allKeys.add(k); >>>> Vals.allVals.add(v); >>>> } >>>> } >>>> >>>> >>>> >>>> But, it doesn't really shine that much; the left is not really much >>>> worse than the right, just a little more fussy. >>>> >>>> In cases where the singleton approach is more natural, the >>>> corresponding "species in static class" idiom isn't so bad either. >>>> But in cases where the species approach is more natural, there's >>>> something unappealing about creating classes (both in source and >>>> runtime footprint) in cases 2/3/4 when we don't need one. The only >>>> place where the singleton approach seems to win big is when there >>>> are multiple variables in the same scope bound by invariants -- >>>> here, the singleton having a ctor is a big win -- but how often >>>> does this happen? >>>> >>>> >>>> So our conclusion is that the species-placement is as good or >>>> better for the identified use cases -- and it also fits cleanly >>>> into the existing model for member placement. >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.goetz at oracle.com Mon May 23 14:20:09 2016 From: brian.goetz at oracle.com (Brian Goetz) Date: Mon, 23 May 2016 07:20:09 -0700 Subject: Species-static members vs singletons In-Reply-To: <57431139.5050401@oracle.com> References: <4219406f-d842-97f4-7206-3a91ffe1e75c@oracle.com> <8abbeaf3-6c46-01cf-8e18-94f40299e9cb@gmail.com> <5742D5EC.70608@oracle.com> <57431139.5050401@oracle.com> Message-ID: <40FBB763-E034-4D4B-8436-ADBC2B3E3C4B@oracle.com> Right. And Peter?s question is: (a) did we think of this (yes) and (b) are we OK with this. Which I think is also yes? > On May 23, 2016, at 7:18 AM, Maurizio Cimadamore wrote: > > Sorry - I now realize that the point I made in my earlier email was unclear. > > What I'm suggesting is to have a single rule for generating unchecked warnings that goes like this: > > "If the qualifier of a species static access is not reifiable, an unchecked warning should occur". > > In the example Peter sent, the only thing worth mentioning is that the qualifier is 'implicit' (i.e. can be omitted and be assumed to be the current class Foo); now since Foo is not reifiable, every unqualified access to 'st' from Foo will get a warning - excluding, of course, accesses occurring in a context where T is restricted (i.e. __WhereVal(T)). > > Maurizio > > On 23/05/16 14:56, Brian Goetz wrote: >> Note that we have this same problem with unchecked warnings today in many of the use cases. For example, in the ?cached empty list? case, we always have to use an unchecked cast to cast the cached list to the desired type. When we use species-static to do the same, and it is possible that the species could correspond to more than one T, we still have to do the same unchecked warning (and as you mention, the singleton form has the same problem.) I think its an unescapable consequence of erasure, but one we?re already sort of comfortable with. >> >> If you use a more constrained type selector (e.g., List), you won?t get a warning, as the compiler will know that st is exactly int. >> >>> On May 23, 2016, at 3:05 AM, Maurizio Cimadamore > wrote: >>> >>> Hi Peter, >>> are you sure we need special treatment for 'it = st' ? After all, the compiler will issue unchecked warnings every time you'll try to access a species static from a non-reifiable type i.e. >>> >>> Foo.st = ""; //warn >>> Foo.st = 42; //no warn >>> >>> In other words, can we put the burden of heap pollution-ness on the client and be happy? >>> >>> Maurizio >>> >>> On 22/05/16 23:58, Peter Levart wrote: >>>> Hi Brian, >>>> >>>> I agree that "species" placement is a better, less verbose option. But how to solve the language problem of having "species" and "instance" members of the same "type-variable" type be assignable to one-another? For example: >>>> >>>> class Foo { >>>> species T st; >>>> T it; >>>> >>>> void m() { >>>> it = st; // this can not be allowed >>>> st = it; // this can be allowed >>>> >>>> // maybe this could be allowed? >>>> @SuppressWarnings("unchecked") >>>> it = (T) st; >>>> } >>>> >>>> >>>> Singleton abstraction has the same problem. >>>> >>>> So while technically possible, it would be weird to have 'T' sometimes not be assignable to 'T'. Can we live with that? >>>> >>>> Regards, Peter >>>> >>>> On 05/19/2016 04:36 PM, Brian Goetz wrote: >>>>> We discussed two primary means to surface species-specific members in the language: a "species" placement (name TBD) as distinct from static and instance, or a "singleton" abstraction (a la Scala's "object" abstraction, as Peter L suggested). We've done some experiments comparing the two approaches. >>>>> >>>>> Separately, we discussed two strategies for handling this at the VM level: having three separate placements (ACC_STATIC, ACC_SPECIES, and instance) or retconning ACC_STATIC to mean "species" and using compiler trickery to simulate traditional statics. In recent discussions with Oracle and IBM VM folks, they seemed happy enough with having a new placement (and possibly new bytecodes, {get,put,invoke}species, or overloading these onto *static with ParamTypes in the owner field of the various XxxRef constants.) >>>>> >>>>> >>>>> There are several places where the language itself can take advantage of species members: >>>>> >>>>> 1. Reifying type variables. For an any-generic class Foo, the compiler can generate public static final reflection-thingie-valued fields called "T" and "U", which means that "aFoo.T" (as an ordinary field ref!) would evaluate to the reflective mirror for the reified T -- if present, otherwise it would evaluate to the reflective mirror for 'erased'. >>>>> >>>>> 2. Representation of generic methods. The current translation strategy has us translating any-generic methods to classes; a static method >>>>> >>>>> static void foo(T t) { } >>>>> >>>>> translates to a class (plus an erased bridge): >>>>> >>>>> bridge static foo(Object o) { ... invoke erased specialization ... } >>>>> >>>>> static class Xxx$foo { >>>>> void foo(T t) { ... } >>>>> } >>>>> >>>>> This means that an instance of Xxx$foo is needed to invoke the method -- but serves solely to carry the type variables -- which is unfortunate. If instead we translate as: >>>>> >>>>> static class Xxx$foo { >>>>> species-static void foo(T t) { ... } >>>>> } >>>>> >>>>> then we can invoke this method via invokespecies: >>>>> >>>>> invokespecies ParamType[Xxx$foo, T_inf].foo(T_inf) >>>>> >>>>> where T_inf is the erasure-normalized type inferred for T (reified if value, `erased` reference.) No fake receiver required. >>>>> >>>>> The translation for generic instance methods is still somewhat messier (will post separately), but still less messy than if we also had to manage / cache a receiver. >>>>> >>>>> >>>>> We also drafted some examples of how such a facility would be used, writing them both with species-static and with singleton. Examples and notes below; the summary is that in all cases, the species-static version is either better or about as good. >>>>> >>>>> >>>>> >>>>> 1. The old favorite, caching an instantiated instance. >>>>> >>>>> Species >>>>> Singleton >>>>> class Collections { >>>>> private static class Holder { >>>>> private species List empty = new EmptyList(); >>>>> } >>>>> >>>>> static List emptyList() { return Holder.empty; } >>>>> } >>>>> class Collections { >>>>> private singleton Holder { >>>>> private empty = new EmptyList(); >>>>> } >>>>> >>>>> static List emptyList() { return Holder.empty; } >>>>> } >>>>> Note that in this case, species by itself isn't enough -- we still need a holder class, and its a bit ugly. Arguably we could merge Holder into EmptyList (if that's under our control) but because Collections is an old-style "static bag" class (aka "sin bin"), we would still need a holder class for state. (Collections could share a single holder for multiple things; empty list, empty set, etc.) >>>>> >>>>> Neither the left nor the right seems particularly better than the other here. (If we were putting this method on Collection, where it would likely go in new code since now interfaces can have statics, the species approach would win, since we'd not need the holder class any more.) >>>>> >>>>> >>>>> 2. Instantiation tracking. >>>>> >>>>> Species >>>>> Singleton >>>>> class Foo { >>>>> private species int count; >>>>> private species List> foos; >>>>> >>>>> public Foo() { >>>>> ++count; >>>>> foos.add(this); >>>>> } >>>>> } >>>>> class Foo { >>>>> private singleton FooStuff { >>>>> private int count; >>>>> private List> foos; >>>>> } >>>>> >>>>> public Foo() { >>>>> ++Foo.count; >>>>> Foo.foos.add(this); >>>>> } >>>>> } >>>>> Because the state is directly tied to the instantiation, the left seems more attractive -- doesn't require an extra artifact, and the constructor body seems more straightforward. >>>>> >>>>> >>>>> 3. Implicit-like associations. Here, we're caching type associations. For example, suppose we have a Box, and we want to cache the associated class for List. >>>>> >>>>> >>>>> Species >>>>> Singleton >>>>> class Box { >>>>> private species Class> listClass >>>>> = Class.forSpecialization(List, T.crass); >>>>> } >>>>> class Box { >>>>> private singleton ListBuddy { >>>>> Class> clazz >>>>> = Class.forSpecialization(List, T.crass); >>>>> } >>>>> } >>>>> The extra singleton declaration feels like "noise" here, because again the association is with the full set of type args for the class. >>>>> >>>>> >>>>> 4. Static factories. Arguably, it makes sense to move factories to the types they describe. >>>>> >>>>> Species >>>>> Singleton >>>>> interface List { >>>>> private species List empty = new EmptyList<>(); >>>>> species List emptyList() { return empty; } >>>>> } >>>>> interface List { >>>>> private singleton Stuff { >>>>> List empty = new EmptyList<>(); >>>>> } >>>>> species List emptyList() { return Stuff.empty; } >>>>> } >>>>> In this model, you'd get an empty list with >>>>> >>>>> List aList = List.empty() >>>>> rather than >>>>> List aList = Collections.empty(); >>>>> >>>>> In the latter, the type witnesses can be omitted; in the former they probably can be as well but that's something new. >>>>> >>>>> >>>>> 5. Typevar shredding. Here, we have separate state for different subsets of variables. This should be the place where the singleton approach shines. >>>>> >>>>> >>>>> Species >>>>> Singleton >>>>> class HashMap { >>>>> private static class Keys { >>>>> species Set allKeys = ... >>>>> } >>>>> >>>>> private static class Vals { >>>>> species Set allVals = ... >>>>> } >>>>> >>>>> void put(K k, V v) { >>>>> Keys.allKeys.add(k); >>>>> Vals.allVals.add(v); >>>>> } >>>>> } >>>>> class HashMap { >>>>> private singleton Keys { >>>>> Set allKeys = ... >>>>> } >>>>> >>>>> private singleton Vals { >>>>> Set allVals = ... >>>>> } >>>>> >>>>> void put(K k, V v) { >>>>> Keys.allKeys.add(k); >>>>> Vals.allVals.add(v); >>>>> } >>>>> } >>>>> >>>>> But, it doesn't really shine that much; the left is not really much worse than the right, just a little more fussy. >>>>> >>>>> In cases where the singleton approach is more natural, the corresponding "species in static class" idiom isn't so bad either. But in cases where the species approach is more natural, there's something unappealing about creating classes (both in source and runtime footprint) in cases 2/3/4 when we don't need one. The only place where the singleton approach seems to win big is when there are multiple variables in the same scope bound by invariants -- here, the singleton having a ctor is a big win -- but how often does this happen? >>>>> >>>>> >>>>> So our conclusion is that the species-placement is as good or better for the identified use cases -- and it also fits cleanly into the existing model for member placement. >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From maurizio.cimadamore at oracle.com Mon May 23 14:24:22 2016 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Mon, 23 May 2016 15:24:22 +0100 Subject: Species-static members vs singletons In-Reply-To: <57431139.5050401@oracle.com> References: <4219406f-d842-97f4-7206-3a91ffe1e75c@oracle.com> <8abbeaf3-6c46-01cf-8e18-94f40299e9cb@gmail.com> <5742D5EC.70608@oracle.com> <57431139.5050401@oracle.com> Message-ID: <57431296.9090508@oracle.com> Also, can you think of cases where the first parameter will be something else other than the receiver class? I.e. do we want to encourage a more OO-like style where you can ask a Complex things like compareTo etc. (rarher than calling a static helper somewhere to do the job) ? Maurizio On 23/05/16 15:18, Maurizio Cimadamore wrote: > Sorry - I now realize that the point I made in my earlier email was > unclear. > > What I'm suggesting is to have a single rule for generating unchecked > warnings that goes like this: > > "If the qualifier of a species static access is not reifiable, an > unchecked warning should occur". > > In the example Peter sent, the only thing worth mentioning is that the > qualifier is 'implicit' (i.e. can be omitted and be assumed to be the > current class Foo); now since Foo is not reifiable, every > unqualified access to 'st' from Foo will get a warning - excluding, > of course, accesses occurring in a context where T is restricted (i.e. > __WhereVal(T)). > > Maurizio > > On 23/05/16 14:56, Brian Goetz wrote: >> Note that we have this same problem with unchecked warnings today in >> many of the use cases. For example, in the ?cached empty list? case, >> we always have to use an unchecked cast to cast the cached list to >> the desired type. When we use species-static to do the same, and it >> is possible that the species could correspond to more than one T, we >> still have to do the same unchecked warning (and as you mention, the >> singleton form has the same problem.) I think its an unescapable >> consequence of erasure, but one we?re already sort of comfortable with. >> >> If you use a more constrained type selector (e.g., List), you >> won?t get a warning, as the compiler will know that st is exactly int. >> >>> On May 23, 2016, at 3:05 AM, Maurizio Cimadamore >>> >> > wrote: >>> >>> Hi Peter, >>> are you sure we need special treatment for 'it = st' ? After all, >>> the compiler will issue unchecked warnings every time you'll try to >>> access a species static from a non-reifiable type i.e. >>> >>> Foo.st = ""; //warn >>> Foo.st = 42; //no warn >>> >>> In other words, can we put the burden of heap pollution-ness on the >>> client and be happy? >>> >>> Maurizio >>> >>> On 22/05/16 23:58, Peter Levart wrote: >>>> Hi Brian, >>>> >>>> I agree that "species" placement is a better, less verbose option. >>>> But how to solve the language problem of having "species" and >>>> "instance" members of the same "type-variable" type be assignable >>>> to one-another? For example: >>>> >>>> class Foo { >>>> species T st; >>>> T it; >>>> >>>> void m() { >>>> it = st; // this can not be allowed >>>> st = it; // this can be allowed >>>> >>>> // maybe this could be allowed? >>>> @SuppressWarnings("unchecked") >>>> it = (T) st; >>>> } >>>> >>>> >>>> Singleton abstraction has the same problem. >>>> >>>> So while technically possible, it would be weird to have 'T' >>>> sometimes not be assignable to 'T'. Can we live with that? >>>> >>>> Regards, Peter >>>> >>>> On 05/19/2016 04:36 PM, Brian Goetz wrote: >>>>> We discussed two primary means to surface species-specific members >>>>> in the language: a "species" placement (name TBD) as distinct from >>>>> static and instance, or a "singleton" abstraction (a la Scala's >>>>> "object" abstraction, as Peter L suggested). We've done some >>>>> experiments comparing the two approaches. >>>>> >>>>> Separately, we discussed two strategies for handling this at the >>>>> VM level: having three separate placements (ACC_STATIC, >>>>> ACC_SPECIES, and instance) or retconning ACC_STATIC to mean >>>>> "species" and using compiler trickery to simulate traditional >>>>> statics. In recent discussions with Oracle and IBM VM folks, they >>>>> seemed happy enough with having a new placement (and possibly new >>>>> bytecodes, {get,put,invoke}species, or overloading these onto >>>>> *static with ParamTypes in the owner field of the various XxxRef >>>>> constants.) >>>>> >>>>> >>>>> There are several places where the language itself can take >>>>> advantage of species members: >>>>> >>>>> 1. Reifying type variables. For an any-generic class Foo, >>>>> the compiler can generate public static final >>>>> reflection-thingie-valued fields called "T" and "U", which means >>>>> that "aFoo.T" (as an ordinary field ref!) would evaluate to the >>>>> reflective mirror for the reified T -- if present, otherwise it >>>>> would evaluate to the reflective mirror for 'erased'. >>>>> >>>>> 2. Representation of generic methods. The current translation >>>>> strategy has us translating any-generic methods to classes; a >>>>> static method >>>>> >>>>> static void foo(T t) { } >>>>> >>>>> translates to a class (plus an erased bridge): >>>>> >>>>> bridge static foo(Object o) { ... invoke erased specialization >>>>> ... } >>>>> >>>>> static class Xxx$foo { >>>>> void foo(T t) { ... } >>>>> } >>>>> >>>>> This means that an instance of Xxx$foo is needed to invoke the >>>>> method -- but serves solely to carry the type variables -- which >>>>> is unfortunate. If instead we translate as: >>>>> >>>>> static class Xxx$foo { >>>>> *species-static *void foo(T t) { ... } >>>>> } >>>>> >>>>> then we can invoke this method via invokespecies: >>>>> >>>>> invokespecies ParamType[Xxx$foo, T_inf].foo(T_inf) >>>>> >>>>> where T_inf is the erasure-normalized type inferred for T (reified >>>>> if value, `erased` reference.) No fake receiver required. >>>>> >>>>> The translation for generic instance methods is still somewhat >>>>> messier (will post separately), but still less messy than if we >>>>> also had to manage / cache a receiver. >>>>> >>>>> >>>>> We also drafted some examples of how such a facility would be >>>>> used, writing them both with species-static and with singleton. >>>>> Examples and notes below; the summary is that in all cases, the >>>>> species-static version is either better or about as good. >>>>> >>>>> >>>>> >>>>> 1. The old favorite, caching an instantiated instance. >>>>> >>>>> Species >>>>> Singleton >>>>> class Collections { >>>>> private static class Holder { >>>>> private species List empty = new EmptyList(); >>>>> } >>>>> >>>>> static List emptyList() { return Holder.empty; } >>>>> } >>>>> class Collections { >>>>> private singleton Holder { >>>>> private empty = new EmptyList(); >>>>> } >>>>> >>>>> static List emptyList() { return Holder.empty; } >>>>> } >>>>> >>>>> >>>>> Note that in this case, species by itself isn't enough -- we still >>>>> need a holder class, and its a bit ugly. Arguably we could merge >>>>> Holder into EmptyList (if that's under our control) but because >>>>> Collections is an old-style "static bag" class (aka "sin bin"), we >>>>> would still need a holder class for state. (Collections could >>>>> share a single holder for multiple things; empty list, empty set, >>>>> etc.) >>>>> >>>>> Neither the left nor the right seems particularly better than the >>>>> other here. (If we were putting this method on Collection, where >>>>> it would likely go in new code since now interfaces can have >>>>> statics, the species approach would win, since we'd not need the >>>>> holder class any more.) >>>>> >>>>> >>>>> 2. Instantiation tracking. >>>>> >>>>> Species >>>>> Singleton >>>>> class Foo { >>>>> private species int count; >>>>> private species List> foos; >>>>> >>>>> public Foo() { >>>>> ++count; >>>>> foos.add(this); >>>>> } >>>>> } >>>>> class Foo { >>>>> private singleton FooStuff { >>>>> private int count; >>>>> private List> foos; >>>>> } >>>>> >>>>> public Foo() { >>>>> ++Foo.count; >>>>> Foo.foos.add(this); >>>>> } >>>>> } >>>>> >>>>> >>>>> Because the state is directly tied to the instantiation, the left >>>>> seems more attractive -- doesn't require an extra artifact, and >>>>> the constructor body seems more straightforward. >>>>> >>>>> >>>>> 3. Implicit-like associations. Here, we're caching type >>>>> associations. For example, suppose we have a Box, and we want >>>>> to cache the associated class for List. >>>>> >>>>> >>>>> Species >>>>> Singleton >>>>> class Box { >>>>> private species Class> listClass >>>>> = Class.forSpecialization(List, T.crass); >>>>> } >>>>> class Box { >>>>> private singleton ListBuddy { >>>>> Class> clazz >>>>> = Class.forSpecialization(List, T.crass); >>>>> } >>>>> } >>>>> >>>>> >>>>> The extra singleton declaration feels like "noise" here, because >>>>> again the association is with the full set of type args for the >>>>> class. >>>>> >>>>> >>>>> 4. Static factories. Arguably, it makes sense to move factories >>>>> to the types they describe. >>>>> >>>>> Species >>>>> Singleton >>>>> interface List { >>>>> private species List empty = new EmptyList<>(); >>>>> species List emptyList() { return empty; } >>>>> } >>>>> interface List { >>>>> private singleton Stuff { >>>>> List empty = new EmptyList<>(); >>>>> } >>>>> species List emptyList() { return Stuff.empty; } >>>>> } >>>>> >>>>> >>>>> In this model, you'd get an empty list with >>>>> >>>>> List aList = List.empty() >>>>> rather than >>>>> List aList = Collections.empty(); >>>>> >>>>> In the latter, the type witnesses can be omitted; in the former >>>>> they probably can be as well but that's something new. >>>>> >>>>> >>>>> 5. Typevar shredding. Here, we have separate state for different >>>>> subsets of variables. This should be the place where the >>>>> singleton approach shines. >>>>> >>>>> >>>>> Species >>>>> Singleton >>>>> class HashMap { >>>>> private static class Keys { >>>>> species Set allKeys = ... >>>>> } >>>>> >>>>> private static class Vals { >>>>> species Set allVals = ... >>>>> } >>>>> >>>>> void put(K k, V v) { >>>>> Keys.allKeys.add(k); >>>>> Vals.allVals.add(v); >>>>> } >>>>> } >>>>> class HashMap { >>>>> private singleton Keys { >>>>> Set allKeys = ... >>>>> } >>>>> >>>>> private singleton Vals { >>>>> Set allVals = ... >>>>> } >>>>> >>>>> void put(K k, V v) { >>>>> Keys.allKeys.add(k); >>>>> Vals.allVals.add(v); >>>>> } >>>>> } >>>>> >>>>> >>>>> >>>>> But, it doesn't really shine that much; the left is not really >>>>> much worse than the right, just a little more fussy. >>>>> >>>>> In cases where the singleton approach is more natural, the >>>>> corresponding "species in static class" idiom isn't so bad >>>>> either. But in cases where the species approach is more natural, >>>>> there's something unappealing about creating classes (both in >>>>> source and runtime footprint) in cases 2/3/4 when we don't need >>>>> one. The only place where the singleton approach seems to win big >>>>> is when there are multiple variables in the same scope bound by >>>>> invariants -- here, the singleton having a ctor is a big win -- >>>>> but how often does this happen? >>>>> >>>>> >>>>> So our conclusion is that the species-placement is as good or >>>>> better for the identified use cases -- and it also fits cleanly >>>>> into the existing model for member placement. >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From maurizio.cimadamore at oracle.com Mon May 23 14:25:50 2016 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Mon, 23 May 2016 15:25:50 +0100 Subject: Species-static members vs singletons In-Reply-To: <57431296.9090508@oracle.com> References: <4219406f-d842-97f4-7206-3a91ffe1e75c@oracle.com> <8abbeaf3-6c46-01cf-8e18-94f40299e9cb@gmail.com> <5742D5EC.70608@oracle.com> <57431139.5050401@oracle.com> <57431296.9090508@oracle.com> Message-ID: <574312EE.2060409@oracle.com> Sorry - wrong thread :-( On 23/05/16 15:24, Maurizio Cimadamore wrote: > Also, can you think of cases where the first parameter will be > something else other than the receiver class? I.e. do we want to > encourage a more OO-like style where you can ask a Complex things like > compareTo etc. (rarher than calling a static helper somewhere to do > the job) ? > > Maurizio > > On 23/05/16 15:18, Maurizio Cimadamore wrote: >> Sorry - I now realize that the point I made in my earlier email was >> unclear. >> >> What I'm suggesting is to have a single rule for generating unchecked >> warnings that goes like this: >> >> "If the qualifier of a species static access is not reifiable, an >> unchecked warning should occur". >> >> In the example Peter sent, the only thing worth mentioning is that >> the qualifier is 'implicit' (i.e. can be omitted and be assumed to be >> the current class Foo); now since Foo is not reifiable, every >> unqualified access to 'st' from Foo will get a warning - >> excluding, of course, accesses occurring in a context where T is >> restricted (i.e. __WhereVal(T)). >> >> Maurizio >> >> On 23/05/16 14:56, Brian Goetz wrote: >>> Note that we have this same problem with unchecked warnings today in >>> many of the use cases. For example, in the ?cached empty list? >>> case, we always have to use an unchecked cast to cast the cached >>> list to the desired type. When we use species-static to do the >>> same, and it is possible that the species could correspond to more >>> than one T, we still have to do the same unchecked warning (and as >>> you mention, the singleton form has the same problem.) I think its >>> an unescapable consequence of erasure, but one we?re already sort of >>> comfortable with. >>> >>> If you use a more constrained type selector (e.g., List), you >>> won?t get a warning, as the compiler will know that st is exactly int. >>> >>>> On May 23, 2016, at 3:05 AM, Maurizio Cimadamore >>>> >>> > wrote: >>>> >>>> Hi Peter, >>>> are you sure we need special treatment for 'it = st' ? After all, >>>> the compiler will issue unchecked warnings every time you'll try to >>>> access a species static from a non-reifiable type i.e. >>>> >>>> Foo.st = ""; //warn >>>> Foo.st = 42; //no warn >>>> >>>> In other words, can we put the burden of heap pollution-ness on the >>>> client and be happy? >>>> >>>> Maurizio >>>> >>>> On 22/05/16 23:58, Peter Levart wrote: >>>>> Hi Brian, >>>>> >>>>> I agree that "species" placement is a better, less verbose option. >>>>> But how to solve the language problem of having "species" and >>>>> "instance" members of the same "type-variable" type be assignable >>>>> to one-another? For example: >>>>> >>>>> class Foo { >>>>> species T st; >>>>> T it; >>>>> >>>>> void m() { >>>>> it = st; // this can not be allowed >>>>> st = it; // this can be allowed >>>>> >>>>> // maybe this could be allowed? >>>>> @SuppressWarnings("unchecked") >>>>> it = (T) st; >>>>> } >>>>> >>>>> >>>>> Singleton abstraction has the same problem. >>>>> >>>>> So while technically possible, it would be weird to have 'T' >>>>> sometimes not be assignable to 'T'. Can we live with that? >>>>> >>>>> Regards, Peter >>>>> >>>>> On 05/19/2016 04:36 PM, Brian Goetz wrote: >>>>>> We discussed two primary means to surface species-specific >>>>>> members in the language: a "species" placement (name TBD) as >>>>>> distinct from static and instance, or a "singleton" abstraction >>>>>> (a la Scala's "object" abstraction, as Peter L suggested). We've >>>>>> done some experiments comparing the two approaches. >>>>>> >>>>>> Separately, we discussed two strategies for handling this at the >>>>>> VM level: having three separate placements (ACC_STATIC, >>>>>> ACC_SPECIES, and instance) or retconning ACC_STATIC to mean >>>>>> "species" and using compiler trickery to simulate traditional >>>>>> statics. In recent discussions with Oracle and IBM VM folks, >>>>>> they seemed happy enough with having a new placement (and >>>>>> possibly new bytecodes, {get,put,invoke}species, or overloading >>>>>> these onto *static with ParamTypes in the owner field of the >>>>>> various XxxRef constants.) >>>>>> >>>>>> >>>>>> There are several places where the language itself can take >>>>>> advantage of species members: >>>>>> >>>>>> 1. Reifying type variables. For an any-generic class Foo, >>>>>> the compiler can generate public static final >>>>>> reflection-thingie-valued fields called "T" and "U", which means >>>>>> that "aFoo.T" (as an ordinary field ref!) would evaluate to the >>>>>> reflective mirror for the reified T -- if present, otherwise it >>>>>> would evaluate to the reflective mirror for 'erased'. >>>>>> >>>>>> 2. Representation of generic methods. The current translation >>>>>> strategy has us translating any-generic methods to classes; a >>>>>> static method >>>>>> >>>>>> static void foo(T t) { } >>>>>> >>>>>> translates to a class (plus an erased bridge): >>>>>> >>>>>> bridge static foo(Object o) { ... invoke erased >>>>>> specialization ... } >>>>>> >>>>>> static class Xxx$foo { >>>>>> void foo(T t) { ... } >>>>>> } >>>>>> >>>>>> This means that an instance of Xxx$foo is needed to invoke the >>>>>> method -- but serves solely to carry the type variables -- which >>>>>> is unfortunate. If instead we translate as: >>>>>> >>>>>> static class Xxx$foo { >>>>>> *species-static *void foo(T t) { ... } >>>>>> } >>>>>> >>>>>> then we can invoke this method via invokespecies: >>>>>> >>>>>> invokespecies ParamType[Xxx$foo, T_inf].foo(T_inf) >>>>>> >>>>>> where T_inf is the erasure-normalized type inferred for T >>>>>> (reified if value, `erased` reference.) No fake receiver required. >>>>>> >>>>>> The translation for generic instance methods is still somewhat >>>>>> messier (will post separately), but still less messy than if we >>>>>> also had to manage / cache a receiver. >>>>>> >>>>>> >>>>>> We also drafted some examples of how such a facility would be >>>>>> used, writing them both with species-static and with singleton. >>>>>> Examples and notes below; the summary is that in all cases, the >>>>>> species-static version is either better or about as good. >>>>>> >>>>>> >>>>>> >>>>>> 1. The old favorite, caching an instantiated instance. >>>>>> >>>>>> Species >>>>>> Singleton >>>>>> class Collections { >>>>>> private static class Holder { >>>>>> private species List empty = new EmptyList(); >>>>>> } >>>>>> >>>>>> static List emptyList() { return Holder.empty; } >>>>>> } >>>>>> class Collections { >>>>>> private singleton Holder { >>>>>> private empty = new EmptyList(); >>>>>> } >>>>>> >>>>>> static List emptyList() { return Holder.empty; } >>>>>> } >>>>>> >>>>>> >>>>>> Note that in this case, species by itself isn't enough -- we >>>>>> still need a holder class, and its a bit ugly. Arguably we could >>>>>> merge Holder into EmptyList (if that's under our control) but >>>>>> because Collections is an old-style "static bag" class (aka "sin >>>>>> bin"), we would still need a holder class for state. >>>>>> (Collections could share a single holder for multiple things; >>>>>> empty list, empty set, etc.) >>>>>> >>>>>> Neither the left nor the right seems particularly better than the >>>>>> other here. (If we were putting this method on Collection, where >>>>>> it would likely go in new code since now interfaces can have >>>>>> statics, the species approach would win, since we'd not need the >>>>>> holder class any more.) >>>>>> >>>>>> >>>>>> 2. Instantiation tracking. >>>>>> >>>>>> Species >>>>>> Singleton >>>>>> class Foo { >>>>>> private species int count; >>>>>> private species List> foos; >>>>>> >>>>>> public Foo() { >>>>>> ++count; >>>>>> foos.add(this); >>>>>> } >>>>>> } >>>>>> class Foo { >>>>>> private singleton FooStuff { >>>>>> private int count; >>>>>> private List> foos; >>>>>> } >>>>>> >>>>>> public Foo() { >>>>>> ++Foo.count; >>>>>> Foo.foos.add(this); >>>>>> } >>>>>> } >>>>>> >>>>>> >>>>>> Because the state is directly tied to the instantiation, the left >>>>>> seems more attractive -- doesn't require an extra artifact, and >>>>>> the constructor body seems more straightforward. >>>>>> >>>>>> >>>>>> 3. Implicit-like associations. Here, we're caching type >>>>>> associations. For example, suppose we have a Box, and we want >>>>>> to cache the associated class for List. >>>>>> >>>>>> >>>>>> Species >>>>>> Singleton >>>>>> class Box { >>>>>> private species Class> listClass >>>>>> = Class.forSpecialization(List, T.crass); >>>>>> } >>>>>> class Box { >>>>>> private singleton ListBuddy { >>>>>> Class> clazz >>>>>> = Class.forSpecialization(List, T.crass); >>>>>> } >>>>>> } >>>>>> >>>>>> >>>>>> The extra singleton declaration feels like "noise" here, because >>>>>> again the association is with the full set of type args for the >>>>>> class. >>>>>> >>>>>> >>>>>> 4. Static factories. Arguably, it makes sense to move factories >>>>>> to the types they describe. >>>>>> >>>>>> Species >>>>>> Singleton >>>>>> interface List { >>>>>> private species List empty = new EmptyList<>(); >>>>>> species List emptyList() { return empty; } >>>>>> } >>>>>> interface List { >>>>>> private singleton Stuff { >>>>>> List empty = new EmptyList<>(); >>>>>> } >>>>>> species List emptyList() { return Stuff.empty; } >>>>>> } >>>>>> >>>>>> >>>>>> In this model, you'd get an empty list with >>>>>> >>>>>> List aList = List.empty() >>>>>> rather than >>>>>> List aList = Collections.empty(); >>>>>> >>>>>> In the latter, the type witnesses can be omitted; in the former >>>>>> they probably can be as well but that's something new. >>>>>> >>>>>> >>>>>> 5. Typevar shredding. Here, we have separate state for >>>>>> different subsets of variables. This should be the place where >>>>>> the singleton approach shines. >>>>>> >>>>>> >>>>>> Species >>>>>> Singleton >>>>>> class HashMap { >>>>>> private static class Keys { >>>>>> species Set allKeys = ... >>>>>> } >>>>>> >>>>>> private static class Vals { >>>>>> species Set allVals = ... >>>>>> } >>>>>> >>>>>> void put(K k, V v) { >>>>>> Keys.allKeys.add(k); >>>>>> Vals.allVals.add(v); >>>>>> } >>>>>> } >>>>>> class HashMap { >>>>>> private singleton Keys { >>>>>> Set allKeys = ... >>>>>> } >>>>>> >>>>>> private singleton Vals { >>>>>> Set allVals = ... >>>>>> } >>>>>> >>>>>> void put(K k, V v) { >>>>>> Keys.allKeys.add(k); >>>>>> Vals.allVals.add(v); >>>>>> } >>>>>> } >>>>>> >>>>>> >>>>>> >>>>>> But, it doesn't really shine that much; the left is not really >>>>>> much worse than the right, just a little more fussy. >>>>>> >>>>>> In cases where the singleton approach is more natural, the >>>>>> corresponding "species in static class" idiom isn't so bad >>>>>> either. But in cases where the species approach is more natural, >>>>>> there's something unappealing about creating classes (both in >>>>>> source and runtime footprint) in cases 2/3/4 when we don't need >>>>>> one. The only place where the singleton approach seems to win big >>>>>> is when there are multiple variables in the same scope bound by >>>>>> invariants -- here, the singleton having a ctor is a big win -- >>>>>> but how often does this happen? >>>>>> >>>>>> >>>>>> So our conclusion is that the species-placement is as good or >>>>>> better for the identified use cases -- and it also fits cleanly >>>>>> into the existing model for member placement. >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From maurizio.cimadamore at oracle.com Mon May 23 14:27:49 2016 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Mon, 23 May 2016 15:27:49 +0100 Subject: Species-static members vs singletons In-Reply-To: <40FBB763-E034-4D4B-8436-ADBC2B3E3C4B@oracle.com> References: <4219406f-d842-97f4-7206-3a91ffe1e75c@oracle.com> <8abbeaf3-6c46-01cf-8e18-94f40299e9cb@gmail.com> <5742D5EC.70608@oracle.com> <57431139.5050401@oracle.com> <40FBB763-E034-4D4B-8436-ADBC2B3E3C4B@oracle.com> Message-ID: <57431365.8060208@oracle.com> On 23/05/16 15:20, Brian Goetz wrote: > Right. And Peter?s question is: (a) did we think of this (yes) and > (b) are we OK with this. Which I think is also yes? I think it's yes; an unfortunate accident of erasure - I don't see any other way around it at the moment. Maurizio > >> On May 23, 2016, at 7:18 AM, Maurizio Cimadamore >> > > wrote: >> >> Sorry - I now realize that the point I made in my earlier email was >> unclear. >> >> What I'm suggesting is to have a single rule for generating unchecked >> warnings that goes like this: >> >> "If the qualifier of a species static access is not reifiable, an >> unchecked warning should occur". >> >> In the example Peter sent, the only thing worth mentioning is that >> the qualifier is 'implicit' (i.e. can be omitted and be assumed to be >> the current class Foo); now since Foo is not reifiable, every >> unqualified access to 'st' from Foo will get a warning - >> excluding, of course, accesses occurring in a context where T is >> restricted (i.e. __WhereVal(T)). >> >> Maurizio >> >> On 23/05/16 14:56, Brian Goetz wrote: >>> Note that we have this same problem with unchecked warnings today in >>> many of the use cases. For example, in the ?cached empty list? >>> case, we always have to use an unchecked cast to cast the cached >>> list to the desired type. When we use species-static to do the >>> same, and it is possible that the species could correspond to more >>> than one T, we still have to do the same unchecked warning (and as >>> you mention, the singleton form has the same problem.) I think its >>> an unescapable consequence of erasure, but one we?re already sort of >>> comfortable with. >>> >>> If you use a more constrained type selector (e.g., List), you >>> won?t get a warning, as the compiler will know that st is exactly int. >>> >>>> On May 23, 2016, at 3:05 AM, Maurizio Cimadamore >>>> >>> > wrote: >>>> >>>> Hi Peter, >>>> are you sure we need special treatment for 'it = st' ? After all, >>>> the compiler will issue unchecked warnings every time you'll try to >>>> access a species static from a non-reifiable type i.e. >>>> >>>> Foo.st = ""; //warn >>>> Foo.st = 42; //no warn >>>> >>>> In other words, can we put the burden of heap pollution-ness on the >>>> client and be happy? >>>> >>>> Maurizio >>>> >>>> On 22/05/16 23:58, Peter Levart wrote: >>>>> Hi Brian, >>>>> >>>>> I agree that "species" placement is a better, less verbose option. >>>>> But how to solve the language problem of having "species" and >>>>> "instance" members of the same "type-variable" type be assignable >>>>> to one-another? For example: >>>>> >>>>> class Foo { >>>>> species T st; >>>>> T it; >>>>> >>>>> void m() { >>>>> it = st; // this can not be allowed >>>>> st = it; // this can be allowed >>>>> >>>>> // maybe this could be allowed? >>>>> @SuppressWarnings("unchecked") >>>>> it = (T) st; >>>>> } >>>>> >>>>> >>>>> Singleton abstraction has the same problem. >>>>> >>>>> So while technically possible, it would be weird to have 'T' >>>>> sometimes not be assignable to 'T'. Can we live with that? >>>>> >>>>> Regards, Peter >>>>> >>>>> On 05/19/2016 04:36 PM, Brian Goetz wrote: >>>>>> We discussed two primary means to surface species-specific >>>>>> members in the language: a "species" placement (name TBD) as >>>>>> distinct from static and instance, or a "singleton" abstraction >>>>>> (a la Scala's "object" abstraction, as Peter L suggested). We've >>>>>> done some experiments comparing the two approaches. >>>>>> >>>>>> Separately, we discussed two strategies for handling this at the >>>>>> VM level: having three separate placements (ACC_STATIC, >>>>>> ACC_SPECIES, and instance) or retconning ACC_STATIC to mean >>>>>> "species" and using compiler trickery to simulate traditional >>>>>> statics. In recent discussions with Oracle and IBM VM folks, >>>>>> they seemed happy enough with having a new placement (and >>>>>> possibly new bytecodes, {get,put,invoke}species, or overloading >>>>>> these onto *static with ParamTypes in the owner field of the >>>>>> various XxxRef constants.) >>>>>> >>>>>> >>>>>> There are several places where the language itself can take >>>>>> advantage of species members: >>>>>> >>>>>> 1. Reifying type variables. For an any-generic class Foo, >>>>>> the compiler can generate public static final >>>>>> reflection-thingie-valued fields called "T" and "U", which means >>>>>> that "aFoo.T" (as an ordinary field ref!) would evaluate to the >>>>>> reflective mirror for the reified T -- if present, otherwise it >>>>>> would evaluate to the reflective mirror for 'erased'. >>>>>> >>>>>> 2. Representation of generic methods. The current translation >>>>>> strategy has us translating any-generic methods to classes; a >>>>>> static method >>>>>> >>>>>> static void foo(T t) { } >>>>>> >>>>>> translates to a class (plus an erased bridge): >>>>>> >>>>>> bridge static foo(Object o) { ... invoke erased >>>>>> specialization ... } >>>>>> >>>>>> static class Xxx$foo { >>>>>> void foo(T t) { ... } >>>>>> } >>>>>> >>>>>> This means that an instance of Xxx$foo is needed to invoke the >>>>>> method -- but serves solely to carry the type variables -- which >>>>>> is unfortunate. If instead we translate as: >>>>>> >>>>>> static class Xxx$foo { >>>>>> *species-static *void foo(T t) { ... } >>>>>> } >>>>>> >>>>>> then we can invoke this method via invokespecies: >>>>>> >>>>>> invokespecies ParamType[Xxx$foo, T_inf].foo(T_inf) >>>>>> >>>>>> where T_inf is the erasure-normalized type inferred for T >>>>>> (reified if value, `erased` reference.) No fake receiver required. >>>>>> >>>>>> The translation for generic instance methods is still somewhat >>>>>> messier (will post separately), but still less messy than if we >>>>>> also had to manage / cache a receiver. >>>>>> >>>>>> >>>>>> We also drafted some examples of how such a facility would be >>>>>> used, writing them both with species-static and with singleton. >>>>>> Examples and notes below; the summary is that in all cases, the >>>>>> species-static version is either better or about as good. >>>>>> >>>>>> >>>>>> >>>>>> 1. The old favorite, caching an instantiated instance. >>>>>> >>>>>> Species >>>>>> Singleton >>>>>> class Collections { >>>>>> private static class Holder { >>>>>> private species List empty = new EmptyList(); >>>>>> } >>>>>> >>>>>> static List emptyList() { return Holder.empty; } >>>>>> } >>>>>> class Collections { >>>>>> private singleton Holder { >>>>>> private empty = new EmptyList(); >>>>>> } >>>>>> >>>>>> static List emptyList() { return Holder.empty; } >>>>>> } >>>>>> >>>>>> >>>>>> Note that in this case, species by itself isn't enough -- we >>>>>> still need a holder class, and its a bit ugly. Arguably we could >>>>>> merge Holder into EmptyList (if that's under our control) but >>>>>> because Collections is an old-style "static bag" class (aka "sin >>>>>> bin"), we would still need a holder class for state. (Collections >>>>>> could share a single holder for multiple things; empty list, >>>>>> empty set, etc.) >>>>>> >>>>>> Neither the left nor the right seems particularly better than the >>>>>> other here. (If we were putting this method on Collection, where >>>>>> it would likely go in new code since now interfaces can have >>>>>> statics, the species approach would win, since we'd not need the >>>>>> holder class any more.) >>>>>> >>>>>> >>>>>> 2. Instantiation tracking. >>>>>> >>>>>> Species >>>>>> Singleton >>>>>> class Foo { >>>>>> private species int count; >>>>>> private species List> foos; >>>>>> >>>>>> public Foo() { >>>>>> ++count; >>>>>> foos.add(this); >>>>>> } >>>>>> } >>>>>> class Foo { >>>>>> private singleton FooStuff { >>>>>> private int count; >>>>>> private List> foos; >>>>>> } >>>>>> >>>>>> public Foo() { >>>>>> ++Foo.count; >>>>>> Foo.foos.add(this); >>>>>> } >>>>>> } >>>>>> >>>>>> >>>>>> Because the state is directly tied to the instantiation, the left >>>>>> seems more attractive -- doesn't require an extra artifact, and >>>>>> the constructor body seems more straightforward. >>>>>> >>>>>> >>>>>> 3. Implicit-like associations. Here, we're caching type >>>>>> associations. For example, suppose we have a Box, and we want >>>>>> to cache the associated class for List. >>>>>> >>>>>> >>>>>> Species >>>>>> Singleton >>>>>> class Box { >>>>>> private species Class> listClass >>>>>> = Class.forSpecialization(List, T.crass); >>>>>> } >>>>>> class Box { >>>>>> private singleton ListBuddy { >>>>>> Class> clazz >>>>>> = Class.forSpecialization(List, T.crass); >>>>>> } >>>>>> } >>>>>> >>>>>> >>>>>> The extra singleton declaration feels like "noise" here, because >>>>>> again the association is with the full set of type args for the >>>>>> class. >>>>>> >>>>>> >>>>>> 4. Static factories. Arguably, it makes sense to move factories >>>>>> to the types they describe. >>>>>> >>>>>> Species >>>>>> Singleton >>>>>> interface List { >>>>>> private species List empty = new EmptyList<>(); >>>>>> species List emptyList() { return empty; } >>>>>> } >>>>>> interface List { >>>>>> private singleton Stuff { >>>>>> List empty = new EmptyList<>(); >>>>>> } >>>>>> species List emptyList() { return Stuff.empty; } >>>>>> } >>>>>> >>>>>> >>>>>> In this model, you'd get an empty list with >>>>>> >>>>>> List aList = List.empty() >>>>>> rather than >>>>>> List aList = Collections.empty(); >>>>>> >>>>>> In the latter, the type witnesses can be omitted; in the former >>>>>> they probably can be as well but that's something new. >>>>>> >>>>>> >>>>>> 5. Typevar shredding. Here, we have separate state for >>>>>> different subsets of variables. This should be the place where >>>>>> the singleton approach shines. >>>>>> >>>>>> >>>>>> Species >>>>>> Singleton >>>>>> class HashMap { >>>>>> private static class Keys { >>>>>> species Set allKeys = ... >>>>>> } >>>>>> >>>>>> private static class Vals { >>>>>> species Set allVals = ... >>>>>> } >>>>>> >>>>>> void put(K k, V v) { >>>>>> Keys.allKeys.add(k); >>>>>> Vals.allVals.add(v); >>>>>> } >>>>>> } >>>>>> class HashMap { >>>>>> private singleton Keys { >>>>>> Set allKeys = ... >>>>>> } >>>>>> >>>>>> private singleton Vals { >>>>>> Set allVals = ... >>>>>> } >>>>>> >>>>>> void put(K k, V v) { >>>>>> Keys.allKeys.add(k); >>>>>> Vals.allVals.add(v); >>>>>> } >>>>>> } >>>>>> >>>>>> >>>>>> >>>>>> But, it doesn't really shine that much; the left is not really >>>>>> much worse than the right, just a little more fussy. >>>>>> >>>>>> In cases where the singleton approach is more natural, the >>>>>> corresponding "species in static class" idiom isn't so bad >>>>>> either. But in cases where the species approach is more natural, >>>>>> there's something unappealing about creating classes (both in >>>>>> source and runtime footprint) in cases 2/3/4 when we don't need >>>>>> one. The only place where the singleton approach seems to win big >>>>>> is when there are multiple variables in the same scope bound by >>>>>> invariants -- here, the singleton having a ctor is a big win -- >>>>>> but how often does this happen? >>>>>> >>>>>> >>>>>> So our conclusion is that the species-placement is as good or >>>>>> better for the identified use cases -- and it also fits cleanly >>>>>> into the existing model for member placement. >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bjornvar at ca.ibm.com Thu May 26 13:35:24 2016 From: bjornvar at ca.ibm.com (Bjorn B Vardal) Date: Thu, 26 May 2016 13:35:24 +0000 Subject: Wildcards -- Models 4 and 5 In-Reply-To: <2421796e-cb8e-6241-9f80-5bb761673f2e@oracle.com> References: <2421796e-cb8e-6241-9f80-5bb761673f2e@oracle.com> Message-ID: <20160526133532.61E2378057@b03ledav004.gho.boulder.ibm.com> An HTML attachment was scrubbed... URL: From bjornvar at ca.ibm.com Fri May 27 18:44:33 2016 From: bjornvar at ca.ibm.com (Bjorn B Vardal) Date: Fri, 27 May 2016 18:44:33 +0000 Subject: Specialization statics - Summary Message-ID: <20160527184438.C0959AE03C@b01ledav005.gho.pok.ibm.com> An HTML attachment was scrubbed... URL: From maurizio.cimadamore at oracle.com Fri May 27 20:56:05 2016 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Fri, 27 May 2016 21:56:05 +0100 Subject: species static prototype Message-ID: <5748B465.3060308@oracle.com> Hi, over the last few days I've been busy putting together a prototype [1, 2] of javac/runtime support for species static. I guess it could be considered an prototype implementation of the approach that Bjorn has described as "Repurpose existing statics" [4] in his nice writeup. Here's what I have learned during the experience. Parser ==== The prototype uses a no fuss approach where '__species' is the modifier to denote species static stuff (of course a better syntax will have to be picked at some point, but that's not the goal of the current exercise). This means you can write: class Foo { String i; //instance field static String s; //static field __species String ss; //species static field } This is obviously good enough for the time being. A complication with parsing occurs when accessing species members; in fact, species members can be accessed via their fully qualified type (including all required type-arguments, if necessary). Foo.ss; Foo.ss; The above are all valid species access expression. Now, adding this kind of support in the parser is always tricky - as we have to battle with ambiguities which might pop up. Luckily, this pattern is similar enough to the one we use for method references - i.e. : Foo::ss Which the compiler already had to special case; so I ended up slightly generalizing what we did in JDK 8 method reference parsing, and I got something working reasonably quick. But this could be an area where coming up with a clean spec might be tricky (as the impl uses abundant lookahead to disambiguate this one). Resolution ====== The basic idea is to divide the world in three static levels, whose properties are summarized in the table below: enclosing type enclosing instance instance yes yes species yes no static no no So, in terms of who can access what, it follows that if we consider 'instance' to be the highest static level and 'static' to be the lowest, then it's ok for a member with static level S1 to access another member of static level S2 provided that S1 >= S2. Or, with a table: from/to instance species static instance yes yes yes species no yes yes static no no yes So, let's look at a concrete example: class TestResolution { static void m_S() { m_S(); //ok m_SS(); //error m_I(); //error } __species void m_SS() { m_S(); //ok m_SS(); //ok m_I(); //error } __species void m_I() { m_S(); //ok m_SS(); //ok m_I(); //ok } } A crucial property, of course, is that species static members can reference to any type vars in the enclosing context: class TestTypeVar { static void m_S() { X x; //error } __species void m_SS() { X x; //ok } __species void m_I() { X x; //ok } } Nesting ===== Another concept that needs generalization is that of allowed nesting; consider the following program: class TestNesting1 { class MemberInner { static String s_S; //error String s_I; //ok } static class StaticInner { static String s_S; //ok String s_I; //ok } } That is, the compiler will only allow you to declare static members in toplevel classes or in static nested classes (which, after all, act as toplevel classes). Now that we are adding a new static level to the picture, how are the nesting rules affected? Looking at the table above, if we consider 'instance' to be the highest static level and 'static' to be the lowest, then it's ok for a member with static level S1 to declare a member of static level S2 provided that S1 <= S2. Again, we can look at this in a tabular fashion: declaring/declared instance species static instance yes no no species yes yes no static yes yes yes This also seems like a nice generalization of the current rules. The rationale behind these rules is to basically, guarantee some invariants during member lookup; let's say that we are in a nested class with static level S1 - then, by the rule above, it follows that any member nested in this class will be able to access another member with static level S1 declared in this class or in any lexically enclosing class. A full example of nesting rules is given below: class TestNesting2 { class MemberInner { static String s_S; //error __species String s_SS; //error String s_I; //ok } __species class StaticInner { static String s_S; //error __species String s_SS; //ok String s_I; //ok } static class StaticInner { static String s_S; //ok __species String s_SS; //ok String s_I; //ok } } Unchecked access =========== Because of an unfortunate interplay between species and erasure, code using species members is potentially unsound (the example below is a variation of an example first discovered by Peter's example [3] in this very mailing list): public class Foo { __species T cache; } Foo.cache = "Hello"; Integer i = Foo.cache; //whoops To prevent cases like these, the compiler implements a check which looks at the qualifier of a species access; if such qualifier (either explicit, or implicit) cannot be proven to be reifiable, an unchecked warning is issued. Note that it is possible to restrict such warnings only to cases where the signature of the accessed species static member changes under erasure. E.g. in the above example, accessing 'cache' is unchecked, because the type of 'cache' contains type-variables; but if another species static field was accessed whose type did not depend on type-variables, then the access should be considered sound. Species initializers =========== In our model we have three static levels - but we have initialization artifacts for only two of those; we need to fix that: instance species static That is, a new method is added to a class containing one or more species variables with an initializer. This method is used to hoist the initialization code for all the species variables. Forward references ============ Rules for detecting forward references have to be extended accordingly. A forward reference occurs whenever there's an attempt to reference a variable from a position P, where the variable declaration occurs in a position P' > P. Currently, the rules for forward references allow an instance variable to forward-reference a static variable - as shown below: class TestForwardRef { String s = s_S; static String s_S = "Hello!"; } The rationale behind this is that, by the time we see the instance initializer for 's' we would have already executed the code for initializing 's_S' (as initialization will occur in different methods, and respectively, see section above). With the new static level, the forward reference rules have to be redefined according to the table below: from/to instance species static instance forward ref ok ok species illegal forward ref ok static illegal illegal forward ref In other words, it's ok to forward reference a variable whose static level is lower than that available where the reference occurs. An example is given below: class TestForwardRef2 { String s1_I = s_S; //ok String s2_I = s_SS; //ok String s1_S = s_S; //error! String s1_SS = s_S; //ok String s2_SS = s_SS; //error! static String s_S = "Hello!"; __species String s_SS = "Hello Species!"; } This is an extension of the above principle: since instance variables are initialized in , they can reference variables initialized in or . If a variable is initialized in it can similarly safely reference a variable initialized in . Another way to think of this is that a forward reference error only occurs if the static level of the referenced symbol is the same as the static level where the reference occurs. All other cases are either illegal (i.e. because it's an attempt to go from a lower static level to an higher one) or valid (because it can be guaranteed that the code initializing the referenced variable has already been executed). Code generation ========== Javac currently emits invokestatic/getstatic/putstatic for both legacy static and species static access. javac will use the 'owner' field of a CONSTANT_MethodRef, CONSTANT_FieldRef constants to point to the sharp type of the species access (through a constant pool type entry). Static access will always see an erased owner. Consider this example: class TestGen { __species void m_SS() { } static void m_S() { } public static void main(String args) { TestGen.m_SS(); TestGen.m_SS(); TestGen.m_S(); TestGen.m_S(); } } The generated code in the 'main' method is reported below: 0: invokestatic #11 // Method TestGen<_>.m_SS:()V 3: invokestatic #15 // Method TestGen.m_SS:()V 6: invokestatic #18 // Method TestGen<_>.m_S:()V 9: invokestatic #18 // Method TestGen<_>.m_S:()V As it can be seen, species static access can cause a sharper type to end up in the 'owner' field of the member reference info; on the other hand, a static access always lead to an erased 'owner'. Another detail worth mentioning is how __species is represented in the bytecode. Given the current lack of flags bit I've opted to use the last remaining bit 0x8000 - this is in fact the last unused bit that can be shared across class, field and method descriptors. Actually, this bit has already been used to encode the ACC_MANDATED flag in the MethodParameters attribute (as of JDK 8) - but since there's no other usage of that flag configuration outside MethodParameters it would seem safe to recycle it. Of course more compact approaches are also possible, but they would lead to different flag configurations for species static fields, methods and classes. Specialization ========= Specializing species access is relatively straightforward: * both instance and species static members are copied in the specialization * static members are only copied in the erased specialization (and skipped otherwise) * ACC_SPECIES classes become regular classes when specialized * ACC_SPECIES methods/fields become static methods/fields in the specialization * becomes the new in the specialization (and is omitted if the specialization is the erased specialization) The last bullet requires some extra care when handling the 'erased' specialization; consider the following example: class TestSpec { static String s_S = "HelloStatic"; __species String s_SS = "HelloSpecies"; } This class will end up with the following two synthetic methods: static void (); descriptor: ()V flags: ACC_STATIC Code: stack=1, locals=0, args_size=0 0: ldc #8 // String HelloStatic 2: putstatic #14 // Field s_S:Ljava/lang/String; 5: ldc #16 // String HelloSpecies 7: putstatic #19 // Field s_SS:Ljava/lang/String; 10: return species void (); descriptor: ()V flags: ACC_SPECIES Code: stack=1, locals=1, args_size=1 0: ldc #16 // String HelloSpecies 2: putstatic #19 // Field s_SS:Ljava/lang/String; 5: return As it can be seen, the method contains initialization code for both static and species static fields! To understand why this is so, let's consider how the specialized bits might be derived from the template class following the rules above. Let's consider a specialization like TestSpec: in this case, we need to drop (it's a static method and TestSpec is not an erased specialization), and we also need to rename as in the new specialization. All is fine - the specialization will contain the relevant code required to initialize its species static fields. Let's now turn to the erased specialization TestSpec<_> - this specialization receives both static and species static members. Now, if we were to follow the same rules for initializers, we'd end up with two different initializer methods - both and . We could ask the specializer to merge them somehow, but that would be tricky and expensive. Instead, we simply (i) drop from the erased specialization and (ii) retain . Of course this means that must also contain initialization code for species static members. Bonus point: Generic methods =================== As pointed out by Brian, if we have species static classes we can translate static and species static specializable generic methods quite effectively. Consider this example: class TestGenMethods { static void m(X x) { ... } void test() { m(42); } } without species static, this would translate to: class TestGenMethods { static class TestGenMethods$m { void m(X z) { ... } } /* bridge */ void m(Object o) { new TestGenMethods$m().m(o); } void test() { new TestGenMethod$m().m(42); // this is really done inside the BSM } } Note how the bridge (called by legacy code) will need to spin a new instance of the synthetic class and then call a method on it. The bootstrap used to dispatch static generic specializable calls also needs to do a very similar operation. But what if we turned the translated generic method into a species static method? class TestGenMethods { class TestGenMethods$m { __species void m(X z) { ... } } /* bridge */ void m(Object o) { TestGenMethods$m.m(o); } void test() { TestGenMethod$m.m(42); // this is really done inside the BSM } } With species static, we can now access the method w/o needing any extra instance. This leads to simplification in both the bridging strategy and the bootstrap implementation. We can apply a similar simplification for dispatch of specializable species static calls - the only difference is that the synthetic holder class has also to be marked as species static (since it could access type-vars from the enclosing context). Bonus point: Access bridges ================= Access bridges are a constant pain in the current translation strategy; such bridges are generated by the compiler to grant access to otherwise inaccessible members. Example: class Outer { private void m() { } class Inner { void test() { m(); } } } This code will be translated as follows: class Outer { /* synthetic */ static access$m(Outer o) { o.m(); } private void m() { } class Inner { /*synthetic*/ Outer this$0; void test() { access$m(this$0); } } } That is, access to private members is translated with an access to an accessor bridge, which then performs access from the right location. Note that the accessor bridge is static (because otherwise it would be possible to maliciously override it to grant access to otherwise inaccessible members); since it's static, usual rules apply, so it cannot refer to type-variables, it cannot be specialized, etc. This means that there are cases with specialization where existing access bridge are not enough to guarantee access - if the access happens to cross specialization boundaries (i.e. accessing m() from an Outer.Inner). Again, species static comes to the rescue: class Outer { /* synthetic */ __species access$m(Outer o) { o.m(); } private void m() { } class Inner { /*synthetic*/ Outer this$0; void test() { Outer.access$m(this$0); } } } Since the accessor bridge is now species static, it means it can now mention type variables (such as X); and it also means that when the bridge is accessed (from Inner), the qualifier type (Outer) is guaranteed to remain sharp from the source code to the bytecode - which means that when this code will get specialized, all references to X will be dealt with accordingly (and the right accessor bridge will be accessed). Parting thoughts ========== On many levels, species statics seem to be the missing ingredient for implementing many of the tricks of our translation strategy, as well as to make it easier to express common idioms (i.e. type-dependent caches) in user code. Adding support for species static has proven to be harder than originally thought. This is mainly because the current world is split in two static levels: static and instance. When something is not static it's implicitly assumed to be instance, and viceversa. If we add a third static level to the picture, a lot of the existing code just doesn't work anymore, or has to be validated to check as to whether 'static' means 'legacy static' or 'species static' (or both). I started the implementation by treating static, species static and instance as completely separate static levels - with different internal flags, etc. but I soon realized that, while clean, this approach was invalidating too much of the existing implementation. More specifically, all the code snippets checking for static would now have been updated to check for static OR species static (overriding vs. hiding, access to 'this', access to 'super', generic bridges, ...). On the other hand, the places where the semantics of species static vs. static was different were quite limited: * membership/type substitution: a species static behaves like an instance member; the type variables of the owner are replaced into the member signature. * resolution: we need to implement the correct access rules as shown in the tables above. * code generation: an invokestatic involving a species static gets a sharp qualifier type This quickly led to the realization that it was instead easier to just treat 'species static' as a special case of 'static' - and then to add finer grained logic whenever we really needed the distinction. This led to a considerably easier patch, and I think that a similar consideration will hold for the JLS. [1] - http://hg.openjdk.java.net/valhalla/valhalla/langtools/rev/6949c3d06e8f [2] - http://hg.openjdk.java.net/valhalla/valhalla/jdk/rev/836efde938c1 [3] - http://mail.openjdk.java.net/pipermail/valhalla-spec-experts/2016-February/000096.html [4] - http://mail.openjdk.java.net/pipermail/valhalla-spec-experts/2016-May/000147.html Maurizio -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrey.breslav at jetbrains.com Sat May 28 18:34:06 2016 From: andrey.breslav at jetbrains.com (Andrey Breslav) Date: Sat, 28 May 2016 18:34:06 +0000 Subject: Wildcards -- Models 4 and 5 In-Reply-To: <20160526133532.61E2378057@b03ledav004.gho.boulder.ibm.com> References: <2421796e-cb8e-6241-9f80-5bb761673f2e@oracle.com> <20160526133532.61E2378057@b03ledav004.gho.boulder.ibm.com> Message-ID: My gut feeling is also for Model 5. And I even dare ask this: can we maybe retire at least some of the raw types legacy somehow? I can't say I've explored that direction in any real depth, but maybe someone else did? On Thu, May 26, 2016 at 4:36 PM Bjorn B Vardal wrote: > We agree that the potential source incompatibility is an acceptable price > for the reduced bytecode complexity in Model 5. If the source > incompatibility turns out to be more severe than expected, does it make > more sense to bring back separate wildcards (?/ref, any), rather than > bringing back the bytecode complexity of Model 4? > > -- > Bj?rn V?rdal > > > ----- Original message ----- > From: Brian Goetz > Sent by: "valhalla-spec-experts" < > valhalla-spec-experts-bounces at openjdk.java.net> > To: valhalla-spec-experts at openjdk.java.net > Cc: > Subject: Wildcards -- Models 4 and 5 > Date: Fri, May 20, 2016 2:36 PM > > > In the 4/20 mail ?Wildcards and raw types: story so far?, we outlined our > explorations for fitting wildcard types into the first several prototypes. > The summary was: > > - > > Model 1: no wildcards at all > - > > Model 2: A pale implementation of wildcards, with lots of problems > that stem from trying to fake wildcards via interfaces > - > > Model 3: basically the same as Model 2, except members are accessed > via indy (which mitigated some of the problems but not all) > > The conclusion was: compiler-driven translation tricks are not going > to cut it (as we suspected all along). We?ve since explored two other > models (call them 4 and 5) which explore a range of options for VM support > for wildcards. The below is a preliminary analysis of these options. > > Reflection, classes, and runtime types > > While it may not be immediately obvious that this subject is deeply > connected to reflection, consider a typical implementation of equals(): > class Box { > T t; > > public boolean equals(Object o) { > if (!(o instanceof Box)) > return false; > Box other = (Box) o; > return (t == null && other.t == null) > || t.equals(other.t); > } > } > > Some implementations use raw types (Box) for the instanceof and cast > target; others use wildcards (Box). While the latter is recommended, > both are widely used in circulation. In any case, as observed in the last > mail, were we to interpret Box or Box as only including erased boxes, > then this code would silently break. > > The term ?class? is horribly overloaded, used to describe the source class > (class Foo { ... }), the binary classfile, the runtime type derived from > the classfile, and the reflective mirror for that runtime type. In the past > these existed in 1:1 correspondence, but no more ? a single source class > now gives rise to a number of runtime types. Having poor terminology causes > confusion, so let?s refine these terms: > > - *class* refers to a source-level class declaration > - *classfile* refers to the binary classfile > - *template* refers to the runtime representation of a classfile > - *runtime type* refers to a primitive, value, class, or interface > type managed by the VM > > So historically, all objects had a class, which equally described the > source class, the classfile, and the runtime type. Going forward, the class > and the runtime type of an object are distinct concepts. So an > ArrayList has a *class* of ArrayList, but a *runtime type* of > ArrayList. Our code name for runtime type is *crass* (obviously a > better name is needed, but we?ll paint that bikeshed later.) > > This allows us to untangle a question that?s been bugging us: what should > Object.getClass() return on an ArrayList? If we return ArrayList, > then we can?t distinguish between an erased and a specialized object (bad); > if we return ArrayList, then existing code that depends on (x.getClass() > == List.class) may break (bad). > > The answer is, of course, that there are two questions the user can ask an > object: what is your *class*, and what is your *crass*, and they need to > be detangled. The existing method getClass() will continue to return the > class mirror; a new method (getCrass()) will return a runtime type mirror > of some form for the runtime type. Similarly, a class literal will evaluate > to a class, and some other form of literal / reflective lookup will be > needed for crass. > > The reflective features built into the language (instanceof, casting, > class literals, getClass()) are mostly tilted towards classes, not types. > (Some exceptions: you can use a wildcard type in an instanceof, and you > can do unchecked static casts to generic types, which are erased.) We need > to extend these to deal in both classes *and* crasses. For getClass() and > literals, there?s an obvious path: have two forms. For casting, we are > mostly there (except for the treatment of raw types for any-generic classes > ? which we need to work out separately.) For instanceof, it seems a forced > move that instanceof Foo is interpreted as ?an instance of any runtime > type projected from class Foo?, but we also would want to apply it to any > reifiable type as well. > Wildcard types > > In Model 3, we express a parameterized type with a ParamType constant, > which names a template class and a set of type parameters, which include > both valid runtime types as well as the special type parameter token > erased. One natural way to express a wildcard type is to introduce a new > special type parameter token, wild, so we?d translate Foo as > ParamType[Foo,wild]. > > In order for wildcard types to work seamlessly, the minimum functionality > we?d need from the VM is to manage subtyping (which is used by the VM for > instanceof, checkcast, verification, array store checks, and array > covariance.) The wildcard must be seen to be a ?top? type for all > parameterizations: > ParamType[Foo,T] <: ParamType[Foo,wild] // for all valid T > > And, wildcard parameterizations must be seen to be subtypes of of their > wildcard-parameterized supertypes. If we have > class Foo extends Bar implements I { ... } > class Moo extends Goo { } > > then we expect > ParamType[Foo,wild] <: ParamType[Bar,wild] > ParamType[Foo,wild] <: ParamType[I,wild] > ParamType[Moo,wild] <: Goo > > Wildcards must also support method invocation and field access to the > members that are in the intersection of the members of all > parameterizations (these are the total members (those not restricted to > particular instantiations) whose member descriptors do not contain any type > variables.) We can continue to implement member access via invokedynamic > (as we do in Model 3, or alternately, the VM can support invoke* > bytecodes on wildcard receivers.) > > We can apply these wildcard behaviors to any of the wildcard models (i.e., > retrofit them onto Model 2/3.) > Partial wildcards > > With multiple type variables, the rules for wildcards generalize cleanly, > but the number of wildcard types that are a supertype of any given > parameterized type grows exponentially in the number of type variables. We > are considering adopting the simplification of erasing all partial > wildcards in the source type system to a total wildcard in the runtime type > system (the costs of this are: some additional boxing on access paths where > boxing might not be necessary, and unchecked casts when casting a broader > wildcard to a narrower one.) > Model 4 > > A constraint we are under is: existing binaries translate the types Foo > (raw type), Foo (erased parameterization), and Foo all as LFoo; > (or its equivalent, CONSTANT_Class[Foo]); since existing code treats this > as meaning an erased class, the natural path would be to continue to > interpret LFoo; as an erased class. > > Model 4 asks the question: ?can we reinterpret legacy LFoo; in > classfiles, and Foo in source files, as any Foo? (restoring the > interpretation of Foo to be more in line with user intuition.) > > Not surprisingly, the cost of reinterpreting the binaries is extensive. > Many bytecodes would have to be reinterpreted, including new, > {get,put}field, invoke*, to make up the difference between the legacy > meaning of these constructs and the desired new meaning. Worse, while > boxing provides us a means to have a common representation of signatures > involving T (T?s bound), in order to get to a common representation for > signatures involving T[], we?d need to either (a) make int[] a subtype of > Object[] or (b) have a ?boxing conversion? from int[] to Object[] (which > would be a proxy box; the data would still live in the original int[].) > Both are intrusive into the aaload and aastore bytecodes and still are > not anomaly-free. > > So, overall, while this seems possible, the implementation cost is very > high, all of which is for the sake of migration, which will remain as > legacy constraints long after the old code has been migrated. > Model 5 > > Model 5 asks the simpler question: can we continue to interpret LFoo; as > erased in legacy classfiles, but upgrade to treating Foo as is > expected in source code? This entails changing the compilation translation > of Foo from ?erased foo? to ParamType[Foo,wild]. > > This is far less intrusive into the bytecode behavior ? legacy code would > continue to mean what it did at compile time. It does require some > migration support for handling the fact that field and method descriptors > have changed (but this is a problem we?re already working on for managing > the migration of reference classes to value classes.) There are also some > possible source incompatibilities in the face of separate compilation (to > be quantified separately). > > Model 5 allows users to keep their Foo and have it mean what they > think it should mean. So we don?t need to introduce a confusing Foo > wildcard, but we will need a way of saying ?erased Foo?, which might be Foo extends Object> or might be something more compact like Foo. > Comparison > > Comparing the three models for wildcards (2, 4, 5): > > - Model 2 defines the source construct Foo to permanently mean Foo ref>, even when Foo is anyfied, and introduces a new wildcard Foo > ? but maintains source and binary compatibility. > - Model 4 let?s us keep Foo, and retroactively redefines bytecode > behavior ? so an old binary can still interoperate with a reified generic > instance, and will think a Foo is really a Foo. > - Model 5 redefines the *source* meaning of Foo to be what users > expect, but because we don?t reinterpret old binaries, allows some source > incompatibility during migration. > > I think this pretty much explores the solution space. Our choices are: > break the user model of what Foo means, take a probably prohibitive > hit to distort the VM to apply new semantics to old bytecode, or accept > some limited source incompatibility under separate compilation but rescue > the source form that users want. > > In my opinion, the Model 5 direction offers the best balance of costs and > benefits ? while there is some short-term migration pain (in relatively > limited cases, and can be mitigated with compiler help), in the long run, > it gets us to the world we want without permanently burdening either the > language (creating confusion between Foo and Foo) or the VM > implementation. > > In all these cases, we still haven?t defined the semantics of *raw types*. > Raw types existed for migration between pre-generic and generic code; we > still have that migration problem, plus the new migration problems of > generic to any-generic, and of pre-generic to any-generic. So in any case, > we?re going to need to define suitable semantics for raw types > corresponding to any-generic classes. > ? > > > -- Andrey Breslav Project Lead of Kotlin JetBrains http://kotlinlang.org/ The Drive to Develop -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.goetz at oracle.com Mon May 30 23:45:43 2016 From: brian.goetz at oracle.com (Brian Goetz) Date: Mon, 30 May 2016 19:45:43 -0400 Subject: Wildcards -- Models 4 and 5 In-Reply-To: References: <2421796e-cb8e-6241-9f80-5bb761673f2e@oracle.com> <20160526133532.61E2378057@b03ledav004.gho.boulder.ibm.com> Message-ID: > And I even dare ask this: can we maybe retire at least some of the raw > types legacy somehow? > I can't say I've explored that direction in any real depth, but maybe > someone else did? > I don't think this would get us very far, because we're stepping back into the same situation that gave rise to raw types in the first place: the desire for gradual compatible migration to anyfied generics. In fact, we now have three transition modes: - (existing) non-generic to erased-generic - (new) erased-generic to any-generic - (new) non-generic to any-generic Taking the last case, if you go from "class Foo" to "class Foo", you still want gradual migration compatibility for clients and subclasses. Which means some sort of "raw type" is needed, so we have a way to interpret occurrences of "Foo" in clients and subclasses after Foo has been any-generified. From brian.goetz at oracle.com Tue May 31 18:37:11 2016 From: brian.goetz at oracle.com (Brian Goetz) Date: Tue, 31 May 2016 14:37:11 -0400 Subject: Compatibility goals Message-ID: <07de2a60-6b4e-b597-1fca-2f3af30fc7f0@oracle.com> The Model 3 document posits several migration compatibility goals. I'd like to drill into these and get consensus on them, as they influence many other decisions (such as the requirements around raw types.) Here's what the doc said: Compatibility Classes will evolve; some evolutions are compatible, and some are not. The following enumerates the compatibility consequences of the proposed approach. * Alpha-renaming a type variable (to a non-shadowed name) should be binary and source compatible. * Reordering or removing type variables is not compatible. (These first two together match the story for method argument lists; you can rename method arguments, but not reorder or remove them.) * Anyfying an existing erased type variable should be binary and source compatible. * Adding a new type variable/at the end/of the argument list should be binary compatible (though not source compatible.) Adding a new type variable other than at the end is not compatible. * Generifying an enclosing scope (evolving|Outer.Inner|to|Outer.Inner|) should be binary compatible. * Changing type variable bounds is not binary compatible. Does anyone have any concerns with these compatibility goals (in either direction -- that they are too constraining, or not constraining enough?) -------------- next part -------------- An HTML attachment was scrubbed... URL: