From hjohn at xs4all.nl Tue Nov 1 08:04:39 2022 From: hjohn at xs4all.nl (John Hendrikx) Date: Tue, 01 Nov 2022 08:04:39 +0000 Subject: Loosening requirements for super() invocation In-Reply-To: References: <11B5833E-2B52-4174-8608-6B0373D6A2A5@oracle.com> Message-ID: I've been looking at the examples, and I find none of them compelling. It seems to be primarily fighting against the symptoms of another problem: `this` references escaping. The `FilteredSet` example is a good example of why calling an instance method before an object is fully constructed is bad. In this case `HashSet` itself is calling a non-final method `addAll`, instead of a final `internalAddAll` or something similar. Furthermore, `FilteredSet` is relying on this undocumented behavior, which is bad as this presumably can still change in a future JDK. Furthermore, even with the current restrictions in place, a `FilteredSet` can be implemented fairly easily, so in itself that's not a good enough reason to relax the super calling requirements. The `BigPositiveValue` example is quite contrived (as you said), and not something I'd ever need to write. First, it perhaps feels more natural to check failing conditions before calling super, but as long as construction is prevented, the end result is the same. A smart compiler can re-order these I suppose. Second, giving the constructors radically different behaviors when numeric input is either `long` or `float` would never pass review. Static factory methods that can be named ("of(long)", "pow2(float)") would be ideally suited to disambiguate these and avoid potential bugs and confusion. I find the idea of having better warnings (or even errors) for `this` references escaping much more compelling. It should flag any method calls that aren't private or final, with implicit or explicit this references. This would highlight the mistakes made in `HashSet` allowing them to be fixed. --John ------ Original Message ------ >From "Archie Cobbs" To amber-dev at openjdk.java.net Cc "Vicente Romero" ; "Brian Goetz" Date 31/10/2022 20:18:19 Subject Re: Loosening requirements for super() invocation >This is an old thread which I've been looking at again lately. > >To summarize, the idea is to relax the JLS rules about code prior to >super()/this() invocation to more closely match what the JVM allows, >which is field assignments and any code that doesn't reference 'this'. >Relevant issues are JDK-8194743 > and JDK-8193760 >. > >In order to make this idea less theoretical, I've prototyped all the >required compiler changes. There is also a write-up describing them >. > >There are a few interesting subtleties, e.g. relating to initialization >blocks, and there is also a sub-question, which is whether to allow >invoking super()/this() within a try block (this would have to be under >the restriction that any catch or finally clause must not return >normally). Currently that's not possible without a small JVM change, >which I also think might be worth considering to really round out this >new feature. See the writeup for details. > >To see some examples of what would be allowed, see this unit test >. >The compiler changes are here >. > >Thoughts? > >-Archie > >On Thu, Jan 17, 2019 at 8:42 AM Brian Goetz >wrote: >>Some things have improved for this feature since we last talked; >>several verifier issues that this would have pushed on have been >>resolved. So it?s moved from the ?way too expensive for the benefit? >>category into the ?there are lots of things we can do, is this really >>what we want to spend our effort and complexity budget on? category. >> >>My view on this is that while there?s nothing wrong with it, it?s also >>a pretty minor wart. If this fell out of a bigger feature, I?d >>certainly not object, but I?d rather spend the effort and complexity >>budget on things that have broader benefit. >> >> > On Jan 16, 2019, at 5:48 PM, Archie Cobbs >>wrote: >> > >> > I'm curious what are people's current thoughts on loosening the >> > requirements for super() invocation in the context of Amber, e.g.: >> > >> > public class MyInputStream extends FilterInputStream { >> > public MyInputStream(InputStream in) { >> > if (in == null) >> > throw new IllegalArgumentException("null input"); >> > super(in); // look ma! >> > } >> > } >> > > >-- >Archie L. Cobbs -------------- next part -------------- An HTML attachment was scrubbed... URL: From archie.cobbs at gmail.com Tue Nov 1 15:43:45 2022 From: archie.cobbs at gmail.com (Archie Cobbs) Date: Tue, 1 Nov 2022 10:43:45 -0500 Subject: Loosening requirements for super() invocation In-Reply-To: <72c76b30-43e8-e9c1-5321-e2a54d242008@oracle.com> References: <11B5833E-2B52-4174-8608-6B0373D6A2A5@oracle.com> <72c76b30-43e8-e9c1-5321-e2a54d242008@oracle.com> Message-ID: Thanks for the various comments. I agree the topic of "this" escaping is definitely an interesting one. Although there is a lot of code out there that currently relies on it (e.g., HashSet and AbstractCollection), perhaps someday we could realize this in a backward-compatible way by adding a notion of a "safe constructor" which would be limited to following more strict rules that disallowed "this" escapes. Perhaps you could declare your constructor (and/or initialized fields) to be "safe" with an annotation, e.g.: @SafeConstruction public MyClass(int x) { this.x = x; // ok this.someMethod(); // error here! } However I'd also like to keep this discussion organized with an important observation... The topic of "this" escape is completely orthogonal to this change. None of the code that would be allowed before super() or this() could allow "this" to escape, because you're not allowed to reference "this" in any way except for assigning values to fields. The only thing you can do is what might be called "static housekeeping". So this change does not make the problem of "this" escape any worse than it already is. That means the two problems are really separate and can be addressed separately and independently. So I agree that Java has issues with "this" escape, and there are some interesting ways we could address it, but I also don't see that as a reason to prevent people from being able to do "static housekeeping" prior to their this() or super() invocation. I think the "try" part is likely to be off the table; too intrusive for > too little value. > There is one other reason to consider the JVM change, which is that currently it's not possible for bytecode rewriting tools (e.g., aspectj) to fully instrument constructors. The StackOverlow link talks about this use case. Another minor factor is that the class file format wouldn't actually need to change, only how it's verified. Have you looked at the spec to see what would need to be adjusted? > I've not gotten that far yet but would be happy to help/participate with some guidance. Thanks, -Archie On Mon, Oct 31, 2022 at 5:08 PM Brian Goetz wrote: > This looks like some nice progress. > > I would really like it if we had some warnings to detect more of the cases > where `this` might escape construction, such as passing `this` explicitly > to a method, invoking a non-final instance method, or invoking an instance > method whose implementation is in a superclass. (Same for use of `this` in > init blocks.) > > I think the "try" part is likely to be off the table; too intrusive for > too little value. > > Have you looked at the spec to see what would need to be adjusted? > > > > > On 10/31/2022 3:18 PM, Archie Cobbs wrote: > > This is an old thread which I've been looking at again lately. > > To summarize, the idea is to relax the JLS rules about code prior to > super()/this() invocation to more closely match what the JVM allows, which > is field assignments and any code that doesn't reference 'this'. Relevant > issues are JDK-8194743 and > JDK-8193760 . > > In order to make this idea less theoretical, I've prototyped all the > required compiler changes. There is also a write-up describing them > > . > > There are a few interesting subtleties, e.g. relating to initialization > blocks, and there is also a sub-question, which is whether to allow > invoking super()/this() within a try block (this would have to be under the > restriction that any catch or finally clause must not return normally). > Currently that's not possible without a small JVM change, which I also > think might be worth considering to really round out this new feature. See > the writeup for details. > > To see some examples of what would be allowed, see this unit test > . > The compiler changes are here > > . > > Thoughts? > > -Archie > > On Thu, Jan 17, 2019 at 8:42 AM Brian Goetz > wrote: > >> Some things have improved for this feature since we last talked; several >> verifier issues that this would have pushed on have been resolved. So it?s >> moved from the ?way too expensive for the benefit? category into the ?there >> are lots of things we can do, is this really what we want to spend our >> effort and complexity budget on? category. >> >> My view on this is that while there?s nothing wrong with it, it?s also a >> pretty minor wart. If this fell out of a bigger feature, I?d certainly not >> object, but I?d rather spend the effort and complexity budget on things >> that have broader benefit. >> >> > On Jan 16, 2019, at 5:48 PM, Archie Cobbs >> wrote: >> > >> > I'm curious what are people's current thoughts on loosening the >> > requirements for super() invocation in the context of Amber, e.g.: >> > >> > public class MyInputStream extends FilterInputStream { >> > public MyInputStream(InputStream in) { >> > if (in == null) >> > throw new IllegalArgumentException("null input"); >> > super(in); // look ma! >> > } >> > } >> > >> > > -- > Archie L. Cobbs > > > -- Archie L. Cobbs -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.goetz at oracle.com Tue Nov 1 16:45:10 2022 From: brian.goetz at oracle.com (Brian Goetz) Date: Tue, 1 Nov 2022 12:45:10 -0400 Subject: Loosening requirements for super() invocation In-Reply-To: References: <11B5833E-2B52-4174-8608-6B0373D6A2A5@oracle.com> <72c76b30-43e8-e9c1-5321-e2a54d242008@oracle.com> Message-ID: <6bdabf6f-c412-46fe-d00c-1e328afb10d5@oracle.com> > I agree the topic of "this" escaping is definitely an interesting one. > Although there is a lot of code out there that currently relies on it > (e.g., HashSet and AbstractCollection), perhaps someday we could > realize this in a backward-compatible way by adding a notion of a > "safe constructor" which would be limited to following more strict > rules that disallowed "this" escapes. So, what you're saying is that we can't immediately promote these to errors, but that doesn't mean we can't have warnings.? (We are fairly disciplined about zero-warnings in the JDK, using @SuppressWarnings to augment type checking with human judgment.) Warnings are an important part of the safety story.? For example, generics are subject to heap pollution because of erasure, but we make a strong guarantee anyway: if the entire codebase compiles without raw or unchecked warnings, then the synthetic casts introduced by the compiler at the boundary of generic and non-generic code are guaranteed to succeed.? Similarly, while it is a high bar, if we can reliably warn about this-escape, we can make a similar guarantee for the final-field initialization safety guarantees, which are voided by this-escape. > So this change does not make the problem of "this" escape any worse > than it already is. That means the two problems are really separate > and can be addressed separately and independently. I get where you're coming from; you have a patch that you'd like to see integrated, so you're trying to keep the requirements narrow. But realistically, this patch surely needs a JEP, because it affects the language spec and the user-visible programming model.? And such a JEP is going to be more compelling if it addresses the closely-related issue of this-escape. -------------- next part -------------- An HTML attachment was scrubbed... URL: From amaembo at gmail.com Tue Nov 1 16:54:28 2022 From: amaembo at gmail.com (Tagir Valeev) Date: Tue, 1 Nov 2022 17:54:28 +0100 Subject: Loosening requirements for super() invocation In-Reply-To: References: <11B5833E-2B52-4174-8608-6B0373D6A2A5@oracle.com> Message-ID: Hello! As an IDE developer, I support allowing statements before this/super invocation, without referencing `this`. This is often an annoying edge case in many refactorings or quick-fixes. E.g.: - If you want to introduce local variable from an argument expression in this/super call, you cannot do this - If you want to inline method called inside this/super call (like inline method() in super(method(x))), you cannot do this, unless method() is a one-liner. Even with one-liner it's complicated, as we usually add variable declarations to assign parameters, then try to inline these declarations, but this approach doesn't work well here, as these temporary declarations are forbidden. - If you have a stream API call inside this/super call (like super(list.stream().map(...).filter(...).toList())), you cannot desugar it to loop, as there's no place to put this loop. On the other hand, I don't think that allowing `this` references (e.g., field initialization) before this/super call is a good idea. This would add more complexity to the specification and more confusion with little advantage. Supporting super() inside try also sounds like an overkill. Btw what about inside the synchronized {} section? With best regards, Tagir Valeev. On Mon, Oct 31, 2022 at 8:18 PM Archie Cobbs wrote: > > This is an old thread which I've been looking at again lately. > > To summarize, the idea is to relax the JLS rules about code prior to super()/this() invocation to more closely match what the JVM allows, which is field assignments and any code that doesn't reference 'this'. Relevant issues are JDK-8194743 and JDK-8193760. > > In order to make this idea less theoretical, I've prototyped all the required compiler changes. There is also a write-up describing them. > > There are a few interesting subtleties, e.g. relating to initialization blocks, and there is also a sub-question, which is whether to allow invoking super()/this() within a try block (this would have to be under the restriction that any catch or finally clause must not return normally). Currently that's not possible without a small JVM change, which I also think might be worth considering to really round out this new feature. See the writeup for details. > > To see some examples of what would be allowed, see this unit test. The compiler changes are here. > > Thoughts? > > -Archie > > On Thu, Jan 17, 2019 at 8:42 AM Brian Goetz wrote: >> >> Some things have improved for this feature since we last talked; several verifier issues that this would have pushed on have been resolved. So it?s moved from the ?way too expensive for the benefit? category into the ?there are lots of things we can do, is this really what we want to spend our effort and complexity budget on? category. >> >> My view on this is that while there?s nothing wrong with it, it?s also a pretty minor wart. If this fell out of a bigger feature, I?d certainly not object, but I?d rather spend the effort and complexity budget on things that have broader benefit. >> >> > On Jan 16, 2019, at 5:48 PM, Archie Cobbs wrote: >> > >> > I'm curious what are people's current thoughts on loosening the >> > requirements for super() invocation in the context of Amber, e.g.: >> > >> > public class MyInputStream extends FilterInputStream { >> > public MyInputStream(InputStream in) { >> > if (in == null) >> > throw new IllegalArgumentException("null input"); >> > super(in); // look ma! >> > } >> > } >> > > > > -- > Archie L. Cobbs From archie.cobbs at gmail.com Tue Nov 1 21:13:06 2022 From: archie.cobbs at gmail.com (Archie Cobbs) Date: Tue, 1 Nov 2022 16:13:06 -0500 Subject: Loosening requirements for super() invocation In-Reply-To: <6bdabf6f-c412-46fe-d00c-1e328afb10d5@oracle.com> References: <11B5833E-2B52-4174-8608-6B0373D6A2A5@oracle.com> <72c76b30-43e8-e9c1-5321-e2a54d242008@oracle.com> <6bdabf6f-c412-46fe-d00c-1e328afb10d5@oracle.com> Message-ID: On Tue, Nov 1, 2022 at 11:45 AM Brian Goetz wrote: > So, what you're saying is that we can't immediately promote these to > errors, but that doesn't mean we can't have warnings. (We are fairly > disciplined about zero-warnings in the JDK, using @SuppressWarnings to > augment type checking with human judgment.) > > Warnings are an important part of the safety story. For example, generics > are subject to heap pollution because of erasure, but we make a strong > guarantee anyway: if the entire codebase compiles without raw or unchecked > warnings, then the synthetic casts introduced by the compiler at the > boundary of generic and non-generic code are guaranteed to succeed. > Similarly, while it is a high bar, if we can reliably warn about > this-escape, we can make a similar guarantee for the final-field > initialization safety guarantees, which are voided by this-escape. > Yes, I really like this idea... so you could compile with something like javac -Xlint:thisEscape and get all the 'this' escapes reported as warnings. Developers could control it using the usual methods, i.e., command line flags and/or @SuppressWarnings annotations. Of course the devil is in the details... First of all, when should this rule apply? At any point in a constructor (one that successfully compiles), the new object is always definitively in one of three states: State 0: neither super() nor this() has been invoked; neither the superclass nor this class are initialized State 1: super() has been invoked, so the superclass is initialized, but at least one final field in this class has not been initialized State 2: super() has been invoked and all final fields in this class have been initialized BTW, the current state is easily tracked within the existing DA/DU analysis of the prototype patch. The possible state transitions are: - super() takes you from: - State 0 ? State 1 if there remain any uninitialized final fields - State 0 ? State 2 if there are no remaining uninitialized final fields - this() takes you from State 0 ? State 2 - Initializing the last final field takes you from: - State 0 ? State 0 - State 1 ? State 2 Presumably the 'this' escape analysis should only apply in State 1. This is when the object is vulnerable because it's half initialized. One might also argue that "uninitialized final fields" should be replaced by "fields that are yet to be assigned later in the constructor" or somesuch. After all, just because a field isn't final, that doesn't mean it's not requiring initialization. Or maybe there should be some other way to denote such fields, e.g., a new @RequiresInit annotation. Assuming you're in State 1, what exactly would constitute a detectable 'this' escape? Could be something like this: - Invoking any non-static method on an object known to be equal to 'this'. - Passing an object known to be equal to 'this' as a parameter to any method or constructor (including implicit outer instance and free variable proxy constructor parameters). - Throwing 'this' In other words, the 'this' instance literally cannot escape the constructor's stack frame. For the meaning of "known to be equal to 'this'", we would be limited in our ability to infer it in the usual ways (Turing complete, blah blah), so you'd still be able to evade the warning e.g.: public class MyClass implements Iterator