RFR: 8333766: Stack overflow with anonymous class in super() parameter

[Maurizio Cimadamore]

On Fri, 7 Jun 2024 18:00:50 GMT, Maurizio Cimadamore <mcimadamore at openjdk.org> wrote:

> This patch fixes an issue in `LambdaToMethod` where the wrong code is generated to access a symbol in the enclosing scope.
> 
> `LambdaTranslationContext::addSymbol` has some logic to avoid referring to `T.this` when `T` is a type under construction. This fix works well when a lambda is directly enclosed in a `super` or `this` call, but does not scale to more complex cases, such as the one in this issue.
> 
> A more complete fix needs to check whether the target `T.this` is really available via a chain of enclosing `this$n` fields. If that's not the case, then `T.this` must be captured by the lambda more directly, as there's no *path* to it from the innermost enclosing this.
> 
> To detect that, I've added a method (`outerThisReachable`) which more or less mimic the lookup logic in `Lower::makeOwnerThisN`.

src/jdk.compiler/share/classes/com/sun/tools/javac/comp/LambdaToMethod.java line 2121:

> 2119:                 if (skind == CAPTURED_THIS && sym != null && sym.kind == TYP) {
> 2120:                     ClassSymbol currentClass = currentClass();
> 2121:                     if (currentClass != null && currentClass != sym && !outerThisReachable(currentClass.type, sym)) {

There is a subtle change here - note that now if `currentClass == sym` we just skip this part (in this case just using `this` is enough).

src/jdk.compiler/share/classes/com/sun/tools/javac/comp/LambdaToMethod.java line 2134:

> 2132:             }
> 2133: 
> 2134:             private boolean outerThisReachable(Type current, Symbol target) {

The idea here is: if we can reach the target symbol from the current one w/o traversing any type under construction, or any type for which `NOOUTERTHIS` is set, then we can consider the target `T.this` reachable.

src/jdk.compiler/share/classes/com/sun/tools/javac/comp/LambdaToMethod.java line 2140:

> 2138:                     return true;
> 2139:                 } else {
> 2140:                     return current.tsym.hasOuterInstance() &&

note that `hasOuterInstance` takes into account the `NOOUTERTHIS` flag, that is, whether the inner class can have an enclosing instance at all.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/19604#discussion_r1631536921
PR Review Comment: https://git.openjdk.org/jdk/pull/19604#discussion_r1631539474
PR Review Comment: https://git.openjdk.org/jdk/pull/19604#discussion_r1631538135