[code-reflection] RFR: Issue errors when quotable constructs are found inside inner classes

[Paul Sandoz]

On Mon, 7 Jul 2025 15:01:34 GMT, Maurizio Cimadamore <mcimadamore at openjdk.org> wrote:

> This PR adds check in `ReflectMethods` to issue errors when quotable methods, lambdas, method references are found inside an inner class. This includes member inner classes, local and anonymous inner classes, but static inner classes (sometimes referred to as *nested classes*) are ok.
> 
> The issue is that if we allowed code reflection in these places, we would need to come up with uniform treatment for compiler-generated captured symbols.
> Currently `ReflectMethods` just allows method in an inner class to refer to a variable or a method in the enclosing class freely. While this is correct from a source code perspective, by the time we translate to bytecode, such references are no longer possible, and are typically mediated by compiler-generated symbols such as `this$0`.
> 
> So, instead of generating a code model that might be unreliable, it is better, for now, to just disallow such cases.
> 
> In principle, we should just "skip over" code elements we cannot support, and generate bytecode as usual. Indeed this is where I started, but then I realized that the support for lambda conversion into `Quoted` does not make sense unless we enable code reflection (as there's no real functional interface target we can use there). For this reason, an error is generated instead.

Marked as reviewed by psandoz (Lead).

I agree it is best to currently disable code reflection for such cases. I don't think the model is correct, as per the design, because the outer class is not referenced correctly e.g.:


    class Inner {
        @CodeReflection
        public void f() {
            TestInner.this.m();
            m();
        }
    }

    void m() {

    }



func @"f" (%0 : java.type:"TestInner::Inner")java.type:"void" -> {
    invoke %0 @java.ref:"TestInner::m():void";
    invoke %0 @java.ref:"TestInner::m():void";
    return @loc="16:9";
};


So we cannot even interpret the invoke operations correctly using reflective method invocation as their operand is not of the correct type.

A correct model might be:


func @"f" (%0 : java.type:"TestInner", %1 : java.type:"TestInner::Inner")java.type:"void" -> {
    invoke %0 @java.ref:"TestInner::m():void";
    invoke %0 @java.ref:"TestInner::m():void";
    return @loc="16:9";
};
``` 

How feasible would it be to, eventually, do this?

-------------

PR Review: https://git.openjdk.org/babylon/pull/486#pullrequestreview-2995254651
PR Comment: https://git.openjdk.org/babylon/pull/486#issuecomment-3046496081