[code-reflection] RFR: Break infinite recursion in ReflectMethods::typeToTypeElement

[Maurizio Cimadamore]

On Thu, 15 Jan 2026 19:25:34 GMT, Adam Sotona <asotona at openjdk.org> wrote:

> Following piece of code to crash our compilation with `StackOverflowError`:
> 
>  @Reflect
>  <E extends Enum<E>> void crash(Enum<E> e) {}
> 
> 
> This PR breaks the infinite recursion in `ReflectMethods::typeToTypeElement`.

The problem of recursive type-variables is quite difficult to address in full.

Some history: initially, our suport for type variables in `JavaType` was restricted to so called "type variable references". E.g. we wanted to capture a "nominal" reference to a variable declaration somewhere (e.g. a method, or a class). This approach would have worked, and would have been free of issue, even in the presence of recursion.

But, as we kept working with the type support, we realized we wanted to have an `erasure` operation on `JavaType` (see `JavaType::erasure`). Now, to correctly compute the erasure of a type variable, we need to know its bound. This is why we eventually augmented `TypeVariable` to also store a bound type. This means that when you call `TypeVariable::erasure` you get the erasure of the type-variable's bound (which is what you want). This operation is used several times in our samples, in the bytecode generator, etc. (so I think we need something like this).

The problem with recursive type-variable definition is that a type-variable bound can contain reference to itself -- as this PR demonstrates:


<X extends Enum<X>>


When that happens, it's important to be able to distinguish between a type-variable _declaration_ and a type-variable _use_. Since a type-variable _use_ doesn't really need any bound (but it implicitly points to a declaration -- which has the bound), no issue arises. But since code models conflate declaration and use, we have an issue.

We could use the logic in this PR to detect a cycle -- and, if so, erase the bound. This is somewhat consistent with what we're doing in other cases -- e.g. when the type attached to a Java expression is too complex and contains types that are not denotable in the source code. One nice property of doing so is that erasing the bound doesn't negatively impact on `JavaType::erasure`. That is, that operation would still work as intended -- only, the type-variable bound would be a bit less sharp.

If we want to do that, and detect cycles in all cases, we need to be careful, as recursion can come in many ways -- here's another:


<X extends List<Y>, Y extends List<X>>


In this case just checking whether, say, `X` is contained inside the bound of of `X` will fail. That bound is `List<Y>`, so `X`'s bound does NOT contain `X` (but it contains `Y`, whose bound contains `X`...)

In other words, in order to properly detect all cycles, we need to _try_ and write down the bound as a `JavaType`. When doing so, we need to keep track of all the type-variables we encounter. If we encounter the same type-variable twice, we need to break out, and fall back with an erased upper bound. (logic such as this is used several times throughout the javac code base).

-------------

PR Comment: https://git.openjdk.org/babylon/pull/849#issuecomment-3759615771