RFR: 8324809: compiler can crash with SOE while proving if two recursive types are disjoint [v2]

[Maurizio Cimadamore]

On Thu, 16 May 2024 18:30:36 GMT, Vicente Romero <vromero at openjdk.org> wrote:

>> javac is crashing with SOE while compiling code like:
>> 
>> 
>> class Criteria<B extends Builder<? extends Criteria>> {
>>     public <D extends Builder<E>, E extends Criteria<D>> D builder() {
>>         return (D) new Builder<>();
>>     }
>> 
>> }
>> 
>> class Builder<C extends Criteria<? extends Builder<C>>> {}
>> 
>> 
>> here while attributing: `return (D)new Builder<>();` the compiler is trying to prove that: `Builder<C>` is castable to `D`, then as the upper bound of `D` is `Builder<E>`, we need to check if `Builder<C>` is castable to `Builder<E>`. Then the next step is to try to check if type variables C and E are disjoint. During this process is when the compiler gets out of resources given that
>> 
>> E <: Criteria<D>
>> D <: Builder<E> here we have E again and we start over in an infinite loop,
>> 
>> Here the proposal is to detect cycles in the type arguments graph and accept these type of casts if we find a cycle and thus can't prove if two types are disjoint,
>> 
>> TIA
>
> Vicente Romero has updated the pull request incrementally with one additional commit since the last revision:
> 
>   addressing review comments

Looks good

-------------

Marked as reviewed by mcimadamore (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/19194#pullrequestreview-2061604999