RFR: 8324809: compiler can crash with SOE while proving if two recursive types are disjoint

[Vicente Romero]

javac is crashing with SOE while compiling code like:


class Criteria<B extends Builder<? extends Criteria>> {
    public <D extends Builder<E>, E extends Criteria<D>> D builder() {
        return (D) new Builder<>();
    }

}

class Builder<C extends Criteria<? extends Builder<C>>> {}


here while attributing: `return (D)new Builder<>();` the compiler is trying to prove that: `Builder<C>` is castable to `D`, then as the upper bound of `D` is `Builder<E>`, we need to check if `Builder<C>` is castable to `Builder<E>`. Then the next step is to try to check if type variables C and E are disjoint. During this process is when the compiler gets out of resources given that

E <: Criteria<D>
D <: Builder<E> here we have E again and we start over in an infinite loop,

Here the proposal is to detect cycles in the type arguments graph and accept these type of casts if we find a cycle and thus can't prove if two types are disjoint,

TIA

-------------

Commit messages:
 - 8324809: compiler can crash with SOE while proving if two recursive types are disjoint

Changes: https://git.openjdk.org/jdk/pull/19194/files
  Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=19194&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8324809
  Stats: 56 lines in 2 files changed: 50 ins; 0 del; 6 mod
  Patch: https://git.openjdk.org/jdk/pull/19194.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/19194/head:pull/19194

PR: https://git.openjdk.org/jdk/pull/19194