RFR: 8337142: StackOverflowError in Types.containsTypeRecursive with deeply nested type hierarchy [v3]

[Maurizio Cimadamore]

On Mon, 3 Nov 2025 18:28:43 GMT, Vicente Romero <vromero at openjdk.org> wrote:

>> javac can crash with SOE for some deeply nested type hierarchies. See the related JBS issue for the code example. Here we have another example of infinite recursion in the compiler. This is what is happening:
>> 
>> - method Types."isSubtype"::containsTypeRecursive uses a cache, which is a Set, the elements in that cache are of type Types.TypePair. Method TypePair::equals invoke Types::isSameType
>> - Types::isSameType for this test case ends up invoking Types."isSubtype"::containsTypeRecursive, closing the infinite loop
>> 
>> The proposed solution is to define current anonymous class isSameTypeVisitor as a named class and derive from it a visitor that will be the one used by Types.TypePair and thus invoked by TypePair::equals every time instances of TypePair are added or removed from a Map, Set, etc.
>> 
>> This new visitor will compare types looking for exactness which is something desirable in a map key.
>> 
>> TIA
>
> Vicente Romero has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains four additional commits since the last revision:
> 
>  - addressing review comments
>  - Merge branch 'master' into JDK-8337142
>  - addressing review comments
>  - 8337142: StackOverflowError in Types.containsTypeRecursive with deeply nested type hierarchy

Looks good, thanks. I hope the performance of the type comparison doesn't regress too much (given that there's more virtual calls in the middle).

-------------

Marked as reviewed by mcimadamore (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/28101#pullrequestreview-3417490888