RFR: 8316396: Endless loop in C2 compilation triggered by AddNode::IdealIL [v2]

[Vladimir Kozlov]

On Tue, 3 Oct 2023 12:27:15 GMT, Roland Westrelin <roland at openjdk.org> wrote:

>> A number of SubI nodes are laid out in a binary tree. There are <100
>> SubI nodes involved but the tree is fairly deep, and balanced and has
>> 200+ millions of nodes that is the relately small number of SubI nodes
>> are arranged in a way that they build a 200+ millions node tree.
>> 
>> The `AddNode::IdealIL()` transform:
>> 
>> "(a-b)+(c-d)" into "(a+c)-(b+d)"
>> 
>> is applied at the root of the tree. Because `tranform()` is called on
>> the resulting a+c and b+d, the same transformation is applied to each
>> and recursively down the tree. So there's no endless loop but very
>> slow progress because c2 is busy walking and transforming a large
>> tree. There's a lot of redundant work done on the way, because the
>> tree is composed of a relatively small number of nodes. The fix I
>> proposed is to not call `tranform()` but enqueue a+c and b+d with
>> igvn. Some subtrees are shared in the large tree of `SubI`
>> nodes. Calling `tranform()` only causes the node in the tree that
>> called it to be updated while, when delaying to igvn, all uses of the
>> subtree are updated with the transformed subtree.
>
> Roland Westrelin has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains three additional commits since the last revision:
> 
>  - review
>  - Merge branch 'master' into JDK-8316396
>  - fix & test

Looks good.

-------------

Marked as reviewed by kvn (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/15923#pullrequestreview-1655996050