RFR: 8316396: Endless loop in C2 compilation triggered by AddNode::IdealIL

[Tobias Hartmann]

On Tue, 26 Sep 2023 13:48:56 GMT, Roland Westrelin <roland at openjdk.org> wrote:

> A number of SubI nodes are laid out in a binary tree. There are <100
> SubI nodes involved but the tree is fairly deep, and balanced and has
> 200+ millions of nodes that is the relately small number of SubI nodes
> are arranged in a way that they build a 200+ millions node tree.
> 
> The `AddNode::IdealIL()` transform:
> 
> "(a-b)+(c-d)" into "(a+c)-(b+d)"
> 
> is applied at the root of the tree. Because `tranform()` is called on
> the resulting a+c and b+d, the same transformation is applied to each
> and recursively down the tree. So there's no endless loop but very
> slow progress because c2 is busy walking and transforming a large
> tree. There's a lot of redundant work done on the way, because the
> tree is composed of a relatively small number of nodes. The fix I
> proposed is to not call `tranform()` but enqueue a+c and b+d with
> igvn. Some subtrees are shared in the large tree of `SubI`
> nodes. Calling `tranform()` only causes the node in the tree that
> called it to be updated while, when delaying to igvn, all uses of the
> subtree are updated with the transformed subtree.

Looks reasonable to me but I'm wondering if other, similar optimizations aren't affected by this as well? Could you please add a comment, explaining why we avoid transform here?

-------------

Marked as reviewed by thartmann (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/15923#pullrequestreview-1654386812