RFR: 8316396: Endless loop in C2 compilation triggered by AddNode::IdealIL [v2]

[Roland Westrelin]

> A number of SubI nodes are laid out in a binary tree. There are <100
> SubI nodes involved but the tree is fairly deep, and balanced and has
> 200+ millions of nodes that is the relately small number of SubI nodes
> are arranged in a way that they build a 200+ millions node tree.
> 
> The `AddNode::IdealIL()` transform:
> 
> "(a-b)+(c-d)" into "(a+c)-(b+d)"
> 
> is applied at the root of the tree. Because `tranform()` is called on
> the resulting a+c and b+d, the same transformation is applied to each
> and recursively down the tree. So there's no endless loop but very
> slow progress because c2 is busy walking and transforming a large
> tree. There's a lot of redundant work done on the way, because the
> tree is composed of a relatively small number of nodes. The fix I
> proposed is to not call `tranform()` but enqueue a+c and b+d with
> igvn. Some subtrees are shared in the large tree of `SubI`
> nodes. Calling `tranform()` only causes the node in the tree that
> called it to be updated while, when delaying to igvn, all uses of the
> subtree are updated with the transformed subtree.

Roland Westrelin has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains three additional commits since the last revision:

 - review
 - Merge branch 'master' into JDK-8316396
 - fix & test

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/15923/files
  - new: https://git.openjdk.org/jdk/pull/15923/files/745e1126..4332a045

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=15923&range=01
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=15923&range=00-01

  Stats: 19558 lines in 597 files changed: 14846 ins; 2621 del; 2091 mod
  Patch: https://git.openjdk.org/jdk/pull/15923.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/15923/head:pull/15923

PR: https://git.openjdk.org/jdk/pull/15923