RFR: 8333258: C2: high memory usage in PhaseCFG::insert_anti_dependences() [v6]

[Roland Westrelin]

> In a debug build, `PhaseCFG::insert_anti_dependences()` is called
> twice for a single node: once for actual processing, once for
> verification.
> 
> In TestAntiDependenciesHighMemUsage, the test has a `Region` that
> merges 337 incoming path. It also has one `Phi` per memory slice that
> are stored to: 1000 `Phi` nodes. Each `Phi` node has 337 inputs that
> are identical except for one. The common input is the memory state on
> method entry. The test has 60 `Load` that needs to be processed for
> anti dependences. All `Load` share the same memory input: the memory
> state on method entry. For each `Load`, all `Phi` nodes are pushed 336
> times on the work lists for anti dependence processing because all of
> them appear multiple times as uses of each `Load`s memory state: `Phi`s
> are pushed 336 000 on 2 work lists. Memory is not reclaimed on exit
> from `PhaseCFG::insert_anti_dependences()` so memory usage grows as
> `Load` nodes are processed:
> 
> 336000 * 2 work lists * 60 loads * 8 bytes pointer = 322 MB. 
> 
> The fix I propose for this is to not push `Phi` nodes more than once
> when they have the same inputs multiple times.
> 
> In TestAntiDependenciesHighMemUsage2, the test has 4000 loads. For
> each of them, when processed for anti dependences, all 4000 loads are
> pushed on the work lists because they share the same memory
> input. Then when they are popped from the work list, they are
> discarded because only stores are of interest:
> 
> 4000 loads processed * 4000 loads pushed * 2 work lists * 8 bytes pointer = 256 MB. 
> 
> The fix I propose for this is to test before pushing on the work list
> whether a node is a store or not.
> 
> Finally, I propose adding a `ResourceMark` so memory doesn't
> accumulate over calls to `PhaseCFG::insert_anti_dependences()`.

Roland Westrelin has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains 13 additional commits since the last revision:

 - review
 - Merge branch 'master' into JDK-8333258
 - more review
 - more review
 - Merge branch 'master' into JDK-8333258
 - review
 - Merge branch 'master' into JDK-8333258
 - refactoring
 - Merge branch 'master' into JDK-8333258
 - review
 - ... and 3 more: https://git.openjdk.org/jdk/compare/a01486a6...4511c175

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/19791/files
  - new: https://git.openjdk.org/jdk/pull/19791/files/15a33090..4511c175

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=19791&range=05
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=19791&range=04-05

  Stats: 11038 lines in 484 files changed: 6574 ins; 1789 del; 2675 mod
  Patch: https://git.openjdk.org/jdk/pull/19791.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/19791/head:pull/19791

PR: https://git.openjdk.org/jdk/pull/19791