RFR: 8333697: C2: Hit MemLimit in PhaseCFG::global_code_motion [v2]

[Roland Westrelin]

> I investigated the failure from the `Test.java` that's attached to the
> bug. The failure with this test is only reproducible up to 8334060
> (Implementation of Late Barrier Expansion for G1) so experiments I
> describe here are from the source code for the commit right before it.
> 
> Peak malloc memory usage reported by NMT is: 1.3GB
> 
> `PhaseCFG::global_code_motion()`, when `OptoRegScheduling` is true,
> creates a `PhaseIFG` that's, when initialized, allocates `_adjs`: a
> `maxlrg` array of `IndexSet`s that can contain up to `maxlrg`.
> 
> `maxlrg` in this case is 122839. An `IndexSet` is an array of pointers
> to a 256 bit bitset: one `IndexSet` array needs: 
> 
> 
> 122839 / 256 * 8 = 3832
> 
> 
> and there are of 122839:
> 
> 
> 3832 * 122839 = ~470 MB
> 
> 
> It turns out the `PhaseIFG` object when used from
> `PhaseCFG::global_code_motion()` doesn't even use the `_adjs`
> array. So a patch like:
> 
> 
> diff --git a/src/hotspot/share/opto/chaitin.hpp b/src/hotspot/share/opto/chaitin.hpp
> index cf02deb6019..4e5333bf181 100644
> --- a/src/hotspot/share/opto/chaitin.hpp
> +++ b/src/hotspot/share/opto/chaitin.hpp
> @@ -258,7 +258,7 @@ class PhaseIFG : public Phase {
>    VectorSet *_yanked;
>  
>    PhaseIFG( Arena *arena );
> -  void init( uint maxlrg );
> +  void init( uint maxlrg, bool no_adjs = false );
>  
>    // Add edge between a and b.  Returns true if actually added.
>    int add_edge( uint a, uint b );
> diff --git a/src/hotspot/share/opto/gcm.cpp b/src/hotspot/share/opto/gcm.cpp
> index ebdefe597ff..fefd75a88c5 100644
> --- a/src/hotspot/share/opto/gcm.cpp
> +++ b/src/hotspot/share/opto/gcm.cpp
> @@ -1704,7 +1704,9 @@ void PhaseCFG::global_code_motion() {
>      rm_live.reset_to_mark();           // Reclaim working storage
>      IndexSet::reset_memory(C, &live_arena);
>      uint node_size = regalloc._lrg_map.max_lrg_id();
> -    ifg.init(node_size); // Empty IFG
> +    ifg.init(node_size, true); // Empty IFG
>      regalloc.set_ifg(ifg);
>      regalloc.set_live(live);
>      regalloc.gather_lrg_masks(false);    // Collect LRG masks
> diff --git a/src/hotspot/share/opto/ifg.cpp b/src/hotspot/share/opto/ifg.cpp
> index d12698121b9..e42121c2254 100644
> --- a/src/hotspot/share/opto/ifg.cpp
> +++ b/src/hotspot/share/opto/ifg.cpp
> @@ -42,18 +42,24 @@
>  PhaseIFG::PhaseIFG( Arena *arena ) : Phase(Interference_Graph), _arena(arena) {
>  }
>  
> -void PhaseIFG::init( uint maxlrg ) {
> +void PhaseIFG::init( uint maxlrg, bool no_adjs ) {
>    _maxlrg = maxlrg;
>    _yanked = new (_arena) VectorSet(_arena);
>    _is_square = false;
>    // Make uninitialized adjacency lists
> -  ...

Roland Westrelin has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains three additional commits since the last revision:

 - review
 - Merge branch 'master' into JDK-8333697
 - fix

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/23075/files
  - new: https://git.openjdk.org/jdk/pull/23075/files/7bdb8c41..a96aa572

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=23075&range=01
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=23075&range=00-01

  Stats: 60570 lines in 3664 files changed: 28295 ins; 16492 del; 15783 mod
  Patch: https://git.openjdk.org/jdk/pull/23075.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/23075/head:pull/23075

PR: https://git.openjdk.org/jdk/pull/23075