RFR: 8315916: assert(C->live_nodes() <= C->max_node_limit()) failed: Live Node limit exceeded [v12]

[Dhamoder Nalla]

> **Problem:**
> In the debug build, the assertion assert(C->live_nodes() <= C->max_node_limit()) is triggered during the parsing phase when the compiler creates more than 80K live nodes while scalarizing large arrays. In the release build, however, compilation proceeds until code generation and then bails out at Compile::check_node_count(), completing execution without crashing.
> 
> This discrepancy occurs because two Phi nodes are added per array element during scalar replacement, leading to a rapid increase in node count—especially when the EliminateAllocationArraySizeLimit JVM option is set high. When the assert is commented out, both builds behave similarly, bailing out during code generation after fully building the ideal graph.
> 
> **Proposed Solution:**
> Introduce a bailout check during graph building in the scalar replacement phase. If the number of live nodes exceeds a defined threshold, the compiler will bail out and trigger a recompilation without Escape Analysis (EA). This prevents the construction of an excessively large graph and ensures consistency between debug and release builds.
> 
> This approach is preferable because the graph may already be partially modified with scalarization-related nodes, and a clean recompilation path helps maintain compiler stability and performance. The bailout logic has been aligned with the existing mechanism in escape.cpp and refactored to reuse the same functionality where appropriate.

Dhamoder Nalla has updated the pull request incrementally with one additional commit since the last revision:

  add additional run with fewer flags

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/20504/files
  - new: https://git.openjdk.org/jdk/pull/20504/files/8581a24f..ac393901

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=20504&range=11
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=20504&range=10-11

  Stats: 5 lines in 1 file changed: 5 ins; 0 del; 0 mod
  Patch: https://git.openjdk.org/jdk/pull/20504.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/20504/head:pull/20504

PR: https://git.openjdk.org/jdk/pull/20504