RFR: JDK-8318183: C2: VM may crash after hitting node limit [v2]

[Thomas Stuefe]

> When processing the node limit, as well as when bailing out of compilation for other reasons, we run `Compiler::record_failure()` which sets the root node of Compile to null.
> 
> We then should stop any further work that may use Compile::_root. That does not always happen. The results are crashes or asserts when the node limit check hit right at the wrong spot. One example for a follow-up assertion after the node limit hit:
> 
> 
> # Internal Error (/shared/projects/openjdk/jdk-jdk/source/src/hotspot/share/opto/loopnode.hpp:623), pid=761308, tid=761322
> # assert(_head != nullptr) failed: precond
> 
> at
> 
> Stack: [0x00007fa068281000,0x00007fa068382000], sp=0x00007fa06837cd40, free space=1007k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
> V [libjvm.so+0x114ad1c] IdealLoopTree::IdealLoopTree(PhaseIdealLoop*, Node*, Node*)+0x14c (loopnode.hpp:623)
> V [libjvm.so+0x11412bf] PhaseIdealLoop::build_and_optimize()+0x237 (loopnode.cpp:4325)
> V [libjvm.so+0x96e7ac] PhaseIdealLoop::PhaseIdealLoop(PhaseIterGVN&, LoopOptsMode)+0x144 (loopnode.hpp:1112)
> V [libjvm.so+0x96e9fa] PhaseIdealLoop::optimize(PhaseIterGVN&, LoopOptsMode)+0x4a (loopnode.hpp:1191)
> V [libjvm.so+0x960394] Compile::Optimize()+0x98c (compile.cpp:2357)
> 
> 
> I saw this during the development of [JDK-8318016](https://bugs.openjdk.org/browse/JDK-8318016), which re-uses the node limit check for memory limit checks, so those code paths are executed more often.
> 
> The patch adds several bailouts to the C2 code. I tried to follow and close all paths that call the node limit check. With this patch above assertion and other crashes and assertions disappear.

Thomas Stuefe has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains four additional commits since the last revision:

 - Fix GraphKit dtor
 - Feedback Tobias
 - Merge branch 'master' into c2-missing-bailouts
 - add bailouts

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/16205/files
  - new: https://git.openjdk.org/jdk/pull/16205/files/eeb64829..af490972

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=16205&range=01
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=16205&range=00-01

  Stats: 964 lines in 57 files changed: 469 ins; 123 del; 372 mod
  Patch: https://git.openjdk.org/jdk/pull/16205.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/16205/head:pull/16205

PR: https://git.openjdk.org/jdk/pull/16205