RFR: JDK-8318183: C2: VM may crash after hitting node limit

[Thomas Stuefe]

When processing the node limit, as well as when bailing out of compilation for other reasons, we run `Compiler::record_failure()` which sets the root node of Compile to null.

We then should stop any further work that may use Compile::_root. That does not always happen. The results are crashes or asserts when the node limit check hit right at the wrong spot. One example for a follow-up assertion after the node limit hit:


# Internal Error (/shared/projects/openjdk/jdk-jdk/source/src/hotspot/share/opto/loopnode.hpp:623), pid=761308, tid=761322
# assert(_head != nullptr) failed: precond

at

Stack: [0x00007fa068281000,0x00007fa068382000], sp=0x00007fa06837cd40, free space=1007k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x114ad1c] IdealLoopTree::IdealLoopTree(PhaseIdealLoop*, Node*, Node*)+0x14c (loopnode.hpp:623)
V [libjvm.so+0x11412bf] PhaseIdealLoop::build_and_optimize()+0x237 (loopnode.cpp:4325)
V [libjvm.so+0x96e7ac] PhaseIdealLoop::PhaseIdealLoop(PhaseIterGVN&, LoopOptsMode)+0x144 (loopnode.hpp:1112)
V [libjvm.so+0x96e9fa] PhaseIdealLoop::optimize(PhaseIterGVN&, LoopOptsMode)+0x4a (loopnode.hpp:1191)
V [libjvm.so+0x960394] Compile::Optimize()+0x98c (compile.cpp:2357)


I saw this during the development of [JDK-8318016](https://bugs.openjdk.org/browse/JDK-8318016), which re-uses the node limit check for memory limit checks, so those code paths are executed more often.

The patch adds several bailouts to the C2 code. I tried to follow and close all paths that call the node limit check. With this patch above assertion and other crashes and assertions disappear.

-------------

Commit messages:
 - add bailouts

Changes: https://git.openjdk.org/jdk/pull/16205/files
 Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=16205&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8318183
  Stats: 24 lines in 7 files changed: 21 ins; 0 del; 3 mod
  Patch: https://git.openjdk.org/jdk/pull/16205.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/16205/head:pull/16205

PR: https://git.openjdk.org/jdk/pull/16205