RFR: JDK-8318183: C2: VM may crash after hitting node limit

[Tobias Hartmann]

On Mon, 16 Oct 2023 16:57:03 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

> When processing the node limit, as well as when bailing out of compilation for other reasons, we run `Compiler::record_failure()` which sets the root node of Compile to null.
> 
> We then should stop any further work that may use Compile::_root. That does not always happen. The results are crashes or asserts when the node limit check hit right at the wrong spot. One example for a follow-up assertion after the node limit hit:
> 
> 
> # Internal Error (/shared/projects/openjdk/jdk-jdk/source/src/hotspot/share/opto/loopnode.hpp:623), pid=761308, tid=761322
> # assert(_head != nullptr) failed: precond
> 
> at
> 
> Stack: [0x00007fa068281000,0x00007fa068382000], sp=0x00007fa06837cd40, free space=1007k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
> V [libjvm.so+0x114ad1c] IdealLoopTree::IdealLoopTree(PhaseIdealLoop*, Node*, Node*)+0x14c (loopnode.hpp:623)
> V [libjvm.so+0x11412bf] PhaseIdealLoop::build_and_optimize()+0x237 (loopnode.cpp:4325)
> V [libjvm.so+0x96e7ac] PhaseIdealLoop::PhaseIdealLoop(PhaseIterGVN&, LoopOptsMode)+0x144 (loopnode.hpp:1112)
> V [libjvm.so+0x96e9fa] PhaseIdealLoop::optimize(PhaseIterGVN&, LoopOptsMode)+0x4a (loopnode.hpp:1191)
> V [libjvm.so+0x960394] Compile::Optimize()+0x98c (compile.cpp:2357)
> 
> 
> I saw this during the development of [JDK-8318016](https://bugs.openjdk.org/browse/JDK-8318016), which re-uses the node limit check for memory limit checks, so those code paths are executed more often.
> 
> The patch adds several bailouts to the C2 code. I tried to follow and close all paths that call the node limit check. With this patch above assertion and other crashes and assertions disappear.

src/hotspot/share/opto/parse1.cpp line 1578:

> 1576: 
> 1577:     do_one_bytecode();
> 1578:     if (C->failing()) return;

I think you can simply call `failing()` here and potentially at other places as well.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/16205#discussion_r1361921816