Integrated: JDK-8318183: C2: VM may crash after hitting node limit

[Thomas Stuefe]

On Mon, 16 Oct 2023 16:57:03 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

> When processing the node limit, as well as when bailing out of compilation for other reasons, we run `Compiler::record_failure()` which sets the root node of Compile to null.
> 
> We then should stop any further work that may use Compile::_root. That does not always happen. The results are crashes or asserts when the node limit check hit right at the wrong spot. One example for a follow-up assertion after the node limit hit:
> 
> 
> # Internal Error (/shared/projects/openjdk/jdk-jdk/source/src/hotspot/share/opto/loopnode.hpp:623), pid=761308, tid=761322
> # assert(_head != nullptr) failed: precond
> 
> at
> 
> Stack: [0x00007fa068281000,0x00007fa068382000], sp=0x00007fa06837cd40, free space=1007k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
> V [libjvm.so+0x114ad1c] IdealLoopTree::IdealLoopTree(PhaseIdealLoop*, Node*, Node*)+0x14c (loopnode.hpp:623)
> V [libjvm.so+0x11412bf] PhaseIdealLoop::build_and_optimize()+0x237 (loopnode.cpp:4325)
> V [libjvm.so+0x96e7ac] PhaseIdealLoop::PhaseIdealLoop(PhaseIterGVN&, LoopOptsMode)+0x144 (loopnode.hpp:1112)
> V [libjvm.so+0x96e9fa] PhaseIdealLoop::optimize(PhaseIterGVN&, LoopOptsMode)+0x4a (loopnode.hpp:1191)
> V [libjvm.so+0x960394] Compile::Optimize()+0x98c (compile.cpp:2357)
> 
> 
> I saw this during the development of [JDK-8318016](https://bugs.openjdk.org/browse/JDK-8318016), which re-uses the node limit check for memory limit checks, so those code paths are executed more often.
> 
> The patch adds several bailouts to the C2 code. I tried to follow and close all paths that call the node limit check. With this patch above assertion and other crashes and assertions disappear.

This pull request has now been integrated.

Changeset: 31ef400f
Author:    Thomas Stuefe <stuefe at openjdk.org>
URL:       https://git.openjdk.org/jdk/commit/31ef400f31b1fd9a861870cce69f6d3eec72aa46
Stats:     26 lines in 8 files changed: 22 ins; 0 del; 4 mod

8318183: C2: VM may crash after hitting node limit

Reviewed-by: kvn, thartmann

-------------

PR: https://git.openjdk.org/jdk/pull/16205