RFR: JDK-8318183: C2: VM may crash after hitting node limit

[Thomas Stuefe]

On Tue, 17 Oct 2023 10:57:57 GMT, Tobias Hartmann <thartmann at openjdk.org> wrote:

>> When processing the node limit, as well as when bailing out of compilation for other reasons, we run `Compiler::record_failure()` which sets the root node of Compile to null.
>> 
>> We then should stop any further work that may use Compile::_root. That does not always happen. The results are crashes or asserts when the node limit check hit right at the wrong spot. One example for a follow-up assertion after the node limit hit:
>> 
>> 
>> # Internal Error (/shared/projects/openjdk/jdk-jdk/source/src/hotspot/share/opto/loopnode.hpp:623), pid=761308, tid=761322
>> # assert(_head != nullptr) failed: precond
>> 
>> at
>> 
>> Stack: [0x00007fa068281000,0x00007fa068382000], sp=0x00007fa06837cd40, free space=1007k
>> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
>> V [libjvm.so+0x114ad1c] IdealLoopTree::IdealLoopTree(PhaseIdealLoop*, Node*, Node*)+0x14c (loopnode.hpp:623)
>> V [libjvm.so+0x11412bf] PhaseIdealLoop::build_and_optimize()+0x237 (loopnode.cpp:4325)
>> V [libjvm.so+0x96e7ac] PhaseIdealLoop::PhaseIdealLoop(PhaseIterGVN&, LoopOptsMode)+0x144 (loopnode.hpp:1112)
>> V [libjvm.so+0x96e9fa] PhaseIdealLoop::optimize(PhaseIterGVN&, LoopOptsMode)+0x4a (loopnode.hpp:1191)
>> V [libjvm.so+0x960394] Compile::Optimize()+0x98c (compile.cpp:2357)
>> 
>> 
>> I saw this during the development of [JDK-8318016](https://bugs.openjdk.org/browse/JDK-8318016), which re-uses the node limit check for memory limit checks, so those code paths are executed more often.
>> 
>> The patch adds several bailouts to the C2 code. I tried to follow and close all paths that call the node limit check. With this patch above assertion and other crashes and assertions disappear.
>
> src/hotspot/share/opto/parse1.cpp line 1578:
> 
>> 1576: 
>> 1577:     do_one_bytecode();
>> 1578:     if (C->failing()) return;
> 
> I think you can simply call `failing()` here and potentially at other places as well.

Okay. Have not found any other places where that is possible though.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/16205#discussion_r1362026463