RFR: JDK-8318183: C2: VM may crash after hitting node limit [v2]

[Vladimir Kozlov]

On Tue, 17 Oct 2023 12:46:03 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

>> When processing the node limit, as well as when bailing out of compilation for other reasons, we run `Compiler::record_failure()` which sets the root node of Compile to null.
>> 
>> We then should stop any further work that may use Compile::_root. That does not always happen. The results are crashes or asserts when the node limit check hit right at the wrong spot. One example for a follow-up assertion after the node limit hit:
>> 
>> 
>> # Internal Error (/shared/projects/openjdk/jdk-jdk/source/src/hotspot/share/opto/loopnode.hpp:623), pid=761308, tid=761322
>> # assert(_head != nullptr) failed: precond
>> 
>> at
>> 
>> Stack: [0x00007fa068281000,0x00007fa068382000], sp=0x00007fa06837cd40, free space=1007k
>> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
>> V [libjvm.so+0x114ad1c] IdealLoopTree::IdealLoopTree(PhaseIdealLoop*, Node*, Node*)+0x14c (loopnode.hpp:623)
>> V [libjvm.so+0x11412bf] PhaseIdealLoop::build_and_optimize()+0x237 (loopnode.cpp:4325)
>> V [libjvm.so+0x96e7ac] PhaseIdealLoop::PhaseIdealLoop(PhaseIterGVN&, LoopOptsMode)+0x144 (loopnode.hpp:1112)
>> V [libjvm.so+0x96e9fa] PhaseIdealLoop::optimize(PhaseIterGVN&, LoopOptsMode)+0x4a (loopnode.hpp:1191)
>> V [libjvm.so+0x960394] Compile::Optimize()+0x98c (compile.cpp:2357)
>> 
>> 
>> I saw this during the development of [JDK-8318016](https://bugs.openjdk.org/browse/JDK-8318016), which re-uses the node limit check for memory limit checks, so those code paths are executed more often.
>> 
>> The patch adds several bailouts to the C2 code. I tried to follow and close all paths that call the node limit check. With this patch above assertion and other crashes and assertions disappear.
>
> Thomas Stuefe has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains four additional commits since the last revision:
> 
>  - Fix GraphKit dtor
>  - Feedback Tobias
>  - Merge branch 'master' into c2-missing-bailouts
>  - add bailouts

Looks good.

-------------

Marked as reviewed by kvn (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/16205#pullrequestreview-1683082532