RFR: 8357781: Deep recursion in PhaseCFG::set_next_call leads to stack overflow [v2]

[Marc Chevalier]

On Tue, 27 May 2025 09:33:32 GMT, Tobias Hartmann <thartmann at openjdk.org> wrote:

>> Marc Chevalier has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Address comments
>
> test/hotspot/jtreg/compiler/c2/StackOverflowInSetNextCall.java line 30:
> 
>> 28:  * @summary Triggered a stack overflow in PhaseCFG::set_next_call due to a legitimately big (mostly deep and not wide) graph.
>> 29:  *
>> 30:  * @run main/othervm -Xcomp -XX:LoopUnrollLimit=8192 -XX:CompileCommand=compileonly,StackOverflowInSetNextCall::test StackOverflowInSetNextCall
> 
> What happens if we set the `LoopUnrollLimit` to its max value, i.e. `max_jint / 4`?

It doesn't crash because the loops that could be unrolled are already unrolled as much as possible with this example.

But what if I replace the `100` with `1000`, then it takes longer, but at some point, the node budget is exceeded and it ends well.

But what if I increase the node budget like crazy (then NodeLimitFudgeFactor is too small, but I can increase that as well). Then it takes 100% CPU for a while until it crashes because it exhausts the memory limit (and in my experiments, we crash even after this, later in code generation).

I could raise the memory limit, but it will be the same: either eventually it works because I can fit all the nodes and everything in memory (including my worklist), or I'll explode the limit and get terminated eventually. But that seems orthogonal to this problem: if you use options that creates many nodes, the memory might be too limited.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/25448#discussion_r2108764834