RFR: 8357781: Deep recursion in PhaseCFG::set_next_call leads to stack overflow [v2]

[Tobias Hartmann]

On Tue, 27 May 2025 09:57:54 GMT, Marc Chevalier <mchevalier at openjdk.org> wrote:

>> test/hotspot/jtreg/compiler/c2/StackOverflowInSetNextCall.java line 30:
>> 
>>> 28:  * @summary Triggered a stack overflow in PhaseCFG::set_next_call due to a legitimately big (mostly deep and not wide) graph.
>>> 29:  *
>>> 30:  * @run main/othervm -Xcomp -XX:LoopUnrollLimit=8192 -XX:CompileCommand=compileonly,StackOverflowInSetNextCall::test StackOverflowInSetNextCall
>> 
>> What happens if we set the `LoopUnrollLimit` to its max value, i.e. `max_jint / 4`?
>
> It doesn't crash because the loops that could be unrolled are already unrolled as much as possible with this example.
> 
> But what if I replace the `100` with `1000`, then it takes longer, but at some point, the node budget is exceeded and it ends well.
> 
> But what if I increase the node budget like crazy (then NodeLimitFudgeFactor is too small, but I can increase that as well). Then it takes 100% CPU for a while until it crashes because it exhausts the memory limit (and in my experiments, we crash even after this, later in code generation).
> 
> I could raise the memory limit, but it will be the same: either eventually it works because I can fit all the nodes and everything in memory (including my worklist), or I'll explode the limit and get terminated eventually. But that seems orthogonal to this problem: if you use options that creates many nodes, the memory might be too limited.

Right, I think that's expected. Thanks for checking!

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/25448#discussion_r2108777197