RFR: 8357781: Deep recursion in PhaseCFG::set_next_call leads to stack overflow [v2]

[Tobias Hartmann]

On Tue, 27 May 2025 09:01:17 GMT, Marc Chevalier <mchevalier at openjdk.org> wrote:

>> There is nothing very wrong here:
>> - the graph is not broken
>> - the algorithm is correct
>> 
>> It just happens that the graph is very deep, it has a very long, narrow chain of nodes because of crazy unrolling, because of `LoopUnrollLimit=8192`. This depth simply makes the algorithm recurse deeper than the stack size allows. This kind of graph shape is not quite trivial to reproduce. The proposed reproducer is very easy to change into a non-reproducer, with many kinds of change, even that seem harmless to me.
>> 
>> The fix is also pretty direct: let's change the recursive traversal, with a worklist-based iterative one. It's not as elegant, but it doesn't overflow.
>> 
>> How sad stacks are still so bounded...
>
> Marc Chevalier has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Address comments

Changes requested by thartmann (Reviewer).

test/hotspot/jtreg/compiler/c2/StackOverflowInSetNextCall.java line 27:

> 25: /**
> 26:  * @test
> 27:  * @bug 8324837

Suggestion:

 * @bug 8357781

test/hotspot/jtreg/compiler/c2/StackOverflowInSetNextCall.java line 30:

> 28:  * @summary Triggered a stack overflow in PhaseCFG::set_next_call due to a legitimately big (mostly deep and not wide) graph.
> 29:  *
> 30:  * @run main/othervm -Xcomp -XX:LoopUnrollLimit=8192 -XX:CompileCommand=compileonly,StackOverflowInSetNextCall::test StackOverflowInSetNextCall

What happens if we set the `LoopUnrollLimit` to its max value, i.e. `max_jint / 4`?

-------------

PR Review: https://git.openjdk.org/jdk/pull/25448#pullrequestreview-2870374384
PR Review Comment: https://git.openjdk.org/jdk/pull/25448#discussion_r2108714363
PR Review Comment: https://git.openjdk.org/jdk/pull/25448#discussion_r2108713495