RFR: 8370519: C2: Hit MemLimit when running with +VerifyLoopOptimizations

[Benoît Maillard]

On Mon, 1 Dec 2025 15:40:00 GMT, Roland Westrelin <roland at openjdk.org> wrote:

> For this failure memory stats are:
> 
> 
> Total Usage: 1095525816 
>     --- Arena Usage by Arena Type and compilation phase, at arena usage peak of 1095525816 ---
>         Phase                         Total        ra      node      comp      type    states   reglive  regsplit   regmask superword     cienv        ha     other
>         none                        5976032    331560   5402064    197512     33712     10200         0         0       984         0         0         0         0
>         parse                       2716464     65456   1145480    196408   1112752         0         0         0         0         0    196368         0         0
>         optimizer                     98184         0     32728         0     65456         0         0         0         0         0         0         0         0
>         connectionGraph               32728         0         0     32728         0         0         0         0         0         0         0         0         0
>         iterGVN                       32728         0     32728         0         0         0         0         0         0         0         0         0         0
>         idealLoop                 918189632         0  38687056 872824784    392776         0         0         0         0         0   6285016         0         0
>         idealLoopVerify             2228144         0         0   2228144         0         0         0         0         0         0         0         0         0
>         macroExpand                   32728         0     32728         0         0         0         0         0         0         0         0         0         0
>         graphReshape                  32728         0     32728         0         0         0         0         0         0         0         0         0         0
>         matcher                    20135944   3369848   9033208   7536400     65456    131032         0         0         0         0         0         0         0
>         postselect_cleanup           294872    294872         0         0         0         0         0         0         0         0         0         0         0
>         scheduler                    752944    196488    556456         0         0         0         0         0         0         0         0         0         0
>         regalloc                     388736    388736         0         0         0         0         0         0         0         0         0         0         0
>         ctorChaitin                  160032    ...

Thanks for fixing this @rwestrel, I agree with the fix. I noticed that this could be a problem while working on [JDK-8366990](https://bugs.openjdk.org/browse/JDK-8366990), but there was no reproducer at the time.

src/hotspot/share/opto/compile.cpp line 656:

> 654:       _stress_seed(0),
> 655:       _comp_arena(mtCompiler, Arena::Tag::tag_comp),
> 656:       _idealloop_arena(mtCompiler, Arena::Tag::tag_idealloop),

To keep the naming consistent with other mentions of `IdealLoop` in variable/field names (such as `_phase_verify_ideal_loop`), I would name this `_ideal_loop_arena`. This will make it easier to find in a code editor. Feel free to ignore if you disagree

test/hotspot/jtreg/compiler/c2/TestVerifyLoopOptimizationsHighMemUsage.java line 28:

> 26:  * @bug 8370519
> 27:  * @summary C2: Hit MemLimit when running with +VerifyLoopOptimizations
> 28:  * @run main/othervm -XX:CompileCommand=compileonly,*TestVerifyLoopOptimizationsHighMemUsage*::* -XX:-TieredCompilation -Xbatch

Out of curiosity, have you try reducing the test with `creduce`? I fixed a similar issue in [JDK-8366990](https://bugs.openjdk.org/browse/JDK-8366990), and initially reviewers were concerned about the long compilation time. I was able to get decent results with `creduce` by using  `-XX:CompileCommand=memlimit`. Not sure if it's worth doing here though.

-------------

PR Review: https://git.openjdk.org/jdk/pull/28581#pullrequestreview-3525878832
PR Review Comment: https://git.openjdk.org/jdk/pull/28581#discussion_r2577760668
PR Review Comment: https://git.openjdk.org/jdk/pull/28581#discussion_r2577804805