RFR: 8370519: C2: Hit MemLimit when running with +VerifyLoopOptimizations [v3]

[Benoît Maillard]

On Tue, 2 Dec 2025 11:21:05 GMT, Roland Westrelin <roland at openjdk.org> wrote:

>> For this failure memory stats are:
>> 
>> 
>> Total Usage: 1095525816 
>>     --- Arena Usage by Arena Type and compilation phase, at arena usage peak of 1095525816 ---
>>         Phase                         Total        ra      node      comp      type    states   reglive  regsplit   regmask superword     cienv        ha     other
>>         none                        5976032    331560   5402064    197512     33712     10200         0         0       984         0         0         0         0
>>         parse                       2716464     65456   1145480    196408   1112752         0         0         0         0         0    196368         0         0
>>         optimizer                     98184         0     32728         0     65456         0         0         0         0         0         0         0         0
>>         connectionGraph               32728         0         0     32728         0         0         0         0         0         0         0         0         0
>>         iterGVN                       32728         0     32728         0         0         0         0         0         0         0         0         0         0
>>         idealLoop                 918189632         0  38687056 872824784    392776         0         0         0         0         0   6285016         0         0
>>         idealLoopVerify             2228144         0         0   2228144         0         0         0         0         0         0         0         0         0
>>         macroExpand                   32728         0     32728         0         0         0         0         0         0         0         0         0         0
>>         graphReshape                  32728         0     32728         0         0         0         0         0         0         0         0         0         0
>>         matcher                    20135944   3369848   9033208   7536400     65456    131032         0         0         0         0         0         0         0
>>         postselect_cleanup           294872    294872         0         0         0         0         0         0         0         0         0         0         0
>>         scheduler                    752944    196488    556456         0         0         0         0         0         0         0         0         0         0
>>         regalloc                     388736    388736         0         0         0         0         0         0         0         0         0         0         0
>>        ...
>
> Roland Westrelin has updated the pull request incrementally with two additional commits since the last revision:
> 
>  - review
>  - review

test/hotspot/jtreg/compiler/c2/TestVerifyLoopOptimizationsHighMemUsage.java line 30:

> 28:  * @summary C2: Hit MemLimit when running with +VerifyLoopOptimizations
> 29:  * @run main/othervm -XX:CompileCommand=compileonly,*TestVerifyLoopOptimizationsHighMemUsage*::* -XX:-TieredCompilation -Xbatch
> 30:  *                   -XX:+UnlockDiagnosticVMOptions -XX:+StressLoopPeeling -XX:+VerifyLoopOptimizations

I think you'll also need `-XX:+IgnoreUnrecognizedVMOptions` (it is causing issues in Github actions).
Suggestion:

 *                   -XX:+UnlockDiagnosticVMOptions -XX:+IgnoreUnrecognizedVMOptions
 *                   -XX:+StressLoopPeeling -XX:+VerifyLoopOptimizations

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/28581#discussion_r2605877498