RFR: 8355488: Add stress mode for C2 loop peeling [v4]

[Tobias Hartmann]

On Fri, 16 May 2025 07:58:36 GMT, Marc Chevalier <mchevalier at openjdk.org> wrote:

>> Adding a `StressLoopPeeling` dev flag that randomize peeling.
>> 
>> ## Semantics
>> 
>> For now, the direction I've taken is to randomly take a decision in case of peeling, otherwise, rely on existing heuristics.
>> 
>> This requires to distinguish two things:
>> - not inlining because it's not legal: see for instance
>>   ```cpp
>>   assert(cl->trip_count() > 0, "peeling a fully unrolled loop");
>>   ```
>>   in `PhaseIdealLoop::do_peeling`
>> - not inlining because it doesn't seem profitable.
>> 
>> Peeling loops without a good reason (not containing an exiting `If` whose condition is not a member of the loop) but without a concrete way to forbid it should always be allowed. Let's stress it!
>> 
>> Peeling too many times is not a great idea either. It uses a lot of memory, of nodes... Also, it may prevent other optimisations from kicking in. And what about interaction with future stress flags? Let's limit peeling: we give a fixed number of opportunities to peel before we give up on peeling for good. That is not the same as limiting the amount of peeling we do. Indeed, if we bound the number of times we say "yes, please, peel" given enough requests, we will always reach the bound. If we limit the number of requests, we have a more evenly distributed amount of peeling, between 0 and the bound.
>> 
>> I've tried without the bound: I couldn't find any bug without the bound that would not reproduce with the bound. It only save some legitimate memory problems. Without a bound on the number of peeling opportunities, hotspot eats a lot of memory, but all the allocations seems reasonable: it just seems we ask too much. We could limit the number of nodes, to prevent peeling before we reach the memory limit, but that would also hinder other optimizations and (future) stress flags.
>> 
>> 
>> 
>> ## The Flag
>> 
>> The flag is very specialized, unlike a `StressLoopOpts` would be. My idea so far is "let's see". My idea is that it's good to be able to enable stress optimizations selectively, and have a flag like `StressLoopOpts` that would turn them all: we could use the general one in testing, and the finer-grain ones when debugging. A reason for that is that I don't see a real use-case for stressing some features but not others (which would make the number of combinations explode): having (for instance) `+StressLoopUnrolling +StressLoopPeeling` would sometimes behave like `+StressLoopUnrolling -StressLoopPeeling`, and so it's not very useful to test the latter.
>> 
>> But once again: let'...
>
> Marc Chevalier has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Address comments

Thanks for making these changes, looks good to me.

-------------

Marked as reviewed by thartmann (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/25140#pullrequestreview-2845907126