RFR: 8374307: Fix deoptimization storm caused by Action_none in GraphKit::uncommon_trap [v3]

[Vladimir Ivanov]

On Wed, 7 Jan 2026 17:28:14 GMT, Boris Ulasevich <bulasevich at openjdk.org> wrote:

>> We observed a deoptimization storm caused by GraphKit::uncommon_trap generator logic. GraphKit::uncommon_trap considers the too_many_recompiles metric. If the threshold is overflowed, it replaces Deoptimization::Action_reinterpret with Deoptimization::Action_none (see code snippet below).
>> 
>> This replacement changes the uncommon_trap logic: once execution hits a trap, the VM performs deoptimization but does not recompile the method anymore. In an "unlucky" case, when the code part calling this uncommon_trap becomes frequent, a deoptimization storm occurs (thousands of deoptimizations per second) causing a significant performance drop.
>> 
>> The original problematic method, which triggered repeated recompilations, is a high-performance compressed binary serialization algorithm with heavy use of conditional branches driven by bitmasks. See a standalone [synthetic benchmark](https://bugs.openjdk.org/secure/attachment/118045/UnstableIf.java) to reproduce the issue.
>> 
>> The issue arises when the method overcomes a global recompilation threshold before stabilizing specific trap counters.
>> 
>> Current thresholds:
>> - Recompilation Limit (too_many_recompiles):
>>   Condition: decompile_count() >= (PerMethodRecompilationCutoff / 2) + 1
>>   Default: 201 (derived from default PerMethodRecompilationCutoff = 400).
>> - Specific Trap Limits (too_many_traps):
>>   Checks if the trap count for a specific reason exceeds:
>>   PerMethodTrapLimit (Default: 100) - for Reason_unstable_if, Reason_unstable_fused_if, etc.
>>   PerMethodSpecTrapLimit (Default: 5000) - for Reason_speculate_class_check, Reason_speculate_null_check, etc.
>> 
>> With the gived defaults, if the only reason for the method recompilation is unstable_if, the system stabilizes after 100 traps (PerMethodTrapLimit). However, if the method experiences traps and recompilations for different reasons, the total number of recompilations can exceed 200 before hitting the limit for unstable_if traps. This triggers Action_none and causes the deopt storm.
>> 
>> The proposal is a minimal change in GraphKit::uncommon_trap: apply the same `too_many_recompiles` threshold inside `Parse::path_is_suitable_for_uncommon_trap` - this ensures that on the final recompilation C2 gets a hint not to speculate on untaken branches anymore.
>> 
>> As an alternative solution, we can revisit GraphKit::uncommon_trap. This "Temporary fix" has persisted in the codebase for 17 years, so it is probably time to change it as well. Any comments are we...
>
> Boris Ulasevich has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. The pull request contains one new commit since the last revision:
> 
>   using too_many_traps_or_recompiles. adding DeoptStorm jtreg test

I believe all places where an uncommon trap with `Action_reinterpret` guarded by `too_many_traps` is susceptible to the very same problem.

The culprit seems to be the discrepancy between `too_many_traps` and `too_many_recompiles` where many places where uncommon traps are inserted are guarded by `too_many_traps` while `GraphKit::uncommon_trap()` checks specifically for `too_many_recompiles`.

As the bug demonstrates, disabling recompilation while keeping the uncommon trap in place (substituting `Action_maybe_recompile`/`Action_maybe_recompile` with `Action_none`) can induce a lot of overhead. So, a better strategy is to avoid an uncommon trap in the first place rather than letting it to degenerate into `Action_none` and, also, assert whenever the situation occurs at runtime.

Speaking of the proposed fix, my concern is that it addresses only one particular instance of the problem. Can we do better and fix similar bugs all at once? That would require aligning `too_many_traps` and `too_many_recompiles` use sites.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/28966#issuecomment-3720372003