RFR: 6960970: Debugger very slow during stepping [v2]

[Jean-Noël Rouvignac]

On Thu, 20 Nov 2025 04:45:24 GMT, Serguei Spitsyn <sspitsyn at openjdk.org> wrote:

>> This change fixes a long standing performance issue related to the debugger single stepping that is using JVMTI `FramePop` events as a part of step over handling. The performance issue is that the target thread continues its execution in very slow `interp-only` mode in a context of frame marked for `FramePop` notification with the JVMTI `NotifyFramePop`. It includes other method calls recursively upon a return from the frame.
>> 
>> This fix is to avoid enabling the `interp-only` mode for threads when `FramePop` events are enabled with JVMTI `SetEventNotificationMode`. Instead, the target frame has been deoptimized and kept interpreted by disabling `OSR` in the function `InterpreterRuntime::frequency_counter_overflow_inner()`. Additionally, some tweaks are applied in several places where the `java_thread->is_interp_only_mode()` is checked.
>> The other details will be provided in the first PR request comment.
>> It is considered to file a SCR for this update a `FramePop` events do not enforce the `interp-only` mode for a target thread anymore which might break some expectations (the behavior has been changed).
>> 
>> Testing:
>>  - test `serviceability/jvmti/vthread/ThreadStateTest` was updated to provide some extra test coverage
>>  - submitted mach5 tiers 1-6
>
> Serguei Spitsyn has updated the pull request incrementally with one additional commit since the last revision:
> 
>   cleanup: removed an old code fragment in frame.cpp

test/jdk/com/sun/jdi/EATests.java line 3068:

> 3066:         // frame[4]: EATestsTarget.main(java.lang.String[])
> 3067: 
> 3068:         env.stepOverLine(thread); // needed to keep target thread interp-only, so dontinline_brkpt_iret is not inligned

inligned => inlined

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/28407#discussion_r2548831869