RFR: 6960970: Debugger very slow during stepping [v2]

[Serguei Spitsyn]

On Thu, 20 Nov 2025 04:45:24 GMT, Serguei Spitsyn <sspitsyn at openjdk.org> wrote:

>> This change fixes a long standing performance issue related to the debugger single stepping that is using JVMTI `FramePop` events as a part of step over handling. The performance issue is that the target thread continues its execution in very slow `interp-only` mode in a context of frame marked for `FramePop` notification with the JVMTI `NotifyFramePop`. It includes other method calls recursively upon a return from the frame.
>> 
>> This fix is to avoid enabling the `interp-only` mode for threads when `FramePop` events are enabled with JVMTI `SetEventNotificationMode`. Instead, the target frame has been deoptimized and kept interpreted by disabling `OSR` in the function `InterpreterRuntime::frequency_counter_overflow_inner()`. Additionally, some tweaks are applied in several places where the `java_thread->is_interp_only_mode()` is checked.
>> The other details will be provided in the first PR request comment.
>> It is considered to file a SCR for this update a `FramePop` events do not enforce the `interp-only` mode for a target thread anymore which might break some expectations (the behavior has been changed).
>> 
>> Testing:
>>  - test `serviceability/jvmti/vthread/ThreadStateTest` was updated to provide some extra test coverage
>>  - submitted mach5 tiers 1-6
>
> Serguei Spitsyn has updated the pull request incrementally with one additional commit since the last revision:
> 
>   cleanup: removed an old code fragment in frame.cpp

Change details:
 - platform-dependent `interp_masm*.cpp` and `zero/bytecodeInterpreter.cpp`:
    removed `interp-only` mode check before call to `InterpreterRuntime::post_method_exit()`
 - `interpreterRuntime.cpp`: disallow OSR for frames marked for `FramePop` event notification
 - `jvmtiEnvBase.cpp`:
   - added frame deoptimization logic to JVMTI `NotifyFramePop` implementation
   - added `check_and_clear_vthread_pending_deopts()` calls to JVMTI `ClearAllFramePops` implementation
 - `jvmtiEnvThreadState.cpp`: removed undesired check `!jvmti_thread_state()->is_interp_only_mode()` from `JvmtiEnvThreadState::is_frame_pop()` function
 - `jvmtiEventController.cpp`:
    - `JvmtiEventControllerPrivate::recompute_thread_enabled()`: removed `ets->has_frame_pops()` check when the `interp-only` node is enabled
 - `jvmtiExport.?pp`: 
   - introduced new function `has_frame_pop_for_top_frame()` used to disallow `OSR` for target frame
   - `JvmtiExport::post_method_exit()`: replaced `state->is_interp_only_mode()` check with `current_frame.is_interpreted_frame()`
 - `jvmtiThreadState.?pp`:
   - implementation of `process_vthread_pending_deopts()`logics
   - `JvmtiThreadState::process_pending_interp_only()`: added `MutexLocker mu(JvmtiThreadState_lock)`
   - `JvmtiThreadState::update_for_pop_top_frame()`: removed `is_interp_only_mode()` check
 - `frame.cpp`:
    - fix provided by Patricio (Thank you!): an assert specific to continuation heap frames is fixed
 - `serviceability/jvmti/events/NotifyFramePopStressTest`: minor tweak to help with debugging
 - `serviceability/jvmti/vthread/ThreadStateTest`: added code to provide extra test coverage from JVMTI `FramePop` events

-------------

PR Comment: https://git.openjdk.org/jdk/pull/28407#issuecomment-3555778224