RFR: 6960970: Debugger very slow during stepping [v5]

[Serguei Spitsyn]

On Wed, 26 Nov 2025 05:05:35 GMT, Serguei Spitsyn <sspitsyn at openjdk.org> wrote:

>> This change fixes a long standing performance issue related to the debugger single stepping that is using JVMTI `FramePop` events as a part of step over handling. The performance issue is that the target thread continues its execution in very slow `interp-only` mode in a context of frame marked for `FramePop` notification with the JVMTI `NotifyFramePop`. It includes other method calls recursively upon a return from the frame.
>> 
>> This fix is to avoid enforcing the `interp-only` execution mode for threads when `FramePop` events are enabled with the JVMTI `SetEventNotificationMode()`. Instead, the target frame has been deoptimized and kept interpreted by disabling `OSR` optimization by the function `InterpreterRuntime::frequency_counter_overflow_inner()`. (Big thanks to @fisk for this suggestion!) Additionally, some tweaks are applied in several places where the `java_thread->is_interp_only_mode()` is checked.
>> The other details will be provided in the first PR request comment.
>> It is considered to file a SCR for this update a `FramePop` events do not enforce the `interp-only` mode for a target thread anymore which might break some expectations (the behavior has been changed).
>> 
>> Testing:
>>  - test `serviceability/jvmti/vthread/ThreadStateTest` was updated to provide some extra test coverage
>>  - submitted mach5 tiers 1-6
>
> Serguei Spitsyn has updated the pull request incrementally with one additional commit since the last revision:
> 
>   review: renamed two functions; FRAME_POP_BIT removed from INTERP_EVENT_BITS

We had a walk through the changes with @alexmenkov , @plummercj and @lmesnik and identified several issues:
 - New function `JvmtiExport::has_frame_pop_for_top_frame()` needs a minor performance tweak
 - The changes in `post_method_exit()` are not as precise and correct as needed:
   - call to `get_jvmti_thread_state()` may create a `JvmtiThreadState` object in a case when it is not needed
   - there can be some unreasonable performance overhead
   - there is a concern about possible incorrect handling of the `cur_stack_depth` (need to double check)
- Need to remove the `FRAME_POP_BIT` from the`INTERP_EVENT_BITS` bit mask (**DONE**)
- Decided to rename a couple of new functions (**DONE**):
   - s/`check_and_clear_vthread_pending_deopts`/`clear_vthread_pending_deopts`/g
   - s/`get_vthread_pending_deopts`/`vthread_pending_deopts`/g
- Need some additional test coverage:
   - for multiple `FramePop` requests handled by `process_vthread_pending_deopts()`
   - a test showing a performance improvement with this PR update

-------------

PR Comment: https://git.openjdk.org/jdk/pull/28407#issuecomment-3579240863