RFR: 6960970: Debugger very slow during stepping [v3]

[Alex Menkov]

On Fri, 21 Nov 2025 19:45:04 GMT, Serguei Spitsyn <sspitsyn at openjdk.org> wrote:

>> This change fixes a long standing performance issue related to the debugger single stepping that is using JVMTI `FramePop` events as a part of step over handling. The performance issue is that the target thread continues its execution in very slow `interp-only` mode in a context of frame marked for `FramePop` notification with the JVMTI `NotifyFramePop`. It includes other method calls recursively upon a return from the frame.
>> 
>> This fix is to avoid enforcing the `interp-only` execution mode for threads when `FramePop` events are enabled with the JVMTI `SetEventNotificationMode()`. Instead, the target frame has been deoptimized and kept interpreted by disabling `OSR` optimization by the function `InterpreterRuntime::frequency_counter_overflow_inner()`. (Big thanks to @fisk for this suggestion!) Additionally, some tweaks are applied in several places where the `java_thread->is_interp_only_mode()` is checked.
>> The other details will be provided in the first PR request comment.
>> It is considered to file a SCR for this update a `FramePop` events do not enforce the `interp-only` mode for a target thread anymore which might break some expectations (the behavior has been changed).
>> 
>> Testing:
>>  - test `serviceability/jvmti/vthread/ThreadStateTest` was updated to provide some extra test coverage
>>  - submitted mach5 tiers 1-6
>
> Serguei Spitsyn has updated the pull request incrementally with one additional commit since the last revision:
> 
>   review: fix typo in a EATests.java comment

src/hotspot/share/prims/jvmtiThreadState.cpp line 707:

> 705:   for (int idx = 0; idx  < deopts->length(); idx++) {
> 706:     int frame_number = deopts->at(idx);
> 707:     deopts->remove_at(idx);

The code forward iterates the array removing the entries? it will skip every other element (indexes change after removal)

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/28407#discussion_r2561326190