RFR: 8330969: scalability issue with loaded JVMTI agent

[Serguei Spitsyn]

This is a fix of the following JVMTI scalability issue. A closed benchmark with millions of virtual threads shows 4X overhead when a JVMTI agent has been loaded. For instance, this is observable when an app is executed under control of the Oracle Studio `collect` utility.

The benchmark takes a little bit more than 3 sec without any JVMTI agent:

Total:               in 3045 ms


The benchmark takes more than ~3.2X of the above when executed with the `collect` utility:

Creating experiment database test.1.er (Process ID: 25262) ...
Picked up JAVA_TOOL_OPTIONS: -agentlib:collector
Total:               in 9864 ms


With the fix in place the overhead of a JVMTI agent is around 1.2X:

Creating experiment database test.1.er (Process ID: 26442) ...
Picked up JAVA_TOOL_OPTIONS: -agentlib:collector
Total:               in 3765 ms


The most of the overhead is taken by two functions:
 - `JvmtiVTMSTransitionDisabler::start_VTMS_transition()`
 - `JvmtiVTMSTransitionDisabler::finish_VTMS_transition()`
 
 Oracle Studio Performance Analyzer `err_print utility shows the following performance data for these functions:
 ```
% er_print -viewmode expert -metrics ie.%totalcpu -csingle JvmtiVTMSTransitionDisabler::start_VTMS_transition test.1.er

Attr. Total     Name
CPU
  sec.      %
=============== Callers
42.930  50.06   SharedRuntime::notify_jvmti_vthread_mount(oopDesc*, unsigned char, JavaThread*)
21.505  25.08   JvmtiVTMSTransitionDisabler::VTMS_vthread_end(_jobject*)
21.315  24.86   JvmtiVTMSTransitionDisabler::VTMS_vthread_unmount(_jobject*, bool)

=============== Stack Fragment
81.407  94.94   JvmtiVTMSTransitionDisabler::start_VTMS_transition(_jobject*, bool)

=============== Callees
 4.083   4.76   java_lang_Thread::set_is_in_VTMS_transition(oopDesc*, bool)
 0.140   0.16   __tls_get_addr
 0.120   0.14   JNIHandles::resolve_external_guard(_jobject*)



% er_print -viewmode expert -metrics ie.%totalcpu -csingle JvmtiVTMSTransitionDisabler::finish_VTMS_transition test.1.er

Attr. Total     Name
CPU
  sec.      %
=============== Callers
47.363  52.59   SharedRuntime::notify_jvmti_vthread_unmount(oopDesc*, unsigned char, JavaThread*)
21.355  23.71   JvmtiVTMSTransitionDisabler::VTMS_vthread_mount(_jobject*, bool)
21.345  23.70   JvmtiVTMSTransitionDisabler::VTMS_vthread_start(_jobject*)

=============== Stack Fragment
64.145  71.22   JvmtiVTMSTransitionDisabler::finish_VTMS_transition(_jobject*, bool)

=============== Callees
25.288  28.08   java_lang_Thread::set_is_in_VTMS_transition(oopDesc*, bool)
 0.240   0.27   __tls_get_addr
 0.200   0.22   JavaThread::set_is_in_VTMS_transition(bool)
 0.190   0.21   JNIHandles::resolve_external_guard(_jobject*)


The main source of this overhead (~90% of overhead) is atomic increment and decrement of the global counter `VTMS_transition_count`:
 - `Atomic::inc(&_VTMS_transition_count)`;
 - `Atomic::dec(&_VTMS_transition_count)`;
 

The fix is to replace this global counter with mark bits `_VTMS_transition_mark` distributed over all `JavaThread`'s.
If these lines are commented out or replaced with the distributed thread-local marks the main performance overhead is gone:

% er_print -viewmode expert -metrics ie.%totalcpu -csingle JvmtiVTMSTransitionDisabler::start_VTMS_transition test.2.er

Attr. Total    Name
CPU
 sec.      %
============== Callers
1.801  64.29   SharedRuntime::notify_jvmti_vthread_mount(oopDesc*, unsigned char, JavaThread*)
0.580  20.71   JvmtiVTMSTransitionDisabler::VTMS_vthread_unmount(_jobject*, bool)
0.420  15.00   JvmtiVTMSTransitionDisabler::VTMS_vthread_end(_jobject*)

============== Stack Fragment
0.630  22.50   JvmtiVTMSTransitionDisabler::start_VTMS_transition(_jobject*, bool)

============== Callees
1.931  68.93   java_lang_Thread::set_is_in_VTMS_transition(oopDesc*, bool)
0.220   7.86   __tls_get_addr
0.020   0.71   JNIHandles::resolve_external_guard(_jobject*)



% er_print -viewmode expert -metrics ie.%totalcpu -csingle JvmtiVTMSTransitionDisabler::finish_VTMS_transition test.2.er

Attr. Total    Name
CPU
 sec.      %
============== Callers
1.661  39.15   JvmtiVTMSTransitionDisabler::VTMS_vthread_mount(_jobject*, bool)
1.351  31.84   JvmtiVTMSTransitionDisabler::VTMS_vthread_start(_jobject*)
1.231  29.01   SharedRuntime::notify_jvmti_vthread_unmount(oopDesc*, unsigned char, JavaThread*)

============== Stack Fragment
0.500  11.79   JvmtiVTMSTransitionDisabler::finish_VTMS_transition(_jobject*, bool)

============== Callees
2.972  70.05   java_lang_Thread::set_is_in_VTMS_transition(oopDesc*, bool)
0.350   8.25   JavaThread::set_is_in_VTMS_transition(bool)
0.340   8.02   __tls_get_addr
0.080   1.89   JNIHandles::resolve_external_guard(_jobject*)


The rest of the overhead (~10% of total overhead) is taken by calls to the function `java_lang_Thread::set_is_in_VTMS_transition()`. The plan is to address this in a separate fix. But it is expected to be a little bit more tricky.


Testing:
 - Tested with mach5 tiers 1-6

-------------

Commit messages:
 - 8330969: scalability issue with loaded JVMTI agent

Changes: https://git.openjdk.org/jdk/pull/18937/files
  Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=18937&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8330969
  Stats: 38 lines in 5 files changed: 14 ins; 11 del; 13 mod
  Patch: https://git.openjdk.org/jdk/pull/18937.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/18937/head:pull/18937

PR: https://git.openjdk.org/jdk/pull/18937