About using SetPriorityClass to reduce reaching safepoint duration

[dmytro sheyko]

Hi,

During gc log analysis of our application I noticed that sometimes we have
abnormally long safepoint pauses. Herewith a noticeable share of the time
is spent on reaching safepoint, not at safepoint itself. These pauses
correlate with the periods of time when CPU is highly loaded. We have
several mostly independent java applications running on the same host. Each
of them has its own thread pool with default parameters (i.e. they utilize
all or almost all available processors). This works fine as long as they
are not loaded simultaneously. But when it happens that they had to do
cpu-intensive work at the same time, there are some noticeable pauses. It
looks like application threads can't reach safepoint promptly because the
threads from the other applications just do not let them to do so, by
consuming CPU.

And here comes an idea: how about to temporarily increase the priority of
application threads to let them reach safepoint as soon as possible and
then decrease back to normal once they reached it or even VM operation
completed? On Windows it can be done by using SetPriorityClass. One system
call changes priority class for whole process (and hence the priorities of
all threads at once). On the other OSes maybe there are something similar
as well.

I made some tests and it seems that indeed SetPriorityClass can keep
"reaching safepoint" pauses within bounds. It makes the pauses a bit longer
when cpu-intensive application running alone (i.e. 50% percentile is
increased approximately from 75us to 100us). However the benefit becomes
evident if there are at least 2 the same cpu-intensive application running.
(In this case 50% percentile is decreased from 125us to 100us and 90% -
from 40millis to 140us). Maybe it makes some sense to add an experimental
VM option that would boost app threads to reach safepoint.

Here is some charts that demonstrate how SetPriorityClass affects "reaching
safepoint" pauses
https://drive.google.com/file/d/1xfdyI6JPs3Qud4s5WwdUnpalj55sO_O-/view

Sheets are named as follows:
COMP_5_G1 - means comparison, 5 simultaneous applications with G1 garbage
collector
COMP_4_PS - means comparison, 4 simultaneous applications with ParallelGC
TEST_3_G1_02 - raw data: all pauses, 3 simultaneous applications with G1GC,
using jvm 02-boostToAboveNormal

00-benchmark - clean jvm, i.e. without any changes
01-dontBoost - there are some changes but PriorityClass remains
NORMAL_PRIORITY_CLASS. It behaves similar as 00-benchmark
02-boostToAboveNormal - there are changes that increase PriorityClass to
ABOVE_NORMAL_PRIORITY_CLASS and then decrease back to NORMAL_PRIORITY_CLASS
03-boostToHigh - there are changes that increase PriorityClass to
HIGH_PRIORITY_CLASS and then decrease back to NORMAL_PRIORITY_CLASS. It
works similar as 02-boostToAboveNormal, so it seems no reason to raise
PriorityClass to HIGH, ABOVE_NORMAL is enough.

Changes were following:

diff --git a/src/hotspot/share/runtime/safepoint.cpp
b/src/hotspot/share/runtime/safepoint.cpp
index c37e8219f3..b282fa191c 100644
--- a/src/hotspot/share/runtime/safepoint.cpp
+++ b/src/hotspot/share/runtime/safepoint.cpp
@@ -334,6 +334,35 @@ void SafepointSynchronize::arm_safepoint() {
   OrderAccess::fence(); // storestore|storeload, global state -> local
state
 }

+
+class BoostProcess {
+    DWORD dwPrevPriorityClass;
+public:
+    enum class Level {
+        Normal, AboveNormal, High
+    };
+    BoostProcess(): dwPrevPriorityClass(0) {}
+
+    void enter(Level level) {
+        if (level != Level::Normal) {
+            HANDLE hCurrentProcess = GetCurrentProcess();
+            DWORD dwCurrPriorityClass = GetPriorityClass(hCurrentProcess);
+
+            if (dwCurrPriorityClass == NORMAL_PRIORITY_CLASS) {
+                if (SetPriorityClass(hCurrentProcess, level == Level::High
? HIGH_PRIORITY_CLASS : ABOVE_NORMAL_PRIORITY_CLASS)) {
+                    dwPrevPriorityClass = dwCurrPriorityClass;
+                }
+            }
+        }
+    }
+
+    void leave() {
+        if (dwPrevPriorityClass != 0) {
+            SetPriorityClass(GetCurrentProcess(), dwPrevPriorityClass);
+        }
+    }
+};
+
 // Roll all threads forward to a safepoint and suspend them all
 void SafepointSynchronize::begin() {
   assert(Thread::current()->is_VM_thread(), "Only VM thread may execute a
safepoint");
@@ -341,6 +370,9 @@ void SafepointSynchronize::begin() {
   EventSafepointBegin begin_event;
   SafepointTracing::begin(VMThread::vm_op_type());

+  BoostProcess boostProcess;
+  boostProcess.enter(BoostProcess::Level::AboveNormal);
+
   Universe::heap()->safepoint_synchronize_begin();

   // By getting the Threads_lock, we assure that no threads are about to
start or
@@ -424,6 +456,7 @@ void SafepointSynchronize::begin() {
   post_safepoint_cleanup_event(cleanup_event, _safepoint_id);

   post_safepoint_begin_event(begin_event, _safepoint_id, nof_threads,
_current_jni_active_count);
+  boostProcess.leave();
   SafepointTracing::cleanup();
 }

Regards,
Dmytro