About using SetPriorityClass to reduce reaching safepoint duration

[David Holmes]

Hi Dmytro,

On 13/05/2020 3:36 am, dmytro sheyko wrote:
> Hi,
> 
> During gc log analysis of our application I noticed that sometimes we have
> abnormally long safepoint pauses. Herewith a noticeable share of the time
> is spent on reaching safepoint, not at safepoint itself. These pauses
> correlate with the periods of time when CPU is highly loaded. We have
> several mostly independent java applications running on the same host. Each
> of them has its own thread pool with default parameters (i.e. they utilize
> all or almost all available processors). This works fine as long as they
> are not loaded simultaneously. But when it happens that they had to do
> cpu-intensive work at the same time, there are some noticeable pauses. It
> looks like application threads can't reach safepoint promptly because the
> threads from the other applications just do not let them to do so, by
> consuming CPU.

Which JDK version? Do you have data on what the safepoints are i.e. all 
for GC or something else? Which GC?

> And here comes an idea: how about to temporarily increase the priority of
> application threads to let them reach safepoint as soon as possible and
> then decrease back to normal once they reached it or even VM operation
> completed? On Windows it can be done by using SetPriorityClass. One system
> call changes priority class for whole process (and hence the priorities of
> all threads at once). On the other OSes maybe there are something similar
> as well.

Messing with OS level thread priorities is a big "can of worms" and can 
lead to more scheduling aberrations than it cures. If you have an 
overloaded system with multiple JVMs running then you are already in a 
bad place. How do you decide which of those JVMs is most important? If 
you boost the priority of threads trying to reach safepoints then you 
skew the CPU allocation to those JVMs that execute a lot of safepoints 
and have a lot of threads. This may make those applications quite happy 
but are they more important than the other applications on the system? 
And you are not just impacting JVMs but all processes running on the 
machine. You'd need to investigate the performance impact on the system 
as a whole. And if different VMs have overlapping safepoints you just 
end up boosting them all. Even within a JVM the boosted GC threads would 
interfere with the boosted JavaThreads.

Functionality wise, Solaris also supported scheduling classes (but we're 
dropping Solaris support), but I don't think Linux/MacOS have this. And 
other OS require special permissions (actually Windows does too!).

Interesting experiment though.

Cheers,
David

> I made some tests and it seems that indeed SetPriorityClass can keep
> "reaching safepoint" pauses within bounds. It makes the pauses a bit longer
> when cpu-intensive application running alone (i.e. 50% percentile is
> increased approximately from 75us to 100us). However the benefit becomes
> evident if there are at least 2 the same cpu-intensive application running.
> (In this case 50% percentile is decreased from 125us to 100us and 90% -
> from 40millis to 140us). Maybe it makes some sense to add an experimental
> VM option that would boost app threads to reach safepoint.
> 
> Here is some charts that demonstrate how SetPriorityClass affects "reaching
> safepoint" pauses
> https://drive.google.com/file/d/1xfdyI6JPs3Qud4s5WwdUnpalj55sO_O-/view
> 
> Sheets are named as follows:
> COMP_5_G1 - means comparison, 5 simultaneous applications with G1 garbage
> collector
> COMP_4_PS - means comparison, 4 simultaneous applications with ParallelGC
> TEST_3_G1_02 - raw data: all pauses, 3 simultaneous applications with G1GC,
> using jvm 02-boostToAboveNormal
> 
> 00-benchmark - clean jvm, i.e. without any changes
> 01-dontBoost - there are some changes but PriorityClass remains
> NORMAL_PRIORITY_CLASS. It behaves similar as 00-benchmark
> 02-boostToAboveNormal - there are changes that increase PriorityClass to
> ABOVE_NORMAL_PRIORITY_CLASS and then decrease back to NORMAL_PRIORITY_CLASS
> 03-boostToHigh - there are changes that increase PriorityClass to
> HIGH_PRIORITY_CLASS and then decrease back to NORMAL_PRIORITY_CLASS. It
> works similar as 02-boostToAboveNormal, so it seems no reason to raise
> PriorityClass to HIGH, ABOVE_NORMAL is enough.
> 
> Changes were following:
> 
> diff --git a/src/hotspot/share/runtime/safepoint.cpp
> b/src/hotspot/share/runtime/safepoint.cpp
> index c37e8219f3..b282fa191c 100644
> --- a/src/hotspot/share/runtime/safepoint.cpp
> +++ b/src/hotspot/share/runtime/safepoint.cpp
> @@ -334,6 +334,35 @@ void SafepointSynchronize::arm_safepoint() {
>     OrderAccess::fence(); // storestore|storeload, global state -> local
> state
>   }
> 
> +
> +class BoostProcess {
> +    DWORD dwPrevPriorityClass;
> +public:
> +    enum class Level {
> +        Normal, AboveNormal, High
> +    };
> +    BoostProcess(): dwPrevPriorityClass(0) {}
> +
> +    void enter(Level level) {
> +        if (level != Level::Normal) {
> +            HANDLE hCurrentProcess = GetCurrentProcess();
> +            DWORD dwCurrPriorityClass = GetPriorityClass(hCurrentProcess);
> +
> +            if (dwCurrPriorityClass == NORMAL_PRIORITY_CLASS) {
> +                if (SetPriorityClass(hCurrentProcess, level == Level::High
> ? HIGH_PRIORITY_CLASS : ABOVE_NORMAL_PRIORITY_CLASS)) {
> +                    dwPrevPriorityClass = dwCurrPriorityClass;
> +                }
> +            }
> +        }
> +    }
> +
> +    void leave() {
> +        if (dwPrevPriorityClass != 0) {
> +            SetPriorityClass(GetCurrentProcess(), dwPrevPriorityClass);
> +        }
> +    }
> +};
> +
>   // Roll all threads forward to a safepoint and suspend them all
>   void SafepointSynchronize::begin() {
>     assert(Thread::current()->is_VM_thread(), "Only VM thread may execute a
> safepoint");
> @@ -341,6 +370,9 @@ void SafepointSynchronize::begin() {
>     EventSafepointBegin begin_event;
>     SafepointTracing::begin(VMThread::vm_op_type());
> 
> +  BoostProcess boostProcess;
> +  boostProcess.enter(BoostProcess::Level::AboveNormal);
> +
>     Universe::heap()->safepoint_synchronize_begin();
> 
>     // By getting the Threads_lock, we assure that no threads are about to
> start or
> @@ -424,6 +456,7 @@ void SafepointSynchronize::begin() {
>     post_safepoint_cleanup_event(cleanup_event, _safepoint_id);
> 
>     post_safepoint_begin_event(begin_event, _safepoint_id, nof_threads,
> _current_jni_active_count);
> +  boostProcess.leave();
>     SafepointTracing::cleanup();
>   }
> 
> Regards,
> Dmytro
>