SelectorManager threads are discriminated in race to get threads_lock mutex

[David Holmes]

On 30/08/2023 8:56 pm, Kern, Joachim wrote:
> Hi guys,
> 
> The ContinuationFrameTest.java 
> <https://github.com/openjdk/jdk/blob/master/test/jdk/java/net/httpclient/http2/ContinuationFrameTest.java> on AIX fails to end within the gracedelay period.
> 
> The gracedelay period function in** ReferenceTracker.java 
> <https://github.com/openjdk/jdk/blob/master/test/jdk/java/net/httpclient/ReferenceTracker.java#L270>effectively does the following:**
> 
> for (i = 0; i < 50; i++) {
> 
>              if (!TRACKERS.stream().anyMatch(hasOutstanding))
> 
>                  break;
> 
>              System.gc();
> 
>              Thread.sleep(10);
> 
> }
> 
> During this grace period there are at least 3 thread categories fighting 
> for the threads_lockmutex.
> 
> 1.TheSafepointSynchronize 
> <http://ccwdfgl9733.glds.c.eu-de-1.cloud.sap:8080/source/s?defs=SafepointSynchronize&project=openjdk-jdk>::begin <http://ccwdfgl9733.glds.c.eu-de-1.cloud.sap:8080/source/s?refs=begin&project=openjdk-jdk>() locks thethreads_lock mutex for the gc. It holds the mutex for about 300ms.
> 
> 2.The wait_until_not_protected()function, which is called a dozen times 
> during one loop iteration. It holds the mutex for about 0.4ms.
> 
> 3.And the SelectorManager threads which need the mutex one time to 
> complete their shutdown. They need the mutex for about 0.5ms.

#2 and #3 are related as wait_until_not_protected is used during thread 
termination and detaching (for JNI attached threads).

> The threads of all three categories are running with _identical_ native 
> thread priority .
> 
> On the various Linux derivates(x86 and ppc64) you can see, that the 
> SelectorManagerthreads are finding slots between the other mutex 
> consumers, so that all threads are shutdown within about 7 loop iterations.
> 
> On AIX the SelectorManagerthreads can’t make it or less often. So, some 
> of them are still alive after the complete grace period, although they 
> were queued up even before the grace period. This means in total 50 
> iterations where their requests were ignored and all later coming 
> requests from 1.+2. were satisfied.
> 
> So, how are the thread priorities handled in hotspot. Why are the AIX 
> SelectorManagerthreads discriminated.

By default they are ignored - see os::set_native_priority for each OS 
and note that ThreadPriorityPolicy == 0 by default.

> Can you please help me with an idea.

It is probably more the barging/queuing policy of the mutex 
implementation than actual thread scheduling priority. But that would be 
a question for the AIX developers.

Cheers,
David
-----

> Greetings
> 
> Joachim
>