Jetty hangs early on Java 12 on OSv with Threadlocal Handshakes ON - shorter version

[Robbin Ehn]

Hi Waldek,


> #0  0x0000100000c86370 in sem_trywait at plt ()
> #1  0x0000100001702bf2 in PosixSemaphore::trywait() ()
> #2  0x000010000121aab4 in VM_HandshakeAllThreads::doit() ()

VMThread is trying to figure out which threads have not executed their handshake 
operation and possible execute it for them (e.g. JavaThread in native).

If you attach gdb and step around in VM_HandshakeAllThreads::doit while looping 
there, you can figure out why we never finish.

> When I explicitly enable thread local handshakes on Java 11 it works as
> well:

In 11 it is only used by ZGC, but in 12 it is used for stack-scanning after 
nmethods.

Sweeper thread:

#12 0x0000100001219a48 in Handshake::execute(ThreadClosure*) ()
#13 0x00001000018b4a0c in NMethodSweeper::possibly_sweep() ()

> One more datapoint - I am able to run a simple Hello World java app on Java
> 12 on OSv with thread-local handshakes on so it looks like this problem
> appears on more complicated apps.

If you never get to needing to sweep nmethod, you will have never executed a 
handshake. (but you will execute safepoints using the handshake poll)

If you can easily reprod, take a slowdebug and when the process hang, attach gdb 
switch to VMThread and step around, you will see a count which decrease once for 
every thread which have executed it's handshake. To figure out if a thread have 
executed it's handshake, VMThread will take a per thread semaphore (decrease 
it), which serialize access to the per thread handshake operation, and then look 
if the operation is completed, return the semaphore (increase it), and look at 
next thread.

/Robbin

> 
> *Lastly, the exact same app works fine on Java 12 on Linux.*
> 
> So given all that could you please point me to what the problem might be?
> What has changed in the implementation of thread local handshakes between
> Java 11 and 12 that may have caused this issue (I see many changes made to
> https://github.com/openjdk/jdk/blob/0fe0312f416add1536a45ecfb292c887ef7e02bd/src/hotspot/share/runtime/handshake.cpp
> but
> I am not even sure which commit matches the version of Java 12 or Java 11 I
> am using)? I am thinking this has to do with some subtle differences
> between relevant ABI implementation between OSv and Linux. Signals related?
> 
> Thanks for your help in advance,
> 
> Waldek
> 
> Thread dump of all Java threads (in other email that was to big)
>