RFR: 8212933: Thread-SMR: requesting a VM operation whilst holding a ThreadsListHandle can cause deadlocks

[Erik Österlund]

Hi Robbin,

Looks good.

Thanks,
/Erik

On 2018-10-26 16:33, Robbin Ehn wrote:
> Hi, please review.
>
> When the VM thread executes a handshake it uses different ThreadsLists 
> during
> the execution. A JavaThread that is armed for the handshake when it is 
> already
> in the exit path in VM will cancel the handshake. Even if the VM 
> thread cannot
> see this thread after the initial ThreadsList which where used for 
> arming, the
> handshake can progress when the exiting thread cancels the handshake.
>
> But if a third thread takes a ThreadsList where the exiting JavaThread 
> is present and tries to execute a VM operation, hence waiting on VM 
> thread to finish the handshake, the JavaThread in the exit path can 
> never reach the handshake cancellation point. VM thread cannot 
> finishes the handshake and the third thread is stuck waiting on the VM 
> thread.
>
> To allow holding a ThreadsList when executing a VM operation we 
> instead let the
> VM thread use the same ThreadsList over the entire handshake making 
> all armed
> threads visible to the VM thread at all time. And if VM thread spots a 
> terminated thread it will count that thread is already done by only 
> clearing
> it's operation.
>
> Passes local stress testing, t1-5 and the deadlock is no longer 
> reproduce-able.
> Added a jtreg handshake + thread suspend test as a reproducer.
>
> Issue: https://bugs.openjdk.java.net/browse/JDK-8212933
> Code: http://cr.openjdk.java.net/~rehn/8212933/v1/webrev/
>
> Thanks, Robbin