RFR: Consistenly order STS scope outside of OOM scope to avoid deadlocks

[Zhengyu Gu]

Make sense, good to me.

Thanks,

-Zhengyu

On 10/16/2018 03:44 PM, Roman Kennke wrote:
> We have observed very spurious deadlocks that involve the OOMEvacScope
> and STSJoiner scope. It goes like this: Thread 1 enters OOM scope, but
> did not yet enter the STS scope. Thread2 is inside both scopes and
> OOM-s, cancelling evac and leaves the STS and attempts to leave the
> OOM-scope, waiting for Thread 1. Apparently, another thread calls for a
> (non-GC) safepoint, which makes Thread 1 block upon entering the STS
> scope, in other words it will never leave the OOM scope.
> 
> Ordering the STS-scope consistently outside of OOM-scope should solve
> this problem: the OOM-leaver will not get blocked by threads trying to
> get into STS first, and will thus always leave all relevant scopes.
> 
> Testing: tier3_gc_shenandoah
> 
> http://cr.openjdk.java.net/~rkennke/sts-oom-deadlock/webrev.00/
> 
> Good?
> 
> Roman
>