RFR: Consistenly order STS scope outside of OOM scope to avoid deadlocks

[Aleksey Shipilev]

On 10/16/2018 09:44 PM, Roman Kennke wrote:
> We have observed very spurious deadlocks that involve the OOMEvacScope
> and STSJoiner scope. It goes like this: Thread 1 enters OOM scope, but
> did not yet enter the STS scope. Thread2 is inside both scopes and
> OOM-s, cancelling evac and leaves the STS and attempts to leave the
> OOM-scope, waiting for Thread 1. Apparently, another thread calls for a
> (non-GC) safepoint, which makes Thread 1 block upon entering the STS
> scope, in other words it will never leave the OOM scope.
> 
> Ordering the STS-scope consistently outside of OOM-scope should solve
> this problem: the OOM-leaver will not get blocked by threads trying to
> get into STS first, and will thus always leave all relevant scopes.
> 
> Testing: tier3_gc_shenandoah
> 
> http://cr.openjdk.java.net/~rkennke/sts-oom-deadlock/webrev.00/

OK. I wonder if we should put the assert somewhere. For example, subclass STS with ShenandoahSTS and
check that OOM scope is not yet acquired?

The patch seems to also miss previous STS change in Traversal, needs retesting, probably.

-Aleksey