RFR: Consistenly order STS scope outside of OOM scope to avoid deadlocks

[Roman Kennke]

> On 10/16/2018 09:44 PM, Roman Kennke wrote:
>> We have observed very spurious deadlocks that involve the OOMEvacScope
>> and STSJoiner scope. It goes like this: Thread 1 enters OOM scope, but
>> did not yet enter the STS scope. Thread2 is inside both scopes and
>> OOM-s, cancelling evac and leaves the STS and attempts to leave the
>> OOM-scope, waiting for Thread 1. Apparently, another thread calls for a
>> (non-GC) safepoint, which makes Thread 1 block upon entering the STS
>> scope, in other words it will never leave the OOM scope.
>>
>> Ordering the STS-scope consistently outside of OOM-scope should solve
>> this problem: the OOM-leaver will not get blocked by threads trying to
>> get into STS first, and will thus always leave all relevant scopes.
>>
>> Testing: tier3_gc_shenandoah
>>
>> http://cr.openjdk.java.net/~rkennke/sts-oom-deadlock/webrev.00/
> 
> OK. I wonder if we should put the assert somewhere. For example, subclass STS with ShenandoahSTS and
> check that OOM scope is not yet acquired?

Good idea! Will do that... tomorrow ;-)

> The patch seems to also miss previous STS change in Traversal, needs retesting, probably.

Right. Will include that tomorrow too.

Cheers,
Roman