RFR (JDK8-only): Fix freeze on OOM-on-evac regarding the PLL

[Roman Kennke]

Oh btw, I tested that by running specjvm with aggressive heuristics,
this used to sometimes freeze before.

Roman

Am Mittwoch, den 21.12.2016, 18:56 +0100 schrieb Roman Kennke:
> This is a complicated one. We may get a freeze under the following
> situation:
> 
> when the final-mark pause is left, the ShenandoahConcurrentThread
> sends
> a message to the SurrogateLockerThread to release the pending-list-
> lock 
> (see VM_ShenandoahReferenceOperation::doit_epilogue()). The
> SurrogateLockerThread is a Java thread that gets kicked off right
> after
> the pause. It attempts to acquire the PLL (a Java lock) and thus
> employs a write-barrier on it. When that write-barrier runs out-of-
> memory, it ends up in our oom_during_evacuation() loop and is waiting
> for the _evacuation_in_progress flag to get cleared. However, since
> the
> ShenandoahConcurrentThread is waiting for the SLT to finish, we never
> get to where we clear that flag (we don't even kick off evacuation
> yet).
> 
> The proposed solution attempts to evacuate the PLL during the pause.
> If
> it succeeds, then the write-barrier will simply pick up the to-space
> object. If it fails, we schedule a full-gc, and turn off evacuation
> before leaving the pause. In no case can the write-barrier on the PLL
> run into OOM, and in all cases will it be correctly unlocked.
> 
> Luckily for us, the whole PLL madness has been changed in a very
> positive way in JDK9, so this change does not apply there.
> 
> http://cr.openjdk.java.net/~rkennke/fixoomevacpllfreeze/webrev.00/
> 
> Ok to push?
> 
> Roman