RFR (JDK8-only): Fix freeze on OOM-on-evac regarding the PLL

[Zhengyu Gu]

Okay.

-Zhengyu


On 12/21/2016 12:56 PM, Roman Kennke wrote:
> This is a complicated one. We may get a freeze under the following
> situation:
>
> when the final-mark pause is left, the ShenandoahConcurrentThread sends
> a message to the SurrogateLockerThread to release the pending-list-lock
> (see VM_ShenandoahReferenceOperation::doit_epilogue()). The
> SurrogateLockerThread is a Java thread that gets kicked off right after
> the pause. It attempts to acquire the PLL (a Java lock) and thus
> employs a write-barrier on it. When that write-barrier runs out-of-
> memory, it ends up in our oom_during_evacuation() loop and is waiting
> for the _evacuation_in_progress flag to get cleared. However, since the
> ShenandoahConcurrentThread is waiting for the SLT to finish, we never
> get to where we clear that flag (we don't even kick off evacuation
> yet).
>
> The proposed solution attempts to evacuate the PLL during the pause. If
> it succeeds, then the write-barrier will simply pick up the to-space
> object. If it fails, we schedule a full-gc, and turn off evacuation
> before leaving the pause. In no case can the write-barrier on the PLL
> run into OOM, and in all cases will it be correctly unlocked.
>
> Luckily for us, the whole PLL madness has been changed in a very
> positive way in JDK9, so this change does not apply there.
>
> http://cr.openjdk.java.net/~rkennke/fixoomevacpllfreeze/webrev.00/
>
> Ok to push?
>
> Roman