RFR: Add new pinned/cset region state for evac-failure-path

[Aleksey Shipilev]

On 11/08/2017 01:43 PM, Roman Kennke wrote:
> We sometimes get an assert failure when running OOM during evac and attempting to pin a region
> afterwards: it would try to make a cset region pinned. This is normally not allowed, but necessary
> on the failure path, as a temporary state until mark-compact cleaned it up.
> 
> I solved it by adding a new region-state pinned-cset: when a region is in cset and is made pinned,
> it checks that it's on cancelled-gc path and goes to pinned-cset. From there it can only go back to
> cset (when unpinning the region) or to pinned (when clearing the cset state in mark-compact).
> 
> http://cr.openjdk.java.net/~rkennke/pinned-cset/webrev.00/

I have a larger problem with the patch like this. Suppose we are on cancellation path, and we have
already half-evacuated the cset region. Then pinning comes, but WB is disabled already, and so we
move the region to pinned_cset state. Now, Full GC code would not move the region data, making sure
pinned objects stay pinned.

Does it update the pointers right, though? E.g. could it happen that we are left with pointers to
pinned_cset after Full GC? Does it recycle those cset regions now, thus corrupting the heap?

Nits:

 *) In make_pinned and make_unpinned: move _pinned case towards _pinned_cset:

      case _cset:
        ...
        // Fall-through
      case _pinned_cset:
      case _pinned:
        _critical_pins++;
        return;

 *) RegionState comment is incomplete:

 108     _pinned_cset,       // region is pinned

Thanks,
-Aleksey