RFR: New evacuation-locking protocol

[Roman Kennke]

I am proposing a new evacuation protocol. Until now we let concurrent
evacuating threads both make a copy and then compete to install the
forwarding pointer using a CAS on the fwdptr field. This worked well,
except for one problem: how to handle OOM during evacuation? We only
came up with the extra protocol in shenandoahEvacOOMHandler* that is
effectively a readers-writer-lock. This has proven to be a major PITA
in the past, has caused several deadlocks, bugs, etc etc.

I am proposing to use a kind of locking-protocol for the whole of
evacuation instead. The proposed patch uses a separate bitmap where
each bit corresponds to a heap-word (or more, see below). Before
evacuating an object, a thread must acquire the corresponding bit, and
when it's done, release it again.

OOM during evac is then handled by a global flag: when we are inside
the per-object-flag, we would raise this flag, and next time we visit
the object we observe the flag and don't attempt any more evacuations.

It also turns out that we can trim the bitmap by making it coarser, by
making each bit correspond to 2, 4, 8 or 2^N heap-words. This can be
achieved by setting the flag ShenandoahEvacLockGranularity to higher
values N. Setting it to 1 means each bit covers 2 words, 2 -> 4 words,
etc. It seems that we can *at least* make it 1 by default, because each
object is at least 2 heap words. Some initial experiments by Aleksey
have shown that we can probably increase it even more without
performance loss, but this warrants more experimentation.

I'm proposing to push it to shenandoah/jdk first, and let it bake there
before going to JDK mainline.

I'm splitting the review into two patches because it seems easier to
review separately. I'd push them as one changeset (unless you prefer it
separately) because the build would be broken between the two.

1. Prune the 'old' evac-handler:
http://cr.openjdk.java.net/~rkennke/evac-locking/webrev-prune-oom-handler.00/

2. Implementation of the new evac-locking:
http://cr.openjdk.java.net/~rkennke/evac-locking/webrev-evac-locking.00/

Testing: hotspot_gc_shenandoah, specjvm2008 doesn't show regressions

Can I please get a review?

Thanks,
Roman