RFR: JDK-8224584: Shenandoah: Eliminate forwarding pointer word

[Roman Kennke]

Until now, Shenandoah had the 'limitation' that it required an extra
word per object to keep the forwarding pointer. It either points to
itself (most of the time) or to the forwarded object when evacuated. The
forwarding pointer is updated atomically using CAS. This is the heart of
Shenandoah's evacuation protocol.

With LRB, we have the opportunity to eliminate that extra word. The
insight is that we really don't need the old copy of an object anymore
and we can store the forwarding pointer in the mark word of it. We would
distinguish a valid mark word from a forwarding pointer by checking the
lowest 2 bits: if they are 0b11 then it's forwarded, otherwise it's not.
0b11 is not a valid combination of bits otherwise. This is the same
protocol that is used by other GCs.

The evacuation protocol changes slightly to CAS into the mark word
instead of the extra word. But the basic mechanics are the same.

The extra decoding does not (significantly) affect performance because
it's only done in the mid/slow-path of the LRB. (As opposed to earlier
attempts at it where it'd have to be in the read-barrier.)

The full-GC needs some significant reworking because we used to store
the forwarding ptr in the extra slot. Storing it in mark-word in full-gc
is destructive because there are no from/to-space copies. Therfore we
need to stash the mark-word when it's not trivial. This is pretty much
what other GCs (G1, parallel, CMS) do, and we re-use the shared
infrastructure for this.

I've run SPECjvm2008. Performance benefits vary between 0 and ~+10%.
SPECjbb2015 shows +~12% improvement in throughput and +~9% improvement
in latency.

Bug:
https://bugs.openjdk.java.net/browse/JDK-8224584
Webrev:
http://cr.openjdk.java.net/~rkennke/JDK-8224584/webrev.00/

Testing:
The change has undergone intensive testing, reviews and bug-fixes in
shenandoah/jdk repository. We have run several workloads on it, some of
them many times. I've run the usual hotspot_gc_shenandoah on it many
times, both on x86 and aarch64.

There is one outstanding C2/matcher issue currently under review that
causes (rare but fatal) failures with this change:
https://bugs.openjdk.java.net/browse/JDK-8224580
https://mail.openjdk.java.net/pipermail/hotspot-compiler-dev/2019-May/033838.html

Roman