[master] RFR: Implement Shenandoah support [v22]

[Zhengyu Gu]

On Tue, 19 Apr 2022 11:02:39 GMT, Roman Kennke <rkennke at openjdk.org> wrote:

>> This implements support for the Shenandoah GC in Lilliput. The following areas require special treatment in order for Shenandoah to work:
>> 
>> ### JVMTI/JFR
>> 
>> I replace the fwdptr-resolve in the object scan loop with a proper LRB to prevent exposing from-space objects to JVMTI runtime.
>> 
>> ### Stack-locking
>> 
>> Accessing the header for the Klass* (and therefore, for size) requires a special protocol to ensure that we're not chasing a stack-locked displaced header that is about to be unlocked, and therefore access potential garbage memory. This is done in #25. However, in Shenandoah it is slightly more complicated, because we need to access the size of an object in from-space, and don't want to observe a stack-lock that is about to be unlocked by another thread. In particular, I am worried about the following scenario:
>> T1:
>> 1. leaves the final-mark safepoint, while holding lock O
>> 2. Starts concurrent evacuation of O (concurrent threads evacuation)
>> 3. CAS fwdptr to header of O
>> 4. Unlocks O
>> 
>> T2 (possibly GC thread):
>> 1. Starts evacuation of O
>> 2. Accesses size/Klass*/header of O *in from space*, observe stack-lock
>> 3. CAS fwdptr to header of O
>> 
>> If context switches after step 2, then T2 loads a stack-lock, then T1 succeeds to evacuate *and* unlock O, and then T2 accesses a dangling stack-lock.
>> 
>> We can use the same protocol that we implemented in #25 to prevent this: whenever we access the header of a from-space object, CAS 0 (INFLATING) into the header to prevent progress by any other thread, while at the same time get a safe hold on the stack-lock (or neutral lock if other thread was faster). In order for this to work, we need to change the evacuation protocol such that it retries (in busy-loop) when it observes a 0.
>> Same goes for any code that loads the mark-word in from-space (not all that many places). For loading the mark-word, we need to extend the protocol a little to allow reaching through the forwarding pointer. This is GC specific for Shenandoah. That is why I put the implementation for this into ShenandoahObjectUtils, which mostly mirrors ObjectSynchronizer implementation to load the mark word, with additional from-space object handling (which would not be necessary in regular runtime accesses to the mark-word, outside of GC code).
>> 
>> ### Monitors
>> 
>> Monitors exhibit a similar problem: when observing a monitor while accessing an object's header, the concurrent deflater thread might concurrently deflate that monitor, and our thread might access a dangling monitor pointer. For Java threads, this is already prevented by the deflating protocol:
>> - First the deflater thread fixes all monitor headers back to neutral. During this phase, it is ok to racily load a monitor header: the monitor is still there, and the displaced header is safe to access.
>> - All Java threads are rendezvous'ed.
>> - Deflater destroys all deflated monitors. At this point, all Java thread would see a neutral header, and cannot access the destroyed monitors anymore.
>> 
>> This protocol is already extended by #27 to also rendezvous GC threads. This only requires that concurrent GC threads participate in SuspendibleThreadSet. Shenandoah has already implemented this, but turned off by default. The remaining step for Shenandoah to safely access monitor headers is to enable Suspendible GC workers.
>> 
>> 
>> Testing:
>>  - [x] hotspot_gc_shenandoah (x86_64, x86_32, aarch64)
>>  - [x] tier1 +UseShenandoahGC  (x86_64, x86_32, aarch64)
>>  - [x] tier2 +UseShenandoahGC  (x86_64, x86_32, aarch64)
>>  - [x] tier3 +UseShenandoahGC (x86_64, x86_32)
>>  - [ ] tier4 +UseShenandoahGC
>>  - [x] specjvm
>
> Roman Kennke has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 44 commits:
> 
>  - Merge remote-tracking branch 'origin/shenandoah-lilliput' into shenandoah-lilliput
>  - Merge remote-tracking branch 'origin/shenandoah-lilliput' into shenandoah-lilliput
>  - Merge branch 'master' into shenandoah-lilliput
>  - Merge branch 'master' into shenandoah-lilliput
>  - Revert some unnecessary changes
>  - Revert JVMTI related changes
>  - Add missing precompiled include
>  - 32bit fixes
>  - More reverts and cleanups
>  - Restore some commented-out parts
>  - ... and 34 more: https://git.openjdk.java.net/lilliput/compare/47392f37...00ca6aea

Changes requested by zgu (no project role).

src/hotspot/share/gc/shenandoah/shenandoahForwarding.inline.hpp line 85:

> 83: 
> 84:   while (true) {
> 85:     markWord new_mark = markWord::encode_pointer_as_mark(update);

You can hoist encoding `new_mark` outside the loop.

src/hotspot/share/gc/shenandoah/shenandoahObjectUtils.inline.hpp line 36:

> 34: #include "runtime/thread.hpp"
> 35: 
> 36: // This is a variant of ObjectSynchronizer::safe_load_mark(), which does the same thing, but also

`ObjectSynchronizer::safe_load_mark()` no longer exists, please update the comment.

src/hotspot/share/gc/shenandoah/shenandoahObjectUtils.inline.hpp line 40:

> 38: // code is supposed to observe from-space objects.
> 39: #ifdef _LP64
> 40: markWord ShenandoahObjectUtils::stable_mark(oop obj) {

There is a lot of duplication with `ObjectSynchronizer::stable_mark()`. I think you can just  call `objectSynchronizer::stable_mark()` after resolve forward pointer.

src/hotspot/share/gc/shenandoah/shenandoah_globals.hpp line 337:

> 335:           "end of concurrent marking.")                                     \
> 336:                                                                             \
> 337:   product(bool, ShenandoahSuspendibleWorkers, true, EXPERIMENTAL,           \

Why this is needed?

-------------

PR: https://git.openjdk.java.net/lilliput/pull/32