RFR: 8267188: gc/stringdedup/TestStringDeduplicationInterned.java fails with Shenandoah

[Zhengyu Gu]

Shenandoah currently enqueues deduplication candidates when marks object gray.

However, it can not do so when it scans roots, as it can potentially result lock rank inversion between stack watermark lock and dedup request buffer lock.

As the result, it can not enqueue as many as candidates as it should be able to, I believe JDK-8271834 is due to the same problem. I purpose we switch to enqueue candidates when we mark object black.

We are still not able to enqueue all candidates, only when they have displaced headers or have monitor inflating in progress. Upstream is working on removing displaced headers, we should revisit the logic afterward, or we can choose to deduplicate all string regardless their ages.



Test:
- [x] hotspot_gc_shenandoah
- [x] tier1 with -XX:+UseShenandoahGC and -XX:+UseStringDeduplication

-------------

Commit messages:
 - fix format
 - v0
 - Merge branch 'master' into JDK-8267188-string-intern
 - v0

Changes: https://git.openjdk.java.net/jdk/pull/5151/files
 Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=5151&range=00
  Issue: https://bugs.openjdk.java.net/browse/JDK-8267188
  Stats: 112 lines in 8 files changed: 35 ins; 28 del; 49 mod
  Patch: https://git.openjdk.java.net/jdk/pull/5151.diff
  Fetch: git fetch https://git.openjdk.java.net/jdk pull/5151/head:pull/5151

PR: https://git.openjdk.java.net/jdk/pull/5151