RFR: JDK-8220671: Initialization race for non-JavaThread PtrQueues

[Roman Kennke]

In Shenandoah testing we discovered an initialization race: A non-Java
GC thread (we have observed it on the StringDedupThread) may be
initialized concurrently while Java and GC are already up an running,
but not (yet) participate in safepointing.

BS::on_thread_attach() usually does propagate global GC state to
thread-local GC state, in this case the SATB active flag.

When doing this concurrently, while not participating in safepointing,
this may propagate the wrong state, and subsequently lead to heap
corruption (e.g. because we missed some SATB updates).

This is related to JDK-8219613 because before that change,
non-Java-threads would simply use a shared SATB queue instead.

The bug appeared in Shenandoah testing, but I don't see why it wouldn't
affect G1 too. It's probably not run with aggressive enough tests to
make it happen. (We run Shenandoah in aggressive mode, which starts
continuous GCing right at the start.)

Bug:
https://bugs.openjdk.java.net/browse/JDK-8220671
Webrev:
http://cr.openjdk.java.net/~rkennke/JDK-8220671/webrev.03/

The problem seems specific to the StringDedupThread (for now), and so is
the solution: in the StringDedupThread's pre_run() and post_run(), take
the STSJoiner. This ensures that the thread doesn't accidentally crosses
safepoints while initializing or exiting, and thus loosing SATB updates.

I tried a couple of other approaches like:
http://cr.openjdk.java.net/~rkennke/JDK-8220671/webrev.02/

But we also need to protect the addition/removal of the thread to the
global NonJavaThread list.

Testing: Running the offending test (TestStringDedupStress.java) 20x in
a row. It used to fail ~1 of 5 runs before. Now it all passes. Also,
hotspot_gc_shenandoah passes. tier1 is fine too. Will push it through
jdk-submit next.

Can I please get reviews?

Thanks,
Roman

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: OpenPGP digital signature
URL: <https://mail.openjdk.org/pipermail/hotspot-gc-dev/attachments/20190315/caf1d7aa/signature.asc>