RFR: 8227175: ZGC: ZHeapIterator visits potentially dead objects

[Stefan Karlsson]

Hi all,

(Sending this RFR to hotspot-dev since it changes CLD claiming.)

Please review this patch to fix the ZHeapIterator to not visit 
potentially dead objects.

https://cr.openjdk.java.net/~stefank/8227175/
https://bugs.openjdk.java.net/browse/JDK-8227175

It changes how all heap iterations are done in ZGC. Previously, the 
marking code visited only the strong CLDs and traced through metadata to 
find all other CLDs that should be considered alive. The verification 
code, serviceability heap iterations, and marking without class 
unloading, skipped the metadata tracing part and visited all CLDs 
instead. Now, with this patch, all these heap iterations starts with the 
strong CLDs and trace through the object graph.

One complication with that scheme is that non-GC heap iterations might 
be executing after concurrent marking has started, but before dead CLDs 
have been unlinked. To allow the GC marking code and one, at a time, 
non-GC heap iteration to run at the same time, I've introduced a new 
claim bit in the CLD claiming byte. I've called it "other", so now we 
have "strong", "finalizable", and "other". The contract is that the 
"other" bits should only be used in a safepoint operation, and must be 
cleared before the operation ends. This way we get mutual exclusion 
between different users of the "other" bits.

The patch also adds more precise verification of ZGC references.

This patch was written a few weeks ago to make the verification of ZGC 
references more precise. I've been using it since then to get better 
verification of other patches and when hunting for bugs. This means I've 
been running it through tier 1-7 multiple times. The intent was to get 
this pushed to JDK 14, but now that we've seen that we have an actual 
bug because of the imprecise nature of the ZHeapIterator, I'd like to 
get this patch pushed to JDK 13. We considered trying to split this up 
into two parts, the first part that fixes the heap iterations and the 
second part that adds the extra ZGC verification, but we thing that 
would take longer time and be riskier.

Thanks,
StefanK