RFR: 8205163: ZGC: Keeps finalizable marked PhantomReference referents strongly alive

[Stefan Karlsson]

Hi all,

Please review this patch to fix a bug where PhantomReferences can cause 
otherwise finalizable objects to become considered as strongly reachable.

http://cr.openjdk.java.net/~stefank/8205163/webrev.01/
https://bugs.openjdk.java.net/browse/JDK-8205163

ZGC has a concept of finalizable marked objects, for objects that are 
only reachable through the referents of Finalizers.

When objects that have been finalizable marked, are later found to also 
be strongly reachable, the marking strength is updated to strongly marked.

When "processing" SoftReferences, WeakReferences, and Finalizers, the GC 
checks if the referents are strongly marked, and if not, later pushes 
the References out to the pending list.

For PhantomReferences, the GC also checks if the referents are 
finalizable marked, and if not, later pushes the PhantomReferences out 
to the pending list.

Before the References are processed, the GC "discovers" References that 
it "encounters" during the marking phase. The GC keeps track of 
discovered References until the processing phase. If a Reference with a 
live referent is encountered, it is not discovered, and instead the 
fields (referent, discovered) are treated as normal fields.

The bug happens when the GC tries to discover a PhantomReference with a 
finalizable marked referent. Since the referent is already live 
according to the PhantomReferences's definition of live (finalizable or 
strongly marked), the PhantomReference is not discovered, and the fields 
are treated as normal fields. This means that the ordinary strong 
marking closure is applied to both the referent and the discovered 
field, causing the marking strength of the referent to be promoted from 
finalizable marked to strongly marked. This increase in strength isn't 
problematic for the PhantomReference, since it already considered the 
referent to be alive. However, any Finalizers that depended on the 
referent to only be finalizable marked, will now be dropped instead of 
pushed to the pending list, and therefore the referent will not be 
finalized.

The proposed patch prevents this problem by always discovering 
PhantomReferences with finalizable marked referents. By doing this, the 
marking code will not mark the referent as strongly reachable, the
processing code will still drop the PhantomReference. And the 
finalizable objects that were incorrectly kept alive, can now be finalized.

This was found in a closed test in higher-tier testing.

Thanks,
StefanK