RFR (9) 8185133: Reference pending list root might not get marked

[Mikael Gerdin]

Hi all,

Please review this fix to a tricky reference processing / conc marking 
bug affecting G1 in 9.

The bug occurs when a weak reference WR is promoted to old and 
discovered during an initial mark pause. The WR is the referent of a 
soft reference SR. The concurrent reference processor determines that SR 
should be treated as a weak reference due to shortage of memory and now 
WR is reachable only from the reference pending list but not explicitly 
marked in the bitmap since objects promoted during the initial mark 
pause are not marked immediately.

The reason we are not saved by the SATB pre-barrier here is that 
clearing of the referent field of a reference object does not trigger 
the pre-barrier (and that would kind of defeat its purpose).

Before JDK-8156500 this worked because the reference pending list was a 
static field in the Reference class and the reference class was scanned 
during concurrent marking, so we would never lose track of the pending 
list head.

My suggested fix is to explicitly mark the reference pending list head 
oop during initial mark, after the reference enqueue phase.
This mirrors how other roots are handled in initial mark, see 
G1Mark::G1MarkPromotedFromRoots.

Webrev: http://cr.openjdk.java.net/~mgerdin/8185133/webrev.0
Bug: https://bugs.openjdk.java.net/browse/JDK-8185133

Testing: JPRT, tier2-5 gc tests, a LOT of runs of the failing test.

Many thanks to Kim and Erik Ö for discussions around this issue!

Thanks
/Mikael