RFR: 8286737: Test vmTestbase/gc/gctests/WeakReference/weak006/weak006.java fails: Last soft reference has not been cleared

[Albert Mingkun Yang]

On Sun, 22 May 2022 22:14:45 GMT, Leonid Mesnik <lmesnik at openjdk.org> wrote:

> The problem is that PSScavenge::invoke()  (young gen GC) never tries to clear soft references. 
> 
> The fix is to update the clear_all_soft_refs to be the same as in full GC if scavenge is run as a part of full GC.  Test passed before https://bugs.openjdk.java.net/browse/JDK-8285951 Replace Algorithms.eatMemory(...) with WB.fullGC() in vmTestbase_vm_gc_ref tests because the eatMemory causes a lot of full GC and all references were propagated into old gen.  
> 
> The failed mem allocation also runs full GC with clear_all_soft_refs = true before throwing OOME so it should be also ok.

It was unclear to me why a Full GC cycle with the correct soft-ref policy couldn't discover/clear that soft ref, so I took a deeper look. My conclusion is that if a non-strong ref, `A`, is discovered in the Young GC cycle, it becomes live/reachable and its enclosing non-strong refs (if any) becomes non-discoverable in the Full GC cycle in the same pause. In contrast, that non-strong ref `A` is actually unreachable at the Java level, so the Full GC cycle can discover its enclosing non-strong refs. IOW, a pause including both Young and Full GC cycles can result in less reclaimed mem (or more kept objs) than a pause with only Full GC. Such behavior, IMO, is a bit counterintuitive.

The fix of adding soft-ref policy to the Young GC doesn't resolve the general problem, because the young/old-gen and obj size could affect which generation various objs lie, which affects what non-strong refs will be discovered and treated as live. (The Weak ref case in the same test can also be triggered due to the same reason.)

I believe one reliable solution here is to disable Full-GC-initiated Young GC using `-XX:-ScavengeBeforeFullGC`. (On my box I can confirm it fixes the failing test.)

-------------

PR: https://git.openjdk.java.net/jdk/pull/8829