RFR: Hole in CAS barrier when using traversal heuristics

[Roman Kennke]

Am 25.01.2018 um 18:15 schrieb Zhengyu Gu:
> I am not complete sure this is right fix. There is hole in CAS barrier 
> when using traversal heuristics.
> 
> E.g. Unsafe_CompareAndSetObject() evacuates target and exchange object, 
> but not the field, so it may hit assertion in ShenandoahBarrier::enqueue().
> 
> I could not come up a reliable reproducer, but I have seen this a few 
> time with specjvm ScimarkLU with options:
> 
> "-Xmx1g -Xms1g -XX:+UseShenandoahGC -XX:+UnlockExperimentalVMOptions 
> -XX:+UnlockDiagnosticVMOptions -XX:ShenandoahGCHeuristics=traversal 
> -Xlog:gc+stats"
> 
> Webrev: http://cr.openjdk.java.net/~zgu/shenandoah/cas_traversal/webrev.00/
> 
> 
> Thanks,
> 
> -Zhengyu

Hi Zhengyu,

I am not sure what you mean by 'evacuates target and exchange object,
but not the field' .. clearly the target object needs to be evacuated, 
because we only write to to-space (write-barrier). Also, the exchange 
object needs to be evacuated, to ensure we end up only with to-space 
references in fields (storeval-barrier). What do you mean by evacuation 
of 'the field' ? The target field is part of the target object.

The issue here seems to be that the usual 'pre-barrier' (i.e. 
SATB-barrier) should not be called at all. However, since we do set the 
MARKING bit, we still get into this code path. We might just want to 
check for traversal-in-progress and return right at the start of the method.

Roman