Shenandoah WB fastpath and optimizations

[Roland Westrelin]

> There are a few interesting observations here:
>
>  *) Enabling Shenandoah WB on this workload is responsible for ~14% throughput hit. This is the
> impact of the WB fastpath, because the workload runs with "passive" that does not do any concurrent
> cycles, and thus never reaches the slowpath.
>
> Shenandoah WB fastpath is basically four instructions:
>
>    movzbl 0x3d8(%rTLS), %rScratch  ; read evac-in-progress
>    test %rScratch, %rScratch
>    jne EVAC-ENABLED-SLOW-PATH
>    mov -0x8(%rObj), %rObj          ; read barrier
>
>  *) CPI numbers agree in both configurations, and the number of instructions had also grown +14%.
> This means the impact is due to larger code path, not some backend effect (like cache misses or such).
>
>  *) If we treat the number of of additional branches as the number of WBs for the workload, then we
> have around 40M WB fastpaths for each benchmark op. This means we should have around 80M
> L1-dcache-loads coming from WB (one for reading TLS flag, and another for RB), and that seems to
> agree with the data, given quite large error bounds.
>
>  *) What is weird is that we have ~18M excess *stores*, which are completely unaccounted by WBs.
>
>  *) ...and to add the insult to injury, 4 insn per WB should add up to 160M excess insns, but
> instead we have around 240M.
>
> The profile is too flat to pinpoint the exact code shape where we lose some of these instructions.
> But this collateral evidence seems to imply WBs make some stores more probable (e.g. by breaking
> some optimizations?), and that is the cause for inflated insn and L1 store counts?
>
> Thoughts?

Could the 18M stores be spills and somewhere in the 77..101M extra loads
would be their counterpart spill loads? The WB needs at least one extra
register and there's also the possibility that the WB slow path messes
up the register allocator heuristics (as we've seen with the XMM
spills).

Roland.