RFR: 8307795: AArch64: Optimize VectorMask.truecount() on Neon

[Andrew Haley]

On Mon, 15 May 2023 02:58:46 GMT, Chang Peng <duke at openjdk.org> wrote:

> In Vector API Java level, vector mask is represented as a boolean array with 0x00/0x01 (8 bits of each element) as values, aka in-memory format. When it is loaded into vector register, e.g. Neon, the in-memory format will be converted to in-register format with 0/-1 value for each lane (lane width aligned to its type) by VectorLoadMask [1] operation, and convert back to in-memory format by VectorStoreMask[2]. In Neon, a typical VectorStoreMask operation will first narrow given vector registers by xtn insn [3] into byte element type, and then do a vector negate to convert to 0x00/0x01 value for each element.
> 
> For most of the vector mask operations, the input mask is in-register format. And a vector mask also works in-register format all through the compilation. But for some operations like VectorMask.trueCount()[4] which counts the elements of true value, the expected input mask is in-memory format. So a VectorStoreMask is generated to convert the mask from in-register format to in-memory format before those operations.
> 
> However, for trueCount() these xtn instructions in VectorStoreMask can be saved, since the narrowing operations will not influence the number of active lane (value of 0x01) of its input.
> 
> This patch adds an optimized rule `VectorMaskTrueCount (VectorStoreMask mask)` to save the unnecessary narrowing operations.
> 
> For example,
> 
> 
> var m = VectorMask.fromArray(IntVector.SPECIES_PREFERRED, ba, 0);
> m.not().trueCount();
> 
> 
> will produce following assembly on a Neon machine before this patch:
> 
> 
> ...
> mvn     v16.16b, v16.16b           // VectorMask.not()
> xtn     v16.4h, v16.4s
> xtn     v16.8b, v16.8h
> neg     v16.8b, v16.8b             // VectorStoreMask
> addv    b17, v16.8b
> umov    w0, v17.b[0]               // VectorMask.trueCount()
> ...
> 
> 
> After this patch:
> 
> 
> ...
> mvn     v16.16b, v16.16b           // VectorMask.not()
> addv    s17, v16.4s
> smov    x0, v17.b[0]
> neg     x0, x0                     // Optimized VectorMask.trueCount()
> ...
> 
> 
> In this case, we can save two xtn insns.
> 
> Performance:
> 
> Benchmark     Before               After                          Unit
> testInt           723.822 ± 1.029  1182.375 ± 12.363    ops/ms
> testLong       632.154 ± 0.197  1382.74 ± 2.188        ops/ms
> testShort      788.665 ± 1.852   1152.38 ± 3.77         ops/ms
> 
> [1]: https://github.com/openjdk/jdk/blob/e1e758a7b43c29840296d337bd2f0213ab0ca3c9/src/hotspot/cpu/aarch64/aarch64_vector.ad#L4740
> [2]: https://github.com/openjdk/jdk/b...

That makes sense. Is it likely that there are more of these combined operations on vector masks that could be matched? if so, it might make sense to do the job earlier, in the C2 optimizer.

test/micro/org/openjdk/bench/jdk/incubator/vector/StoreMaskTrueCount.java line 80:

> 78:             m = m.not();
> 79:             res += m.trueCount();
> 80:         }

This looks like it might be removed by loop opts. I think you might need a blackhole somewhere.

-------------

Marked as reviewed by aph (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/13974#pullrequestreview-1426088017
PR Review Comment: https://git.openjdk.org/jdk/pull/13974#discussion_r1193540231