RFR: 8370863: VectorAPI: Optimize the VectorMaskCast chain in specific patterns [v11]

[Eric Fang]

On Wed, 25 Feb 2026 13:11:28 GMT, Emanuel Peter <epeter at openjdk.org> wrote:

>> Eric Fang has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 16 commits:
>> 
>>  - Improve the code comment and tests
>>  - Merge branch 'master' into JDK-8370863-mask-cast-opt
>>  - Refine the JTReg tests
>>  - Add clearer comments to VectorMaskCastIdentityTest.java
>>  - Update copyright year to 2026
>>  - Merge branch 'master' into JDK-8370863-mask-cast-opt
>>  - Convert the check condition for vector length into an assertion
>>    
>>    Also refined the tests.
>>  - Refine code comments
>>  - Merge branch 'master' into JDK-8370863-mask-cast-opt
>>  - Merge branch 'master' into JDK-8370863-mask-cast-opt
>>  - ... and 6 more: https://git.openjdk.org/jdk/compare/c0c1775a...dcd64ad1
>
> I'm getting some failures with your new test:
> 
> 
> Failed IR Rules (1) of Methods (1)
> ----------------------------------
> 1) Method "compiler.vectorapi.VectorStoreMaskIdentityTest::testVectorMaskStoreIdentityFloat256" - [Failed IR rules: 1]:
>    * @IR rule 1: "@compiler.lib.ir_framework.IR(phase={DEFAULT}, applyIfPlatformAnd={}, applyIfCPUFeatureOr={"sve", "true", "avx2", "true"}, counts={"_#V#LOAD_VECTOR_Z#_", "_ at 4", ">= 3", "_#VECTOR_LOAD_MASK#_", "= 0", "_#VECTOR_STORE_MASK#_", "= 0", "_#VECTOR_MASK_CAST#_", "= 0"}, failOn={}, applyIfPlatform={}, applyIfPlatformOr={}, applyIfOr={}, applyIfCPUFeatureAnd={}, applyIf={"MaxVectorSize", "> 16"}, applyIfCPUFeature={}, applyIfAnd={}, applyIfNot={})"
>      > Phase "PrintIdeal":
>        - counts: Graph contains wrong number of nodes:
>          * Constraint 2: "(\\d+(\\s){2}(VectorLoadMask.*)+(\\s){2}===.*)"
>            - Failed comparison: [found] 1 = 0 [given]
>              - Matched node:
>                * 277  VectorLoadMask  === _ 106  [[ 275 ]]  #vectormask<F,4> !jvms: VectorMask::fromArray @ bci:47 (line 209) VectorStoreMaskIdentityTest::testTwoCastsKernel @ bci:5 (line 75) VectorStoreMaskIdentityTest::testVectorMaskStoreIdentityFloat256 @ bci:29 (line 271)
>          * Constraint 3: "(\\d+(\\s){2}(VectorStoreMask.*)+(\\s){2}===.*)"
>            - Failed comparison: [found] 1 = 0 [given]
>              - Matched node:
>                * 254  VectorStoreMask  === _ 271 272  [[ 216 ]]  #vectors<Z,4> !jvms: AbstractMask::intoArray @ bci:50 (line 75) VectorStoreMaskIdentityTest::testTwoCastsKernel @ bci:20 (line 77) VectorStoreMaskIdentityTest::testVectorMaskStoreIdentityFloat256 @ bci:29 (line 271)
>          * Constraint 4: "(\\d+(\\s){2}(VectorMaskCast.*)+(\\s){2}===.*)"
>            - Failed comparison: [found] 2 = 0 [given]
>              - Matched nodes (2):
>                * 271  VectorMaskCast  === _ 275  [[ 254 ]]  #vectormask<J,4> !jvms: ShortVector64$ShortMask64::cast @ bci:54 (line 665) VectorStoreMaskIdentityTest::testTwoCastsKernel @ bci:13 (line 77) VectorStoreMaskIdentityTest::testVectorMaskStoreIdentityFloat256 @ bci:29 (line 271)
>                * 275  VectorMaskCast  === _ 277  [[ 271 ]]  #vectormask<S,4> !orig=[5222],[2038] !jvms: FloatVector128$FloatMask128::cast @ bci:54 (line 654) VectorStoreMaskIdentityTest::testTwoCastsKernel @ bci:9 (line 76) VectorStoreMaskIdentityTest::testVectorMaskStoreIdentityFloat256 @ bci:29 (line 271)
> 
> 
> This was run on an x64 machine with extra flags:
> `-ea -esa -XX:CompileThreshold=100 ...

@eme64 Thank you for your help! I can reproduce the test failure. It appears to be a random test failure, but we can trigger it in multiple tests. I haven't found the root cause yet. Looking at the dump's IR, it seems an expected optimization isn't working; `(VectorStoreMask (VectorMaskCast (VectorMaskCast (VectorMaskCast (VectorLoadMask (LoadVector m))))))` should have been optimized to `(LoadVector m)`, but it wasn't. I'm continuing to debug. Thanks!

-------------

PR Comment: https://git.openjdk.org/jdk/pull/28313#issuecomment-3964784818