RFR: 8337791: VectorAPI jtreg ABSMaskedByteMaxVectorTests crashes with UseAVX=0 -XX:MaxVectorSize=8 [v2]

[Jatin Bhateja]

On Thu, 27 Nov 2025 13:19:24 GMT, Jatin Bhateja <jbhateja at openjdk.org> wrote:

>> This bug patch fixes a crash seen while querying the bottom type of MachTempNode corresponding to [rxmm0 operand](https://github.com/openjdk/jdk/blob/master/src/hotspot/cpu/x86/x86.ad#L22509) of blend pattern during late scheduling. Here, MaxVectorSize is contrainted to 8 bytes thus during C2 type system initialization, [TypeVect::VECTX ](https://github.com/openjdk/jdk/blob/master/src/hotspot/share/opto/type.cpp#L719) guarded by target supprted vector size remains uninitialized.
>> 
>> Its better to reject matching of VectorBlend in such a scenario.
>> 
>> All exisitng VectorAPI jtreg tests are passing with -XX:UseAVX=0 and -XX:MaxVectorSize=8
>> 
>> Kindly review and share your feedback.
>> 
>> Best Regards,
>> Jatin
>
> Jatin Bhateja has refreshed the contents of this pull request, and previous commits have been removed. The incremental views will show differences compared to the previous content of the PR. The pull request contains one new commit since the last revision:
> 
>   8337791: VectorAPI jtreg ABSMaskedByteMaxVectorTests crash with UseAVX=0 -XX:MaxVectorSize=8

src/hotspot/cpu/x86/x86.ad line 3324:

> 3322:       break;
> 3323:     case Op_VectorBlend:
> 3324:       if (UseAVX == 0 && MaxVectorSize < 16) {

While its technically fesable to create a 64-bit wide byte vector blend for AVX level 0 using blendvp selection pattern and emit pblendv instruction for it, but the machine nodes inputs will not be iso-morphic, ie. MachTemp node for xmm0 will be 128 bit wide TypeVect::VECTX and other input will be of type 64bit TypeVect::VECTD type. Given that blend is a lanewise operation hence IR type must be compatible with input types. Its better to use size_in_bits rather than MaxVectorSize in guard check.

We could have pushed this check to actual pattern predicate but its better to uplift the checks to match_rule_supported_vector since it prevents creation non-matchable IR during construction.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/28533#discussion_r2569238618