RFR: 8337791: VectorAPI jtreg ABSMaskedByteMaxVectorTests crashes with UseAVX=0 -XX:MaxVectorSize=8 [v3]

[Emanuel Peter]

On Thu, 27 Nov 2025 15:32:15 GMT, Jatin Bhateja <jbhateja at openjdk.org> wrote:

>> This bug patch fixes a crash seen while querying the bottom type of MachTempNode corresponding to [rxmm0 operand](https://github.com/openjdk/jdk/blob/master/src/hotspot/cpu/x86/x86.ad#L22509) of blend pattern during late scheduling. Here, MaxVectorSize is contrainted to 8 bytes thus during C2 type system initialization, [TypeVect::VECTX ](https://github.com/openjdk/jdk/blob/master/src/hotspot/share/opto/type.cpp#L719) guarded by target supprted vector size remains uninitialized.
>> 
>> Its better to reject matching of VectorBlend in such a scenario.
>> 
>> All exisitng VectorAPI jtreg tests are passing with -XX:UseAVX=0 and -XX:MaxVectorSize=8
>> 
>> Kindly review and share your feedback.
>> 
>> Best Regards,
>> Jatin
>
> Jatin Bhateja has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Fine tune matcher check

@jatin-bhateja Thanks for looking into this!

I think it is fine to just disable the small vector size for AVX=0. Those platforms are quite rare now anyway.

Should we also adjust the predicate in the matching rule here?

  22509 instruct blendvp(vec dst, vec src, vec mask, rxmm0 tmp) %{
  22510   predicate(UseAVX == 0);   
  22511   match(Set dst (VectorBlend (Binary dst src) mask));                                                                                                                                                                                 
  22512   format %{ "vector_blend  $dst,$src,$mask\t! using $tmp as TEMP" %}
  22513   effect(TEMP tmp);  
  22514   ins_encode %{
  22515     assert(UseSSE >= 4, "required");
  22516 
  22517     if ($mask$$XMMRegister != $tmp$$XMMRegister) {
  22518       __ movdqu($tmp$$XMMRegister, $mask$$XMMRegister);
  22519     }
  22520     __ pblendvb($dst$$XMMRegister, $src$$XMMRegister); // uses xmm0 as mask
  22521   %}
  22522   ins_pipe( pipe_slow );
  22523 %}

-------------

PR Review: https://git.openjdk.org/jdk/pull/28533#pullrequestreview-3516189610