RFR: 8371964: C2 compilation asserts with "Unexpected load/store size" [v3]

[Emanuel Peter]

On Mon, 24 Nov 2025 18:10:50 GMT, Quan Anh Mai <qamai at openjdk.org> wrote:

>> Hi,
>> 
>> This fixes the crash in `Load/StoreVectorMaskedNode::Ideal`. The issue here is that the graph is not canonical during idealization, which leads to us processing a dead node. The fix I propose is to bail-out when that happens.
>> 
>> To be more specific, for this issue, we have the graph that looks like:
>> 
>>     ConI -> ConvI2L -> CastLL(0..32) -> VectorMaskGen
>> 
>> with `ConI` being 45 and `MaxVectorSize` being 32. In this instance, `CastLL` is processed before `ConvI2L`, and when it is processed, it sees that the type of `ConvI2L` being its bottom type. As a result, it does not know that it is top, and since we are after macro expansion, which is after loop opts, the `CastLL` goes away, leaving us with:
>> 
>>     ConI -> ConvI2L -> VectorMaskGen
>> 
>> After `ConvI2L` is processed, we know that the input of `VectorMaskGen` is a constant 45, which is larger than `MaxVectorSize`, leading to the assert failure.
>> 
>> Please take a look and leave your thoughts, thanks a lot.
>
> Quan Anh Mai has updated the pull request incrementally with one additional commit since the last revision:
> 
>   reviews

Looks good to me, thanks for fixing this @merykitty !

src/hotspot/share/opto/vectornode.cpp line 1177:

> 1175:       int load_sz = type2aelembytes(mask_bt) * ty->get_con();
> 1176:       if (load_sz > MaxVectorSize) {
> 1177:         // See LoadVectorMaskedNode::Ideal

Suggestion:

        // After loop opts, cast nodes are aggressively removed, if the input is then transformed
        // into a constant that is outside the range of the removed cast, we may encounter it here.
        // This should be a dead node then.

Optional: Might as well just repeat the explanation. If the code in `LoadVectorMaskedNode::Ideal` changes it is unlikely that we would notice here, and then we'd have a dead link.

-------------

Marked as reviewed by epeter (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/28410#pullrequestreview-3528671396
PR Review Comment: https://git.openjdk.org/jdk/pull/28410#discussion_r2580010355