RFR: 8374043: C2: assert(_base >= VectorMask && _base <= VectorZ) failed: Not a Vector

[Xiaohong Gong]

On Tue, 13 Jan 2026 09:11:25 GMT, Quan Anh Mai <qamai at openjdk.org> wrote:

>> ### Problem:
>> 
>> Test `compiler/vectorapi/VectorMaskToLongTest.java` crashes intermittently (approximately once per 200+ runs) with stress VM options such as `-XX:+StressIGVN`:
>> 
>> 
>> // A fatal error has been detected by the Java Runtime Environment:
>> //
>> // Internal Error (jdk/src/hotspot/share/opto/type.hpp:2287), pid=69056, tid=28419
>> // assert(_base >= VectorMask && _base <= VectorZ) failed: Not a Vector
>> // ...
>> 
>> 
>> The crash occurs in following code when calling `is_vect()` in the assertion added by JDK-8367292 [1]:
>> 
>> https://github.com/openjdk/jdk/blob/2cb228e142369ec73d768d8a69653a984b1c5908/src/hotspot/share/opto/vectornode.cpp#L1920-L1924
>> 
>> ### Root Cause:
>> 
>> The mask's type becomes TOP (unreachable) during compiler optimizations when the mask node is marked as dead before all its users are removed from the ideal graph. If `Ideal()` is subsequently called on a user node, it may access the TOP type, triggering the assertion.
>> 
>> Here is the simplified ideal graph showing the crash scenario:
>> 
>> 
>>       Con #top
>>        |       ConI
>>          \      /
>>            \  /
>>      VectorStoreMask
>>              |
>>          VectorMaskToLong  # !jvms: IntMaxVector$IntMaxMask::toLong
>> 
>> 
>> ### Detailed Scenario:
>> 
>> Following is the method in the test case that hits the assertion: 
>> 
>> https://github.com/openjdk/jdk/blob/2cb228e142369ec73d768d8a69653a984b1c5908/test/hotspot/jtreg/compiler/vectorapi/VectorMaskToLongTest.java#L65-L70
>> 
>> This method accepts a `VectorSpecies<?>` parameter and calls vector APIs `VectorMask.fromLong()` and `toLong()`. It is called with species ranging from `ByteVector.SPECIES_MAX` to `DoubleVector.SPECIES_MAX`. During compilation, C2 speculatively generates fast paths for `toLong()` for all possible species. 
>> 
>> When compiling a specific test case such as:
>> https://github.com/openjdk/jdk/blob/6eaabed55ca4670d8c317f0a4323ccea4dd0b9ca/test/hotspot/jtreg/compiler/vectorapi/VectorMaskToLongTest.java#L177-L179
>> 
>> the compiler inlines the method and attempts to optimize away unreachable branches. The following graph shows the situation before the mask becomes `TOP`:
>> 
>> 
>>                      VectorBox # DoubleMaxMask, generated by VectorMask.fromLong()
>>                        /    \
>>                      AddP     \
>>                       |         \
>>                   LoadNClass      \
>>    ConP #IntMaxMask    |            |
>>       \                |             |
>>         \        DecodeN...
>
> src/hotspot/share/opto/vectornode.cpp line 1923:
> 
>> 1921:     Node* mask = in1->in(1);
>> 1922:     const TypeVect* mask_vt = mask->bottom_type()->isa_vect();
>> 1923:     if (mask_vt == nullptr) {
> 
> It is better to filter the exact `Type::TOP` instance and assert that otherwise, this must be a `TypeVect`. Additionally, if the type of the input is `Type::TOP`, we can eagerly return `C->top()` to kill it.

OK, I will check TOP input instead and convert back the assertion changes. Thanks a lot for your input @merykitty and @iwanowww !

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/29057#discussion_r2693069575