RFR: 8374043: C2: assert(_base >= VectorMask && _base <= VectorZ) failed: Not a Vector [v3]

[Vladimir Ivanov]

On Wed, 21 Jan 2026 08:56:19 GMT, Xiaohong Gong <xgong at openjdk.org> wrote:

>> ### Problem:
>> 
>> Test `compiler/vectorapi/VectorMaskToLongTest.java` crashes intermittently (approximately once per 200+ runs) with stress VM options such as `-XX:+StressIGVN`:
>> 
>> 
>> // A fatal error has been detected by the Java Runtime Environment:
>> //
>> // Internal Error (jdk/src/hotspot/share/opto/type.hpp:2287), pid=69056, tid=28419
>> // assert(_base >= VectorMask && _base <= VectorZ) failed: Not a Vector
>> // ...
>> 
>> 
>> The crash occurs in following code when calling `is_vect()` in the assertion added by JDK-8367292 [1]:
>> 
>> https://github.com/openjdk/jdk/blob/2cb228e142369ec73d768d8a69653a984b1c5908/src/hotspot/share/opto/vectornode.cpp#L1920-L1924
>> 
>> ### Root Cause:
>> 
>> The mask's type becomes TOP (unreachable) during compiler optimizations when the mask node is marked as dead before all its users are removed from the ideal graph. If `Ideal()` is subsequently called on a user node, it may access the TOP type, triggering the assertion.
>> 
>> Here is the simplified ideal graph showing the crash scenario:
>> 
>> 
>>       Con #top
>>        |       ConI
>>          \      /
>>            \  /
>>      VectorStoreMask
>>              |
>>          VectorMaskToLong  # !jvms: IntMaxVector$IntMaxMask::toLong
>> 
>> 
>> ### Detailed Scenario:
>> 
>> Following is the method in the test case that hits the assertion: 
>> 
>> https://github.com/openjdk/jdk/blob/2cb228e142369ec73d768d8a69653a984b1c5908/test/hotspot/jtreg/compiler/vectorapi/VectorMaskToLongTest.java#L65-L70
>> 
>> This method accepts a `VectorSpecies<?>` parameter and calls vector APIs `VectorMask.fromLong()` and `toLong()`. It is called with species ranging from `ByteVector.SPECIES_MAX` to `DoubleVector.SPECIES_MAX`. During compilation, C2 speculatively generates fast paths for `toLong()` for all possible species. 
>> 
>> When compiling a specific test case such as:
>> https://github.com/openjdk/jdk/blob/6eaabed55ca4670d8c317f0a4323ccea4dd0b9ca/test/hotspot/jtreg/compiler/vectorapi/VectorMaskToLongTest.java#L177-L179
>> 
>> the compiler inlines the method and attempts to optimize away unreachable branches. The following graph shows the situation before the mask becomes `TOP`:
>> 
>> 
>>                      VectorBox # DoubleMaxMask, generated by VectorMask.fromLong()
>>                        /    \
>>                      AddP     \
>>                       |         \
>>                   LoadNClass      \
>>    ConP #IntMaxMask    |            |
>>       \                |             |
>>         \        DecodeN...
>
> Xiaohong Gong has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Ensure it is vector type for vector unbox result

Testing passed (hs-tier1 - hs-tier4).

-------------

Marked as reviewed by vlivanov (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/29057#pullrequestreview-3699793380