RFR: 8374043: C2: assert(_base >= VectorMask && _base <= VectorZ) failed: Not a Vector

[Quan Anh Mai]

On Tue, 6 Jan 2026 09:18:32 GMT, Xiaohong Gong <xgong at openjdk.org> wrote:

> ### Problem:
> 
> Test `compiler/vectorapi/VectorMaskToLongTest.java` crashes intermittently (approximately once per 200+ runs) with stress VM options such as `-XX:+StressIGVN`:
> 
> 
> // A fatal error has been detected by the Java Runtime Environment:
> //
> // Internal Error (jdk/src/hotspot/share/opto/type.hpp:2287), pid=69056, tid=28419
> // assert(_base >= VectorMask && _base <= VectorZ) failed: Not a Vector
> // ...
> 
> 
> The crash occurs in following code when calling `is_vect()` in the assertion added by JDK-8367292 [1]:
> 
> https://github.com/openjdk/jdk/blob/2cb228e142369ec73d768d8a69653a984b1c5908/src/hotspot/share/opto/vectornode.cpp#L1920-L1924
> 
> ### Root Cause:
> 
> The mask's type becomes TOP (unreachable) during compiler optimizations when the mask node is marked as dead before all its users are removed from the ideal graph. If `Ideal()` is subsequently called on a user node, it may access the TOP type, triggering the assertion.
> 
> Here is the simplified ideal graph showing the crash scenario:
> 
> 
>       Con #top
>        |       ConI
>          \      /
>            \  /
>      VectorStoreMask
>              |
>          VectorMaskToLong  # !jvms: IntMaxVector$IntMaxMask::toLong
> 
> 
> ### Detailed Scenario:
> 
> Following is the method in the test case that hits the assertion: 
> 
> https://github.com/openjdk/jdk/blob/2cb228e142369ec73d768d8a69653a984b1c5908/test/hotspot/jtreg/compiler/vectorapi/VectorMaskToLongTest.java#L65-L70
> 
> This method accepts a `VectorSpecies<?>` parameter and calls vector APIs `VectorMask.fromLong()` and `toLong()`. It is called with species ranging from `ByteVector.SPECIES_MAX` to `DoubleVector.SPECIES_MAX`. During compilation, C2 speculatively generates fast paths for `toLong()` for all possible species. 
> 
> When compiling a specific test case such as:
> https://github.com/openjdk/jdk/blob/6eaabed55ca4670d8c317f0a4323ccea4dd0b9ca/test/hotspot/jtreg/compiler/vectorapi/VectorMaskToLongTest.java#L177-L179
> 
> the compiler inlines the method and attempts to optimize away unreachable branches. The following graph shows the situation before the mask becomes `TOP`:
> 
> 
>                      VectorBox # DoubleMaxMask, generated by VectorMask.fromLong()
>                        /    \
>                      AddP     \
>                       |         \
>                   LoadNClass      \
>    ConP #IntMaxMask    |            |
>       \                |             |
>         \        DecodeNClass       |
>           \       /                |
>             \   /                 |
>              CmpP    ...

src/hotspot/share/opto/vectornode.cpp line 1923:

> 1921:     Node* mask = in1->in(1);
> 1922:     const TypeVect* mask_vt = mask->bottom_type()->isa_vect();
> 1923:     if (mask_vt == nullptr) {

It is better to filter the exact `Type::TOP` instance and assert that otherwise, this must be a `TypeVect`. Additionally, if the type of the input is `Type::TOP`, we can eagerly return `C->top()` to kill it.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/29057#discussion_r2685541505