RFR: 8374043: C2: assert(_base >= VectorMask && _base <= VectorZ) failed: Not a Vector

[Manuel Hässig]

On Tue, 6 Jan 2026 09:18:32 GMT, Xiaohong Gong <xgong at openjdk.org> wrote:

> ### Problem:
> 
> Test `compiler/vectorapi/VectorMaskToLongTest.java` crashes intermittently (approximately once per 200+ runs) with stress VM options such as `-XX:+StressIGVN`:
> 
> 
> // A fatal error has been detected by the Java Runtime Environment:
> //
> // Internal Error (jdk/src/hotspot/share/opto/type.hpp:2287), pid=69056, tid=28419
> // assert(_base >= VectorMask && _base <= VectorZ) failed: Not a Vector
> // ...
> 
> 
> The crash occurs in following code when calling `is_vect()` in the assertion added by JDK-8367292 [1]:
> 
> https://github.com/openjdk/jdk/blob/2cb228e142369ec73d768d8a69653a984b1c5908/src/hotspot/share/opto/vectornode.cpp#L1920-L1924
> 
> ### Root Cause:
> 
> The mask's type becomes TOP (unreachable) during compiler optimizations when the mask node is marked as dead before all its users are removed from the ideal graph. If `Ideal()` is subsequently called on a user node, it may access the TOP type, triggering the assertion.
> 
> Here is the simplified ideal graph showing the crash scenario:
> 
> 
>       Con #top
>        |       ConI
>          \      /
>            \  /
>      VectorStoreMask
>              |
>          VectorMaskToLong  # !jvms: IntMaxVector$IntMaxMask::toLong
> 
> 
> ### Detailed Scenario:
> 
> Following is the method in the test case that hits the assertion: 
> 
> https://github.com/openjdk/jdk/blob/2cb228e142369ec73d768d8a69653a984b1c5908/test/hotspot/jtreg/compiler/vectorapi/VectorMaskToLongTest.java#L65-L70
> 
> This method accepts a `VectorSpecies<?>` parameter and calls vector APIs `VectorMask.fromLong()` and `toLong()`. It is called with species ranging from `ByteVector.SPECIES_MAX` to `DoubleVector.SPECIES_MAX`. During compilation, C2 speculatively generates fast paths for `toLong()` for all possible species. 
> 
> When compiling a specific test case such as:
> https://github.com/openjdk/jdk/blob/6eaabed55ca4670d8c317f0a4323ccea4dd0b9ca/test/hotspot/jtreg/compiler/vectorapi/VectorMaskToLongTest.java#L177-L179
> 
> the compiler inlines the method and attempts to optimize away unreachable branches. The following graph shows the situation before the mask becomes `TOP`:
> 
> 
>                      VectorBox # DoubleMaxMask, generated by VectorMask.fromLong()
>                        /    \
>                      AddP     \
>                       |         \
>                   LoadNClass      \
>    ConP #IntMaxMask    |            |
>       \                |             |
>         \        DecodeNClass       |
>           \       /                |
>             \   /                 |
>              CmpP    ...

A drive-by comment on the reproducibility:
 - Does this only reproduce for specific hardware features or on all relatively new vector instruction sets?
 - Have you tried to reproduce this using the `StressSeed` flag? In the hs-error file you should find it with all the hotspot flags and rerunning the test with that seed often leads to a reproducible failure.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/29057#issuecomment-3722723846