RFR: 8374043: C2: assert(_base >= VectorMask && _base <= VectorZ) failed: Not a Vector [v3]

[Xiaohong Gong]

> ### Problem:
> 
> Test `compiler/vectorapi/VectorMaskToLongTest.java` crashes intermittently (approximately once per 200+ runs) with stress VM options such as `-XX:+StressIGVN`:
> 
> 
> // A fatal error has been detected by the Java Runtime Environment:
> //
> // Internal Error (jdk/src/hotspot/share/opto/type.hpp:2287), pid=69056, tid=28419
> // assert(_base >= VectorMask && _base <= VectorZ) failed: Not a Vector
> // ...
> 
> 
> The crash occurs in following code when calling `is_vect()` in the assertion added by JDK-8367292 [1]:
> 
> https://github.com/openjdk/jdk/blob/2cb228e142369ec73d768d8a69653a984b1c5908/src/hotspot/share/opto/vectornode.cpp#L1920-L1924
> 
> ### Root Cause:
> 
> The mask's type becomes TOP (unreachable) during compiler optimizations when the mask node is marked as dead before all its users are removed from the ideal graph. If `Ideal()` is subsequently called on a user node, it may access the TOP type, triggering the assertion.
> 
> Here is the simplified ideal graph showing the crash scenario:
> 
> 
>       Con #top
>        |       ConI
>          \      /
>            \  /
>      VectorStoreMask
>              |
>          VectorMaskToLong  # !jvms: IntMaxVector$IntMaxMask::toLong
> 
> 
> ### Detailed Scenario:
> 
> Following is the method in the test case that hits the assertion: 
> 
> https://github.com/openjdk/jdk/blob/2cb228e142369ec73d768d8a69653a984b1c5908/test/hotspot/jtreg/compiler/vectorapi/VectorMaskToLongTest.java#L65-L70
> 
> This method accepts a `VectorSpecies<?>` parameter and calls vector APIs `VectorMask.fromLong()` and `toLong()`. It is called with species ranging from `ByteVector.SPECIES_MAX` to `DoubleVector.SPECIES_MAX`. During compilation, C2 speculatively generates fast paths for `toLong()` for all possible species. 
> 
> When compiling a specific test case such as:
> https://github.com/openjdk/jdk/blob/6eaabed55ca4670d8c317f0a4323ccea4dd0b9ca/test/hotspot/jtreg/compiler/vectorapi/VectorMaskToLongTest.java#L177-L179
> 
> the compiler inlines the method and attempts to optimize away unreachable branches. The following graph shows the situation before the mask becomes `TOP`:
> 
> 
>                      VectorBox # DoubleMaxMask, generated by VectorMask.fromLong()
>                        /    \
>                      AddP     \
>                       |         \
>                   LoadNClass      \
>    ConP #IntMaxMask    |            |
>       \                |             |
>         \        DecodeNClass       |
>           \       /                |
>             \   /                 |
>              CmpP    ...

Xiaohong Gong has updated the pull request incrementally with one additional commit since the last revision:

  Ensure it is vector type for vector unbox result

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/29057/files
  - new: https://git.openjdk.org/jdk/pull/29057/files/cf73b3ad..328140eb

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=29057&range=02
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=29057&range=01-02

  Stats: 8 lines in 1 file changed: 4 ins; 2 del; 2 mod
  Patch: https://git.openjdk.org/jdk/pull/29057.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/29057/head:pull/29057

PR: https://git.openjdk.org/jdk/pull/29057