RFR: 8374043: C2: assert(_base >= VectorMask && _base <= VectorZ) failed: Not a Vector

[Xiaohong Gong]

### Problem:

Test `compiler/vectorapi/VectorMaskToLongTest.java` crashes intermittently (approximately once per 200+ runs) with stress VM options such as `-XX:+StressIGVN`:


// A fatal error has been detected by the Java Runtime Environment:
//
// Internal Error (jdk/src/hotspot/share/opto/type.hpp:2287), pid=69056, tid=28419
// assert(_base >= VectorMask && _base <= VectorZ) failed: Not a Vector
// ...


The crash occurs in following code when calling `is_vect()` in the assertion added by JDK-8367292 [1]:

https://github.com/openjdk/jdk/blob/2cb228e142369ec73d768d8a69653a984b1c5908/src/hotspot/share/opto/vectornode.cpp#L1920-L1924

### Root Cause:

The mask's type becomes TOP (unreachable) during compiler optimizations when the mask node is marked as dead before all its users are removed from the ideal graph. If `Ideal()` is subsequently called on a user node, it may access the TOP type, triggering the assertion.

Here is the simplified ideal graph showing the crash scenario:


      Con #top
       |       ConI
         \      /
           \  /
     VectorStoreMask
             |
         VectorMaskToLong  # !jvms: IntMaxVector$IntMaxMask::toLong


### Detailed Scenario:

Following is the method in the test case that hits the assertion: 

https://github.com/openjdk/jdk/blob/2cb228e142369ec73d768d8a69653a984b1c5908/test/hotspot/jtreg/compiler/vectorapi/VectorMaskToLongTest.java#L65-L70

This method accepts a `VectorSpecies<?>` parameter and calls vector APIs `VectorMask.fromLong()` and `toLong()`. It is called with species ranging from `ByteVector.SPECIES_MAX` to `DoubleVector.SPECIES_MAX`. During compilation, C2 speculatively generates fast paths for `toLong()` for all possible species. 

When compiling a specific test case such as:
https://github.com/openjdk/jdk/blob/6eaabed55ca4670d8c317f0a4323ccea4dd0b9ca/test/hotspot/jtreg/compiler/vectorapi/VectorMaskToLongTest.java#L177-L179

the compiler inlines the method and attempts to optimize away unreachable branches. The following graph shows the situation before the mask becomes `TOP`:


                     VectorBox # DoubleMaxMask, generated by VectorMask.fromLong()
                       /    \
                     AddP     \
                      |         \
                  LoadNClass      \
   ConP #IntMaxMask    |            |
      \                |             |
        \        DecodeNClass       |
          \       /                |
            \   /                 |
             CmpP                |
              |                 |
             Bool #ne          |
              |              /
             If            /
              |          /
           IfFalse     /
              |      /
              |    /
          CheckCastPP  # IntMaxMask
              |
         VectorUnbox  # Start of inlining IntMaxMask::toLong()
              |
               \     ConI
                \    /
           VectorStoreMask
                   |
            VectorMaskToLong


The generated mask (`VectorBox`) is a `DoubleMaxMask`, but the code path expects an `IntMaxMask` for `IntMaxMask::toLong()`. Since this is an unreachable branch, the control input of `CheckCastPP` becomes `TOP` during IGVN, propagating the `TOP` type to subsequent data nodes until reaching `VectorStoreMask`. `VectorStoreMask` has another non-TOP input (`ConI`), which stops further `TOP` propagation.

With stress VM options, the IGVN worklist order is shuffled, causing `VectorMaskToLongNode::Ideal()` to be invoked before dead path cleanup completes, which triggers the assertion failure.

### Solution:

Replace `is_vect()` with the safer `isa_vect()`, which checks whether the type is a vector type before casting and returns `nullptr` if it is not. Additionally, check for `nullptr` and skip the optimization if the type check fails.

An alternative solution would be to detect `top` inputs during IGVN for the relevant vector nodes and skip certain optimizations when such inputs are encountered. That is probably the right long-term direction. However, because this handling is currently missing for all vector nodes, I'd like to leave it as a separate follow-up topic for discussion.

### Testing:

Ran the test 800 times on SVE/NEON/AVX2 systems with no failures observed.

Note that no new test case was added because it is so challenging to me to reproduce this issue reliably. The issue depends on a specific IGVN optimization sequence that occurs non-deterministically due to the worklist shuffling behavior under stress VM options.

[1] https://bugs.openjdk.org/browse/JDK-8367292

-------------

Commit messages:
 - 8374043: C2: assert(_base >= VectorMask && _base <= VectorZ) failed: Not a Vector

Changes: https://git.openjdk.org/jdk/pull/29057/files
  Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=29057&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8374043
  Stats: 12 lines in 2 files changed: 7 ins; 0 del; 5 mod
  Patch: https://git.openjdk.org/jdk/pull/29057.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/29057/head:pull/29057

PR: https://git.openjdk.org/jdk/pull/29057