RFR: 8316533: C2 compilation fails with assert(verify(phase)) failed: missing Value() optimization

[Roberto Castañeda Lozano]

On Wed, 25 Oct 2023 14:29:07 GMT, Emanuel Peter <epeter at openjdk.org> wrote:

> **Problem**
> We have a `abstract` class `A` with no subtype. Hence, a reference of type `A` must always be `null` (unless a subclass were to be loaded, which we guard against with a compile dependency).
> 
> But there are at least these two ways a `A:NotNull` can be created:
> - Null-Check: CastPP after null-check improves type from `A` to `A:NotNull`.
> - Forced compilation (eg CTW) of a member method of `A`. Then `Parm0` has type `A`, which is improved to `A:NotNull` because the `this/self` pointer cannot be `null`.
> 
> This means we are now left with an impossible type `A:NotNull`, a path that uses this type will never be executed.
> 
> The question is now what should happen at a `SubTypeCheck` if we do:
> `SubTypeCheck(  oop #A:NotNull   ,   constant-classptr-A-exact )`
> 
> The verification happens because we do these two different things:
> - `SubTypeCheck`: we first detect that we have a constant classptr of a class `A`, which is abstract and has no subtype. Hence, we conclude that any oop compared to it cannot be a subtype (there are no subtypes), and it cannot be of the same type (class is abstract). Hence, any oop must be a supertype (TypeInt::CC_GT).
> - The verification code computes the subtype check by computing the klass of the oop via `LoadKlass` (this constant folds to `constant-classptr-A-exact`, because the type of the oop is `A:NotNull`). The `CmpP` node compares the two klasses, and sees that they are identical, and returns an `TypeInt::CC_EQ`.
> 
> **Alternatives**
> 
> Both results are reasonable, but they are in fact both supersets of the true result. We should take the intersection of the two and get `Type:TOP`, since the input type is already impossible. In fact, it would be best if the impossible type was never created. We could do that by improving `CmpP` to detect the impossible type and constant fold towards the `null` path, removing the `A:NotNull` path. It is harder to deal with the forced-compilation of non-static methods of an abstract class with no subclasses - here we would basically have to forbid compilation or replace the compilation with a `Halt`.
> 
> **Solution**
> 
> Instead, I have now decided to change the logic in `SubTypeCheckNode` to return `EQ` in case the oop has the same klass and is `NotNull`.
> 
> **Testing**
> 
> Tier1-6 and stress testing. Running...

test/hotspot/jtreg/compiler/types/TestSubTypeOfAsbtractClass.java line 67:

> 65: }
> 66: 
> 67: 

Please remove these extra line breaks.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/16361#discussion_r1380063425