[lworld] Integrated: 8373878: [lworld] C2: Assertions about flat in array properties are too strong in TypePtr::compute_flat_in_array_if_unknown()

[Christian Hagedorn]

On Wed, 25 Feb 2026 12:16:53 GMT, Christian Hagedorn <chagedorn at openjdk.org> wrote:

> `compute_flat_in_array_if_unknown()` computes the flat in array property if we don't know it. We additionally added some assertions that when we have `FlatInArray::NotFlat` or `FlatInArray::Flat`, we are either dealing with a non-value class or a value class, respectively. This worked in most cases.
> 
> We now found cases with type speculation where this breaks: We could speculatively assume that something is `FlatInArray::NotFlat/Flat`, propagate that further and then later find that this contradicts the flat in array property of some non-speculative type.
> 
> I've first tried to avoid introducing these inconsistencies when starting to use a speculative type. This sometimes work but not always: We could find an inconsistency way further down in the graph later which is then hard to fix.
> 
> I therefore propose to accept the fact that the flat in array property can be inconsistent (i.e. being not flat in array, when actually dealing with an always flat in array value class etc.):
> - We set not flat in array even though the type could theoretically be flat in array.
> - We set flat in array even though the type could theoretically be non flat in array.
> 
> We only use the flat in array property for `SubTypeCheck` and `CmpP` nodes in `Value()`. When facing such an inconsistency, we could end up "wrongly" folding a compare node. But in that case, we would trap earlier because the speculated type turns out to be wrong (see `testSuperNotFlatSubFlat()` as an example).
> 
> To make this work properly, we also need to adjust `SubTypeCheck::sub()` to not only check the case when the super class is flat in array and the sub class is not but also the other way round (this missing piece was now also found as a missed optimization opportunity in the CI independently of these changes).
> 
> Thanks,
> Christian

This pull request has now been integrated.

Changeset: 94f14342
Author:    Christian Hagedorn <chagedorn at openjdk.org>
Committer: Tobias Hartmann <thartmann at openjdk.org>
URL:       https://git.openjdk.org/valhalla/commit/94f14342eaa6ef40bcb7926eac401affa9807312
Stats:     274 lines in 5 files changed: 255 ins; 7 del; 12 mod

8373878: [lworld] C2: Assertions about flat in array properties are too strong in TypePtr::compute_flat_in_array_if_unknown()

Reviewed-by: thartmann

-------------

PR: https://git.openjdk.org/valhalla/pull/2162