[lworld] RFR: 8373878: [lworld] C2: Assertions about flat in array properties are too strong in TypePtr::compute_flat_in_array_if_unknown() [v2]

[Tobias Hartmann]

On Wed, 25 Feb 2026 12:46:15 GMT, Christian Hagedorn <chagedorn at openjdk.org> wrote:

>> `compute_flat_in_array_if_unknown()` computes the flat in array property if we don't know it. We additionally added some assertions that when we have `FlatInArray::NotFlat` or `FlatInArray::Flat`, we are either dealing with a non-value class or a value class, respectively. This worked in most cases.
>> 
>> We now found cases with type speculation where this breaks: We could speculatively assume that something is `FlatInArray::NotFlat/Flat`, propagate that further and then later find that this contradicts the flat in array property of some non-speculative type.
>> 
>> I've first tried to avoid introducing these inconsistencies when starting to use a speculative type. This sometimes work but not always: We could find an inconsistency way further down in the graph later which is then hard to fix.
>> 
>> I therefore propose to accept the fact that the flat in array property can be inconsistent (i.e. being not flat in array, when actually dealing with an always flat in array value class etc.):
>> - We set not flat in array even though the type could theoretically be flat in array.
>> - We set flat in array even though the type could theoretically be non flat in array.
>> 
>> We only use the flat in array property for `SubTypeCheck` and `CmpP` nodes in `Value()`. When facing such an inconsistency, we could end up "wrongly" folding a compare node. But in that case, we would trap earlier because the speculated type turns out to be wrong (see `testSuperNotFlatSubFlat()` as an example).
>> 
>> To make this work properly, we also need to adjust `SubTypeCheck::sub()` to not only check the case when the super class is flat in array and the sub class is not but also the other way round (this missing piece was now also found as a missed optimization opportunity in the CI independently of these changes).
>> 
>> Thanks,
>> Christian
>
> Christian Hagedorn has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains three commits:
> 
>  - Fix problemlist
>  - Merge branch 'lworld' into JDK-8373878
>    
>    # Conflicts:
>    #	test/jdk/ProblemList-enable-preview.txt
>  - 8373878: [lworld] C2: Assertions about flat in array properties are too strong in TypePtr::compute_flat_in_array_if_unknown()

Thanks for investigating this thoroughly. Looks good to me!

-------------

Marked as reviewed by thartmann (Committer).

PR Review: https://git.openjdk.org/valhalla/pull/2162#pullrequestreview-3854272439