RFR: 8375653: C2: CmpUNode::sub is not monotonic [v3]

[Christian Hagedorn]

On Thu, 22 Jan 2026 07:49:32 GMT, Quan Anh Mai <qamai at openjdk.org> wrote:

>> Hi,
>> 
>> This PR fixes the issue that `CmpUNode::sub` is not monotonic. The root cause is that it returns different values for several cases, but the cases are not mutually exclusive and the return values are not a subset of each other. This leads to the possibilities that a node satisfying both cases will return the first value, but if upon being widen it ceases to satisfy the first case but still satisfies the second case, the method will return the second value, which is not a superset of the previous result.
>> 
>> For example, given `r = CmpU(x, y)`.
>> 
>> At the first iteration, `type(x) = {0}` and `type(y) = {1, -1}`, then `CmpUNode::sub` returns `TypeInt::CC_LE` since it sees that `x` is the constant `0`.
>> 
>> At the second iteration, `type(x) = {0, 2}` and `type(y) = {-1, 1}`, then `CmpUNode::sub` returns `TypeInt::CC_NE` since it sees that `x` and `y` do not overlap. This is not a superset of `TypeInt::CC_LE`, which leads to an assertion.
>> 
>> Please take a look and leave your reviews, thanks a lot.
>
> Quan Anh Mai has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Test Long::compareUnsigned

Otherwise, the fix looks good and is nicely solved with unsigned types!

I agree with @eme64 that we should have better coverage either with IR tests and/or gtests. Since gtests are out of question for JDK 26, I suggest to have some basic IR test coverage for your improvement using unsigned types. We can then still come back in JDK 27 with gtests.

test/hotspot/jtreg/compiler/ccp/TestCmpUMonotonicity.java line 29:

> 27:  * @bug 8375653
> 28:  * @summary Test that CmpUNode::sub conforms monotonicity
> 29:  * @run main/othervm -Xcomp -XX:CompileCommand=compileonly,${test.main.class}::test ${test.main.class}

Was hard to spot but this will not compile anything since we match on `test` instead of `test*`:

Suggestion:

 * @run main/othervm -Xcomp -XX:CompileCommand=compileonly,${test.main.class}::test* ${test.main.class}

-------------

Changes requested by chagedorn (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/29308#pullrequestreview-3692119300
PR Review Comment: https://git.openjdk.org/jdk/pull/29308#discussion_r2716625343