RFR: 8375653: C2: CmpUNode::sub is not monotonic [v5]

[Christian Hagedorn]

On Thu, 22 Jan 2026 15:01:36 GMT, Quan Anh Mai <qamai at openjdk.org> wrote:

>> Hi,
>> 
>> This PR fixes the issue that `CmpUNode::sub` is not monotonic. The root cause is that it returns different values for several cases, but the cases are not mutually exclusive and the return values are not a subset of each other. This leads to the possibilities that a node satisfying both cases will return the first value, but if upon being widen it ceases to satisfy the first case but still satisfies the second case, the method will return the second value, which is not a superset of the previous result.
>> 
>> For example, given `r = CmpU(x, y)`.
>> 
>> At the first iteration, `type(x) = {0}` and `type(y) = {1, -1}`, then `CmpUNode::sub` returns `TypeInt::CC_LE` since it sees that `x` is the constant `0`.
>> 
>> At the second iteration, `type(x) = {0, 2}` and `type(y) = {-1, 1}`, then `CmpUNode::sub` returns `TypeInt::CC_NE` since it sees that `x` and `y` do not overlap. This is not a superset of `TypeInt::CC_LE`, which leads to an assertion.
>> 
>> Please take a look and leave your reviews, thanks a lot.
>
> Quan Anh Mai has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Improve IR tests

Update looks good, thanks!

test/hotspot/jtreg/compiler/c2/gvn/CmpUNodeValueTests.java line 36:

> 34:  * @run driver ${test.main.class}
> 35:  */
> 36: public class CmpUNodeValueTests {

Thanks for adding an IR test!

test/hotspot/jtreg/compiler/ccp/TestCmpUMonotonicity.java line 35:

> 33: public class TestCmpUMonotonicity {
> 34:     public static void main(String[] args) {
> 35:         for (int i = 0; i < 20000; i++) {

But do you really need 20000 iterations to trigger the issue? We end up executing the inner loop 20000*50 times. But it's not much work in the loop, so probably does not matter too much.

-------------

Marked as reviewed by chagedorn (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/29308#pullrequestreview-3696101563
PR Review Comment: https://git.openjdk.org/jdk/pull/29308#discussion_r2719978794
PR Review Comment: https://git.openjdk.org/jdk/pull/29308#discussion_r2719977624