RFR: 8369902: C2 SuperWord: wrong result because filterin NaN instead of zero in MemPointerParser::canonicalize_raw_summands [v4]

[Vladimir Kozlov]

On Mon, 20 Oct 2025 07:39:49 GMT, Emanuel Peter <epeter at openjdk.org> wrote:

>> **TLDR** `is_NaN` -> `is_zero`, just like the code comment says.
>> 
>> Thanks to @mhaessig for debugging the ARM32 bug below. He found the buggy line of code.
>> 
>> ----------------------------------------
>> 
>> **Details**
>> 
>> It seems there is a little "typo" (logic error) in `MemPointerParser::canonicalize_raw_summands` that slipped through the cracks in https://github.com/openjdk/jdk/pull/24278. The JavaFuzzer now found an example, and independently the issue was also reported on ARM32 [JDK-8368578](https://bugs.openjdk.org/browse/JDK-8368578).
>> 
>> Filtering out `NaN` instead of `zero` for the `scaleL` has two manifestations:
>> - If `scaleL` is zero, but does not get filtered out even though it should be: we hit the assert in `MemPointerSummand` constructor, `assert(!_scale.is_zero(), "non-zero scale");`.
>>   - See [JDK-8368578](https://bugs.openjdk.org/browse/JDK-8368578), though those tests seem to only fail on ARM32, and nowhere else.
>>   - I was able to construct a `MemorySegment` regression test, see `TestMemorySegmentFilterSummands.test1`. I suspect that the ARM32 failures happened on an array, as it failed in places like `BigInteger::implMultiplyToLen`. But now I was able to reproduce it with native memory, to get a pointer expression that has the same cancellation issue.
>> - If `scaleL` is `NaN`, and gets filtered even though it should not be: We get a non-trivial MemPointer that is missing a summand. So we will succeed in optimizing, but with wrong assumptions. We generate a runtime aliasing check that is incorrect, leading to wrong results.
>>   - This was reported by the fuzzer, see attached `TestDoNotFilterNaNSummands`.
>>   - I was also able to create a simpler example with `MemorySegments`, see attached `TestMemorySegmentFilterSummands.test2`.
>> 
>> **Why did this slip through the cracks?**
>> 
>> In https://github.com/openjdk/jdk/pull/24278 I added pretty extensive testing, even fuzzer style tests, see `TestAliasingFuzzer.java`. But I think all of those tests exercise `scale` that are in "nice" [int ranges](https://github.com/openjdk/jdk/pull/24278/files#diff-26de03e864a492fe8aa8178818968f2097b99cf36a763605e2fb11fbc04eedffR303-R322). Also the JavaFuzzer does not directly generate such long constants for array accesses (not possible without Unsafe I think), we were lucky that it generated the index with `%` that got optimized to some magic long constant.
>> 
>> There is already an RFE filed for improvements to `TestAliasingFuzzer.java`: [JDK-83...
>
> Emanuel Peter has updated the pull request incrementally with one additional commit since the last revision:
> 
>   fix up manuel's suggestion

Looks fine.

-------------

Marked as reviewed by kvn (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/27848#pullrequestreview-3357105039