RFR: 8349637: Integer.numberOfLeadingZeros outputs incorrectly in certain cases

[Jasmine Karthikeyan]

On Wed, 12 Feb 2025 06:45:33 GMT, Quan Anh Mai <qamai at openjdk.org> wrote:

>> Hi all,
>> This is a fix for a miscompile in the AVX2 implementation of `CountLeadingZerosV` for int types. Currently, the implementation turns ints into floats, in order to calculating the leading zeros based on the exponent part of the float. Unfortunately, floats can only accurately represent integers up to 2^24. After that, multiple integer values can map onto the same floating point value. The issue manifests when an int is converted to a floating point representation that is higher than it, crossing a bit boundary. As an example, `(float)0x01FFFFFF == (float)0x02000000`, but `lzcnt(0x01FFFFFF) == 7` and `lzcnt(0x02000000) == 6`. The values are incorrectly rounded up.
>> 
>> This patch fixes the issue by masking the input in the cases where it is larger than 2^24, to set the low bits to 0. Removing these bits prevents the accidental rounding behavior. I've added these cases to`TestNumberOfContinuousZeros`, and removed the set random seed so that it can produce random inputs to test with.
>> 
>> Reviews would be appreciated!
>
> src/hotspot/cpu/x86/c2_MacroAssembler_x86.cpp line 6242:
> 
>> 6240:   vpxor(xtmp2, xtmp2, xtmp2, vec_enc);
>> 6241:   vpsrld(xtmp2, src, 24, vec_enc);
>> 6242:   vpandn(src, xtmp2, src, vec_enc);
> 
> Are you sure this will work? I don't see how `x &~ (x >> 24)` can zero out all lower bits. Also, we should not kill `src` here.
> 
> I think a better solution is to do: `x > 0xFF ? x &~ 0xFF : x`. I chose this value because we have `0xFF` in `xtmp1` here.

I believe the solution should work since my idea was to get rid of low bits when the high bits past 24 are all `1`, as that is the case where rounding behavior can incorrectly bump up the value. In the other cases, not removing all the low bits shouldn't have an impact on rounding and the exponent. At least, when running an [exhaustive test](https://gist.github.com/jaskarth/6b05352c3007a2650bf084fcb4c50c13) I wrote this patch fixes the output discrepancy. I can change it to the compare and blend you suggested, but I think that will be slower than this approach. Killing `src` is my mistake though, I will fix it in the next commit.

> test/hotspot/jtreg/compiler/vectorization/TestNumberOfContinuousZeros.java line 62:
> 
>> 60:         inputInt = new int[LEN];
>> 61:         outputInt = new int[LEN];
>> 62:         rng = new Random();
> 
> Please use `Util.getRandomInstance()`

Nice catch! I've updated it.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/23579#discussion_r1952983061
PR Review Comment: https://git.openjdk.org/jdk/pull/23579#discussion_r1952983344