RFR: 8349637: Integer.numberOfLeadingZeros outputs incorrectly in certain cases

[Jasmine Karthikeyan]

On Wed, 12 Feb 2025 06:02:05 GMT, Joe Darcy <darcy at openjdk.org> wrote:

>> Hi all,
>> This is a fix for a miscompile in the AVX2 implementation of `CountLeadingZerosV` for int types. Currently, the implementation turns ints into floats, in order to calculating the leading zeros based on the exponent part of the float. Unfortunately, floats can only accurately represent integers up to 2^24. After that, multiple integer values can map onto the same floating point value. The issue manifests when an int is converted to a floating point representation that is higher than it, crossing a bit boundary. As an example, `(float)0x01FFFFFF == (float)0x02000000`, but `lzcnt(0x01FFFFFF) == 7` and `lzcnt(0x02000000) == 6`. The values are incorrectly rounded up.
>> 
>> This patch fixes the issue by masking the input in the cases where it is larger than 2^24, to set the low bits to 0. Removing these bits prevents the accidental rounding behavior. I've added these cases to`TestNumberOfContinuousZeros`, and removed the set random seed so that it can produce random inputs to test with.
>> 
>> Reviews would be appreciated!
>
> src/hotspot/cpu/x86/c2_MacroAssembler_x86.cpp line 6237:
> 
>> 6235:   vpsrld(xtmp1, xtmp1, 24, vec_enc);
>> 6236: 
>> 6237:   // As 2^24 is the largest possible integer that can be exactly represented by a float value, special handling has to be
> 
> More exactly, +/- 2^24 is the region were all contiguous integers can be represented as floats. All sufficiently large finite floating-point numbers are integers, but the distance between adjacent floating-point numbers is more than 1.

Thanks for the clarification! I've adjusted the wording to be more clear.

> test/hotspot/jtreg/compiler/vectorization/TestNumberOfContinuousZeros.java line 1:
> 
>> 1: /*
> 
> As a one-off test, in other words not a test to be checked in and run continuously, it would be reassuring to test al the int values are make sure the intrinsic is computing the same result as the Java library version.

I ended up writing this test while debugging the patch to exhaustively check the int range: https://gist.github.com/jaskarth/6b05352c3007a2650bf084fcb4c50c13

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/23579#discussion_r1952926301
PR Review Comment: https://git.openjdk.org/jdk/pull/23579#discussion_r1952927124