RFR: 8349637: Integer.numberOfLeadingZeros outputs incorrectly in certain cases

[Jatin Bhateja]

On Wed, 12 Feb 2025 09:08:14 GMT, Jatin Bhateja <jbhateja at openjdk.org> wrote:

>> Hi all,
>> This is a fix for a miscompile in the AVX2 implementation of `CountLeadingZerosV` for int types. Currently, the implementation turns ints into floats, in order to calculating the leading zeros based on the exponent part of the float. Unfortunately, floats can only accurately represent integers up to 2^24. After that, multiple integer values can map onto the same floating point value. The issue manifests when an int is converted to a floating point representation that is higher than it, crossing a bit boundary. As an example, `(float)0x01FFFFFF == (float)0x02000000`, but `lzcnt(0x01FFFFFF) == 7` and `lzcnt(0x02000000) == 6`. The values are incorrectly rounded up.
>> 
>> This patch fixes the issue by masking the input in the cases where it is larger than 2^24, to set the low bits to 0. Removing these bits prevents the accidental rounding behavior. I've added these cases to`TestNumberOfContinuousZeros`, and removed the set random seed so that it can produce random inputs to test with.
>> 
>> Reviews would be appreciated!
>
> test/hotspot/jtreg/compiler/vectorization/TestNumberOfContinuousZeros.java line 45:
> 
>> 43: 
>> 44: public class TestNumberOfContinuousZeros {
>> 45:     private static final int[] SPECIAL = { 0x01FFFFFF, 0x03FFFFFE, 0x07FFFFFC, 0x0FFFFFF8, 0x1FFFFFF0, 0x3FFFFFE0 };
> 
> Can you also check for 0xFFFFFFFF, even though we have special handling for -ve signed numbers not affected by this patch.

Can you kindly write an exhaustive functional correctness test that covers entier positive integral value range. Idea here is to test all the cases where unintended exponent increment can lead to incorrect results.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/23579#discussion_r1952484248