RFR: 8341470: BigDecimal.stripTrailingZeros() optimization [v33]

[Raffaello Giulietti]

On Fri, 11 Oct 2024 18:20:50 GMT, fabioromano1 <duke at openjdk.org> wrote:

>> IIUC, the code assumes that the floating-point computation `Math.ceil(intVal.bitLength() * LOG_5_OF_2)` has the same value as the mathematical ⌈ `intVal.bitLength()` × log5(2) ⌉.
>> I don't think this is safe, as it might happen that the computed and mathematical values are off by ±1.
>> To ensure 5^`maxPowsOf5` >= `intVal` (that is, maxPowsOf5 >= log5(intVal)) it would be more prudent to have
>> 
>> long maxPowsOf5 = (long) Math.ceil(intVal.bitLength() * LOG_5_OF_2) + 1;
>> 
>> 
>> But I think what you really want is maybe to meet 5^`maxPowsOf5` <= `intVal` < 5^(`maxPowsOf5` + 1) instead?
>
> Actually, if we reason in terms of "ulp vs precision", the computation should be safe: `Math.log()`'s results are within 1 ulp of the exact result, and the floating point operations are a multiplication and a division. The division to compute `LOG_5_OF_2` costs 1/2 ulp plus the errors of the operands, so 2.5 ulps. Same for multiplication, but `intVal.bitLength()` has an exact `double` value, so the total roundoff error of `intVal.bitLength() * LOG_5_OF_2` is 3 ulps. Since the integer part of `intVal.bitLength() * LOG_5_OF_2` is representable with 31 bits, and double has 53 bits of precision, we can reasonably say that `Math.ceil()` can always guarantee `maxPowsOf5 >= log5(intVal)`.

If the mathematical value v of the product and its floating-point value fp are separated by an integer i in such a way that fp < i < v, we are in trouble: the ceilings will be different, even if the values are very close to each other.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/21323#discussion_r1797302468