JDK 14 RFR of JDK-8233452: java.math.BigDecimal.sqrt() with RoundingMode.FLOOR results in incorrect result

[Brian Burkhalter]

Hi Joe,

> On Nov 5, 2019, at 6:04 PM, Joe Darcy <joe.darcy at oracle.com> wrote:
> 
> Please review the changes to fix
> 
>     JDK-8233452: java.math.BigDecimal.sqrt() with RoundingMode.FLOOR results in incorrect result
>     http://cr.openjdk.java.net/~darcy/8233452.0/ <http://cr.openjdk.java.net/~darcy/8233452.0/>
> 
> Some background on the problem and fix.
> 
> The core of the BigDecimal.sqrt method is a Newton-Raphson loop.
> […]
> The Newton iteration reduces the error at each step and, as currently written, it can settle on a value like 2, which is closer to the actual answer, but *incorrect* according to the requested rounding mode.
> 
> There are a few ways to fix this one. One is to run the Newton loop to a higher-precision where these finer distinctions can be teased out. […]
> 
> Another approach is to detect when the result is too large/too small and then subtract/add an ulp as a fix-up. This is the approach taken for BigDecimal.sqrt. […]

This implementation change looks reasonable to me.

> An approach not explored for this fix would be to arrange for the iteration to start from the "right" side of the number line depending on the rounding mode and then monotonically increase/decrease to approach the square root. […] I believe this is technically possible, but would require some additional analysis to setup.

This approach seems more appealing and could perhaps be investigated at a later date.

In the test I think you want to replace “down” with “up" at line 238.

Brian