<i18n dev> RFR: 8262744: Formatter '%g' conversion uses wrong format for BigDecimal rounding up to limits [v3]

[Stuart Marks]

On Fri, 16 Apr 2021 16:08:53 GMT, Ian Graves <igraves at openjdk.org> wrote:

>> This fixes a bug where the formatting code for `%g` flags incorrectly tries to round `BigDecimal` after determining whether it should be a decimal numeric format or a scientific numeric format. The solution rounds before determining the correct format.
>
> Ian Graves has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Inlining some single use variables

src/java.base/share/classes/java/util/Formatter.java line 3827:

> 3825:                 if ((value.equals(BigDecimal.ZERO))
> 3826:                     || ((value.compareTo(BigDecimal.valueOf(1, 4)) != -1)
> 3827:                         && (value.compareTo(BigDecimal.valueOf(1, -prec)) == -1))) {

Note that `compareTo` in general specifies a negative, zero, or positive return value, but BigDecimal and BigInteger specify a return value of -1, 0, and 1. So the code here that compares against -1 is strictly correct. However, the BigDecimal/BigInteger.compareTo docs say "The suggested idiom..." is a relative comparison against zero.

Indeed, the BigDecimal::compareTo method does always seem to return -1, 0, or 1 so this code is not incorrect. Well, maybe. I checked quickly and the BigDecimal comparison logic is fairly intricate (and also runs through BigInteger) so I might have missed something. Also, BigDecimal is subclassable, so an override of `compareTo` might return something other than -1, 0, or 1, even though strictly speaking this would violate the BigDecimal spec.

I'm wondering if there should be a followup bug that changes these tests to `>= 0` and `< 0`.

-------------

PR: https://git.openjdk.java.net/jdk/pull/3363