Arithmetic error in SunEC

[Tobias Wagner]

Hi,

we found an error in the GF(p)-arithmetics of SunEC, while adding
support for brainpool-curves in ECDHE for TLS connections as
suggested in RFC 7027. BrainpoolP256r1 and brainpoolP512r1 worked 
out of the box, but brainpoolP384r1 did not. The calculated public keys were
not on that curve and thus we got handshake failures.

After debugging the key generation during an TLS handshake, we came to
the conclusion, that there must be a flaw in ec_GFp_sub_6 from ecl_gf.c.
Using ec_GFp_sub instead worked with brainpoolP384r1.

Researching further on that issue, we learned that SunEC originates
from Mozilla's NSS library and that the same error was reported to
Mozilla three years ago
(https://bugzilla.mozilla.org/show_bug.cgi?id=1128140):

> The functions ec_GFp_sub_5 and ec_GFp_sub_6 are missing an additional:
> 
> MP_ADD_CARRY(b4, r4, r4, borrow, borrow)
> 
> and
> 
> MP_ADD_CARRY(b5, r5, r5, borrow, borrow)
> 
> in the /* Do quick 'add' if we've gone under 0
> * (subtract the 2's complement of the curve field) *
> check.

It has eventually been fixed last year
(https://hg.mozilla.org/projects/nss/rev/d81d6127781e).

The attached patch would fix the issue on JDK9 in the same manner as it
was done in Mozilla's NSS. The issue is also present in JDK8 and
probably in earlier JDKs as well.

Regards
Tobias

-- 
phone: +49 221 222896 17
fax: +49 221 222896 11
keybase: https://keybase.io/toebix

n - d e s i g n   G m b H 
www.n-design.de
Alpenerstr. 16
D-50825 Köln

Amtsgericht Köln HRB 33766 B
Geschäftsführer Andy Kohl
-------------- next part --------------
A non-text attachment was scrubbed...
Name: jdk9_jdk_17287.patch
Type: application/octet-stream
Size: 1217 bytes
Desc: not available
URL: <https://mail.openjdk.org/pipermail/security-dev/attachments/20171017/3d486789/jdk9_jdk_17287.patch>