Integrated: 8309569: sun/security/pkcs11/Signature/TestRSAKeyLength.java fails after JDK-8301553

[Martin Balao]

On Wed, 7 Jun 2023 20:24:12 GMT, Martin Balao <mbalao at openjdk.org> wrote:

> We would like to propose a fix for 8309569. In this bug, a Java signature buffer of length 0 is passed to sun.security.pkcs11.wrapper.PKCS11::C_VerifyFinal to finish an ongoing signature verification operation. Notice that finishing the operation by means of C_SessionCancel was either not tried —as in software tokens implementing a standard of PKCS # 11 previous to 3— or failed, and finishing the operation with C_VerifyFinal is used as a last resort. Before calling the native C_VerifyFinal function, the signature buffer is converted from Java to native in jByteArrayToCKByteArray. Previous to 8301553, calloc was called with length 0 and, in most platforms, returns an address different from NULL. This non-NULL value was okay to the NSS Software Token [1], nothing was read because the length was 0 and the operation was finally cancelled. After 8301553, we made jByteArrayToCKByteArray to return NULL in cases of length 0 in the spirit of aligning different calloc implementations to 
 a single path. The scope of the change is not limited to jByteArrayToCKByteArray but to other 4 analogous conversion functions in p11_util.c.
> 
> We propose to revert the behavior to the state previous to 8301553 —in other words, calling calloc in cases of length 0— but avoiding an OOM exception if calloc returns NULL because of a length 0. For implementations where calloc returns NULL upon a 0 length, an OOM exception does not reflect what really happened. In these cases, it's up to the native PKCS # 11 library to handle the case.
> 
> --
> [1] - https://github.com/nss-dev/nss/blob/NSS_3_67_RTM/lib/softoken/pkcs11c.c#L3823

This pull request has now been integrated.

Changeset: 760cb04a
Author:    Martin Balao <mbalao at openjdk.org>
URL:       https://git.openjdk.org/jdk/commit/760cb04a2e099a3af9199d77a234af75a18cce5d
Stats:     39 lines in 2 files changed: 10 ins; 5 del; 24 mod

8309569: sun/security/pkcs11/Signature/TestRSAKeyLength.java fails after JDK-8301553

Co-authored-by: Martin Balao <mbalao at openjdk.org>
Co-authored-by: Francisco Ferrari Bihurriet <fferrari at redhat.com>
Reviewed-by: valeriep

-------------

PR: https://git.openjdk.org/jdk/pull/14369