RFR: 8309569: sun/security/pkcs11/Signature/TestRSAKeyLength.java fails after JDK-8301553

[Valerie Peng]

On Wed, 7 Jun 2023 20:24:12 GMT, Martin Balao <mbalao at openjdk.org> wrote:

> We would like to propose a fix for 8309569. In this bug, a Java signature buffer of length 0 is passed to sun.security.pkcs11.wrapper.PKCS11::C_VerifyFinal to finish an ongoing signature verification operation. Notice that finishing the operation by means of C_SessionCancel was either not tried —as in software tokens implementing a standard of PKCS # 11 previous to 3— or failed, and finishing the operation with C_VerifyFinal is used as a last resort. Before calling the native C_VerifyFinal function, the signature buffer is converted from Java to native in jByteArrayToCKByteArray. Previous to 8301553, calloc was called with length 0 and, in most platforms, returns an address different from NULL. This non-NULL value was okay to the NSS Software Token [1], nothing was read because the length was 0 and the operation was finally cancelled. After 8301553, we made jByteArrayToCKByteArray to return NULL in cases of length 0 in the spirit of aligning different calloc implementations to 
 a single path. The scope of the change is not limited to jByteArrayToCKByteArray but to other 4 analogous conversion functions in p11_util.c.
> 
> We propose to revert the behavior to the state previous to 8301553 —in other words, calling calloc in cases of length 0— but avoiding an OOM exception if calloc returns NULL because of a length 0. For implementations where calloc returns NULL upon a 0 length, an OOM exception does not reflect what really happened. In these cases, it's up to the native PKCS # 11 library to handle the case.
> 
> --
> [1] - https://github.com/nss-dev/nss/blob/NSS_3_67_RTM/lib/softoken/pkcs11c.c#L3823

Looks good.

-------------

Marked as reviewed by valeriep (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/14369#pullrequestreview-1468783187