RFR: 8258833: Cancel multi-part cipher operations in SunPKCS11 after failures

[Martin Balao]

On Thu, 7 Jan 2021 18:56:03 GMT, Martin Balao <mbalao at openjdk.org> wrote:

> In summary, I believe we need changes in the OpenJDK side to properly handle CKR_BUFFER_TOO_SMALL errors when C_SignFinal or C_Sign PKCS#11 functions are called from P11Signature. Even if other error types or functions such as C_VerifyFinal, C_Verify, NSC_SignUpdate and NSC_VerifyUpdate should not need an explicit cancel; we might want to do it anyways to be more defensive and do not depend on a specific NSS implementation.
> 

I'm not entirely sure that we can trigger the CKR_BUFFER_TOO_SMALL potential bug from OpenJDK because the output buffer is allocated in the OpenJDK native code [1] for C_Sign and the length is equal to 4K [2]. In the case of C_SignFinal, the CKR_BUFFER_TOO_SMALL error is handled in native code [3].

My understanding is that we still want to be defensive here, and do not depend on a specific NSS version that may leak active operations on some types of errors. The difference is that this change in P11Signature won't have a regression test.

@valeriepeng  are you okay with this reasoning?

--
[1] - https://github.com/openjdk/jdk/blob/master/src/jdk.crypto.cryptoki/share/native/libj2pkcs11/p11_sign.c#L142
[2] - https://github.com/openjdk/jdk/blob/master/src/jdk.crypto.cryptoki/share/native/libj2pkcs11/pkcs11wrapper.h#L166
[3] - https://github.com/openjdk/jdk/blob/master/src/jdk.crypto.cryptoki/share/native/libj2pkcs11/p11_sign.c#L252

-------------

PR: https://git.openjdk.java.net/jdk/pull/1901