RFR: 8258833: Cancel multi-part cipher operations in SunPKCS11 after failures

[Valerie Peng]

On Mon, 28 Dec 2020 16:24:43 GMT, Martin Balao <mbalao at openjdk.org> wrote:

> When a multi-part cipher operation fails in SunPKCS11 (i.e. because of an invalid block size), we now cancel the operation before returning the underlying Session to the Session Manager. This allows to use the returned Session for a different purpose. Otherwise, an CKR_OPERATION_ACTIVE error would be raised from the PKCS#11 library.
> 
> The jdk/sun/security/pkcs11/Cipher/CancelMultipart.java regression test is introduced as part of this PR.
> 
> No regressions found in jdk/sun/security/pkcs11.

Thanks for finding this! There have been intermittent CKR_OPERATION_ACTIVE errors which is likely the result of this bug...

Existing SunPKCS11 code is based on the PKCS#11 spec, e.g. a call to C_EncryptUpdate which results in an error other than CKR_BUFFER_TOO_SMALL terminates the current encryption operation, and thus avoid the canceling operation for such scenarios. Ideally, NSS should have properly terminated the operation as the spec described. Knowing the NSS behavior, this defensive-cancellation fix is good.

There are additional SunPKCS11 impl classes which follow the same model as P11Cipher class and may have to be updated with this defensive-cancellation fix as well. How about P11AEADCipher.java, P11RSACipher.java, P11Signature.java, P11PSSSignature.java, P11Mac.java?

Thanks,
Valerie

-------------

PR: https://git.openjdk.java.net/jdk/pull/1901