RFR: 8294906: Memory leak in PKCS11 NSS TLS server [v2]

[Valerie Peng]

On Fri, 7 Oct 2022 07:42:25 GMT, Daniel Jeliński <djelinski at openjdk.org> wrote:

>> C_DeriveKey with mechanisms `CKM_*_KEY_AND_MAC_DERIVE` always returns mac keys, even if macBits is zero. These keys must be free'd when no longer needed.
>> 
>> Verified that:
>> - SSL server configured with PKCS11-NSS provider leaks memory without this patch, does not leak memory with this patch
>> - The same server continues to function correctly
>> - Existing tier1-3 tests continue to pass with NSS; did not test any other PKCS11 providers
>> - new tests for AES-128-GCM-SHA256 and AES-256-GCM-SHA384 key derivation pass
>
> Daniel Jeliński has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Check if mac keys are present

Changes look fine, just have one nit.
Thanks for finding and fixing this!

BTW, if this NSS behavior is still present in newer NSS releases, we should raise the priority of this and backport it.

test/jdk/sun/security/pkcs11/tls/TestKeyMaterial.java line 150:

> 148:                         clientRandom, serverRandom, cipherAlgorithm,
> 149:                         keyLength, expandedKeyLength, ivLength, macLength,
> 150:                         hashAlgorithm, -1, -1);

nit: add a comment that PKCS11 does not use the prfHashLength and prfBlockSize fields. So that's why this test does not set the values as in the com/sun/crypto/provider/TLS/TestKeyMaterial.java file.

-------------

Marked as reviewed by valeriep (Reviewer).

PR: https://git.openjdk.org/jdk/pull/10594