RFR: 8324585: JVM native memory leak in PCKS11-NSS security provider [v2]

[Valerie Peng]

On Wed, 31 Jan 2024 10:23:22 GMT, Daniel Jeliński <djelinski at openjdk.org> wrote:

>> Please review this patch that fixes a memory leak in P11TlsPrfGenerator, which is triggered during TLS1.2 Finished message generation and verification.
>> 
>> The patch changes C_SignInit JNI method to free the mechanism data immediately after use. This matches the behavior of other Init methods (like C_EncryptInit). The patch also fixes a similar issue in other signature-related methods.
>> 
>> The change essentially reverts part of [JDK-8080462](https://bugs.openjdk.org/browse/JDK-8080462).
>> 
>> All sun/security/pkcs11 tests still pass with NSS ~3.35 and~ 3.91. All tier1-3 tests still pass.
>> 
>> EDIT:
>> Some sun/security/pkcs11 tests fail with NSS 3.64 and older, see [comment](https://github.com/openjdk/jdk/pull/17584#issuecomment-1914665234)
>
> Daniel Jeliński has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Disable RSA-PSS in known bad NSS versions

Just a minor wording suggestion. 
Disabling older NSS releases due to known issue is the best that we can do to work with multiple NSS releases out there.

src/jdk.crypto.cryptoki/share/classes/sun/security/pkcs11/SunPKCS11.java line 1311:

> 1309:             if (brokenMechanisms.contains(longMech)) {
> 1310:                 if (showInfo) {
> 1311:                     System.out.println("DISABLED due to broken provider");

nit: how about changing "broken provider" to "known issue with NSS"?

-------------

Marked as reviewed by valeriep (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/17584#pullrequestreview-1855140171
PR Review Comment: https://git.openjdk.org/jdk/pull/17584#discussion_r1473648397