RFR: 8325247: Memory leak in SessionKeyRef class def when using PKCS11 security provider

Daniel Jeliński djelinski at openjdk.org
Thu Feb 15 11:42:14 UTC 2024


The reported leak was caused by the death of the `Cleanup-SunPKCS11` thread. The cleanup thread in turn died because of an exception thrown from `removeNativeKey` that resulted from 2 threads executing that method at the same time.

This PR adds a reachabilityFence to ensure that the key will only be enqueued for cleanup after the user thread is done with the `removeNativeKey` call.

No new regression test; the issue is extremely hard to reproduce in a reasonable time. Existing tier1-3 tests continue to pass.

In JBS I attached a PoC patch that changes the relative timing of operations; with that patch and without the changes from this PR I am able to reproduce the issue within a few seconds. With the changes from this PR the issue did not reproduce after 10 minutes of testing.

-------------

Commit messages:
 - Fix P11Key double-free

Changes: https://git.openjdk.org/jdk/pull/17870/files
 Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=17870&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8325247
  Stats: 3 lines in 1 file changed: 2 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/jdk/pull/17870.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/17870/head:pull/17870

PR: https://git.openjdk.org/jdk/pull/17870



More information about the security-dev mailing list