RFR: 8313202: MutexLocker should disallow null Mutexes

[David Holmes]

On Wed, 26 Jul 2023 17:06:02 GMT, Aleksey Shipilev <shade at openjdk.org> wrote:

> As seen in [JDK-8313081](https://bugs.openjdk.org/browse/JDK-8313081), it is fairly easy to pass nullptr `Mutex` to `MutexLocker` by accident, which would just silently avoid the lock.
> 
> There are a few places in Hotspot where we pass `nullptr` to simulate re-entrancy and/or conditionally take the lock. Those places can be more explicit, and the default `MutexLocker` can disallow nullptrs for extra safety.
> 
> Open for some bikeshedding on the names of the new `MutexLockers`. Particularly `ReentrantMutexLocker` might lull readers into believing it does safepoint checks on re-entrant "lock", which it actually does not do.
> 
> More thorough testing with different GC/JIT combinations is running now, we might find more issues there. Meanwhile, please comment on the approach.
> 
> Additional testing:
>  - [x] `grep -R "MutexLocker " src/hotspot | grep -i null`, no hits
>  - [x] `grep -R "MutexLocker " src/hotspot | grep -i ?`, no hits
>  - [x] Linux AArch64 fastdebug, `tier1 tier2 tier3` (re-run in progress)

The issue with JDK-8313081 was NOT due to null mutexes - that was the last line of defense if you like. With JDK-8313081 the real problem was that we took a mutex intended only for G1 and used it outside its intended domain without realizing it was conditionally created. Sure a null check would eventually have found it but as I said last line of defense. We screwed up and missed the problem. Having a null mutex mean "no locking" is a feature not a bug.

Case (b) should be detected by proper sub-system initialization checks post VM init when the mutexes should all be initialized.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/15043#issuecomment-1653496319
PR Comment: https://git.openjdk.org/jdk/pull/15043#issuecomment-1653498807