Integrated: 8375125: assert(false) failed: "Attempting to acquire lock NativeHeapTrimmer_lock/nosafepoint out of order with lock ConcurrentHashTableResize_lock/nosafepoint-2 -- possible deadlock" when using native heap trimmer

[Guanqiang Han]

On Wed, 14 Jan 2026 00:08:47 GMT, Guanqiang Han <ghan at openjdk.org> wrote:

> Please review this change. Thanks!
> 
> **Description:**
> 
> When -XX:TrimNativeHeapInterval=non-zero is enabled, running Test6892265 can trigger a VM abort due to a detected deadlock (or lock-order issue) during SymbolTable cleanup.
> SymbolTable::clean_dead_entries() calls BulkDeleteTask::prepare(), which may take ConcurrentHashTableResize_lock (nosafepoint-2). 
> https://github.com/openjdk/jdk/blob/0d19d91b44e5232dbd99d34dcdf6500f892e3048/src/hotspot/share/classfile/symbolTable.cpp#L765-L769
> https://github.com/openjdk/jdk/blob/0d19d91b44e5232dbd99d34dcdf6500f892e3048/src/hotspot/share/utilities/concurrentHashTableTasks.inline.hpp#L159-L160
> https://github.com/openjdk/jdk/blob/0d19d91b44e5232dbd99d34dcdf6500f892e3048/src/hotspot/share/utilities/concurrentHashTable.inline.hpp#L307-L310
> 
> 
> It then constructs a NativeHeapTrimmer::SuspendMark, which may take NativeHeapTrimmer_lock (nosafepoint). This can invert the lock order with the trimmer thread.
> https://github.com/openjdk/jdk/blob/0d19d91b44e5232dbd99d34dcdf6500f892e3048/src/hotspot/share/classfile/symbolTable.cpp#L773
> https://github.com/openjdk/jdk/blob/0d19d91b44e5232dbd99d34dcdf6500f892e3048/src/hotspot/share/runtime/trimNativeHeap.hpp#L56-L59
> https://github.com/openjdk/jdk/blob/0d19d91b44e5232dbd99d34dcdf6500f892e3048/src/hotspot/share/runtime/trimNativeHeap.cpp#L177-L181
> 
> **Fix:**
> 
> Move NativeHeapTrimmer::SuspendMark to the beginning of SymbolTable::clean_dead_entries() before BulkDeleteTask::prepare() to enforce a consistent lock order
> 
> **Test:**
> 
> GHA

This pull request has now been integrated.

Changeset: a67979c4
Author:    Guanqiang Han <ghan at openjdk.org>
Committer: David Holmes <dholmes at openjdk.org>
URL:       https://git.openjdk.org/jdk/commit/a67979c4e6dcea70e63cc79a105be12a9306c660
Stats:     119 lines in 3 files changed: 115 ins; 2 del; 2 mod

8375125: assert(false) failed: "Attempting to acquire lock NativeHeapTrimmer_lock/nosafepoint out of order with lock ConcurrentHashTableResize_lock/nosafepoint-2 -- possible deadlock" when using native heap trimmer

Reviewed-by: dholmes, stuefe

-------------

PR: https://git.openjdk.org/jdk/pull/29212