RFR: 8375125: assert(false) failed: "Attempting to acquire lock NativeHeapTrimmer_lock/nosafepoint out of order with lock ConcurrentHashTableResize_lock/nosafepoint-2 -- possible deadlock" when using native heap trimmer [v4]

[Thomas Stuefe]

On Thu, 15 Jan 2026 15:13:46 GMT, Guanqiang Han <ghan at openjdk.org> wrote:

>> Please review this change. Thanks!
>> 
>> **Description:**
>> 
>> When -XX:TrimNativeHeapInterval=non-zero is enabled, running Test6892265 can trigger a VM abort due to a detected deadlock (or lock-order issue) during SymbolTable cleanup.
>> SymbolTable::clean_dead_entries() calls BulkDeleteTask::prepare(), which may take ConcurrentHashTableResize_lock (nosafepoint-2). 
>> https://github.com/openjdk/jdk/blob/0d19d91b44e5232dbd99d34dcdf6500f892e3048/src/hotspot/share/classfile/symbolTable.cpp#L765-L769
>> https://github.com/openjdk/jdk/blob/0d19d91b44e5232dbd99d34dcdf6500f892e3048/src/hotspot/share/utilities/concurrentHashTableTasks.inline.hpp#L159-L160
>> https://github.com/openjdk/jdk/blob/0d19d91b44e5232dbd99d34dcdf6500f892e3048/src/hotspot/share/utilities/concurrentHashTable.inline.hpp#L307-L310
>> 
>> 
>> It then constructs a NativeHeapTrimmer::SuspendMark, which may take NativeHeapTrimmer_lock (nosafepoint). This can invert the lock order with the trimmer thread.
>> https://github.com/openjdk/jdk/blob/0d19d91b44e5232dbd99d34dcdf6500f892e3048/src/hotspot/share/classfile/symbolTable.cpp#L773
>> https://github.com/openjdk/jdk/blob/0d19d91b44e5232dbd99d34dcdf6500f892e3048/src/hotspot/share/runtime/trimNativeHeap.hpp#L56-L59
>> https://github.com/openjdk/jdk/blob/0d19d91b44e5232dbd99d34dcdf6500f892e3048/src/hotspot/share/runtime/trimNativeHeap.cpp#L177-L181
>> 
>> **Fix:**
>> 
>> Move NativeHeapTrimmer::SuspendMark to the beginning of SymbolTable::clean_dead_entries() before BulkDeleteTask::prepare() to enforce a consistent lock order
>> 
>> **Test:**
>> 
>> GHA
>
> Guanqiang Han has updated the pull request incrementally with one additional commit since the last revision:
> 
>   remove Whitespace

Thank you for fixing this. Small remarks, otherwise this looks okay. Thanks for the repro case. Does it fire reliably with the bug preset, or is it intermittent?

test/hotspot/jtreg/runtime/nativeheaptrimmer/TestTrimNativeHeapIntervalTablesCleanup.java line 40:

> 38:  */
> 39: 
> 40: package runtime.nativeheaptrimmer;

This looks odd. Other tests don't use packages.

Also, the trimnative tests lives in runtime/os, maybe this should too.

-------------

Marked as reviewed by stuefe (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/29212#pullrequestreview-3666360370
PR Review Comment: https://git.openjdk.org/jdk/pull/29212#discussion_r2694980674