RFR: 8375125: assert(false) failed: "Attempting to acquire lock NativeHeapTrimmer_lock/nosafepoint out of order with lock ConcurrentHashTableResize_lock/nosafepoint-2 -- possible deadlock" when using native heap trimmer [v5]

[David Holmes]

On Fri, 16 Jan 2026 01:02:52 GMT, Guanqiang Han <ghan at openjdk.org> wrote:

>> Please review this change. Thanks!
>> 
>> **Description:**
>> 
>> When -XX:TrimNativeHeapInterval=non-zero is enabled, running Test6892265 can trigger a VM abort due to a detected deadlock (or lock-order issue) during SymbolTable cleanup.
>> SymbolTable::clean_dead_entries() calls BulkDeleteTask::prepare(), which may take ConcurrentHashTableResize_lock (nosafepoint-2). 
>> https://github.com/openjdk/jdk/blob/0d19d91b44e5232dbd99d34dcdf6500f892e3048/src/hotspot/share/classfile/symbolTable.cpp#L765-L769
>> https://github.com/openjdk/jdk/blob/0d19d91b44e5232dbd99d34dcdf6500f892e3048/src/hotspot/share/utilities/concurrentHashTableTasks.inline.hpp#L159-L160
>> https://github.com/openjdk/jdk/blob/0d19d91b44e5232dbd99d34dcdf6500f892e3048/src/hotspot/share/utilities/concurrentHashTable.inline.hpp#L307-L310
>> 
>> 
>> It then constructs a NativeHeapTrimmer::SuspendMark, which may take NativeHeapTrimmer_lock (nosafepoint). This can invert the lock order with the trimmer thread.
>> https://github.com/openjdk/jdk/blob/0d19d91b44e5232dbd99d34dcdf6500f892e3048/src/hotspot/share/classfile/symbolTable.cpp#L773
>> https://github.com/openjdk/jdk/blob/0d19d91b44e5232dbd99d34dcdf6500f892e3048/src/hotspot/share/runtime/trimNativeHeap.hpp#L56-L59
>> https://github.com/openjdk/jdk/blob/0d19d91b44e5232dbd99d34dcdf6500f892e3048/src/hotspot/share/runtime/trimNativeHeap.cpp#L177-L181
>> 
>> **Fix:**
>> 
>> Move NativeHeapTrimmer::SuspendMark to the beginning of SymbolTable::clean_dead_entries() before BulkDeleteTask::prepare() to enforce a consistent lock order
>> 
>> **Test:**
>> 
>> GHA
>
> Guanqiang Han has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Polish regression test and move it to runtime/os

Functional fix looks good. A query on the test.

Thanks

test/hotspot/jtreg/runtime/os/TestTrimNativeHeapIntervalTablesCleanup.java line 32:

> 30:  * @library /test/lib
> 31:  * @modules java.compiler
> 32:  * @run main/othervm -XX:+UseG1GC -Xms128m -Xmx128m

If the test needs G1 then it should have an appropriate `@requires` clause. No point running this test when we are testing ZGC for example.

But does it really require G1? I admit I have no idea how this test is causing the StringTable or SymbolTable cleanup to be triggered.

-------------

PR Review: https://git.openjdk.org/jdk/pull/29212#pullrequestreview-3668482304
PR Review Comment: https://git.openjdk.org/jdk/pull/29212#discussion_r2696641793