RFR: 8374862: assert(false) failed: Attempting to acquire lock MDOExtraData_lock/nosafepoint-1 out of order with lock tty_lock/tty -- possible deadlock (running with -XX:+Verbose -XX:+WizardMode -XX:+PrintDeoptimizationDetails)

[David Holmes]

On Tue, 13 Jan 2026 08:42:07 GMT, Guanqiang Han <ghan at openjdk.org> wrote:

> Please review this change. Thanks!
> 
> **Description:**
> 
> When -XX:+PrintDeoptimizationDetails is enabled, vframeArray.cpp prints interpreted frames under ttyLocker. 
> https://github.com/openjdk/jdk/blob/578204f8c49f06be8b9c4855359ca61c9e107678/src/hotspot/share/runtime/vframeArray.cpp#L493-L503
> In the WizardMode && Verbose branch it calls Method::print_codes() / print_codes_on(). The bytecode printing path may acquire MDOExtraData_lock (rank nosafepoint-1). 
> https://github.com/openjdk/jdk/blob/578204f8c49f06be8b9c4855359ca61c9e107678/src/hotspot/share/interpreter/bytecodeTracer.cpp#L597-L602
> Calling it while holding tty_lock violates the global lock ranking order and triggers a lock rank inversion assertion.
> 
> **Fix:**
> 
> Generate the bytecode dump before taking tty_lock, and print it afterwards under ttyLocker to keep the output coherent while preserving correct lock acquisition order. 
> To avoid double buffering when the target stream is already a thread-local stringStream, extend BytecodeTracer::print_method_codes with a buffered flag . The new call site uses buffered=false when dumping into the temporary stringStream.
> 
> **Test:** 
> 
> GHA

I think this looks like a good solution. I flagged the compiler folk as I'm unsure about the test location - but I see it is where the only other test that uses `PrintDeoptimizationDetails` exists.

One small change requested (while you await the second review).

Thanks

src/hotspot/share/runtime/vframeArray.cpp line 494:

> 492: #ifndef PRODUCT
> 493:   if (PrintDeoptimizationDetails) {
> 494:     const bool dump_codes = WizardMode && Verbose;

Suggestion:

    const bool print_codes = WizardMode && Verbose;

src/hotspot/share/runtime/vframeArray.cpp line 497:

> 495:     ResourceMark rm(thread);
> 496:     stringStream codes_ss;
> 497:     if (dump_codes) {

Suggestion:

    if (print_codes) {

src/hotspot/share/runtime/vframeArray.cpp line 511:

> 509:     vframe* f = vframe::new_vframe(iframe(), &map, thread);
> 510:     f->print();
> 511:     if (dump_codes) {

Suggestion:

    if (print_codes) {

-------------

Changes requested by dholmes (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/29186#pullrequestreview-3658551282
PR Review Comment: https://git.openjdk.org/jdk/pull/29186#discussion_r2688674709
PR Review Comment: https://git.openjdk.org/jdk/pull/29186#discussion_r2688680012
PR Review Comment: https://git.openjdk.org/jdk/pull/29186#discussion_r2688680468