RFR: 8374862: assert(false) failed: Attempting to acquire lock MDOExtraData_lock/nosafepoint-1 out of order with lock tty_lock/tty -- possible deadlock (running with -XX:+Verbose -XX:+WizardMode -XX:+PrintDeoptimizationDetails) [v9]

[Guanqiang Han]

> Please review this change. Thanks!
> 
> **Description:**
> 
> When -XX:+PrintDeoptimizationDetails is enabled, vframeArray.cpp prints interpreted frames under ttyLocker. 
> https://github.com/openjdk/jdk/blob/578204f8c49f06be8b9c4855359ca61c9e107678/src/hotspot/share/runtime/vframeArray.cpp#L493-L503
> In the WizardMode && Verbose branch it calls Method::print_codes() / print_codes_on(). The bytecode printing path may acquire MDOExtraData_lock (rank nosafepoint-1). 
> https://github.com/openjdk/jdk/blob/578204f8c49f06be8b9c4855359ca61c9e107678/src/hotspot/share/interpreter/bytecodeTracer.cpp#L597-L602
> Calling it while holding tty_lock violates the global lock ranking order and triggers a lock rank inversion assertion.
> 
> **Fix:**
> 
> Generate the bytecode dump before taking tty_lock, and print it afterwards under ttyLocker to keep the output coherent while preserving correct lock acquisition order. 
> To avoid double buffering when the target stream is already a thread-local stringStream, extend BytecodeTracer::print_method_codes with a buffered flag . The new call site uses buffered=false when dumping into the temporary stringStream.
> 
> **Test:** 
> 
> GHA

Guanqiang Han has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains 13 additional commits since the last revision:

 - change variable name
 - Merge remote-tracking branch 'upstream/master' into 8374862
 - remove unused code
 - revert
 - Merge remote-tracking branch 'upstream/master' into 8374862
 - fix a compile error
 - remove unnecessary blank line
 - correct copyright year
 - Add outputStream::is_buffered()
 - change variable name
 - ... and 3 more: https://git.openjdk.org/jdk/compare/76d3f6a3...a63c4613

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/29186/files
  - new: https://git.openjdk.org/jdk/pull/29186/files/601ff0da..a63c4613

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=29186&range=08
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=29186&range=07-08

  Stats: 132 lines in 9 files changed: 117 ins; 2 del; 13 mod
  Patch: https://git.openjdk.org/jdk/pull/29186.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/29186/head:pull/29186

PR: https://git.openjdk.org/jdk/pull/29186