RFR: JDK-8277056: Combining several C2 Print* flags asserts in xmlStream::pop_tag

[Dean Long]

On Tue, 12 Apr 2022 13:15:28 GMT, Tobias Holenstein <duke at openjdk.java.net> wrote:

> Sometimes the writing to xmlStream is mixed from several threads, and therefore the `xmlStream` tag stack can end up in a bad state. When this occurs, the VM crashes in `xmlStream::pop_tag` with `assert(false) failed: bad tag in log`.
> 
> The logging between `xtty->head` and `xtty->tail` is guarded by a ttyLocker which also locks `VMThread::should_terminate()` (ttyLocker is needed here to serializes the output info about the termination of the VMThread). 
> 
> The problem is that `print_metadata` and `dump_asm` may block for safepoint which then releases the ttyLocker (`break_tty_lock_for_safepoint`). When the `print_metadata` or `dump_asm` continues after the safepoint other threads could have taken the ttyLocker and may be busy printing and interfere with the tag stack of `xmlStream`.
> 
> The solution is to call `print_metadata` and `dump_asm` first and let them write to a local stringStream. Then acquire the ttyLocker to do the printing (using the local stringStream) and use a separate ttyLocker for `VMThread::should_terminate()`
> 
> The same issue was already targeted in https://bugs.openjdk.java.net/browse/JDK-8153527 but the fix was incomplete.

If the problem is the native --> VM transition while holding the tty lock, then how about if we enter VM mode first, then grab the tty lock?

-------------

PR: https://git.openjdk.java.net/jdk/pull/8203