RFR: 8273608: Deadlock when jcmd of OnError attaches to itself

[Patricio Chilano Mateo]

On Thu, 23 Sep 2021 05:27:58 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

>> src/hotspot/share/utilities/vmError.cpp line 1341:
>> 
>>> 1339:   ~VMErrorThreadToNativeFromVM() {
>>> 1340:     if (_thread != nullptr) {
>>> 1341:       ThreadStateTransition::transition_from_native(_thread, _thread_in_vm);
>> 
>> Just some thought on this part. Ideally we should avoid calling process_if_requested_with_exit_check() since attempting to process handshakes/stackwatermarks at this point might lead to all sorts of other issues. An alternative could be to just set the original state back and continue. But maybe we don't care about this because we are almost done with the error reporting and the OnError commands were already executed. That last part would argue to move the wrapper before the while loop.
>
> I agree with @pchilano (if I understand him correctly) in that I think an RAII object here is not even needed. There is no need to re-instate the prior thread state.

As long as we don't call process_if_requested_with_exit_check() I think either way should be fine, i.e. manually restoring the state or just continue, since after this the only thing left is to call os::die/os::abort. If we reuse this logic with jfrEmergencyDump::on_vm_shutdown() then we would have to restore it, since I see print_bug_submit_message() which happens after that checks the state.

-------------

PR: https://git.openjdk.java.net/jdk/pull/5590