RFR: 8273608: Deadlock when jcmd of OnError attaches to itself

[David Holmes]

On Wed, 22 Sep 2021 06:56:12 GMT, Xin Liu <xliu at openjdk.org> wrote:

>> src/hotspot/share/utilities/vmError.cpp line 1334:
>> 
>>> 1332: 
>>> 1333:     if (_thread != nullptr) {
>>> 1334:       assert(!_thread->owns_locks(), "must release all locks when leaving VM");
>> 
>> This can't be an assertion as the thread is not knowingly leaving the VM without first releasing locks. If it does hold locks then that could lead to additional problems and strange errors when running the external command. We have to decide whether it is safest/best to simply not transition to native if holding locks, or whether it is okay to proceed knowing that there are risks of secondary crashes, or hangs, if we do.
>
> In debug build, a JavaThread can't transit to Native if it owns any lock.  Even I remove the assert here, it will hit another assert later in `ThreadStateTransition::transition_from_vm`.
> 
> 
> // Checks safepoint allowed and clears unhandled oops at potential safepoints.
> void JavaThread::check_possible_safepoint() {
>   if (_no_safepoint_count > 0) {
>     print_owned_locks();
>     assert(false, "Possible safepoint reached by thread that does not allow it");
>   }
> 
> 
> I'd like to make VMErrorThreadToNativeFromVM only change state if _thread doesn't own any mutex, but 'Thread::own_lock()` is only available in debug build. 
> 
> Only one test  ’runtime/ErrorHandling/TestOnError.java‘ will call VMError::report_and_die with Threads_lock.

I did flag this problem originally. It is unfortunate that we can't tell if a thread holds any locks in a product build. Not sure how to deal with this.

-------------

PR: https://git.openjdk.java.net/jdk/pull/5590