RFR: 8273608: Deadlock when jcmd of OnError attaches to itself

[Patricio Chilano Mateo]

On Wed, 22 Sep 2021 07:57:05 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

>> I did flag this problem originally. It is unfortunate that we can't tell if a thread holds any locks in a product build. Not sure how to deal with this.
>
>> In debug build, a JavaThread can't transit to Native if it owns any lock. Even I remove the assert here, it will hit another assert later in `ThreadStateTransition::transition_from_vm`.
>> 
>> ```
>> // Checks safepoint allowed and clears unhandled oops at potential safepoints.
>> void JavaThread::check_possible_safepoint() {
>>   if (_no_safepoint_count > 0) {
>>     print_owned_locks();
>>     assert(false, "Possible safepoint reached by thread that does not allow it");
>>   }
>> ```
>> 
>> I'd like to make VMErrorThreadToNativeFromVM only change state if _thread doesn't own any mutex, but 'Thread::own_lock()` is only available in debug build.
> 
> Which may be fine, since asserts only fire in debug builds.

We can always walk _mutex_array like we do in print_owned_locks_on_error(). Note that locks created outside mutex_init() will not be visible though. Maybe we should fix that.

-------------

PR: https://git.openjdk.java.net/jdk/pull/5590