RFR: 8273608: Deadlock when jcmd of OnError attaches to itself

[Thomas Stuefe]

On Wed, 22 Sep 2021 07:43:43 GMT, David Holmes <dholmes at openjdk.org> wrote:

>> In debug build, a JavaThread can't transit to Native if it owns any lock.  Even I remove the assert here, it will hit another assert later in `ThreadStateTransition::transition_from_vm`.
>> 
>> 
>> // Checks safepoint allowed and clears unhandled oops at potential safepoints.
>> void JavaThread::check_possible_safepoint() {
>>   if (_no_safepoint_count > 0) {
>>     print_owned_locks();
>>     assert(false, "Possible safepoint reached by thread that does not allow it");
>>   }
>> 
>> 
>> I'd like to make VMErrorThreadToNativeFromVM only change state if _thread doesn't own any mutex, but 'Thread::own_lock()` is only available in debug build. 
>> 
>> Only one test  ’runtime/ErrorHandling/TestOnError.java‘ will call VMError::report_and_die with Threads_lock.
>
> I did flag this problem originally. It is unfortunate that we can't tell if a thread holds any locks in a product build. Not sure how to deal with this.

> In debug build, a JavaThread can't transit to Native if it owns any lock. Even I remove the assert here, it will hit another assert later in `ThreadStateTransition::transition_from_vm`.
> 
> ```
> // Checks safepoint allowed and clears unhandled oops at potential safepoints.
> void JavaThread::check_possible_safepoint() {
>   if (_no_safepoint_count > 0) {
>     print_owned_locks();
>     assert(false, "Possible safepoint reached by thread that does not allow it");
>   }
> ```
> 
> I'd like to make VMErrorThreadToNativeFromVM only change state if _thread doesn't own any mutex, but 'Thread::own_lock()` is only available in debug build.

Which may be fine, since asserts only fire in debug builds.

-------------

PR: https://git.openjdk.java.net/jdk/pull/5590