RFR: 8273608: Deadlock when jcmd of OnError attaches to itself [v5]

[David Holmes]

On 7/10/2021 4:22 am, Xin Liu wrote:
> On Wed, 6 Oct 2021 06:30:36 GMT, Xin Liu <xliu at openjdk.org> wrote:
>> Xin Liu has updated the pull request incrementally with one additional commit since the last revision:
>>
>>    Update per reviewers' feedbacks.
>>    
>>    1. rename the test to TestOnErrorWithSelfAttachingJCmd.java.
>>    2. rename java_current_transition_into_native() and add the static qualifier.
>>    3. make unlock_locks_on_error more general. ensure formal name is consistent
>>    with argument name.
> 
> 1. `HeapDumpOnOutOfMemoryError` can't cover OOME from Java world and thread exhaustion.
> 2. I admit that unlocking mutexes is hacky.  To be fair, this problem predates this patch.   There is [Jfr::on_vm_shutdown(true)](https://github.com/openjdk/jdk/blob/master/src/hotspot/share/utilities/vmError.cpp#L1560) in VMError::report_and_die. if you use ""-XX:+FlightRecorder", "-XX:StartFlightRecording:dumponexit=true", this will tamper the coredump because it unlocks all holding mutexes.

Two wrongs don't make a right. :) While JFR may need to do some hackery 
like this for its own purposes, the more it is done the more likely you 
will run into problems because of it. I'd much prefer to see if we can 
handle this without resorting to such hacks.

Thanks,
David

> 3. I just learned from David that "you can't unlock a lock that belongs to another thread!". I need to clean up this.
> 
> I would like to provide a general solution to the temporary Stop-The-world. Not only for OnError=jcmd %, but also for JFR dumponexit. I think I can have a RAII object to do the bookkeeping of unlocked mutexes.  I will clean up everything for coredump.
> 
> -------------
> 
> PR: https://git.openjdk.java.net/jdk/pull/5590
>