RFR: 8274687: JDWP can deadlock VM if Java thread waits in blockOnDebuggerSuspend

[Richard Reingruber]

On Mon, 4 Oct 2021 14:07:18 GMT, Richard Reingruber <rrich at openjdk.org> wrote:

> This change fixes the deadlock described in the JBS-bug by:
> 
> * Releasing `handlerLock` before waiting on `threadLock` in `blockOnDebuggerSuspend()`
> 
> * Notifying on `threadLock` after resuming all threads in `threadControl_reset()`
> 
> The PR has 3 commits:
> 
> The first commit delays the cleanup actions after leaving the loop in
> `debugLoop_run()` because of a Dispose command. The delay allows to reproduce the deadlock
> running the dispose003 test with the command
> 
> 
> make run-test TEST=test/hotspot/jtreg/vmTestbase/nsk/jdi/VirtualMachine/dispose/dispose003
> 
> 
> The second commit contains the fix described above. With it the deadlock
> cannot be reproduced anymore.
> 
> The third commit removes the diagnostic code introduced with the first commit again.
> 
> The fix passed our nightly regression testing: JCK and JTREG, also in Xcomp mode with fastdebug and release builds on all platforms.

I've implemented a test that shows the jdwp agent deadlocks also trying to resume one/all threads if a thread is waiting in `blockOnDebuggerSuspend`. I will open a new pr that will include the test and a new attempt to fix the issues.

-------------

PR: https://git.openjdk.java.net/jdk/pull/5805