RFR: 8274687: JDWP deadlocks if some Java thread reaches wait in blockOnDebuggerSuspend [v4]

[Chris Plummer]

On Tue, 12 Oct 2021 08:03:22 GMT, Richard Reingruber <rrich at openjdk.org> wrote:

>> This change fixes deadlocks described in the JBS-bug by:
>> 
>> * Releasing `handlerLock` before waiting on `threadLock` in `blockOnDebuggerSuspend()`
>> 
>> * Notifying on `threadLock` in `threadControl_reset()`
>> 
>> Also the actions in handleAppResumeBreakpoint() are moved/deferred until
>> doPendingTasks() runs. This is necessary because an event handler must not
>> release handlerLock first and foremost because handlers are called while
>> iterating the handler chain for an event type which is protected by handlerLock
>> (see https://github.com/openjdk/jdk/pull/5805)
>> 
>> The first commit delays the cleanup actions after leaving the loop in
>> `debugLoop_run()`. It allows to reproduce the deadlock running the dispose003
>> test with the command
>> 
>> 
>> make run-test TEST=test/hotspot/jtreg/vmTestbase/nsk/jdi/VirtualMachine/dispose/dispose003
>> 
>> 
>> The second commit adds a new test that reproduces the deadlock when calling
>> threadControl_resumeThread() while a thread is waiting in
>> blockOnDebuggerSuspend().
>> 
>> The third commit contains the fix described above. With it the deadlocks
>> cannot be reproduced anymore.
>> 
>> The forth commit removes the diagnostic code introduced with the first commit again.
>> 
>> The fix passed
>> 
>> test/hotspot/jtreg/serviceability/jdwp
>> test/jdk/com/sun/jdi
>> test/hotspot/jtreg/vmTestbase/nsk/jdwp
>> test/hotspot/jtreg/vmTestbase/nsk/jdi
>
> Richard Reingruber has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Improve @summary section of test.

src/jdk.jdwp.agent/share/native/libjdwp/threadControl.c line 750:

> 748:  * handlerLock and threadLock are owned when returning and the suspendCount of
> 749:  * the given thread is 0.
> 750:  */

How about:

/*
 * The caller must own handlerLock and threadLock.
 * If the suspendCount of the given thread is greater than 0, then the
 * current thread will release the handlerLock and wait on the threadLock. It
 * must release the handlerLock first, because threadControl_resumeThread()
 * and threadControl_resumeAll() need it, and calling them is how suspendCount
 * will eventually be decremented to 0.
 * handlerLock and threadLock are owned when returning and the suspendCount of
 * the given thread is 0.
 */

-------------

PR: https://git.openjdk.java.net/jdk/pull/5849