RFR: 8241390: 'Deadlock' with VM_RedefineClasses::lock_classes() [v2]

[Serguei Spitsyn]

On Thu, 17 Sep 2020 20:57:24 GMT, Daniil Titov <dtitov at openjdk.org> wrote:

>> The change fixes a 'deadlock' situation in VM_RedefineClasses::lock_classes() when the current thread is in the middle
>> of redefining the same class. The change is based on the fix [1] suggested in the Jira issue [2] comment.
>> [1] http://cr.openjdk.java.net/~jiangli/8241390/webrev.00/
>> [2] https://bugs.openjdk.java.net/browse/JDK-8241390
>> 
>> Testing: :jdk_instrument, tier1-tier3, and tier5 tests pass.
>
> Daniil Titov has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Changed test to not use shell scripts
>   
>   Additional changes:
>    - Replaced 'cls' identifier with 'redef_classes'
>    - Added a comment that the same class can be pushed to the list multiple times.
>   
>   Testing: tier1-tier3 tests pass.

The following test @summary needs some tweaks:
 * @summary Test recursively redefines the same class. The test hangs if
 * a deadlock happens.
Please, replace: redefines => retransforms.
Also, it'd be nice to align "a deadlock happens" with the "Test recursively ...".

-------------

Changes requested by sspitsyn (Reviewer).

PR: https://git.openjdk.java.net/jdk/pull/190