RFR(s) 8153711: [REDO] JDWP: Memory Leak: GlobalRefs never deleted when processing invokeMethod command

[Severin Gehwolf]

Hi,

Here is a patch which is a redo of the fix for JDK-4858370 which got
backed out due to it causing test regressions. Specifically problems
were reported for com/sun/jdi/InvokeTest.java
and com/sun/jdi/InterfaceMethodsTest.java with the fix for JDK-4858370
applied.

Those test regressions were caused because:
a.) The fix for JDK-4858370 deleted refs from the request object
    while *not* holding the invoker lock.
b.) Invokes done via invoker_doInvoke() save global references, but
    don't hold the invoker lock either.

This new fix grabs relevant locks for both cases above.

Testing done:
 - com/sun/jdi test set. No regressions + added regression test.
 - com/sun/jdi/InterfaceMethodsTest.java did not fail in 1300
   invocations. Same for com/sun/jdi/InvokeTest.java.
 - I haven't seen any crashes in new OomDebugTest.java either.

Bug: https://bugs.openjdk.java.net/browse/JDK-8153711
webrev: http://cr.openjdk.java.net/~sgehwolf/webrevs/JDK-8153711/webrev.01/

Once reviewed, I'd need somebody to sponsor this patch.

Thanks,
Severin