[15] RFR (XS) 8247560: Shenandoah: heap iteration holds root locks all the time

[Aleksey Shipilev]

Bug:
  https://bugs.openjdk.java.net/browse/JDK-8247560

Newly added compressed hprof test exposes a trouble with Shenandoah heap iteration: "Attempting to
wait on monitor HProf Compression Backend/11 while holding lock CodeCache_lock/6 -- possible deadlock".

ShenandoahHeapIterationRootScanner holds the CodeCache_lock for code roots iteration, and it lingers
for the entirety of heap iteration. The fix is to scope it properly:

diff -r a39eb5a4f1c1 src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp
--- a/src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp        Thu Jun 11 18:16:32 2020 +0200
+++ b/src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp        Mon Jun 15 09:19:21 2020 +0200
@@ -1298,9 +1298,14 @@
   Stack<oop,mtGC> oop_stack;

-  // First, we process GC roots according to current GC cycle. This populates the work stack with
initial objects.
-  ShenandoahHeapIterationRootScanner rp;
   ObjectIterateScanRootClosure oops(&_aux_bit_map, &oop_stack);

-  rp.roots_do(&oops);
+  {
+    // First, we process GC roots according to current GC cycle.
+    // This populates the work stack with initial objects.
+    // It is important to relinquish the associated locks before diving
+    // into heap dumper.
+    ShenandoahHeapIterationRootScanner rp;
+    rp.roots_do(&oops);
+  }

   // Work through the oop stack to traverse heap.

Testing: hotspot_gc_shenandoah, affected tests (many times)

-- 
Thanks,
-Aleksey