RFR: Out-of-cycle Degenerated GC should process references and unload classes

[Aleksey Shipilev]

Hopefully the comment describes the rationale.

diff -r 90a268e66f81 src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp
--- a/src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp        Tue Aug 28 07:58:08 2018 -0400
+++ b/src/hotspot/share/gc/shenandoah/shenandoahHeap.cpp        Wed Aug 29 10:22:59 2018 +0200
@@ -1800,10 +1800,21 @@
       op_final_traversal();
       op_cleanup_traversal();
       return;

     case _degenerated_outside_cycle:
+      // We have degenerated from outside the cycle, which means something is bad with
+      // the heap, most probably heavy humongous fragmentation, or we are very low on free
+      // space. It makes little sense to wait for Full GC to reclaim as much as it can, when
+      // we can do the most aggressive degen cycle, which includes processing references and
+      // class unloading, unless those features are explicitly disabled.
+      //
+      // Note that we can only do this for "outside-cycle" degens, otherwise we would risk
+      // changing the cycle parameters mid-cycle during concurrent -> degenerated handover.
+      set_process_references(ShenandoahRefProcFrequency != 0);
+      set_unload_classes(ClassUnloading);
+
       if (heuristics()->can_do_traversal_gc()) {
         // Not possible to degenerate from here, upgrade to Full GC right away.
         cancel_gc(GCCause::_shenandoah_upgrade_to_full_gc);
         op_degenerated_fail();
         return;

Testing: tier3_gc_shenandoah (running)

Thanks,
-Aleksey