RFR (S): [9+8u] Stack overflow in choose_collection_set on high region count

[Aleksey Shipilev]

Hi,

Again, running on large machine with lots of regions, you will see this:

#  SIGSEGV (0xb) at pc=0x00007f3d9ff76235, pid=26678, tid=26928
#
# JRE version: OpenJDK Runtime Environment (9.0) (fastdebug build
9-internal+0-adhoc.root.shenandoah-jdk9)
# Java VM: OpenJDK 64-Bit Server VM (fastdebug
9-internal+0-adhoc.root.shenandoah-jdk9, mixed mode, tiered, Shenandoah gc,
linux-amd64)
# Problematic frame:
# V  [libjvm.so+0x140c235]
ShenandoahHeuristics::choose_collection_set(ShenandoahCollectionSet*, int*)+0x145

This is because "candidates" array in choose_collection_set is automatically
allocated on stack, and for large number of regions, it overflows! At best, we
will crash with SEGV, but may also silently corrupt something. My fault for not
allocating it on heap from the start.

Fix:
  http://cr.openjdk.java.net/~shade/shenandoah/cset-stackoverflow/webrev.01/

Going to push to both 9 and 8u.

Testing: hotspot_gc_shenandoah (9/8u) + failing configuration on large machine


Thanks,
-Aleksey