RFR: 8260267: vmTestbase/gc/gctests/FinalizeLock/FinalizeLock.java fails with fatal error: Mark stack space exhausted

[王超]

ZGC mark stack will be exhausted when the live object graph is very large, which due to the pseudo-BFS style visiting of the object graph (ZGC gc worker will push children to other stripes which behave like BFS, so the maximum mark stack size is O(num_objects).). But other gc like g1 or Shenandoah use DFS to visiting the object graph, and the mark stack usage merely exceeds 32M, which may due to the maximum mark stack size is related to the depth of the object graph O(depth_of_object_graph).

The following is the test case used to reproduce the crash,
import java.util.HashMap;
import java.util.HashSet;

class Test2 {
  public static int NODE_COUNT = 25000000;
  public static int NODE_COUNT_UB = 1 << 25;
  public static int STRING_GEN_INDEX = 1;
  public static int WATCHER_COUNT = 100;
  public static int MAP_COUNT = 8;
  public static int BIN_COUNT = 16;
  public static Object[] watchTable = new Object[MAP_COUNT];

  public static class Watcher {
    public static long counter = 0;
    public long index;
    Watcher() {
      index = counter;
      counter++;
    }
  }

  public static int gen_name() {
    int cur = STRING_GEN_INDEX++;
    int reminder = cur % BIN_COUNT;
    cur = cur - reminder + NODE_COUNT_UB * reminder;
    return cur;
  }

  public static HashMap<Integer, HashSet<Watcher>> get_map(int index) {
    return (HashMap<Integer, HashSet<Watcher>>)watchTable[index];
  }

  public static void gen_watcher() {
    int name = gen_name();
    HashSet<Watcher> set = new HashSet<Watcher>();
    for (int i = 0; i < WATCHER_COUNT; i++) {
       set.add(new Watcher());
    }
    for (int i = 0; i < MAP_COUNT; i++) {
      get_map(i).put(name, set);
    }
  }

  public static void setup() {
    for (int i = 0; i < MAP_COUNT; i++) {
      watchTable[i] = new HashMap<Integer, HashSet<Watcher>>();
    }
    for (int i = 0; i < NODE_COUNT; i++) {
      gen_watcher();
    }
  }

  public static void main(String[] args) {
    setup();
    for (int i = 0; i < 10; i++) {
      System.gc();
    }
    System.out.println("pass");
  }
}

The command to run the testcase is,
java -XX:+UseZGC -Xmx280g -XX:ZCollectionInterval=10  -Xlog:gc*=debug:file=gc.log:time,level,tags:filesize=5g Test2

-------------

Commit messages:
 - Fix mark stack overflow

Changes: https://git.openjdk.java.net/jdk/pull/3262/files
 Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=3262&range=00
  Issue: https://bugs.openjdk.java.net/browse/JDK-8260267
  Stats: 61 lines in 6 files changed: 58 ins; 0 del; 3 mod
  Patch: https://git.openjdk.java.net/jdk/pull/3262.diff
  Fetch: git fetch https://git.openjdk.java.net/jdk pull/3262/head:pull/3262

PR: https://git.openjdk.java.net/jdk/pull/3262