Possible bug parking virtual threads with large stacks

[Mark Lippincott]

Hello all,

A few of us at PayPal have been experimenting with the Loom binaries released over the past few months and while testing limitations, we ran into what we believe may be a bug. It seems that with many virtual threads with a large enough stack and light workload, a StackOverflowError occurs when parking a virtual thread.

Using the code below, on a GCP instance with 4 cores and 16GB ram, a StackOverflowError is thrown with a stack depth of roughly 2000 and 20 virtual threads with Thread.startVirtualThread(). Under 20 virtual threads does not produce this error. Adding some workload between each stack depth function call causes the error to disappear at 20 threads.

Using the ExecutorService from Executors.newVirtualThreadPerTaskExecutor() seems to increase these limitations.

public class TestStackDepth {
    static final int sleepDuration = 3600000;
    static int numThreads;
    static int stackDepth;
    static int workload;

    public static void run(int threadId, int stack, boolean logData) {
        BigInteger a = BigInteger.ZERO;
        BigInteger b = BigInteger.ONE;
        for (int i = 0; i < workload; i++) {
            b = a.add(b);
            a = b.subtract(a);
        }
        String s = b.toString();

        if (stack > 0) {
            if (logData && stack % 100 == 0) {
                System.out.printf("Thread %d: %d\n", threadId, stack);
            }
            run(threadId, stack - 1, logData);
        } else {
            if (logData) {
                System.out.printf("Thread %d in final depth %s\n", threadId, s.substring(0, Math.min(4, s.length())));
            }

            try {
                Thread.sleep(sleepDuration);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
        }
    }

    public static void main(String[] argv) {
        numThreads = Integer.parseInt(argv[0]);
        stackDepth = Integer.parseInt(argv[1]);
        int execType = Integer.parseInt(argv[2]);
        workload = Integer.parseInt(argv[3]);

        ExecutorService executor = Executors.newVirtualThreadPerTaskExecutor();

        for (int i = 0; i < numThreads; i++) {
            int threadId = i + 1;
            boolean logData = threadId % (Math.max(1, numThreads / 10)) == 0;
            switch (execType) {
                case 1 -> Thread.startVirtualThread(() -> run(threadId, stackDepth, logData));
                case 2 -> run(threadId, stackDepth, logData);
                case 3 -> Thread.ofPlatform().start(() -> run(threadId, stackDepth, logData));
                case 4 -> Thread.ofVirtual().start(() -> run(threadId, stackDepth, logData));
                case 5 -> executor.submit(() -> run(threadId, stackDepth, logData));
            }
        }

        try {
            Thread.sleep(sleepDuration * 2);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }
}

Running above code with arguments "20 2000 1 0" results in StackOverflowError.
Increasing workload to 1000 removes error: "20 2000 1 1000".
Using executor, stack depth can be increased to 3000: "20 3000 5 0".

Altering the code even slightly seems to change the depth at which the error is thrown, but it seems to be consistent when run on the same machine.

Thanks,
Mark