Virtual Threads Deadlock on Trival Tail-Call Recursion

[Simon König]

# Problem
Project Loom's virtual threads exhibit unexpected behavior when the 
virtual thread's stack grows. The following example shows a 
tail-recursive function `recursion` that stops the recursion at a depth 
of 1 million. Hence, it is not an infinite recursion. However, when 
submitting a virtual thread that executes this function, the program 
hangs indefinitely.

# Code
```java
package main;

import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Semaphore;

public class Main {

     private static Semaphore sem = new Semaphore(0);

     public static void recursion(int depth) {
         if(depth < 1_000_000) {
             System.out.println(depth);
             recursion(depth + 1);
             return;
         }
         System.out.println("End :)");
         sem.release();
     }

     public static ExecutorService executor = 
Executors.newVirtualThreadPerTaskExecutor();

     public static void main(String[] args) throws InterruptedException {

         executor.submit(() -> {
             recursion(0);
         });
         // wait for recursion task to end
         sem.acquire();
     }
}
```

I am running
```sh
$> java --version
openjdk 22.0.2 2024-07-16
OpenJDK Runtime Environment (build 22.0.2+9)
OpenJDK 64-Bit Server VM (build 22.0.2+9, mixed mode, sharing)
$> javac --version
javac 22.0.2
```
And
```sh
$> jstat -gc <pid>
     S0C         S1C         S0U         S1U          EC           EU 
        OC           OU          MC         MU       CCSC      CCSU 
YGC     YGCT     FGC    FGCT     CGC    CGCT       GCT
         0.0         0.0         0.0         0.0      24576.0 
4096.0     471040.0          0.0        0.0        0.0       0.0 
0.0      0     0.000     0     0.000     0     0.000     0.000
```

# Expected Behavior
The tail call should be optimized out and the program should exit 
normally. At the very least, a stack overflow exception should be 
raised, if this is the true cause of the deadlock.

# Actual Behavior
The program outputs
```
<...>
12961
12962
12963
12964
12965
```
and hangs.


-- 
Simon König

University of Stuttgart
Institute of Parallel and Distributed Systems (IPVS)
Distributed Systems Department

Universitätsstr. 38
70569 Stuttgart
Germany

e-mail: simon.koenig at ipvs.uni-stuttgart.de
-------------- next part --------------
A non-text attachment was scrubbed...
Name: simon_koenig.vcf
Type: text/vcard
Size: 122 bytes
Desc: not available
URL: <https://mail.openjdk.org/pipermail/loom-dev/attachments/20240731/42aef147/simon_koenig.vcf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5073 bytes
Desc: S/MIME Cryptographic Signature
URL: <https://mail.openjdk.org/pipermail/loom-dev/attachments/20240731/42aef147/smime.p7s>