Netty Support / VirtualThread Selector#select Issue

[Johannes Schüth]

Hello,

I started some experiments on how to add Loom VirtualThread support to
Netty.

During my tests I noticed that Netty was no longer able to handle requests
when using 500+ virtual threads for its reactor pool.

I created a dedicated test without Netty to replicate this issue and found
that the JVM would not run / start a new virtual thread if 400+ previous
threads were calling new `Selector().select().`
I'm not yet sure why this is the case. The thread dump shows all threads as
'Running'.

--snip--
static final int THREAD_COUNT = 1500;

@Test
public void testSelectorThreads() throws InterruptedException,
TimeoutException {
    CountDownLatch latch = new CountDownLatch(THREAD_COUNT);
    AtomicInteger startCounter = new AtomicInteger();
    for (int i = 0; i < THREAD_COUNT; i++) {
        Thread.startVirtualThread(() -> {
            try {
                System.out.println("Running: " +
startCounter.incrementAndGet());
                latch.countDown();
                Selector selector = Selector.open();
                selector.select();
                Thread.sleep(10_000);
            } catch (Exception e) {
                e.printStackTrace();
            }
        });
    }

    if (!latch.await(5_000, TimeUnit.MILLISECONDS)) {
        dumpThreads();
    }
    assertEquals("Not all thread did start.", THREAD_COUNT,
startCounter.get());
}
--snap--

The full repository can be found on Github:
https://github.com/Jotschi/netty-loom-experiment

Aside of this problem my two major concerns for Netty Loom support are
Garbage/GC overhead due to newly created threadlocals for on-demand virtual
threads and JNI usage within Netty itself (e.g. EPoll and soon hopefully
io_uring).
I'm however not a Netty expert and merely experimenting out of curiosity.
As mentioned in the async-await thread the thread local problem could be
addressed by the Foreign Memory API of Project Panama.
Any other feedback would be welcome.

Thanks for the great work.

Greetings,

Johannes