Resource Constrained Thread Per Task Executor

[Robert Engels]

I think you can simply this. You don’t need to pool the virtual threads. Just use a semaphore to control how many are running at once. You can put this behind and Executor for a standardized api 

> On May 13, 2024, at 2:10 PM, Colin Redmond <colin.redmond at outlook.com> wrote:
> 
> 
> Hello,
> 
> I have been experimenting with Virtual Threads in a Spring Integration project that is essentially a producer consumer project. it has a small number of threads that reads from an external queue (pubsub) and passes the received message off to an ThreadPerTaskExecutor executor service. It then executes several IO heavy steps.
> 
> Overall Virtual threads have been amazing, before I was running with 500-600 platform threads. Switching to virtual threads has enabled me to scale 20-30% more for the same resources (CPU, Memory).
> 
> However I am running into one problem. Since this is a spring integration project I don't have a lot of control over how the work is propagated between threads. The thread that reads from the external queue runs much faster than the internal threads processing the work. When running with high load the service fails OOM because it can start 100k+ virtual threads, and more importantly those virtual threads have 100k+ messages to be processed which take up a lot of memory.
> 
> I resolved the issue by using a ThreadPoolExecutor a with small core pool but large a max pool size (1000), a virtual thread factory,  a limited size LinkedBlockingQueue and the RejectedExecutionHandler uses the spring CallerBlocksPolicy which retries to offer() the work to the executor and blocks. This causes back pressure and ensure that the service never pulls in more work than it can manage, so it no longer OOM. It has enough threads to saturate the CPU and by using virtual threads I save a lot of memory vs platform threads.
> 
> It has been suggested we use a semaphore to control flow with virtual threads instead of a Thread Pool. The best place I can think of for controlling the number of worker threads is in the Executor Service itself. So, I am investigating creating a ResourceConstrainedThreadPerTaskExecutor that will request a resource before launching a new thread. This resource could be a semaphore or even a guava ratelimiter, but I have been facing some issues. Unlike the ThreadPoolExecutor, the ThreadPerTaskExecutor does not allow you to customize it. I can not extend it, I tried to make my own ExecutorService that has a ThreadPerTaskExecutor, but the best place to check for a resource is the private Thread start(Runnable task)  method that I cant wrap. Finally I was going to borrow code from the ThreadPerTaskExecutor but it uses several jdk.internal such as ThreadContainer.
> 
> So finally my questions:
> 
> I am aware that virtual threads should not be pooled, but is it safe to hold a virtual thread for an extended period of time in a thread pool?
> Is there a perfered method to customize the ThreadPerTaskExecutor?
> If I write my own executor service what is the goal of  jdk.internal.ThreadContainer? Other than holding the threads, does it integrate with the JVM on a lower level?
> Can I write an executor service without a ThreadContainer? Or can I use ThreadContainer, is it acceptable to use jdk.internal classes?
> 
> If this is not possible, I can modify Springs, ExecutorChannel to control the flow at a level higher, but I felt that it would be useful to have a New Thread Per Task Executor that could be throttled or limited.  Thanks for any and all comments.
> 
> 
> 
> 
> 
> 
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/loom-dev/attachments/20240513/d614aafe/attachment-0001.htm>