Resource Constrained Thread Per Task Executor

Mon May 13 21:05:56 UTC 2024

I may be overthink this. I was hoping for an easy way to extend the ThreadPerTaskExecutor like the ThreadPoolExecutior is extensible. But I Agree in this case that it should wrap another executor as getting threading correct is hard! For the Executor Service, Execute and Submit would he easy to manage, as it is a single task so easy to get a acquire a single resource from a semaphore. But I will need to be more careful around invokeAll and invokeAny, as I may need to acquire more permits than are available or one invoke all may block another invoke from executing in parallel. So maybe I can submit them one at time, or just go ahead and get the number of permits at once.

It is good to know that i dont need to use any jdk.internal classes. I saw both ThreadPerTaskExecutor  and ThreadPoolExecutor using ThreadContainer and i wasn't sure if it was some special class to manage threads. I can implement something without this.

________________________________
From: masoud parvari <masoud.parvari at gmail.com>
Sent: May 13, 2024 1:45 PM
To: Colin Redmond <Colin.Redmond at outlook.com>
Cc: Robert Engels <rengels at ix.netcom.com>; loom-dev at openjdk.org <loom-dev at openjdk.org>
Subject: Re: Resource Constrained Thread Per Task Executor

If your api takes an ExecutorService, then that is what you need to implement and not customizing ThreadPerTaskExecutor. Just Implement ExecutorService using a so called WrappingExecutorService which uses delegate pattern
to offload the actual implementation to a delegate in this case ThreadPerTaskExecutor. Only in submit methods, make sure you acquire the semaphore before offloading to the delegate.

On Mon, May 13, 2024 at 10:19 PM Colin Redmond <Colin.Redmond at outlook.com<mailto:Colin.Redmond at outlook.com>> wrote:
Thanks for the feedback, and I agree completely. Spring Integration takes an Executor Service so I was trying to customize the  ThreadPerTaskExecutor. Unfortunately it is closed for inheritance and uses several classes from jdk.internal.vm, so I was not sure how best to proceed. To Write a new Executor Service without a ThreadContainer, or to use jdk.internal.vm classes.
________________________________
From: Robert Engels <rengels at ix.netcom.com<mailto:rengels at ix.netcom.com>>
Sent: May 13, 2024 12:45 PM
To: Colin Redmond <colin.redmond at outlook.com<mailto:colin.redmond at outlook.com>>
Cc: loom-dev at openjdk.org<mailto:loom-dev at openjdk.org> <loom-dev at openjdk.org<mailto:loom-dev at openjdk.org>>
Subject: Re: Resource Constrained Thread Per Task Executor

I think you can simply this. You don’t need to pool the virtual threads. Just use a semaphore to control how many are running at once. You can put this behind and Executor for a standardized api

On May 13, 2024, at 2:10 PM, Colin Redmond <colin.redmond at outlook.com<mailto:colin.redmond at outlook.com>> wrote:

Hello,

I have been experimenting with Virtual Threads in a Spring Integration project that is essentially a producer consumer project. it has a small number of threads that reads from an external queue (pubsub) and passes the received message off to an ThreadPerTaskExecutor executor service. It then executes several IO heavy steps.

Overall Virtual threads have been amazing, before I was running with 500-600 platform threads. Switching to virtual threads has enabled me to scale 20-30% more for the same resources (CPU, Memory).

However I am running into one problem. Since this is a spring integration project I don't have a lot of control over how the work is propagated between threads. The thread that reads from the external queue runs much faster than the internal threads processing the work. When running with high load the service fails OOM because it can start 100k+ virtual threads, and more importantly those virtual threads have 100k+ messages to be processed which take up a lot of memory.

I resolved the issue by using a ThreadPoolExecutor a with small core pool but large a max pool size (1000), a virtual thread factory,  a limited size LinkedBlockingQueue and the RejectedExecutionHandler uses the spring CallerBlocksPolicy which retries to offer() the work to the executor and blocks. This causes back pressure and ensure that the service never pulls in more work than it can manage, so it no longer OOM. It has enough threads to saturate the CPU and by using virtual threads I save a lot of memory vs platform threads.

It has been suggested we use a semaphore to control flow with virtual threads instead of a Thread Pool. The best place I can think of for controlling the number of worker threads is in the Executor Service itself. So, I am investigating creating a ResourceConstrainedThreadPerTaskExecutor that will request a resource before launching a new thread. This resource could be a semaphore or even a guava ratelimiter, but I have been facing some issues. Unlike the ThreadPoolExecutor, the ThreadPerTaskExecutor does not allow you to customize it. I can not extend it, I tried to make my own ExecutorService that has a ThreadPerTaskExecutor, but the best place to check for a resource is the private Thread start(Runnable task)  method that I cant wrap. Finally I was going to borrow code from the ThreadPerTaskExecutor but it uses several jdk.internal such as ThreadContainer.

So finally my questions:

  1.
I am aware that virtual threads should not be pooled, but is it safe to hold a virtual thread for an extended period of time in a thread pool?
  2.
Is there a perfered method to customize the ThreadPerTaskExecutor?
  3.
If I write my own executor service what is the goal of  jdk.internal.ThreadContainer? Other than holding the threads, does it integrate with the JVM on a lower level?
  4.
Can I write an executor service without a ThreadContainer? Or can I use ThreadContainer, is it acceptable to use jdk.internal classes?

If this is not possible, I can modify Springs, ExecutorChannel to control the flow at a level higher, but I felt that it would be useful to have a New Thread Per Task Executor that could be throttled or limited.  Thanks for any and all comments.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/loom-dev/attachments/20240513/56b790d0/attachment-0001.htm>