Resource Constrained Thread Per Task Executor

[masoud parvari]

If your api takes an ExecutorService, then that is what you need to
implement and not customizing ThreadPerTaskExecutor. Just Implement
ExecutorService using a so called WrappingExecutorService which uses
delegate pattern
to offload the actual implementation to a delegate in this case
ThreadPerTaskExecutor. Only in submit methods, make sure you acquire the
semaphore before offloading to the delegate.

On Mon, May 13, 2024 at 10:19 PM Colin Redmond <Colin.Redmond at outlook.com>
wrote:

> Thanks for the feedback, and I agree completely. Spring Integration takes
> an Executor Service so I was trying to customize the
> ThreadPerTaskExecutor. Unfortunately it is closed for inheritance and uses
> several classes from jdk.internal.vm, so I was not sure how best to
> proceed. To Write a new Executor Service without a ThreadContainer, or to
> use jdk.internal.vm classes.
> ------------------------------
> *From:* Robert Engels <rengels at ix.netcom.com>
> *Sent:* May 13, 2024 12:45 PM
> *To:* Colin Redmond <colin.redmond at outlook.com>
> *Cc:* loom-dev at openjdk.org <loom-dev at openjdk.org>
> *Subject:* Re: Resource Constrained Thread Per Task Executor
>
> I think you can simply this. You don’t need to pool the virtual threads.
> Just use a semaphore to control how many are running at once. You can put
> this behind and Executor for a standardized api
>
> On May 13, 2024, at 2:10 PM, Colin Redmond <colin.redmond at outlook.com>
> wrote:
>
> 
> Hello,
>
> I have been experimenting with Virtual Threads in a Spring Integration
> project that is essentially a producer consumer project. it has a small
> number of threads that reads from an external queue (pubsub) and passes the
> received message off to an ThreadPerTaskExecutor executor service. It then
> executes several IO heavy steps.
>
> Overall Virtual threads have been amazing, before I was running with
> 500-600 platform threads. Switching to virtual threads has enabled me to
> scale 20-30% more for the same resources (CPU, Memory).
>
> However I am running into one problem. Since this is a spring integration
> project I don't have a lot of control over how the work is propagated
> between threads. The thread that reads from the external queue runs much
> faster than the internal threads processing the work. When running with
> high load the service fails OOM because it can start 100k+ virtual threads,
> and more importantly those virtual threads have 100k+ messages to be
> processed which take up a lot of memory.
>
> I resolved the issue by using a ThreadPoolExecutor a with small core pool
> but large a max pool size (1000), a virtual thread factory,  a limited size
> LinkedBlockingQueue and the RejectedExecutionHandler uses the spring
> CallerBlocksPolicy which retries to offer() the work to the executor and
> blocks. This causes back pressure and ensure that the service never pulls
> in more work than it can manage, so it no longer OOM. It has enough threads
> to saturate the CPU and by using virtual threads I save a lot of memory vs
> platform threads.
>
> It has been suggested we use a semaphore to control flow with virtual
> threads instead of a Thread Pool. The best place I can think of for
> controlling the number of worker threads is in the Executor Service itself.
> So, I am investigating creating a ResourceConstrainedThreadPerTaskExecutor
> that will request a resource before launching a new thread. This resource
> could be a semaphore or even a guava ratelimiter, but I have been facing
> some issues. Unlike the ThreadPoolExecutor, the ThreadPerTaskExecutor does
> not allow you to customize it. I can not extend it, I tried to make my own
> ExecutorService that has a ThreadPerTaskExecutor, but the best place to
> check for a resource is the private Thread start(Runnable task)  method
> that I cant wrap. Finally I was going to borrow code from the
> ThreadPerTaskExecutor but it uses several jdk.internal such as
> ThreadContainer.
>
> So finally my questions:
>
>
>    1. I am aware that virtual threads should not be pooled, but is it
>    safe to hold a virtual thread for an extended period of time in a thread
>    pool?
>    2. Is there a perfered method to customize the ThreadPerTaskExecutor?
>    3. If I write my own executor service what is the goal of
>     jdk.internal.ThreadContainer? Other than holding the threads, does it
>    integrate with the JVM on a lower level?
>    4. Can I write an executor service without a ThreadContainer? Or can I
>    use ThreadContainer, is it acceptable to use jdk.internal classes?
>
>
> If this is not possible, I can modify Springs, ExecutorChannel to control
> the flow at a level higher, but I felt that it would be useful to have a
> New Thread Per Task Executor that could be throttled or limited.  Thanks
> for any and all comments.
>
>
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/loom-dev/attachments/20240513/5bcc3999/attachment.htm>