Fast graceful shutdown of ThreadPerTaskExecutor (when expected WAITING Threads)

[Ron Pressler]

I am not familiar with Helidon’s internals or with the details of the particular issue, but you can separate the “idle” and “busy” threads by spawning them in different ExecutorServices. For example, threads that wait an accept new requests are spawned in one ES and they, in turn, spawn a “request processing” thread in another. To shut down the server you will then forcefully shut down the first ES, stopping new requests from being accepted, and gracefully shut down the second, giving the requests some time to finish.

— Ron

On 21 Dec 2022, at 22:22, Rob Bygrave <robin.bygrave at gmail.com<mailto:robin.bygrave at gmail.com>> wrote:

A "solution / fix" to this issue that modifies the internals of java.util.concurrent.ThreadPerTaskExector is to:


1) Add a new interface InterruptableTask

public interface InterruptableTask {

    /**
     * Return true if this is a task that is deemed to be in an idle state that can be interrupted.
     */
    boolean canInterrupt();
}


2) Modify java.util.concurrent.ThreadPerTaskExector by:

2i) Changing the threads field from storing just Thread to additionally storing the associated task (Runnable or Callable)

// instead of ...
private final Set<Thread> threads = ConcurrentHashMap.newKeySet();

// use a Map with values being the tasks (Runnable or Callable)
private final Map<Thread, Object> threadTasks = new ConcurrentHashMap<>();



2ii) Add a new internal method tryStopInterruptableTask():

private void tryStopInterruptableTask() {
  threadTasks.entrySet().stream()
       .filter(entry -> entry.getKey().isAlive()) // Thread isAlive
       .filter(entry -> entry.getKey().getState() == Thread.State.WAITING) // Thread WAITING state
       .forEach(entry -> {
         if (entry.getValue() instanceof InterruptableTask task) {
           if (task.canInterrupt()) { // application deems this task to NOT be in-flight (e.g. not IO waiting on database response)
             entry.getKey().interrupt();
           }
         }
       });
}

2 iii) Call this method as part of shutdown() (or perhaps the internal tryShutdownAndTerminate())



With these changes, then the Nima specific ConnectionHandler just needs to implement InterruptableTask and then the usual
shutdown of the ExecutorService operates nicely.

executor.shutdown();
if (!executor.awaitTermination(gracefulShutdownMillis, TimeUnit.MILLISECONDS)) {
    List<Runnable> running = executor.shutdownNow();
    if (!running.isEmpty()) {
        LOGGER.log(INFO, running.size() + " channel tasks did not terminate gracefully");
    }
}



Cheers, Rob.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/loom-dev/attachments/20221222/97a43413/attachment-0001.htm>