<div dir="ltr"><div>Hi Jaikiran,</div><div><br></div><div>Thanks a lot for replying.<br><br>Our application is a client that communicates to the server for<br>request/response. The client creates a secure (TLS) connection to the server,<br>that is, on top of the SocketChannel, we implement a Wrapper class called<br>SSLDataChannel for reading and writing. The SSLDataChannel uses the<br>javax.net.ssl.SSLEngine. Before any read and write can happen, we need to do<br>SSL handshakes by calling methods in SSLEngine. One of the methods is<br>SSLEngine#getDelegatedTask(). The returned task needs to be executed before the<br>handshake can proceed. After the task is done, we need to continue processing<br>read and write events on the connection. The connection read and write events<br>are all handled by a class called NioEndpointHandler. One requirement for our<br>client is that it supports an asynchronous API and therefore the whole stack<br>must all implement non-blocking methods. The tasks from the SSLEngine could<br>take a long time and we do not want them to block our other connection events,<br>and this is when the ForkJoinPool is used. We run the SSL tasks in the<br>ForkJoinPool and after the task is done we arrange to run the<br>NioEndpointHandler callbacks to proceed with the read and write events. The<br>much simplified code looks somewhat like the following.<br><br>```<br>class NioEndpointHandler {<br><br> /** The ssl channel */<br> private final SSLDataChannel sslDataChannel;<br> /** The runnable to execute to handle read after ssl tasks is done. */<br> private final Runnable handleReadAfterSSLTask = () -> onRead();<br> /** The handler state. */<br> State state;<br><br> /** Executes the SSL tasks until no task to run, then run the callback. */<br> private void executeSSLTask(ExecutorService executor, Runnable callback) {<br> executor.submit(() -> {<br> Runnable task;<br> while ((task = sslDataChannel.getSSLEngine().getDelegatedTask()) != null) {<br> task.run();<br> }<br> try {<br> callback.run();<br> } catch (Throwable t) {<br></div><div> /* logging the exception. */<br></div><div> }<br> });<br> }<br><br> /** Handle a read event. */<br> private void onRead() {<br> if (sslDataChannel.needsHandshake()) {<br> /* do handshake */<br><br> /* One of the handshake step is to check if there is any SSL task to run. */<br> if (sslDataChannel.needExecuteTask()) {<br> executeSSLTask(ForkJoinPool.commonPool(), handleReadAfterSSLTask);<br> }<br> }<br> }<br><br> private void terminate() {<br> state = TERMINATED;<br><br> /* Other clean up tasks, however, tasks submitted to the ForkJoinPool are not cancelled. */<br> }<br>}<br>```<br><br>> What are these handlers? Are they classes which implement Runnable or<br>> are they something else? What does termination of handler mean in this<br>> context? Do you use any java.util.concurrent.* APIs to "cancel" such<br>> terminated handlers?<br><br>The much simplified handler code please see above.<br><br>The tasks submitted to the ForkJoinPool queue are Runnables that are fields to<br>the NioEndpointHandler. What we have observed is that there are a lot of tasks<br>in the fork join pool that have a reference to the lambda inside<br>NioEndpointHandler#executeSSLTask which eventually have a reference to the<br>NioEndpointHandler. Those NioEndpointHandler are in the TERMINATED state. The<br>only reference to those NioEndpointHandler are these tasks or otherwise they<br>can be garbage collected after the termination cleans up all the other<br>references.</div><div><br></div><div>Termination of the handler means those connections are at the end of their life<br>cycle. We clean up things such as signal end of life cycle for all the<br>associated request/response pairs and closing the SSLDataChannel, etc.<br><br>No, we have not use the cancel method to cancel the submitted tasks. I agree<br>that this is an oversight and it would be cleaner to cancel them. However, my<br>current theory is that this is not the root cause. From my understanding of the<br>code, the cancel method only changes the state of the task. It does not remove<br>the task from the queue of the ForkJoinPool. Therefore, those tasks, even if<br>got cancelled, would still stay in the queue preventing the terminated<br>NioEndpointHandler from being garbage collected. Currently, I am strongly<br>biased to my own theory that somehow there is no ForkJoinPool thread that<br>polling tasks out of the queue and I am trying to use the ctl field in the<br>ForkJoinPool as the evidence to backup my theory. I am wondering if I am making<br>some mistake with my theory.<br><br>> Finally, what does the OutOfMemoryError exception stacktrace look like<br>> and what is the JVM parameters (heap size for example) used to launch<br>> this application?<br><br>Our clients creates about 155 threads and quite a lot of them have OOME on<br>their stack. I am not quite sure how to reply to this question. Going through<br>the stack traces, I do not find anything very suspicious. They are just<br>exercising their most frequent code path: some I/O threads waiting for I/O<br>events and some execution threads waiting for more work to do, etc.<br><br>It is worth mentioning that there is no ForkJoinPoolWorkerThread stacks in the<br>thread dump from the heap dump. From my understanding, the only time when there<br>is no such thread is when there is no tasks to run. But there are quite a lot<br>of tasks in the queue.<br><br>Here are our JVM arguments:<br><br>```<br>-Xms1G<br>-Xmx1G<br>-Djava.util.logging.config.file=/var/lib/andc/config/params/sender.logging.properties<br>-Djavax.net.ssl.trustStore=/var/lib/andc/wallet/client.trust<br>-Doracle.kv.security=/var/lib/andc/config/security/login.properties<br>-Doci.javasdk.extra.stream.logs.enabled=false<br>-XX:G1HeapRegionSize=32m<br>-XX:+DisableExplicitGC<br>-Xlog:all=warning,gc*=info,safepoint=info:file=/var/lib/andc/log/sender/sender.gc:utctime:filecount=10,filesize=10000000<br>-XX:+HeapDumpOnOutOfMemoryError<br>-XX:HeapDumpPath=/var/lib/andc/log/sender/<br>```<br><br>We have creation and termination timestamps in the NioEndpointHandler object.<br>From what I can see in the heap dump, the SSL tasks in the ForkJoinPool are<br>associated with NioEndpointHandler that are created at an interval on the<br>magnitude of seconds (retry attempt with second-magnitude backoff). Each<br>NioEndpointHandler are terminated after a fixed 5-second timeout due to unable<br>to connect. The time span for those NioEndpointHandler is about 2 hours. This<br>creates<br>```<br>2 hours * 3600 seconds / hour * 1 NioEndpointHandler / second * 1 SSLDataChannel / NioEndpointHandler * 65K bytes / SSLDataChannel ~= 468M bytes.<br>```<br>With 1G heap size, this eventually caused OOME.<br><br>We are adding fixes so that the SSL tasks would not preventing the<br>NioEndpointHandler from being garbage collected. However, the root cause is<br>still a mystery and I am wondering if I am on the right tracker to figure it<br>out.<br><br>Thanks a lot for your time and patience.<br><br><br></div><div><br></div><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature">Xiao Yu</div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Feb 2, 2024 at 5:35 AM Jaikiran Pai <<a href="mailto:jai.forums2013@gmail.com">jai.forums2013@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello Xiao,<br>
<br>
I don't have enough knowledge of this area to provide any insight into <br>
the issue. However, just to try and get the discussion started, do you <br>
have any sample code of your application which shows how the application <br>
uses the ForkJoinPool? More specifically what APIs do you use in the <br>
application?<br>
<br>
Few other questions inline below.<br>
<br>
On 12/01/24 11:30 am, Xiao Yu wrote:<br>
> ....<br>
> Here is the full background. One of our process experienced an OOME <br>
> and a heap<br>
> dump was obtained. We know there was a concurrent issue of our system <br>
> happening<br>
> on some other machines such that network failure and retries occurred <br>
> in this<br>
> process at the same time. Upon analyzing the heap dump, we observed a <br>
> lot of<br>
> our network connection handlers being frequently created and <br>
> terminated which<br>
> is expected due to the network failure and retry attempts mentioned above.<br>
> However, those terminated handlers are not being GC'ed because of <br>
> there were<br>
> references to tasks submitted to the ForkJoinPool during the connection<br>
> attempts. The tasks stayed in the queue until OOME happened as there is no<br>
> threads to execute them.<br>
<br>
What are these handlers? Are they classes which implement Runnable or <br>
are they something else? What does termination of handler mean in this <br>
context? Do you use any java.util.concurrent.* APIs to "cancel" such <br>
terminated handlers?<br>
<br>
Finally, what does the OutOfMemoryError exception stacktrace look like <br>
and what is the JVM parameters (heap size for example) used to launch <br>
this application?<br>
<br>
-Jaikiran<br>
</blockquote></div>