<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;
panose-1:2 11 0 4 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:10.0pt;
font-family:"Aptos",sans-serif;}
h3
{mso-style-priority:9;
mso-style-link:"Heading 3 Char";
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:13.5pt;
font-family:"Aptos",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#467886;
text-decoration:underline;}
span.Heading3Char
{mso-style-name:"Heading 3 Char";
mso-style-priority:9;
mso-style-link:"Heading 3";
font-family:"Aptos",sans-serif;
color:#0F4761;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Aptos",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#467886" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Hi Alan, thanks for response<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">> </span><span style="font-size:12.0pt">Would it be possible to do some digging to say for sure if this happens after StackOverflowError (SOE)?</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We saw a thread throwing an SOE while holding a DB connection. Later, there was another thread getting stuck while trying to cleanup a connection. Unfortunately, we haven’t collected the thread/memory dumps
and can’t state it with 100% confidence. But everything we see in logs and the code is pointing to the scenario described below.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">> </span><span style="font-size:12.0pt">Can you confirm you created this to demonstrate your point and isn't representative of what the code is doing?</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The code is demonstrating the point, but it comes from real code. It starts with our DB Connection Pool checking the connection has no stale data in its socket buffer before handing it out. If it detects stale
data (which was the case) it realizes the connection can’t be trusted and kills it by calling the most brutal API method available: abort(). In MariaDB JDBC driver, abort() is calling its internal method closeSocket():<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><a href="https://github.com/mariadb-corporation/mariadb-connector-j/blob/2.3.0/src/main/java/org/mariadb/jdbc/internal/protocol/AbstractConnectProtocol.java#L275">https://github.com/mariadb-corporation/mariadb-connector-j/blob/2.3.0/src/main/java/org/mariadb/jdbc/internal/protocol/AbstractConnectProtocol.java#L275</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Now, for some reason the closeSocket() is trying to ensure the socket buffer is empty by attempting to read a byte from it. To protect from hangs, the driver is setting a 3-msec timeout on the socket:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><a href="https://github.com/mariadb-corporation/mariadb-connector-j/blob/2.3.0/src/main/java/org/mariadb/jdbc/internal/protocol/AbstractConnectProtocol.java#L203">https://github.com/mariadb-corporation/mariadb-connector-j/blob/2.3.0/src/main/java/org/mariadb/jdbc/internal/protocol/AbstractConnectProtocol.java#L203</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The NIO’s read() first tries to protect the operation by obtaining the lock which makes total sense. But the gap is that it tries to acquire the lock w/o any timeout. Since the lock is already held by the
compromised thread, our thread also hangs.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">As you’ve pointed out, this isn’t a normal or intended scenario. Moreover, NIO has every right and obligation to protect multi-threaded access to the socket.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The only ask is to respect the timeout while doing that. Feels like this can be easily achieved by using tryLock() similarly to accept().<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Calibri",sans-serif">-- <o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Calibri",sans-serif">Regards, Alexei (team: Data Scale/Dev-Persistence, dept: Platform Engineering, Loc: San Diego, Mgr: Venkata Koya, Teams/MM: alexei.olkhovskii)</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div id="mail-editor-reference-message-container">
<div>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Alan Bateman <alan.bateman@oracle.com><br>
<b>Date: </b>Tuesday, May 27, 2025 at 22:59<br>
<b>To: </b>Alexei Olkhovskii <alexei.olkhovskii@servicenow.com>, nio-dev@openjdk.org <nio-dev@openjdk.org><br>
<b>Subject: </b>Re: NIO Socket.read() may not respect the socket timeout<o:p></o:p></span></p>
</div>
<div>
<h3><strong><span style="font-family:Helvetica;color:black;background:yellow">[External Email]</span></strong><span style="font-family:Helvetica"><o:p></o:p></span></h3>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:Helvetica"><o:p> </o:p></span></p>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-size:12.0pt;font-family:Helvetica">
<hr size="0" width="100%" align="center">
</span></div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:12.0pt"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">On 27/05/2025 23:50, Alexei Olkhovskii wrote:<o:p></o:p></span></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">Hello,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">We’ve ran into a corner-case issue with socket read(). In short, it doesn’t honor socket timeout when trying to acquire the read lock:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""><a href="https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/sun/nio/ch/NioSocketImpl.java#L336">https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/sun/nio/ch/NioSocketImpl.java#L336</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> readLock.lock();</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">In our case one thread obtained a database connection and got stuck in read() (we think because of a StackOverflow).</span><o:p></o:p></p>
</div>
</blockquote>
<p class="MsoNormal"><span style="font-size:12.0pt"><br>
Would it be possible to do some digging to say for sure if this happens after StackOverflowError (SOE)? There have been a few reports of issues that appear to be lock "corruption" when continuing after SOE and it may be that the reserved stack area for critical
sections needs to be re-visited.<br>
<br>
As regards the test. Can you confirm you created this to demonstrate your point and isn't representative of what the code is doing? For a TCP/stream connection then having several threads reading from the same connection would require coordination at a high
level. If it does arise that one thread is blocked reading and another thread attempts to read from the same connection then it is blocked until the first thread finishes reading. It would be a strange scenario to arise and different to the accept case where
it's okay to have concurrently threads attempt to accept connections, they don't interfere.
<br>
<br>
-Alan<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</body>
</html>