<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div style="display: block;">Recently we came across a very surprising situation.</div><div style="display: block;"><br></div><div style="display: block;">In our code we were doing something similar to:</div><div style="display: block;"><br></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"><div style="display: block;">echo 'new java.io.ByteArrayInputStream(new byte[256*1024*1024]).transferTo(new java.io.FileOutputStream("/tmp/aaa"));' | jshell -R-XX:MaxDirectMemorySize=25M</div></blockquote><div style="display: block;"><br></div><div style="display: block;">Which works perfectly well. For various reasons some of the code was adapted to use Paths, instead of Files, and this was changed to (effectively):</div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"><div style="display: block;"><br></div><div style="display: block;">echo 'java.nio.file.Files.copy(new java.io.ByteArrayInputStream(new byte[256*1024*1024]), java.nio.file.Path.of("/tmp/aaa"), java.nio.file.StandardCopyOption.REPLACE_EXISTING);' | jshell -R-XX:MaxDirectMemorySize=25M</div></blockquote><div style="display: block;"><br></div><div style="display: block;">Whereupon it started to fail with a message similar to:</div><div style="display: block;"><pre class="notranslate" style="box-sizing: border-box; font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, "Liberation Mono", monospace; font-size: 11.9px; margin-top: 0px; margin-bottom: 16px; overflow-wrap: normal; padding: 16px; overflow: auto; line-height: 1.45; color: rgb(31, 35, 40); border-radius: 6px; caret-color: rgb(31, 35, 40);"><code class="notranslate" style="box-sizing: border-box; font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, "Liberation Mono", monospace; font-size: 11.9px; padding: 0px; margin: 0px; background: none; border-radius: 6px; word-break: normal; border: 0px; display: inline; overflow: visible; line-height: inherit; overflow-wrap: normal;">Exception java.lang.OutOfMemoryError: Cannot reserve 268435456 bytes of direct buffer memory (allocated: 127, limit: 26214400)</code></pre></div><div style="display: block;"><span style="caret-color: rgb(31, 35, 40); color: rgb(31, 35, 40); font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 14px; background-color: rgb(255, 255, 255);">Note that this isn't strictly a problem of </span><code class="notranslate" style="box-sizing: border-box; font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, "Liberation Mono", monospace; font-size: 11.9px; padding: 0.2em 0.4em; margin: 0px; white-space: break-spaces; border-radius: 6px; caret-color: rgb(31, 35, 40); color: rgb(31, 35, 40);">Files.copy</code><span style="caret-color: rgb(31, 35, 40); color: rgb(31, 35, 40); font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 14px; background-color: rgb(255, 255, 255);">, but rather of the output stream returned by</span><code class="notranslate" style="box-sizing: border-box; font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, "Liberation Mono", monospace; font-size: 11.9px; padding: 0.2em 0.4em; margin: 0px; white-space: break-spaces; border-radius: 6px; caret-color: rgb(31, 35, 40); color: rgb(31, 35, 40);">Files.newOutputStream</code><span style="caret-color: rgb(31, 35, 40); color: rgb(31, 35, 40); font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"; font-size: 14px; background-color: rgb(255, 255, 255);">, as evidenced by </span><code class="notranslate" style="box-sizing: border-box; font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, "Liberation Mono", monospace; font-size: 11.9px; padding: 0.2em 0.4em; margin: 0px; white-space: break-spaces; border-radius: 6px; caret-color: rgb(31, 35, 40); color: rgb(31, 35, 40);">echo 'new java.io.ByteArrayInputStream(new byte[256*1024*1024]).transferTo(java.nio.file.Files.newOutputStream(java.nio.file.Path.of("/tmp/aaa")));' | jshell -R-XX:MaxDirectMemorySize=25M</code><span style="caret-color: rgb(31, 35, 40); color: rgb(31, 35, 40); font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, "Liberation Mono", monospace; font-size: 11.9px; white-space: break-spaces;"> </span></div><div style="display: block;"><br></div><div style="display: block;">This makes it much harder to use a Path, since we cannot know whether or not it will be possible to write to the file or not.</div><div style="display: block;"><br></div><div style="display: block;"><div>This fails on all of Java 11 (11.0.18), Java 17 (17.0.8), and Java 20 (openjdk 20.0.2 2023-07-18), and has been tested on both Mac and Linux.</div><div><br></div><div><div>In this particular code path, the easiest fix is to replace <a href="https://github.com/adoptium/jdk/blob/d0be73a78038faf9509623bc4ba71eb4385cd645/src/java.base/share/classes/sun/nio/ch/IOUtil.java#L94">https://github.com/adoptium/jdk/blob/d0be73a78038faf9509623bc4ba71eb4385cd645/src/java.base/share/classes/sun/nio/ch/IOUtil.java#L94</a> with something like:</div><div><br></div><div> bb = Util.getTemporaryDirectBuffer(Math.min(rem, VM.maxDirectMemory()/2));</div><div>Since the code outside of this method already loops until the ByteBuffer is full, at least on this code path. Other code paths should also retry, since Channel.write() doesn't guarantee to write all bytes anyway. A better fix would be to look at how much memory is available, and allocate that much, or to fall back to a streaming copy, similar to what the default InputStream.transferTo already does.</div><div><br></div><div>You might wonder _why_ we limit the MaxDirectMemorySize, and we do so since we found that if we don’t it can lead to the OOMKiller killing our containers, due to memory being allocated outside of the heap. Nonetheless, whether it should be limited or not, then I still find it very surprising that I can do inputStream.transferTo(fileOutputStream) and expect that it will work, regardless of the InputStream, but I cannot do the same with inputStream.transferTo(channelOutputStream), as that might fail depending on how much data is available in the input stream, and the amount of available Direct Memory.</div></div></div><div style="display: block;"><br></div><div style="display: block;">Discussion also raised with Adoptium, where they suggested to post it here.</div><div style="display: block;"><div style="-webkit-user-select: all; -webkit-user-drag: element; display: inline-block;" class="apple-rich-link" draggable="true" role="link" data-url="https://github.com/adoptium/adoptium-support/issues/894"><a style="border-radius:10px;font-family:-apple-system, Helvetica, Arial, sans-serif;display:block;-webkit-user-select:none;width:300px;user-select:none;-webkit-user-modify:read-only;user-modify:read-only;overflow:hidden;text-decoration:none;" class="lp-rich-link" rel="nofollow" href="https://github.com/adoptium/adoptium-support/issues/894" dir="ltr" role="button" draggable="false" width="300"><table style="table-layout:fixed;border-collapse:collapse;width:300px;background-color:#E5E6E9;font-family:-apple-system, Helvetica, Arial, sans-serif;" class="lp-rich-link-emailBaseTable" cellpadding="0" cellspacing="0" border="0" width="300"><tbody><tr><td vertical-align="center"><table bgcolor="#E5E6E9" cellpadding="0" cellspacing="0" width="300" style="font-family:-apple-system, Helvetica, Arial, sans-serif;table-layout:fixed;background-color:rgba(229, 230, 233, 1);" class="lp-rich-link-captionBar"><tbody><tr><td style="padding:8px 0px 8px 0px;" class="lp-rich-link-captionBar-textStackItem"><div style="max-width:100%;margin:0px 16px 0px 16px;overflow:hidden;" class="lp-rich-link-captionBar-textStack"><div style="word-wrap:break-word;font-weight:500;font-size:12px;overflow:hidden;text-overflow:ellipsis;text-align:left;" class="lp-rich-link-captionBar-textStack-topCaption-leading"><a rel="nofollow" href="https://github.com/adoptium/adoptium-support/issues/894" style="text-decoration: none" draggable="false"><font color="#272727" style="color: rgba(0, 0, 0, 0.847059);">Files.copy(InputStream, Path, CopyOption...) can fail with OOM if direct memory is limited · Issue #894 · adoptium/adoptium-support</font></a></div><div style="word-wrap:break-word;font-weight:400;font-size:11px;overflow:hidden;text-overflow:ellipsis;text-align:left;" class="lp-rich-link-captionBar-textStack-bottomCaption-leading"><a rel="nofollow" href="https://github.com/adoptium/adoptium-support/issues/894" style="text-decoration: none" draggable="false"><font color="#808080" style="color: rgba(0, 0, 0, 0.498039);">github.com</font></a></div></div></td><td style="padding:6px 12px 6px 0px;" class="lp-rich-link-captionBar-rightIconItem" width="36"><a rel="nofollow" href="https://github.com/adoptium/adoptium-support/issues/894" draggable="false"><img style="pointer-events:none !important;display:inline-block;width:36px;height:36px;border-radius:3px;" width="36" height="36" draggable="false" class="lp-rich-link-captionBar-rightIcon" alt="apple-touch-icon-180x180.png" src="cid:439DCE1A-EE20-45FF-82A4-A0D38362CEBD"></a></td></tr></tbody></table></td></tr></tbody></table></a></div></div><div><br></div><div>For our product, I have changed Files.copy() everywhere to use InputStream.transferTo(fileOutputStream), but this doesn’t seem like it is the intended result. I look forward to your responses on how we can best deal with this within Java.</div><div><br></div><div>Cheers,</div><div>Paul</div></body></html>