Files.copy(InputStream, Path, CopyOption...) can fail with OOM if direct memory is limited

[Alan Bateman]

On 12/09/2023 18:05, github at paul.kungfoocoder.org wrote:
> Recently we came across a very surprising situation.
>
> In our code we were doing something similar to:
>
>     echo 'new java.io.ByteArrayInputStream(new
>     byte[256*1024*1024]).transferTo(new
>     java.io.FileOutputStream("/tmp/aaa"));' | jshell
>     -R-XX:MaxDirectMemorySize=25M
>
>
> Which works perfectly well. For various reasons some of the code was 
> adapted to use Paths, instead of Files, and this was changed to 
> (effectively):
>
>
>     echo 'java.nio.file.Files.copy(new
>     java.io.ByteArrayInputStream(new byte[256*1024*1024]),
>     java.nio.file.Path.of("/tmp/aaa"),
>     java.nio.file.StandardCopyOption.REPLACE_EXISTING);' | jshell
>     -R-XX:MaxDirectMemorySize=25M
>
>
> Whereupon it started to fail with a message similar to:
> |Exception java.lang.OutOfMemoryError: Cannot reserve 268435456 bytes 
> of direct buffer memory (allocated: 127, limit: 26214400)|
> Note that this isn't strictly a problem of |Files.copy|, but rather of 
> the output stream returned by|Files.newOutputStream|, as evidenced by 
> |echo 'new java.io.ByteArrayInputStream(new 
> byte[256*1024*1024]).transferTo(java.nio.file.Files.newOutputStream(java.nio.file.Path.of("/tmp/aaa")));' 
> | jshell -R-XX:MaxDirectMemorySize=25M|
>
Right, it's ChannelOutputStream.writeFully that doesn't limit the size 
of the writes to the underlying FileChannel. Improving that would reduce 
direct buffer usage for several operations (not just copy).

-Alan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/nio-dev/attachments/20230912/d1a5d3d2/attachment-0001.htm>