RFR: 8190753: (zipfs): Accessing a large entry (> 2^31 bytes) leads to a negative initial size for ByteArrayOutputStream [v2]

[Jaikiran Pai]

Hello Lance,

On 29/06/21 11:31 pm, Lance Andersen wrote:
>
> I ran your current test 150 times and the max runtime was 25 seconds, most cases were in the 18-20 second range on our slower test boxes.

Thank you for running those tests. Do you think those timings are good 
enough to let that test stay as a regular automated jtreg test, in 
tier1? I'm guessing this falls in tier1? I haven't yet looked in detail 
the tier definitions of the build.


> As part of looking at what happens with a file whose deflated size is > 2gb, I would add a specific test which is a manual test to validate that there is no issue when we cross the 2gb threshold.

I added a (manual) test to see what happens in this case. I have 
committed the test as part of this PR just for the sake of reference. 
The test is named LargeCompressedEntrySizeTest. The test uses ZipFS to 
create a (new) zip file and attempts to write out a zip entry whose 
deflated/compressed size is potentially greater than 2gb. When I run 
this test case, I consistenly run into the following exception:

test LargeCompressedEntrySizeTest.testLargeCompressedSizeWithZipFS(): 
failure
java.lang.OutOfMemoryError: Required array length 2147483639 + 419 is 
too large
     at 
java.base/jdk.internal.util.ArraysSupport.hugeLength(ArraysSupport.java:649)
     at 
java.base/jdk.internal.util.ArraysSupport.newLength(ArraysSupport.java:642)
     at 
java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:100)
     at 
java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:130)
     at 
java.base/java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:252)
     at 
java.base/java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:210)
     at 
jdk.zipfs/jdk.nio.zipfs.ZipFileSystem$DeflatingEntryOutputStream.write(ZipFileSystem.java:2016)
     at 
java.base/java.io.FilterOutputStream.write(FilterOutputStream.java:108)
     at 
LargeCompressedEntrySizeTest.testLargeCompressedSizeWithZipFS(LargeCompressedEntrySizeTest.java:104)

which to me is understandable. Is this what you and Alan wanted 
tested/checked? In its current form I don't see a way to write out a 
entry whose deflated size exceeds 2gb, unless the user/caller use the 
"useTempFile=true" option while creating the zip filesystem. FWIW - if I 
do set this "useTempFile=true" while creating that zip filesystem, in 
the LargeCompressedEntrySizeTest, that test passes fine and the 
underlying zip that is created shows a compressed/deflated size as follows:

unzip -lv JTwork/scratch/8190753-test-compressed-size.zip
Archive:  JTwork/scratch/8190753-test-compressed-size.zip
  Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- ----- --------  ----
2147483649  Defl:N 2148138719   0% 06-30-2021 21:39 52cab9f8 
LargeZipEntry.txt
--------          -------  ---                            -------
2147483649         2148138719   0%                            1 file


I understand that Alan's suggestion holds good and we should have some 
logic in place which switches to using a temp file once we notice that 
the sizes we are dealing with can exceed some threshold, but I guess 
that is something we need to do separately outside of this PR?

-Jaikiran