Native zlib libraries

[Xueming Shen]

totalbytes_Read/Written are being tested though not the size > 4G. That 
test mainly tests
the > 64K entries and total size > 4G (lots of > 32-bit offsets...), but 
the total bytes read/
written are all < 4G (for each entry). An easy manual test I always do 
after run that test is to
jar the resulting > 6g file and unjar it. That will actually check the 
total bytes read/written
 > 4G case.

If you keep using the same JRE  which uses the system zlib, you might 
still be able to jar and
unjar a single > 4G file. The total bytes read/written are used as one 
of the validation in zip
format, ZipOutputStream writes these two number as size/csize fields of 
the loc and cen
table, the ZipInputStream then verifies these numbers when inflating. 
This round-trip happens
to work if the zos and zis both use the same JRE, in which both throw 
away the high bits of
the total numbers (but their low bits are still match) But a simple jar 
tvf tells you the size number
is incorrect. Also , if you jar some other files together with this > 4G 
file,  you probably will
get error when trying to extract individual entry file(s) out, as the 
position info of  those files
probably will get messed up.

Btw, the code to support > 4G total_in/out at java level is ready here, 
need some more tests
and the bugid for this one and yours.

http://cr.openjdk.java.net/~sherman/undozip/webrev/

-Sherman


On 7/30/12 3:20 AM, Andrew Hughes wrote:
> ----- Original Message -----
>> On 07/27/2012 10:37 AM, Andrew Hughes wrote:
>>> ----- Original Message -----
>>>> The zip64 support (total_in/out) part probably can be done at Java
>>>> level
>>>> (ignore
>>>> the total_in/out in z_tream_s).  Need to remove this dependency.
>>>> Will
>>>> take a look later.
>>>>
>>> Yes, it seems they still mention the size of total_in/out on the
>>> website on the zlib site, and that they shouldn't be relied on:
>>>
>>> http://www.zlib.net/zlib_faq.html#faq32
>>>
>>> "Note however that the strm.total_in and strm_total_out counters
>>> may be limited to 4 GB. These counters are provided as a
>>> convenience and are not used internally by inflate() or deflate().
>>> The application can easily set up its own counters updated after
>>> each call of inflate() or deflate() to count beyond 4 GB"
>>>
>>> "The word "may" appears several times above since there is a 4 GB
>>> limit only if the compiler's long type is 32 bits. If the
>>> compiler's long type is 64 bits, then the limit is 16 exabytes."
>>>
>>> I notice a test went in with the 64-bit support, but I assume it
>>> can't test these counters as the Deflater for a ZipStream is
>>> protected.  At least, they aren't failing on our builds with
>>> system zlib.
>> That test is not configured to be run for "auto testing", it just
>> takes
>> too long to zip/unzip
>> a 4G+ file. I use it manually to test the ZIP64 support.
>>
> Yes, sorry, I noticed this after posting.  I've since run it manually and it passes
> with a system zlib, both on the zip it creates and one I created containing just
> a single 4.2gb file (a RHEL ISO image zipped).  But I'm not sure if it's testing total bytes
> read for overflow.
>
>> I will give it a try to remove this dependency next week.
> Thanks.
>
>> It would be
>> helpful if you can
>> help "migrate" the icetea patch. How does the icetea patch work now?
>> Always use the
>> system zlib, if it presents? any configurable option to switch on and
>> off?
> It adds a switch and also support for setting the CFLAGS/LIBS:
>
>      USE_SYSTEM_ZLIB=true \
>      ZLIB_LIBS="$(pkg-config --libs zlib)" \
>      ZLIB_CFLAGS="$(pkg-config --cflags zlib)" \
>
> This is the same nomenclature we use for the other libraries too
> (lcms, jpeg, png, gif, cups, etc.)
>
> I'll try and post the patch later today.
>
>> -Sherman
>>
>>> Are you actively working on this now or shall I take a look?
>>>
>>>> -Sherman
>>>>
>>>> On 7/11/2012 12:47 AM, Alan Bateman wrote:
>>>>> On 05/07/2012 17:11, Andrew Hughes wrote:
>>>>>> ----- Original Message -----
>>>>>>> Is there a way to get the native zlib libraries to get picked
>>>>>>> up
>>>>>>> instead of the hardcoded version within the JVM?
>>>>>>>
>>>>>>> --
>>>>>>> Azeem Jiva
>>>>>>> @javawithjiva
>>>>>> We have this in IcedTea (USE_SYSTEM_ZLIB=true) and intend to get
>>>>>> it
>>>>>> upstream.
>>>>>>
>>>>>> However, I don't see how this is related to HotSpot, as the zlib
>>>>>> usage
>>>>>> is in the jdk tree.
>>>>> I think we need to (re)start the discussion on core-libs-dev with
>>>>> a
>>>>> view to eliminating the patches that the JDK has to zlib, see:
>>>>>
>>>>> http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/tip/src/share/native/java/util/zip/zlib-1.2.5/patches/ChangeLog_java
>>>>>
>>>>>
>>>>> One of these changes relates to the zip64 support and I believe
>>>>> there
>>>>> are corner cases when inflating or deflating>2GB that won't work
>>>>> if
>>>>> using the system zlib. Sherman will likely recall the details.
>>>>> Given
>>>>> that the new build already supports using the system zlib (at
>>>>> least
>>>>> on
>>>>> Linux) then it would be good to sort this out so that it just
>>>>> works.
>>>>>
>>>>> -Alan
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>