RFR: JDK-8031767 Support system or alternative implementations of zlib

[Viswanathan, Sandhya]

I would like to reiterate the need for system zlib support by the JVM instead of bundled zlib from the performance point of view.

Compression is one of the hotspots in Big Data, Genomics and Middleware applications.   
There are many solutions in market today which help improve compression performance either through software or hardware acceleration.
Intel has faster compression libraries as part of IPP as Sherman mentioned. Also there are hardware compression accelerators from Intel like QuickAssist.
Both of these provide the acceleration through the zlib interface. 
Support for system zlib by the JVM would make these acceleration capabilities available to Java applications almost seamlessly through java.util.zip.

Best Regards,
Sandhya

-----Original Message-----
From: core-libs-dev [mailto:core-libs-dev-bounces at openjdk.java.net] On Behalf Of Martin Buchholz
Sent: Thursday, February 11, 2016 10:49 AM
To: Seán Coffey
Cc: build-dev; core-libs-dev
Subject: Re: RFR: JDK-8031767 Support system or alternative implementations of zlib

Currently the pendulum is swinging away from multiple applications
sharing common libraries towards every application being
self-contained, perhaps because disk space is dirt cheap and because
of the rise of "containers".  It may be that much of the packaging of
jdks will be picked up by third parties who package up the JDK and any
of its dependencies - e.g. I already see
https://hub.docker.com/_/java/

On Thu, Feb 11, 2016 at 2:34 AM, Seán Coffey <sean.coffey at oracle.com> wrote:
>
> On 11/02/2016 00:25, Xueming Shen wrote:
>>
>>
>> One of the benefits of moving to the system libz is actually for
>> better/easy
>> maintenance. Just replacing the offending version of libz with an
>> earlier/later
>> version that works, instead of waiting for a customized jdk/jre image with
>> a
>> working/bundled libz, or the next update release. Especially given the
>> fact
>> that we have decided not to touch the libz at source level. Having
>> dependency
>> on the system libz is really not that bad. The experience suggests those
>> binaries are quite stable. And it should always be easier to replace the
>> system libz than a java runtime, in case of problem.
>
>
> I think people overestimate people's ability to 'just replace' a zlib
> library. Adding a new system property is a struggle for some enterprises.
> We'll enjoy supporting a many to one matrix of zlib:JDK scenarios now rather
> than old style 1:1. It's great to have system library support - I'm just
> highlighting a possible risk. A fall back option solves that but there's no
> appetite for such a solution. Let's see how it goes.
>
> regards,
> Sean.
>
>>
>> Sherman
>>
>> On 02/10/2016 06:41 AM, Seán Coffey wrote:
>>>
>>>
>>> On 10/02/16 14:29, Alan Bateman wrote:
>>>>
>>>>
>>>> On 10/02/2016 13:57, Seán Coffey wrote:
>>>>>
>>>>> I'm all for allowing one to introduce a new version of zlib to their
>>>>> JDK at runtime. I just don't think it's in the interests of enterprises and
>>>>> stability to introduce a dependency to the JDK on the underlying OS zlib
>>>>> libraries. Could we at least consider a runtime property to allow linking to
>>>>> the (currently bundled) zlib v.1.2.8 port in case issues arise ?
>>>>
>>>> Don't the LD_* environment variables serve this need already? Once the
>>>> JDK is using the system zlib then this is the simplest way to get it to use
>>>> a different libz library at runtime.
>>>
>>> No - I don't see that as a solution. You've still made the default JDK
>>> config become dependent on OS environment for all libzip operations. I don't
>>> think we even capture the zlib version that the JDK would be operating with
>>> in any diagnostics. An application wanting a tried/tested and stable libzip
>>> version has extra work to do now. Letting the default be system dependent
>>> has just increased risk for QA teams also. A system property just makes this
>>> all go away. In fact - I would say that for JDK9, the default should be the
>>> JDK bundled libzip library. For those looking for libzip experimenting and
>>> performance benefits, they could take the system property approach.
>>>
>>> regards,
>>> Sean.
>>>>
>>>>
>>>> -Alan
>>>
>>>
>>>
>>> On 08/02/16 09:55, Alan Bateman wrote:
>>>>
>>>>
>>>> On 08/02/2016 10:42, Seán Coffey wrote:
>>>>>
>>>>> Is there an option to fall back to the older v.1.2.8 implementation if
>>>>> necessary ? It would certainly alleviate issues for any applications that
>>>>> might run into issues with the latest and greatest zlib libraries.
>>>>> JDK-8133206 would be one such example.
>>>>
>>>> Just at build time
>>>
>>> so - we introduce a runtime dependency on the underlying zlib libraries
>>> on the OS by default. I would be very concerned with this approach. We've
>>> run JDK 6 for 10+ years with zlib v1.1.3. It was consistent and reliable for
>>> the most part. When we moved JDK7/8 to zlib v1.2.5, we encountered an
>>> inflation issue[1]. When JDK was upgraded to zlib v1.2.8, we received
>>> reports of performance/OOM issues [2].
>>>>
>>>> but if the zlib on the platform is broken then it impacts tools and
>>>> probably lots of things. I would assume the OS would patch such issues
>>>> quickly. In the case of JDK-8133206 then was the issue addressed in the
>>>> libzip wrapper or in the zlib code? I thought it was the former.
>>>
>>> The code change is proposed in the libzip wrapper but the issue was
>>> triggered by the zlib library update.
>>>>
>>>>
>>>> On a fallback or some way to configure at launch time then Sandhya
>>>> Viswanathan (Intel) has a proposal here last year. I think we mostly agreed
>>>> on that thread that switching the build to use the system zlib by default
>>>> would be better.
>>>
>>> I'm all for allowing one to introduce a new version of zlib to their JDK
>>> at runtime. I just don't think it's in the interests of enterprises and
>>> stability to introduce a dependency to the JDK on the underlying OS zlib
>>> libraries. Could we at least consider a runtime property to allow linking to
>>> the (currently bundled) zlib v.1.2.8 port in case issues arise ?
>>>
>>> regards,
>>> Sean.
>>>
>>> [1] https://bugs.openjdk.java.net/browse/JDK-8044725
>>> [2] https://bugs.openjdk.java.net/browse/JDK-8133206
>>>
>>
>