Compression acceleration for Java

[Xueming Shen]

We can/should/will do it.

-Sherman

On 03/10/2015 11:39 AM, Viswanathan, Sandhya wrote:
> Yes, it will be nice if the OpenJDK and Oracle builds for Linux/Solaris used the system rather than
> the bundled zlib. In which case, there will be no need for JVM command line option. Also the HW/SW
> compression accelerators can then be used easily with Java. Is it possible to get this feature in JDK9?
>
> Thanks,
> Sandhya
>
> -----Original Message-----
> From: Alan Bateman [mailto:Alan.Bateman at oracle.com]
> Sent: Saturday, March 07, 2015 1:32 AM
> To: Viswanathan, Sandhya; core-libs-dev at openjdk.java.net
> Subject: Re: Compression acceleration for Java
>
> On 07/03/2015 01:19, Viswanathan, Sandhya wrote:
>> Hi All,
>>
>> This is a request for support in JVM to use system zlib or alternative implementations of zlib.
>>
>> Compression is heavily used in Java based big data, genomics and middleware applications. There are many products in market today which help in compression performance either through software or hardware acceleration. Intel has faster compression libraries as part of IPP and also there are hardware compression accelerator products from Intel. Both of these products are available in the market today and support the zlib interface as an API. Support in the JVM to use system zlib on a JVM command line option would make these acceleration capabilities available to Linux java users through java.util.zip.
>> The JVM could support this on an option UseSystemZlib which can be set to false by default. When the option is set to true on command line by the user, the load_zip_library() function in classLoader.cpp can load libszip.so instead of libzip.so. Also the JVM can set the java.util.zip.UseSystemZlib property accordingly.
>>
>> The JDK would need to include two libraries for zip in jre/lib/amd64: libzip.so and libszip.so. The libzip.so will be built as before with statically linking system zlib with it and libszip.so will be built without statically linking system zlib with it. Also the java.lang.System.initializeSystemClass will load libzip.so or libszip.so based on the java.util.zip.UseSystemZlib property.
>>
>> The users can then use system zlib by including the -XX:+UseSystemZlib option on the command line and compression accelerators by setting the LD_LIBRARY_PATH and LD_PRELOAD appropriately.
>>
>> I am very interested to know your thoughts on this feature support.
>>
> You may know this already but there is a configure option to use the
> system zlib (configure --with-zlib=system). I will assume that if you
> build with this option and run with LD_* set to the IPP libraries then
> it should work.
>
> That said, I think your mail is timely as we do need to re-examine how
> zlib is used. It would be nice if the build used the system rather than
> the bundled zlib when building on Linux and Solaris (OS X already uses
> the system zlib). Ditto for Oracle builds. That would avoid needing to
> split libzip and avoid needing to introduce a runtime option to select
> between the bundled and system zlib.
>
> Another long standing suggestion is to just move to a pure java
> implementation, something that Sherman Shen prototyped some time ago and
> mentioned in a recent thread here on memory mapping the zip central
> directory. If we were to go down that route then use of zlib couldn't be
> dropped completely, I think it would still be needed for HotSpot to
> support -Xbootclasspath/a and the JVM TI agent equivalent (with the JEP
> 220 work then rt.jar/friends go away so HotSpot only rarely needs to
> call into libzip now). Moving to a pure java implementation might nobble
> your proposal of course but it might open other opportunities for the
> compiler.
>
> -Alan