Compression acceleration for Java

[Viswanathan, Sandhya]

Yes, it will be nice if the OpenJDK and Oracle builds for Linux/Solaris used the system rather than 
the bundled zlib. In which case, there will be no need for JVM command line option. Also the HW/SW 
compression accelerators can then be used easily with Java. Is it possible to get this feature in JDK9?

Thanks,
Sandhya

-----Original Message-----
From: Alan Bateman [mailto:Alan.Bateman at oracle.com] 
Sent: Saturday, March 07, 2015 1:32 AM
To: Viswanathan, Sandhya; core-libs-dev at openjdk.java.net
Subject: Re: Compression acceleration for Java

On 07/03/2015 01:19, Viswanathan, Sandhya wrote:
> Hi All,
>
> This is a request for support in JVM to use system zlib or alternative implementations of zlib.
>
> Compression is heavily used in Java based big data, genomics and middleware applications. There are many products in market today which help in compression performance either through software or hardware acceleration. Intel has faster compression libraries as part of IPP and also there are hardware compression accelerator products from Intel. Both of these products are available in the market today and support the zlib interface as an API. Support in the JVM to use system zlib on a JVM command line option would make these acceleration capabilities available to Linux java users through java.util.zip.
> The JVM could support this on an option UseSystemZlib which can be set to false by default. When the option is set to true on command line by the user, the load_zip_library() function in classLoader.cpp can load libszip.so instead of libzip.so. Also the JVM can set the java.util.zip.UseSystemZlib property accordingly.
>
> The JDK would need to include two libraries for zip in jre/lib/amd64: libzip.so and libszip.so. The libzip.so will be built as before with statically linking system zlib with it and libszip.so will be built without statically linking system zlib with it. Also the java.lang.System.initializeSystemClass will load libzip.so or libszip.so based on the java.util.zip.UseSystemZlib property.
>
> The users can then use system zlib by including the -XX:+UseSystemZlib option on the command line and compression accelerators by setting the LD_LIBRARY_PATH and LD_PRELOAD appropriately.
>
> I am very interested to know your thoughts on this feature support.
>
You may know this already but there is a configure option to use the 
system zlib (configure --with-zlib=system). I will assume that if you 
build with this option and run with LD_* set to the IPP libraries then 
it should work.

That said, I think your mail is timely as we do need to re-examine how 
zlib is used. It would be nice if the build used the system rather than 
the bundled zlib when building on Linux and Solaris (OS X already uses 
the system zlib). Ditto for Oracle builds. That would avoid needing to 
split libzip and avoid needing to introduce a runtime option to select 
between the bundled and system zlib.

Another long standing suggestion is to just move to a pure java 
implementation, something that Sherman Shen prototyped some time ago and 
mentioned in a recent thread here on memory mapping the zip central 
directory. If we were to go down that route then use of zlib couldn't be 
dropped completely, I think it would still be needed for HotSpot to 
support -Xbootclasspath/a and the JVM TI agent equivalent (with the JEP 
220 work then rt.jar/friends go away so HotSpot only rarely needs to 
call into libzip now). Moving to a pure java implementation might nobble 
your proposal of course but it might open other opportunities for the 
compiler.

-Alan