state of openjdks zlib on s390x

[Eduard Stefes]

Hello,

Disclaimer: This is email could cause cause discussions and work.

# TLDR

zlib (and zlib-ng) on s390x use hardware compression. This
hardware compression implementation has currently some problems when
used by openjdk.



# current situation

- openjdk uses zlib(or zlib-ng) for most(all?) its data compression. 
  This includes also jar file creation. 
- s390x has deflate implemented on hardware level(called dfltcc). This 
  implementation is up to 10x faster(empirical value) compared to the
  standard software algorithm.
- The dfltcc implementation has some drawbacks vs the software 
  implementation:
	- dfltcc will ALWAYS write to the output buffer, indifferent
of 	  the flushing parameter passed to deflate()
	- dfltcc does not generate reproducible output. This means
	  that it will always generate valid deflate data streams
	  that uncompress(inflate) to the same input. But the 	  	 
compressed data may look different between two calls with 
	  the same input data, because the hardware compressor depends
	  also on hardware variables that not visible to the external 
	  api user.




# the problem
 
due to the differences of the hardware implementation, some tests in
the openjdk JREG tests fail(tracked here[1][2])

- java/util/zip/CloseInflaterDeflaterTest.java
    fails due to dfltcc's flushing behaviour. The test should check if 
    the openjdk wrapper around the jni and native library will 
    successfully detect writes to closed output streams. This was 
    created because there where bugs with and race conditions in the 
    write/close/flush.

- tools/jlink/JLinkReproducibleTest.java:
- tools/jar/modularJar/JarToolModuleDescriptorReproducibilityTest.java:
    This tests fail due to two calls to dfltcc will not generate the 
    same compressed data for the same input data. I want to 
    emphasize that RFC 1950 and 1951 do not guarantee the same output 
    for two deflate/zlib data streams if feed the same input data. 


# proposed solutions
## flushing problem

IMHO the CloseInflaterDeflaterTest.java test relays on zlib behavior
where zlib explicitly has the freedom to change its behavior[3][1].
As a quick and dirty solution i would propose to backport the
disablement[4] of this test to openjdk-17 21 and 24.

I read and traced through the test, and for me it looks like the
behavior of the openjdk and jni wrappers will be the same regardless of
the underlying zlib. Therefor it seems ok to assume that: "If its okay
on x86 it will be ok on s390x"

However I think that this test might need a redesign. It relies on
flushing behavior of zlib the the library explicitly stated can change.



## reproducible compression

I don't know how much openjdk relies on reproducible data compression.
I assume it is preferable if its possible to create the same .jar twice
and get the same binary data. The zlib dfltcc implementation could be
controlled via an env variable to disable the hardware compression[5].
Also usually dftcc will only be used for certain compression levels and
this can also be changed via env variables[6]. 

Unfortunately this env variables are evaluated at library load and can
not be adjusted during runtime.

Moreover ubuntu setzt the default LEVEL_MASK to 0x7e[7]. This means
that the compression level is also not a reliabl method in order to
force zlib to use software instead of hardware compression.

Also zlib-ng lacks this env variables. Here the use of dfltcc cannot be
influenced at all. 

This all leads me to the conclusion that there has to be a decision if
and how openjdk on s390x will be able to generate reproducible jars.
For the time being I suppose to also disable this tests and backport
the disablements down till openjdk-17. 


# Finally

I hope i did not overwhelm you all with this email. As I don't have a
bugs.openjdk.org account, I thought the mailing list is the best place
to discuss this mater. If all are okay with disabling the tests I can
create the PR in gh. 

cheers Eddy



[1] https://bugs.launchpad.net/ubuntu/+source/openjdk-21/+bug/2109016
[2] https://bugs.openjdk.org/browse/JDK-8339216
[3] https://github.com/madler/zlib/blob/develop/zlib.h#L286-L288
[4]
https://github.com/openjdk/jdk/commit/537447f8816129dad9a1edd21bd30f3edf69ea60
[5]
https://github.com/iii-i/zlib/blob/dfltcc/contrib/s390/dfltcc.c#L705-L706
[6]
https://github.com/iii-i/zlib/blob/dfltcc/contrib/s390/dfltcc.c#L714-L715
[7] https://git.launchpad.net/ubuntu/+source/zlib/tree/debian/rules#n53