state of openjdk-zlib on s390x

[eddy]

Hello,

# TLDR

zlib (and zlib-ng) on s390x use hardware compression. This
hardware compression implementation has currently some problems when
used by openjdk.



# current situation

- openjdk uses zlib(or zlib-ng) for most(all?) its data compression.
   This includes also jar file creation.
- s390x has deflate implemented on hardware level(called dfltcc). This
   implementation is up to 10x faster(empirical value) compared to the
   standard software algorithm.
- The dfltcc implementation has some drawbacks vs the software
   implementation:
	- dfltcc will ALWAYS write to the output buffer, indifferent
of 	  the flushing parameter passed to deflate()
	- dfltcc does not generate reproducible output. This means
	  that it will always generate valid deflate data streams
	  that uncompress(inflate) to the same input. But the
compressed data may look different between two calls with
	  the same input data, because the hardware compressor depends
	  also on hardware variables that not visible to the external
	  api user.




# the problem

due to the differences of the hardware implementation, some tests in
the openjdk JREG tests fail(tracked here[1][2])

- java/util/zip/CloseInflaterDeflaterTest.java
     fails due to dfltcc's flushing behaviour. The test should check if
     the openjdk wrapper around the jni and native library will
     successfully detect writes to closed output streams. This was
     created because there where bugs with and race conditions in the
     write/close/flush.

- tools/jlink/JLinkReproducibleTest.java:
- tools/jar/modularJar/JarToolModuleDescriptorReproducibilityTest.java:
     This tests fail due to two calls to dfltcc will not generate the
     same compressed data for the same input data. I want to
     emphasize that RFC 1950 and 1951 do not guarantee the same output
     for two deflate/zlib data streams if feed the same input data.


# proposed solutions
## flushing problem

IMHO the CloseInflaterDeflaterTest.java test relays on zlib behavior
where zlib explicitly has the freedom to change its behavior[3][1].
As a quick and dirty solution i would propose to backport the
disablement[4] of this test to openjdk-17 21 and 24.

I read and traced through the test, and for me it looks like the
behavior of the openjdk and jni wrappers will be the same regardless of
the underlying zlib. Therefor it seems ok to assume that: "If its okay
on x86 it will be ok on s390x"

However I think that this test might need a redesign. It relies on
flushing behavior of zlib the the library explicitly stated can change.



## reproducible compression

I don't know how much openjdk relies on reproducible data compression.
I assume it is preferable if its possible to create the same .jar twice
and get the same binary data. The zlib dfltcc implementation could be
controlled via an env variable to disable the hardware compression[5].
Also usually dftcc will only be used for certain compression levels and
this can also be changed via env variables[6].

Unfortunately this env variables are evaluated at library load and can
not be adjusted during runtime.

Moreover ubuntu setzt the default LEVEL_MASK to 0x7e[7]. This means
that the compression level is also not a reliabl method in order to
force zlib to use software instead of hardware compression.

Also zlib-ng lacks this env variables. Here the use of dfltcc cannot be
influenced at all.

This all leads me to the conclusion that there has to be a decision if
and how openjdk on s390x will be able to generate reproducible jars.
For the time being I suppose to also disable this tests and backport
the disablements down till openjdk-17.



# Finally

I hope i did not overwhelm you all with this email. As I don't have a
bugs.openjdk.org account, I thought the mailing list is the best place
to discuss this mater.

cheers Eddy



[1] https://bugs.launchpad.net/ubuntu/+source/openjdk-21/+bug/2109016
[2] https://bugs.openjdk.org/browse/JDK-8339216
[3] https://github.com/madler/zlib/blob/develop/zlib.h#L286-L288
[4]
https://github.com/openjdk/jdk/commit/537447f8816129dad9a1edd21bd30f3edf69ea60
[5]
https://github.com/iii-i/zlib/blob/dfltcc/contrib/s390/dfltcc.c#L705-L706
[6]
https://github.com/iii-i/zlib/blob/dfltcc/contrib/s390/dfltcc.c#L714-L715
[7] https://git.launchpad.net/ubuntu/+source/zlib/tree/debian/rules#n53