RFR: 8276743: Make openjdk build Zip Archive generation "reproducible"

[Erik Joelsson]

On Tue, 9 Nov 2021 12:59:17 GMT, Andrew Leonard <aleonard at openjdk.org> wrote:

> This PR adds a new openjdk build tool GenerateZip, which generates a final "zip" file from an input folder, and creates it in a deterministic way, ensuring ordering and timestamps are set as specified.
> 
> Using this tool in ZipArchive.gmk will ensure src.zip is then created deterministically.
> 
> Signed-off-by: Andrew Leonard <anleonar at redhat.com>

I think it would it make sense add a configure parameter to to enable/disable this new functionality. I think developers would want to opt out of extra unnecessary build steps when building and testing locally. Not sure what the default should be. For other reproducible measures, we require active enabling today. Maybe default on for release builds and off for debug builds.

make/common/ZipArchive.gmk line 29:

> 27: _ZIP_ARCHIVE_GMK := 1
> 28: 
> 29: include ../ToolsJdk.gmk

Should probably add a comment about inclusion of this file now requires an explicit dependency on build-tools in Main.gmk.

make/common/ZipArchive.gmk line 178:

> 176: 	    (cd $$(SUPPORT_OUTPUTDIR)/ziptmp/$1/files && \
> 177: 	     $(RM) $$@ && \
> 178: 	     $(UNZIP) -q $$(SUPPORT_OUTPUTDIR)/ziptmp/$1/tmp.zip && \

Having to explode the zip here is unfortunate. This means we are creating an almost full copy of the whole src tree in the build directory, something I tried to avoid by leveraging the include/exclude functionality of zip, instead of generating make rules for copying the files I wanted into a source tree to run zip on. This may be a small overhead on Linux, but I'm pretty sure it will be very noticeable on Windows.

When reading about your tool at first, I assumed it would read the intermediate zip file directly when rebuilding the zip. I don't think modifying it to do that would be too complicated, basically read and processing ZipEntrys instead of walking the file system.

make/jdk/src/classes/build/tools/generatezip/GenerateZip.java line 161:

> 159:         boolean pathIsDir = fpath.isDirectory();
> 160: 
> 161:         // Keep a sorted Set of files to be processed, so that the Jmod is reproducible

Aren't we generating zip files rather than jmod files here?

-------------

PR: https://git.openjdk.java.net/jdk/pull/6311