RFR: 8276400: openjdk image Jars, Zips and Jmods built from the same source are not reproducible

[Magnus Ihse Bursie]

On Thu, 4 Nov 2021 20:56:45 GMT, Andrew Leonard <aleonard at openjdk.org> wrote:

> This PR enables reproducible Jars, Jmods and openjdk image zip files (eg.src.zip).
> It provides support for SOURCE_DATE_EPOCH for Jar, Jmod and underlying ZipOutputStream's.
> It fixes the following keys issues relating to reproducibility:
> - Jar and ZipOutputStream are not SOURCE_DATE_EPOCH aware
>   - Jar and ZipOutputStream now detect SOURCE_DATE_EPOCH environment setting
> - Jar and Jmod file content ordering was non-determinsitic
>   - Fixes to Jar and Jmod main's to ensure sorted classes content ordering
> - openjdk image zip file generation used "zip" which is non-determinsitic
>   - New openjdk build tool "GenerateZip" which produces the final determinsitic zip files as part of the build and also detects SOURCE_DATE_EPOCH
> 
> Signed-off-by: Andrew Leonard <anleonar at redhat.com>

I agree with Alan's comment above. First of all, to be absolutely clear, I think this is a very worthy goal, and much overdue. I'll do my best to help get this implemented. 

But.

This PR conflates two different issues. One of them is "changing the zip implementation in the JDK". This will affect all users of the JDK, and as Alan says, this will require a thorough discussion to get it right, including a proper CSR. Otoh, the benefit of doing this will affect *all* Java projects out there in the wild wanting to do reproducible builds, so this has actually a much higher leverage for the reproducible build community as a whole. 

And, also, it is a prerequisite for issue number two, which is "use the new zip implementation in the build system". Once the former is in place, this is more or less a no-brainer. Most will come automatically, but there are some things to fix. But these require just the approval of developers in the build team; no long discussions or CSRs. (Also, when we get to this point, I believe we should get rid of the "zip" utility and write our own in Java, rather than unzipping and then re-zipping. But we can take that discussion when we get there.)

In a way, Alan is right when saying a PR is premature for issue #1. OTOH, my personal opinion is that "working code" is a good start for any discussion, as long as one is prepared to throw it all out and start over, or rewrite it over and over again, like @jaikiran did in the Property storage PR. So my suggestion is that your break out these changes to the zip and jar implementation, open a new PR, and start a discussion around them.

-------------

PR: https://git.openjdk.java.net/jdk/pull/6268