RFR: 8264449: Enable reproducible builds with SOURCE_DATE_EPOCH [v2]

[John Neffenger]

On Sat, 3 Apr 2021 01:49:47 GMT, John Neffenger <jgneff at openjdk.org> wrote:

>>> Silly question, is the difference with Windows just the nature of the native support on each platform (Unix based vs Windows) and libraries used as part of that?
>> 
>> That's a good question. I'm hoping the answer is no. So far, the difference with Windows is the linker. The other systems use the GNU linker where it's enough to sort the input object files. The Microsoft linker, though, creates reproducible builds with the flag `/experimental:deterministic` (or `-experimental:deterministic`). So on Windows, we just need to figure out where to put the flag. I'm finding that to be a challenge for the WebKit library.
>
>> /contributor add ...
> 
> @bmwiedemann I added you as co-author of pull request #422, which is my continuation of your pull request #99 from last year. This pull request #446 temporarily includes our co-authored commit, but those changes will disappear when #422 is integrated and I merge the master branch into this one. Sorry for the confusion.

> It would be better to add the flag in `libxml/CMakeLists.txt` or in `build.gradle` via `cmakeArgs`.

Thank you, Arun, for that helpful suggestion. I updated this pull request for the WebKit library on Windows and ran my tests again, as described below.

I saved the output of the first build in the `build1.log` file and the `build1` directory:

$ export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)
$ bash gradlew -PCONF=Release -PPROMOTED_BUILD_NUMBER=5 \
  -PHUDSON_BUILD_NUMBER=101 -PHUDSON_JOB_NAME=jfx -PCOMPILE_WEBKIT=true \
  -PCOMPILE_MEDIA=true -PBUILD_LIBAV_STUBS=true sdk jmods javadoc \
  | tee build1.log
$ mv build build1

Then I ran another clean build, this time saving the output in the `build2.log` file and the `build2` directory:

$ bash gradlew cleanAll
$ bash gradlew -PCONF=Release -PPROMOTED_BUILD_NUMBER=5 \
  -PHUDSON_BUILD_NUMBER=101 -PHUDSON_JOB_NAME=jfx -PCOMPILE_WEBKIT=true \
  -PCOMPILE_MEDIA=true -PBUILD_LIBAV_STUBS=true sdk jmods javadoc \
  | tee build2.log
$ mv build build2

I copied the builds to my Linux workstation and ran the following command on the JMOD files for each pair of builds:

$ strip-nondeterminism -v -T $SOURCE_DATE_EPOCH build?/jmods/*.jmod
Normalizing build1/jmods/javafx.base.jmod
Normalizing build1/jmods/javafx.controls.jmod
Normalizing build1/jmods/javafx.fxml.jmod
Normalizing build1/jmods/javafx.graphics.jmod
Normalizing build1/jmods/javafx.media.jmod
Normalizing build1/jmods/javafx.swing.jmod
Normalizing build1/jmods/javafx.web.jmod
Normalizing build2/jmods/javafx.base.jmod
Normalizing build2/jmods/javafx.controls.jmod
Normalizing build2/jmods/javafx.fxml.jmod
Normalizing build2/jmods/javafx.graphics.jmod
Normalizing build2/jmods/javafx.media.jmod
Normalizing build2/jmods/javafx.swing.jmod
Normalizing build2/jmods/javafx.web.jmod

After normalizing the JMOD files, the output of the two builds on each of the three operating systems is identical:

$ diff -qr linux/build1 linux/build2
$ diff -qr macos/build1 macos/build2
$ diff -qr win10/build1 win10/build2
$

The following commands show the count of identical output files for each system:

$ diff -sr linux/build1 linux/build2 | wc -l
10676
$ diff -sr macos/build1 macos/build2 | wc -l
10680
$ diff -sr win10/build1 win10/build2 | wc -l
10633

Considering the time it takes to run all these builds, I'd like to take a week to get feedback and mull over the code changes before anyone spends a lot of time testing. I wish the changes for Windows were not so scattered about, for example.

-------------

PR: https://git.openjdk.java.net/jfx/pull/446