RFR: 8264449: Enable reproducible builds with SOURCE_DATE_EPOCH [v6]

Mon Nov 22 23:44:21 UTC 2021

On Mon, 22 Nov 2021 06:43:30 GMT, John Neffenger <jgneff at openjdk.org> wrote:

>> This pull request allows for reproducible builds of JavaFX on Linux, macOS, and Windows by defining the `SOURCE_DATE_EPOCH` environment variable. For example, the following commands create a reproducible build:
>> 
>> 
>> $ export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)
>> $ bash gradlew sdk jmods javadoc
>> $ strip-nondeterminism -v -T $SOURCE_DATE_EPOCH build/jmods/*.jmod
>> 
>> 
>> The three commands:
>> 
>> 1. set the build timestamp to the date of the latest source code change,
>> 2. build the JavaFX SDK libraries, JMOD archives, and API documentation, and
>> 3. recreate the JMOD files with stable file modification times and ordering.
>> 
>> The third command won't be necessary once Gradle can build the JMOD archives or the `jmod` tool itself has the required support. For more information on the environment variable, see the [`SOURCE_DATE_EPOCH`][1] page. For more information on the command to recreate the JMOD files, see the [`strip-nondeterminism`][2] repository. I'd like to propose that we allow for reproducible builds in JavaFX 17 and consider making them the default in JavaFX 18.
>> 
>> #### Fixes
>> 
>> There are at least four sources of non-determinism in the JavaFX builds:
>> 
>> 1. Build timestamp
>> 
>>     The class `com.sun.javafx.runtime.VersionInfo` in the JavaFX Base module stores the time of the build. Furthermore, for builds that don't run on the Hudson continuous integration tool, the class adds the build time to the system property `javafx.runtime.version`.
>> 
>> 2. Modification times
>> 
>>     The JAR, JMOD, and ZIP archives store the modification time of each file.
>> 
>> 3. File ordering
>> 
>>     The JAR, JMOD, and ZIP archives store their files in the order returned by the file system. The native shared libraries also store their object files in the order returned by the file system. Most file systems, though, do not guarantee the order of a directory's file listing.
>> 
>> 4. Build path
>> 
>>     The class `com.sun.javafx.css.parser.Css2Bin` in the JavaFX Graphics module stores the absolute path of its `.css` input file in the corresponding `.bss` output file, which is then included in the JavaFX Controls module.
>> 
>> This pull request modifies the Gradle and Groovy build files to fix the first three sources of non-determinism. A later pull request can modify the Java files to fix the fourth.
>> 
>> [1]: https://reproducible-builds.org/docs/source-date-epoch/
>> [2]: https://salsa.debian.org/reproducible-builds/strip-nondeterminism
>
> John Neffenger has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains ten commits:
> 
>  - Get build timestamp in UTC and set ZIP timestamps
>    
>    Create the build timestamp as a zoned date and time in UTC instead
>    of a local date and time. If SOURCE_DATE_EPOCH is defined, set the
>    last modification time of ZIP and JAR entries to the local date and
>    time in UTC of the instant defined by SOURCE_DATE_EPOCH.
>    
>    Add a comment as a reminder to make JMOD files deterministic when
>    the following enhancements are complete:
>    
>    * Enable deterministic file content ordering for Jar and Jmod
>      https://bugs.openjdk.java.net/browse/JDK-8276764
>      https://github.com/openjdk/jdk/pull/6395
>    
>    * Enable jar and jmod to produce deterministic timestamped content
>      https://bugs.openjdk.java.net/browse/JDK-8276766
>      https://github.com/openjdk/jdk/pull/6481
>  - Merge branch 'master' into allow-reproducible-builds
>  - Make build of SDK ZIP bundle reproducible
>  - Merge branch 'master' into allow-reproducible-builds
>  - Merge branch 'master' into allow-reproducible-builds
>  - Include WebKit shared library for Windows
>    
>    Enable reproducible builds of the native WebKit shared library for
>    Windows (jfxwebkit.dll) when SOURCE_DATE_EPOCH is defined.
>  - Include media shared libraries for Windows
>    
>    Enable reproducible builds of the native media shared libraries for
>    Windows when SOURCE_DATE_EPOCH is defined. The libraries are:
>    
>      fxplugins.dll
>      glib-lite.dll
>      gstreamer-lite.dll
>      jfxmedia.dll
>  - Enable reproducible builds with SOURCE_DATE_EPOCH
>  - 8238650: Allow to override buildDate with SOURCE_DATE_EPOCH

The results of 33 builds are shown in the table below:

| Builds  | Linux | macOS | Win10 |
| ------- |:-----:|:-----:|:-----:|
| Develop | ✔ | ✔ | ✔ |
| Actions | ✔ | ✔ | ✔ |
| Release | ✔ | ✔ | ✔ |

where the check mark (✔) means that the unit tests ran successfully and that the files in the `build` directory where identical between two builds on the same system except for the `libjfxwebkit.dylib` file on macOS and the JMOD files on all of the systems.

The build types are shown below by their Gradle options and tasks:

* **Develop:** `bash gradlew --no-daemon sdk jmods javadoc test -x :web:test`
* **Actions:** `bash gradlew --no-daemon all test -x :web:test`
* **Release:** `bash gradlew --no-daemon -PCONF=Release -PPROMOTED_BUILD_NUMBER=7 -PHUDSON_BUILD_NUMBER=101 -PHUDSON_JOB_NAME=jfx -PCOMPILE_WEBKIT=true -PCOMPILE_MEDIA=true -PBUILD_LIBAV_STUBS=true sdk jmods javadoc test`

I ran each set of builds as shown by the following example:

$ export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)
$ bash gradlew --no-daemon cleanAll
$ bash gradlew --no-daemon all
$ mv build build1
$ bash gradlew --no-daemon cleanAll
$ bash gradlew --no-daemon all
$ mv build build2
$ bash gradlew --no-daemon all test -x :web:test

I ran two more builds with `SOURCE_DATE_EPOCH` undefined on each system: a reference *Develop* build and a *Release* build with its unit tests.

I also confirmed that the builds are reproducible, except for the JMOD files, on all of the following Linux architectures running Ubuntu 18.04 LTS:

| Builds  | amd64 | arm64 | armhf | i386 | ppc64el | s390x |
| ------- |:-----:|:-----:|:-----:|:----:|:-------:|:-----:|
| Develop | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |

One bonus with reproducible builds is that you can compare across systems. For example, the following JAR files are now identical across operating systems and hardware architectures, just as one would expect:

* javafx.base.jar
* javafx.fxml.jar
* javafx.swing.jar
* javafx-swt.jar
* javafx.web.jar

-------------

PR: https://git.openjdk.java.net/jfx/pull/446