RFR: 8264449: Enable reproducible builds with SOURCE_DATE_EPOCH [v5]

Fri Sep 17 23:28:03 UTC 2021

On Mon, 14 Jun 2021 20:53:50 GMT, John Neffenger <jgneff at openjdk.org> wrote:

>> This pull request allows for reproducible builds of JavaFX on Linux, macOS, and Windows by defining the `SOURCE_DATE_EPOCH` environment variable. For example, the following commands create a reproducible build:
>> 
>> 
>> $ export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)
>> $ bash gradlew sdk jmods javadoc
>> $ strip-nondeterminism -v -T $SOURCE_DATE_EPOCH build/jmods/*.jmod
>> 
>> 
>> The three commands:
>> 
>> 1. set the build timestamp to the date of the latest source code change,
>> 2. build the JavaFX SDK libraries, JMOD archives, and API documentation, and
>> 3. recreate the JMOD files with stable file modification times and ordering.
>> 
>> The third command won't be necessary once Gradle can build the JMOD archives or the `jmod` tool itself has the required support. For more information on the environment variable, see the [`SOURCE_DATE_EPOCH`][1] page. For more information on the command to recreate the JMOD files, see the [`strip-nondeterminism`][2] repository. I'd like to propose that we allow for reproducible builds in JavaFX 17 and consider making them the default in JavaFX 18.
>> 
>> #### Fixes
>> 
>> There are at least four sources of non-determinism in the JavaFX builds:
>> 
>> 1. Build timestamp
>> 
>>     The class `com.sun.javafx.runtime.VersionInfo` in the JavaFX Base module stores the time of the build. Furthermore, for builds that don't run on the Hudson continuous integration tool, the class adds the build time to the system property `javafx.runtime.version`.
>> 
>> 2. Modification times
>> 
>>     The JAR, JMOD, and ZIP archives store the modification time of each file.
>> 
>> 3. File ordering
>> 
>>     The JAR, JMOD, and ZIP archives store their files in the order returned by the file system. The native shared libraries also store their object files in the order returned by the file system. Most file systems, though, do not guarantee the order of a directory's file listing.
>> 
>> 4. Build path
>> 
>>     The class `com.sun.javafx.css.parser.Css2Bin` in the JavaFX Graphics module stores the absolute path of its `.css` input file in the corresponding `.bss` output file, which is then included in the JavaFX Controls module.
>> 
>> This pull request modifies the Gradle and Groovy build files to fix the first three sources of non-determinism. A later pull request can modify the Java files to fix the fourth.
>> 
>> [1]: https://reproducible-builds.org/docs/source-date-epoch/
>> [2]: https://salsa.debian.org/reproducible-builds/strip-nondeterminism
>
> John Neffenger has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains seven commits:
> 
>  - Make build of SDK ZIP bundle reproducible
>  - Merge branch 'master' into allow-reproducible-builds
>  - Merge branch 'master' into allow-reproducible-builds
>  - Include WebKit shared library for Windows
>    
>    Enable reproducible builds of the native WebKit shared library for
>    Windows (jfxwebkit.dll) when SOURCE_DATE_EPOCH is defined.
>  - Include media shared libraries for Windows
>    
>    Enable reproducible builds of the native media shared libraries for
>    Windows when SOURCE_DATE_EPOCH is defined. The libraries are:
>    
>      fxplugins.dll
>      glib-lite.dll
>      gstreamer-lite.dll
>      jfxmedia.dll
>  - Enable reproducible builds with SOURCE_DATE_EPOCH
>  - 8238650: Allow to override buildDate with SOURCE_DATE_EPOCH

I did a set of full builds on all three platforms today, and can verify that the builds are reproducible, when run on the same system and build dir, for all artifacts except the jmods, which you point out as a known issue. There is one caveat related to time zones listed below.

All but two of the changes only affect reproducible builds, meaning that they only take effect when `SOURCE_DATE_EPOCH` is set. The following two changes affect all builds unconditionally, even when not using SOURCE_DATE_EPOCH:

* removal of the (obsolete) jar indexing for javafx-swt.jar
* sorting the list of files to be linked

I think it's OK to not qualify the above with a check for `SOURCE_DATE_EPOCH`.

When `SOURCE_DATE_EPOCH` is set, I see the following problems:

* The timestamps for all files in the zip archives are set to a hard-coded "1980-02-01", rather than the date and time specified by `SOURCE_DATE_EPOCH`. This is both annoying, and a potential for bugs (e.g., a program looking at the time stamps of the various files in the archive could get confused). If no good solution can be found now, I want to see a follow-up bug for this. I would be reluctant to recommend the use of SOURCE_DATE_EPOCH until this can be fixed.
* The conversion of `SOURCE_DATE_EPOCH` into `buildTimeStamp` depends on the time zone of the build machine. This means that you won't be able to compare two builds using the same time stamp on two different builds machines with different time zones. One solution would be to do the conversion such that the buildTimeStamp was always shown in UTC. This can be done as a follow-up issue, if you like.

I'm doing a CI build and will verify the results (and probably do a bit more testing) when I get a chance, but I wanted to pass on this feedback today.

build.gradle line 559:

> 557:     buildDate = new java.util.Date(ms)
> 558: }
> 559: def buildTimestamp = new java.text.SimpleDateFormat("yyyy-MM-dd-HHmmss").format(buildDate)

I think it would be useful to format `buildDate` using UTC as the time zone when `SOURCE_DATE_EPOCH` is set.

build.gradle line 3518:

> 3516:                         def lFlags = webkitProperties.linkFlags?.join(' ') ?: ''
> 3517:                         cmakeArgs = "$cmakeArgs -DCMAKE_C_FLAGS='${cFlags}' -DCMAKE_CXX_FLAGS='${cFlags}'"
> 3518:                         cmakeArgs = "$cmakeArgs -DCMAKE_SHARED_LINKER_FLAGS='${lFlags}'"

I presume you've tested this both with and without `SOURCE_DATE_EPOCH`?

build.gradle line 3914:

> 3912:     tasks.withType(AbstractArchiveTask) {
> 3913:         if (sourceDateEpoch != null) {
> 3914:             preserveFileTimestamps = false

This is a problem given how gradle generates a zip archive when this is set. How hard would it be to force all time stamps to be `SOURCE_DATE_EPOCH`?

build.gradle line 4238:

> 4236:     def sdkTask = task("sdk$t.capital") {
> 4237:         group = "Basic"
> 4238:         dependsOn(javafxSwtTask)

As mentioned in the general comments, this change affects builds even when not using `SOURCE_DATE_EPOCH`. I think this is fine.

-------------

PR: https://git.openjdk.java.net/jfx/pull/446