RFR: 8264449: Enable reproducible builds with SOURCE_DATE_EPOCH [v15]

John Neffenger jgneff at openjdk.org
Thu Jun 15 22:07:20 UTC 2023


On Tue, 13 Jun 2023 14:27:32 GMT, John Neffenger <jgneff at openjdk.org> wrote:

>> This pull request allows for reproducible builds of JavaFX on Linux, macOS, and Windows by defining the `SOURCE_DATE_EPOCH` environment variable. For example, the following commands create a reproducible build:
>> 
>> 
>> $ export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)
>> $ bash gradlew sdk jmods javadoc
>> $ strip-nondeterminism -v -T $SOURCE_DATE_EPOCH build/jmods/*.jmod
>> 
>> 
>> The three commands:
>> 
>> 1. set the build timestamp to the date of the latest source code change,
>> 2. build the JavaFX SDK libraries, JMOD archives, and API documentation, and
>> 3. recreate the JMOD files with stable file modification times and ordering.
>> 
>> The third command won't be necessary once Gradle can build the JMOD archives or the `jmod` tool itself has the required support. For more information on the environment variable, see the [`SOURCE_DATE_EPOCH`][1] page. For more information on the command to recreate the JMOD files, see the [`strip-nondeterminism`][2] repository. I'd like to propose that we allow for reproducible builds in JavaFX 17 and consider making them the default in JavaFX 18.
>> 
>> #### Fixes
>> 
>> There are at least four sources of non-determinism in the JavaFX builds:
>> 
>> 1. Build timestamp
>> 
>>     The class `com.sun.javafx.runtime.VersionInfo` in the JavaFX Base module stores the time of the build. Furthermore, for builds that don't run on the Hudson continuous integration tool, the class adds the build time to the system property `javafx.runtime.version`.
>> 
>> 2. Modification times
>> 
>>     The JAR, JMOD, and ZIP archives store the modification time of each file.
>> 
>> 3. File ordering
>> 
>>     The JAR, JMOD, and ZIP archives store their files in the order returned by the file system. The native shared libraries also store their object files in the order returned by the file system. Most file systems, though, do not guarantee the order of a directory's file listing.
>> 
>> 4. Build path
>> 
>>     The class `com.sun.javafx.css.parser.Css2Bin` in the JavaFX Graphics module stores the absolute path of its `.css` input file in the corresponding `.bss` output file, which is then included in the JavaFX Controls module.
>> 
>> This pull request modifies the Gradle and Groovy build files to fix the first three sources of non-determinism. A later pull request can modify the Java files to fix the fourth.
>> 
>> [1]: https://reproducible-builds.org/docs/source-date-epoch/
>> [2]: https://salsa.debian.org/reproducible-builds/strip-nondeterminism
>
> John Neffenger has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 24 commits:
> 
>  - Merge branch 'master' into allow-reproducible-builds
>  - Merge branch 'master' into allow-reproducible-builds
>  - Revert format of timestamp in version OPT field
>  - Merge branch 'master' into allow-reproducible-builds
>    
>    Include two commits that fix WebKit build failures on Windows and macOS:
>    
>      8282359: Intermittent WebKit build failure on Windows:
>               C1090: PDB API call failed, error code 23
>      8286089: Intermittent WebKit build failure on macOS in JavaScriptCore
>  - Merge branch 'master' into allow-reproducible-builds
>  - Support JDK 17 GA or later for building JavaFX
>  - Merge branch 'master' into allow-reproducible-builds
>  - Add '--date' argument for deterministic JMOD files
>  - Merge branch 'master' into allow-reproducible-builds
>  - Merge branch 'master' into allow-reproducible-builds
>  - ... and 14 more: https://git.openjdk.org/jfx/compare/72be85ec...fd39e6d1

I upgraded Visual Studio 2022 to version 17.5.5 on Windows and CMake to version 3.26.4 on all of the platforms, so now I'm using the following tools:

### Toolchains

* **Linux:** gcc (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0
* **macOS:** Command Line Tools for Xcode 14.2 version 14.2.0.0.1.1668646533
* **Windows:** Visual Studio Build Tools 2022 version 17.5.5, build number 17.5.33627.172

### Build Tools

* **JDK:** OpenJDK Runtime Environment (build 19.0.2+7-44)
* **Ant:** Apache Ant(TM) version 1.10.13 compiled on January 4 2023
* **CMake:** cmake version 3.26.4

I ran the builds as described in the [jgneff/jfxbuild][1] repository. The results are shown below.

### Linux

Only the file `libjfxwebkit.so` differs between builds:


ubuntu at openjfx:~/src/jfx$ for n in 1,2 3,4 5,6; do (set -x; diff -qr build[$n]); done
+ diff -qr build1 build2
+ diff -qr build3 build4
+ diff -qr build5 build6
Files build5/jmods/javafx.web.jmod and build6/jmods/javafx.web.jmod differ
Files build5/modular-sdk/modules_libs/javafx.web/libjfxwebkit.so and build6/modular-sdk/modules_libs/javafx.web/libjfxwebkit.so differ
Files build5/publications/javafx.web-linux.jar and build6/publications/javafx.web-linux.jar differ
Files build5/sdk/lib/libjfxwebkit.so and build6/sdk/lib/libjfxwebkit.so differ


### macOS

Only the file `libjavafx_iio.dylib` differs between builds, but I believe that was just by chance. Other test runs often show no differences at all or differences in other native libraries. Xcode version 14.3 may fix these differences, but my late 2014 Mac mini only supports up to Xcode 14.2.


john at macwifi:~/src/jfx$ for n in 1,2 3,4 5,6; do (set -x; diff -qr build[$n]); done
+ diff -qr build1 build2
Files build1/jmods/javafx.graphics.jmod and build2/jmods/javafx.graphics.jmod differ
Files build1/modular-sdk/modules_libs/javafx.graphics/libjavafx_iio.dylib and build2/modular-sdk/modules_libs/javafx.graphics/libjavafx_iio.dylib differ
Files build1/publications/javafx.graphics-mac.jar and build2/publications/javafx.graphics-mac.jar differ
Files build1/sdk/lib/libjavafx_iio.dylib and build2/sdk/lib/libjavafx_iio.dylib differ
+ diff -qr build3 build4
+ diff -qr build5 build6


### Windows

Only the file `jfxwebkit.dll` differs between builds:


john at windows:~/src/jfx$ for n in 1,2 3,4 5,6; do (set -x; diff -qr build[$n]); done
+ diff -qr build1 build2
+ diff -qr build3 build4
+ diff -qr build5 build6
Files build5/jmods/javafx.web.jmod and build6/jmods/javafx.web.jmod differ
Files build5/modular-sdk/modules_libs/javafx.web/jfxwebkit.dll and build6/modular-sdk/modules_libs/javafx.web/jfxwebkit.dll differ
Files build5/publications/javafx.web-win.jar and build6/publications/javafx.web-win.jar differ
Files build5/sdk/bin/jfxwebkit.dll and build6/sdk/bin/jfxwebkit.dll differ


I found another build failure on Windows when running the latest Visual Studio 2022 [Build Tools version 17.6.3][2], dated June 13, 2023. I opened the bug report, "[JDK-8310161][3]: Error C2666 compiling ICU 73.1 with VS 2022 17.6," as a *heads-up* for when we upgrade Visual Studio.

[1]: https://github.com/jgneff/jfxbuild
[2]: https://learn.microsoft.com/en-us/visualstudio/releases/2022/release-history#fixed-version-bootstrappers
[3]: https://bugs.openjdk.org/browse/JDK-8310161

-------------

PR Comment: https://git.openjdk.org/jfx/pull/446#issuecomment-1593779934


More information about the openjfx-dev mailing list