RFR: 8256127: Add cross-compiled foreign architectures builds to submit workflow [v7]
Aleksey Shipilev
shade at openjdk.java.net
Fri Nov 13 11:19:30 UTC 2020
> It is [possible](https://github.com/openjdk/jdk/blob/master/doc/building.md#creating-and-using-sysroots-with-qemu-deboostrap) to efficiently cross-compile to foreign architectures on current GH actions that are driven by Ubuntu. I have been using this method for years to produce binaries at [builds.shipilev.net](https://builds.shipilev.net).
>
> These cross-compilation targets frequently find arch-specific build bugs. Hotspot is rather well insulated from the fundamental problems in arch-support, so the overwhelming majority of those arch-specific bugs are simple omissions of `#include`-s, inconsistent `#ifdef`-ing, missed method renames / signature changes, or incorrect removals of methods that are used by arch-specific code. Those are straightforward to fix, if the contributor knows about them. My experience says the common attitude to these fixes is: "Oh, I would have fixed it in the original change if I knew about this."
>
> This improvement adds several foreign architectures to GH actions workflow to provide the automatic safety net to give early warning for these cases. In fact, many Hotspot contributors already build AArch64, ARM, PPC64 targets as pre-integration sanity checks, and this does the important checks automatically for them.
>
> To optimize workflow costs, the change does the following:
> - The build is only done for hotspot-debug-no-pch, as the most frequent place where arch-specific build bugs show up.
> - There are no tests, and in fact the arch-specific binary does not even run. The build is purely about the build.
> - Foreign architectures reuse the Linux x86_64 release build as build JDK. This avoids building "host" build JDK as it would otherwise happen with cross-compiling.
> - The created sysroot is cached and keyed on `submit.yml` hash. So it would regenerate only if the workflow itself changes. This saves about 10 minutes per arch.
>
> Space-wise, sysroots in GH cache take about 580M uncompressed, and 270M zstd-compressed bundle. In the end, this adds 4x270 = 1080M into local cache.
>
> Time-wise, ball-parking the current workflow budget [looks like this](https://github.com/shipilev/jdk/runs/1389151071):
> - Linux x86_64:
> - builds: 120m
> - tests: 200m
> - Linux x86_32:
> - builds: 75m
> - tests: 235m
> - Windows x86_64
> - builds: 120m
> - tests: 290m
> - MacOS x86_64
> - builds: 75m
> - tests: 165m
> - Linux aarch64:
> - builds: 25m (+10m to create uncached sysroot)
> - Linux arm:
> - builds: 20m (+10m to create uncached sysroot)
> - Linux ppc64le:
> - builds: 20m (+15m to create uncached sysroot)
> - Linux s390x:
> - builds: 20m (+10m to create uncached sysroot)
>
> In other words, new workflow takes about 715 Linux-host-minutes, 410 Windows-host-minutes, 240 Mac-host-minutes. Out of which new jobs take about 85 Linux-host-minutes. So the cost of new jobs is roughly:
> - 11.9% of Linux-host-minutes
> - 6.2% of all-host-minutes
> - 2.2% of runner-minutes, if you [weigh in the proportional cost of non-Linux runners](https://docs.github.com/en/free-pro-team@latest/github/setting-up-and-managing-billing-and-payments-on-github/about-billing-for-github-actions)
>
> In other words, the additional runner costs are pale in comparison with what is already done in build and test jobs.
Aleksey Shipilev has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains eight additional commits since the last revision:
- Merge branch 'master' into JDK-8256127-foreign-build
- Build only hotspot reduced build
- Merge branch 'master' into JDK-8256127-foreign-build
- Merge branch 'master' into JDK-8256127-foreign-build
- Incomplete fix: remove special dirs
- Merge branch 'master' into JDK-8256127-foreign-build
- Remove special directories from chroot to make it archivable
- 8256127: Add cross-compiled foreign architectures builds to submit workflow
-------------
Changes:
- all: https://git.openjdk.java.net/jdk/pull/1147/files
- new: https://git.openjdk.java.net/jdk/pull/1147/files/81e07fe2..b0dfab5f
Webrevs:
- full: https://webrevs.openjdk.java.net/?repo=jdk&pr=1147&range=06
- incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=1147&range=05-06
Stats: 13479 lines in 272 files changed: 7889 ins; 3996 del; 1594 mod
Patch: https://git.openjdk.java.net/jdk/pull/1147.diff
Fetch: git fetch https://git.openjdk.java.net/jdk pull/1147/head:pull/1147
PR: https://git.openjdk.java.net/jdk/pull/1147
More information about the build-dev
mailing list