RFR(S): 8213698: Improve devkit creation and add support for linux/ppc64/ppc64le/s390x
Magnus Ihse Bursie
magnus.ihse.bursie at oracle.com
Thu Nov 22 16:09:55 UTC 2018
On 2018-11-22 11:27, Volker Simonis wrote:
> On Tue, Nov 20, 2018 at 1:46 PM Magnus Ihse Bursie
> <magnus.ihse.bursie at oracle.com> wrote:
>> Hi Volker,
>>
>> Glad to see you fix and clean up the devkit scripts! And you're more
>> than welcome to add documentation to the building.md.
>>
> OK, I've added a section about devkits to build.{md,html}. Please have
> a look and let me know what you think. Any comments are welcome :)
>
> http://cr.openjdk.java.net/~simonis/webrevs/2018/8213698.v2/
Looks good, all changes below, and the additional readme text!
A hint (you don't need to update your changes): if you have long links
in markdown, instead of having to line break them
like this (and move a word to the new line)
described in [section 3.4 of the GNU
Autobook](https://sourceware.org/autobook/autobook/autobook_17.html). If
you can put a line break right before the URL, after the opening
parenthesis, like this:
described in [section 3.4 of the GNU Autobook]( https://sourceware.org/autobook/autobook/autobook_17.html). If
It usually improves readability of the markdown file.
/Magnus
>
> Please find my comments to the other changes inline
>
>> I think all your changes are good, and can go in as-is, but I see some
>> further potential for cleanup, so if you feel like some more fixes while
>> you're at it, please go ahead. Otherwise you can ignore the rest of what
>> I'm writing. :)
>>
>>> make cross_compile_target="ppc64-linux-gnu" BASE_OS=Fedora BASE_OS_VERSION=17
>> For consistency, maybe we should rename cross_compile_target to TARGET?
>> At least as written on the make command line; you can keep the
>> cross_compile_target name in the Makefile itself, and only assign to it
>> from TARGET.
>>
> I've renamed it to TARGETS because it can actually take a whole list
> of targets (and also to distinguish it from the TARGET parameter used
> in Tools.gmk which only accepts a single TARGET).
>
>>> OEL_URL :=
>>> https://archives.fedoraproject.org/pub/archive/$(FEDORA_TYPE)/releases/$(BASE_OS_VERSION)/Everything/$(ARCH)/os/Packages/
>> Maybe it's time to rename this to something more distro-agnostic than
>> OEL_URL...
>>
> Renamed it to BASE_URL
>
>>> PREFIX=$(RESULT)/$@-to-$$p
>> I really like this! Devkits are much more better described as X-to-Y
>> (even if that was not originally the case, as Erik said).
>>
> Thanks :)
>
>>> Notice that we have to build "ccache" for each target because ccache
>>> will be installed into the directory specified by "--prefix" at
>>> configure time and this is now different for every target. However
>>> that's not a big problem, because the time for compiling ccache is
>>> negligible compared to the download time of the RPMs and the build
>>> time of GCC.
>> Is it still worth the effort of compiling ccache? At least in Oracle,
>> we're not using it anymore. OTOH, if it works, we can just keep it
>> there. It's no big cost.
>>
> I'm not using it either, but I leave it in for now.
>
>>> - Cleanup arch selection:
>>>
>>> ifeq ($(ARCH),x86_64)
>>> - RPM_ARCHS := x86_64 noarch
>>> + RPM_ARCHS := $(ARCH) noarch
>>> ifeq ($(BUILD),$(HOST))
>>> ifeq ($(TARGET),$(HOST))
>>> # When building the native compiler for x86_64, enable mixed mode.
>>> @@ -199,7 +206,7 @@
>>> endif
>>> endif
>>> else ifeq ($(ARCH),i686)
>>> - RPM_ARCHS := i386 i686 noarch
>>> + RPM_ARCHS := $(ARCH) i386 noarch
>>> else ifeq ($(ARCH), armhfp)
>> Looks like you could always to RPM_ARCHS := $(ARCH) noarch, and then use
>> += for the few platforms that need additionals RPM archs.
>>
> Done.
>
>> There was something more I thought of when I reviewed your changes, but
>> it slipped my mind now.
>>
> There's still room for improvements :) For example the GCC version
> could be made configurable, as well as the OEL version. But I leave
> that to the person who updates the compiler the next time :)
>
>> /Magnus
>>
>>
>> On 2018-11-12 12:19, Volker Simonis wrote:
>>> Hi,
>>>
>>> can I please have a review for the following change which ads support
>>> for linux/ppc64/ppc64le/s390x devkits and hopefully improves the
>>> creation of devkits a little bit :)
>>>
>>> http://cr.openjdk.java.net/~simonis/webrevs/2018/8213698/
>>> https://bugs.openjdk.java.net/browse/JDK-8213698
>>>
>>> With these changes it becomes possible to say any of the following:
>>>
>>> make cross_compile_target="ppc64le-linux-gnu s390x-linux-gnu" BASE_OS=Fedora
>>> make cross_compile_target="ppc64-linux-gnu" BASE_OS=Fedora BASE_OS_VERSION=17
>>> make onlytars cross_compile_target="ppc64-linux-gnu ppc64le-linux-gnu
>>> s390x-linux-gnu"
>>>
>>> and get the following devkits under "build/devkit/result":
>>>
>>> sdk-x86_64-unknown-linux-gnu-to-ppc64le-linux-gnu-20181112.tar.gz
>>> sdk-x86_64-unknown-linux-gnu-to-ppc64-linux-gnu-20181112.tar.gz
>>> sdk-x86_64-unknown-linux-gnu-to-s390x-linux-gnu-20181112.tar.gz
>>> x86_64-unknown-linux-gnu-to-ppc64le-linux-gnu/
>>> x86_64-unknown-linux-gnu-to-ppc64-linux-gnu/
>>> x86_64-unknown-linux-gnu-to-s390x-linux-gnu/
>>>
>>> Below you can find a more detailed description of the various changes.
>>> Once we've discussed and agreed on the changes I'd like to add a small
>>> documentation about how to build and use devkits to
>>> "doc/building.{md,html}" which describes the creation and usage of
>>> devkits. This documentation should be right at the top of the
>>> "Cross-compiling" section which is quite complex now. It was not clear
>>> to me until yet how trivial the creation and usage of a devkit can be
>>> :)
>>>
>>>
>>> - The changes required for supporting linux/ppc64/ppc64le/s390x are trivial:
>>>
>>> make/devkit/Tools.gmk
>>>
>>> +ifneq ($(filter ppc64 ppc64le s390x, $(ARCH)), )
>>> + # We only support 64-bit on these platforms anyway
>>> + CONFIG += --disable-multilib
>>> +endif
>>>
>>> This is required to prevent building of multilib toolchains which
>>> arent't needed anyway. The problem with the multilib toolchain build
>>> is that it requires some special 32-bit headers which arn't installed
>>> by default from the current RPM list.
>>>
>>> - The following change allows users to choose the version of Fedora
>>> which is used to create the sysroot environment by setting
>>> "BASE_OS_VERSION" (with "27" being the default). This works "BASE_OS"
>>> will be set to "Fedora" (as opposed to "Fedora27" before). Notice that
>>> older Fedora versions have a sligthly different download URL:
>>>
>>> make/devkit/Tools.gmk
>>>
>>> ifeq ($(BASE_OS), OEL6)
>>> OEL_URL := http://yum.oracle.com/repo/OracleLinux/OL6/4/base/$(ARCH)/
>>> LINUX_VERSION := OEL6.4
>>> -else ifeq ($(BASE_OS), Fedora27)
>>> - ifeq ($(ARCH), aarch64)
>>> +else ifeq ($(BASE_OS), Fedora)
>>> + DEFAULT_OS_VERSION := 27
>>> + ifeq ($(BASE_OS_VERSION), )
>>> + BASE_OS_VERSION := $(DEFAULT_OS_VERSION)
>>> + endif
>>> + ifeq ($(filter x86_64 armhfp, $(ARCH)), )
>>> FEDORA_TYPE=fedora-secondary
>>> else
>>> FEDORA_TYPE=fedora/linux
>>> endif
>>> - OEL_URL := https://dl.fedoraproject.org/pub/$(FEDORA_TYPE)/releases/27/Everything/$(ARCH)/os/Packages/
>>> - LINUX_VERSION := Fedora 27
>>> + ARCHIVED := $(shell [ $(BASE_OS_VERSION) -lt $(DEFAULT_OS_VERSION)
>>> ] && echo true)
>>> + ifeq ($(ARCHIVED),true)
>>> + OEL_URL :=
>>> https://archives.fedoraproject.org/pub/archive/$(FEDORA_TYPE)/releases/$(BASE_OS_VERSION)/Everything/$(ARCH)/os/Packages/
>>> + else
>>> + OEL_URL :=
>>> https://dl.fedoraproject.org/pub/$(FEDORA_TYPE)/releases/$(BASE_OS_VERSION)/Everything/$(ARCH)/os/Packages/
>>> + endif
>>> + LINUX_VERSION := Fedora $(BASE_OS_VERSION)
>>> else
>>> $(error Unknown base OS $(BASE_OS))
>>> endif
>>>
>>> - Enable the creation of several different devkits at once (e.g. 'make
>>> cross_compile_target="ppc64-linux-gnu ppc64le-linux-gnu
>>> s390x-linux-gnu"') or one after another but all into the same
>>> 'build/devkit/result' directory. The result directory will contain
>>> $HOST-to-$TARGET sub-directories with the corresponding devkits:
>>>
>>> make/devkit/Makefile
>>>
>>> -submakevars = HOST=$@ BUILD=$(me) \
>>> - RESULT=$(RESULT) PREFIX=$(RESULT)/$@ \
>>> - OUTPUT_ROOT=$(OUTPUT_ROOT)
>>> +submakevars = HOST=$@ BUILD=$(me) RESULT=$(RESULT) OUTPUT_ROOT=$(OUTPUT_ROOT)
>>> +
>>> $(host_platforms) :
>>> @echo 'Building compilers for $@'
>>> @echo 'Targets: $(target_platforms)'
>>> for p in $(filter $@, $(target_platforms)) $(filter-out $@,
>>> $(target_platforms)); do \
>>> - $(MAKE) -f Tools.gmk download-rpms $(submakevars) TARGET=$$p && \
>>> + $(MAKE) -f Tools.gmk download-rpms $(submakevars) \
>>> + TARGET=$$p PREFIX=$(RESULT)/$@-to-$$p && \
>>> $(MAKE) -f Tools.gmk all $(submakevars) \
>>> - TARGET=$$p || exit 1 ; \
>>> + TARGET=$$p PREFIX=$(RESULT)/$@-to-$$p && \
>>> + $(MAKE) -f Tools.gmk ccache $(submakevars) \
>>> + TARGET=$@ PREFIX=$(RESULT)/$@-to-$$p
>>> BUILDDIR=$(OUTPUT_ROOT)/$@/$$p || exit 1 ; \
>>> done
>>> - @echo 'Building ccache program for $@'
>>> - $(MAKE) -f Tools.gmk ccache $(submakevars) TARGET=$@
>>> @echo 'All done"'
>>>
>>> Notice that we have to build "ccache" for each target because ccache
>>> will be installed into the directory specified by "--prefix" at
>>> configure time and this is now different for every target. However
>>> that's not a big problem, because the time for compiling ccache is
>>> negligible compared to the download time of the RPMs and the build
>>> time of GCC.
>>>
>>> define Mktar
>>> - $(1)_tar = $$(RESULT)/sdk-$(1)-$$(today).tar.gz
>>> - $$($(1)_tar) : PLATFORM = $(1)
>>> - TARFILES += $$($(1)_tar)
>>> - $$($(1)_tar) : $(1) $$(shell find $$(RESULT)/$(1))
>>> + $(1)-to-$(2)_tar = $$(RESULT)/sdk-$(1)-to-$(2)-$$(today).tar.gz
>>> + $$($(1)-to-$(2)_tar) : PLATFORM = $(1)-to-$(2)
>>> + TARFILES += $$($(1)-to-$(2)_tar)
>>> + $$($(1)-to-$(2)_tar) : $$(shell find $$(RESULT)/$(1)-to-$(2) -type f)
>>> endef
>>>
>>> -$(foreach p,$(host_platforms),$(eval $(call Mktar,$(p))))
>>> +$(foreach p,$(host_platforms),$(foreach t,$(target_platforms),$(eval
>>> $(call Mktar,$(p),$(t)))))
>>>
>>> make/devkit/Tools.gmk
>>>
>>> -PATHEXT = $(RESULT)/$(BUILD)/bin:
>>> +PATHEXT = $(PREFIX)/bin:
>>>
>>>
>>> - Various small cleanups
>>>
>>> make/devkit/Tools.gmk
>>>
>>> - Don't set "RESULT" and "PREFIX" in Tools.gmk because the values are
>>> overridden by the settings in the calling Makefile anyway:
>>>
>>> # Define directories
>>> -RESULT := $(OUTPUT_ROOT)/result
>>> BUILDDIR := $(OUTPUT_ROOT)/$(HOST)/$(TARGET)
>>> -PREFIX := $(RESULT)/$(HOST)
>>> TARGETDIR := $(PREFIX)/$(TARGET)
>>>
>>> - Cleanup arch selection:
>>>
>>> ifeq ($(ARCH),x86_64)
>>> - RPM_ARCHS := x86_64 noarch
>>> + RPM_ARCHS := $(ARCH) noarch
>>> ifeq ($(BUILD),$(HOST))
>>> ifeq ($(TARGET),$(HOST))
>>> # When building the native compiler for x86_64, enable mixed mode.
>>> @@ -199,7 +206,7 @@
>>> endif
>>> endif
>>> else ifeq ($(ARCH),i686)
>>> - RPM_ARCHS := i386 i686 noarch
>>> + RPM_ARCHS := $(ARCH) i386 noarch
>>> else ifeq ($(ARCH), armhfp)
>>>
>>> - Don't create 'devkit.info' unconditinally. Only build it as part of
>>> the "all" target invocation. This prevents the creation of a
>>> 'devkit.info' with the wrong "BASE_OS_VERSION" if the Makefile is
>>> invoked several times with different "BASE_OS_VERSION" values:
>>>
>>> -$(PREFIX)/devkit.info: FRC
>>> +$(PREFIX)/devkit.info:
>>> @echo 'Creating devkit.info in the root of the kit'
>>> rm -f $@
>>> touch $@
>>> @@ -611,7 +623,4 @@
>>> # this is only built for host. so separate.
>>> ccache : $(ccache)
>>>
>>> -# Force target
>>> -FRC:
>>>
>>> - put the base directory of the devkits into the tar archives. I don't
>>> like tar archives which don't have a single top-level directory and
>>> expand right into the current working directory :)
>>>
>>> make/devkit/Makefile
>>>
>>> %.tar.gz :
>>> @echo 'Creating compiler package $@'
>>> - cd $(RESULT)/$(PLATFORM) && tar -czf $@ *
>>> + cd $(RESULT) && tar -czf $@ $(PLATFORM)/*
>>> touch $@
More information about the build-dev
mailing list