RFR(S): 8213698: Improve devkit creation and add support for linux/ppc64/ppc64le/s390x
Erik Joelsson
erik.joelsson at oracle.com
Thu Nov 15 16:57:46 UTC 2018
On 2018-11-15 08:17, Volker Simonis wrote:
> Hi Erik,
>
> thanks for looking at my changes (and providing some history, which is
> always interesting) Can I consider your answer a review from your
> site? I'd like to wait for Magnus to double check my patch but it's
> definitely good to know if you're fine with my changes :)
Yes, I'm ok with it.
Magnus will most likely not be able to answer until next week.
/Erik
> Thank you and best regards,
> Volker
>
> On Tue, Nov 13, 2018 at 6:34 PM Erik Joelsson <erik.joelsson at oracle.com> wrote:
>> Hello Volker,
>>
>> I think your changes look reasonable and I'm happy to see some external
>> interest in the devkits!
>>
>> To provide some history on the structure of these makefiles. They used
>> to support building full cross toolchains from and to all listed
>> platforms (creating one mega kit for each host platform with all the
>> others as targets), which was also the default behavior. Since we
>> weren't using this, I simplified it a bit. I think your design with
>> base-to-target directories makes more sense. You get separate kits for
>> each base and target combination, which is how we use them in practice
>> anyway.
>>
>> The presence of a top level dir in the tars is something I have debated
>> with myself a lot. The different devkit scripts aren't even behaving the
>> same in this regard. I agree that it's cleaner with a top level dir, but
>> in the world of automated installs, it adds yet another level of
>> directories to already deep paths. I'm ok with changing it however.
>>
>> /Erik
>>
>> On 2018-11-13 08:25, Volker Simonis wrote:
>>> As it seems that nobody has looked at this until now, I take the
>>> liberty to slightly update my proposal :)
>>>
>>> http://cr.openjdk.java.net/~simonis/webrevs/2018/8213698.v1/
>>>
>>> - add the Fedora version to the sub-directory under
>>> build/devkit/download/rpms/ which contains the target specific .rpm
>>> files. This saves a lot of time if experimenting with various sysroot
>>> versions:
>>>
>>> endif
>>> - LINUX_VERSION := Fedora $(BASE_OS_VERSION)
>>> + LINUX_VERSION := Fedora_$(BASE_OS_VERSION)
>>> else
>>> DOWNLOAD := $(OUTPUT_ROOT)/download
>>> -DOWNLOAD_RPMS := $(DOWNLOAD)/rpms/$(TARGET)
>>> +DOWNLOAD_RPMS := $(DOWNLOAD)/rpms/$(TARGET)-$(LINUX_VERSION)
>>> SRCDIR := $(OUTPUT_ROOT)/src
>>>
>>> - remove "unknown" from the host platform triplet. According to the
>>> "Autobook, §3.4 Configuration Names" [1] "..it is normally not
>>> necessary to specify an entire name. In particular, the middle field
>>> (manufacturer - which defaults to "unknown") is often omitted."
>>>
>>> # Figure out what platform this is building on.
>>> -me := $(cpu)-$(if $(findstring Linux,$(os)),unknown-linux-gnu)
>>> +me := $(cpu)-$(if $(findstring Linux,$(os)),linux-gnu)
>>>
>>> Thank you and best regards,
>>> Volker
>>>
>>> [1] https://sourceware.org/autobook/autobook/autobook_17.html
>>> On Mon, Nov 12, 2018 at 12:19 PM Volker Simonis
>>> <volker.simonis at gmail.com> wrote:
>>>> Hi,
>>>>
>>>> can I please have a review for the following change which ads support
>>>> for linux/ppc64/ppc64le/s390x devkits and hopefully improves the
>>>> creation of devkits a little bit :)
>>>>
>>>> http://cr.openjdk.java.net/~simonis/webrevs/2018/8213698/
>>>> https://bugs.openjdk.java.net/browse/JDK-8213698
>>>>
>>>> With these changes it becomes possible to say any of the following:
>>>>
>>>> make cross_compile_target="ppc64le-linux-gnu s390x-linux-gnu" BASE_OS=Fedora
>>>> make cross_compile_target="ppc64-linux-gnu" BASE_OS=Fedora BASE_OS_VERSION=17
>>>> make onlytars cross_compile_target="ppc64-linux-gnu ppc64le-linux-gnu
>>>> s390x-linux-gnu"
>>>>
>>>> and get the following devkits under "build/devkit/result":
>>>>
>>>> sdk-x86_64-unknown-linux-gnu-to-ppc64le-linux-gnu-20181112.tar.gz
>>>> sdk-x86_64-unknown-linux-gnu-to-ppc64-linux-gnu-20181112.tar.gz
>>>> sdk-x86_64-unknown-linux-gnu-to-s390x-linux-gnu-20181112.tar.gz
>>>> x86_64-unknown-linux-gnu-to-ppc64le-linux-gnu/
>>>> x86_64-unknown-linux-gnu-to-ppc64-linux-gnu/
>>>> x86_64-unknown-linux-gnu-to-s390x-linux-gnu/
>>>>
>>>> Below you can find a more detailed description of the various changes.
>>>> Once we've discussed and agreed on the changes I'd like to add a small
>>>> documentation about how to build and use devkits to
>>>> "doc/building.{md,html}" which describes the creation and usage of
>>>> devkits. This documentation should be right at the top of the
>>>> "Cross-compiling" section which is quite complex now. It was not clear
>>>> to me until yet how trivial the creation and usage of a devkit can be
>>>> :)
>>>>
>>>>
>>>> - The changes required for supporting linux/ppc64/ppc64le/s390x are trivial:
>>>>
>>>> make/devkit/Tools.gmk
>>>>
>>>> +ifneq ($(filter ppc64 ppc64le s390x, $(ARCH)), )
>>>> + # We only support 64-bit on these platforms anyway
>>>> + CONFIG += --disable-multilib
>>>> +endif
>>>>
>>>> This is required to prevent building of multilib toolchains which
>>>> arent't needed anyway. The problem with the multilib toolchain build
>>>> is that it requires some special 32-bit headers which arn't installed
>>>> by default from the current RPM list.
>>>>
>>>> - The following change allows users to choose the version of Fedora
>>>> which is used to create the sysroot environment by setting
>>>> "BASE_OS_VERSION" (with "27" being the default). This works "BASE_OS"
>>>> will be set to "Fedora" (as opposed to "Fedora27" before). Notice that
>>>> older Fedora versions have a sligthly different download URL:
>>>>
>>>> make/devkit/Tools.gmk
>>>>
>>>> ifeq ($(BASE_OS), OEL6)
>>>> OEL_URL := http://yum.oracle.com/repo/OracleLinux/OL6/4/base/$(ARCH)/
>>>> LINUX_VERSION := OEL6.4
>>>> -else ifeq ($(BASE_OS), Fedora27)
>>>> - ifeq ($(ARCH), aarch64)
>>>> +else ifeq ($(BASE_OS), Fedora)
>>>> + DEFAULT_OS_VERSION := 27
>>>> + ifeq ($(BASE_OS_VERSION), )
>>>> + BASE_OS_VERSION := $(DEFAULT_OS_VERSION)
>>>> + endif
>>>> + ifeq ($(filter x86_64 armhfp, $(ARCH)), )
>>>> FEDORA_TYPE=fedora-secondary
>>>> else
>>>> FEDORA_TYPE=fedora/linux
>>>> endif
>>>> - OEL_URL := https://dl.fedoraproject.org/pub/$(FEDORA_TYPE)/releases/27/Everything/$(ARCH)/os/Packages/
>>>> - LINUX_VERSION := Fedora 27
>>>> + ARCHIVED := $(shell [ $(BASE_OS_VERSION) -lt $(DEFAULT_OS_VERSION)
>>>> ] && echo true)
>>>> + ifeq ($(ARCHIVED),true)
>>>> + OEL_URL :=
>>>> https://archives.fedoraproject.org/pub/archive/$(FEDORA_TYPE)/releases/$(BASE_OS_VERSION)/Everything/$(ARCH)/os/Packages/
>>>> + else
>>>> + OEL_URL :=
>>>> https://dl.fedoraproject.org/pub/$(FEDORA_TYPE)/releases/$(BASE_OS_VERSION)/Everything/$(ARCH)/os/Packages/
>>>> + endif
>>>> + LINUX_VERSION := Fedora $(BASE_OS_VERSION)
>>>> else
>>>> $(error Unknown base OS $(BASE_OS))
>>>> endif
>>>>
>>>> - Enable the creation of several different devkits at once (e.g. 'make
>>>> cross_compile_target="ppc64-linux-gnu ppc64le-linux-gnu
>>>> s390x-linux-gnu"') or one after another but all into the same
>>>> 'build/devkit/result' directory. The result directory will contain
>>>> $HOST-to-$TARGET sub-directories with the corresponding devkits:
>>>>
>>>> make/devkit/Makefile
>>>>
>>>> -submakevars = HOST=$@ BUILD=$(me) \
>>>> - RESULT=$(RESULT) PREFIX=$(RESULT)/$@ \
>>>> - OUTPUT_ROOT=$(OUTPUT_ROOT)
>>>> +submakevars = HOST=$@ BUILD=$(me) RESULT=$(RESULT) OUTPUT_ROOT=$(OUTPUT_ROOT)
>>>> +
>>>> $(host_platforms) :
>>>> @echo 'Building compilers for $@'
>>>> @echo 'Targets: $(target_platforms)'
>>>> for p in $(filter $@, $(target_platforms)) $(filter-out $@,
>>>> $(target_platforms)); do \
>>>> - $(MAKE) -f Tools.gmk download-rpms $(submakevars) TARGET=$$p && \
>>>> + $(MAKE) -f Tools.gmk download-rpms $(submakevars) \
>>>> + TARGET=$$p PREFIX=$(RESULT)/$@-to-$$p && \
>>>> $(MAKE) -f Tools.gmk all $(submakevars) \
>>>> - TARGET=$$p || exit 1 ; \
>>>> + TARGET=$$p PREFIX=$(RESULT)/$@-to-$$p && \
>>>> + $(MAKE) -f Tools.gmk ccache $(submakevars) \
>>>> + TARGET=$@ PREFIX=$(RESULT)/$@-to-$$p
>>>> BUILDDIR=$(OUTPUT_ROOT)/$@/$$p || exit 1 ; \
>>>> done
>>>> - @echo 'Building ccache program for $@'
>>>> - $(MAKE) -f Tools.gmk ccache $(submakevars) TARGET=$@
>>>> @echo 'All done"'
>>>>
>>>> Notice that we have to build "ccache" for each target because ccache
>>>> will be installed into the directory specified by "--prefix" at
>>>> configure time and this is now different for every target. However
>>>> that's not a big problem, because the time for compiling ccache is
>>>> negligible compared to the download time of the RPMs and the build
>>>> time of GCC.
>>>>
>>>> define Mktar
>>>> - $(1)_tar = $$(RESULT)/sdk-$(1)-$$(today).tar.gz
>>>> - $$($(1)_tar) : PLATFORM = $(1)
>>>> - TARFILES += $$($(1)_tar)
>>>> - $$($(1)_tar) : $(1) $$(shell find $$(RESULT)/$(1))
>>>> + $(1)-to-$(2)_tar = $$(RESULT)/sdk-$(1)-to-$(2)-$$(today).tar.gz
>>>> + $$($(1)-to-$(2)_tar) : PLATFORM = $(1)-to-$(2)
>>>> + TARFILES += $$($(1)-to-$(2)_tar)
>>>> + $$($(1)-to-$(2)_tar) : $$(shell find $$(RESULT)/$(1)-to-$(2) -type f)
>>>> endef
>>>>
>>>> -$(foreach p,$(host_platforms),$(eval $(call Mktar,$(p))))
>>>> +$(foreach p,$(host_platforms),$(foreach t,$(target_platforms),$(eval
>>>> $(call Mktar,$(p),$(t)))))
>>>>
>>>> make/devkit/Tools.gmk
>>>>
>>>> -PATHEXT = $(RESULT)/$(BUILD)/bin:
>>>> +PATHEXT = $(PREFIX)/bin:
>>>>
>>>>
>>>> - Various small cleanups
>>>>
>>>> make/devkit/Tools.gmk
>>>>
>>>> - Don't set "RESULT" and "PREFIX" in Tools.gmk because the values are
>>>> overridden by the settings in the calling Makefile anyway:
>>>>
>>>> # Define directories
>>>> -RESULT := $(OUTPUT_ROOT)/result
>>>> BUILDDIR := $(OUTPUT_ROOT)/$(HOST)/$(TARGET)
>>>> -PREFIX := $(RESULT)/$(HOST)
>>>> TARGETDIR := $(PREFIX)/$(TARGET)
>>>>
>>>> - Cleanup arch selection:
>>>>
>>>> ifeq ($(ARCH),x86_64)
>>>> - RPM_ARCHS := x86_64 noarch
>>>> + RPM_ARCHS := $(ARCH) noarch
>>>> ifeq ($(BUILD),$(HOST))
>>>> ifeq ($(TARGET),$(HOST))
>>>> # When building the native compiler for x86_64, enable mixed mode.
>>>> @@ -199,7 +206,7 @@
>>>> endif
>>>> endif
>>>> else ifeq ($(ARCH),i686)
>>>> - RPM_ARCHS := i386 i686 noarch
>>>> + RPM_ARCHS := $(ARCH) i386 noarch
>>>> else ifeq ($(ARCH), armhfp)
>>>>
>>>> - Don't create 'devkit.info' unconditinally. Only build it as part of
>>>> the "all" target invocation. This prevents the creation of a
>>>> 'devkit.info' with the wrong "BASE_OS_VERSION" if the Makefile is
>>>> invoked several times with different "BASE_OS_VERSION" values:
>>>>
>>>> -$(PREFIX)/devkit.info: FRC
>>>> +$(PREFIX)/devkit.info:
>>>> @echo 'Creating devkit.info in the root of the kit'
>>>> rm -f $@
>>>> touch $@
>>>> @@ -611,7 +623,4 @@
>>>> # this is only built for host. so separate.
>>>> ccache : $(ccache)
>>>>
>>>> -# Force target
>>>> -FRC:
>>>>
>>>> - put the base directory of the devkits into the tar archives. I don't
>>>> like tar archives which don't have a single top-level directory and
>>>> expand right into the current working directory :)
>>>>
>>>> make/devkit/Makefile
>>>>
>>>> %.tar.gz :
>>>> @echo 'Creating compiler package $@'
>>>> - cd $(RESULT)/$(PLATFORM) && tar -czf $@ *
>>>> + cd $(RESULT) && tar -czf $@ $(PLATFORM)/*
>>>> touch $@
More information about the build-dev
mailing list