RFR: JDK-8217739: Cannot reuse java.base UnixConstants.java from target in BuildJDK when cross compiling

Thu Jun 6 14:31:33 UTC 2019

Hello,

On 2019-06-05 20:28, Ao Qi wrote:
> On Thu, Jun 6, 2019 at 10:49 AM Ao Qi <aoqi at loongson.cn> wrote:
>> On Thu, Jun 6, 2019 at 4:46 AM Erik Joelsson <erik.joelsson at oracle.com> wrote:
>>> Here is a new webrev: http://cr.openjdk.java.net/~erikj/8217739/webrev.03
> Another question: should "[bootcycle] " be displayed during make
> bootcycle-images? If yes, should we remove line 164 "LOG_PREFIX :=" of
> http://cr.openjdk.java.net/~erikj/8217739/webrev.03/make/common/MakeBase.gmk.html
> ?

It seems you are right. I assumed that by setting LOG_PREFIX on the make 
command line, it would override. I think it's cleaner to initiate the 
variable to empty so that any accidental setting in the environment 
doesn't interfere. This works for the buildjdk call, which calls 
Main.gmk, but doesn't seem to work for bootcycle, which calls Init.gmk.

For now I will just refrain from initializing it in MakeBase.gmk.

Webrev: http://cr.openjdk.java.net/~erikj/8217739/webrev.04

>> I (as an Author) am ok with this fix. mips64el zero cross build passed
>> and javac (to test open a file) worked.

Thanks!

/Erik

>>
>> UnixConstants.java of build:
>> .../usr/bin/cc -E -x c -I...
>>
>> UnixConstants.java of target:
>> .../usr/bin/mips64el-linux-gnuabi64-gcc -E -x c
>> --sysroot=/home/aoqi//chroots/mips64el_stretch/ -I...
>>
>> Thanks,
>> Ao Qi
>>> The change is in buildjdk-spec.gmk.in where CPP is now also overridden.
>>>
>>> /Erik
>>>
>>> On 2019-06-05 10:33, Erik Joelsson wrote:
>>>> Never mind, I found the issue. The reason it worked for me was that we
>>>> set SYSROOT_CFLAGS for both the target and the build compiler (from
>>>> using devkits for them). The preprocessor for the target would then
>>>> act correctly given the --sysroot parameter pointing to a sysroot for
>>>> the build system.
>>>>
>>>> I'm working on a patch where PPC gets overridden with the build
>>>> compiler in buildjdk-spec.gmk.
>>>>
>>>> /Erik
>>>>
>>>> On 2019-06-05 08:37, Erik Joelsson wrote:
>>>>> Hello Ao Qi,
>>>>>
>>>>> In my testing, I tried building ppc64le (which I first identified as
>>>>> generating a UnixConstants.java that differed from an x64 build).
>>>>> With this patch, the buildjdk and target version of
>>>>> UnixConstants.java differ the same way. How did you configure your
>>>>> build? I'm suspecting something in the setup of the buildjdk
>>>>> compilers is different from my configuration.
>>>>>
>>>>> /Erik
>>>>>
>>>>> On 2019-06-04 21:48, Ao Qi wrote:
>>>>>> Hi Erik,
>>>>>>
>>>>>> Thank you a lot for taking care of this. I mainly like your approach.
>>>>>> I tested the patch by cross building mips64-zero, by which we found
>>>>>> this problem. However, it seems the problem still exists. The reason I
>>>>>> found is that UnixConstants.java of buildjdk is generated by cross
>>>>>> compiler, so the macro definitions are defined by the target. Here is
>>>>>> a preliminary workaround for your reference only (only worked for
>>>>>> gcc):
>>>>>>
>>>>>> $ hg diff make/gensrc/GensrcMisc.gmk
>>>>>> diff -r f4df9d4201cc make/gensrc/GensrcMisc.gmk
>>>>>> --- a/make/gensrc/GensrcMisc.gmk Tue Jun 04 16:50:25 2019 -0700
>>>>>> +++ b/make/gensrc/GensrcMisc.gmk Wed Jun 05 12:39:43 2019 +0800
>>>>>> @@ -61,6 +61,13 @@
>>>>>>      CPP_FLAGS += -nologo
>>>>>>    endif
>>>>>>
>>>>>> +ifeq ($(CREATING_BUILDJDK), true)
>>>>>> +  # Only worked for gcc
>>>>>> +  TEMP_CPP=gcc -E
>>>>>> +else
>>>>>> +  TEMP_CPP=$(CPP)
>>>>>> +endif
>>>>>> +
>>>>>>    # Generate a java source file from a template through the C
>>>>>> preprocessor for the
>>>>>>    # target system. First extract the copyright notice at the start
>>>>>> of the file.
>>>>>>    # Run the preprocessor. Filter out the default compiler stderr
>>>>>> output on
>>>>>> @@ -71,7 +78,7 @@
>>>>>>    define generate-preproc-src
>>>>>>     $(call MakeDir, $(@D))
>>>>>>     ( $(NAWK) '/@@END_COPYRIGHT@@/{exit}1' $< && \
>>>>>> -   $(CPP) $(CPP_FLAGS) $(SYSROOT_CFLAGS) $(CFLAGS_JDKLIB) $< \
>>>>>> +   $(TEMP_CPP) $(CPP_FLAGS) $(SYSROOT_CFLAGS) $(CFLAGS_JDKLIB) $< \
>>>>>>           2> >($(GREP) -v '^$(<F)$$' >&2) \
>>>>>>           | $(NAWK) '/@@START_HERE@@/,0' \
>>>>>>           |  $(SED) -e 's/@@START_HERE@@/\/\/ AUTOMATICALLY GENERATED
>>>>>> FILE - DO NOT EDIT/' \
>>>>>>
>>>>>>
>>>>>> Cheers,
>>>>>> Ao Qi
>>>>>>
>>>>>> On Wed, Jun 5, 2019 at 12:00 AM Erik Joelsson
>>>>>> <erik.joelsson at oracle.com> wrote:
>>>>>>> Thanks!
>>>>>>>
>>>>>>> During testing, I found an intermittent race issue. The command line
>>>>>>> that runs the interim langtools version of javac still always
>>>>>>> points to
>>>>>>> the buildtools/interim-langtools dir of the normal build. This
>>>>>>> variable
>>>>>>> also needs to be rewritten in buildjdk-spec.gmk.in. New webrev, only
>>>>>>> changed in buildjdk-spec.gmk.in:
>>>>>>>
>>>>>>> http://cr.openjdk.java.net/~erikj/8217739/webrev.02/
>>>>>>>
>>>>>>> /Erik
>>>>>>>
>>>>>>> On 2019-06-04 07:53, Tim Bell wrote:
>>>>>>>> Erik:
>>>>>>>>
>>>>>>>> Looks good.
>>>>>>>>
>>>>>>>> Tim
>>>>>>>>
>>>>>>>>> This is a fix for a problem brought up on this list [1]. The main
>>>>>>>>> issue
>>>>>>>>> is that when cross compiling, and creating a buildjdk during the
>>>>>>>>> build,
>>>>>>>>> the UnixConstants.java cannot necessarily be reused between the
>>>>>>>>> target
>>>>>>>>> JDK and the buildjdk. While investigating this, I came to the
>>>>>>>>> conclusion
>>>>>>>>> that we really should stop trying to reuse anything between the
>>>>>>>>> buildjdk
>>>>>>>>> and target JDK. It's just becoming too complex to maintain (due
>>>>>>>>> to the
>>>>>>>>> intricate interactions with the interim-image,
>>>>>>>>> generate-link-opt-data
>>>>>>>>> and exploded-image-optimize targets, causing circular
>>>>>>>>> dependencies and
>>>>>>>>> endless incremental rebuilds).
>>>>>>>>>
>>>>>>>>> So this fix simply removes all the reuse of buildtools, gensrc
>>>>>>>>> and java
>>>>>>>>> compilation when creating a buildjdk. Hopefully this is making
>>>>>>>>> Main.gmk
>>>>>>>>> a bit less convoluted.
>>>>>>>>>
>>>>>>>>> While working on this, I came up with the idea of adding a log
>>>>>>>>> prefix
>>>>>>>>> functionality, to more easily discern which log lines were
>>>>>>>>> printed by
>>>>>>>>> the buildjdk build and which were part of the normal build. I
>>>>>>>>> liked this
>>>>>>>>> a lot so including that in the same fix, and applied it to the
>>>>>>>>> bootcycle
>>>>>>>>> build as well.
>>>>>>>>>
>>>>>>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8217739
>>>>>>>>>
>>>>>>>>> Webrev:
>>>>>>>>> http://cr.openjdk.java.net/~erikj/8217739/webrev.01/index.html
>>>>>>>>>
>>>>>>>>> /Erik
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> http://mail.openjdk.java.net/pipermail/build-dev/2019-January/024729.html
>>>>>>>>>
>>>>>>>>>
>>>>>>>>