RFR: JDK-8217739: Cannot reuse java.base UnixConstants.java from target in BuildJDK when cross compiling

[Ao Qi]

On Thu, Jun 6, 2019 at 10:49 AM Ao Qi <aoqi at loongson.cn> wrote:
>
> On Thu, Jun 6, 2019 at 4:46 AM Erik Joelsson <erik.joelsson at oracle.com> wrote:
> >
> > Here is a new webrev: http://cr.openjdk.java.net/~erikj/8217739/webrev.03

Another question: should "[bootcycle] " be displayed during make
bootcycle-images? If yes, should we remove line 164 "LOG_PREFIX :=" of
http://cr.openjdk.java.net/~erikj/8217739/webrev.03/make/common/MakeBase.gmk.html
?

>
> I (as an Author) am ok with this fix. mips64el zero cross build passed
> and javac (to test open a file) worked.
>
> UnixConstants.java of build:
> .../usr/bin/cc -E -x c -I...
>
> UnixConstants.java of target:
> .../usr/bin/mips64el-linux-gnuabi64-gcc -E -x c
> --sysroot=/home/aoqi//chroots/mips64el_stretch/ -I...
>
> Thanks,
> Ao Qi
> >
> > The change is in buildjdk-spec.gmk.in where CPP is now also overridden.
> >
> > /Erik
> >
> > On 2019-06-05 10:33, Erik Joelsson wrote:
> > > Never mind, I found the issue. The reason it worked for me was that we
> > > set SYSROOT_CFLAGS for both the target and the build compiler (from
> > > using devkits for them). The preprocessor for the target would then
> > > act correctly given the --sysroot parameter pointing to a sysroot for
> > > the build system.
> > >
> > > I'm working on a patch where PPC gets overridden with the build
> > > compiler in buildjdk-spec.gmk.
> > >
> > > /Erik
> > >
> > > On 2019-06-05 08:37, Erik Joelsson wrote:
> > >> Hello Ao Qi,
> > >>
> > >> In my testing, I tried building ppc64le (which I first identified as
> > >> generating a UnixConstants.java that differed from an x64 build).
> > >> With this patch, the buildjdk and target version of
> > >> UnixConstants.java differ the same way. How did you configure your
> > >> build? I'm suspecting something in the setup of the buildjdk
> > >> compilers is different from my configuration.
> > >>
> > >> /Erik
> > >>
> > >> On 2019-06-04 21:48, Ao Qi wrote:
> > >>> Hi Erik,
> > >>>
> > >>> Thank you a lot for taking care of this. I mainly like your approach.
> > >>> I tested the patch by cross building mips64-zero, by which we found
> > >>> this problem. However, it seems the problem still exists. The reason I
> > >>> found is that UnixConstants.java of buildjdk is generated by cross
> > >>> compiler, so the macro definitions are defined by the target. Here is
> > >>> a preliminary workaround for your reference only (only worked for
> > >>> gcc):
> > >>>
> > >>> $ hg diff make/gensrc/GensrcMisc.gmk
> > >>> diff -r f4df9d4201cc make/gensrc/GensrcMisc.gmk
> > >>> --- a/make/gensrc/GensrcMisc.gmk Tue Jun 04 16:50:25 2019 -0700
> > >>> +++ b/make/gensrc/GensrcMisc.gmk Wed Jun 05 12:39:43 2019 +0800
> > >>> @@ -61,6 +61,13 @@
> > >>>     CPP_FLAGS += -nologo
> > >>>   endif
> > >>>
> > >>> +ifeq ($(CREATING_BUILDJDK), true)
> > >>> +  # Only worked for gcc
> > >>> +  TEMP_CPP=gcc -E
> > >>> +else
> > >>> +  TEMP_CPP=$(CPP)
> > >>> +endif
> > >>> +
> > >>>   # Generate a java source file from a template through the C
> > >>> preprocessor for the
> > >>>   # target system. First extract the copyright notice at the start
> > >>> of the file.
> > >>>   # Run the preprocessor. Filter out the default compiler stderr
> > >>> output on
> > >>> @@ -71,7 +78,7 @@
> > >>>   define generate-preproc-src
> > >>>    $(call MakeDir, $(@D))
> > >>>    ( $(NAWK) '/@@END_COPYRIGHT@@/{exit}1' $< && \
> > >>> -   $(CPP) $(CPP_FLAGS) $(SYSROOT_CFLAGS) $(CFLAGS_JDKLIB) $< \
> > >>> +   $(TEMP_CPP) $(CPP_FLAGS) $(SYSROOT_CFLAGS) $(CFLAGS_JDKLIB) $< \
> > >>>          2> >($(GREP) -v '^$(<F)$$' >&2) \
> > >>>          | $(NAWK) '/@@START_HERE@@/,0' \
> > >>>          |  $(SED) -e 's/@@START_HERE@@/\/\/ AUTOMATICALLY GENERATED
> > >>> FILE - DO NOT EDIT/' \
> > >>>
> > >>>
> > >>> Cheers,
> > >>> Ao Qi
> > >>>
> > >>> On Wed, Jun 5, 2019 at 12:00 AM Erik Joelsson
> > >>> <erik.joelsson at oracle.com> wrote:
> > >>>> Thanks!
> > >>>>
> > >>>> During testing, I found an intermittent race issue. The command line
> > >>>> that runs the interim langtools version of javac still always
> > >>>> points to
> > >>>> the buildtools/interim-langtools dir of the normal build. This
> > >>>> variable
> > >>>> also needs to be rewritten in buildjdk-spec.gmk.in. New webrev, only
> > >>>> changed in buildjdk-spec.gmk.in:
> > >>>>
> > >>>> http://cr.openjdk.java.net/~erikj/8217739/webrev.02/
> > >>>>
> > >>>> /Erik
> > >>>>
> > >>>> On 2019-06-04 07:53, Tim Bell wrote:
> > >>>>> Erik:
> > >>>>>
> > >>>>> Looks good.
> > >>>>>
> > >>>>> Tim
> > >>>>>
> > >>>>>> This is a fix for a problem brought up on this list [1]. The main
> > >>>>>> issue
> > >>>>>> is that when cross compiling, and creating a buildjdk during the
> > >>>>>> build,
> > >>>>>> the UnixConstants.java cannot necessarily be reused between the
> > >>>>>> target
> > >>>>>> JDK and the buildjdk. While investigating this, I came to the
> > >>>>>> conclusion
> > >>>>>> that we really should stop trying to reuse anything between the
> > >>>>>> buildjdk
> > >>>>>> and target JDK. It's just becoming too complex to maintain (due
> > >>>>>> to the
> > >>>>>> intricate interactions with the interim-image,
> > >>>>>> generate-link-opt-data
> > >>>>>> and exploded-image-optimize targets, causing circular
> > >>>>>> dependencies and
> > >>>>>> endless incremental rebuilds).
> > >>>>>>
> > >>>>>> So this fix simply removes all the reuse of buildtools, gensrc
> > >>>>>> and java
> > >>>>>> compilation when creating a buildjdk. Hopefully this is making
> > >>>>>> Main.gmk
> > >>>>>> a bit less convoluted.
> > >>>>>>
> > >>>>>> While working on this, I came up with the idea of adding a log
> > >>>>>> prefix
> > >>>>>> functionality, to more easily discern which log lines were
> > >>>>>> printed by
> > >>>>>> the buildjdk build and which were part of the normal build. I
> > >>>>>> liked this
> > >>>>>> a lot so including that in the same fix, and applied it to the
> > >>>>>> bootcycle
> > >>>>>> build as well.
> > >>>>>>
> > >>>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8217739
> > >>>>>>
> > >>>>>> Webrev:
> > >>>>>> http://cr.openjdk.java.net/~erikj/8217739/webrev.01/index.html
> > >>>>>>
> > >>>>>> /Erik
> > >>>>>>
> > >>>>>> [1]
> > >>>>>> http://mail.openjdk.java.net/pipermail/build-dev/2019-January/024729.html
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>>