Building OpenJDK9 on MSYS2
Magnus Ihse Bursie
magnus.ihse.bursie at oracle.com
Thu Oct 5 10:13:05 UTC 2017
On 2017-10-05 11:59, Peter Budai wrote:
>
> Hi Magnus and Erik,
>
> I really appreciate your quick feedback. I assumed that it won’t be
> easy, but I just don’t feel I should give up now - maybe later when I
> see the real scale of work. So bear with me for a time being.
>
> Attached is a patch which already includes Magnus’ changes, plus a few
> which I have added:
>
> * basically enabling gcc for windows,
> * and modifying a logic for compiling fixpath (before that it was
> using hard-coded MS VSC compile flags)
>
Actually, you must make sure fixpath is *not* used for the toolchain,
since gcc uses unix style paths.
(However, other tools such as java will still need it.)
> So here is what I have as the result of configure:
>
> ====================================================
>
> The existing configuration has been successfully updated in
>
> /C/msys64/home/peterbud/jdk9/build/windows-x86_64-normal-server-release
>
> using configure arguments '--disable-freetype-bundling
> --disable-javac-server'.
>
> Configuration summary:
>
> * Debug level: release
>
> * HS debug level: product
>
> * JDK variant: normal
>
> * JVM variants: server
>
> * OpenJDK target: OS: windows, CPU architecture: x86, address length: 64
>
> * Version string: 9-internal+0-adhoc.peterbud.jdk9 (9-internal)
>
> Tools summary:
>
> * Environment: msys version 2.9.0(0.318/5/3) (root at /C/msys64)
>
> * Boot JDK: java version "1.8.0_144" Java(TM) SE Runtime
> Environment (build 1.8.0_144-b01) Java HotSpot(TM) 64-Bit Server VM
> (build 25.144-b01, mixed mode) (at /c/progra~1/java/jdk18~1.0_1)
>
> * Toolchain: gcc (GNU Compiler Collection)
>
> * C Compiler: Version 7.2.0 (at /C/msys64/mingw64/bin/gcc)
>
> * C++ Compiler: Version 7.2.0 (at /c/msys64/mingw64/bin/g++)
>
> Build performance summary:
>
> * Cores to use: 4
>
> * Memory limit: 16216 MB
>
> Its clear says that the toolchain is gcc 7.2 (BTW there is no Visual
> Studio on this machine)
>
> Now for the details of the config log, you can see here:
> https://pastebin.com/MN2ZYcHH <https://pastebin.com/MN2ZYcHH>
>
> And about the build process and the error I get:
>
> $ make JOBS=1
>
> Building target 'default (exploded-image)' in configuration
> 'windows-x86_64-normal-server-release'
>
> Compiling 8 files for BUILD_TOOLS_LANGTOOLS
>
> Compiling 17 properties into resource bundles for jdk.compiler
>
> Parsing 1 properties into enum-like class for jdk.compiler
>
> Compiling 19 properties into resource bundles for jdk.javadoc
>
> Compiling 12 properties into resource bundles for jdk.jdeps
>
> Compiling 7 properties into resource bundles for jdk.jshell
>
> Compiling 117 files for BUILD_INTERIM_java.compiler
>
> Compiling 396 files for BUILD_INTERIM_jdk.compiler
>
> Compiling 61 files for BUILD_INTERIM_jdk.jdeps
>
> Compiling 457 files for BUILD_INTERIM_jdk.javadoc
>
> Note: Some input files use or override a deprecated API.
>
> Note: Recompile with -Xlint:deprecation for details.
>
> Compiling 159 files for BUILD_TOOLS_JDK
>
> Note: Some input files use unchecked or unsafe operations.
>
> Note: Recompile with -Xlint:unchecked for details.
>
> make[3]: *** [GensrcMisc.gmk:78:
> /C/msys64/home/peterbud/jdk9/build/windows-x86_64-normal-server-release/support/gensrc/java.base/sun/nio/ch/SocketOptionRegistry.java]
> Error 1
>
> make[3]: *** Deleting file
> '/C/msys64/home/peterbud/jdk9/build/windows-x86_64-normal-server-release/support/gensrc/java.base/sun/nio/ch/SocketOptionRegistry.java'
>
> make[2]: *** [make/Main.gmk:115: java.base-gensrc-jdk] Error 2
>
> ERROR: Build failed for target 'default (exploded-image)' in
> configuration 'windows-x86_64-normal-server-release' (exit code 2)
>
> No indication of failed target found.
>
> Hint: Try searching the build log for '] Error'.
>
> Hint: See common/doc/building.html#troubleshooting for assistance.
>
> make[1]: *** [/home/peterbud/jdk9/make/Init.gmk:296: main] Error 2
>
> make: *** [/home/peterbud/jdk9/make/Init.gmk:185: default] Error 2
>
> If I run here
>
> make JOBS=1 LOG=debug
>
> The failing line seems to be this:
>
> ( /usr/bin/gawk '/@@END_COPYRIGHT@@/{exit}1'
> /C/msys64/home/peterbud/jdk9/jdk/src/java.base/share/classes/sun/nio/ch/SocketOptionRegistry.java.template
> &&
> /C/msys64/home/peterbud/jdk9/build/windows-x86_64-normal-server-release/configure-support/bin/fixpath.exe
> -m/C/msys64/@/c/msys64/@/c/progra~ /C/msys64/mingw64/bin/gcc -E -x c
> /C/msys64/home/peterbud/jdk9/jdk/src/java.base/share/classes/sun/nio/ch/SocketOptionRegistry.java.template
> 2> >(/usr/bin/grep -v '^SocketOptionRegistry.java.template$' >&2) |
> /usr/bin/gawk '/@@START_HERE@@/,0' | /usr/bin/sed -e
> 's/@@START_HERE@@/\/\/ AUTOMATICALLY GENERATED FILE - DO NOT EDIT/' -e
> 's/PREFIX_//' -e 's/^#.*//' ) >
> /C/msys64/home/peterbud/jdk9/build/windows-x86_64-normal-server-release/support/gensrc/java.base/sun/nio/ch/SocketOptionRegistry.java
>
> make[3]: *** [GensrcMisc.gmk:78:
> /C/msys64/home/peterbud/jdk9/build/windows-x86_64-normal-server-release/support/gensrc/java.base/sun/nio/ch/SocketOptionRegistry.java]
> Error 1
>
> Now the interesting is: if I copy this line above to the bash prompt,
> it runs without problem, and the file
> support/gensrc/java.base/sun/nio/ch/SocketOptionRegistry.java
>
That's surprising, since gcc is prefixed with fixpath, which it should not.
I have a hard time believing this is a race condition. On the other
hand, this stuff is weird, we're misusing the C preprocessor to process
defines in java code, so I'm not suprised it breaks down. I don't know
why it succeeded when run on the command line, though. My suggestion is
to just do some quick and dirty hack around this: take the file you
manage to generate and just copy it in during the build instead. If you
can get round this, you might start seeing some *real* problems. :-)
Also, my suggestion is that you try running "make hotspot" to cut to the
chase. Compiling hotspot will likely be the hardest thing. Or even "make
-k hotspot" to get an assessment of the amount of work ahead of you.
/Magnus
> Is produced.
>
> Then I can again issue
>
> make JOBS=1 LOG=debug
>
> And the compile process is being continued, until a similar error pops
> up with a different generated file. I have an assumption that this
> happens because make is still running parallel jobs, despite JOBS=1
> but I’m not sure.
>
> How could I best tackle this?
>
> Thank you and best regards,
>
> Peter
>
> Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for
> Windows 10
>
> *From: *Magnus Ihse Bursie <mailto:magnus.ihse.bursie at oracle.com>
> *Sent: *Thursday, October 5, 2017 11:33 AM
> *To: *Erik Joelsson <mailto:erik.joelsson at oracle.com>; Peter Budai
> <mailto:peterbudai at hotmail.com>; build-dev at openjdk.java.net
> <mailto:build-dev at openjdk.java.net>
> *Subject: *Re: Building OpenJDK9 on MSYS2
>
> On 2017-10-05 10:10, Erik Joelsson wrote:
> > Hello Peter,
> >
> >
> > On 2017-10-04 21:15, Peter Budai wrote:
> >> Hi Magnus,
> >>
> >> Thanks for the quick reply I’ll check these patches with msys2.
> >>
> >> Let me specify with more details what I’d like to achieve: I’d like
> >> to build OpenJDK9 with MSYS2 MINGW64 environment using gcc toolchain.
> >> (I’m not sure how familiar are you with MSYS2, but there are 3
> >> different environments: MSYS2, MINGW32 and MINGW64). In theory
> >> MINGW64 with gcc is the closes you can get on Windows platform as a
> >> gcc unix like build environment, which produces still a native 64-bit
> >> executable on Windows.
> >>
> >> I’m not very familiar with OpenJDK yet, so therefore I’d like to hear
> >> your opinion: how realistic is that?
> > Sorry to disappoint, but I would say that requires major work. There
> > is a strong historic assumption that windows builds are done using
> > Visual Studio. We have abstracted away some of it in configure (see
> > TOOLCHAIN_TYPE), but it's very far from enough to change compiler
> > environment for a Windows build. The native sources are also bound to
> > make a lot of such assumptions. I would expect the changes needed to
> > be in the thousands of lines of code.
>
> I agree that it requires hard work (even if "thousands" might be an
> overestimation I think, but "hundreds" is not enough, so it's the right
> magnitude). On the other hand, it would be really good if we did sort
> things out, so that we had proper conditions based on OS vs
> compiler/toolchain.
>
> If you really want to start, the first thing is to patch toolchain.m4 to
> VALID_TOOLCHAINS_windows="microsoft gcc"
> and then call configure using "bash configure --with-toolchain-type=gcc".
>
> As Erik, I doubt you will come very far before things starts tumbling
> down.
>
> >
> > When we say supporting the build in msys2 instead of cygwin, we just
> > mean using msys2 as the unix emulating layer for our tools like
> > make/bash/grep/sed etc.
> >
> > One think I have done successfully is running the build in WSL
> > (Windows Subsystem for Linux), but that isn't all that helpful as WSL
> > for practical purposes is more or less like running Linux in a VM, so
> > the build sees a Linux system and builds a Linux binary.
> >> As a side note: with MINGW64 I have managed to run configure phase
> >> successfully for OpenJDK. The compile process has also started and
> >> went for a while, but interestingly I run into some kind of race
> >> conditions as make stopped with an error. Using LOG=debug I have fond
> >> the failing line and then copying the failed command and pasting it
> >> to the bash prompt it successfully generated the output target, and
> >> then the build process run further when a similar situation happened.
> >> Also pasting the failed command run in the bash without any problem,
> >> and build continued… until the next.
> > Without seeing the errors I can't say much. I very much doubt that you
> > are running with gcc as the compiler though. Configure isn't easily
> > fooled into using a different compiler to what it prefers, and I would
> > expect things to crash and burn pretty early if you actually did.
> >
> > /Erik
> >>
> >> I have tried to run make JOBS=1, but did not help, strangely I have
> >> still seen in the log make[3] and make[4] logs which suggested that
> >> there are more than one make jobs were running. Also tried .configure
> >> --with-output-sync=recurse without success (same symptoms)
> >>
> >> Let me know your thoughts.
> >>
> >> Best regards,
> >>
> >> Peter
> >>
> >> Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for
> >> Windows 10
> >>
> >> From: Magnus Ihse Bursie<mailto:magnus.ihse.bursie at oracle.com>
> >> Sent: Wednesday, October 4, 2017 1:04 AM
> >> To: Peter Budai<mailto:peterbudai at hotmail.com>;
> >> build-dev at openjdk.java.net<mailto:build-dev at openjdk.java.net>
> >> Subject: Re: Building OpenJDK9 on MSYS2
> >>
> >> Actually, it wasn't so much remaining trouble. :-) I fired up msys2 and
> >> checked out where I left off. It turned out that the remaining snag was
> >> that msys2 tries to convert command lines automatically, from "unix"
> >> style paths to "windows" style paths. Unfortunately, it does not do
> this
> >> very well and it breaks all sorts of things. We already have a FIXPATH
> >> solution in place which deals with this, so basically all I had to do
> >> was disable this (by setting MSYS2_ARG_CONV_EXCL to "*"). However, this
> >> broke our cygpath replacement hack (!) so I had to disable it there.
> >> Sigh. Anyway, with those fixes it ran and worked well. (I also
> >> discovered and fixed a bug related to how we set up the FIXPATH
> variable
> >> on msys, but it only triggers in certain circumstances).
> >>
> >> With this patch I now jdk9 seems to build fine on msys2. It should
> apply
> >> cleanly on jdk9/jdk9. Since it turned out to be so trivial, I'll try to
> >> get it in in jdk10.
> >>
> >> Here's the patch if you want to apply it yourself:
> >>
> >> diff -r a08cbfc0e4ec common/autoconf/basics_windows.m4
> >> --- a/common/autoconf/basics_windows.m4 Thu Aug 03 18:56:56 2017
> >> +0000
> >> +++ b/common/autoconf/basics_windows.m4 Wed Oct 04 00:53:58 2017
> >> +0200
> >> @@ -42,7 +42,7 @@
> >> windows_path=`$CYGPATH -m "$unix_path"`
> >> $1="$windows_path"
> >> elif test "x$OPENJDK_BUILD_OS_ENV" = "xwindows.msys"; then
> >> - windows_path=`cmd //c echo $unix_path`
> >> + windows_path=`MSYS2_ARG_CONV_EXCL= cmd //c echo $unix_path`
> >> $1="$windows_path"
> >> fi
> >> ])
> >> @@ -136,6 +136,16 @@
> >> fi
> >> ])
> >>
> >> +AC_DEFUN([BASIC_MSYS_UPDATE_FIXPATH],
> >> +[
> >> + # Take all collected prefixes and turn them into a
> >> -m/c/foo@/c/bar at ... command line
> >> + # @ was chosen as separator to minimize risk of other tools messing
> >> around with it
> >> + all_unique_prefixes=`echo "${all_fixpath_prefixes@<:@@@:>@}" \
> >> + | tr ' ' '\n' | $GREP '^/./' | $SORT | $UNIQ`
> >> + fixpath_argument_list=`echo $all_unique_prefixes | tr ' ' '@'`
> >> + FIXPATH="$FIXPATH_BIN -m$fixpath_argument_list"
> >> +])
> >> +
> >> AC_DEFUN([BASIC_FIXUP_PATH_MSYS],
> >> [
> >> path="[$]$1"
> >> @@ -143,7 +153,7 @@
> >> new_path="$path"
> >> if test "x$has_colon" = x; then
> >> # Not in mixed or Windows style, start by that.
> >> - new_path=`cmd //c echo $path`
> >> + new_path=`MSYS2_ARG_CONV_EXCL= cmd //c echo $path`
> >> fi
> >>
> >> BASIC_MAKE_WINDOWS_SPACE_SAFE_MSYS([$new_path])
> >> @@ -155,6 +165,8 @@
> >>
> >> # Save the first 10 bytes of this path to the storage, so fixpath
> >> can work.
> >> all_fixpath_prefixes=("${all_fixpath_prefixes@<:@@@:>@}"
> >> "${new_path:0:10}")
> >> + # We might need to re-evaluate FIXPATH.
> >> + BASIC_MSYS_UPDATE_FIXPATH
> >> ])
> >>
> >> AC_DEFUN([BASIC_FIXUP_EXECUTABLE_CYGWIN],
> >> @@ -293,7 +305,7 @@
> >> # Do not save /bin paths to all_fixpath_prefixes!
> >> else
> >> # Not in mixed or Windows style, start by that.
> >> - new_path=`cmd //c echo $new_path`
> >> + new_path=`MSYS2_ARG_CONV_EXCL= cmd //c echo $new_path`
> >> BASIC_MAKE_WINDOWS_SPACE_SAFE_MSYS([$new_path])
> >> # Output is in $new_path
> >> BASIC_WINDOWS_REWRITE_AS_UNIX_PATH(new_path)
> >> @@ -302,6 +314,8 @@
> >>
> >> # Save the first 10 bytes of this path to the storage, so
> fixpath
> >> can work.
> >> all_fixpath_prefixes=("${all_fixpath_prefixes@<:@@@:>@}"
> >> "${new_path:0:10}")
> >> + # We might need to re-evaluate FIXPATH.
> >> + BASIC_MSYS_UPDATE_FIXPATH
> >> fi
> >> ])
> >>
> >> @@ -347,6 +361,10 @@
> >> WINDOWS_ENV_VENDOR='msys'
> >> WINDOWS_ENV_VERSION="$MSYS_VERSION"
> >>
> >> + # Prohibit msys2 path conversion from trying to be "intelligent",
> >> and rely
> >> + # on fixpath instead.
> >> + export MSYS2_ARG_CONV_EXCL="*"
> >> +
> >> AC_MSG_CHECKING([msys root directory as unix-style path])
> >> # The cmd output ends with Windows line endings (CR/LF), the
> grep
> >> command will strip that away
> >> MSYS_ROOT_PATH=`cd / ; cmd /c cd | $GREP ".*"`
> >> @@ -391,10 +409,7 @@
> >> elif test "x$OPENJDK_BUILD_OS_ENV" = xwindows.msys; then
> >> # Take all collected prefixes and turn them into a
> >> -m/c/foo@/c/bar at ... command line
> >> # @ was chosen as separator to minimize risk of other tools
> >> messing around with it
> >> - all_unique_prefixes=`echo "${all_fixpath_prefixes@<:@@@:>@}" \
> >> - | tr ' ' '\n' | $GREP '^/./' | $SORT | $UNIQ`
> >> - fixpath_argument_list=`echo $all_unique_prefixes | tr ' ' '@'`
> >> - FIXPATH="$FIXPATH_BIN -m$fixpath_argument_list"
> >> + BASIC_MSYS_UPDATE_FIXPATH
> >> fi
> >> FIXPATH_SRC_W="$FIXPATH_SRC"
> >> FIXPATH_BIN_W="$FIXPATH_BIN"
> >> diff -r a08cbfc0e4ec common/autoconf/build-aux/config.sub
> >> --- a/common/autoconf/build-aux/config.sub Thu Aug 03 18:56:56
> >> 2017 +0000
> >> +++ b/common/autoconf/build-aux/config.sub Wed Oct 04 00:53:58
> >> 2017 +0200
> >> @@ -30,7 +30,7 @@
> >> DIR=`dirname $0`
> >>
> >> # First, filter out everything that doesn't begin with "aarch64-"
> >> -if ! echo $* | grep '^aarch64-' >/dev/null ; then
> >> +if ! echo $* | grep -e '^aarch64-' -e 'msys' >/dev/null ; then
> >> . $DIR/autoconf-config.sub "$@"
> >> # autoconf-config.sub exits, so we never reach here, but just in
> >> # case we do:
> >> @@ -38,13 +38,17 @@
> >> fi
> >>
> >> while test $# -gt 0 ; do
> >> - case $1 in
> >> + case $1 in
> >> -- ) # Stop option processing
> >> shift; break ;;
> >> aarch64-* )
> >> config=`echo $1 | sed 's/^aarch64-/arm-/'`
> >> sub_args="$sub_args $config"
> >> shift; ;;
> >> + *-msys )
> >> + config=`echo $1 | sed 's/msys/mingw32/'`
> >> + sub_args="$sub_args $config"
> >> + shift; ;;
> >> - ) # Use stdin as input.
> >> sub_args="$sub_args $1"
> >> shift; break ;;
> >> diff -r a08cbfc0e4ec common/autoconf/spec.gmk.in
> >> --- a/common/autoconf/spec.gmk.in Thu Aug 03 18:56:56 2017 +0000
> >> +++ b/common/autoconf/spec.gmk.in Wed Oct 04 00:53:58 2017 +0200
> >> @@ -120,6 +120,13 @@
> >> # On Windows, the Visual Studio toolchain needs the PATH to be
> >> adjusted
> >> # to include Visual Studio tools (this needs to be in cygwin/msys
> >> style).
> >> export PATH:=@VS_PATH@
> >> +
> >> +endif
> >> +
> >> +ifeq ($(OPENJDK_TARGET_OS_ENV), windows.msys)
> >> + # On msys2, prohibit msys path conversion from trying to be
> >> + # "intelligent", and rely on fixpath instead.
> >> + export MSYS2_ARG_CONV_EXCL:=*
> >> endif
> >>
> >> SYSROOT_CFLAGS := @SYSROOT_CFLAGS@
> >>
> >> /Magnus
> >>
> >> On 2017-10-03 22:34, Magnus Ihse Bursie wrote:
> >>> I gave msys2 a shot some time ago, but it ended up too much trouble.
> >>> I'll share some of my notes from that attempt, for what it's worth.
> >>>
> >>> To install package X/Y, run "pacman -S X/Y". Missing tools and
> >>> packages where to find them:
> >>> cmp: msys/diffutils
> >>> tar: msys/tar
> >>> make: msys/make
> >>> unzip: msys/unzip
> >>> zip: msys/zip
> >>>
> >>> config.sub reports msys as "x86_64-pc-mingw32" but msys2 as
> >>> "x86_64-pc-msys". This patch adds postprocessing in "our" config.sub
> >>> to report msys2 similar to msys. (Opinions, including my own :-) may
> >>> vary if this really is the best way..)
> >>>
> >>> diff -r b88023f46daa common/autoconf/build-aux/config.sub
> >>> --- a/common/autoconf/build-aux/config.sub Fri Jan 27 10:15:41
> >>> 2017 +0100
> >>> +++ b/common/autoconf/build-aux/config.sub Fri Feb 03 05:00:25
> >>> 2017 -0700
> >>> @@ -30,7 +30,7 @@
> >>> DIR=`dirname $0`
> >>>
> >>> # First, filter out everything that doesn't begin with "aarch64-"
> >>> -if ! echo $* | grep '^aarch64-' >/dev/null ; then
> >>> +if ! echo $* | grep -e '^aarch64-' -e 'msys' >/dev/null ; then
> >>> . $DIR/autoconf-config.sub "$@"
> >>> # autoconf-config.sub exits, so we never reach here, but just in
> >>> # case we do:
> >>> @@ -45,6 +45,10 @@
> >>> config=`echo $1 | sed 's/^aarch64-/arm-/'`
> >>> sub_args="$sub_args $config"
> >>> shift; ;;
> >>> + *-msys )
> >>> + config=`echo $1 | sed 's/msys/mingw32/'`
> >>> + sub_args="$sub_args $config"
> >>> + shift; ;;
> >>> - ) # Use stdin as input.
> >>> sub_args="$sub_args $1"
> >>> shift; break ;;
> >>>
> >>> If I remember correctly, this got me past the configure stage at the
> >>> time.
> >>>
> >>> I don't think it's very hard to get it to work on msys2, I just ran
> >>> into one snag too many and didn't think msys2 would be used by anyone.
> >>>
> >>> /Magnus
> >>>
> >>> On 2017-10-03 17:20, Peter Budai wrote:
> >>>> Hello,
> >>>>
> >>>> According to
> >>>>
> http://hg.openjdk.java.net/jdk9/jdk9/file/a08cbfc0e4ec/common/doc/building.html
>
> >>>>
> >>>> “msys2 and the new Windows Subsystem for Linux (WSL) would likely be
> >>>> possible to support in a future version but that would require a
> >>>> community effort to implement”
> >>>>
> >>>> I’d like to help making the OpenJDK 9 build working on msys2. What is
> >>>> the best way to move forward? Is there a similar effort in progress?
> >>>>
> >>>> Thank you and best regards,
> >>>>
> >>>> Peter
> >>>>
> >>>>
> >
>
More information about the build-dev
mailing list