Building OpenJDK9 on MSYS2
David Holmes
david.holmes at oracle.com
Thu Oct 5 10:12:36 UTC 2017
On 5/10/2017 8:00 PM, Peter Budai wrote:
> Now the attachment as well…
Attachments get stripped by the mail list servers.
David
> Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10
>
> From: Peter Budai<mailto:peterbudai at hotmail.com>
> Sent: Thursday, October 5, 2017 11:59 AM
> To: Magnus Ihse Bursie<mailto:magnus.ihse.bursie at oracle.com>; Erik Joelsson<mailto:erik.joelsson at oracle.com>; build-dev at openjdk.java.net<mailto:build-dev at openjdk.java.net>
> Subject: RE: Building OpenJDK9 on MSYS2
>
> Hi Magnus and Erik,
>
> I really appreciate your quick feedback. I assumed that it won’t be easy, but I just don’t feel I should give up now - maybe later when I see the real scale of work. So bear with me for a time being.
>
> Attached is a patch which already includes Magnus’ changes, plus a few which I have added:
>
> * basically enabling gcc for windows,
> * and modifying a logic for compiling fixpath (before that it was using hard-coded MS VSC compile flags)
>
>
> So here is what I have as the result of configure:
> ====================================================
> The existing configuration has been successfully updated in
> /C/msys64/home/peterbud/jdk9/build/windows-x86_64-normal-server-release
> using configure arguments '--disable-freetype-bundling --disable-javac-server'.
>
> Configuration summary:
> * Debug level: release
> * HS debug level: product
> * JDK variant: normal
> * JVM variants: server
> * OpenJDK target: OS: windows, CPU architecture: x86, address length: 64
> * Version string: 9-internal+0-adhoc.peterbud.jdk9 (9-internal)
>
> Tools summary:
> * Environment: msys version 2.9.0(0.318/5/3) (root at /C/msys64)
> * Boot JDK: java version "1.8.0_144" Java(TM) SE Runtime Environment (build 1.8.0_144-b01) Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode) (at /c/progra~1/java/jdk18~1.0_1)
> * Toolchain: gcc (GNU Compiler Collection)
> * C Compiler: Version 7.2.0 (at /C/msys64/mingw64/bin/gcc)
> * C++ Compiler: Version 7.2.0 (at /c/msys64/mingw64/bin/g++)
>
> Build performance summary:
> * Cores to use: 4
> * Memory limit: 16216 MB
>
> Its clear says that the toolchain is gcc 7.2 (BTW there is no Visual Studio on this machine)
>
> Now for the details of the config log, you can see here: https://pastebin.com/MN2ZYcHH
>
> And about the build process and the error I get:
>
> $ make JOBS=1
> Building target 'default (exploded-image)' in configuration 'windows-x86_64-normal-server-release'
> Compiling 8 files for BUILD_TOOLS_LANGTOOLS
> Compiling 17 properties into resource bundles for jdk.compiler
> Parsing 1 properties into enum-like class for jdk.compiler
> Compiling 19 properties into resource bundles for jdk.javadoc
> Compiling 12 properties into resource bundles for jdk.jdeps
> Compiling 7 properties into resource bundles for jdk.jshell
> Compiling 117 files for BUILD_INTERIM_java.compiler
> Compiling 396 files for BUILD_INTERIM_jdk.compiler
> Compiling 61 files for BUILD_INTERIM_jdk.jdeps
> Compiling 457 files for BUILD_INTERIM_jdk.javadoc
> Note: Some input files use or override a deprecated API.
> Note: Recompile with -Xlint:deprecation for details.
> Compiling 159 files for BUILD_TOOLS_JDK
> Note: Some input files use unchecked or unsafe operations.
> Note: Recompile with -Xlint:unchecked for details.
> make[3]: *** [GensrcMisc.gmk:78: /C/msys64/home/peterbud/jdk9/build/windows-x86_64-normal-server-release/support/gensrc/java.base/sun/nio/ch/SocketOptionRegistry.java] Error 1
> make[3]: *** Deleting file '/C/msys64/home/peterbud/jdk9/build/windows-x86_64-normal-server-release/support/gensrc/java.base/sun/nio/ch/SocketOptionRegistry.java'
> make[2]: *** [make/Main.gmk:115: java.base-gensrc-jdk] Error 2
>
> ERROR: Build failed for target 'default (exploded-image)' in configuration 'windows-x86_64-normal-server-release' (exit code 2)
>
> No indication of failed target found.
> Hint: Try searching the build log for '] Error'.
> Hint: See common/doc/building.html#troubleshooting for assistance.
>
> make[1]: *** [/home/peterbud/jdk9/make/Init.gmk:296: main] Error 2
> make: *** [/home/peterbud/jdk9/make/Init.gmk:185: default] Error 2
>
> If I run here
> make JOBS=1 LOG=debug
> The failing line seems to be this:
>
> ( /usr/bin/gawk '/@@END_COPYRIGHT@@/{exit}1' /C/msys64/home/peterbud/jdk9/jdk/src/java.base/share/classes/sun/nio/ch/SocketOptionRegistry.java.template && /C/msys64/home/peterbud/jdk9/build/windows-x86_64-normal-server-release/configure-support/bin/fixpath.exe -m/C/msys64/@/c/msys64/@/c/progra~ /C/msys64/mingw64/bin/gcc -E -x c /C/msys64/home/peterbud/jdk9/jdk/src/java.base/share/classes/sun/nio/ch/SocketOptionRegistry.java.template 2> >(/usr/bin/grep -v '^SocketOptionRegistry.java.template$' >&2) | /usr/bin/gawk '/@@START_HERE@@/,0' | /usr/bin/sed -e 's/@@START_HERE@@/\/\/ AUTOMATICALLY GENERATED FILE - DO NOT EDIT/' -e 's/PREFIX_//' -e 's/^#.*//' ) > /C/msys64/home/peterbud/jdk9/build/windows-x86_64-normal-server-release/support/gensrc/java.base/sun/nio/ch/SocketOptionRegistry.java
> make[3]: *** [GensrcMisc.gmk:78: /C/msys64/home/peterbud/jdk9/build/windows-x86_64-normal-server-release/support/gensrc/java.base/sun/nio/ch/SocketOptionRegistry.java] Error 1
>
> Now the interesting is: if I copy this line above to the bash prompt, it runs without problem, and the file support/gensrc/java.base/sun/nio/ch/SocketOptionRegistry.java
> Is produced.
>
> Then I can again issue
> make JOBS=1 LOG=debug
>
> And the compile process is being continued, until a similar error pops up with a different generated file. I have an assumption that this happens because make is still running parallel jobs, despite JOBS=1 but I’m not sure.
>
> How could I best tackle this?
>
> Thank you and best regards,
>
> Peter
>
> Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10
>
> From: Magnus Ihse Bursie<mailto:magnus.ihse.bursie at oracle.com>
> Sent: Thursday, October 5, 2017 11:33 AM
> To: Erik Joelsson<mailto:erik.joelsson at oracle.com>; Peter Budai<mailto:peterbudai at hotmail.com>; build-dev at openjdk.java.net<mailto:build-dev at openjdk.java.net>
> Subject: Re: Building OpenJDK9 on MSYS2
>
> On 2017-10-05 10:10, Erik Joelsson wrote:
>> Hello Peter,
>>
>>
>> On 2017-10-04 21:15, Peter Budai wrote:
>>> Hi Magnus,
>>>
>>> Thanks for the quick reply I’ll check these patches with msys2.
>>>
>>> Let me specify with more details what I’d like to achieve: I’d like
>>> to build OpenJDK9 with MSYS2 MINGW64 environment using gcc toolchain.
>>> (I’m not sure how familiar are you with MSYS2, but there are 3
>>> different environments: MSYS2, MINGW32 and MINGW64). In theory
>>> MINGW64 with gcc is the closes you can get on Windows platform as a
>>> gcc unix like build environment, which produces still a native 64-bit
>>> executable on Windows.
>>>
>>> I’m not very familiar with OpenJDK yet, so therefore I’d like to hear
>>> your opinion: how realistic is that?
>> Sorry to disappoint, but I would say that requires major work. There
>> is a strong historic assumption that windows builds are done using
>> Visual Studio. We have abstracted away some of it in configure (see
>> TOOLCHAIN_TYPE), but it's very far from enough to change compiler
>> environment for a Windows build. The native sources are also bound to
>> make a lot of such assumptions. I would expect the changes needed to
>> be in the thousands of lines of code.
>
> I agree that it requires hard work (even if "thousands" might be an
> overestimation I think, but "hundreds" is not enough, so it's the right
> magnitude). On the other hand, it would be really good if we did sort
> things out, so that we had proper conditions based on OS vs
> compiler/toolchain.
>
> If you really want to start, the first thing is to patch toolchain.m4 to
> VALID_TOOLCHAINS_windows="microsoft gcc"
> and then call configure using "bash configure --with-toolchain-type=gcc".
>
> As Erik, I doubt you will come very far before things starts tumbling down.
>
>>
>> When we say supporting the build in msys2 instead of cygwin, we just
>> mean using msys2 as the unix emulating layer for our tools like
>> make/bash/grep/sed etc.
>>
>> One think I have done successfully is running the build in WSL
>> (Windows Subsystem for Linux), but that isn't all that helpful as WSL
>> for practical purposes is more or less like running Linux in a VM, so
>> the build sees a Linux system and builds a Linux binary.
>>> As a side note: with MINGW64 I have managed to run configure phase
>>> successfully for OpenJDK. The compile process has also started and
>>> went for a while, but interestingly I run into some kind of race
>>> conditions as make stopped with an error. Using LOG=debug I have fond
>>> the failing line and then copying the failed command and pasting it
>>> to the bash prompt it successfully generated the output target, and
>>> then the build process run further when a similar situation happened.
>>> Also pasting the failed command run in the bash without any problem,
>>> and build continued… until the next.
>> Without seeing the errors I can't say much. I very much doubt that you
>> are running with gcc as the compiler though. Configure isn't easily
>> fooled into using a different compiler to what it prefers, and I would
>> expect things to crash and burn pretty early if you actually did.
>>
>> /Erik
>>>
>>> I have tried to run make JOBS=1, but did not help, strangely I have
>>> still seen in the log make[3] and make[4] logs which suggested that
>>> there are more than one make jobs were running. Also tried .configure
>>> --with-output-sync=recurse without success (same symptoms)
>>>
>>> Let me know your thoughts.
>>>
>>> Best regards,
>>>
>>> Peter
>>>
>>> Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for
>>> Windows 10
>>>
>>> From: Magnus Ihse Bursie<mailto:magnus.ihse.bursie at oracle.com>
>>> Sent: Wednesday, October 4, 2017 1:04 AM
>>> To: Peter Budai<mailto:peterbudai at hotmail.com>;
>>> build-dev at openjdk.java.net<mailto:build-dev at openjdk.java.net>
>>> Subject: Re: Building OpenJDK9 on MSYS2
>>>
>>> Actually, it wasn't so much remaining trouble. :-) I fired up msys2 and
>>> checked out where I left off. It turned out that the remaining snag was
>>> that msys2 tries to convert command lines automatically, from "unix"
>>> style paths to "windows" style paths. Unfortunately, it does not do this
>>> very well and it breaks all sorts of things. We already have a FIXPATH
>>> solution in place which deals with this, so basically all I had to do
>>> was disable this (by setting MSYS2_ARG_CONV_EXCL to "*"). However, this
>>> broke our cygpath replacement hack (!) so I had to disable it there.
>>> Sigh. Anyway, with those fixes it ran and worked well. (I also
>>> discovered and fixed a bug related to how we set up the FIXPATH variable
>>> on msys, but it only triggers in certain circumstances).
>>>
>>> With this patch I now jdk9 seems to build fine on msys2. It should apply
>>> cleanly on jdk9/jdk9. Since it turned out to be so trivial, I'll try to
>>> get it in in jdk10.
>>>
>>> Here's the patch if you want to apply it yourself:
>>>
>>> diff -r a08cbfc0e4ec common/autoconf/basics_windows.m4
>>> --- a/common/autoconf/basics_windows.m4 Thu Aug 03 18:56:56 2017
>>> +0000
>>> +++ b/common/autoconf/basics_windows.m4 Wed Oct 04 00:53:58 2017
>>> +0200
>>> @@ -42,7 +42,7 @@
>>> windows_path=`$CYGPATH -m "$unix_path"`
>>> $1="$windows_path"
>>> elif test "x$OPENJDK_BUILD_OS_ENV" = "xwindows.msys"; then
>>> - windows_path=`cmd //c echo $unix_path`
>>> + windows_path=`MSYS2_ARG_CONV_EXCL= cmd //c echo $unix_path`
>>> $1="$windows_path"
>>> fi
>>> ])
>>> @@ -136,6 +136,16 @@
>>> fi
>>> ])
>>>
>>> +AC_DEFUN([BASIC_MSYS_UPDATE_FIXPATH],
>>> +[
>>> + # Take all collected prefixes and turn them into a
>>> -m/c/foo@/c/bar at ... command line
>>> + # @ was chosen as separator to minimize risk of other tools messing
>>> around with it
>>> + all_unique_prefixes=`echo "${all_fixpath_prefixes@<:@@@:>@}" \
>>> + | tr ' ' '\n' | $GREP '^/./' | $SORT | $UNIQ`
>>> + fixpath_argument_list=`echo $all_unique_prefixes | tr ' ' '@'`
>>> + FIXPATH="$FIXPATH_BIN -m$fixpath_argument_list"
>>> +])
>>> +
>>> AC_DEFUN([BASIC_FIXUP_PATH_MSYS],
>>> [
>>> path="[$]$1"
>>> @@ -143,7 +153,7 @@
>>> new_path="$path"
>>> if test "x$has_colon" = x; then
>>> # Not in mixed or Windows style, start by that.
>>> - new_path=`cmd //c echo $path`
>>> + new_path=`MSYS2_ARG_CONV_EXCL= cmd //c echo $path`
>>> fi
>>>
>>> BASIC_MAKE_WINDOWS_SPACE_SAFE_MSYS([$new_path])
>>> @@ -155,6 +165,8 @@
>>>
>>> # Save the first 10 bytes of this path to the storage, so fixpath
>>> can work.
>>> all_fixpath_prefixes=("${all_fixpath_prefixes@<:@@@:>@}"
>>> "${new_path:0:10}")
>>> + # We might need to re-evaluate FIXPATH.
>>> + BASIC_MSYS_UPDATE_FIXPATH
>>> ])
>>>
>>> AC_DEFUN([BASIC_FIXUP_EXECUTABLE_CYGWIN],
>>> @@ -293,7 +305,7 @@
>>> # Do not save /bin paths to all_fixpath_prefixes!
>>> else
>>> # Not in mixed or Windows style, start by that.
>>> - new_path=`cmd //c echo $new_path`
>>> + new_path=`MSYS2_ARG_CONV_EXCL= cmd //c echo $new_path`
>>> BASIC_MAKE_WINDOWS_SPACE_SAFE_MSYS([$new_path])
>>> # Output is in $new_path
>>> BASIC_WINDOWS_REWRITE_AS_UNIX_PATH(new_path)
>>> @@ -302,6 +314,8 @@
>>>
>>> # Save the first 10 bytes of this path to the storage, so fixpath
>>> can work.
>>> all_fixpath_prefixes=("${all_fixpath_prefixes@<:@@@:>@}"
>>> "${new_path:0:10}")
>>> + # We might need to re-evaluate FIXPATH.
>>> + BASIC_MSYS_UPDATE_FIXPATH
>>> fi
>>> ])
>>>
>>> @@ -347,6 +361,10 @@
>>> WINDOWS_ENV_VENDOR='msys'
>>> WINDOWS_ENV_VERSION="$MSYS_VERSION"
>>>
>>> + # Prohibit msys2 path conversion from trying to be "intelligent",
>>> and rely
>>> + # on fixpath instead.
>>> + export MSYS2_ARG_CONV_EXCL="*"
>>> +
>>> AC_MSG_CHECKING([msys root directory as unix-style path])
>>> # The cmd output ends with Windows line endings (CR/LF), the grep
>>> command will strip that away
>>> MSYS_ROOT_PATH=`cd / ; cmd /c cd | $GREP ".*"`
>>> @@ -391,10 +409,7 @@
>>> elif test "x$OPENJDK_BUILD_OS_ENV" = xwindows.msys; then
>>> # Take all collected prefixes and turn them into a
>>> -m/c/foo@/c/bar at ... command line
>>> # @ was chosen as separator to minimize risk of other tools
>>> messing around with it
>>> - all_unique_prefixes=`echo "${all_fixpath_prefixes@<:@@@:>@}" \
>>> - | tr ' ' '\n' | $GREP '^/./' | $SORT | $UNIQ`
>>> - fixpath_argument_list=`echo $all_unique_prefixes | tr ' ' '@'`
>>> - FIXPATH="$FIXPATH_BIN -m$fixpath_argument_list"
>>> + BASIC_MSYS_UPDATE_FIXPATH
>>> fi
>>> FIXPATH_SRC_W="$FIXPATH_SRC"
>>> FIXPATH_BIN_W="$FIXPATH_BIN"
>>> diff -r a08cbfc0e4ec common/autoconf/build-aux/config.sub
>>> --- a/common/autoconf/build-aux/config.sub Thu Aug 03 18:56:56
>>> 2017 +0000
>>> +++ b/common/autoconf/build-aux/config.sub Wed Oct 04 00:53:58
>>> 2017 +0200
>>> @@ -30,7 +30,7 @@
>>> DIR=`dirname $0`
>>>
>>> # First, filter out everything that doesn't begin with "aarch64-"
>>> -if ! echo $* | grep '^aarch64-' >/dev/null ; then
>>> +if ! echo $* | grep -e '^aarch64-' -e 'msys' >/dev/null ; then
>>> . $DIR/autoconf-config.sub "$@"
>>> # autoconf-config.sub exits, so we never reach here, but just in
>>> # case we do:
>>> @@ -38,13 +38,17 @@
>>> fi
>>>
>>> while test $# -gt 0 ; do
>>> - case $1 in
>>> + case $1 in
>>> -- ) # Stop option processing
>>> shift; break ;;
>>> aarch64-* )
>>> config=`echo $1 | sed 's/^aarch64-/arm-/'`
>>> sub_args="$sub_args $config"
>>> shift; ;;
>>> + *-msys )
>>> + config=`echo $1 | sed 's/msys/mingw32/'`
>>> + sub_args="$sub_args $config"
>>> + shift; ;;
>>> - ) # Use stdin as input.
>>> sub_args="$sub_args $1"
>>> shift; break ;;
>>> diff -r a08cbfc0e4ec common/autoconf/spec.gmk.in
>>> --- a/common/autoconf/spec.gmk.in Thu Aug 03 18:56:56 2017 +0000
>>> +++ b/common/autoconf/spec.gmk.in Wed Oct 04 00:53:58 2017 +0200
>>> @@ -120,6 +120,13 @@
>>> # On Windows, the Visual Studio toolchain needs the PATH to be
>>> adjusted
>>> # to include Visual Studio tools (this needs to be in cygwin/msys
>>> style).
>>> export PATH:=@VS_PATH@
>>> +
>>> +endif
>>> +
>>> +ifeq ($(OPENJDK_TARGET_OS_ENV), windows.msys)
>>> + # On msys2, prohibit msys path conversion from trying to be
>>> + # "intelligent", and rely on fixpath instead.
>>> + export MSYS2_ARG_CONV_EXCL:=*
>>> endif
>>>
>>> SYSROOT_CFLAGS := @SYSROOT_CFLAGS@
>>>
>>> /Magnus
>>>
>>> On 2017-10-03 22:34, Magnus Ihse Bursie wrote:
>>>> I gave msys2 a shot some time ago, but it ended up too much trouble.
>>>> I'll share some of my notes from that attempt, for what it's worth.
>>>>
>>>> To install package X/Y, run "pacman -S X/Y". Missing tools and
>>>> packages where to find them:
>>>> cmp: msys/diffutils
>>>> tar: msys/tar
>>>> make: msys/make
>>>> unzip: msys/unzip
>>>> zip: msys/zip
>>>>
>>>> config.sub reports msys as "x86_64-pc-mingw32" but msys2 as
>>>> "x86_64-pc-msys". This patch adds postprocessing in "our" config.sub
>>>> to report msys2 similar to msys. (Opinions, including my own :-) may
>>>> vary if this really is the best way..)
>>>>
>>>> diff -r b88023f46daa common/autoconf/build-aux/config.sub
>>>> --- a/common/autoconf/build-aux/config.sub Fri Jan 27 10:15:41
>>>> 2017 +0100
>>>> +++ b/common/autoconf/build-aux/config.sub Fri Feb 03 05:00:25
>>>> 2017 -0700
>>>> @@ -30,7 +30,7 @@
>>>> DIR=`dirname $0`
>>>>
>>>> # First, filter out everything that doesn't begin with "aarch64-"
>>>> -if ! echo $* | grep '^aarch64-' >/dev/null ; then
>>>> +if ! echo $* | grep -e '^aarch64-' -e 'msys' >/dev/null ; then
>>>> . $DIR/autoconf-config.sub "$@"
>>>> # autoconf-config.sub exits, so we never reach here, but just in
>>>> # case we do:
>>>> @@ -45,6 +45,10 @@
>>>> config=`echo $1 | sed 's/^aarch64-/arm-/'`
>>>> sub_args="$sub_args $config"
>>>> shift; ;;
>>>> + *-msys )
>>>> + config=`echo $1 | sed 's/msys/mingw32/'`
>>>> + sub_args="$sub_args $config"
>>>> + shift; ;;
>>>> - ) # Use stdin as input.
>>>> sub_args="$sub_args $1"
>>>> shift; break ;;
>>>>
>>>> If I remember correctly, this got me past the configure stage at the
>>>> time.
>>>>
>>>> I don't think it's very hard to get it to work on msys2, I just ran
>>>> into one snag too many and didn't think msys2 would be used by anyone.
>>>>
>>>> /Magnus
>>>>
>>>> On 2017-10-03 17:20, Peter Budai wrote:
>>>>> Hello,
>>>>>
>>>>> According to
>>>>> http://hg.openjdk.java.net/jdk9/jdk9/file/a08cbfc0e4ec/common/doc/building.html
>>>>>
>>>>> “msys2 and the new Windows Subsystem for Linux (WSL) would likely be
>>>>> possible to support in a future version but that would require a
>>>>> community effort to implement”
>>>>>
>>>>> I’d like to help making the OpenJDK 9 build working on msys2. What is
>>>>> the best way to move forward? Is there a similar effort in progress?
>>>>>
>>>>> Thank you and best regards,
>>>>>
>>>>> Peter
>>>>>
>>>>>
>>
>
>
More information about the build-dev
mailing list