Preliminary review for new WINENV support

Thu Jul 9 04:47:22 UTC 2020

On 2020/07/09 12:12, Derek Keeler wrote:
> After applying Suenaga's patch, I see this error:
> 
> checking wsl distribution... /home/dekeeler/dev/openjdk.net/sandbox/build/.configure-support/generated-configure.sh: line 32952: -d: command not found
> 
> I'm running Debian 10 in my WSL2 instance and it does not include `lsb_release` by default. Installed it with `sudo apt-get install lsb-release`.

It's better if we add AC_CHECK_PROG for lsb_release for Windows (for just WSL?)

> The next trouble I ran into is that I only have Visual Studio 2019 installed. I tried to make it work but lack the necessary experience to wrangle this code. Will start again in the morning after installing VS2017.

FYI: I can run configure script with VS2019 community edition on Ubuntu 20.04 on both WSL 1 and 2, however I could not complete building OpenJDK image on WSL 2 due to I/O performance issue.

Thanks,

Yasumasa

> -Derek
> 
> ________________________________
> From: build-dev <build-dev-retn at openjdk.java.net> on behalf of Derek Keeler <dekeeler at microsoft.com>
> Sent: July 8, 2020 5:53 PM
> To: Monica Beckwith <Monica.Beckwith at microsoft.com>; Magnus Ihse Bursie <magnus.ihse.bursie at oracle.com>; build-dev <build-dev at openjdk.java.net>
> Subject: Re: Preliminary review for new WINENV support
> 
> This is fantastic!
> 
> I'm synching to this branch tonight and will start building it locally on my WSL2 environment. I'll send questions/comments to this thread unless there is a better place?
> 
> (Note: our internal email system may obfuscate the links below in future replies, my apologies).
> 
> -Derek
> 
> ________________________________
> From: build-dev <build-dev-retn at openjdk.java.net> on behalf of Monica Beckwith <Monica.Beckwith at microsoft.com>
> Sent: July 5, 2020 11:34 AM
> To: Magnus Ihse Bursie <magnus.ihse.bursie at oracle.com>; build-dev <build-dev at openjdk.java.net>
> Subject: Re: Preliminary review for new WINENV support
> 
> Wow! Impressive work, @Magnus Ihse Bursie<mailto:magnus.ihse.bursie at oracle.com> . When working with cross-compilation mods for aarch64-win port, I realized we needed to clean up/better support the quirks around fixpath and also I was hoping to expand the support to wsl2, etc. From what I am reading, you seem to have touched all of these and more! I am mostly offline for the next two days, but will start testing your changes when I am back online. Thank you for doing the needful.
> 
> Monica
> 
> Get Outlook for Android<https://aka.ms/ghei36>
> 
> ________________________________
> From: build-dev <build-dev-retn at openjdk.java.net> on behalf of Magnus Ihse Bursie <magnus.ihse.bursie at oracle.com>
> Sent: Saturday, July 4, 2020 7:08:20 PM
> To: build-dev <build-dev at openjdk.java.net>
> Subject: Preliminary review for new WINENV support
> 
> I've been working for some time on a complete rewrite of how we handle the pecularities of the Windows build environment. This new solution supports Cygwin, Msys2, WSL1 and WSL2 (after a fashion, see below), what I have termed different "winenvs".
> 
> One of the main design goals has been to minimize the difference for the configure script and make files for the different winenvs, and leverage as much shared code between them as possible. Another has been to try to collect all the "trickiness" needed in as few places as possible, ideally just one, instead of having it spread out all over the configure script. A third design goal has been to prepare for cross-compilation for Windows from Linux, using Wine.
> 
> It pretty soon turned out that I needed to get a better grip on how we detect tools in configure, so a complete overhaul of this is included in the change. Now we have more or less fully parted with the original autoconf functions, which has long been too limited for us, and now finally has reached their end of usefulness.
> 
> At this point, I have a prototype / preview that basically works, but has some limitations.
> 
> I'd like to ask anyone interested in building OpenJDK on Windows to take the patch for a spin. Especially if you have an esoteric or exotic setup!
> 
> Webrev: https://nam06.safelinks.protection.outlook.com/?url=http:%2F%2Fcr.openjdk.java.net%2F~ihse%2Fwinenv-preview-1%2Fwebrev.01%2F&data=02%7C01%7Cdekeeler%40microsoft.com%7C28e792b26f604f86be7108d823a29bf5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637298528593747355&sdata=LkTMXBZRcaOe%2FRCvcckFViwDOysi7nS9H1Edve%2B4Alk%3D&reserved=0
> 
> (If you prefer, you can check out the branch "ihse-winenv-branch" on https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhg.openjdk.java.net%2Fjdk%2Fsandbox&data=02%7C01%7Cdekeeler%40microsoft.com%7C28e792b26f604f86be7108d823a29bf5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637298528593747355&sdata=g8UElT7Zx15fMItd8yGGjS%2BuPcyKd3i1EEVHJLumfUo%3D&reserved=0 instead of downloading the patch from the webrev.)
> 
> 
> I am leaving on vacation next week, so I won't be doing any more work on this for a while, but I promise to read all emails when I get back and try to rectify all issues that are reported. This means you have some time to try it out, too.
> 
> Here are some technical notes of what is changing, compared to the current Windows build.
> 
> The native "fixpath.exe" tool is gone. This means that we do not need to compile it during configure, which was always tricky to get right (mostly since we did not have fixpath in place to help us...).
> 
> Instead, there is a new fixpath.sh shell script, that does the same job, and more. The script is written in highly optimized shell code (yeah, I see the oxymoron) so only bash internal functionality is used, to avoid calling external tools, which is expensive on Windows. This makes the performance practically roughly at par with the native fixpath.exe.
> 
> Fixpath also has a "print" and "import" mode, apart from the traditional"exec" mode. This makes it possible to use the same tool for converting individual paths at runtime to Windows style ("print"), and it takes the logic needed to "import" paths given by the user to configure, into a format usable internally by the build system, and moves it into a centralized location in the fixpath script.
> 
> A "winenv" is defined by a quartet of variables: PATHTOOL, DRIVEPREFIX, ENVROOT and WINTEMP. For instance, for "cygwin", these are:
>   PATHTOOL=cygpath
>   DRIVEPREFIX=/cygdrive (typically)
>   ENVROOT=C:\Cygwin64 (typically)
>   WINTEMP=/tmp
> 
> These are needed for fixpath to do it's magic. Fixpath can auto-detect those, but to save on execution time they are normally detected by configure and sent as arguments to fixpath.
> 
> Detection of the Visual Studio environment has been massively simplified. Using fixpath, conversion between Windows and unix paths is not so complex anymore. The bridge Windows batch file that is needed to extract the environment variables is no longer created on the fly, but is instead stored in make/scripts/extract-vs-env.cmd. This is called with fixpath, so all arguments to it can be unix paths.
> 
> Furthermore, TOOLCHAIN_SETUP_VISUAL_STUDIO_ENV is now clearly designed to have a single responsibility, to set SYSROOT flags (and TOOLCHAIN_PATH) for the microsoft toolchain. As a result of this, it is now called from FLAGS_SETUP_SYSROOT_FLAGS. Also, the file toolchain_windows.m4 is now more correctly named toolchain_microsoft.m4. A price we had to pay for this was that the old idea that you should be able to start a "Visual Studio console" and then run configure from it to extract the variables do not work anymore. (It had not been tested for ages, and might have been broken anyway.)
> 
> Fixpath also knows about the difference between unix path lists (/foo/bar:/my/dir) and windows path lists (c:\dir;d:\data) and can convert freely and automatically between them. This furthermore means that PATH_SEP is removed, and we only use unix style (colon separated) path lists internally.
> 
> The logic automatically detects if and when .exe is needed, so EXE_SUFFIX is removed too. There are some limitations; when code needs to explicitly test for the presence of a file, the suffix needs to be correct. Also when make files check for e.g. the generated bin/java.exe, the suffix needs to be present. For this, I have introduced an EXECUTABLE_SUFFIX, that has the same value as EXE_SUFFIX -- but not the same semantics! The old code added EXE_SUFFIX here and there, not caring about if we meant "microsoft" as a toolchain or if we were running on Windows as a build platform. Clearly not well adapted for future cross-compilation.
> 
> The old ways of locating programs in configure were messy, complicated and not always correct. We used a mixture of the original autoconf macros, and our own UTIL_PATH_PROGS and UTIL_CHECK_TOOLS. These did not have very well defined semantics, and were frequently mixed up. Also, UTIL_CHECK_TOOLS required UTIL_FIXUP_EXECUTABLE tobe called afterwards, to "rewrite" the result in a second step. This was not needed after UTIL_PATH_PROGS but was frequently done anyway to "be on the safe side".
> 
> I have now replaced:
>    AC_(PATH|CHECK)_(PROG|PROGS) with UTIL_LOOKUP_PROGS
>    AC_(PATH|CHECK)_(TOOL|TOOLS) with UTIL_LOOKUP_TOOLCHAIN_PROGS
> 
> This is actually almost the same semantic, but unless you're an autoconf aficionado, you ar not likely to understand the difference     between "PROG" and "TOOL". The only difference is that UTIL_LOOKUP_TOOLCHAIN_PROGS will try to look for "host-prefixed" tools first, when cross-compiling, and should therefore be used for all toolchain lookups.
> 
> There is also a fail-fast version of both, UTIL_REQUIRE_PROGS and UTIL_REQUIRE_TOOLCHAIN_PROGS.
> 
> UTIL_LOOKUP_PROGS is the core function, with the rest being thin wrappers around it. This function is created from scratch, to do exactly what we want, on Unix platforms and Windows. So there is no need anymore to call UTIL_FIXUP_EXECUTABLE, unless you have input from elsewhere (e.g. user flag) that you need to verify. I have also collected all this logic in a single file, util_paths.m4, moving parts from util.m4, and just removing util_windows.m4.
> 
> UTIL_LOOKUP_PROGS will use the new and nifty function UTIL_CHECK_WINENV_EXEC_TYPE, which will automatically determine if an executable needs to be prefixed by fixpath! That means that you can match and mix Windows-style and Unix-style programs however you like, with very few limitations. For instance, you can have a Linux version of the BootJDK on WSL. For this to work, the $FIXPATH prefix is now stored in the variables themselves (e.g. in $CC), rather than added as @FIXPATH@ in spec.gmk.in. This has generally worked out OK, but caused some headaches when the code thought that $CC (etc) was not a way to launch a program, but was a file that could be tested for existence, etc.
> 
> I reintroduced support for msys2. This was mostly free, since msys2 is so close to cygwin (which msys never where). To underline the difference, I renamed the winenv windows.msys2 (instead of windows.msys). Msys (version 1) will never be supported in modern OpenJDK due to critical packages being far too old, and not updated. I also clearly separate between WSL1 (which is using a kernel emulation layer, somewhat like Wine) and WSL2 (which is using a Hyper-V VM to run an actual Linux kernel), as windows.wsl1 and windows.wsl2.
> 
> I have also done a ton of small fixes, to make things more convenient and smooth when working in these winenvs. I have adjusted the output from configure to be less verbose and more streamlined. Overall, a lot of odd code in configure has been removed. A few changes that are strictly unneeded for this patch has also been included; mostly removal of dead code I came across, and a few fixes for additional debuggability that I needed when developing this patch, like ExecuteithLog.
> 
> I have also temporarily disabled the javac server, and building without absolute paths. I believe a better way forward is to address these with improved functionality in separate patches, instead of trying to work around the issues introduced by them in this patch.
> 
> I have done substantial testing of the core functionality (building jdk-images) on Cygwin, Msys2 and WSL2, both on my personal machine and on Oracle's CI system. The performance on Cygwin (which we use for our CI builds) is roughly on par with the old Cygwin performance, but the WSL1 performance is roughly 20% faster, so I think we should investigate if it is possible for us to switch to WSL1. Everything seems stable so far, but more testing is definitely needed. I have also not even started testing autxillary     functionality, such as the compare script, IDE project file generation etc.
> 
> However, the big disappointment in all of this is WSL2. That was the main driver that got me started on this rewrite. But it turned out that WSL2 is still very immature. :-( There are lot of issues, like stdout and stderr getting messed up, time synchronization issues causing make to complain about "Clock skew detected", extreme slowness when accessing disks cross the OS boundary. But worst of all has been that WSL2 is *extremly* unstable. After a few calls of Window executables, I get the impression that a conversion daemon dies. All further calls to Window binaries just stalls, and the WSL2 instance is unusable until I terminate it using "wsl -t Ubuntu-2004" from a cmd shell. :-(
> 
> So the WSL2 functionality is not very well tested, since I have not been able to build it completely a single time. I do believe that everything is correct, in "theory". So in case this is something broken with my WSL2 installation, I'd encourage anyone with a WSL2 installation to try it out.
> 
> /Magnus
> 