Preliminary review for new WINENV support

Tue Nov 10 11:30:21 UTC 2020

Hi Bernhard,

Thank you for you efforts in keeping this patch viable and getting it 
closer to integration!

I have no intention of letting this bitrot. The patch is, still, blocked 
by major simplifications I made:

  * There is no support for the javac server
  * There is no support for building without absolute pathnames

This functionality is a must-have. The good news is that there is a 
relative clear way forward for me to achieve both of these, which in 
both cases involves fixing the code base: to simplify how we pass 
arguments to the javac server and how we build without absolute 
pathnames on Windows. And in both cases these will be good additions in 
their own right.

After that, all irregularities must be fixed, so there is no regression 
in terms of building functionality or major performance regressions on 
already supported platforms. Expanding to support new compilation 
combinations is certainly a nice addition, but not one that's required 
for an initial checkin.

I apologize for the slow progress on this patch. Due to personal and 
family issues, I am unfortunately unable to work on the OpenJDK build as 
much as I'd like, and the time I can spend is somewhat unpredictable. 
But I can assure you that this patch is important for me as well, and 
I'm trying to prioritize it as much as I can.

/Magnus

On 2020-11-10 11:01, Bernhard Urban-Forster wrote:
> Hello Magnus,
>
> Sorry for the late reply! This is awesome work and I don't want to see that to bit rot :-)
>
> I rebased your changes on top of current master: https://github.com/openjdk/jdk/compare/master...lewurm:winenv-testing
>
> This branch is by no means ready to be reviewed, it's just whatever I've used to do my experiments.
>
> Also note that it was a bit painful to merge the changes in toolchain_windows.m4 due to the renaming. While I agree that toolchain_microsoft.m4 is the better name, I suggest to defer that change to a later point.
>
> I've tested the following scenarios:
>
> 1. cygwin on x86_64 to build x86_64
> 2. cygwin on x86_64 to build arm64
> 3. WSL1 on x86_64 to build x86_64
> 4. WSL1 on x86_64 to build arm64
> 5. WSL2 on x86_64 to build x86_64
> 6. WSL1 on arm64 to build arm64
>
> Some context: Microsoft doesn't ship native Arm64 binaries for the VS toolchain (yet). Instead one can run x86 binaries via xtajit (a binary translator), but as one can imagine this comes with some performance penalty.
> Cygwin is also not available on Arm64. Cygwin already being slow, it made sense to avoid running Cygwin x86 on Arm64, and instead go the cross-compilation route in the OpenJDK build for Windows+Arm64.
> Previously the OpenJDK build did not support cross-compiling on Windows. Even today, on openjdk/jdk:master a dirty workaround for fixpath.exe is needed to build Windows+Arm64 ( https://github.com/openjdk/jdk/pull/212#issuecomment-695024586 ). So we have been very excited to see this WINENV patch :-)
>
> Here are comments for each scenario.
>
> ======================================
> ## 1. cygwin on x86_64 to build x86_64
> ======================================
>
> ```
> $ bash configure --with-devkit=/cygdrive/c/work/VS2019-16.6.1-devkit --with-build-devkit=/cygdrive/c/work/VS2019-16.6.1-devkit --with-boot-jdk=/cygdrive/c/work/jdk-16+22 --openjdk-target=aarch64-unknown-cygwin
> configure: Configuration created at Mon Nov  9 21:33:42 CET 2020.
> checking for basename... /usr/bin/basename
> checking for dirname... /usr/bin/dirname
> checking for file... /usr/bin/file
> checking for ldd... /usr/bin/ldd
> checking for bash... /usr/bin/bash
> checking for cat... /usr/bin/cat
> checking for chmod... /usr/bin/chmod
> checking for cp... /usr/bin/cp
> checking for cut... /usr/bin/cut
> checking for date... /usr/bin/date
> checking for gdiff... /cygdrive/c/work/winenv-cygwin/build/.configure-support/generated-configure.sh: line 10043: test: too many arguments
> /cygdrive/c/work/winenv-cygwin/build/.configure-support/generated-configure.sh: line 10047: test: too many arguments
> /cygdrive/c/work/winenv-cygwin/build/.configure-support/generated-configure.sh: line 10043: test: too many arguments
> /cygdrive/c/work/winenv-cygwin/build/.configure-support/generated-configure.sh: line 10047: test: too many arguments
> /cygdrive/c/work/winenv-cygwin/build/.configure-support/generated-configure.sh: line 10043: test: too many arguments
> /cygdrive/c/work/winenv-cygwin/build/.configure-support/generated-configure.sh: line 10047: test: too many arguments
> /cygdrive/c/work/winenv-cygwin/build/.configure-support/generated-configure.sh: line 10043: test: /cygdrive/c/Program: binary operator expected
> /cygdrive/c/work/winenv-cygwin/build/.configure-support/generated-configure.sh: line 10047: test: /cygdrive/c/Program: binary operator expected
> /cygdrive/c/work/winenv-cygwin/build/.configure-support/generated-configure.sh: line 10043: test: too many arguments
> /cygdrive/c/work/winenv-cygwin/build/.configure-support/generated-configure.sh: line 10047: test: too many arguments
> [not found]
> [...]
> ```
>
> I didn't investigate it any further, as this has already been reported on the mailing list.
>
> =====================================
> ## 2. cygwin on x86_64 to build arm64
> =====================================
> ```
> $ bash configure --with-devkit=/cygdrive/c/work/VS2019-16.6.1-devkit --with-build-devkit=/cygdrive/c/work/VS2019-16.6.1-devkit --with-boot-jdk=/cygdrive/c/work/jdk-16+22 --openjdk-target=aarch64-unknown-cygwin
> ```
> Same as scenario 1)
>
> ====================================
> ## 3. WSL1 on x86_64 to build x86_64
> ====================================
> I do get some weird permission errors, so I run everything with sudo (chmod -R $user doesn't help):
> ```
> $ sudo bash configure --with-build-devkit="/mnt/c/work/VS2019-16.6.1-devkit" --with-devkit="/mnt/c/work/VS2019-16.6.1-devkit" --with-boot-jdk="/mnt/c/work/jdk-16+22"
> [...]
>
> Configuration summary:
> * Debug level:    release
> * HS debug level: product
> * JVM variants:   server
> * JVM features:   server: 'aot cds compiler1 compiler2 epsilongc g1gc graal jfr jni-check jvmci jvmti management nmt parallelgc serialgc services shenandoahgc vm-structs zgc'
> * OpenJDK target: OS: windows, CPU architecture: x86, address length: 64
> * Version string: 16-internal+0-adhoc.root.winenv-wsl1 (16-internal)
>
> Tools summary:
> * Environment:    wsl1 version 4.4.0-19041-Microsoft, #488-Microsoft Mon Sep 01 13:43:00 PST 2020 (Ubuntu 20.04.1 LTS); windows version 10.0.19041.572; prefix "/mnt"; root "[unavailable]"
> * Boot JDK:       openjdk version "16" 2021-03-16 OpenJDK Runtime Environment AdoptOpenJDK (build 16+22-202011040337) OpenJDK 64-Bit Server VM AdoptOpenJDK (build 16+22-202011040337, mixed mode, sharing) (at /mnt/c/work/jdk-16~1)
> * Toolchain:      microsoft (Microsoft Visual Studio 2019 16.6.1 (devkit))
> * C Compiler:     Version 19.26.28806 (at /mnt/c/work/vs2019-16.6.1-devkit/vc/bin/x64/cl.exe)
> * C++ Compiler:   Version 19.26.28806 (at /mnt/c/work/vs2019-16.6.1-devkit/vc/bin/x64/cl.exe)
>
> Build performance summary:
> * Cores to use:   12
> * Memory limit:   32605 MB
> ```
>
> The build fails eventually:
> ```
> $ sudo make jdk-image JOBS=12 LOG=info
> [...]
> /mnt/c/work/winenv-wsl1/make/scripts/fixpath.sh: line 405: /mnt/c/work/winenv-wsl1/build/windows-x86_64-server-release/jdk/bin/java: No such file or directory
> make[3]: *** [ExplodedImageOptimize.gmk:41: /mnt/c/work/winenv-wsl1/build/windows-x86_64-server-release/jdk/_optimize_image_exec.marker] Error 127
> make[2]: *** [make/Main.gmk:437: exploded-image-optimize] Error 2
> make[2]: *** Waiting for unfinished jobs....
> ```
> Which looks like a path issue, that should be relatively easy to fix (I didn't investigate). HotSpot and parts of the JDK built successfully though:
> ```
> $ ./build/windows-x86_64-server-release/jdk/bin/java.exe -version
> openjdk version "16-internal" 2021-03-16
> OpenJDK Runtime Environment (build 16-internal+0-adhoc.root.winenv-wsl1)
> OpenJDK 64-Bit Server VM (build 16-internal+0-adhoc.root.winenv-wsl1, mixed mode)
> ```
> So I declare success on this one :-)
>
> ===================================
> ## 4. WSL1 on x86_64 to build arm64
> ===================================
> ```
> $ bash configure --with-build-devkit="/mnt/c/work/VS2019-16.6.1-devkit" --with-devkit="/mnt/c/work/VS2019-16.6.1-devkit" --with-boot-jdk="/mnt/c/work/jdk-16+22" --openjdk-target=aarch64-unknown-cygwin
> [...]
> checking for link... $FIXPATH /mnt/c/work/vs2019-16.6.1-devkit/vc/bin/arm64/link.exe
> checking if the found link.exe is actually the Visual Studio linker... yes
> configure: Using microsoft linker version 14.26.28806.0 [Microsoft (R) Incremental Linker Version 14.26.28806.0]
> checking for aarch64-unknown-cygwin-ml... [not found]
> checking for ml... [not found]
> configure: WARNING: using cross tools not prefixed with host triplet
> configure: WARNING: Ignoring value of AS from the environment. Use command line variables instead.
> checking for ml... [not found]
> checking for aarch64-unknown-cygwin-lib... [not found]
> checking for lib... $FIXPATH /mnt/c/work/vs2019-16.6.1-devkit/vc/bin/arm64/lib.exe
> checking for aarch64-unknown-cygwin-mt... [not found]
> checking for mt... $FIXPATH /mnt/c/work/vs2019-16.6.1-devkit/10/bin/x64/mt.exe
> checking for aarch64-unknown-cygwin-rc... [not found]
> checking for rc... $FIXPATH /mnt/c/work/vs2019-16.6.1-devkit/10/bin/x64/rc.exe
> checking for aarch64-unknown-cygwin-dumpbin... [not found]
> checking for dumpbin... $FIXPATH /mnt/c/work/vs2019-16.6.1-devkit/vc/bin/arm64/dumpbin.exe
> checking for aarch64-unknown-cygwin-gobjdump... [not found]
> checking for gobjdump... [not found]
> checking for aarch64-unknown-cygwin-objdump... [not found]
> checking for objdump... /usr/bin/objdump
> checking for build platform devkit... Microsoft Visual Studio 2019 16.6.1 (devkit) in /mnt/c/work/vs2019-16.6.1-devkit
> checking for cl... fixpath: failure: Cannot locate cygpath or wslpath
>
> configure: error: Could not find required tool for BUILD_CC
> /mnt/c/work/winenv-wsl1/build/.configure-support/generated-configure.sh: line 12: sed: command not found
> /mnt/c/work/winenv-wsl1/build/.configure-support/generated-configure.sh: line 38: sort: command not found
> /mnt/c/work/winenv-wsl1/build/.configure-support/generated-configure.sh: line 35: sed: command not found
> /mnt/c/work/winenv-wsl1/build/.configure-support/generated-configure.sh: line 53: sort: command not found
> /mnt/c/work/winenv-wsl1/build/.configure-support/generated-configure.sh: line 77: cat: command not found
> /mnt/c/work/winenv-wsl1/build/.configure-support/generated-configure.sh: line 40: rm: command not found
> ```
> It looks like it's still not suitable for cross compilation. I might have messed up resolving some merge conflicts. I haven't investigated that further, but that would probably be a scenario that would be nice to have eventually.
>
> ====================================
> ## 5. WSL2 on x86_64 to build x86_64
> ====================================
>
> I second the clock skew observations :-( I had a problem like that on WSL2 in a different context before https://github.com/microsoft/WSL/issues/4898
>
> Apart from that, crossing the "file system boundary" got worse with WSL2, and as long checking out the sources on a Windows visible path is a requirment, I'm not sure if it's worth going down this road.
>
> ==================================
> ## 6. WSL1 on arm64 to build arm64
> ==================================
> Same problem with permissions. I must be doing something wrong.
> ```
> $ sudo bash configure --with-devkit="/mnt/c/lewurm/VS2019-16.6.1-devkit" --with-boot-jdk="/mnt/c/lewurm/jdk-16-ea+19-windows-aarch64" --disable-precompiled-headers
> [...]
> Configuration summary:
> * Debug level:    release
> * HS debug level: product
> * JVM variants:   server
> * JVM features:   server: 'cds compiler1 compiler2 epsilongc g1gc jfr jni-check jvmti management nmt parallelgc serialgc services shenandoahgc vm-structs zgc'
> * OpenJDK target: OS: windows, CPU architecture: aarch64, address length: 64
> * Version string: 16-internal+0-adhoc.root.openjdk-jdk (16-internal)
>
> Tools summary:
> * Environment:    wsl1 version 4.4.0-19607-Microsoft, #1000-Microsoft Wed Apr 09 15:02:00 PST 2020 (Ubuntu 20.04 LTS); windows version 10.0.19607.1000; prefix "/mnt"; root "[unavailable]"
> * Boot JDK:       openjdk version "16-ea" 2020-10-09 OpenJDK Runtime Environment Microsoft (build 16-ea+19-windows-aarch64) OpenJDK 64-Bit Server VM Microsoft (build 16-ea+19-windows-aarch64, mixed mode) (at /mnt/c/lewurm/jdk-16~1)
> * Toolchain:      microsoft (Microsoft Visual Studio 2019 16.6.1 (devkit))
> * C Compiler:     Version 19.27.29112 (at /mnt/c/lewurm/vs2019-16.6.1-devkit/vc/bin/arm64/cl.exe)
> * C++ Compiler:   Version 19.27.29112 (at /mnt/c/lewurm/vs2019-16.6.1-devkit/vc/bin/arm64/cl.exe)
>
> Build performance summary:
> * Cores to use:   64
> * Memory limit:   524204 MB
> ```
> So a couple of things that I hacked up:
> * I manually copied x86 MSVC binaries into /mnt/c/lewurm/vs2019-16.6.1-devkit/vc/bin/arm64
> * I manually copied arm64 binaries from the Windows SDK into /mnt/c/lewurm/vs2019-16.6.1-devkit/10/bin/arm64 Eventually I need to update the devkit script to create a proper archive and/or update the build script to be aware that x86 MSVC binaries must be used on arm64 (the hope is that this eventually won't be true anymore).
> * add detection for WSL on aarch64: https://github.com/lewurm/openjdk/commit/4da4b6df16d05c97b7f48bef166e6d2080d37511
> * The boot-jdk is from https://github.com/microsoft/openjdk-aarch64/releases
> * `--disable-precompiled-headers` is needed because otherwise a bug in the binary translator (xtajit) is triggered when running x86 cl.exe with it. Note to myself: File a bug report.
>
>
> Let's build a jdk image:
> ```
> $ sudo make jdk-image JOBS=64 LOG=info
> [...]
> /mnt/c/lewurm/openjdk-jdk/make/scripts/fixpath.sh: line 405: /mnt/c/lewurm/openjdk-jdk/build/windows-aarch64-server-release/jdk/bin/java: No such file or directory
> make[3]: *** [ExplodedImageOptimize.gmk:41: /mnt/c/lewurm/openjdk-jdk/build/windows-aarch64-server-release/jdk/_optimize_image_exec.marker] Error 127
> make[2]: *** [make/Main.gmk:437: exploded-image-optimize] Error 2
> $ ./build/windows-aarch64-server-release/jdk/bin/java.exe -version
> openjdk version "16-internal" 2021-03-16
> OpenJDK Runtime Environment (build 16-internal+0-adhoc.root.openjdk-jdk)
> OpenJDK 64-Bit Server VM (build 16-internal+0-adhoc.root.openjdk-jdk, mixed mode)
> ```
> So same issue as in scenario 3 (WSL1 on x86_64 building x86_64), which is _great_ news :-)
>
> ==============
> ## Performance
> ==============
> I did a comparision run on WSL1 on x86_64 to build x86_64 (scenario 3), once with the WINENV patches applied and once without it.
> Overall it seems there is some performance regression, but not too bad. The profile logs aren't very meaningful to me, but if I would need to guess the difference is due to having a native `fixpath.exe` vs. `fixpath.sh`.
>
> Logs are here: https://gist.github.com/lewurm/2fbf2971a55a3a4a4f5933b105925976
>
>
> =========
> ## Epilog
> =========
>
> I hope this has met your bar of exotic setup.  Unfortunately I can't provide much feedback on the automagic.  I would love to see your work landing eventually!
>
> Thank you,
> -Bernhard
>
> ________________________________________
> From: build-dev <build-dev-retn at openjdk.java.net> on behalf of Magnus Ihse Bursie <magnus.ihse.bursie at oracle.com>
> Sent: Friday, August 28, 2020 11:09
> To: Ludovic Henry; Yasumasa Suenaga; Monica Beckwith; build-dev
> Subject: Re: Preliminary review for new WINENV support
>
> On 2020-08-24 19:21, Ludovic Henry wrote:
>> Hi Magnus,
>>
>> I want to follow up on this work of yours, as we've particular interest in it for the Windows-AArch64 port.
>>
>> Let me know how I could assist you in this effort.
> The big blockers for this fix is basically these two issues:
>
>>> I have also temporarily disabled the javac server, and building without absolute paths. I believe a better way forward is to address these with improved functionality in separate patches, instead of trying to work around the issues introduced by them in this patch.
> Until that is fixed, any integration of this patch is a no-go.
>
> I have been working on those, but a lot of other stuff has kept me from
> gaining much progress; everything from long vacations to preparations to
> the switch to github. Unfortunately, I don't think there's an easy way
> for you to help on those two blockers.
>
> However, there are two more things that I'd like to look into, if
> possible, before integrating my patch, and there you might be able to help.
>
> One is performance. I got a somewhat different performance profile with
> my patch than without -- some things were faster, and some were slower.
> I'd like to understand why some things got slower, and if it is possible
> to do anything about. (Why things got faster I don't care about, I'm
> just happy :))
>
> So maybe you can help me with some performance analysis? With the
> limited availability of tooling for makefiles, this is not really
> trivial. What I usually do is I run with JOBS=1 (to be able to track
> each task individually without effects of concurrency), and
> LOG=info,profile. This will give you time stamps for each individual
> execution in build.log.  And see if there is any interesting differences
> with and without the patch.
>
> The other thing is that WSL2 works awfully bad. :( Maybe there's nothing
> to be done about it, if the thing being shipped is just too buggy, but
> maybe we do things incorrectly, or that unnecessarily triggers bugs. Or
> maybe there are workarounds we could use. But I think it could be worth
> some more effort to try to get it working. Maybe you can help with this?
>
> /Magnus