[PATCH] Support for building using WSL (Windows Subsystem for Linux) on Windows

[Magnus Ihse Bursie]

On 2018-12-11 06:25, Andrew Luo wrote:
> Hi,
>
> Yes, I've signed an OCA (I've also contributed changes to other groups before, but not build).
>
> Okay, I have fixed the autconf-config* files.
>
> Unfortunately, as Erik mentioned, there is no (supported/reliable) way to access the WSL root / from /cygdrive/c, or even from Windows (there is a way in reality, however, it's not documented/supported by Microsoft and the location changes depending on the distribution/store app id/etc. so best to avoid using it.)  I can see if we can print information about versions however.
>
> Right, WSL requires the .exe extension when accessing an executable, as this is Linux behavior (Linux doesn't have extensions for executables generally, but that's besides the point)...
>
> I fixed BASIC_REMOVE_SYMBOLIC_LINKS - a leftover from another approach I tried.
>
> For the redirect, redirect doesn't seem to be working when you have a bash shell input piped into a Win32 executable reading from stdin using WINAPI.  I'm not sure this is supported by the OpenJDK, more likely it might be a Microsoft issue.  For some reason, the stdin would be cut off (or I would see an exception thrown from available0 in FileInputStream).  I personally didn't see any harm in changing piping into input/output files (since all the inputs/outputs are files anyways!).
Ok, let me be sure I get this right. It is only the redirect of *input* 
that fails? (But you fixed both because of consistency). I agree that 
the change itself is fine, even better than it is right now -- I was 
mostly worried about the consequences of redirects is not working; there 
might be other places that fail. But if redirecting output works, I 
think we're mostly fine. That's something we do all the time, for each 
executed command, so if that did not work reliably it would be really bad.

But still... I tried greping for "<" and there's a lot of places, 20+, 
that redirects input.

Or did this problem only happen when running *java* as the recipient of 
the redirected input?

This worries me, and while I do think your change makes the tools have a 
better UI, I don't like this as a workaround that will not solve all 
potential problems. :(

>
> I fixed TOOLCHAIN_FIND_VISUAL_STUDIO_BAT_FILE - this was also from a few things I had tried earlier.
>
> I disabled the $BASH code because to call bash from Win32 the correct way is either "wsl /bin/bash" or just "bash".  $BASH correctly evaluates to /bin/bash, however BASIC_WINDOWS_REWRITE_AS_WINDOWS_MIXED_PATH is implemented in terms of wslpath, which can only convert a path under /mnt/c back to a Windows path.  Other files under /, for example /bin and /bin/bash, cannot be converted to a Windwos path.
>
> The escaping changes I made because it wasn't working.  This does work with spaces in the path on WSL.  I don't have a Cygwin environment to check, perhaps someone else here could help out?  Otherwise I can refactor that code to use that echo statement for WSL and use the old echo statement for Cygwin.
I can check it out the next time I'm on a Windows machine.

>
> I have fixed the extraneous debug print statement.
>
> As for Windows vs Linux output - you can still force it to build a Linux output binary.  You just need to run configure as follows:
>
> ./configure --with-boot-jdk=/home/andrew/jdk-11.0.1 ----build=x86_64-unknown-linux-gnu --host=x86_64-unknown-linux-gnu
>
> However, there is a behavior change: now, on WSL, by default, Windows binaries are targeted.  Previously, Linux binaries were the default target.  (Also, you can run configure twice and two sets of configurations will be generated, you can actually build both images by setting CONF=linux-x86_64-server-release or CONF=windows-x86_64-server-release)
If you run on WLS, it's reasonable that the default is Windows. The 
--build --host combo is good enough for me as a way to force a linux 
build; we don't need an extra flag for this somewhat odd build 
configuration.

>
> As for BASIC_MAKE_WINDOWS_SPACE_SAFE_CYGWIN, wslpath does not support 8.3 names.  But perhaps the symlink workaround is acceptable for now and we can handle the 8.3 naming on WSL in a separate change, what do you guys think - personally I think what we have (assuming Cygwin still works) is at least a MVP for WSL devs.  Anyways, at least some people may have to use the symlink workaround if they've disabled 8.3 on NTFS.
That's too bad, since it really helped with getting around the issue 
with spaces in path that's mandatory on Windows using default 
installation of Visual Studio. :(

Again, sorry if I don't know enough about WSL to know if this is 
possible, but on msys we do the following:
     new_path=`cmd /c "for %A in (\"$input_path\") do @echo %~sA"|$TR 
\\\\\\\\ / | $TR 'ABCDEFGHIJKLMNOPQRSTUVWXYZ' 'abcdefghijklmnopqrstuvwxyz'`

That is, we call the Windows cmd.exe using the "%~sA" variable syntax to 
print the 8.3 version of the path (input_path is a "normal" Windows 
path). Is there any way it's possible to do this on WSL? It seems 
reasonable that you should be able to call cmd.exe and redirect the output.

I think it will be worth trying to jump through some loops or doing some 
dirty tricks to get this to work, because everything will be *soooo* 
much simpler if you can reliably turn paths into space-safe paths; our 
normal Windows build depends on it.

/Magnus
>
> Attaching my latest patch (generated using powershell, so to properly import you may need to convert UTF16 -> UTF8 and change CRLF to just LF, hg tends to be picky)...
>
> Thanks,
>
> -Andrew
>
>
> -----Original Message-----
> From: Erik Joelsson <erik.joelsson at oracle.com>
> Sent: Monday, December 10, 2018 9:19 AM
> To: Magnus Ihse Bursie <magnus.ihse.bursie at oracle.com>; Andrew Luo <andrewluotechnologies at outlook.com>; build-dev at openjdk.java.net
> Subject: Re: [PATCH] Support for building using WSL (Windows Subsystem for Linux) on Windows
>
> Hello,
>
> On 2018-12-10 02:06, Magnus Ihse Bursie wrote:
>> On 2018-12-09 20:11, Andrew Luo wrote:
>>> One important thing to note is that the WSL build targets Windows. It
>>> is also possible to use WSL to target itself (a WSL Linux binary) or
>>> even other distributions of Linux.  I have not implemented that yet,
>>> but I think I could do that as a next step if you guys think it would
>>> be useful (at least I think it would be useful, then you can test
>>> your changes for both Windows and Linux on one system...).
>> I think if you just run configure ordinarily, it will behave like a
>> Linux system and build the Linux image right out-of-the-box..? But
>> then again, maybe that behavior is negated by your changes to
>> config.guess and platform.m4. So maybe we need a flag to configure to
>> control this...
> It is indeed possible to build a pure Linux binary in WSL today so I think it would be bad to lose that functionality. We certainly need a configure flag to control if a Windows or Linux build should be produced in this case. This is something I have been thinking about when I started tackling WSL builds some time ago but didn't really come up with a good solution. I didn't have the time to spend to really see it through though, so it's nice to see that someone else is trying.
>
> We could simply use the --with-openjdk-target, that would perhaps be the cleanest, but it's also a bit cumbersome. We may need some simplification similar to how we have --with-target-bits=32/64 as a simple switch (e.g. --with-wsl-target=linux/windows?).
>
>>> Steps in case you want to try this out:
>>>
>>>
>>> 1.       Due to autotools not handling spaces well, you have to
>>> create symlinks in Windows that will allow you to access Windows Kits
>>> and the VC++ compiler without spaces in the path:
>>>
>>> mklink /D C:\VS "C:\Program Files (x86)\Microsoft Visual Studio"
>>>
>>> mklink /D C:\WindowsKits "C:\Program Files (x86)\Windows Kits"
>> That's a bit odd. We encounter spaces in paths on Windows normally on
>> cygwin and msys, and that works fine. I suspect there is something
>> missing with the rewriting functions. What we do, is that we rewrite
>> paths with spaces to paths without spaces, by using the old 8+3
>> compatibility names, so we get something like
>> "/cygdrive/c/progra~1/microso~2" from "C:\Program Files
>> (x86)\Microsoft Visual Studio". Have a look at
>> BASIC_MAKE_WINDOWS_SPACE_SAFE_CYGWIN. I think you need a WSL version
>> of that, as well as of BASIC_FIXUP_PATH_CYGWIN. (And you need to call
>> the BASIC_FIXUP_PATH_WSL from BASIC_FIXUP_PATH.)
>>
>> If you get these parts right, I don't think you will need any of the
>> special instructions below to build. In fact, as long as C:\... is
>> properly remapped, the normal VS autodetect code should work just
>> fine. And perhaps you can even revert some of the scarier changes in
>> toolchain_windows.m4.
>>
> I definitely agree with Magnus that to make WSL truly supported, the
> path handling macros need to be replicated. I'm not sure how to solve it
> properly. The root path Magnus is asking for is not defined in WSL. In
> fact, from windows you cannot reach any path in the WSL filesystem. Only
> Windows drives are mounted in WSL, not the other way around. To convert
> to old style paths in Cygwin we rely on the cygpath utility. There is a
> wslpath utility but does it support old style path conversions? If not,
> maybe it's possible to write such a tool in CMD/PowerShell?
>
> /Erik
>
>
>>> 2.       wsl must be started from a Windows Developer command
>>> prompt.  To ensure the correct environment variables are propagated
>>> from Windows to WSL, you can run the following commands:
>>>
>>> set WSLENV=INCLUDE/l:LIBPATH/l
>>>
>>> 3.       Start wsl (bash):
>>>
>>> wsl
>>>
>>> 4.       After starting bash you must set your compiler variables to
>>> explicitly point to the correct tools:
>>>
>>> export
>>> AR=/mnt/c/VS/2017/Enterprise/VC/Tools/MSVC/14.16.27023/bin/Hostx64/x64/lib.exe
>>>
>>> export
>>> CC=/mnt/c/VS/2017/Enterprise/VC/Tools/MSVC/14.16.27023/bin/Hostx64/x64/cl.exe
>>>
>>> export
>>> CXX=/mnt/c/VS/2017/Enterprise/VC/Tools/MSVC/14.16.27023/bin/Hostx64/x64/cl.exe
>>>
>>> export
>>> LD=/mnt/c/VS/2017/Enterprise/VC/Tools/MSVC/14.16.27023/bin/Hostx64/x64/link.exe
>>>
>>> export RC=/mnt/c/WindowsKits/10/bin/10.0.17763.0/x64/rc.exe
>>>
>>> export MT=/mnt/c/WindowsKits/10/bin/10.0.17763.0/x64/mt.exe
>>>
>>> export
>>> DUMPBIN=/mnt/c/VS/2017/Enterprise/VC/Tools/MSVC/14.16.27023/bin/Hostx64/x64/dumpbin.exe
>>>
>>> 5.       Run configure:
>>>
>>> ./configure
>>> --with-boot-jdk=/mnt/c/Users/Andrew/Downloads/openjdk-11.0.1_windows-x64_bin/jdk-11.0.1
>>> --with-tools-dir="C:\VS\2017\Enterprise\VC\Auxiliary"
>>> --with-ucrt-dll-dir="/mnt/c/WindowsKits/10/Redist/ucrt/DLLs/x64"
>>>
>>> 6.       Run make
>>>
>>> I've tested make with the default target as well as "make images"
>>>
>>> Let me know if you have any feedback/comments.
>>>
>>> Thanks,
>>>
>>> -Andrew
>>>