[PATCH] Support for building using WSL (Windows Subsystem for Linux) on Windows

Wed Dec 12 17:30:01 UTC 2018

Hello,

I had the same trouble you describe trying to call cmd to create a short 
path from WSL with an inline script. I managed to it working by creating 
a script file like this:

shortName.cmd:

---
@ECHO OFF
if '%1' NEQ '' echo %~s1
---

$ /mnt/c/Windows/System32/cmd.exe /c shortName.cmd "C:\Program Files"
C:\PROGRA~1

We could put such a script in make/scripts.

/Erik

On 2018-12-11 22:44, Andrew Luo wrote:
> For the stdin/stdout redirection: basically, the issue was only occurring with those two executables.  Oddly enough, it would occur every time I tried (for SPP the output would be cutoff, presumably because the input was cutoff, and for the other executable, available0 would throw an exception consistently for System.in).  I have a hunch this is due to using WINAPI console functions for reading from a WSL shell, but I'm not 100% certain.  I plan to try to build a smaller sample that can reproduce the issue, and if it seems to be a Windows bug I will file a bug with Microsoft.
>
> As for the short paths, calling cmd.exe from WSL bash seems to be a bit buggy.  cmd.exe, when called from a standard Windows environment, works properly and prints the path, however, when called from WSL, I get:
>
> "(C:\Program Files (x86))" was unexpected at this time.
>
> I verified that the correct path was being passed to cmd.exe (not a bash escaping issue) by creating my own test executable (C++) that just prints argv[0] ... argv[argc - 1] and when running my text executable from both Windows and WSL I get the same result, however when running cmd.exe with the exact same arguments, it works in Windows but not WSL.  I will file a bug with Microsoft for this issue.
>
> Thanks,
>
> -Andrew
>
>
> -----Original Message-----
> From: Magnus Ihse Bursie <magnus.ihse.bursie at oracle.com>
> Sent: Tuesday, December 11, 2018 6:18 AM
> To: Andrew Luo <andrewluotechnologies at outlook.com>; Erik Joelsson <erik.joelsson at oracle.com>; build-dev at openjdk.java.net
> Subject: Re: [PATCH] Support for building using WSL (Windows Subsystem for Linux) on Windows
>
>
>
> On 2018-12-11 14:37, Magnus Ihse Bursie wrote:
>> On 2018-12-11 06:25, Andrew Luo wrote:
>>> Hi,
>>>
>>> Yes, I've signed an OCA (I've also contributed changes to other
>>> groups before, but not build).
>>>
>>> Okay, I have fixed the autconf-config* files.
>>>
>>> Unfortunately, as Erik mentioned, there is no (supported/reliable)
>>> way to access the WSL root / from /cygdrive/c, or even from Windows
>>> (there is a way in reality, however, it's not documented/supported by
>>> Microsoft and the location changes depending on the
>>> distribution/store app id/etc. so best to avoid using it.)  I can see
>>> if we can print information about versions however.
>>>
>>> Right, WSL requires the .exe extension when accessing an executable,
>>> as this is Linux behavior (Linux doesn't have extensions for
>>> executables generally, but that's besides the point)...
>>>
>>> I fixed BASIC_REMOVE_SYMBOLIC_LINKS - a leftover from another
>>> approach I tried.
>>>
>>> For the redirect, redirect doesn't seem to be working when you have a
>>> bash shell input piped into a Win32 executable reading from stdin
>>> using WINAPI.  I'm not sure this is supported by the OpenJDK, more
>>> likely it might be a Microsoft issue.  For some reason, the stdin
>>> would be cut off (or I would see an exception thrown from available0
>>> in FileInputStream).  I personally didn't see any harm in changing
>>> piping into input/output files (since all the inputs/outputs are
>>> files anyways!).
>> Ok, let me be sure I get this right. It is only the redirect of
>> *input* that fails? (But you fixed both because of consistency). I
>> agree that the change itself is fine, even better than it is right now
>> -- I was mostly worried about the consequences of redirects is not
>> working; there might be other places that fail. But if redirecting
>> output works, I think we're mostly fine. That's something we do all
>> the time, for each executed command, so if that did not work reliably
>> it would be really bad.
>>
>> But still... I tried greping for "<" and there's a lot of places, 20+,
>> that redirects input.
>>
>> Or did this problem only happen when running *java* as the recipient
>> of the redirected input?
>>
>> This worries me, and while I do think your change makes the tools have
>> a better UI, I don't like this as a workaround that will not solve all
>> potential problems. :(
>>
>>> I fixed TOOLCHAIN_FIND_VISUAL_STUDIO_BAT_FILE - this was also from a
>>> few things I had tried earlier.
>>>
>>> I disabled the $BASH code because to call bash from Win32 the correct
>>> way is either "wsl /bin/bash" or just "bash".  $BASH correctly
>>> evaluates to /bin/bash, however
>>> BASIC_WINDOWS_REWRITE_AS_WINDOWS_MIXED_PATH is implemented in terms
>>> of wslpath, which can only convert a path under /mnt/c back to a
>>> Windows path.  Other files under /, for example /bin and /bin/bash,
>>> cannot be converted to a Windwos path.
>>>
>>> The escaping changes I made because it wasn't working.  This does
>>> work with spaces in the path on WSL.  I don't have a Cygwin
>>> environment to check, perhaps someone else here could help out?
>>> Otherwise I can refactor that code to use that echo statement for WSL
>>> and use the old echo statement for Cygwin.
>> I can check it out the next time I'm on a Windows machine.
>>
>>> I have fixed the extraneous debug print statement.
>>>
>>> As for Windows vs Linux output - you can still force it to build a
>>> Linux output binary.  You just need to run configure as follows:
>>>
>>> ./configure --with-boot-jdk=/home/andrew/jdk-11.0.1
>>> ----build=x86_64-unknown-linux-gnu --host=x86_64-unknown-linux-gnu
>>>
>>> However, there is a behavior change: now, on WSL, by default, Windows
>>> binaries are targeted.  Previously, Linux binaries were the default
>>> target.  (Also, you can run configure twice and two sets of
>>> configurations will be generated, you can actually build both images
>>> by setting CONF=linux-x86_64-server-release or
>>> CONF=windows-x86_64-server-release)
>> If you run on WLS, it's reasonable that the default is Windows. The
>> --build --host combo is good enough for me as a way to force a linux
>> build; we don't need an extra flag for this somewhat odd build
>> configuration.
>>
>>> As for BASIC_MAKE_WINDOWS_SPACE_SAFE_CYGWIN, wslpath does not support
>>> 8.3 names.  But perhaps the symlink workaround is acceptable for now
>>> and we can handle the 8.3 naming on WSL in a separate change, what do
>>> you guys think - personally I think what we have (assuming Cygwin
>>> still works) is at least a MVP for WSL devs.  Anyways, at least some
>>> people may have to use the symlink workaround if they've disabled 8.3
>>> on NTFS.
>> That's too bad, since it really helped with getting around the issue
>> with spaces in path that's mandatory on Windows using default
>> installation of Visual Studio. :(
>>
>> Again, sorry if I don't know enough about WSL to know if this is
>> possible, but on msys we do the following:
>>      new_path=`cmd /c "for %A in (\"$input_path\") do @echo %~sA"|$TR
>> \\\\\\\\ / | $TR 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
>> 'abcdefghijklmnopqrstuvwxyz'`
>>
>> That is, we call the Windows cmd.exe using the "%~sA" variable syntax
>> to print the 8.3 version of the path (input_path is a "normal" Windows
>> path). Is there any way it's possible to do this on WSL? It seems
>> reasonable that you should be able to call cmd.exe and redirect the
>> output.
>>
>> I think it will be worth trying to jump through some loops or doing
>> some dirty tricks to get this to work, because everything will be
>> *soooo* much simpler if you can reliably turn paths into space-safe
>> paths; our normal Windows build depends on it.
> I also realized that if you make a WSL version of BASIC_FIXUP_EXECUTABLE, then you might be able to add .exe in it. You will still need the EXE_SUFFIX when e.g. looking for the java.exe binary in locating the Boot JDK, but perhaps it will simplify some of the places.
>
> I see now that the call to java in Images.gmk really should have been prefixed with $(FIXPATH) instead. Then you would not have needed the EXE_SUFFIX fix there.
>
> Also, I suggest you look more closely on how we do things on msys than on cygwin; I have the feeling wsl is closer to msys (in terms of functionality from our perspective) than cygwin.
>> /Magnus
>>> Attaching my latest patch (generated using powershell, so to properly
>>> import you may need to convert UTF16 -> UTF8 and change CRLF to just
>>> LF, hg tends to be picky)...
> Looks much better now!
>
> Before accepting it, I'd like to understand more about the redirect issue, and see if there really is no way to rewrite the paths in a space-safe manner. Then I think this is good to go.
>
> /Magnus
>
>>> Thanks,
>>>
>>> -Andrew
>>>
>>>
>>> -----Original Message-----
>>> From: Erik Joelsson <erik.joelsson at oracle.com>
>>> Sent: Monday, December 10, 2018 9:19 AM
>>> To: Magnus Ihse Bursie <magnus.ihse.bursie at oracle.com>; Andrew Luo
>>> <andrewluotechnologies at outlook.com>; build-dev at openjdk.java.net
>>> Subject: Re: [PATCH] Support for building using WSL (Windows
>>> Subsystem for Linux) on Windows
>>>
>>> Hello,
>>>
>>> On 2018-12-10 02:06, Magnus Ihse Bursie wrote:
>>>> On 2018-12-09 20:11, Andrew Luo wrote:
>>>>> One important thing to note is that the WSL build targets Windows.
>>>>> It is also possible to use WSL to target itself (a WSL Linux
>>>>> binary) or even other distributions of Linux.  I have not
>>>>> implemented that yet, but I think I could do that as a next step if
>>>>> you guys think it would be useful (at least I think it would be
>>>>> useful, then you can test your changes for both Windows and Linux on one system...).
>>>> I think if you just run configure ordinarily, it will behave like a
>>>> Linux system and build the Linux image right out-of-the-box..? But
>>>> then again, maybe that behavior is negated by your changes to
>>>> config.guess and platform.m4. So maybe we need a flag to configure
>>>> to control this...
>>> It is indeed possible to build a pure Linux binary in WSL today so I
>>> think it would be bad to lose that functionality. We certainly need a
>>> configure flag to control if a Windows or Linux build should be
>>> produced in this case. This is something I have been thinking about
>>> when I started tackling WSL builds some time ago but didn't really
>>> come up with a good solution. I didn't have the time to spend to
>>> really see it through though, so it's nice to see that someone else
>>> is trying.
>>>
>>> We could simply use the --with-openjdk-target, that would perhaps be
>>> the cleanest, but it's also a bit cumbersome. We may need some
>>> simplification similar to how we have --with-target-bits=32/64 as a
>>> simple switch (e.g. --with-wsl-target=linux/windows?).
>>>
>>>>> Steps in case you want to try this out:
>>>>>
>>>>>
>>>>> 1.       Due to autotools not handling spaces well, you have to
>>>>> create symlinks in Windows that will allow you to access Windows
>>>>> Kits and the VC++ compiler without spaces in the path:
>>>>>
>>>>> mklink /D C:\VS "C:\Program Files (x86)\Microsoft Visual Studio"
>>>>>
>>>>> mklink /D C:\WindowsKits "C:\Program Files (x86)\Windows Kits"
>>>> That's a bit odd. We encounter spaces in paths on Windows normally
>>>> on cygwin and msys, and that works fine. I suspect there is
>>>> something missing with the rewriting functions. What we do, is that
>>>> we rewrite paths with spaces to paths without spaces, by using the
>>>> old 8+3 compatibility names, so we get something like
>>>> "/cygdrive/c/progra~1/microso~2" from "C:\Program Files
>>>> (x86)\Microsoft Visual Studio". Have a look at
>>>> BASIC_MAKE_WINDOWS_SPACE_SAFE_CYGWIN. I think you need a WSL version
>>>> of that, as well as of BASIC_FIXUP_PATH_CYGWIN. (And you need to
>>>> call the BASIC_FIXUP_PATH_WSL from BASIC_FIXUP_PATH.)
>>>>
>>>> If you get these parts right, I don't think you will need any of the
>>>> special instructions below to build. In fact, as long as C:\... is
>>>> properly remapped, the normal VS autodetect code should work just
>>>> fine. And perhaps you can even revert some of the scarier changes in
>>>> toolchain_windows.m4.
>>>>
>>> I definitely agree with Magnus that to make WSL truly supported, the
>>> path handling macros need to be replicated. I'm not sure how to solve
>>> it properly. The root path Magnus is asking for is not defined in
>>> WSL. In fact, from windows you cannot reach any path in the WSL
>>> filesystem. Only Windows drives are mounted in WSL, not the other way
>>> around. To convert to old style paths in Cygwin we rely on the
>>> cygpath utility. There is a wslpath utility but does it support old
>>> style path conversions? If not, maybe it's possible to write such a tool in CMD/PowerShell?
>>>
>>> /Erik
>>>
>>>
>>>>> 2.       wsl must be started from a Windows Developer command
>>>>> prompt.  To ensure the correct environment variables are propagated
>>>>> from Windows to WSL, you can run the following commands:
>>>>>
>>>>> set WSLENV=INCLUDE/l:LIBPATH/l
>>>>>
>>>>> 3.       Start wsl (bash):
>>>>>
>>>>> wsl
>>>>>
>>>>> 4.       After starting bash you must set your compiler variables to
>>>>> explicitly point to the correct tools:
>>>>>
>>>>> export
>>>>> AR=/mnt/c/VS/2017/Enterprise/VC/Tools/MSVC/14.16.27023/bin/Hostx64/
>>>>> x64/lib.exe
>>>>>
>>>>>
>>>>> export
>>>>> CC=/mnt/c/VS/2017/Enterprise/VC/Tools/MSVC/14.16.27023/bin/Hostx64/
>>>>> x64/cl.exe
>>>>>
>>>>>
>>>>> export
>>>>> CXX=/mnt/c/VS/2017/Enterprise/VC/Tools/MSVC/14.16.27023/bin/Hostx64
>>>>> /x64/cl.exe
>>>>>
>>>>>
>>>>> export
>>>>> LD=/mnt/c/VS/2017/Enterprise/VC/Tools/MSVC/14.16.27023/bin/Hostx64/
>>>>> x64/link.exe
>>>>>
>>>>>
>>>>> export RC=/mnt/c/WindowsKits/10/bin/10.0.17763.0/x64/rc.exe
>>>>>
>>>>> export MT=/mnt/c/WindowsKits/10/bin/10.0.17763.0/x64/mt.exe
>>>>>
>>>>> export
>>>>> DUMPBIN=/mnt/c/VS/2017/Enterprise/VC/Tools/MSVC/14.16.27023/bin/Hos
>>>>> tx64/x64/dumpbin.exe
>>>>>
>>>>>
>>>>> 5.       Run configure:
>>>>>
>>>>> ./configure
>>>>> --with-boot-jdk=/mnt/c/Users/Andrew/Downloads/openjdk-11.0.1_window
>>>>> s-x64_bin/jdk-11.0.1
>>>>>
>>>>> --with-tools-dir="C:\VS\2017\Enterprise\VC\Auxiliary"
>>>>> --with-ucrt-dll-dir="/mnt/c/WindowsKits/10/Redist/ucrt/DLLs/x64"
>>>>>
>>>>> 6.       Run make
>>>>>
>>>>> I've tested make with the default target as well as "make images"
>>>>>
>>>>> Let me know if you have any feedback/comments.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> -Andrew
>>>>>