[aarch64-port-dev ] OpenJDK extension to AArch64 and Windows

[Magnus Ihse Bursie]

On 2020-06-30 01:20, Ludovic Henry wrote:
> Hi Magnus,
>
>> I have now looked a bit more closely at the code. This is what I have
>> found so far that attracted my eye. Please note that this is not a
>> complete review. When you have a JEP and a test plan for how to verify
>> these changes and make sure you do not break existing platforms, you can
>> post a new RFR and I'll do a full review.
> Understood. I'll answer some of your remarks here, taking into account that Monica is currently working on the JEP.
>
>> * In flags-cflags.m4: You don't have to set  $1_CFLAGS_CPU_JVM="", an
>> empty value is default for unspecified variables.
> I'll revert that.
>
>> * In flags-ldflags.m4: The stack size seems dependent on CPU_BITS, not
>> the CPU arch. Please break out the -stack argument setting. Also, have
>> you verified if -machine is really needed? The comment says that it's
>> probably not; if it's just an old, unnecessary, precaution, we should
>> probably remove it instead, to simplify the logic.
> I've verified that it works without the `-machine` bit for ARM64, I'll revert it. I'll also split the `-stack` arguments into 32 and 64 bits checks.
Great! If you can test if -machine can be removed for the other 
platforms as well, I'd be grateful. Old code which adds special cases, 
and besides that is flagged as "probably not needed", should really be 
cleaned away.
>
>> * In platform.m4: These changes worries me. Neither of them were
>> necessary for the linux-aarch64 port. But now you are changing the
>> values for all aarch64 builds, not just windows-aarch64. Have you
>> discovered a bug for linux-aarch64? Otherwise, these changes looks like
>> they are going to break linux-aarch64. If you believe you need to modify
>> these legacy values (which we'd rather move away from), please see if
>> you have made changes elsewhere that can be resolved without resorting
>> to adding new identifiers to the legacy values.
> It seems to have been a mistake in the early stage of our port, I'll revert it.
I see. Good.
>
>> * In basic.m4: Please move BASIC_EVAL_BUILD_DEVKIT_VARIABLE to be
>> positioned next to BASIC_EVAL_DEVKIT_VARIABLE.
> Ok
>
>> * In toolchain_windows.m4:
>>    In TOOLCHAIN_CHECK_POSSIBLE_MSVC_DLL, if you know about the upcoming
>> changes to file, why don't you add both the old and the new pattern to
>> the test?
> The comment is slightly inaccurate as it won't break, it is just not going to be as precise as it could be. The future version of file(1) will still return `PE32+ executable` (like it does for x86_64 executables), but it will also return something specifically for aarch64.
I see. Since we will need to support old versions of file for a long 
time to come, the upcoming fix is really not relevant for us then. We'll 
just have to accept the fact that the check is not really useful for 
aarch64. Perhaps you can adjust the wording of the comment, so it's 
clear that it's irrelevant if the bug fix is present or not in the 
version of file that the user has.
>
>>    In TOOLCHAIN_SETUP_MSVC_DLL, when it was just two instances, the code
>> duplication could be accepted, but with three instances you need to
>> generalize this and refactor out the changing platform part of the path
>> only to the if statement. This applies, with some variation, to all four
>> changed places.
> Ok, I'll refactor by adding a variable containing x86, x64 or arm64.
Perfect!
>
>>    In the new TOOLCHAIN_SETUP_VISUAL_STUDIO_SYSROOT_FLAGS, you are doing
>>     AC_SUBST($1SYSROOT_CFLAGS)
>>     AC_SUBST($1SYSROOT_LDFLAGS)
>>
>> However, that seems superfluous, since it is already done by
>> FLAGS_SETUP_SYSROOT_FLAGS in flags.m4. In fact, now that I checked this,
>> I spotted that we *already* do an unnecessary AC_SUBST in
>> FLAGS_POST_TOOLCHAIN for the build sysroot flags! :-o
> This part is what caused the most problems in getting cross-compilation with MSVC working, and the current solution is the simplest changeset I could come up with. I've to admit that this part stretched my understanding of the build system, and thus I'm not even sure what I propose as good as it can be.
>
> FLAGS_SETUP_SYSROOT_FLAGS for example, only sets $1SYSROOT_CFLAGS and $1SYSROOT_LDLAGS to a valid value on non-"microsoft" toolchain. That is because at the time FLAGS_PRE_TOOLCHAIN is called, the "microsoft" toolchain has not been initialized yet. It only gets properly initialized once TOOLCHAIN_SETUP_VISUAL_STUDIO_ENV is called.
I understand if it's confusing. Autoconf is a beast to work with; I've 
done that for more years than I want to think about, and I still don't 
understand it fully. However, in this case, it's "easy". The call to 
AC_SUBST only flags a variable for export. The actual exportation to 
spec.gmk is done at the end of the configure script, with the call to 
AC_OUTPUT, and the value exported is the value the variables has at that 
point. So the order of execution does not matter here. This also means 
that it is not an error to call AC_SUBST multiple times, but it is a 
style issue. On the other hand, we try to follow the pattern of "create 
variable, call AC_SUBST" for our own sanity.

Conceptually, the microsoft toolchain setup is almost equivalent to 
FLAGS_PRE_TOOLCHAIN. That is, the result should be that we have a set of 
"sysroot" (think "global") flags that should always be added. I have a 
patch in the works that changes this somewhat, and makes the 
dependencies clearer.
>
>> * In GensrcMisc,gmk: You are changing this for all users of the
>> microsoft toolchain. I don't recall seeing any problems with this on
>> x64. What version of Visual Studio are you using? Is this a limitation
>> in the aarch64 version of CL.EXE, or does it apply to other platforms as
>> well? Finally, if we do need to keep it, please use "-" as prefix for
>> options (even though the microsoft tooling normally suggests the
>> non-standard "/" -- this can all too easily be confused with path names.)
> Ok.
>
>> * In GensrcAdlc.gmk: You are adding -D_WIN64=1 for all 64-bit platforms,
>> i.e. also for x64, which has apparently worked fine until now without
>> that define. What does the define do, and what is the rationale for not
>> only adding it on your platform?
> It is the default define used on Windows-compatible codebases to specify whether it's targetting a 64-bit platform or not. We need to define it to have Windows-specific code in the existing aarch64 code of the OpenJDK. It makes sense to define it for x64 as well as arm64, but I agree it's not a requirement.
Ok, sounds good. Keep it as it is then.
>
>> * In jib-profiles.js: I appreciate the effort, but this file is
>> basically just for Oracle-internal usage. If and when we will add
>> support for building on windows-aarch64, we can update the file to work
>> properly with the JIB tool.
> Ok.
>
>> I am impressed that you manage to get cross-compilation working for
>> Windows with that small amount of changes, though! If you had asked med
>> beforehand, I'd have guessed that it would require more substantial
>> changes. As you say, this is not something we have done before.
> The meat of the change is in `toolchain.m4` with the support functions in `toolchain_windows.m4`. It is where I iterated the most to find the right encapsulation across the different components relying on it - and I've to say it wasn't straightforward to understand how all the pieces fit together. I'd be happy to work out with you a better way to do the foundational work of supporting cross-compilation with the Microsoft toolchain, just le me know how you'd like to proceed.
I think you have found more or less exactly the right spots to inject 
the support for cross-compilation. I think I'd ended up with something 
similar if I was tasked with getting cross-compilation work on Windows. 
Maybe there's room for more generalisations, but it's a bit hard to say 
at this moment. I think things might clear up a bit more when I get the 
"unified winenv" patch in. It makes the sysroot cflag handling a bit 
more transparent.

/Magnus
>
> Thank you,
>
> --
> Ludovic
>
> ________________________________________
> From: Magnus Ihse Bursie <magnus.ihse.bursie at oracle.com>
> Sent: Monday, June 29, 2020 07:12
> To: Ludovic Henry; Monica Beckwith; hotspot-runtime-dev at openjdk.java.net; aarch64-port-dev at openjdk.java.net; build-dev
> Cc: openjdk-aarch64
> Subject: Re: OpenJDK extension to AArch64 and Windows
>
> I have now looked a bit more closely at the code. This is what I have
> found so far that attracted my eye. Please note that this is not a
> complete review. When you have a JEP and a test plan for how to verify
> these changes and make sure you do not break existing platforms, you can
> post a new RFR and I'll do a full review.
>
> * In flags-cflags.m4: You don't have to set  $1_CFLAGS_CPU_JVM="", an
> empty value is default for unspecified variables.
>
> * In flags-ldflags.m4: The stack size seems dependent on CPU_BITS, not
> the CPU arch. Please break out the -stack argument setting. Also, have
> you verified if -machine is really needed? The comment says that it's
> probably not; if it's just an old, unnecessary, precaution, we should
> probably remove it instead, to simplify the logic.
>
> * In platform.m4: These changes worries me. Neither of them were
> necessary for the linux-aarch64 port. But now you are changing the
> values for all aarch64 builds, not just windows-aarch64. Have you
> discovered a bug for linux-aarch64? Otherwise, these changes looks like
> they are going to break linux-aarch64. If you believe you need to modify
> these legacy values (which we'd rather move away from), please see if
> you have made changes elsewhere that can be resolved without resorting
> to adding new identifiers to the legacy values.
>
> * In basic.m4: Please move BASIC_EVAL_BUILD_DEVKIT_VARIABLE to be
> positioned next to BASIC_EVAL_DEVKIT_VARIABLE.
>
> * In toolchain_windows.m4:
>    In TOOLCHAIN_CHECK_POSSIBLE_MSVC_DLL, if you know about the upcoming
> changes to file, why don't you add both the old and the new pattern to
> the test?
>
>    In TOOLCHAIN_SETUP_MSVC_DLL, when it was just two instances, the code
> duplication could be accepted, but with three instances you need to
> generalize this and refactor out the changing platform part of the path
> only to the if statement. This applies, with some variation, to all four
> changed places.
>
>    In the new TOOLCHAIN_SETUP_VISUAL_STUDIO_SYSROOT_FLAGS, you are doing
>     AC_SUBST($1SYSROOT_CFLAGS)
>     AC_SUBST($1SYSROOT_LDFLAGS)
>
> However, that seems superfluous, since it is already done by
> FLAGS_SETUP_SYSROOT_FLAGS in flags.m4. In fact, now that I checked this,
> I spotted that we *already* do an unnecessary AC_SUBST in
> FLAGS_POST_TOOLCHAIN for the build sysroot flags! :-o
>
> * In GensrcMisc,gmk: You are changing this for all users of the
> microsoft toolchain. I don't recall seeing any problems with this on
> x64. What version of Visual Studio are you using? Is this a limitation
> in the aarch64 version of CL.EXE, or does it apply to other platforms as
> well? Finally, if we do need to keep it, please use "-" as prefix for
> options (even though the microsoft tooling normally suggests the
> non-standard "/" -- this can all too easily be confused with path names.)
>
> * In GensrcAdlc.gmk: You are adding -D_WIN64=1 for all 64-bit platforms,
> i.e. also for x64, which has apparently worked fine until now without
> that define. What does the define do, and what is the rationale for not
> only adding it on your platform?
>
> * In jib-profiles.js: I appreciate the effort, but this file is
> basically just for Oracle-internal usage. If and when we will add
> support for building on windows-aarch64, we can update the file to work
> properly with the JIB tool.
>
> I am impressed that you manage to get cross-compilation working for
> Windows with that small amount of changes, though! If you had asked med
> beforehand, I'd have guessed that it would require more substantial
> changes. As you say, this is not something we have done before.
>
> /Magnus
>
>
>
> On 2020-06-24 23:33, Ludovic Henry wrote:
>> Hi Magnus,
>>
>> Happy to answer any question you have on the build system changes.
>>
>> A lot of the changes were due to the build system not supporting cross-compilation well when targetting the microsoft toolchain (it just never really had to support it). We had to go through a few hoops to remove as many of our own quick hacks, initially there just to get things building - like hardcoding the target being windows-aarch64 for example.
>>
>> Thank you for your review,
>>
>> --
>> Ludovic
>>
>> ________________________________________
>> From: Magnus Ihse Bursie <magnus.ihse.bursie at oracle.com>
>> Sent: Wednesday, June 24, 2020 13:44
>> To: Monica Beckwith; hotspot-runtime-dev at openjdk.java.net; aarch64-port-dev at openjdk.java.net; build-dev
>> Cc: openjdk-aarch64
>> Subject: Re: OpenJDK extension to AArch64 and Windows
>>
>> Hi Monica,
>>
>> All build system changes must be sent to build-dev for review by the
>> build team, and you are doing quite a lot of build changes. (I'm cc:ing
>> build-dev now.)
>>
>> I did a quick scan and found some changes that looked odd enough to draw
>> my attention.
>>
>> I will need some time to fully understand what you are trying to
>> accomplish here, before I can give a full review.
>>
>> /Magnus
>>
>>
>> On 2020-06-24 18:40, Monica Beckwith wrote:
>>> Hello OpenJDK community,
>>>
>>> As the project lead here @Microsoft, I am pleased to share that we have been working towards a Windows addition to the OpenJDK AArch64 port. We are very thankful to all that have contributed to the Linux+aarch64 and Windows+x86-64. Both these codebases came to our rescue on numerous occasions.
>>>
>>> Support status: We have successfully ported C2 and can build the server release (cross-compiled environment)
>>> Test coverage: C2 + ParallelGC (No AOT, JVMCI, ZGC, ShenandoahGC, G1GC)
>>> Tests and benchmarks covered [1]: JTReg [2], JCStress, jmh-jdk-microbenchmarks, SPEC SERT, SPECJBB2015 [3], SPEC JVM2008, Scimark2, SPEC JBB2005.
>>> Umbrella Bug ID: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.openjdk.java.net%2Fbrowse%2FJDK-8248238&data=02%7C01%7Cluhenry%40microsoft.com%7Cebbae1ef103e4cb5b2cf08d81c36ccca%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637290368978180090&sdata=ntF3w0wtSa2%2BbN9ikZeJc3OZTYgWsjbvQxqudBSa%2B4I%3D&reserved=0
>>> Webrevs:
>>> `Webrev P1`: https://nam06.safelinks.protection.outlook.com/?url=http:%2F%2Fcr.openjdk.java.net%2F~burban%2Fwinarm64_p1_llp64%2F&data=02%7C01%7Cluhenry%40microsoft.com%7Cebbae1ef103e4cb5b2cf08d81c36ccca%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637290368978180090&sdata=PG7oj%2FDCporqoF96ocIxidkAlTOJYs6gZwfGRWDnhYE%3D&reserved=0  &
>>> `Webrev P2`: https://nam06.safelinks.protection.outlook.com/?url=http:%2F%2Fcr.openjdk.java.net%2F~burban%2Fwinarm64_p2_new-target%2F&data=02%7C01%7Cluhenry%40microsoft.com%7Cebbae1ef103e4cb5b2cf08d81c36ccca%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637290368978190084&sdata=Br9TkpMvgiQyN1xop6IE2CA%2BZCqIVDpEzw%2Bx6XnZrmg%3D&reserved=0
>>>
>>> The first patch `Webrev P1` (patch 1 aka P1 in our tests) helps integrate support for Windows (LLP64) on Linux + AArch64
>>> The second patch `Webrev P2` (patch 2 aka P2 in our tests) adds the 'windows-aarch64' support in `os_cpu`. We also had to modify shared code, and I am highlighting a few details here:
>>>         * In windows_x86 such as the `get_frame_at_stack_banging_point` in `os_windows_x86.cpp`,
>>>         * In `os/windows os_windows.cpp` to make it aware of Windows + Arm64
>>>         * `os/windows` in `threadCritical_windows.cpp`,
>>>         * Windbg support
>>>         * `globalDefinitions_visCPP.hpp` in `share/utilities`
>>>         * We also added Vectored Exception Handling (VEH) to P2, as it is a requirement on Windows + Arm64 (due to ABI specifications).
>>> Also, in `Webrev P2`, you will find that we have made some significant changes to `cpu/aarch64` around register usage since on Windows + Arm64, register R18 points to TEB [4]. We have discussed this with Andrew Haley and Andrew Dinn, and they are helping us with a cleaner implementation of the same. Their constant support and guidance have humbled me.
>>>
>>> I'd also like to recognize the great work done by Ludovic Henry (our resident runtime expert) in driving the C2 support and by Bernhard Urban-Forster, (who recently joined our team) in helping expand the coverage to G1 GC.
>>>
>>> As a part of this project, we have also worked on two additional patches:
>>>         * Expanding VEH on Windows to x86-64 (patch 3 aka P3 in our tests). Details here: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.openjdk.java.net%2Fbrowse%2FJDK-8247941&data=02%7C01%7Cluhenry%40microsoft.com%7Cebbae1ef103e4cb5b2cf08d81c36ccca%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637290368978190084&sdata=95tmUIxEmDJP0AZ2GpxhCkII7SefU4%2BRRkkae0fRdRA%3D&reserved=0
>>>         * Improvements in the shared cross-platform code on Windows (patch 4 aka P4 in our tests) - We will send out a separate patch soon.
>>>
>>> We welcome any feedback and comments from the community and are looking forward to working together.
>>>
>>> Regards,
>>> Monica
>>>
>>> [1] https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fopenjdk-aarch64%2Fblob%2Fmaster%2Fwkload-status-on-Win%252BArm64.md&data=02%7C01%7Cluhenry%40microsoft.com%7Cebbae1ef103e4cb5b2cf08d81c36ccca%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637290368978190084&sdata=RSeWSplQ%2F7t%2Fq8aaZwjVolXBCWsp0dNfebeDddjBvUM%3D&reserved=0
>>> [2] https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fopenjdk-aarch64%2Fblob%2Fmaster%2FJTRegtests.md&data=02%7C01%7Cluhenry%40microsoft.com%7Cebbae1ef103e4cb5b2cf08d81c36ccca%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637290368978190084&sdata=6q37AVqwlxpXl%2BV0nTLp6h24n3AuuEmbQod5tKnwrhs%3D&reserved=0
>>> [3] https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fopenjdk-aarch64%2Fblob%2Fmaster%2FSPECJBB2015-test-matrices.md&data=02%7C01%7Cluhenry%40microsoft.com%7Cebbae1ef103e4cb5b2cf08d81c36ccca%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637290368978190084&sdata=AXkooXGVJMqzeBn29ZUeabrW8mzfFfJrFcoHkfRSVMQ%3D&reserved=0
>>> [4] https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fcpp%2Fbuild%2Farm64-windows-abi-conventions%3Fview%3Dvs-2019%23integer-registers&data=02%7C01%7Cluhenry%40microsoft.com%7Cebbae1ef103e4cb5b2cf08d81c36ccca%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637290368978190084&sdata=aJN9ocV4DF0d3bKP74E%2FkoNgq336BTXd8%2BQPgJzKDaU%3D&reserved=0