[11u] RFC: Update Graal in OpenJDK 11 Updates?

[Doug Simon]

Andrew,

That is a good summary. Indeed, libgraal is only a compilation time and resource optimization and thus not necessary for GraalVM.

Note that the JDK11 based Travis gates run on openjdk11 <https://github.com/oracle/graal/blob/master/.travis.yml#L17> so we should recognize any regression of the GraalVM compiler there (as we did recently <https://github.com/oracle/graal/pull/2278#issuecomment-602513391>). That said, also note that the Travis gate does not include any native image tests. That would require use of open-labsjdk-11 for access to the static libs.

It’s also worth pointing out that we’re also working with the HotSpot team at Oracle to ensure any JVMCI change we’re pushing into 11 also goes into JDK 15 where relevant.

-Doug

> On 23 Mar 2020, at 17:19, Andrew Dinn <adinn at redhat.com> wrote:
> 
> Hi Doug,
> 
> Ok, so let me summarize and please correct me if I misrepresent the case:
> 
> 1) The first patch is critical to the ability to use jdk11u to build
> GraalVM and successfully use the resulting build products.
> 
> 2) The second patch is essentially an enhancement to performance of
> JVMCI-based compilation on OpenJDK jdk11u. Not having it will impinge on
> performance (but not change functionality) both for OpenJDK-internal use
> e.g. jaotc or Graal as C2 replacement and also when using OpenJDK jdk11u
> to build GraalVM.
> 
> 3) You plan to continue to backport upstream changes to the the labs
> jdk11u in order to be able to support ongoing development in GraalVM.
> 
> n.b. I classify the 2nd patch as only having performance implications
> because, as I understand it, the point of the libgraal update is simply
> to allow the GraalVM JIT to execute as precompiled native code with its
> own managed memory pool rather than having to be interpreted then JIT
> compile itself and employ the JVM heap for compiler data. i.e. this
> change has no effect on what functionality is supported.
> 
> If that is correct then:
> 
> 1) I think it would probably be wise to include the first patch into
> jdk11u, modulo suitable review.
> 
> 2) It doesn't sound to me like the second patch really needs to be
> backported into jdk11u. jdk13+ and the jdk dev tree head already support
> the the use of libgraal for those who need to use it.
> 
> 3) As regards any further changes that might be needed to retain the
> ability to buidl GraalVM, my view is that I don't think the jdk11u
> project can or ought to offer a blank check regarding further updates to
> JVMCI that imply hotspot changes (i.e. ones that will have to be met by
> backporting rather than by overlaying the GraalVM code).
> 
> Of course, any further changes the GraalVM project would like to be
> backported could and, I think, should still be considered on their
> merits. However, in my view the project's priority has to be continuity
> of operation, reliability and security. That said, it's certainly not
> for me to make any final decision here. I am sure the project lead and
> maintainers an, indeed, other updates project members will wish to
> follow up on this point.
> 
> regards,
> 
> 
> Andrew Dinn
> -----------
> Senior Principal Software Engineer
> Red Hat UK Ltd
> Registered in England and Wales under Company Registration No. 03798903
> Directors: Michael Cunningham, Michael ("Mike") O'Neill
> 
> On 23/03/2020 15:03, Doug Simon wrote:
>> 
>> 
>>> On 23 Mar 2020, at 15:29, Andrew Dinn <adinn at redhat.com> wrote:
>>> 
>>> Hi Doug,
>>> 
>>> On 23/03/2020 13:36, Doug Simon wrote:
>>>> Sorry, this thread is long and has mentioned a number of patches.
>>>> Which ones are you referring to here?
>>> 
>>> The specific changes in question were derived from the discussion in
>>> github/graal issue #2196. The patches are detailed in the 2 webrevs
>>> Christoph posted:
>>> 
>>> http://cr.openjdk.java.net/~clanger/webrevs/8208686.11u/
>>> 
>>> and
>>> 
>>> http://cr.openjdk.java.net/~clanger/webrevs/graal13.11u/
>>> 
>>> The first webrev is the fix for JDK-8208686. The second one is
>>> essentially the bundled update that Tom pushed to the labs version of
>>> jkd11u after Christoph tweaked ti to make it apply and build correctly.
>>> 
>>>> I’m having trouble parsing that question. Are you asking “is there
>>>> anything in GitHub Graal that depends on the JVMCI changes in
>>>> labs-openjdk-11”?
>>> 
>>> I am asking two things:
>>> 
>>> 1) Is there anything in the current GitHub Graal repo that critically
>>> depends on the above changes getting into jdk11u?
>> 
>> JDK-8208686 fixes a JCK failure so is critical if you want to pass the JCK, especially with jaotc.
>> 
>> However, the general answer depends on the goal. If you want to be able to build and use libgraal with GitHub Graal + jdk11u, then the bundled update is necessary.
>> 
>>> 
>>> 2) Is there anything planned for addition to the GitHub Graal repo that
>>> critically depends on the above changes getting into jdk11u?
>> 
>> Once again, it depends on what the goal is. Since GraalVM is built on labs-openjdk-11, the extent to which jdk11u matches open-labsjdk-11 determines the extent to which the same GraalVM can be built in jdk11u.
>> 
>>> In other words why do this backport instead of just use the current jdk11u?
>> 
>> Because jdk11u misses all the JVMCI implementation changes needed for libgraal.
>> 
>>> 
>>>> I believe Tom has previously provided a breakdown of the patches in
>>>> labs-openjdk-11. Everything in labs-openjdk-11 is essential for
>>>> supporting complete GraalVM functionality as far as I understand
>>>> otherwise it would not be there.
>>> 
>>> First, I think all the patches mentioned in that list that involve
>>> hotspot changes have been backported except for the ones listed above --
>>> which is why we are currently able to build GraalVM using the latest
>>> OpenJDK jdkd11u.
>> 
>> Yes but it’s a GraalVM missing libgraal.
>> 
>>> Now, I'm not suggesting that there is any /redundancy/ as regards these
>>> remaining patches. However, what we would like to know is why we need to
>>> backport them i.e. what problem do they resolve and what would be the
>>> upshot if we stop further backports where we are now.
>>> 
>>> The existing jdk11u code appears to work to build Graal VM. It also
>>> works perfectly well to build and run the in-tree Graal code. So, by
>>> 'upshot' above I mean what would it imply for the ability to continue
>>> using jdk11u to build Graal if we don't backport the above changes? This
>>> is one side of a benefits equation we need to assess.
>> 
>> Again, libgraal is the main difference.
>> 
>>>> I think the high order point is that porting the changes in
>>>> labs-openjdk-11 to jdk11u is going to be complex due the way we
>>>> back-ported changes from JDK 13. With more time, we would have tried
>>>> to be more principled about it but on the other hand we were keen to
>>>> release GraalVM on JDK 11 given how much demand there was for it.
>>> 
>>> Yes, I had noticed that ;-). Also, I fully understand why a mega-patch
>>> was chosen at the time (especially since Red Hat were responsible for a
>>> lot of the demand you mention). I'm really trying to assess the degree
>>> to which we can justify bringing jdk11u up to date with the jdk13u base
>>> level.
>>> 
>>>> It might help to have a Zoom call to discuss this further. I want Tom
>>>> to be on the call so some time between 18:00 - 21:00 CET would
>>>> probably work. I can do any day except Tuesday.
>>> 
>>> Well, we normally try very hard to keep discussions like this in public
>>> so that they are visible to all who might have an interest and also on
>>> record. It's not really good open source practice to do otherwise.
>>> However, I'd be happy to try to use the tech to help expedite resolution
>>> of this issue -- so long as we report back ideas, suggestions and
>>> conclusions here to all interested parties. That's also assuming that
>>> others are happy to join in. Perhaps anyone interested in taking up your
>>> offer could respond and then we can take arrangement of a Zoom call offline.
>> 
>> FWIW, I’d be happy to record and publish a Zoom call. My only motivation for that option is that we can probably get to a resolution a lot faster. But for now, we can keep discussing on this thread.
>> 
>> -Doug
>> 
>>> Andrew Dinn
>>> -----------
>>> Senior Principal Software Engineer
>>> Red Hat UK Ltd
>>> Registered in England and Wales under Company Registration No. 03798903
>>> Directors: Michael Cunningham, Michael ("Mike") O'Neill
>>> 
>> 
>