JDK11 & Polyglot

[Bob McWhirter]

And my "solution":

https://github.com/oracle/graal/pull/1326

On Thu, May 23, 2019 at 11:55 AM Bob McWhirter <bmcwhirt at redhat.com> wrote:

> Progress... maybe?
>
> In Module$ReflectionData
>
> /**
>  * A module (1st key) exports or opens a package to another module
>  * (2nd key). The map value is a map of package name to a boolean
>  * that indicates if the package is opened.
>  */
> static final WeakPairMap<Module, Module, Map<String, Boolean>> exports =
>     new WeakPairMap<>();
>
>
> Ultimately used by Module.isReflectivelyExportedOrOpen(...):
>
> private boolean isReflectivelyExportedOrOpen(String pn, Module other, boolean open) {
>     // exported or open to all modules
>     Map<String, Boolean> exports = ReflectionData.exports.get(this, EVERYONE_MODULE);
>
>
>
> Is the source of things-changing-while-writing-the-heap.
>
> Not sure how to fix, or if there is a way to whitelist/ignore this.
>
> I do have an inbound PR to push in a moment helping fix the debug reason
> trace, to avoid the "root=null" in the current master#HEAD.
>
> Caused by: com.oracle.svm.core.util.UserError$UserException: Static field
> or an object referenced from a static field changed during native image
> generation?
>   object:com.oracle.truffle.api=true  of class:
> java.util.concurrent.ConcurrentHashMap$Node
>   reachable through:
>     object: [Ljava.util.concurrent.ConcurrentHashMap$Node;@3cb34219  of
> class: java.util.concurrent.ConcurrentHashMap$Node[]
>     object: {com.oracle.truffle.api.dsl=true,
> com.oracle.truffle.api.library=true, com.oracle.truffle.api.nodes=true,
> com.oracle.truffle.api=true}  of class:
> java.util.concurrent.ConcurrentHashMap
>     object: java.lang.WeakPairMap$Pair$Weak at 248dd9c1={com.oracle.truffle.api.dsl=true,
> com.oracle.truffle.api.library=true, com.oracle.truffle.api.nodes=true,
> com.oracle.truffle.api=true}  of class:
> java.util.concurrent.ConcurrentHashMap$Node
>     object: java.lang.WeakPairMap$Pair$Weak at 40620d6e={jdk.internal.vm.annotation=false,
> jdk.internal=false, jdk.internal.reflect=false}  of class:
> java.util.concurrent.ConcurrentHashMap$Node
>     object: java.lang.WeakPairMap$Pair$Weak at 40620d6e={com.sun.proxy.jdk.proxy1=false}
> of class: java.util.concurrent.ConcurrentHashMap$Node
>     object: [Ljava.util.concurrent.ConcurrentHashMap$Node;@6e0bf27b  of
> class: java.util.concurrent.ConcurrentHashMap$Node[]
>     object: {java.lang.WeakPairMap$Pair$Weak at 64486d2a={org.graalvm.nativeimage.c.constant=true},
> java.lang.WeakPairMap$Pair$Weak at 708aa969={jdk.vm.ci.aarch64=true,
> jdk.vm.ci.services=true, jdk.vm.ci.common=true, jdk.vm.ci.code=true,
> jdk.vm.ci.hotspot.aarch64=true, jdk.vm.ci.runtime=true,
> jdk.vm.ci.hotspot.sparc=true, jdk.vm.ci.code.stack=true,
> jdk.vm.ci.hotspot.amd64=true, jdk.vm.ci.meta=true,
> jdk.vm.ci.code.site=true, jdk.vm.ci.sparc=true, jdk.vm.ci.hotspot=true,
> jdk.vm.ci.services.internal=true, jdk.vm.ci.amd64=true},
> java.lang.WeakPairMap$Pair$Weak at 6f715c77={jdk.vm.ci.common=false},
> java.lang.WeakPairMap$Pair$Weak at 40620d6e={com.sun.proxy.jdk.proxy1=false},
> java.lang.WeakPairMap$Pair$Weak at 40620d6e={jdk.internal.vm.annotation=false,
> jdk.internal=false, jdk.internal.reflect=false},
> java.lang.WeakPairMap$Pair$Weak at 248dd9c1={com.oracle.truffle.api.dsl=true,
> com.oracle.truffle.api.library=true, com.oracle.truffle.api.nodes=true,
> com.oracle.truffle.api=true}, java.lang.WeakPairMap$Pair$Weak at 7c968d5b={org.graalvm.compiler.debug=true,
> org.graalvm.compiler.nodes=true}, java.lang.WeakPairMap$Pair$Weak at 5ba3fc8d={jdk.vm.ci.aarch64=true,
> jdk.vm.ci.services=true, jdk.vm.ci.common=true, jdk.vm.ci.code=true,
> jdk.vm.ci.hotspot.aarch64=true, jdk.vm.ci.runtime=true,
> jdk.vm.ci.hotspot.sparc=true, jdk.vm.ci.code.stack=true,
> jdk.vm.ci.hotspot.amd64=true, jdk.vm.ci.meta=true,
> jdk.vm.ci.code.site=true, jdk.vm.ci.sparc=true, jdk.vm.ci.hotspot=true,
> jdk.vm.ci.services.internal=true, jdk.vm.ci.amd64=true},
> java.lang.WeakPairMap$Pair$Weak at 6112b142={com.oracle.truffle.api.impl=true,
> com.oracle.truffle.polyglot=true}, java.lang.WeakPairMap$Pair$Weak at 3909e5d8={org.graalvm.compiler.graph=true,
> org.graalvm.compiler.core.match=true},
> java.lang.WeakPairMap$Pair$Weak at 49834cb1={jdk.vm.ci.aarch64=true,
> jdk.vm.ci.services=true, jdk.vm.ci.common=true, jdk.vm.ci.code=true,
> jdk.vm.ci.hotspot.aarch64=true, jdk.vm.ci.runtime=true,
> jdk.vm.ci.hotspot.sparc=true, jdk.vm.ci.code.stack=true,
> jdk.vm.ci.hotspot.amd64=true, jdk.vm.ci.meta=true,
> jdk.vm.ci.code.site=true, jdk.vm.ci.sparc=true, jdk.vm.ci.hotspot=true,
> jdk.vm.ci.services.internal=true, jdk.vm.ci.amd64=true},
> java.lang.WeakPairMap$Pair$Weak at 2cc8e91b={org.graalvm.word.impl=false},
> java.lang.WeakPairMap$Pair$Weak at 1a0690f3={java.util=true, java.nio=true,
> jdk.internal.misc=true, jdk.internal.logger=true, java.lang=true,
> java.lang.ref=true, jdk.internal.vm.annotation=true, java.net=true,
> java.lang.invoke=true, jdk.internal.ref=true, jdk.internal.module=true,
> jdk.internal.reflect=true}, java.lang.WeakPairMap$Pair$Weak at 17197c41={com.sun.proxy.jdk.proxy2=false},
> java.lang.WeakPairMap$Pair$Weak at 21d705a9={org.graalvm.nativeimage.impl=true,
> org.graalvm.polyglot=true}, java.lang.WeakPairMap$Pair$Weak at 3515c1ea={jdk.vm.ci.aarch64=false,
> jdk.vm.ci.meta=false, jdk.vm.ci.code.site=false, jdk.vm.ci.services=false,
> jdk.vm.ci.code=false, jdk.vm.ci.common=false, jdk.vm.ci.hotspot=false,
> jdk.vm.ci.runtime=false, jdk.vm.ci.amd64=false}}  of class:
> java.util.concurrent.ConcurrentHashMap
>     object: java.lang.WeakPairMap at 3b3c322f  of class:
> java.lang.WeakPairMap
>     root: java.lang.Module.isReflectivelyExportedOrOpen(String, Module,
> boolean)
>
>
>
> On Thu, May 23, 2019 at 9:19 AM Bob McWhirter <bmcwhirt at redhat.com> wrote:
>
>> Seems like an intermittent/transient failure, as this morning it all
>> built cleanly.
>>
>> Good times.
>>
>> -Bob
>>
>> On Thu, May 23, 2019 at 6:36 AM Bob McWhirter <bmcwhirt at redhat.com>
>> wrote:
>>
>>> Thanks Andrew!
>>>
>>> I’ll continue to poke around. When I was getting the ObjectIdentifier
>>> variant there simple aren’t many statics available. There’s a none-static
>>> stringForm field of type String. Maybe I’m just misunderstanding the error
>>> message also.
>>>
>>> Time to git log.
>>>
>>> Bob
>>>
>>> On Thu, May 23, 2019 at 5:02 AM Andrew Dinn <adinn at redhat.com> wrote:
>>>
>>>> On 22/05/2019 19:20, Bob McWhirter wrote:
>>>> > With the new JDK11 changes, at one point I could build
>>>> polyglot/libpolyglot
>>>> > with JDK11.
>>>> >
>>>> > Currently, though, I'm hitting:
>>>> > . . .
>>>> > I sense that it might be the transient static boolean COMPACT_STRINGS
>>>> on
>>>> > String.class ultimately causing it, but that's not necessarily a
>>>> warranted
>>>> > opinion.  It always seems to be a String problem, and COMPACT_STRINGS
>>>> is
>>>> > designated as set by the JVM at runtime, not to the true that appears
>>>> in
>>>> > the source.
>>>>
>>>> I'm not sure how this could be the issue. I say that because I do
>>>> understand what goes on with this field in the JVM. Here is the context:
>>>>
>>>> During JVM startup various Java classes are constructed 'by hand' i.e.
>>>>
>>>>   the bytecodes are loaded
>>>>   the JVM-internal metadata model for the class is created
>>>>     (C++ object InstanceKlass plus auxiliary objects like Symbol. method
>>>> etc)
>>>>   the JVM-heap allocated Class<?> is created
>>>>   the Class<?>'s <clinit> method is run
>>>>
>>>> This is done in a very specific sequence starting with some Exception
>>>> classes that have no dependencies on supers or linked classes and
>>>> proceeding to classes like String, Thread, ThreadGroup etc.
>>>>
>>>> At certain points during this process a <clinit> run may require
>>>> creating certain linked Class<?> instances (most obviously supers but
>>>> also classes referenced from the <clinit> method) and this will initiate
>>>> the above steps recursively. So, the explicit hand initialization
>>>> sequence only defines a nominal load+init order. Recursive dependency
>>>> chasing means that the actual load order may be different.
>>>>
>>>> Now as for class String: very early on in this init sequence String gets
>>>> explicitly loaded. It is one of many classes whose static fields need to
>>>> be initialized specially to accord with HW, OS or user config setting --
>>>> String.COMPACT_STRINGS is one such example. In most normal cases where
>>>> runtime-specific settings need to be detected the <clinit> code calls
>>>> native methods from <clinit> code to read them. However, for a few of
>>>> these early-load classes the <clinit> methods instead initialize to a
>>>> dummy and the C++ code that hand-loads the class then overwrites the
>>>> correct values into the static field slots immediately after the load
>>>> and init of the class has completed.
>>>>
>>>> So, in the case of String field COMPACT_STRINGS will be updated by the
>>>> JVM immediately after running its <clinit> method. I don't really see
>>>> how this can make any difference visible to the Graal native image
>>>> generator. The change to COMPACT_STRINGS wil have happened long before
>>>> any of the Graal NativeImageGenerator code could even have started
>>>> running (indeed well before the JVM runtime code like ForkJoinPool that
>>>> drives the native image process is loaded). So, sorry Bob, but I think
>>>> you are probably barking up the wrong tree here.
>>>>
>>>> > Anyone have a thought or a pointer on what might've changed in the
>>>> past 3
>>>> > weeks, or if I'm completely off base and would be willing to point me
>>>> in a
>>>> > more useful direction.
>>>> Looking at recent changes sounds to me like the best approach.
>>>>
>>>> regards,
>>>>
>>>>
>>>> Andrew Dinn
>>>> -----------
>>>> Senior Principal Software Engineer
>>>> Red Hat UK Ltd
>>>> Registered in England and Wales under Company Registration No. 03798903
>>>> Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander
>>>>
>>>