JDK11 & Polyglot

[Bob McWhirter]

Progress... maybe?

In Module$ReflectionData

/**
 * A module (1st key) exports or opens a package to another module
 * (2nd key). The map value is a map of package name to a boolean
 * that indicates if the package is opened.
 */
static final WeakPairMap<Module, Module, Map<String, Boolean>> exports =
    new WeakPairMap<>();


Ultimately used by Module.isReflectivelyExportedOrOpen(...):

private boolean isReflectivelyExportedOrOpen(String pn, Module other,
boolean open) {
    // exported or open to all modules
    Map<String, Boolean> exports = ReflectionData.exports.get(this,
EVERYONE_MODULE);



Is the source of things-changing-while-writing-the-heap.

Not sure how to fix, or if there is a way to whitelist/ignore this.

I do have an inbound PR to push in a moment helping fix the debug reason
trace, to avoid the "root=null" in the current master#HEAD.

Caused by: com.oracle.svm.core.util.UserError$UserException: Static field
or an object referenced from a static field changed during native image
generation?
  object:com.oracle.truffle.api=true  of class:
java.util.concurrent.ConcurrentHashMap$Node
  reachable through:
    object: [Ljava.util.concurrent.ConcurrentHashMap$Node;@3cb34219  of
class: java.util.concurrent.ConcurrentHashMap$Node[]
    object: {com.oracle.truffle.api.dsl=true,
com.oracle.truffle.api.library=true, com.oracle.truffle.api.nodes=true,
com.oracle.truffle.api=true}  of class:
java.util.concurrent.ConcurrentHashMap
    object: java.lang.WeakPairMap$Pair$Weak at 248dd9c1={com.oracle.truffle.api.dsl=true,
com.oracle.truffle.api.library=true, com.oracle.truffle.api.nodes=true,
com.oracle.truffle.api=true}  of class:
java.util.concurrent.ConcurrentHashMap$Node
    object: java.lang.WeakPairMap$Pair$Weak at 40620d6e={jdk.internal.vm.annotation=false,
jdk.internal=false, jdk.internal.reflect=false}  of class:
java.util.concurrent.ConcurrentHashMap$Node
    object: java.lang.WeakPairMap$Pair$Weak at 40620d6e={com.sun.proxy.jdk.proxy1=false}
of class: java.util.concurrent.ConcurrentHashMap$Node
    object: [Ljava.util.concurrent.ConcurrentHashMap$Node;@6e0bf27b  of
class: java.util.concurrent.ConcurrentHashMap$Node[]
    object: {java.lang.WeakPairMap$Pair$Weak at 64486d2a={org.graalvm.nativeimage.c.constant=true},
java.lang.WeakPairMap$Pair$Weak at 708aa969={jdk.vm.ci.aarch64=true,
jdk.vm.ci.services=true, jdk.vm.ci.common=true, jdk.vm.ci.code=true,
jdk.vm.ci.hotspot.aarch64=true, jdk.vm.ci.runtime=true,
jdk.vm.ci.hotspot.sparc=true, jdk.vm.ci.code.stack=true,
jdk.vm.ci.hotspot.amd64=true, jdk.vm.ci.meta=true,
jdk.vm.ci.code.site=true, jdk.vm.ci.sparc=true, jdk.vm.ci.hotspot=true,
jdk.vm.ci.services.internal=true, jdk.vm.ci.amd64=true},
java.lang.WeakPairMap$Pair$Weak at 6f715c77={jdk.vm.ci.common=false},
java.lang.WeakPairMap$Pair$Weak at 40620d6e={com.sun.proxy.jdk.proxy1=false},
java.lang.WeakPairMap$Pair$Weak at 40620d6e={jdk.internal.vm.annotation=false,
jdk.internal=false, jdk.internal.reflect=false},
java.lang.WeakPairMap$Pair$Weak at 248dd9c1={com.oracle.truffle.api.dsl=true,
com.oracle.truffle.api.library=true, com.oracle.truffle.api.nodes=true,
com.oracle.truffle.api=true},
java.lang.WeakPairMap$Pair$Weak at 7c968d5b={org.graalvm.compiler.debug=true,
org.graalvm.compiler.nodes=true},
java.lang.WeakPairMap$Pair$Weak at 5ba3fc8d={jdk.vm.ci.aarch64=true,
jdk.vm.ci.services=true, jdk.vm.ci.common=true, jdk.vm.ci.code=true,
jdk.vm.ci.hotspot.aarch64=true, jdk.vm.ci.runtime=true,
jdk.vm.ci.hotspot.sparc=true, jdk.vm.ci.code.stack=true,
jdk.vm.ci.hotspot.amd64=true, jdk.vm.ci.meta=true,
jdk.vm.ci.code.site=true, jdk.vm.ci.sparc=true, jdk.vm.ci.hotspot=true,
jdk.vm.ci.services.internal=true, jdk.vm.ci.amd64=true},
java.lang.WeakPairMap$Pair$Weak at 6112b142={com.oracle.truffle.api.impl=true,
com.oracle.truffle.polyglot=true},
java.lang.WeakPairMap$Pair$Weak at 3909e5d8={org.graalvm.compiler.graph=true,
org.graalvm.compiler.core.match=true},
java.lang.WeakPairMap$Pair$Weak at 49834cb1={jdk.vm.ci.aarch64=true,
jdk.vm.ci.services=true, jdk.vm.ci.common=true, jdk.vm.ci.code=true,
jdk.vm.ci.hotspot.aarch64=true, jdk.vm.ci.runtime=true,
jdk.vm.ci.hotspot.sparc=true, jdk.vm.ci.code.stack=true,
jdk.vm.ci.hotspot.amd64=true, jdk.vm.ci.meta=true,
jdk.vm.ci.code.site=true, jdk.vm.ci.sparc=true, jdk.vm.ci.hotspot=true,
jdk.vm.ci.services.internal=true, jdk.vm.ci.amd64=true},
java.lang.WeakPairMap$Pair$Weak at 2cc8e91b={org.graalvm.word.impl=false},
java.lang.WeakPairMap$Pair$Weak at 1a0690f3={java.util=true, java.nio=true,
jdk.internal.misc=true, jdk.internal.logger=true, java.lang=true,
java.lang.ref=true, jdk.internal.vm.annotation=true, java.net=true,
java.lang.invoke=true, jdk.internal.ref=true, jdk.internal.module=true,
jdk.internal.reflect=true},
java.lang.WeakPairMap$Pair$Weak at 17197c41={com.sun.proxy.jdk.proxy2=false},
java.lang.WeakPairMap$Pair$Weak at 21d705a9={org.graalvm.nativeimage.impl=true,
org.graalvm.polyglot=true},
java.lang.WeakPairMap$Pair$Weak at 3515c1ea={jdk.vm.ci.aarch64=false,
jdk.vm.ci.meta=false, jdk.vm.ci.code.site=false, jdk.vm.ci.services=false,
jdk.vm.ci.code=false, jdk.vm.ci.common=false, jdk.vm.ci.hotspot=false,
jdk.vm.ci.runtime=false, jdk.vm.ci.amd64=false}}  of class:
java.util.concurrent.ConcurrentHashMap
    object: java.lang.WeakPairMap at 3b3c322f  of class: java.lang.WeakPairMap
    root: java.lang.Module.isReflectivelyExportedOrOpen(String, Module,
boolean)



On Thu, May 23, 2019 at 9:19 AM Bob McWhirter <bmcwhirt at redhat.com> wrote:

> Seems like an intermittent/transient failure, as this morning it all built
> cleanly.
>
> Good times.
>
> -Bob
>
> On Thu, May 23, 2019 at 6:36 AM Bob McWhirter <bmcwhirt at redhat.com> wrote:
>
>> Thanks Andrew!
>>
>> I’ll continue to poke around. When I was getting the ObjectIdentifier
>> variant there simple aren’t many statics available. There’s a none-static
>> stringForm field of type String. Maybe I’m just misunderstanding the error
>> message also.
>>
>> Time to git log.
>>
>> Bob
>>
>> On Thu, May 23, 2019 at 5:02 AM Andrew Dinn <adinn at redhat.com> wrote:
>>
>>> On 22/05/2019 19:20, Bob McWhirter wrote:
>>> > With the new JDK11 changes, at one point I could build
>>> polyglot/libpolyglot
>>> > with JDK11.
>>> >
>>> > Currently, though, I'm hitting:
>>> > . . .
>>> > I sense that it might be the transient static boolean COMPACT_STRINGS
>>> on
>>> > String.class ultimately causing it, but that's not necessarily a
>>> warranted
>>> > opinion.  It always seems to be a String problem, and COMPACT_STRINGS
>>> is
>>> > designated as set by the JVM at runtime, not to the true that appears
>>> in
>>> > the source.
>>>
>>> I'm not sure how this could be the issue. I say that because I do
>>> understand what goes on with this field in the JVM. Here is the context:
>>>
>>> During JVM startup various Java classes are constructed 'by hand' i.e.
>>>
>>>   the bytecodes are loaded
>>>   the JVM-internal metadata model for the class is created
>>>     (C++ object InstanceKlass plus auxiliary objects like Symbol. method
>>> etc)
>>>   the JVM-heap allocated Class<?> is created
>>>   the Class<?>'s <clinit> method is run
>>>
>>> This is done in a very specific sequence starting with some Exception
>>> classes that have no dependencies on supers or linked classes and
>>> proceeding to classes like String, Thread, ThreadGroup etc.
>>>
>>> At certain points during this process a <clinit> run may require
>>> creating certain linked Class<?> instances (most obviously supers but
>>> also classes referenced from the <clinit> method) and this will initiate
>>> the above steps recursively. So, the explicit hand initialization
>>> sequence only defines a nominal load+init order. Recursive dependency
>>> chasing means that the actual load order may be different.
>>>
>>> Now as for class String: very early on in this init sequence String gets
>>> explicitly loaded. It is one of many classes whose static fields need to
>>> be initialized specially to accord with HW, OS or user config setting --
>>> String.COMPACT_STRINGS is one such example. In most normal cases where
>>> runtime-specific settings need to be detected the <clinit> code calls
>>> native methods from <clinit> code to read them. However, for a few of
>>> these early-load classes the <clinit> methods instead initialize to a
>>> dummy and the C++ code that hand-loads the class then overwrites the
>>> correct values into the static field slots immediately after the load
>>> and init of the class has completed.
>>>
>>> So, in the case of String field COMPACT_STRINGS will be updated by the
>>> JVM immediately after running its <clinit> method. I don't really see
>>> how this can make any difference visible to the Graal native image
>>> generator. The change to COMPACT_STRINGS wil have happened long before
>>> any of the Graal NativeImageGenerator code could even have started
>>> running (indeed well before the JVM runtime code like ForkJoinPool that
>>> drives the native image process is loaded). So, sorry Bob, but I think
>>> you are probably barking up the wrong tree here.
>>>
>>> > Anyone have a thought or a pointer on what might've changed in the
>>> past 3
>>> > weeks, or if I'm completely off base and would be willing to point me
>>> in a
>>> > more useful direction.
>>> Looking at recent changes sounds to me like the best approach.
>>>
>>> regards,
>>>
>>>
>>> Andrew Dinn
>>> -----------
>>> Senior Principal Software Engineer
>>> Red Hat UK Ltd
>>> Registered in England and Wales under Company Registration No. 03798903
>>> Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander
>>>
>>