[foreign] programmatic access to native libraries w/o binder

Maurizio Cimadamore maurizio.cimadamore at oracle.com
Wed Nov 28 15:58:49 UTC 2018 

On 28/11/2018 15:41, Jorn Vernee wrote:
>> Internally, we can have a NativeMethodType abstraction, since for the
>> binder it seems to make sense - I'm wondering how much that should be
>> public API, given that low level users will probably see that as a
>> somewhat redundant intermediate step.
>
> Well, if it makes sense for the binder, wouldn't it make sense for 
> framework writers as well? I'd imagine they'd want to have some kind 
> of MethodType analog for native binding.

Let me unpack what I said a bit. The useful part of NativeMethodType is 
that is an aggregation of layout types (return and arguments).

The not-so-useful part is, I think, the ability to construct one such 
NMT from a j.l.Method and a Function; as that is making a specific 
assumption that you have a Java method somewhere embodying the desired 
Java signature for a given method call. Maybe frameworks will use Java 
method for that, maybe they will use something else (even a config 
file/API?). In any case, I think that saying NMT <-> j.l.Method is a 
step too far.

And, if the interesting part of NMT is really that is a bunch of 
LayoutTypes, is it worth adding an extra public API point which looks so 
similar to j.l.i.MethodType? What is the risk that people will get 
confused by it? I don't have a 100% answer to that, but, let's say, I 
wish there were stronger arguments on favor of having NMT (and maybe 
we'll find them, but maybe not).

Maurizio


>
> Jorn
>
> Maurizio Cimadamore schreef op 2018-11-28 16:32:
>> On 28/11/2018 15:27, Jorn Vernee wrote:
>>> Looking good!
>>>
>>>> * having to create a NativeMethodType when I just want to pass the
>>>> list of layout types leads to more verbose code
>>>>
>>>> * we might need NativeTypes.POINTER - doing NativeTypes.VOID.pointer()
>>>> is an explicit workaround, but a tad verbose
>>>
>>> Reading through the code, I prefer both of these actually, since it 
>>> makes the code more visually explicit as to what's happening. The 
>>> `new NativeType(...)` call also gives a visual grouping to the 
>>> arguments. Though I admit, I like being very descriptive when 
>>> writing code.
>>>
>>> Tbh, I mostly expect the NativeMethodType to be constructed 
>>> automatically, e.g. by deriving the signature from a mangled symbol 
>>> name, so I'm not sure how much verbosity will be a problem in practice.
>>>
>>> To save on characters I'd suggest using `import static 
>>> NativeTypes.*`, and perhaps we could add a convenience overload of 
>>> upcallStub which doesn't take the target MethodHandle, and instead 
>>> tries to infer it from the passed receiver object?
>>>
>>>     default UpcallHandle upcallStub(Object receiver, 
>>> NativeMethodType nmt) {
>>>         Method m = receiver.getClass().getDeclaredMethods()[0]; // 
>>> you get the gist
>>>         return upcallStub(cc, receiver, publicLookup().unreflect(m), 
>>> nmt);
>>>     }
>>
>> It's not about saving characters - I was trying to imagine what a
>> typical usage of this API would look like. I assume that people that
>> reach low level to SystemABI directly (bypassing the binder) basically
>> want to be more in control on how the mapping is done. This could be a
>> powerful options for framework writers. But those users, I believe,
>> will almost never use the API point that takes a j.l.Method and a
>> Function - they would prefer to specific a list of
>> NativeTypes/LayoutTypes directly.
>>
>> Internally, we can have a NativeMethodType abstraction, since for the
>> binder it seems to make sense - I'm wondering how much that should be
>> public API, given that low level users will probably see that as a
>> somewhat redundant intermediate step.
>>
>>>
>>> Btw, now that we're starting to have NativeTypes and 
>>> NativeMethodType maybe we should consider renaming LayoutType to 
>>> NativeType as well?
>>
>> Sure, I was thinking along the same lines
>>
>> Maurizio
>>
>>>
>>> Jorn
>>>
>>> Maurizio Cimadamore schreef op 2018-11-28 15:00:
>>>> Hi,
>>>> I took the patch at [1] for a spin with jshell. I wanted to see how
>>>> easy it was to call native functions using the SystemABI interface -
>>>> e.g. w/o binder and annotated interface. I think I'm pleasantly
>>>> impressed! I did some changes to the proposed patch as I wanted to put
>>>> everything SystemABI-specific in a public package (java.foreign) and I
>>>> also added a method on UpcallHandle to get a pointer. But overall
>>>> these were minor modification, mostly in order to be able to use this
>>>> setup on jshell. But let's dive on some code.
>>>>
>>>> First, some setup:
>>>>
>>>>
>>>> $ jshell
>>>> |  Welcome to JShell -- Version 12-internal
>>>> |  For an introduction type: /help intro
>>>>
>>>> jshell> import java.foreign.*
>>>>
>>>> jshell> import java.foreign.memory.*
>>>>
>>>> jshell> import java.lang.invoke.*
>>>>
>>>> jshell> var abi = SystemABI.getInstance()
>>>> abi ==> jdk.internal.foreign.abi.sysv.x64.SysVx64ABI at 41cf53f9
>>>>
>>>> Now that we have an ABI, let's try getpid:
>>>>
>>>>
>>>> jshell> var getpid =
>>>> abi.downcallHandle(Libraries.getDefaultLibrary().lookup("getpid"), new
>>>> NativeMethodType(NativeTypes.INT32))
>>>> getpid ==> MethodHandle()int
>>>>
>>>> jshell> (int)getpid.invokeExact()
>>>> $6 ==> 20308
>>>>
>>>>
>>>> That was easy; let's try a function which actually takes some args:
>>>>
>>>>
>>>> jshell> var pow =
>>>> abi.downcallHandle(Libraries.getDefaultLibrary().lookup("pow"), new
>>>> NativeMethodType(NativeTypes.DOUBLE, NativeTypes.DOUBLE,
>>>> NativeTypes.DOUBLE))
>>>> pow ==> MethodHandle(double,double)double
>>>>
>>>> jshell> (double)pow.invokeExact(2d, 3d)
>>>> $12 ==> 8.0
>>>>
>>>>
>>>> That works too. Finally, let's see if we can make qsort work - first
>>>> let's declare the comparison logic:
>>>>
>>>>
>>>> jshell> class Comp {
>>>>    ...>       int comp(Pointer<Integer> p1, Pointer<Integer> p2) {
>>>>    ...>          return p1.get() - p2.get();
>>>>    ...>       }
>>>>    ...>    }
>>>> |  created class Comp
>>>>
>>>>
>>>> And allocate the stub:
>>>>
>>>>
>>>>
>>>> jshell> var up = abi.upcallStub(abi.defaultCallingConvention(), new 
>>>> Comp(),
>>>>    ...> MethodHandles.lookup().findVirtual(Comp.class, "comp",
>>>> MethodType.methodType(int.class, Pointer.class, Pointer.class)),
>>>>    ...>                         new
>>>> NativeMethodType(NativeTypes.INT32, NativeTypes.INT.pointer(),
>>>> NativeTypes.INT.pointer()));
>>>> up ==> jdk.internal.foreign.invokers.DirectUpcallHandle at dcf3e99
>>>>
>>>>
>>>> And finally, let's call qsort and print the results.
>>>>
>>>>
>>>> jshell> var qsort =
>>>> abi.downcallHandle(Libraries.getDefaultLibrary().lookup("qsort"),
>>>>    ...>                         new NativeMethodType(NativeTypes.VOID,
>>>> NativeTypes.INT.pointer(), NativeTypes.INT, NativeTypes.INT,
>>>> NativeTypes.VOID.pointer()));
>>>> qsort ==> MethodHandle(Pointer,int,int,Pointer)void
>>>>
>>>> jshell> Scope sc = Scope.newNativeScope()
>>>> sc ==> jdk.internal.foreign.ScopeImpl$NativeScope at 5fdef03a
>>>>
>>>> jshell> var arr = sc.allocateArray(NativeTypes.INT32, new int[] { 2,
>>>> 6, 4, 8, 1, 10 });
>>>> arr ==> jdk.internal.foreign.memory.BoundedArray at 311d617d
>>>>
>>>> jshell> qsort.invoke(arr.elementPointer(), 6, 4, up.entryPoint());
>>>> $13 ==> null
>>>>
>>>> jshell> System.err.println(Arrays.toString(arr.toArray(int[]::new)));
>>>> [1, 2, 4, 6, 8, 10]
>>>>
>>>> Cool! We have been able to do all this without using a single
>>>> annotated interface, which suggests that SystemABI might indeed be
>>>> general enough to build on top of it.
>>>>
>>>> There are, I think, some hiccups in the above snippets which I 
>>>> found annoying:
>>>>
>>>> * having to create a NativeMethodType when I just want to pass the
>>>> list of layout types leads to more verbose code
>>>>
>>>> * we might need NativeTypes.POINTER - doing NativeTypes.VOID.pointer()
>>>> is an explicit workaround, but a tad verbose
>>>>
>>>> * as we already know, the upcallStub API point could use some
>>>> simplification: omit the receiver argument and directly return the
>>>> pointer; that will also simplify things
>>>>
>>>> Maurizio
>>>>
>>>> [1] - 
>>>> https://cr.openjdk.java.net/~henryjen/panama/SystemABI/webrev.01/webrev/ 
>>>>

More information about the panama-dev mailing list