[foreign] programmatic access to native libraries w/o binder

Wed Nov 28 15:41:00 UTC 2018

> Internally, we can have a NativeMethodType abstraction, since for the
> binder it seems to make sense - I'm wondering how much that should be
> public API, given that low level users will probably see that as a
> somewhat redundant intermediate step.

Well, if it makes sense for the binder, wouldn't it make sense for 
framework writers as well? I'd imagine they'd want to have some kind of 
MethodType analog for native binding.

Jorn

Maurizio Cimadamore schreef op 2018-11-28 16:32:
> On 28/11/2018 15:27, Jorn Vernee wrote:
>> Looking good!
>> 
>>> * having to create a NativeMethodType when I just want to pass the
>>> list of layout types leads to more verbose code
>>> 
>>> * we might need NativeTypes.POINTER - doing 
>>> NativeTypes.VOID.pointer()
>>> is an explicit workaround, but a tad verbose
>> 
>> Reading through the code, I prefer both of these actually, since it 
>> makes the code more visually explicit as to what's happening. The `new 
>> NativeType(...)` call also gives a visual grouping to the arguments. 
>> Though I admit, I like being very descriptive when writing code.
>> 
>> Tbh, I mostly expect the NativeMethodType to be constructed 
>> automatically, e.g. by deriving the signature from a mangled symbol 
>> name, so I'm not sure how much verbosity will be a problem in 
>> practice.
>> 
>> To save on characters I'd suggest using `import static NativeTypes.*`, 
>> and perhaps we could add a convenience overload of upcallStub which 
>> doesn't take the target MethodHandle, and instead tries to infer it 
>> from the passed receiver object?
>> 
>>     default UpcallHandle upcallStub(Object receiver, NativeMethodType 
>> nmt) {
>>         Method m = receiver.getClass().getDeclaredMethods()[0]; // you 
>> get the gist
>>         return upcallStub(cc, receiver, publicLookup().unreflect(m), 
>> nmt);
>>     }
> 
> It's not about saving characters - I was trying to imagine what a
> typical usage of this API would look like. I assume that people that
> reach low level to SystemABI directly (bypassing the binder) basically
> want to be more in control on how the mapping is done. This could be a
> powerful options for framework writers. But those users, I believe,
> will almost never use the API point that takes a j.l.Method and a
> Function - they would prefer to specific a list of
> NativeTypes/LayoutTypes directly.
> 
> Internally, we can have a NativeMethodType abstraction, since for the
> binder it seems to make sense - I'm wondering how much that should be
> public API, given that low level users will probably see that as a
> somewhat redundant intermediate step.
> 
>> 
>> Btw, now that we're starting to have NativeTypes and NativeMethodType 
>> maybe we should consider renaming LayoutType to NativeType as well?
> 
> Sure, I was thinking along the same lines
> 
> Maurizio
> 
>> 
>> Jorn
>> 
>> Maurizio Cimadamore schreef op 2018-11-28 15:00:
>>> Hi,
>>> I took the patch at [1] for a spin with jshell. I wanted to see how
>>> easy it was to call native functions using the SystemABI interface -
>>> e.g. w/o binder and annotated interface. I think I'm pleasantly
>>> impressed! I did some changes to the proposed patch as I wanted to 
>>> put
>>> everything SystemABI-specific in a public package (java.foreign) and 
>>> I
>>> also added a method on UpcallHandle to get a pointer. But overall
>>> these were minor modification, mostly in order to be able to use this
>>> setup on jshell. But let's dive on some code.
>>> 
>>> First, some setup:
>>> 
>>> 
>>> $ jshell
>>> |  Welcome to JShell -- Version 12-internal
>>> |  For an introduction type: /help intro
>>> 
>>> jshell> import java.foreign.*
>>> 
>>> jshell> import java.foreign.memory.*
>>> 
>>> jshell> import java.lang.invoke.*
>>> 
>>> jshell> var abi = SystemABI.getInstance()
>>> abi ==> jdk.internal.foreign.abi.sysv.x64.SysVx64ABI at 41cf53f9
>>> 
>>> Now that we have an ABI, let's try getpid:
>>> 
>>> 
>>> jshell> var getpid =
>>> abi.downcallHandle(Libraries.getDefaultLibrary().lookup("getpid"), 
>>> new
>>> NativeMethodType(NativeTypes.INT32))
>>> getpid ==> MethodHandle()int
>>> 
>>> jshell> (int)getpid.invokeExact()
>>> $6 ==> 20308
>>> 
>>> 
>>> That was easy; let's try a function which actually takes some args:
>>> 
>>> 
>>> jshell> var pow =
>>> abi.downcallHandle(Libraries.getDefaultLibrary().lookup("pow"), new
>>> NativeMethodType(NativeTypes.DOUBLE, NativeTypes.DOUBLE,
>>> NativeTypes.DOUBLE))
>>> pow ==> MethodHandle(double,double)double
>>> 
>>> jshell> (double)pow.invokeExact(2d, 3d)
>>> $12 ==> 8.0
>>> 
>>> 
>>> That works too. Finally, let's see if we can make qsort work - first
>>> let's declare the comparison logic:
>>> 
>>> 
>>> jshell> class Comp {
>>>    ...>       int comp(Pointer<Integer> p1, Pointer<Integer> p2) {
>>>    ...>          return p1.get() - p2.get();
>>>    ...>       }
>>>    ...>    }
>>> |  created class Comp
>>> 
>>> 
>>> And allocate the stub:
>>> 
>>> 
>>> 
>>> jshell> var up = abi.upcallStub(abi.defaultCallingConvention(), new 
>>> Comp(),
>>>    ...> MethodHandles.lookup().findVirtual(Comp.class, "comp",
>>> MethodType.methodType(int.class, Pointer.class, Pointer.class)),
>>>    ...>                         new
>>> NativeMethodType(NativeTypes.INT32, NativeTypes.INT.pointer(),
>>> NativeTypes.INT.pointer()));
>>> up ==> jdk.internal.foreign.invokers.DirectUpcallHandle at dcf3e99
>>> 
>>> 
>>> And finally, let's call qsort and print the results.
>>> 
>>> 
>>> jshell> var qsort =
>>> abi.downcallHandle(Libraries.getDefaultLibrary().lookup("qsort"),
>>>    ...>                         new 
>>> NativeMethodType(NativeTypes.VOID,
>>> NativeTypes.INT.pointer(), NativeTypes.INT, NativeTypes.INT,
>>> NativeTypes.VOID.pointer()));
>>> qsort ==> MethodHandle(Pointer,int,int,Pointer)void
>>> 
>>> jshell> Scope sc = Scope.newNativeScope()
>>> sc ==> jdk.internal.foreign.ScopeImpl$NativeScope at 5fdef03a
>>> 
>>> jshell> var arr = sc.allocateArray(NativeTypes.INT32, new int[] { 2,
>>> 6, 4, 8, 1, 10 });
>>> arr ==> jdk.internal.foreign.memory.BoundedArray at 311d617d
>>> 
>>> jshell> qsort.invoke(arr.elementPointer(), 6, 4, up.entryPoint());
>>> $13 ==> null
>>> 
>>> jshell> System.err.println(Arrays.toString(arr.toArray(int[]::new)));
>>> [1, 2, 4, 6, 8, 10]
>>> 
>>> Cool! We have been able to do all this without using a single
>>> annotated interface, which suggests that SystemABI might indeed be
>>> general enough to build on top of it.
>>> 
>>> There are, I think, some hiccups in the above snippets which I found 
>>> annoying:
>>> 
>>> * having to create a NativeMethodType when I just want to pass the
>>> list of layout types leads to more verbose code
>>> 
>>> * we might need NativeTypes.POINTER - doing 
>>> NativeTypes.VOID.pointer()
>>> is an explicit workaround, but a tad verbose
>>> 
>>> * as we already know, the upcallStub API point could use some
>>> simplification: omit the receiver argument and directly return the
>>> pointer; that will also simplify things
>>> 
>>> Maurizio
>>> 
>>> [1] - 
>>> https://cr.openjdk.java.net/~henryjen/panama/SystemABI/webrev.01/webrev/