[foreign] programmatic access to native libraries w/o binder

Wed Nov 28 15:32:28 UTC 2018

On 28/11/2018 15:27, Jorn Vernee wrote:
> Looking good!
>
>> * having to create a NativeMethodType when I just want to pass the
>> list of layout types leads to more verbose code
>>
>> * we might need NativeTypes.POINTER - doing NativeTypes.VOID.pointer()
>> is an explicit workaround, but a tad verbose
>
> Reading through the code, I prefer both of these actually, since it 
> makes the code more visually explicit as to what's happening. The `new 
> NativeType(...)` call also gives a visual grouping to the arguments. 
> Though I admit, I like being very descriptive when writing code.
>
> Tbh, I mostly expect the NativeMethodType to be constructed 
> automatically, e.g. by deriving the signature from a mangled symbol 
> name, so I'm not sure how much verbosity will be a problem in practice.
>
> To save on characters I'd suggest using `import static NativeTypes.*`, 
> and perhaps we could add a convenience overload of upcallStub which 
> doesn't take the target MethodHandle, and instead tries to infer it 
> from the passed receiver object?
>
>     default UpcallHandle upcallStub(Object receiver, NativeMethodType 
> nmt) {
>         Method m = receiver.getClass().getDeclaredMethods()[0]; // you 
> get the gist
>         return upcallStub(cc, receiver, publicLookup().unreflect(m), 
> nmt);
>     }

It's not about saving characters - I was trying to imagine what a 
typical usage of this API would look like. I assume that people that 
reach low level to SystemABI directly (bypassing the binder) basically 
want to be more in control on how the mapping is done. This could be a 
powerful options for framework writers. But those users, I believe, will 
almost never use the API point that takes a j.l.Method and a Function - 
they would prefer to specific a list of NativeTypes/LayoutTypes directly.

Internally, we can have a NativeMethodType abstraction, since for the 
binder it seems to make sense - I'm wondering how much that should be 
public API, given that low level users will probably see that as a 
somewhat redundant intermediate step.

>
> Btw, now that we're starting to have NativeTypes and NativeMethodType 
> maybe we should consider renaming LayoutType to NativeType as well?

Sure, I was thinking along the same lines

Maurizio

>
> Jorn
>
> Maurizio Cimadamore schreef op 2018-11-28 15:00:
>> Hi,
>> I took the patch at [1] for a spin with jshell. I wanted to see how
>> easy it was to call native functions using the SystemABI interface -
>> e.g. w/o binder and annotated interface. I think I'm pleasantly
>> impressed! I did some changes to the proposed patch as I wanted to put
>> everything SystemABI-specific in a public package (java.foreign) and I
>> also added a method on UpcallHandle to get a pointer. But overall
>> these were minor modification, mostly in order to be able to use this
>> setup on jshell. But let's dive on some code.
>>
>> First, some setup:
>>
>>
>> $ jshell
>> |  Welcome to JShell -- Version 12-internal
>> |  For an introduction type: /help intro
>>
>> jshell> import java.foreign.*
>>
>> jshell> import java.foreign.memory.*
>>
>> jshell> import java.lang.invoke.*
>>
>> jshell> var abi = SystemABI.getInstance()
>> abi ==> jdk.internal.foreign.abi.sysv.x64.SysVx64ABI at 41cf53f9
>>
>> Now that we have an ABI, let's try getpid:
>>
>>
>> jshell> var getpid =
>> abi.downcallHandle(Libraries.getDefaultLibrary().lookup("getpid"), new
>> NativeMethodType(NativeTypes.INT32))
>> getpid ==> MethodHandle()int
>>
>> jshell> (int)getpid.invokeExact()
>> $6 ==> 20308
>>
>>
>> That was easy; let's try a function which actually takes some args:
>>
>>
>> jshell> var pow =
>> abi.downcallHandle(Libraries.getDefaultLibrary().lookup("pow"), new
>> NativeMethodType(NativeTypes.DOUBLE, NativeTypes.DOUBLE,
>> NativeTypes.DOUBLE))
>> pow ==> MethodHandle(double,double)double
>>
>> jshell> (double)pow.invokeExact(2d, 3d)
>> $12 ==> 8.0
>>
>>
>> That works too. Finally, let's see if we can make qsort work - first
>> let's declare the comparison logic:
>>
>>
>> jshell> class Comp {
>>    ...>       int comp(Pointer<Integer> p1, Pointer<Integer> p2) {
>>    ...>          return p1.get() - p2.get();
>>    ...>       }
>>    ...>    }
>> |  created class Comp
>>
>>
>> And allocate the stub:
>>
>>
>>
>> jshell> var up = abi.upcallStub(abi.defaultCallingConvention(), new 
>> Comp(),
>>    ...> MethodHandles.lookup().findVirtual(Comp.class, "comp",
>> MethodType.methodType(int.class, Pointer.class, Pointer.class)),
>>    ...>                         new
>> NativeMethodType(NativeTypes.INT32, NativeTypes.INT.pointer(),
>> NativeTypes.INT.pointer()));
>> up ==> jdk.internal.foreign.invokers.DirectUpcallHandle at dcf3e99
>>
>>
>> And finally, let's call qsort and print the results.
>>
>>
>> jshell> var qsort =
>> abi.downcallHandle(Libraries.getDefaultLibrary().lookup("qsort"),
>>    ...>                         new NativeMethodType(NativeTypes.VOID,
>> NativeTypes.INT.pointer(), NativeTypes.INT, NativeTypes.INT,
>> NativeTypes.VOID.pointer()));
>> qsort ==> MethodHandle(Pointer,int,int,Pointer)void
>>
>> jshell> Scope sc = Scope.newNativeScope()
>> sc ==> jdk.internal.foreign.ScopeImpl$NativeScope at 5fdef03a
>>
>> jshell> var arr = sc.allocateArray(NativeTypes.INT32, new int[] { 2,
>> 6, 4, 8, 1, 10 });
>> arr ==> jdk.internal.foreign.memory.BoundedArray at 311d617d
>>
>> jshell> qsort.invoke(arr.elementPointer(), 6, 4, up.entryPoint());
>> $13 ==> null
>>
>> jshell> System.err.println(Arrays.toString(arr.toArray(int[]::new)));
>> [1, 2, 4, 6, 8, 10]
>>
>> Cool! We have been able to do all this without using a single
>> annotated interface, which suggests that SystemABI might indeed be
>> general enough to build on top of it.
>>
>> There are, I think, some hiccups in the above snippets which I found 
>> annoying:
>>
>> * having to create a NativeMethodType when I just want to pass the
>> list of layout types leads to more verbose code
>>
>> * we might need NativeTypes.POINTER - doing NativeTypes.VOID.pointer()
>> is an explicit workaround, but a tad verbose
>>
>> * as we already know, the upcallStub API point could use some
>> simplification: omit the receiver argument and directly return the
>> pointer; that will also simplify things
>>
>> Maurizio
>>
>> [1] - 
>> https://cr.openjdk.java.net/~henryjen/panama/SystemABI/webrev.01/webrev/