[foreign] programmatic access to native libraries w/o binder
Jorn Vernee
jbvernee at xs4all.nl
Wed Nov 28 16:41:30 UTC 2018
Using LayoutType[] is not good enough for me, and for the same reason
the JDK doesn't have a Tuple or Pair type I think; it removes the
meta-data from your data. It would also mean having to unpack the array
manually:
LayoutType<?>[] types = deriveNativeMethodType(target);
LayoutType<?>[] params = Arrays.copyOfRange(types, 1, types.length -
1);
return abi.downcallHandle(sym, types[0], params);
As far as stronger arguments go; imagine having to work with
MethodHandles without having MethodType, and all the lookup methods
taking `Class<?>, Class<?>...` as arguments instead. The benefits of
having an API type for that are that you get all the utility methods
with it, and code becomes more share-able because it's all using the
same API.
Even if it's a simple API, having it can still be very useful.
Jorn
Maurizio Cimadamore schreef op 2018-11-28 17:17:
> I think that's probably true. Although, in the absence of that, your
> deriveNativeMethodType could be rewritten to return a LayoutType[], I
> guess.
>
> Is there any additional value in reifying the vararg-ness inside the
> NMT? To me that is likely to be what moves the balance in one
> direction rather than another. I'd say that there is some value in
> having a description of the native method type that is self-contained
> (including varargs) - as opposed to pass an extra boolean flag to the
> SystemABI methods. That is the strongest argument I can find for NMT.
>
> Maurizio
>
> On 28/11/2018 16:10, Jorn Vernee wrote:
>> I don't imagine framework code to look like this;
>>
>> var getpid =
>> abi.downcallHandle(Libraries.getDefaultLibrary().lookup("getpid"), new
>> NativeMethodType(NativeTypes.INT32))
>>
>> But more like this:
>>
>> public MetodHandle bind(BindTarget target) { // some framework
>> type
>> Library.Symbol sym = Libraries.loadLibrary(lookup(),
>> target.libName()).lookup(target.funcName());
>> NativeMethodType nmt = deriveNativeMethodType(target);
>> return abi.downcallHandle(sym, nmt);
>> }
>>
>> In which case NativeMethodType is a nice abstraction to have,
>> otherwise a framework writer will have to write it themselves.
>>
>> The only cases where I expect the first form to appear is when you
>> already know the exact target of the binding when writing the program.
>>
>> Jorn
>>
>> Jorn Vernee schreef op 2018-11-28 16:41:
>>>> Internally, we can have a NativeMethodType abstraction, since for
>>>> the
>>>> binder it seems to make sense - I'm wondering how much that should
>>>> be
>>>> public API, given that low level users will probably see that as a
>>>> somewhat redundant intermediate step.
>>>
>>> Well, if it makes sense for the binder, wouldn't it make sense for
>>> framework writers as well? I'd imagine they'd want to have some kind
>>> of MethodType analog for native binding.
>>>
>>> Jorn
>>>
>>> Maurizio Cimadamore schreef op 2018-11-28 16:32:
>>>> On 28/11/2018 15:27, Jorn Vernee wrote:
>>>>> Looking good!
>>>>>
>>>>>> * having to create a NativeMethodType when I just want to pass the
>>>>>> list of layout types leads to more verbose code
>>>>>>
>>>>>> * we might need NativeTypes.POINTER - doing
>>>>>> NativeTypes.VOID.pointer()
>>>>>> is an explicit workaround, but a tad verbose
>>>>>
>>>>> Reading through the code, I prefer both of these actually, since it
>>>>> makes the code more visually explicit as to what's happening. The
>>>>> `new NativeType(...)` call also gives a visual grouping to the
>>>>> arguments. Though I admit, I like being very descriptive when
>>>>> writing code.
>>>>>
>>>>> Tbh, I mostly expect the NativeMethodType to be constructed
>>>>> automatically, e.g. by deriving the signature from a mangled symbol
>>>>> name, so I'm not sure how much verbosity will be a problem in
>>>>> practice.
>>>>>
>>>>> To save on characters I'd suggest using `import static
>>>>> NativeTypes.*`, and perhaps we could add a convenience overload of
>>>>> upcallStub which doesn't take the target MethodHandle, and instead
>>>>> tries to infer it from the passed receiver object?
>>>>>
>>>>> default UpcallHandle upcallStub(Object receiver,
>>>>> NativeMethodType nmt) {
>>>>> Method m = receiver.getClass().getDeclaredMethods()[0]; //
>>>>> you get the gist
>>>>> return upcallStub(cc, receiver,
>>>>> publicLookup().unreflect(m), nmt);
>>>>> }
>>>>
>>>> It's not about saving characters - I was trying to imagine what a
>>>> typical usage of this API would look like. I assume that people that
>>>> reach low level to SystemABI directly (bypassing the binder)
>>>> basically
>>>> want to be more in control on how the mapping is done. This could be
>>>> a
>>>> powerful options for framework writers. But those users, I believe,
>>>> will almost never use the API point that takes a j.l.Method and a
>>>> Function - they would prefer to specific a list of
>>>> NativeTypes/LayoutTypes directly.
>>>>
>>>> Internally, we can have a NativeMethodType abstraction, since for
>>>> the
>>>> binder it seems to make sense - I'm wondering how much that should
>>>> be
>>>> public API, given that low level users will probably see that as a
>>>> somewhat redundant intermediate step.
>>>>
>>>>>
>>>>> Btw, now that we're starting to have NativeTypes and
>>>>> NativeMethodType maybe we should consider renaming LayoutType to
>>>>> NativeType as well?
>>>>
>>>> Sure, I was thinking along the same lines
>>>>
>>>> Maurizio
>>>>
>>>>>
>>>>> Jorn
>>>>>
>>>>> Maurizio Cimadamore schreef op 2018-11-28 15:00:
>>>>>> Hi,
>>>>>> I took the patch at [1] for a spin with jshell. I wanted to see
>>>>>> how
>>>>>> easy it was to call native functions using the SystemABI interface
>>>>>> -
>>>>>> e.g. w/o binder and annotated interface. I think I'm pleasantly
>>>>>> impressed! I did some changes to the proposed patch as I wanted to
>>>>>> put
>>>>>> everything SystemABI-specific in a public package (java.foreign)
>>>>>> and I
>>>>>> also added a method on UpcallHandle to get a pointer. But overall
>>>>>> these were minor modification, mostly in order to be able to use
>>>>>> this
>>>>>> setup on jshell. But let's dive on some code.
>>>>>>
>>>>>> First, some setup:
>>>>>>
>>>>>>
>>>>>> $ jshell
>>>>>> | Welcome to JShell -- Version 12-internal
>>>>>> | For an introduction type: /help intro
>>>>>>
>>>>>> jshell> import java.foreign.*
>>>>>>
>>>>>> jshell> import java.foreign.memory.*
>>>>>>
>>>>>> jshell> import java.lang.invoke.*
>>>>>>
>>>>>> jshell> var abi = SystemABI.getInstance()
>>>>>> abi ==> jdk.internal.foreign.abi.sysv.x64.SysVx64ABI at 41cf53f9
>>>>>>
>>>>>> Now that we have an ABI, let's try getpid:
>>>>>>
>>>>>>
>>>>>> jshell> var getpid =
>>>>>> abi.downcallHandle(Libraries.getDefaultLibrary().lookup("getpid"),
>>>>>> new
>>>>>> NativeMethodType(NativeTypes.INT32))
>>>>>> getpid ==> MethodHandle()int
>>>>>>
>>>>>> jshell> (int)getpid.invokeExact()
>>>>>> $6 ==> 20308
>>>>>>
>>>>>>
>>>>>> That was easy; let's try a function which actually takes some
>>>>>> args:
>>>>>>
>>>>>>
>>>>>> jshell> var pow =
>>>>>> abi.downcallHandle(Libraries.getDefaultLibrary().lookup("pow"),
>>>>>> new
>>>>>> NativeMethodType(NativeTypes.DOUBLE, NativeTypes.DOUBLE,
>>>>>> NativeTypes.DOUBLE))
>>>>>> pow ==> MethodHandle(double,double)double
>>>>>>
>>>>>> jshell> (double)pow.invokeExact(2d, 3d)
>>>>>> $12 ==> 8.0
>>>>>>
>>>>>>
>>>>>> That works too. Finally, let's see if we can make qsort work -
>>>>>> first
>>>>>> let's declare the comparison logic:
>>>>>>
>>>>>>
>>>>>> jshell> class Comp {
>>>>>> ...> int comp(Pointer<Integer> p1, Pointer<Integer> p2) {
>>>>>> ...> return p1.get() - p2.get();
>>>>>> ...> }
>>>>>> ...> }
>>>>>> | created class Comp
>>>>>>
>>>>>>
>>>>>> And allocate the stub:
>>>>>>
>>>>>>
>>>>>>
>>>>>> jshell> var up = abi.upcallStub(abi.defaultCallingConvention(),
>>>>>> new Comp(),
>>>>>> ...> MethodHandles.lookup().findVirtual(Comp.class, "comp",
>>>>>> MethodType.methodType(int.class, Pointer.class, Pointer.class)),
>>>>>> ...> new
>>>>>> NativeMethodType(NativeTypes.INT32, NativeTypes.INT.pointer(),
>>>>>> NativeTypes.INT.pointer()));
>>>>>> up ==> jdk.internal.foreign.invokers.DirectUpcallHandle at dcf3e99
>>>>>>
>>>>>>
>>>>>> And finally, let's call qsort and print the results.
>>>>>>
>>>>>>
>>>>>> jshell> var qsort =
>>>>>> abi.downcallHandle(Libraries.getDefaultLibrary().lookup("qsort"),
>>>>>> ...> new
>>>>>> NativeMethodType(NativeTypes.VOID,
>>>>>> NativeTypes.INT.pointer(), NativeTypes.INT, NativeTypes.INT,
>>>>>> NativeTypes.VOID.pointer()));
>>>>>> qsort ==> MethodHandle(Pointer,int,int,Pointer)void
>>>>>>
>>>>>> jshell> Scope sc = Scope.newNativeScope()
>>>>>> sc ==> jdk.internal.foreign.ScopeImpl$NativeScope at 5fdef03a
>>>>>>
>>>>>> jshell> var arr = sc.allocateArray(NativeTypes.INT32, new int[] {
>>>>>> 2,
>>>>>> 6, 4, 8, 1, 10 });
>>>>>> arr ==> jdk.internal.foreign.memory.BoundedArray at 311d617d
>>>>>>
>>>>>> jshell> qsort.invoke(arr.elementPointer(), 6, 4, up.entryPoint());
>>>>>> $13 ==> null
>>>>>>
>>>>>> jshell>
>>>>>> System.err.println(Arrays.toString(arr.toArray(int[]::new)));
>>>>>> [1, 2, 4, 6, 8, 10]
>>>>>>
>>>>>> Cool! We have been able to do all this without using a single
>>>>>> annotated interface, which suggests that SystemABI might indeed be
>>>>>> general enough to build on top of it.
>>>>>>
>>>>>> There are, I think, some hiccups in the above snippets which I
>>>>>> found annoying:
>>>>>>
>>>>>> * having to create a NativeMethodType when I just want to pass the
>>>>>> list of layout types leads to more verbose code
>>>>>>
>>>>>> * we might need NativeTypes.POINTER - doing
>>>>>> NativeTypes.VOID.pointer()
>>>>>> is an explicit workaround, but a tad verbose
>>>>>>
>>>>>> * as we already know, the upcallStub API point could use some
>>>>>> simplification: omit the receiver argument and directly return the
>>>>>> pointer; that will also simplify things
>>>>>>
>>>>>> Maurizio
>>>>>>
>>>>>> [1] -
>>>>>> https://cr.openjdk.java.net/~henryjen/panama/SystemABI/webrev.01/webrev/
More information about the panama-dev
mailing list