[foreign] programmatic access to native libraries w/o binder

Maurizio Cimadamore maurizio.cimadamore at oracle.com
Wed Nov 28 16:17:16 UTC 2018 

I think that's probably true. Although, in the absence of that, your 
deriveNativeMethodType could be rewritten to return a LayoutType[], I guess.

Is there any additional value in reifying the vararg-ness inside the 
NMT? To me that is likely to be what moves the balance in one direction 
rather than another. I'd say that there is some value in having a 
description of the native method type that is self-contained (including 
varargs) - as opposed to pass an extra boolean flag to the SystemABI 
methods. That is the strongest argument I can find for NMT.

Maurizio

On 28/11/2018 16:10, Jorn Vernee wrote:
> I don't imagine framework code to look like this;
>
>    var getpid = 
> abi.downcallHandle(Libraries.getDefaultLibrary().lookup("getpid"), new 
> NativeMethodType(NativeTypes.INT32))
>
> But more like this:
>
>     public MetodHandle bind(BindTarget target) { // some framework type
>         Library.Symbol sym = Libraries.loadLibrary(lookup(), 
> target.libName()).lookup(target.funcName());
>         NativeMethodType nmt = deriveNativeMethodType(target);
>         return abi.downcallHandle(sym, nmt);
>     }
>
> In which case NativeMethodType is a nice abstraction to have, 
> otherwise a framework writer will have to write it themselves.
>
> The only cases where I expect the first form to appear is when you 
> already know the exact target of the binding when writing the program.
>
> Jorn
>
> Jorn Vernee schreef op 2018-11-28 16:41:
>>> Internally, we can have a NativeMethodType abstraction, since for the
>>> binder it seems to make sense - I'm wondering how much that should be
>>> public API, given that low level users will probably see that as a
>>> somewhat redundant intermediate step.
>>
>> Well, if it makes sense for the binder, wouldn't it make sense for
>> framework writers as well? I'd imagine they'd want to have some kind
>> of MethodType analog for native binding.
>>
>> Jorn
>>
>> Maurizio Cimadamore schreef op 2018-11-28 16:32:
>>> On 28/11/2018 15:27, Jorn Vernee wrote:
>>>> Looking good!
>>>>
>>>>> * having to create a NativeMethodType when I just want to pass the
>>>>> list of layout types leads to more verbose code
>>>>>
>>>>> * we might need NativeTypes.POINTER - doing 
>>>>> NativeTypes.VOID.pointer()
>>>>> is an explicit workaround, but a tad verbose
>>>>
>>>> Reading through the code, I prefer both of these actually, since it 
>>>> makes the code more visually explicit as to what's happening. The 
>>>> `new NativeType(...)` call also gives a visual grouping to the 
>>>> arguments. Though I admit, I like being very descriptive when 
>>>> writing code.
>>>>
>>>> Tbh, I mostly expect the NativeMethodType to be constructed 
>>>> automatically, e.g. by deriving the signature from a mangled symbol 
>>>> name, so I'm not sure how much verbosity will be a problem in 
>>>> practice.
>>>>
>>>> To save on characters I'd suggest using `import static 
>>>> NativeTypes.*`, and perhaps we could add a convenience overload of 
>>>> upcallStub which doesn't take the target MethodHandle, and instead 
>>>> tries to infer it from the passed receiver object?
>>>>
>>>>     default UpcallHandle upcallStub(Object receiver, 
>>>> NativeMethodType nmt) {
>>>>         Method m = receiver.getClass().getDeclaredMethods()[0]; // 
>>>> you get the gist
>>>>         return upcallStub(cc, receiver, 
>>>> publicLookup().unreflect(m), nmt);
>>>>     }
>>>
>>> It's not about saving characters - I was trying to imagine what a
>>> typical usage of this API would look like. I assume that people that
>>> reach low level to SystemABI directly (bypassing the binder) basically
>>> want to be more in control on how the mapping is done. This could be a
>>> powerful options for framework writers. But those users, I believe,
>>> will almost never use the API point that takes a j.l.Method and a
>>> Function - they would prefer to specific a list of
>>> NativeTypes/LayoutTypes directly.
>>>
>>> Internally, we can have a NativeMethodType abstraction, since for the
>>> binder it seems to make sense - I'm wondering how much that should be
>>> public API, given that low level users will probably see that as a
>>> somewhat redundant intermediate step.
>>>
>>>>
>>>> Btw, now that we're starting to have NativeTypes and 
>>>> NativeMethodType maybe we should consider renaming LayoutType to 
>>>> NativeType as well?
>>>
>>> Sure, I was thinking along the same lines
>>>
>>> Maurizio
>>>
>>>>
>>>> Jorn
>>>>
>>>> Maurizio Cimadamore schreef op 2018-11-28 15:00:
>>>>> Hi,
>>>>> I took the patch at [1] for a spin with jshell. I wanted to see how
>>>>> easy it was to call native functions using the SystemABI interface -
>>>>> e.g. w/o binder and annotated interface. I think I'm pleasantly
>>>>> impressed! I did some changes to the proposed patch as I wanted to 
>>>>> put
>>>>> everything SystemABI-specific in a public package (java.foreign) 
>>>>> and I
>>>>> also added a method on UpcallHandle to get a pointer. But overall
>>>>> these were minor modification, mostly in order to be able to use this
>>>>> setup on jshell. But let's dive on some code.
>>>>>
>>>>> First, some setup:
>>>>>
>>>>>
>>>>> $ jshell
>>>>> |  Welcome to JShell -- Version 12-internal
>>>>> |  For an introduction type: /help intro
>>>>>
>>>>> jshell> import java.foreign.*
>>>>>
>>>>> jshell> import java.foreign.memory.*
>>>>>
>>>>> jshell> import java.lang.invoke.*
>>>>>
>>>>> jshell> var abi = SystemABI.getInstance()
>>>>> abi ==> jdk.internal.foreign.abi.sysv.x64.SysVx64ABI at 41cf53f9
>>>>>
>>>>> Now that we have an ABI, let's try getpid:
>>>>>
>>>>>
>>>>> jshell> var getpid =
>>>>> abi.downcallHandle(Libraries.getDefaultLibrary().lookup("getpid"), 
>>>>> new
>>>>> NativeMethodType(NativeTypes.INT32))
>>>>> getpid ==> MethodHandle()int
>>>>>
>>>>> jshell> (int)getpid.invokeExact()
>>>>> $6 ==> 20308
>>>>>
>>>>>
>>>>> That was easy; let's try a function which actually takes some args:
>>>>>
>>>>>
>>>>> jshell> var pow =
>>>>> abi.downcallHandle(Libraries.getDefaultLibrary().lookup("pow"), new
>>>>> NativeMethodType(NativeTypes.DOUBLE, NativeTypes.DOUBLE,
>>>>> NativeTypes.DOUBLE))
>>>>> pow ==> MethodHandle(double,double)double
>>>>>
>>>>> jshell> (double)pow.invokeExact(2d, 3d)
>>>>> $12 ==> 8.0
>>>>>
>>>>>
>>>>> That works too. Finally, let's see if we can make qsort work - first
>>>>> let's declare the comparison logic:
>>>>>
>>>>>
>>>>> jshell> class Comp {
>>>>>    ...>       int comp(Pointer<Integer> p1, Pointer<Integer> p2) {
>>>>>    ...>          return p1.get() - p2.get();
>>>>>    ...>       }
>>>>>    ...>    }
>>>>> |  created class Comp
>>>>>
>>>>>
>>>>> And allocate the stub:
>>>>>
>>>>>
>>>>>
>>>>> jshell> var up = abi.upcallStub(abi.defaultCallingConvention(), 
>>>>> new Comp(),
>>>>>    ...> MethodHandles.lookup().findVirtual(Comp.class, "comp",
>>>>> MethodType.methodType(int.class, Pointer.class, Pointer.class)),
>>>>>    ...>                         new
>>>>> NativeMethodType(NativeTypes.INT32, NativeTypes.INT.pointer(),
>>>>> NativeTypes.INT.pointer()));
>>>>> up ==> jdk.internal.foreign.invokers.DirectUpcallHandle at dcf3e99
>>>>>
>>>>>
>>>>> And finally, let's call qsort and print the results.
>>>>>
>>>>>
>>>>> jshell> var qsort =
>>>>> abi.downcallHandle(Libraries.getDefaultLibrary().lookup("qsort"),
>>>>>    ...>                         new 
>>>>> NativeMethodType(NativeTypes.VOID,
>>>>> NativeTypes.INT.pointer(), NativeTypes.INT, NativeTypes.INT,
>>>>> NativeTypes.VOID.pointer()));
>>>>> qsort ==> MethodHandle(Pointer,int,int,Pointer)void
>>>>>
>>>>> jshell> Scope sc = Scope.newNativeScope()
>>>>> sc ==> jdk.internal.foreign.ScopeImpl$NativeScope at 5fdef03a
>>>>>
>>>>> jshell> var arr = sc.allocateArray(NativeTypes.INT32, new int[] { 2,
>>>>> 6, 4, 8, 1, 10 });
>>>>> arr ==> jdk.internal.foreign.memory.BoundedArray at 311d617d
>>>>>
>>>>> jshell> qsort.invoke(arr.elementPointer(), 6, 4, up.entryPoint());
>>>>> $13 ==> null
>>>>>
>>>>> jshell> System.err.println(Arrays.toString(arr.toArray(int[]::new)));
>>>>> [1, 2, 4, 6, 8, 10]
>>>>>
>>>>> Cool! We have been able to do all this without using a single
>>>>> annotated interface, which suggests that SystemABI might indeed be
>>>>> general enough to build on top of it.
>>>>>
>>>>> There are, I think, some hiccups in the above snippets which I 
>>>>> found annoying:
>>>>>
>>>>> * having to create a NativeMethodType when I just want to pass the
>>>>> list of layout types leads to more verbose code
>>>>>
>>>>> * we might need NativeTypes.POINTER - doing 
>>>>> NativeTypes.VOID.pointer()
>>>>> is an explicit workaround, but a tad verbose
>>>>>
>>>>> * as we already know, the upcallStub API point could use some
>>>>> simplification: omit the receiver argument and directly return the
>>>>> pointer; that will also simplify things
>>>>>
>>>>> Maurizio
>>>>>
>>>>> [1] - 
>>>>> https://cr.openjdk.java.net/~henryjen/panama/SystemABI/webrev.01/webrev/ 
>>>>>

More information about the panama-dev mailing list