RFC: Refactoring SystemABI

Tue Dec 4 11:22:20 UTC 2018

On 04/12/2018 01:17, Jorn Vernee wrote:
>> 1) pass it as argument to a native function (this is the use case you
>> have in mind)
>
> I think Henry has a point here; A function that takes a function 
> pointer currently mostly takes a Callback as argument on the Java side 
> (at least that's how jextract models it right?). So to satisfy this 
> scenario, we either need a way to convert a Pointer to a Callback, or 
> return a Callback from SystemABI::upcallStub outright.

Yes and no - callback is mostly a fiction here. As demonstrated in an 
email I sent few days ago, you can just take the pointer and pass it to 
the native function and you can call qsort just fine.

I think we should try to keep this as low level as possible - just 
Pointers, LayoutType, NativeMethodType, Symbol and very little else.

You can think of all the stuff that we normally do with jextract as 
'built on top' e.g. if you have a special LayouType with a special 
getter/setter MH pair which converts a blob of bytes into an annotated 
struct interface or annotated callback, fine - but that's higher up.

>
>> But Callback is a higher level construct
>> in my view - it requires a Scope and a functional interface, none of
>> which are necessarily available in this low level view. Is there a
>> lower level view to speak about function pointers?
>
> Java doesn't have function types. The closest to that is a functional 
> interface type, add a pointer to that and you basically have a 
> Callback. The association with a scope only comes from the Pointer 
> being associated with a scope. In a sense I feel like Callback is just 
> a way to put a (pseudo) function type on a Pointer<?>.
Function pointers is one thing - for that Pointer<?> is more than 
enough; if you want to *call* the function pointer, what do you do in C? 
You take the function pointer and you call it - meaning that the 
function pointer is a valid entry point; to me this means that there 
must be a way to play back a (function) Pointer into the 
SystemABI::downcallHandle machinery.
>
> I think Callback is a low-level enough abstraction. Maybe 
> SystemABI::upcallStub could return a `Callback<?>` ? Initially the 
> functional interface class would be null, and we could add a `<R> 
> Callback<R> cast(Class<R> cls);` method to Callback to convert it to 
> another type. The Callback implementation could save the 
> NativeMethodType to check if a cast is valid.
I disagree here - different frameworks will have different ways to model 
callbacks/function pointers. Our Callback interface is just a way to do 
that. I don't think at this level we want to impose more semantic 
mismatch than what's absolutely necessary.
>
>> I believe one way to solve the issue could be by tweaking the
>> SystemABI to accept a Pointer<?> also as an entry point for a downcall
>
> This seems like a good idea to me; split the Library.Symbol parameter 
> into `String name` and `Pointer<?> address` parameters, and then make 
> the name parameter optional (with an overload). If we end up going 
> with SystemABI::upcallStub returning a Callback, it already has a way 
> to retrieve the entry point Pointer as well.

Sure, callback works - and maybe there are cases where the user will 
want to use a return LayoutType that creates a Callback - fine. I think 
the point of contention here is what should SystemABI::upcallHandle 
return - everything else is customizable at higher or lower level (or 
should be) by picking the right LayoutType with the right carrier, MH pairs.

And I think the answer is that upcallStub should return a Symbol. After 
all what you just created is a native function - you can even call it, 
if you want. So what you get back should go right back into the downcall 
machinery. Note that if you return Callback you still have an issue - as 
you cannot pass a Callback as the entry point of SystemABI::downcallHandle.

To conclude, there are two problems here:

1) SystemABI::upcallStub should return something that is consistent with 
the type used to describe the entry point of SystemABI::downcallHandle, 
or we have a mismatch. A possible alternative is to allow a conversion 
from whatever SystemABI::upcallStub returns and whatever 
SystemABI::downcallHandle accepts (e.g. Pointer -> Symbol). But I 
honestly prefer the former.

2) When passing and returning function pointers to native function, the 
user of SystemABI is free to choose the level of abstraction he wants to 
work at. If it's just pointers (as I demonstrated in my qsort example) 
fine, if it's something more, the user can pick a pre-baked LayoutType 
(e.g. the Callback one) and use that instead. Or he can build a new, 
custom one.

I think this leaves us with a lot of flexibility and configurability.

>
>> - we can check the validity of the Pointer in Linux by calling dladdr
>
> Not all function pointers are associated with a symbol. Symbols of 
> internal functions are entirely optional for native code to have, but 
> you still want to be able to have function pointers for them. There's 
> also generated code which might not have a symbol associated with it. 
> I think trying to validate the pointers is going a step too far.

Right, this occurred to me as well.

Maurizio

>
> Jorn
>
> Maurizio Cimadamore schreef op 2018-12-04 01:08:
>> On 03/12/2018 22:02, Henry Jen wrote:
>>>
>>>> On Dec 3, 2018, at 1:21 PM, Maurizio Cimadamore 
>>>> <maurizio.cimadamore at oracle.com> wrote:
>>>>
>>>>
>>>> On 03/12/2018 20:19, Henry Jen wrote:
>>>>> This looks good, I tried to have bound MH before, but because of 
>>>>> that test, I decided to keep it as is as I thought that’s what we 
>>>>> wanted.
>>>>>
>>>>> By adding the UpcallInvoker, we basically achieve the same thing 
>>>>> for Java code invoke callback from Java land, but that add cost to 
>>>>> every NativeInvoker allocation, perhaps it should remains in 
>>>>> CallbackImplGenerator as I expect this will only be used by binder?
>>>> Not really - if you only interact with SystemABI, you don't even 
>>>> see CallbackImplGenerator. So, SystemABI hands you back a 
>>>> Pointer<?>, how do you call it?
>>>>
>>> This is what I meant, upcall is for native to call java. This 
>>> Pointer<?> is only interact with native calls, that is, when pass a 
>>> function pointer to native API.
>>>
>>> For Java to call Java, there is absolutely no reason to go via this 
>>> path.
>>>
>>> In case you get back a function pointer from native call and want to 
>>> make call into it, isn’ t that suppose to deref the pointer and get 
>>> the functional interface?
>>
>> There are two possible uses for a Pointer you get back from
>> SystemABI::upcallStub:
>>
>> 1) pass it as argument to a native function (this is the use case you
>> have in mind)
>>
>> 2) pass it as the entry point of a native function
>>
>> I think that, for completeness, both should be supported; let's say I
>> have a native function returning a pointer to function; how do I call
>> that function pointer using SystemABI?
>>
>> Perhaps, you are saying that a function that returns a function
>> pointer will return a Callback object - which then can be used to get
>> to the functional interface. But Callback is a higher level construct
>> in my view - it requires a Scope and a functional interface, none of
>> which are necessarily available in this low level view. Is there a
>> lower level view to speak about function pointers? My bet was that you
>> can do 99% of function pointer just by passing void pointers around,
>> and teaching SystemABI how to deal with 'its own' pointers.
>>
>> Which is almost correct, but, as noted in my previous email, there's a
>> slight mismatch between what is the input of SystemABI::downcallHandle
>> and what's the output of SystemABI::upcallStub (or the return value of
>> a native function returning a function pointer modeled as a
>> Pointer<?>).
>>
>> I believe one way to solve the issue could be by tweaking the
>> SystemABI to accept a Pointer<?> also as an entry point for a downcall
>> - we can check the validity of the Pointer in Linux by calling dladdr:
>>
>> http://man7.org/linux/man-pages/man3/dladdr.3.html
>>
>> Dunno if there is something similar for windows... I found this:
>>
>> https://docs.microsoft.com/en-us/windows/desktop/api/Dbghelp/nf-dbghelp-symfromaddr 
>>
>>
>> but seems related to debugging (although that seems to be used by
>> Hotspot as well).
>>
>> Maurizio