RFC: Refactoring SystemABI

[Henry Jen]

The other option I am exploring is simply use Symbol as API point for upcall/downcall. 

Symbol {
	MethodHandle methodHandle();
}

Then we can have Symbol to represent any entry point, and have some sort of JavaLibrary to deal with the symbol and use SystemABI internally.

JavaLibrary {
	Symbol lookup(Method, NativeMethodType);
	Symbol lookup(MethodHandle, NativeMethodType);
        void free(Symbol);
}

This provide consistent view of represent method/variable as a Symbol for both native and Java. 

To expose SystemABI, particularly if we have freeUpcallStub(Pointer) concerns me. As on one hand, we have Scope calling it with free, on the other hand, one can shoot oneself by calling freeUpcallStub. I think it’s better to have single API point. Using Library to manage lifecycle seems more reasonable to me as code stub is part of Library loading.

What do you think?

Cheers,
Henry



> On Dec 3, 2018, at 5:17 PM, Jorn Vernee <jbvernee at xs4all.nl> wrote:
> 
>> 1) pass it as argument to a native function (this is the use case you
>> have in mind)
> 
> I think Henry has a point here; A function that takes a function pointer currently mostly takes a Callback as argument on the Java side (at least that's how jextract models it right?). So to satisfy this scenario, we either need a way to convert a Pointer to a Callback, or return a Callback from SystemABI::upcallStub outright.
> 
>> But Callback is a higher level construct
>> in my view - it requires a Scope and a functional interface, none of
>> which are necessarily available in this low level view. Is there a
>> lower level view to speak about function pointers?
> 
> Java doesn't have function types. The closest to that is a functional interface type, add a pointer to that and you basically have a Callback. The association with a scope only comes from the Pointer being associated with a scope. In a sense I feel like Callback is just a way to put a (pseudo) function type on a Pointer<?>.
> 
> I think Callback is a low-level enough abstraction. Maybe SystemABI::upcallStub could return a `Callback<?>` ? Initially the functional interface class would be null, and we could add a `<R> Callback<R> cast(Class<R> cls);` method to Callback to convert it to another type. The Callback implementation could save the NativeMethodType to check if a cast is valid.
> 
>> I believe one way to solve the issue could be by tweaking the
>> SystemABI to accept a Pointer<?> also as an entry point for a downcall
> 
> This seems like a good idea to me; split the Library.Symbol parameter into `String name` and `Pointer<?> address` parameters, and then make the name parameter optional (with an overload). If we end up going with SystemABI::upcallStub returning a Callback, it already has a way to retrieve the entry point Pointer as well.
> 
>> - we can check the validity of the Pointer in Linux by calling dladdr
> 
> Not all function pointers are associated with a symbol. Symbols of internal functions are entirely optional for native code to have, but you still want to be able to have function pointers for them. There's also generated code which might not have a symbol associated with it. I think trying to validate the pointers is going a step too far.
> 
> Jorn
> 
> Maurizio Cimadamore schreef op 2018-12-04 01:08:
>> On 03/12/2018 22:02, Henry Jen wrote:
>>>> On Dec 3, 2018, at 1:21 PM, Maurizio Cimadamore <maurizio.cimadamore at oracle.com> wrote:
>>>> On 03/12/2018 20:19, Henry Jen wrote:
>>>>> This looks good, I tried to have bound MH before, but because of that test, I decided to keep it as is as I thought that’s what we wanted.
>>>>> By adding the UpcallInvoker, we basically achieve the same thing for Java code invoke callback from Java land, but that add cost to every NativeInvoker allocation, perhaps it should remains in CallbackImplGenerator as I expect this will only be used by binder?
>>>> Not really - if you only interact with SystemABI, you don't even see CallbackImplGenerator. So, SystemABI hands you back a Pointer<?>, how do you call it?
>>> This is what I meant, upcall is for native to call java. This Pointer<?> is only interact with native calls, that is, when pass a function pointer to native API.
>>> For Java to call Java, there is absolutely no reason to go via this path.
>>> In case you get back a function pointer from native call and want to make call into it, isn’ t that suppose to deref the pointer and get the functional interface?
>> There are two possible uses for a Pointer you get back from
>> SystemABI::upcallStub:
>> 1) pass it as argument to a native function (this is the use case you
>> have in mind)
>> 2) pass it as the entry point of a native function
>> I think that, for completeness, both should be supported; let's say I
>> have a native function returning a pointer to function; how do I call
>> that function pointer using SystemABI?
>> Perhaps, you are saying that a function that returns a function
>> pointer will return a Callback object - which then can be used to get
>> to the functional interface. But Callback is a higher level construct
>> in my view - it requires a Scope and a functional interface, none of
>> which are necessarily available in this low level view. Is there a
>> lower level view to speak about function pointers? My bet was that you
>> can do 99% of function pointer just by passing void pointers around,
>> and teaching SystemABI how to deal with 'its own' pointers.
>> Which is almost correct, but, as noted in my previous email, there's a
>> slight mismatch between what is the input of SystemABI::downcallHandle
>> and what's the output of SystemABI::upcallStub (or the return value of
>> a native function returning a function pointer modeled as a
>> Pointer<?>).
>> I believe one way to solve the issue could be by tweaking the
>> SystemABI to accept a Pointer<?> also as an entry point for a downcall
>> - we can check the validity of the Pointer in Linux by calling dladdr:
>> http://man7.org/linux/man-pages/man3/dladdr.3.html
>> Dunno if there is something similar for windows... I found this:
>> https://docs.microsoft.com/en-us/windows/desktop/api/Dbghelp/nf-dbghelp-symfromaddr
>> but seems related to debugging (although that seems to be used by
>> Hotspot as well).
>> Maurizio