jextract generates downcallHandles where I need upcallStubs

Tue Dec 15 12:45:04 UTC 2020

On 15/12/2020 12:13, Sebastian Stenzel wrote:
> Hi Maurizio,
>
> thank you for the detailed response! What you write makes a lot of 
> sense. I got confused by the fact that for each of the fields of said 
> struct there is an equivalent downcall (!) method defined in fuse.h as 
> well [1]. You're right, of course, that I'm free to ignore any 
> generated upcall stub if I don't need it.
I noticed the downcall later - with quite a similar name, and got 
similarly puzzled by it! But yes, they are different entities.
>
> Regarding the fields in a struct: If I understand you correctly, there 
> is (currently) no way other than to manually create a stub for such 
> methods.
>
> I have no idea how such structs look like for the compiler and if the 
> "type information" for such fields get lost, but is there any chance, 
> jextract can generate a FunctionDescriptor when encountering _any_ 
> function-like expression? This would not only save a lot of time and 
> prevent mistakes, but would fail early at compile time after 
> re-extracting FunctionDescriptors in case of an api-breaking upstream 
> library update.

Aha - now I get the issue - back on my simple example:

```
cat foo.h
struct Foo {
   int (*foo)(float, double);
};
```

While I see that jextrract is generating getter/setters for the struct, 
I don't see the usual functional interface being generated - which would 
allow you to create a callback in a very straightforward fashion (e.g. 
with a lambda expression).

This is, I think, the bit of ingredient that's missing - and I think 
it's a plain bug: while jextract looks at regular function to see if 
their signature mentions a function pointer, the same is not done for 
struct fields.

We'll fix this.

Thanks for bearing with me.

Maurizio

>
> Thanks again!
> Sebastian
>
> [1]: 
> https://github.com/libfuse/libfuse/blob/fuse_2_9_5/include/fuse.h#L817 
> <https://urldefense.com/v3/__https://github.com/libfuse/libfuse/blob/fuse_2_9_5/include/fuse.h*L817__;Iw!!GqivPVa7Brio!OLoMSc1W4YZlzJZB1Au71dTyx1Vi2tSn9hC4cFFNxN9lonhyi4U6BpH9LNsOpC4lBJaiuRI$>
>
>> On 15. Dec 2020, at 12:31, Maurizio Cimadamore 
>> <maurizio.cimadamore at oracle.com 
>> <mailto:maurizio.cimadamore at oracle.com>> wrote:
>>
>> Hi Sebastian,
>> I had a look at the header you mention, and I also tried to extract 
>> it on my machine.
>>
>> What I see in that header is a struct, which defines several function 
>> pointer fields.
>>
>> Let's focus on a simpler example:
>>
>> ```
>> cat foo.h
>> struct Foo {
>>   int (*foo)(float, double);
>> };
>> ```
>>
>> When extracting such an header, jextract will see a struct with one 
>> field - only, that field will have a function pointer type. But it's 
>> still a field - so jextract will generate getter/setters for that 
>> field. You get no downcall at all for this - this is not a function - 
>> there is no shared symbol for it anywhere, so talking about downcall 
>> here is inappropriate.
>>
>> With fuse.h the same thing is happening - and I see a lot of 
>> getters/setters being generated for the various pieces of 
>> functionality declared in the fuse_operations struct - e.g:
>>
>> ```
>> public static  MemoryAddress readdir$get(MemorySegment seg) { ... }
>>
>> public static void readdir$set(MemorySegment seg, MemoryAddress x) { 
>> ... }
>>
>> ```
>>
>> This means that the "readdir" field of fuse_operation can be get/set 
>> using these accessors; to do both get/set you need to pass the memory 
>> segment corresponding to fuse_operation (the first argument in the 
>> accessor). Note that the type to get/set is a MemoryAddress, which is 
>> the readdir function pointer.
>>
>> So, the way this works is (I think):
>>
>> 1) you create a readdir implementation using upcallStub - this gives 
>> you a segment
>> 2) you set the above implementation on fuse_operation.readdir, using 
>> the set accessor above (to turn the segment into an address, just 
>> call the MemorySegment::address method)
>>
>> The same structure seems to be present here:
>>
>> https://github.com/libfuse/libfuse/blob/fuse_2_9_5/example/fusexmp.c 
>> <https://urldefense.com/v3/__https://github.com/libfuse/libfuse/blob/fuse_2_9_5/example/fusexmp.c__;!!GqivPVa7Brio!OLoMSc1W4YZlzJZB1Au71dTyx1Vi2tSn9hC4cFFNxN9lonhyi4U6BpH9LNsOpC4l0Y0tSk0$>
>>
>> Where a struct fuse_operation is allocated and then all its fields 
>> are set to some functions declared in that example. If you want to do 
>> the same using Panama you will have to create all the function 
>> pointers to set on fuse_operations using CLinker::upcallStub.
>>
>> Now, it's true that the function pointer type behind "readdir" takes 
>> its own function pointer (filler), but that shouldn't be a concern 
>> when creating the upcall for (1) - a function pointer is just some 
>> MemoryAddress parameter that the upcall receives (and that the 
>> runtime of the libfuse library will provide). And, it is true that, 
>> in this case jextract will also generate functional interfaces for 
>> e.g. the filler upcall, but you don't have to use them in this case 
>> (since the filler function pointer will never be generated by you - 
>> but by the library) - of course jextract sees a function pointer and 
>> generates the corresponding functional interface, it has no way to 
>> know which party will allocate the function pointer.
>>
>> I hope this helps.
>>
>> Maurizio
>>
>>
>>
>> On 15/12/2020 08:50, Sebastian Stenzel wrote:
>>> Hi all,
>>>
>>> I'm pretty new to this topic and therefore I may misunderstand what 
>>> is going on here, but I'm pretty certain that jextract swaps 
>>> downcalls and upcalls in some scenarios:
>>>
>>> I jextracted [0] libfuse, which mainly consists of the 
>>> fuse_operations struct [1], which contains a set of file system 
>>> operations that need to be implemented by library users. These 
>>> methods are then called by fuse. I.e. this is a typical upcall 
>>> scenario (native code calls java code). Or am I mistaken?
>>>
>>> However, jextract thinks otherwise: While I get correct 
>>> downcallHandles for methods declared on "top level" in the .h file, 
>>> such as fuse_main() [2], everything inside of aforementioned struct 
>>> is reversed. Take for example readdir() [3], which gets called by 
>>> fuse, so the library user can list directory contents. For this 
>>> purpuse, fuse passes a "filler" callback as a parameter to this 
>>> function. But with jextract will generate a downcallHandle for 
>>> readdir and an upcallStub for the filler.
>>>
>>> Tested with build 16-panama+3-385.
>>>
>>> Might it be possible that guessing the "direction" of a call is not 
>>> yet supported for methods inside of structs? Is it even _possible_ 
>>> to always predict this direction correctly? If not, is this too much 
>>> of a restriction and shouldn't it rather be the developers choice 
>>> what to do with a generated FunctionDescriptor?
>>>
>>> Regards,
>>> Sebastian
>>>
>>> [0]: jextract --source -l fuse -d someDir -t com.example.pkg 
>>> -C-D_FILE_OFFSET_BITS=64 -C-DFUSE_USE_VERSION=29 --filter fuse.h 
>>> /usr/local/include/fuse/fuse.h
>>> [1]:https://github.com/libfuse/libfuse/blob/fuse_2_9_5/include/fuse.h#L88 
>>> <https://urldefense.com/v3/__https://github.com/libfuse/libfuse/blob/fuse_2_9_5/include/fuse.h*L88__;Iw!!GqivPVa7Brio!OLoMSc1W4YZlzJZB1Au71dTyx1Vi2tSn9hC4cFFNxN9lonhyi4U6BpH9LNsOpC4lwfG-bfs$><https://github.com/libfuse/libfuse/blob/fuse_2_9_5/include/fuse.h#L88 
>>> <https://urldefense.com/v3/__https://github.com/libfuse/libfuse/blob/fuse_2_9_5/include/fuse.h*L88__;Iw!!GqivPVa7Brio!OLoMSc1W4YZlzJZB1Au71dTyx1Vi2tSn9hC4cFFNxN9lonhyi4U6BpH9LNsOpC4lwfG-bfs$>>
>>> [2]:https://github.com/libfuse/libfuse/blob/fuse_2_9_5/include/fuse.h#L766 
>>> <https://urldefense.com/v3/__https://github.com/libfuse/libfuse/blob/fuse_2_9_5/include/fuse.h*L766__;Iw!!GqivPVa7Brio!OLoMSc1W4YZlzJZB1Au71dTyx1Vi2tSn9hC4cFFNxN9lonhyi4U6BpH9LNsOpC4lkJHXQlQ$><https://github.com/libfuse/libfuse/blob/fuse_2_9_5/include/fuse.h#L766 
>>> <https://urldefense.com/v3/__https://github.com/libfuse/libfuse/blob/fuse_2_9_5/include/fuse.h*L766__;Iw!!GqivPVa7Brio!OLoMSc1W4YZlzJZB1Au71dTyx1Vi2tSn9hC4cFFNxN9lonhyi4U6BpH9LNsOpC4lkJHXQlQ$>>
>>> [3]:https://github.com/libfuse/libfuse/blob/fuse_2_9_5/include/fuse.h#L304 
>>> <https://urldefense.com/v3/__https://github.com/libfuse/libfuse/blob/fuse_2_9_5/include/fuse.h*L304__;Iw!!GqivPVa7Brio!OLoMSc1W4YZlzJZB1Au71dTyx1Vi2tSn9hC4cFFNxN9lonhyi4U6BpH9LNsOpC4li8C-7DA$><https://github.com/libfuse/libfuse/blob/fuse_2_9_5/include/fuse.h#L304 
>>> <https://urldefense.com/v3/__https://github.com/libfuse/libfuse/blob/fuse_2_9_5/include/fuse.h*L304__;Iw!!GqivPVa7Brio!OLoMSc1W4YZlzJZB1Au71dTyx1Vi2tSn9hC4cFFNxN9lonhyi4U6BpH9LNsOpC4li8C-7DA$>>
>