[foreign-jextract] jextract-generated MethodHandle is null

[Maurizio Cimadamore]

On 18/09/2020 21:09, Filip Krakowski wrote:
> Hi,
>
> Got to the bottom of this. "ibv_post_send" is an inline function 
> declared in the "verbs.h" header file 
> (https://github.com/linux-rdma/rdma-core/blob/master/libibverbs/verbs.h#L3222-L3226). 
> Many other functions are also declared inline within this header file. 
> Inline functions will most likely never be supported, right?

Wouldn't say "never" ;-)

Note that, in principle, to support them you need to define an extra 
native function which wraps the function-like macro.

You can do that on your own - e.g. by defining an header that includes 
verbs.h and that adds the required wrapped macros.

Or, you can work with the jextract API - and use it to parse the 
contents of the header file, and then define a plugin to add macro 
wrappers for the ones you need - this will maybe generate another plain 
C header which you can then re-process with the jextract API w/o 
modifications.

A good starting point for the API is the jextract tool itself:

https://github.com/openjdk/panama-foreign/blob/41bd821d179d342b9245b4bc6795d55369811255/src/jdk.incubator.jextract/share/classes/jdk/incubator/jextract/JextractTool.java

This class exposes many static methods which can be used to process the 
contents of header files:

* this does the parsing:

https://github.com/openjdk/panama-foreign/blob/41bd821d179d342b9245b4bc6795d55369811255/src/jdk.incubator.jextract/share/classes/jdk/incubator/jextract/JextractTool.java#L108

* this does the filtering:

https://github.com/openjdk/panama-foreign/blob/41bd821d179d342b9245b4bc6795d55369811255/src/jdk.incubator.jextract/share/classes/jdk/incubator/jextract/JextractTool.java#L113

* this does the code generation

https://github.com/openjdk/panama-foreign/blob/41bd821d179d342b9245b4bc6795d55369811255/src/jdk.incubator.jextract/share/classes/jdk/incubator/jextract/JextractTool.java#L117

* this writes stuff into class/source files

https://github.com/openjdk/panama-foreign/blob/41bd821d179d342b9245b4bc6795d55369811255/src/jdk.incubator.jextract/share/classes/jdk/incubator/jextract/JextractTool.java#L128


Note that the result of the parsing is a relatively high-level AST 
called Declaration - there are basically two parallel hierarchies in the 
jextract API - Declaration and Type; a Declaration models an entity in 
an header file, like a function, or a struct variable, while a Type 
models... well a Type. Both provide a visitor interface so you can use 
the visitor pattern to walk through the contents of what has been parsed.

Parsed value-like macros are represented in the API as 
"Declaration.Constant" nodes:

https://github.com/openjdk/panama-foreign/blob/33bccec768f7dfd3366d79946445940fcbb4783a/src/jdk.incubator.jextract/share/classes/jdk/incubator/jextract/Declaration.java#L255


Unfortunately, at the moment, this is not enough to do your DYI macro 
parsing - we need to expose a better node for macros which not only 
gives you the parsed value, but also gives you the raw string if the 
value could not be parsed - at which point you could decide how to deal 
with those - e.g. by creating additonal Function declarations and 
attaching them to the main tree.

That's the plan, we're not quite there at the moment, but we'll get 
there (as right now the priority is to get the ABI support up and running).

Maurizio


>
> Best regards
> Filip
>
> On 18.09.20 21:44, Filip Krakowski wrote:
>> Hi,
>>
>> please ignore the last mail. Just tried looking up the symbol 
>> manually using LibraryLookup which resulted in an Exception.
>> For some strange reason the symbol "ibv_post_send" is indeed missing 
>> within the shared library.
>>
>>    nm -D libibverbs.so | grep send
>>
>>    0000000000007f70 T ibv_cmd_post_send
>>                      U nl_send_auto
>>                      U nl_send_simple
>>                      U sendto
>>
>>
>> Best regards,
>> Filip
>>
>> On 18.09.20 21:23, Filip Krakowski wrote:
>>> Hi,
>>>
>>> I had some time to code again and found the following bug in 
>>> jextract. Since jextract's source mode is not working at the moment 
>>> (https://bugs.openjdk.java.net/browse/JDK-8252799) I switched to 
>>> class file mode. The following code (only the important parts) was 
>>> generated using source mode for better explanation.
>>>
>>>    static final FunctionDescriptor ibv_post_send$FUNC_ 
>>> =FunctionDescriptor.of(C_INT,
>>>         C_POINTER,
>>>         C_POINTER,
>>>         C_POINTER );
>>>
>>>    static final MethodHandle ibv_post_send$MH_ 
>>> =RuntimeHelper.downcallHandle(
>>>         LIBRARIES,"ibv_post_send",
>>> "(Ljdk/incubator/foreign/MemoryAddress;Ljdk/incubator/foreign/MemoryAddress;Ljdk/incubator/foreign/MemoryAddress;)I", 
>>>
>>>         ibv_post_send$FUNC_,false );
>>>
>>>    public static java.lang.invoke.MethodHandle ibv_post_send$MH() 
>>> {return ibv_post_send$MH_; }
>>>
>>>    public static MethodHandle ibv_post_send$MH() {
>>>         return header_h$constants$0.ibv_post_send$MH();
>>>    }
>>>
>>>    public static @C("int")int ibv_post_send (@C("struct 
>>> ibv_qp*")Addressable qp, at C("struct ibv_send_wr*")Addressable 
>>> wr, at C("struct ibv_send_wr**")Addressable bad_wr) {
>>>         try {
>>>             return 
>>> (int)header_h$constants$0.ibv_post_send$MH().invokeExact(qp.address(), 
>>> wr.address(), bad_wr.address());
>>>         }catch (Throwable ex) {
>>>             throw new AssertionError(ex);
>>>         }
>>>    }
>>>
>>> Calling the generated "ibv_post_send" 
>>> (https://linux.die.net/man/3/ibv_post_send) method results in a 
>>> NullPointerException since RuntimeHelper#downcallHandle returns null 
>>> instead the correct MethodHandle.
>>>
>>>    /java.lang.AssertionError: java.lang.NullPointerException: Cannot
>>>    invoke
>>> "java.lang.invoke.MethodHandle.invokeExact(jdk.incubator.foreign.MemoryAddress, 
>>>
>>>    jdk.incubator.foreign.MemoryAddress,
>>>    jdk.incubator.foreign.MemoryAddress)" because the return value of
>>>    "org.linux.rdma.header_h$constants$0.ibv_post_send$MH()" is null/
>>>
>>> I also tried creating my own MethodHandle using 
>>> RuntimeHelper#downcallHandle which also returned null. 
>>> Interestingly, all previous calls to methods from the same shared 
>>> library work without problems.
>>>
>>> Best regards,
>>> Filip
>>
>