[foreign-jextract] Generated MemoryLayout contains padding instead of fields

Wed Sep 2 20:07:30 UTC 2020

Hi Filip,
I'm afraid this might be an issue we know about - and is ultimately 
caused by clang - e.g. libclang will, if the struct contains an 
unspecified-size array, refuse to give information about _other fields_ 
in that struct. To avoid completely omitting generation of the struct, 
we have a workaround which just emits a struct with some padding (enough 
to fit the struct size) - but it seems the workaround is then causing 
further issues, as jextract will still try to generate struct getters 
and setters even if no field is really there (which I suspect is a 
problem, again, only in source generation mode).

Thanks
Maurizio

On 02/09/2020 21:01, Filip Krakowski wrote:
> Hi,
>
> my last encountered bug seems to be fixed with 
> https://git.openjdk.java.net/panama-foreign/commit/1dc9149d 
> ("constants are made package static final instead of private static 
> final"). The code compiles now but throws an Exception during runtime.
>
>    java.lang.IllegalArgumentException: Bad layout path: cannot resolve
>    'cmsg_len' in layout [x128]
>
>
> I searched for the field and found that the associated MemoryLayout 
> for "cmsghdr" only consists of padding in the generated code.
>
>    static final MemoryLayout cmsghdr$struct$LAYOUT_ =
>    MemoryLayout.ofStruct(
>       MemoryLayout.ofPaddingBits(128)
>    );
>
>    public static jdk.incubator.foreign.MemoryLayout
>    cmsghdr$struct$LAYOUT() { return cmsghdr$struct$LAYOUT_; }
>
>    static final MemoryLayout cmsghdr$cmsg_len$LAYOUT_ = C_LONG;
>    public static jdk.incubator.foreign.MemoryLayout
>    cmsghdr$cmsg_len$LAYOUT() { return cmsghdr$cmsg_len$LAYOUT_; }
>
>    static final VarHandle cmsghdr$cmsg_len$VH_ =
>    cmsghdr$struct$LAYOUT_.varHandle(long.class,
>    MemoryLayout.PathElement.groupElement("cmsg_len"));
>    public static java.lang.invoke.VarHandle cmsghdr$cmsg_len$VH() {
>    return cmsghdr$cmsg_len$VH_; }
>
>    ...
>
>
> According to https://man7.org/linux/man-pages/man3/cmsg.3.html the 
> "cmsghdr" struct looks like this.
>
>    struct cmsghdr {
>       size_t cmsg_len;    /* Data byte count, including header
>                                           (type is socklen_t in POSIX) */
>       int    cmsg_level;  /* Originating protocol */
>       int    cmsg_type;   /* Protocol-specific type */
>       /* followed by
>       unsigned char cmsg_data[]; */
>    };
>
>
> Best regards,
> Filip
> Email Signature
> On 02.09.20 12:30, Filip Krakowski wrote:
>> Hi Jorn,
>>
>> I just gave it a try and it looks good.
>>
>> Now unfortunately I have another bug. A source file (in my case 
>> "constants$3") is referencing a private field in another source file 
>> ("constants$2"). It looks like the VarHandles for the struct got 
>> split in two different source files. I am using the same header file 
>> (https://github.com/linux-rdma/rdma-core/blob/master/libibverbs/verbs.h) 
>> as before.
>>
>> constants$2.java
>>
>>     private static final MemoryLayout
>>     ib_uverbs_alloc_mw_resp$struct$LAYOUT_ =MemoryLayout.ofStruct(
>>          C_INT.withName("mw_handle"),
>>          C_INT.withName("rkey"),
>> MemoryLayout.ofSequence(0,C_LONG).withName("driver_data")
>>     ).withName("ib_uverbs_alloc_mw_resp");
>>     public static jdk.incubator.foreign.MemoryLayout
>>     ib_uverbs_alloc_mw_resp$struct$LAYOUT() {return 
>> ib_uverbs_alloc_mw_resp$struct$LAYOUT_; }
>>
>>     private static final MemoryLayout
>>     ib_uverbs_alloc_mw_resp$mw_handle$LAYOUT_ =C_INT;
>>     public static jdk.incubator.foreign.MemoryLayout
>>     ib_uverbs_alloc_mw_resp$mw_handle$LAYOUT() {return 
>> ib_uverbs_alloc_mw_resp$mw_handle$LAYOUT_; }
>>
>>     private static final VarHandle 
>> ib_uverbs_alloc_mw_resp$mw_handle$VH_ 
>> =ib_uverbs_alloc_mw_resp$struct$LAYOUT_.varHandle(int.class,MemoryLayout.PathElement.groupElement("mw_handle"));
>>     public static java.lang.invoke.VarHandle
>>     ib_uverbs_alloc_mw_resp$mw_handle$VH() {return 
>> ib_uverbs_alloc_mw_resp$mw_handle$VH_; }
>>
>>
>> constants$3.java
>>
>>     private static final MemoryLayout
>>     ib_uverbs_alloc_mw_resp$rkey$LAYOUT_ =C_INT;
>>     public static jdk.incubator.foreign.MemoryLayout
>>     ib_uverbs_alloc_mw_resp$rkey$LAYOUT() {return 
>> ib_uverbs_alloc_mw_resp$rkey$LAYOUT_; }
>>
>>     private static final VarHandle ib_uverbs_alloc_mw_resp$rkey$VH_ 
>> =ib_uverbs_alloc_mw_resp$struct$LAYOUT_.varHandle(int.class,MemoryLayout.PathElement.groupElement("rkey"));
>>     public static java.lang.invoke.VarHandle
>>     ib_uverbs_alloc_mw_resp$rkey$VH() {return 
>> ib_uverbs_alloc_mw_resp$rkey$VH_; }
>>
>>
>> Since "ib_uverbs_alloc_mw_resp$struct$LAYOUT_" is private within 
>> "constants$2" the code does not compile.
>>
>> Best regards,
>> Filip
>>
>> On 01.09.20 15:08, Jorn Vernee wrote:
>>> Hi Filip,
>>>
>>> This issue should now be fixed [1], can you give it a try?
>>>
>>> Thanks,
>>> Jorn
>>>
>>> [1] : https://github.com/openjdk/panama-foreign/commit/afdbc657
>>>
>>> On 31/08/2020 12:35, Filip Krakowski wrote:
>>>> Hi,
>>>>
>>>> I just noticed that jextract's generated getters and setters always 
>>>> access the provided MemorySegment at offset 0. This looks like a 
>>>> bug to me.
>>>>
>>>> This is the struct I used to reproduce this issue.
>>>>
>>>>    struct coordinate {
>>>>       int x;
>>>>       int y;
>>>>    };
>>>>
>>>>
>>>> And this is the code jextract generated from it (source mode).
>>>>
>>>>    private static final MemoryLayout coordinate$struct$LAYOUT_ 
>>>> =MemoryLayout.ofStruct(
>>>>         C_INT.withName("x"),
>>>>         C_INT.withName("y")
>>>>    ).withName("coordinate");
>>>>
>>>>    public static jdk.incubator.foreign.MemoryLayout
>>>>    coordinate$struct$LAYOUT() {return coordinate$struct$LAYOUT_; }
>>>>
>>>>    private static final MemoryLayout coordinate$x$LAYOUT_ =C_INT;
>>>>    public static jdk.incubator.foreign.MemoryLayout 
>>>> coordinate$x$LAYOUT() {return coordinate$x$LAYOUT_; }
>>>>
>>>>    private static final VarHandle coordinate$x$VH_ 
>>>> =coordinate$x$LAYOUT_.varHandle(int.class);
>>>>    public static java.lang.invoke.VarHandle coordinate$x$VH() 
>>>> {return coordinate$x$VH_; }
>>>>
>>>>    private static final MemoryLayout coordinate$y$LAYOUT_ =C_INT;
>>>>    public static jdk.incubator.foreign.MemoryLayout 
>>>> coordinate$y$LAYOUT() {return coordinate$y$LAYOUT_; }
>>>>
>>>>    private static final VarHandle coordinate$y$VH_ 
>>>> =coordinate$y$LAYOUT_.varHandle(int.class);
>>>>    public static java.lang.invoke.VarHandle coordinate$y$VH() 
>>>> {return coordinate$y$VH_; }
>>>>
>>>>
>>>> Here I allocate the struct, set both fields and print them afterwards.
>>>>
>>>>    try (var segment =coordinate.allocate()) {
>>>>         coordinate.x$set(segment,1);
>>>>         coordinate.y$set(segment,2);
>>>>
>>>>         System.out.println(coordinate.x$get(segment));
>>>>         System.out.println(coordinate.y$get(segment));
>>>>    }
>>>>
>>>>
>>>> This will print "2" twice, although "x" should be set to "1" in my 
>>>> understanding.
>>>>
>>>> I am using the latest build (commit 
>>>> 4d7888c040767760b6250130ef6024ea16b43461).
>>>>
>>>>
>>>> Best regards,
>>>> Filip
>>
>