Growing jextract
Maurizio Cimadamore
maurizio.cimadamore at oracle.com
Sun Feb 18 11:15:15 UTC 2024
Thanks for the reports Manuel.
It seems to me that the very-recently-introduced feature to catch broken
dependencies needs some more ironing out (understandable, as it's just
fresh out of the press).
Overall, you seem to have reported the following issues:
* spurious missing dependency errors for stuff that is not generated
anyway (e.g. missing struct depends on another missing struct)
* no error generated for missing dependencies in array types
* NullPointerException when creating enum constants (it seems like
jextract here is returning a null declaration tree for an enum
constant... seems a very odd situation).
* Bad alignment when creating layouts for packed structs
Is this list correct? Of these, it seems to me the latter two are more
important to fix, as they are clear regression in behavior. The first
two seem more (perhaps annoying) "quality of life" issues with a newly
introduced feature.
Cheers
Maurizio
On 18/02/2024 09:00, Manuel Bleichenbacher wrote:
> With regards to "struct udev". I very much support your last comment
> that this should work without warnings.
>
> "struct udev" is an opaque data structure. The API only uses pointers
> to the struct but never reveals its actual layout as it is an
> implementation detail and can change from version to version. This is
> an established and good pattern. And jextract should be able to handle it.
>
> Regards
> Manuel
>
>
> Am So., 18. Feb. 2024 um 09:53 Uhr schrieb Manuel Bleichenbacher
> <manuel.bleichenbacher at gmail.com>:
>
> Hi Jorn
>
> Regarding the dependencies: Do I get it correctly? If a struct,
> e.g. epoll_event, includes another struct, e.g. epoll_data, I have
> to explicitly include both of them?
>
> If this is the case, it's a bit annoying but doable. It should
> probably be better documented (or did I miss the documentation of
> this fact)?
>
> Anyhow, it leads to the next issue (likely a bug or limitation of
> jextract) with the generated code for epoll_event and epoll_data
> (from <sys/epoll.h>). When the generated class epoll_event is
> accessed for the first time, it crashes with:
>
> Caused by: java.lang.IllegalArgumentException: Invalid alignment
> constraint for member layout:
> [a8(ptr):[*:b1]|i4(fd)|i4(u32)|j8(u64)](data)
> at
> java.base/jdk.internal.foreign.layout.StructLayoutImpl.of(StructLayoutImpl.java:49)
> at
> java.base/java.lang.foreign.MemoryLayout.lambda$structLayout$1(MemoryLayout.java:1063)
> at java.base/jdk.internal.foreign.Utils.wrapOverflow(Utils.java:277)
> at
> java.base/java.lang.foreign.MemoryLayout.structLayout(MemoryLayout.java:1062)
> at
> net.codecrete.usb/net.codecrete.usb.linux.gen.epoll.epoll_event.<clinit>(epoll_event.java:29)
>
> epoll_event is a tricky structure. It looks like so:
>
> typedef union epoll_data
> {
> void *ptr;
> int fd;
> uint32_t u32;
> uint64_t u64;
> } epoll_data_t;
>
> struct epoll_event
> {
> uint32_t events;
> epoll_data_t data;
> } __EPOLL_PACKED;
>
> Because it's packed, the fields data.ptr and data.u64 are not aligned.
>
> It can be made to work if epoll_data.ptr and epoll_data.u64 are
> set to a byte alignment of 4. But that's not actually correct.
> epoll_data is not a packed or unaligned struct. It's only
> unaligned if used within another packed struct.
>
> Regards
> Manuel
>
>
>
>
>
> Am So., 18. Feb. 2024 um 09:34 Uhr schrieb Manuel Bleichenbacher
> <manuel.bleichenbacher at gmail.com>:
>
> Hi Jorn
>
> Thanks for the prompt response. Using the pre-built binaries,
> I've made some progress on Linux but run into new issues. I
> will provide the feedback one by one.
>
> So let's start with the macOS. Here's the output run with verbose.
>
> I also propose to suppress the unrelated warning that occurs
> on all platforms. It's just annoying. And it will send
> newcomers in the wrong direction when they actually have to
> investigate real warnings or errors.
>
> Regards
> Manuel
>
> WARNING: A restricted method in
> java.lang.foreign.AddressLayout has been called
> WARNING: java.lang.foreign.AddressLayout::withTargetLayout has
> been called by org.openjdk.jextract.clang.libclang.Index_h in
> module org.openjdk.jextract
> WARNING: Use --enable-native-access=org.openjdk.jextract to
> avoid a warning for callers in this module
> WARNING: Restricted methods will be blocked in a future
> release unless native access is enabled
>
> Cannot invoke
> "org.openjdk.jextract.Declaration$Constant.name()" because
> "<parameter2>" is null
> java.lang.NullPointerException: Cannot invoke
> "org.openjdk.jextract.Declaration$Constant.name()" because
> "<parameter2>" is null
> at
> org.openjdk.jextract at 22/org.openjdk.jextract.impl.TreeMaker.enumConstantString(TreeMaker.java:532)
> at
> org.openjdk.jextract at 22/org.openjdk.jextract.impl.TreeMaker.lambda$createEnum$7(TreeMaker.java:375)
> at
> java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
> at
> org.openjdk.jextract at 22/org.openjdk.jextract.impl.TreeMaker.createEnum(TreeMaker.java:373)
> at
> org.openjdk.jextract at 22/org.openjdk.jextract.impl.TreeMaker.createTreeInternal(TreeMaker.java:133)
> at
> org.openjdk.jextract at 22/org.openjdk.jextract.impl.TreeMaker.createTree(TreeMaker.java:119)
> at
> org.openjdk.jextract at 22/org.openjdk.jextract.impl.Parser.lambda$parse$2(Parser.java:86)
> at
> org.openjdk.jextract at 22/org.openjdk.jextract.clang.Cursor$CursorChildren.lambda$forEach$1(Cursor.java:261)
> at
> org.openjdk.jextract at 22/org.openjdk.jextract.clang.Cursor$CursorChildren$Context.visit(Cursor.java:234)
> at
> org.openjdk.jextract at 22/org.openjdk.jextract.clang.Cursor$CursorChildren.lambda$static$0(Cursor.java:252)
> at
> org.openjdk.jextract at 22/org.openjdk.jextract.clang.libclang.Index_h.clang_visitChildren(Index_h.java:8275)
> at
> org.openjdk.jextract at 22/org.openjdk.jextract.clang.Cursor$CursorChildren.forEachShortCircuit(Cursor.java:271)
> at
> org.openjdk.jextract at 22/org.openjdk.jextract.clang.Cursor$CursorChildren.forEach(Cursor.java:260)
> at
> org.openjdk.jextract at 22/org.openjdk.jextract.clang.Cursor.forEach(Cursor.java:201)
> at
> org.openjdk.jextract at 22/org.openjdk.jextract.impl.Parser.parse(Parser.java:64)
> at
> org.openjdk.jextract at 22/org.openjdk.jextract.JextractTool.parse(JextractTool.java:119)
> at
> org.openjdk.jextract at 22/org.openjdk.jextract.JextractTool.run(JextractTool.java:516)
> at
> org.openjdk.jextract at 22/org.openjdk.jextract.JextractTool.main(JextractTool.java:218)
>
>
> Am Sa., 17. Feb. 2024 um 20:46 Uhr schrieb Jorn Vernee
> <jorn.vernee at oracle.com>:
>
> One more thing I realized about the warnings you get for
> /usr/include/libudev.h: the structs about which you get
> warnings are not being included (looking at your script,
> there are no --include-struct for them), so there
> shouldn't be any need for us to warn about them either.
>
> We currently check whether things are included first, and
> then mark things that aren't as 'skipped'. However, in the
> later pass which checks for unsupported types, we also
> issue warnings for those skipped elements. I think we
> should just skip issuing warnings for things that are
> marked as skipped by the earlier pass.
>
> Jorn
>
> On 17/02/2024 17:59, Jorn Vernee wrote:
>>
>> Hey Manuel,
>>
>> Thanks for giving jextract 22 a try, and sorry for the
>> frustrating experience.
>>
>> - The NullPointerException is definitely a bug. Jextract
>> should not just report an exception. We'll have a closer
>> look at this on Monday, and see if we can find what code
>> causes the issue. It would help if you could edit the
>> jextract launch script, and add
>> `JLINK_VM_OPTIONS=-Djextract.debug=true`
>> (`JLINK_VM_OPTIONS=` should already be there), and then
>> retry the failed extraction. This should print out more
>> information, including the entire stack trace. that's the
>> part I'm interested in.
>>
>>
>> > I don't understand the error message. What does "xxx
>> depends on yyy which has been excluded" mean? Have I used
>> it incorrectly? The error message mostly mentions types
>> (and possibly functions) that I don't use, neither
>> directly nor indirectly.
>>
>> It is meant to tell the user that they've included
>> something that depends on something else which was not
>> included. This would previously lead to uncompileable
>> code, since FunctionDescriptors and MemoryLayouts now
>> refer directly to the layouts defined in struct classes.
>> For example, if I have a function like `void foo(struct
>> A)`, then the FunctionDescriptor for `foo` will depend
>> directly on the class we generate for `struct A`:
>>
>> FunctionDescriptor.ofVoid(
>> A.layout()
>> );
>>
>> If `A` itself is not included, this code would be
>> uncompileable. So, in the latest version we (try to)
>> issue an error about that.
>>
>> - For the 'ERROR's due to missing dependencies you are
>> seeing: this also looks like a bug. We are currently
>> issuing errors if a _skipped_ symbols depends on a
>> skipped symbol as well, which makes no sense. We can just
>> ignore missing dependencies of skipped things. I've tried
>> to implement a quick fix here [1]. Though, you might be
>> better off using the pre-built binaries for now, which
>> don't have this dependency scanning behavior in the first
>> place (i.e the builds at jdk.java.net/jextract
>> <http://jdk.java.net/jextract>)
>>
>>
>> > And why are udev, udev_list_entry, udev_device etc.
>> skipped? They used to work with jextract 21.
>>
>> - I took a look at libudev.h, and the warnings about
>> unsupported types seem correct to me. e.g. `struct udev`
>> does not have a definition in libudev.h [2], so jextract
>> can not generate a class for it (since the layout and
>> fields are not known). That is all the the warning is
>> trying to say. The other structs about which you get a
>> warning seems to be the same in that they don't have a
>> definition in that file. (FWIW this is one of the areas
>> we put a lot of effort into, so it makes sense that you
>> didn't seen a warning in JDK 21). I think we need to
>> clarify the error message to say more than "is not
>> supported" though.
>>
>>
>> > /usr/include/linux/usbdevice_fs.h: The code generated
>> for the struct usbdevfs_urb does not compile (it refers
>> to a type usbdevfs_iso_packet_desc, which is missing).
>>
>> This is actually the situation we were trying to detect
>> with the ERRORs you encountered. There should have been
>> an error about usbdevfs_iso_packet_desc being excluded,
>> but it seems like we are not scanning the element types
>> of arrays at the moment (linked PR tries to fix this as
>> well).
>>
>>
>> And finally, yeah: we need to update the README on the
>> master branch to reference JDK 22 instead of 21.
>>
>> Thanks for reporting back! While we try to test as much
>> as possible, some issues slip through the cracks, and are
>> only found through user feedback like yours.
>> Jorn
>>
>> [1]: https://github.com/openjdk/jextract/pull/217
>> [2]:
>> https://github.com/systemd/systemd/blob/main/src/libudev/libudev.h#L20C1-L20C13
>>
>> On 17/02/2024 15:40, Manuel Bleichenbacher wrote:
>>> Hi Maurizio
>>>
>>> I have tried to upgrade the JavaDoesUSB library
>>> (https://github.com/manuelbl/JavaDoesUSB) from JDK 21 to
>>> 22, and it was very frustrating. It failed on macOS,
>>> Windows nor Linux. And the problem is jextract.
>>>
>>> (The header files I'm trying the process are operating
>>> system header files.)
>>>
>>>
>>> On macOS, it always crashes:
>>>
>>> FATAL: Unexpected exception
>>> java.lang.NullPointerException: Cannot invoke
>>> "org.openjdk.jextract.Declaration$Constant.name()"
>>> because "<parameter2>" is null occurred
>>>
>>> I've tried different branches (master, panama, jdk22)
>>> and the pre-built binaries. They all behave the same
>>> except the error message is sometimes shorter and starts
>>> at "Cannot invoke..."
>>>
>>> The full commands can be found here (remove "--source"):
>>> https://github.com/manuelbl/JavaDoesUSB/blob/main/java-does-usb/jextract/macos/gen_macos.sh
>>>
>>>
>>> On Linux, jextract worked partially. But for several
>>> files, it refused to generate anything or generated code
>>> that didn't compile:
>>>
>>> /usr/include/fcntl.h:
>>> ERROR: stat depends on timespec which has been excluded
>>> ERROR: stat depends on timespec which has been excluded
>>> ERROR: stat depends on timespec which has been excluded
>>>
>>> /usr/include/libudev.h:
>>> ERROR: __pthread_list_t depends on
>>> __pthread_internal_list which has been excluded
>>> ERROR: __pthread_slist_t depends on
>>> __pthread_internal_slist which has been excluded
>>> ERROR: __pthread_mutex_s depends on
>>> __pthread_internal_list which has been excluded
>>> WARNING: Skipping va_list (type <error: struct
>>> __va_list_tag> is not supported)
>>> WARNING: Skipping __gnuc_va_list (type <error: struct
>>> __va_list_tag> is not supported)
>>> WARNING: Skipping udev (type Declared(udev) is not
>>> supported)
>>> WARNING: Skipping udev_list_entry (type
>>> Declared(udev_list_entry) is not supported)
>>> WARNING: Skipping udev_device (type
>>> Declared(udev_device) is not supported)
>>> WARNING: Skipping udev_monitor (type
>>> Declared(udev_monitor) is not supported)
>>> WARNING: Skipping udev_enumerate (type
>>> Declared(udev_enumerate) is not supported)
>>> WARNING: Skipping udev_queue (type Declared(udev_queue)
>>> is not supported)
>>> WARNING: Skipping udev_hwdb (type Declared(udev_hwdb) is
>>> not supported)
>>>
>>> sys/epoll.h:
>>> ERROR: __pthread_list_t depends on
>>> __pthread_internal_list which has been excluded
>>> ERROR: __pthread_slist_t depends on
>>> __pthread_internal_slist which has been excluded
>>> ERROR: __pthread_mutex_s depends on
>>> __pthread_internal_list which has been excluded
>>> ERROR: epoll_data_t depends on epoll_data which has been
>>> excluded
>>> ERROR: epoll_event depends on epoll_data which has been
>>> excluded
>>>
>>> /usr/include/linux/usbdevice_fs.h:
>>> The code generated for the structusbdevfs_urb does not
>>> compile (it refers to a type usbdevfs_iso_packet_desc,
>>> which is missing).
>>>
>>> The full commands for jextract can be found here. I've
>>> only removed "--source" as it is no longer needed.
>>> https://github.com/manuelbl/JavaDoesUSB/blob/main/java-does-usb/jextract/linux/gen_linux.sh
>>>
>>> All these files could successfully be processed with
>>> jextract 21.
>>>
>>> I don't understand the error message. What does "xxx
>>> depends on yyy which has been excluded" mean? Have I
>>> used it incorrectly? The error message mostly mentions
>>> types (and possibly functions) that I don't use, neither
>>> directly nor indirectly.
>>>
>>> And why are udev, udev_list_entry, udev_device etc.
>>> skipped? They used to work with jextract 21.
>>>
>>>
>>> On Windows, it completely failed as well. jextract
>>> outputs more than 5000 lines of errors and warnings.
>>> Here is just a random selection:
>>>
>>> ERROR: mbstate_t depends on _Mbstatet which has been
>>> excluded
>>> ERROR: LIST_ENTRY depends on _LIST_ENTRY which has been
>>> excluded
>>> ERROR: depends on _M128A which has been excluded
>>> ERROR: WIN32_FIND_STREAM_DATA depends on
>>> _WIN32_FIND_STREAM_DATA which has been excluded
>>> ERROR: tagEXTLOGFONTW depends on tagLOGFONTW which has
>>> been excluded
>>> ERROR: MULTIKEYHELPW depends on tagMULTIKEYHELPW which
>>> has been excluded
>>> WARNING: Skipping MEM_EXTENDED_PARAMETER.Reserved
>>> (bitfields are not supported)
>>> WARNING: Skipping Flags.Reserved (bitfields are not
>>> supported)
>>>
>>> The full command can be found here (remove "--source"):
>>> https://github.com/manuelbl/JavaDoesUSB/blob/main/java-does-usb/jextract/windows/gen_win.cmd
>>>
>>>
>>>
>>> The REAMDE on https://github.com/openjdk/jextract could
>>> also profit from an update:
>>>
>>> The command line "$ sh ./gradlew
>>> -Pjdk21_home=<jdk21_home_dir> ..." sent me on search for
>>> the branch with the JDK 22 code. But it's just the
>>> README that's outdated.
>>>
>>> And the instruction "Run the Gradle build with a Java
>>> version appropriate for the Gradle version. For example,
>>> Gradle 7.5.1 supports JDK 21." sent me down another
>>> rabbit hole. I have the latest Gradle version installed,
>>> which nicely runs with JDK 21. But it failed anyway.
>>> Turns out that the installed gradle is irrelevant but
>>> the gradle wrapper relevant (see command line above).
>>> And the gradle wrapper is at version 7.3.3. So there is
>>> nothing to choose really. You must use JDK 17 (or even
>>> earlier). It has never worked with JDK 21.
>>>
>>>
>>> I would appreciate some feedback. Are these bugs? Or has
>>> jextract become so much more restrictive?
>>>
>>> Regards
>>> Manuel
>>>
>>>
>>>
>>> Am Fr., 16. Feb. 2024 um 16:50 Uhr schrieb Maurizio
>>> Cimadamore <maurizio.cimadamore at oracle.com>:
>>>
>>> Hi,
>>> with JDK 22 near us, we have spent some quality time
>>> with jextract, to
>>> make sure the code it generates is as good as it can
>>> be ahead of the
>>> finalizaton of the FFM API. This resulted in several
>>> changes, both in
>>> the implementation (so, invisible to jextract users)
>>> and in the
>>> generated code, as we cleaned up the translation
>>> strategy to better
>>> adhere with the core principles behind the jextract
>>> tool. These changes
>>> are captured in details in this document:
>>>
>>> https://cr.openjdk.org/~mcimadamore/panama/jextract_changes.html
>>>
>>> It might be a good time to take the latest jextract
>>> for a spin (using
>>> your favourite C library!) and report back, in case
>>> we missed anything.
>>> You can find the latest sources in this branch:
>>>
>>> https://github.com/openjdk/jextract/tree/panama
>>>
>>> Binary snapshots of this newer version are also
>>> available here (note
>>> that MacOS/Arm64 builds is also supported now):
>>>
>>> https://jdk.java.net/jextract/
>>>
>>> Cheers
>>> Maurizio
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jextract-dev/attachments/20240218/60957032/attachment-0001.htm>
More information about the jextract-dev
mailing list