Growing jextract
Manuel Bleichenbacher
manuel.bleichenbacher at gmail.com
Sun Feb 18 14:00:39 UTC 2024
Are the bitfield warnings part of the "spurious missing dependency errors"?
I would prioritize the issues differently:
1. Spurious missing dependency errors: They rendered jextract completely
useless, on all platforms. I could only make progress because I reverted to
the older pre-built binaries.
2. NullPointerException: In my case, it prevents me from making any
progress on macOS.
3. Bitfield warnings: With Windows.h, they result in more than 1000
warnings. Now try to imagine a person that has to figure out why jextract
doesn't do what he/she expects?
4. Bad alignment when creating layouts for packed structs: I'm not sure
this has ever worked. I also had custom code before. The resulting error
message isn't really helpful.
5. no error generated for missing dependencies in array types: It
results in a compilation error that exactly points at the right location,
is easy to understand, and the problem is easy to fix
1 and 2 far more important because they have no workaround.
Regards
Manuel
Am So., 18. Feb. 2024 um 12:15 Uhr schrieb Maurizio Cimadamore <
maurizio.cimadamore at oracle.com>:
> Thanks for the reports Manuel.
>
> It seems to me that the very-recently-introduced feature to catch broken
> dependencies needs some more ironing out (understandable, as it's just
> fresh out of the press).
>
> Overall, you seem to have reported the following issues:
>
> * spurious missing dependency errors for stuff that is not generated
> anyway (e.g. missing struct depends on another missing struct)
> * no error generated for missing dependencies in array types
> * NullPointerException when creating enum constants (it seems like
> jextract here is returning a null declaration tree for an enum constant...
> seems a very odd situation).
> * Bad alignment when creating layouts for packed structs
>
> Is this list correct? Of these, it seems to me the latter two are more
> important to fix, as they are clear regression in behavior. The first two
> seem more (perhaps annoying) "quality of life" issues with a newly
> introduced feature.
>
> Cheers
> Maurizio
>
>
> On 18/02/2024 09:00, Manuel Bleichenbacher wrote:
>
> With regards to "struct udev". I very much support your last comment that
> this should work without warnings.
>
> "struct udev" is an opaque data structure. The API only uses pointers to
> the struct but never reveals its actual layout as it is an implementation
> detail and can change from version to version. This is an established and
> good pattern. And jextract should be able to handle it.
>
> Regards
> Manuel
>
>
> Am So., 18. Feb. 2024 um 09:53 Uhr schrieb Manuel Bleichenbacher <
> manuel.bleichenbacher at gmail.com>:
>
>> Hi Jorn
>>
>> Regarding the dependencies: Do I get it correctly? If a struct, e.g.
>> epoll_event, includes another struct, e.g. epoll_data, I have to explicitly
>> include both of them?
>>
>> If this is the case, it's a bit annoying but doable. It should probably
>> be better documented (or did I miss the documentation of this fact)?
>>
>> Anyhow, it leads to the next issue (likely a bug or limitation of
>> jextract) with the generated code for epoll_event and epoll_data (from
>> <sys/epoll.h>). When the generated class epoll_event is accessed for the
>> first time, it crashes with:
>>
>> Caused by: java.lang.IllegalArgumentException: Invalid alignment
>> constraint for member layout: [a8(ptr):[*:b1]|i4(fd)|i4(u32)|j8(u64)](data)
>> at
>> java.base/jdk.internal.foreign.layout.StructLayoutImpl.of(StructLayoutImpl.java:49)
>> at
>> java.base/java.lang.foreign.MemoryLayout.lambda$structLayout$1(MemoryLayout.java:1063)
>> at java.base/jdk.internal.foreign.Utils.wrapOverflow(Utils.java:277)
>> at
>> java.base/java.lang.foreign.MemoryLayout.structLayout(MemoryLayout.java:1062)
>> at
>> net.codecrete.usb/net.codecrete.usb.linux.gen.epoll.epoll_event.<clinit>(epoll_event.java:29)
>>
>> epoll_event is a tricky structure. It looks like so:
>>
>> typedef union epoll_data
>> {
>> void *ptr;
>> int fd;
>> uint32_t u32;
>> uint64_t u64;
>> } epoll_data_t;
>>
>> struct epoll_event
>> {
>> uint32_t events;
>> epoll_data_t data;
>> } __EPOLL_PACKED;
>>
>> Because it's packed, the fields data.ptr and data.u64 are not aligned.
>>
>> It can be made to work if epoll_data.ptr and epoll_data.u64 are set to a
>> byte alignment of 4. But that's not actually correct. epoll_data is not a
>> packed or unaligned struct. It's only unaligned if used within another
>> packed struct.
>>
>> Regards
>> Manuel
>>
>>
>>
>>
>>
>> Am So., 18. Feb. 2024 um 09:34 Uhr schrieb Manuel Bleichenbacher <
>> manuel.bleichenbacher at gmail.com>:
>>
>>> Hi Jorn
>>>
>>> Thanks for the prompt response. Using the pre-built binaries, I've made
>>> some progress on Linux but run into new issues. I will provide the feedback
>>> one by one.
>>>
>>> So let's start with the macOS. Here's the output run with verbose.
>>>
>>> I also propose to suppress the unrelated warning that occurs on all
>>> platforms. It's just annoying. And it will send newcomers in the wrong
>>> direction when they actually have to investigate real warnings or errors.
>>>
>>> Regards
>>> Manuel
>>>
>>>
>>> WARNING: A restricted method in java.lang.foreign.AddressLayout has been
>>> called
>>> WARNING: java.lang.foreign.AddressLayout::withTargetLayout has been
>>> called by org.openjdk.jextract.clang.libclang.Index_h in module
>>> org.openjdk.jextract
>>> WARNING: Use --enable-native-access=org.openjdk.jextract to avoid a
>>> warning for callers in this module
>>> WARNING: Restricted methods will be blocked in a future release unless
>>> native access is enabled
>>>
>>> Cannot invoke "org.openjdk.jextract.Declaration$Constant.name()" because
>>> "<parameter2>" is null
>>> java.lang.NullPointerException: Cannot invoke
>>> "org.openjdk.jextract.Declaration$Constant.name()" because "<parameter2>"
>>> is null
>>> at
>>> org.openjdk.jextract at 22/org.openjdk.jextract.impl.TreeMaker.enumConstantString
>>> (TreeMaker.java:532)
>>> at
>>> org.openjdk.jextract at 22/org.openjdk.jextract.impl.TreeMaker.lambda$createEnum$7
>>> (TreeMaker.java:375)
>>> at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
>>> at
>>> org.openjdk.jextract at 22/org.openjdk.jextract.impl.TreeMaker.createEnum
>>> (TreeMaker.java:373)
>>> at
>>> org.openjdk.jextract at 22/org.openjdk.jextract.impl.TreeMaker.createTreeInternal
>>> (TreeMaker.java:133)
>>> at
>>> org.openjdk.jextract at 22/org.openjdk.jextract.impl.TreeMaker.createTree
>>> (TreeMaker.java:119)
>>> at
>>> org.openjdk.jextract at 22/org.openjdk.jextract.impl.Parser.lambda$parse$2
>>> (Parser.java:86)
>>> at
>>> org.openjdk.jextract at 22/org.openjdk.jextract.clang.Cursor$CursorChildren.lambda$forEach$1
>>> (Cursor.java:261)
>>> at
>>> org.openjdk.jextract at 22/org.openjdk.jextract.clang.Cursor$CursorChildren$Context.visit
>>> (Cursor.java:234)
>>> at
>>> org.openjdk.jextract at 22/org.openjdk.jextract.clang.Cursor$CursorChildren.lambda$static$0
>>> (Cursor.java:252)
>>> at
>>> org.openjdk.jextract at 22/org.openjdk.jextract.clang.libclang.Index_h.clang_visitChildren
>>> (Index_h.java:8275)
>>> at
>>> org.openjdk.jextract at 22/org.openjdk.jextract.clang.Cursor$CursorChildren.forEachShortCircuit
>>> (Cursor.java:271)
>>> at
>>> org.openjdk.jextract at 22/org.openjdk.jextract.clang.Cursor$CursorChildren.forEach
>>> (Cursor.java:260)
>>> at
>>> org.openjdk.jextract at 22/org.openjdk.jextract.clang.Cursor.forEach
>>> (Cursor.java:201)
>>> at
>>> org.openjdk.jextract at 22/org.openjdk.jextract.impl.Parser.parse
>>> (Parser.java:64)
>>> at
>>> org.openjdk.jextract at 22/org.openjdk.jextract.JextractTool.parse
>>> (JextractTool.java:119)
>>> at org.openjdk.jextract at 22/org.openjdk.jextract.JextractTool.run
>>> (JextractTool.java:516)
>>> at
>>> org.openjdk.jextract at 22/org.openjdk.jextract.JextractTool.main
>>> (JextractTool.java:218)
>>>
>>>
>>> Am Sa., 17. Feb. 2024 um 20:46 Uhr schrieb Jorn Vernee <
>>> jorn.vernee at oracle.com>:
>>>
>>>> One more thing I realized about the warnings you get for
>>>> /usr/include/libudev.h: the structs about which you get warnings are not
>>>> being included (looking at your script, there are no --include-struct for
>>>> them), so there shouldn't be any need for us to warn about them either.
>>>>
>>>> We currently check whether things are included first, and then mark
>>>> things that aren't as 'skipped'. However, in the later pass which checks
>>>> for unsupported types, we also issue warnings for those skipped elements. I
>>>> think we should just skip issuing warnings for things that are marked as
>>>> skipped by the earlier pass.
>>>>
>>>> Jorn
>>>> On 17/02/2024 17:59, Jorn Vernee wrote:
>>>>
>>>> Hey Manuel,
>>>>
>>>> Thanks for giving jextract 22 a try, and sorry for the frustrating
>>>> experience.
>>>>
>>>> - The NullPointerException is definitely a bug. Jextract should not
>>>> just report an exception. We'll have a closer look at this on Monday, and
>>>> see if we can find what code causes the issue. It would help if you could
>>>> edit the jextract launch script, and add
>>>> `JLINK_VM_OPTIONS=-Djextract.debug=true` (`JLINK_VM_OPTIONS=` should
>>>> already be there), and then retry the failed extraction. This should print
>>>> out more information, including the entire stack trace. that's the part I'm
>>>> interested in.
>>>>
>>>>
>>>> > I don't understand the error message. What does "xxx depends on yyy
>>>> which has been excluded" mean? Have I used it incorrectly? The error
>>>> message mostly mentions types (and possibly functions) that I don't use,
>>>> neither directly nor indirectly.
>>>>
>>>> It is meant to tell the user that they've included something that
>>>> depends on something else which was not included. This would previously
>>>> lead to uncompileable code, since FunctionDescriptors and MemoryLayouts now
>>>> refer directly to the layouts defined in struct classes. For example, if I
>>>> have a function like `void foo(struct A)`, then the FunctionDescriptor for
>>>> `foo` will depend directly on the class we generate for `struct A`:
>>>>
>>>> FunctionDescriptor.ofVoid(
>>>> A.layout()
>>>> );
>>>>
>>>> If `A` itself is not included, this code would be uncompileable. So, in
>>>> the latest version we (try to) issue an error about that.
>>>>
>>>> - For the 'ERROR's due to missing dependencies you are seeing: this
>>>> also looks like a bug. We are currently issuing errors if a _skipped_
>>>> symbols depends on a skipped symbol as well, which makes no sense. We can
>>>> just ignore missing dependencies of skipped things. I've tried to implement
>>>> a quick fix here [1]. Though, you might be better off using the pre-built
>>>> binaries for now, which don't have this dependency scanning behavior in the
>>>> first place (i.e the builds at jdk.java.net/jextract)
>>>>
>>>>
>>>> > And why are udev, udev_list_entry, udev_device etc. skipped? They
>>>> used to work with jextract 21.
>>>>
>>>> - I took a look at libudev.h, and the warnings about unsupported types
>>>> seem correct to me. e.g. `struct udev` does not have a definition in
>>>> libudev.h [2], so jextract can not generate a class for it (since the
>>>> layout and fields are not known). That is all the the warning is trying to
>>>> say. The other structs about which you get a warning seems to be the same
>>>> in that they don't have a definition in that file. (FWIW this is one of the
>>>> areas we put a lot of effort into, so it makes sense that you didn't seen a
>>>> warning in JDK 21). I think we need to clarify the error message to say
>>>> more than "is not supported" though.
>>>>
>>>>
>>>> > /usr/include/linux/usbdevice_fs.h: The code generated for the struct
>>>> usbdevfs_urb does not compile (it refers to a
>>>> type usbdevfs_iso_packet_desc, which is missing).
>>>>
>>>> This is actually the situation we were trying to detect with the ERRORs
>>>> you encountered. There should have been an error about
>>>> usbdevfs_iso_packet_desc being excluded, but it seems like we are not
>>>> scanning the element types of arrays at the moment (linked PR tries to fix
>>>> this as well).
>>>>
>>>>
>>>> And finally, yeah: we need to update the README on the master branch to
>>>> reference JDK 22 instead of 21.
>>>>
>>>> Thanks for reporting back! While we try to test as much as possible,
>>>> some issues slip through the cracks, and are only found through user
>>>> feedback like yours.
>>>> Jorn
>>>>
>>>> [1]: https://github.com/openjdk/jextract/pull/217
>>>> [2]:
>>>> https://github.com/systemd/systemd/blob/main/src/libudev/libudev.h#L20C1-L20C13
>>>> On 17/02/2024 15:40, Manuel Bleichenbacher wrote:
>>>>
>>>> Hi Maurizio
>>>>
>>>> I have tried to upgrade the JavaDoesUSB library (
>>>> https://github.com/manuelbl/JavaDoesUSB) from JDK 21 to 22, and it was
>>>> very frustrating. It failed on macOS, Windows nor Linux. And the problem is
>>>> jextract.
>>>>
>>>> (The header files I'm trying the process are operating system header
>>>> files.)
>>>>
>>>>
>>>> On macOS, it always crashes:
>>>>
>>>> FATAL: Unexpected exception java.lang.NullPointerException: Cannot
>>>> invoke "org.openjdk.jextract.Declaration$Constant.name()" because
>>>> "<parameter2>" is null occurred
>>>>
>>>> I've tried different branches (master, panama, jdk22) and the pre-built
>>>> binaries. They all behave the same except the error message is sometimes
>>>> shorter and starts at "Cannot invoke..."
>>>>
>>>> The full commands can be found here (remove "--source"):
>>>> https://github.com/manuelbl/JavaDoesUSB/blob/main/java-does-usb/jextract/macos/gen_macos.sh
>>>>
>>>>
>>>> On Linux, jextract worked partially. But for several files, it refused
>>>> to generate anything or generated code that didn't compile:
>>>>
>>>> /usr/include/fcntl.h:
>>>> ERROR: stat depends on timespec which has been excluded
>>>> ERROR: stat depends on timespec which has been excluded
>>>> ERROR: stat depends on timespec which has been excluded
>>>>
>>>> /usr/include/libudev.h:
>>>> ERROR: __pthread_list_t depends on __pthread_internal_list which has
>>>> been excluded
>>>> ERROR: __pthread_slist_t depends on __pthread_internal_slist which has
>>>> been excluded
>>>> ERROR: __pthread_mutex_s depends on __pthread_internal_list which has
>>>> been excluded
>>>> WARNING: Skipping va_list (type <error: struct __va_list_tag> is not
>>>> supported)
>>>> WARNING: Skipping __gnuc_va_list (type <error: struct __va_list_tag> is
>>>> not supported)
>>>> WARNING: Skipping udev (type Declared(udev) is not supported)
>>>> WARNING: Skipping udev_list_entry (type Declared(udev_list_entry) is
>>>> not supported)
>>>> WARNING: Skipping udev_device (type Declared(udev_device) is not
>>>> supported)
>>>> WARNING: Skipping udev_monitor (type Declared(udev_monitor) is not
>>>> supported)
>>>> WARNING: Skipping udev_enumerate (type Declared(udev_enumerate) is not
>>>> supported)
>>>> WARNING: Skipping udev_queue (type Declared(udev_queue) is not
>>>> supported)
>>>> WARNING: Skipping udev_hwdb (type Declared(udev_hwdb) is not supported)
>>>>
>>>> sys/epoll.h:
>>>> ERROR: __pthread_list_t depends on __pthread_internal_list which has
>>>> been excluded
>>>> ERROR: __pthread_slist_t depends on __pthread_internal_slist which has
>>>> been excluded
>>>> ERROR: __pthread_mutex_s depends on __pthread_internal_list which has
>>>> been excluded
>>>> ERROR: epoll_data_t depends on epoll_data which has been excluded
>>>> ERROR: epoll_event depends on epoll_data which has been excluded
>>>>
>>>> /usr/include/linux/usbdevice_fs.h:
>>>> The code generated for the structusbdevfs_urb does not compile (it
>>>> refers to a type usbdevfs_iso_packet_desc, which is missing).
>>>>
>>>> The full commands for jextract can be found here. I've only removed
>>>> "--source" as it is no longer needed.
>>>>
>>>> https://github.com/manuelbl/JavaDoesUSB/blob/main/java-does-usb/jextract/linux/gen_linux.sh
>>>>
>>>> All these files could successfully be processed with jextract 21.
>>>>
>>>> I don't understand the error message. What does "xxx depends on yyy
>>>> which has been excluded" mean? Have I used it incorrectly? The error
>>>> message mostly mentions types (and possibly functions) that I don't use,
>>>> neither directly nor indirectly.
>>>>
>>>> And why are udev, udev_list_entry, udev_device etc. skipped? They used
>>>> to work with jextract 21.
>>>>
>>>>
>>>> On Windows, it completely failed as well. jextract outputs more than
>>>> 5000 lines of errors and warnings. Here is just a random selection:
>>>>
>>>> ERROR: mbstate_t depends on _Mbstatet which has been excluded
>>>> ERROR: LIST_ENTRY depends on _LIST_ENTRY which has been excluded
>>>> ERROR: depends on _M128A which has been excluded
>>>> ERROR: WIN32_FIND_STREAM_DATA depends on _WIN32_FIND_STREAM_DATA which
>>>> has been excluded
>>>> ERROR: tagEXTLOGFONTW depends on tagLOGFONTW which has been excluded
>>>> ERROR: MULTIKEYHELPW depends on tagMULTIKEYHELPW which has been excluded
>>>> WARNING: Skipping MEM_EXTENDED_PARAMETER.Reserved (bitfields are not
>>>> supported)
>>>> WARNING: Skipping Flags.Reserved (bitfields are not supported)
>>>>
>>>> The full command can be found here (remove "--source"):
>>>> https://github.com/manuelbl/JavaDoesUSB/blob/main/java-does-usb/jextract/windows/gen_win.cmd
>>>>
>>>>
>>>>
>>>> The REAMDE on https://github.com/openjdk/jextract could also profit
>>>> from an update:
>>>>
>>>> The command line "$ sh ./gradlew -Pjdk21_home=<jdk21_home_dir> ..."
>>>> sent me on search for the branch with the JDK 22 code. But it's just the
>>>> README that's outdated.
>>>>
>>>> And the instruction "Run the Gradle build with a Java version
>>>> appropriate for the Gradle version. For example, Gradle 7.5.1 supports JDK
>>>> 21." sent me down another rabbit hole. I have the latest Gradle version
>>>> installed, which nicely runs with JDK 21. But it failed anyway. Turns out
>>>> that the installed gradle is irrelevant but the gradle wrapper relevant
>>>> (see command line above). And the gradle wrapper is at version 7.3.3. So
>>>> there is nothing to choose really. You must use JDK 17 (or even earlier).
>>>> It has never worked with JDK 21.
>>>>
>>>>
>>>> I would appreciate some feedback. Are these bugs? Or has jextract
>>>> become so much more restrictive?
>>>>
>>>> Regards
>>>> Manuel
>>>>
>>>>
>>>>
>>>> Am Fr., 16. Feb. 2024 um 16:50 Uhr schrieb Maurizio Cimadamore <
>>>> maurizio.cimadamore at oracle.com>:
>>>>
>>>>> Hi,
>>>>> with JDK 22 near us, we have spent some quality time with jextract, to
>>>>> make sure the code it generates is as good as it can be ahead of the
>>>>> finalizaton of the FFM API. This resulted in several changes, both in
>>>>> the implementation (so, invisible to jextract users) and in the
>>>>> generated code, as we cleaned up the translation strategy to better
>>>>> adhere with the core principles behind the jextract tool. These
>>>>> changes
>>>>> are captured in details in this document:
>>>>>
>>>>> https://cr.openjdk.org/~mcimadamore/panama/jextract_changes.html
>>>>>
>>>>> It might be a good time to take the latest jextract for a spin (using
>>>>> your favourite C library!) and report back, in case we missed
>>>>> anything.
>>>>> You can find the latest sources in this branch:
>>>>>
>>>>> https://github.com/openjdk/jextract/tree/panama
>>>>>
>>>>> Binary snapshots of this newer version are also available here (note
>>>>> that MacOS/Arm64 builds is also supported now):
>>>>>
>>>>> https://jdk.java.net/jextract/
>>>>>
>>>>> Cheers
>>>>> Maurizio
>>>>>
>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jextract-dev/attachments/20240218/aa695ef0/attachment-0001.htm>
More information about the jextract-dev
mailing list