Growing jextract
Manuel Bleichenbacher
manuel.bleichenbacher at gmail.com
Sun Feb 18 16:56:00 UTC 2024
I've check the bad alignment for packed structs. It's indeed a regression.
The old code looked like so:
static final StructLayout const$0 = MemoryLayout.structLayout(
JAVA_INT.withByteAlignment(1).withName("events"),
MemoryLayout.unionLayout(
RuntimeHelper.POINTER.withByteAlignment(1).withName("ptr"),
JAVA_INT.withByteAlignment(1).withName("fd"),
JAVA_INT.withByteAlignment(1).withName("u32"),
JAVA_LONG.withByteAlignment(1).withName("u64")
).withName("data")
).withName("epoll_event");
Not that it generated an unnamed union for epoll_data even though the union
is named. I guess that's the only way it can be achieved with FFM's
alignment rules, which do not match those of C.
The new code looks like so:
private static final GroupLayout $LAYOUT = MemoryLayout.structLayout(
epoll.C_INT.withByteAlignment(1).withName("events"),
epoll_data.layout().withName("data")
).withName("epoll_event");
But I guess there is no way to make the named epoll_data union forget its
alignment constraints. So something similar to the old version will be
needed.
What is a real-world use case where a developer would profit from the
alignment constraints?
-- Manuel
Am So., 18. Feb. 2024 um 15:00 Uhr schrieb Manuel Bleichenbacher <
manuel.bleichenbacher at gmail.com>:
> Are the bitfield warnings part of the "spurious missing dependency
> errors"? I would prioritize the issues differently:
>
>
> 1. Spurious missing dependency errors: They rendered jextract
> completely useless, on all platforms. I could only make progress because I
> reverted to the older pre-built binaries.
> 2. NullPointerException: In my case, it prevents me from making any
> progress on macOS.
> 3. Bitfield warnings: With Windows.h, they result in more than 1000
> warnings. Now try to imagine a person that has to figure out why jextract
> doesn't do what he/she expects?
> 4. Bad alignment when creating layouts for packed structs: I'm not
> sure this has ever worked. I also had custom code before. The resulting
> error message isn't really helpful.
> 5. no error generated for missing dependencies in array types: It
> results in a compilation error that exactly points at the right location,
> is easy to understand, and the problem is easy to fix
>
>
> 1 and 2 far more important because they have no workaround.
>
> Regards
> Manuel
>
> Am So., 18. Feb. 2024 um 12:15 Uhr schrieb Maurizio Cimadamore <
> maurizio.cimadamore at oracle.com>:
>
>> Thanks for the reports Manuel.
>>
>> It seems to me that the very-recently-introduced feature to catch broken
>> dependencies needs some more ironing out (understandable, as it's just
>> fresh out of the press).
>>
>> Overall, you seem to have reported the following issues:
>>
>> * spurious missing dependency errors for stuff that is not generated
>> anyway (e.g. missing struct depends on another missing struct)
>> * no error generated for missing dependencies in array types
>> * NullPointerException when creating enum constants (it seems like
>> jextract here is returning a null declaration tree for an enum constant...
>> seems a very odd situation).
>> * Bad alignment when creating layouts for packed structs
>>
>> Is this list correct? Of these, it seems to me the latter two are more
>> important to fix, as they are clear regression in behavior. The first two
>> seem more (perhaps annoying) "quality of life" issues with a newly
>> introduced feature.
>>
>> Cheers
>> Maurizio
>>
>>
>> On 18/02/2024 09:00, Manuel Bleichenbacher wrote:
>>
>> With regards to "struct udev". I very much support your last comment that
>> this should work without warnings.
>>
>> "struct udev" is an opaque data structure. The API only uses pointers to
>> the struct but never reveals its actual layout as it is an implementation
>> detail and can change from version to version. This is an established and
>> good pattern. And jextract should be able to handle it.
>>
>> Regards
>> Manuel
>>
>>
>> Am So., 18. Feb. 2024 um 09:53 Uhr schrieb Manuel Bleichenbacher <
>> manuel.bleichenbacher at gmail.com>:
>>
>>> Hi Jorn
>>>
>>> Regarding the dependencies: Do I get it correctly? If a struct, e.g.
>>> epoll_event, includes another struct, e.g. epoll_data, I have to explicitly
>>> include both of them?
>>>
>>> If this is the case, it's a bit annoying but doable. It should probably
>>> be better documented (or did I miss the documentation of this fact)?
>>>
>>> Anyhow, it leads to the next issue (likely a bug or limitation of
>>> jextract) with the generated code for epoll_event and epoll_data (from
>>> <sys/epoll.h>). When the generated class epoll_event is accessed for the
>>> first time, it crashes with:
>>>
>>> Caused by: java.lang.IllegalArgumentException: Invalid alignment
>>> constraint for member layout: [a8(ptr):[*:b1]|i4(fd)|i4(u32)|j8(u64)](data)
>>> at
>>> java.base/jdk.internal.foreign.layout.StructLayoutImpl.of(StructLayoutImpl.java:49)
>>> at
>>> java.base/java.lang.foreign.MemoryLayout.lambda$structLayout$1(MemoryLayout.java:1063)
>>> at java.base/jdk.internal.foreign.Utils.wrapOverflow(Utils.java:277)
>>> at
>>> java.base/java.lang.foreign.MemoryLayout.structLayout(MemoryLayout.java:1062)
>>> at
>>> net.codecrete.usb/net.codecrete.usb.linux.gen.epoll.epoll_event.<clinit>(epoll_event.java:29)
>>>
>>> epoll_event is a tricky structure. It looks like so:
>>>
>>> typedef union epoll_data
>>> {
>>> void *ptr;
>>> int fd;
>>> uint32_t u32;
>>> uint64_t u64;
>>> } epoll_data_t;
>>>
>>> struct epoll_event
>>> {
>>> uint32_t events;
>>> epoll_data_t data;
>>> } __EPOLL_PACKED;
>>>
>>> Because it's packed, the fields data.ptr and data.u64 are not aligned.
>>>
>>> It can be made to work if epoll_data.ptr and epoll_data.u64 are set to a
>>> byte alignment of 4. But that's not actually correct. epoll_data is not a
>>> packed or unaligned struct. It's only unaligned if used within another
>>> packed struct.
>>>
>>> Regards
>>> Manuel
>>>
>>>
>>>
>>>
>>>
>>> Am So., 18. Feb. 2024 um 09:34 Uhr schrieb Manuel Bleichenbacher <
>>> manuel.bleichenbacher at gmail.com>:
>>>
>>>> Hi Jorn
>>>>
>>>> Thanks for the prompt response. Using the pre-built binaries, I've made
>>>> some progress on Linux but run into new issues. I will provide the feedback
>>>> one by one.
>>>>
>>>> So let's start with the macOS. Here's the output run with verbose.
>>>>
>>>> I also propose to suppress the unrelated warning that occurs on all
>>>> platforms. It's just annoying. And it will send newcomers in the wrong
>>>> direction when they actually have to investigate real warnings or errors.
>>>>
>>>> Regards
>>>> Manuel
>>>>
>>>>
>>>> WARNING: A restricted method in java.lang.foreign.AddressLayout has
>>>> been called
>>>> WARNING: java.lang.foreign.AddressLayout::withTargetLayout has been
>>>> called by org.openjdk.jextract.clang.libclang.Index_h in module
>>>> org.openjdk.jextract
>>>> WARNING: Use --enable-native-access=org.openjdk.jextract to avoid a
>>>> warning for callers in this module
>>>> WARNING: Restricted methods will be blocked in a future release unless
>>>> native access is enabled
>>>>
>>>> Cannot invoke "org.openjdk.jextract.Declaration$Constant.name()"
>>>> because "<parameter2>" is null
>>>> java.lang.NullPointerException: Cannot invoke
>>>> "org.openjdk.jextract.Declaration$Constant.name()" because "<parameter2>"
>>>> is null
>>>> at
>>>> org.openjdk.jextract at 22/org.openjdk.jextract.impl.TreeMaker.enumConstantString
>>>> (TreeMaker.java:532)
>>>> at
>>>> org.openjdk.jextract at 22/org.openjdk.jextract.impl.TreeMaker.lambda$createEnum$7
>>>> (TreeMaker.java:375)
>>>> at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
>>>> at
>>>> org.openjdk.jextract at 22/org.openjdk.jextract.impl.TreeMaker.createEnum
>>>> (TreeMaker.java:373)
>>>> at
>>>> org.openjdk.jextract at 22/org.openjdk.jextract.impl.TreeMaker.createTreeInternal
>>>> (TreeMaker.java:133)
>>>> at
>>>> org.openjdk.jextract at 22/org.openjdk.jextract.impl.TreeMaker.createTree
>>>> (TreeMaker.java:119)
>>>> at
>>>> org.openjdk.jextract at 22/org.openjdk.jextract.impl.Parser.lambda$parse$2
>>>> (Parser.java:86)
>>>> at
>>>> org.openjdk.jextract at 22/org.openjdk.jextract.clang.Cursor$CursorChildren.lambda$forEach$1
>>>> (Cursor.java:261)
>>>> at
>>>> org.openjdk.jextract at 22/org.openjdk.jextract.clang.Cursor$CursorChildren$Context.visit
>>>> (Cursor.java:234)
>>>> at
>>>> org.openjdk.jextract at 22/org.openjdk.jextract.clang.Cursor$CursorChildren.lambda$static$0
>>>> (Cursor.java:252)
>>>> at
>>>> org.openjdk.jextract at 22/org.openjdk.jextract.clang.libclang.Index_h.clang_visitChildren
>>>> (Index_h.java:8275)
>>>> at
>>>> org.openjdk.jextract at 22/org.openjdk.jextract.clang.Cursor$CursorChildren.forEachShortCircuit
>>>> (Cursor.java:271)
>>>> at
>>>> org.openjdk.jextract at 22/org.openjdk.jextract.clang.Cursor$CursorChildren.forEach
>>>> (Cursor.java:260)
>>>> at
>>>> org.openjdk.jextract at 22/org.openjdk.jextract.clang.Cursor.forEach
>>>> (Cursor.java:201)
>>>> at
>>>> org.openjdk.jextract at 22/org.openjdk.jextract.impl.Parser.parse
>>>> (Parser.java:64)
>>>> at
>>>> org.openjdk.jextract at 22/org.openjdk.jextract.JextractTool.parse
>>>> (JextractTool.java:119)
>>>> at
>>>> org.openjdk.jextract at 22/org.openjdk.jextract.JextractTool.run
>>>> (JextractTool.java:516)
>>>> at
>>>> org.openjdk.jextract at 22/org.openjdk.jextract.JextractTool.main
>>>> (JextractTool.java:218)
>>>>
>>>>
>>>> Am Sa., 17. Feb. 2024 um 20:46 Uhr schrieb Jorn Vernee <
>>>> jorn.vernee at oracle.com>:
>>>>
>>>>> One more thing I realized about the warnings you get for
>>>>> /usr/include/libudev.h: the structs about which you get warnings are not
>>>>> being included (looking at your script, there are no --include-struct for
>>>>> them), so there shouldn't be any need for us to warn about them either.
>>>>>
>>>>> We currently check whether things are included first, and then mark
>>>>> things that aren't as 'skipped'. However, in the later pass which checks
>>>>> for unsupported types, we also issue warnings for those skipped elements. I
>>>>> think we should just skip issuing warnings for things that are marked as
>>>>> skipped by the earlier pass.
>>>>>
>>>>> Jorn
>>>>> On 17/02/2024 17:59, Jorn Vernee wrote:
>>>>>
>>>>> Hey Manuel,
>>>>>
>>>>> Thanks for giving jextract 22 a try, and sorry for the frustrating
>>>>> experience.
>>>>>
>>>>> - The NullPointerException is definitely a bug. Jextract should not
>>>>> just report an exception. We'll have a closer look at this on Monday, and
>>>>> see if we can find what code causes the issue. It would help if you could
>>>>> edit the jextract launch script, and add
>>>>> `JLINK_VM_OPTIONS=-Djextract.debug=true` (`JLINK_VM_OPTIONS=` should
>>>>> already be there), and then retry the failed extraction. This should print
>>>>> out more information, including the entire stack trace. that's the part I'm
>>>>> interested in.
>>>>>
>>>>>
>>>>> > I don't understand the error message. What does "xxx depends on yyy
>>>>> which has been excluded" mean? Have I used it incorrectly? The error
>>>>> message mostly mentions types (and possibly functions) that I don't use,
>>>>> neither directly nor indirectly.
>>>>>
>>>>> It is meant to tell the user that they've included something that
>>>>> depends on something else which was not included. This would previously
>>>>> lead to uncompileable code, since FunctionDescriptors and MemoryLayouts now
>>>>> refer directly to the layouts defined in struct classes. For example, if I
>>>>> have a function like `void foo(struct A)`, then the FunctionDescriptor for
>>>>> `foo` will depend directly on the class we generate for `struct A`:
>>>>>
>>>>> FunctionDescriptor.ofVoid(
>>>>> A.layout()
>>>>> );
>>>>>
>>>>> If `A` itself is not included, this code would be uncompileable. So,
>>>>> in the latest version we (try to) issue an error about that.
>>>>>
>>>>> - For the 'ERROR's due to missing dependencies you are seeing: this
>>>>> also looks like a bug. We are currently issuing errors if a _skipped_
>>>>> symbols depends on a skipped symbol as well, which makes no sense. We can
>>>>> just ignore missing dependencies of skipped things. I've tried to implement
>>>>> a quick fix here [1]. Though, you might be better off using the pre-built
>>>>> binaries for now, which don't have this dependency scanning behavior in the
>>>>> first place (i.e the builds at jdk.java.net/jextract)
>>>>>
>>>>>
>>>>> > And why are udev, udev_list_entry, udev_device etc. skipped? They
>>>>> used to work with jextract 21.
>>>>>
>>>>> - I took a look at libudev.h, and the warnings about unsupported types
>>>>> seem correct to me. e.g. `struct udev` does not have a definition in
>>>>> libudev.h [2], so jextract can not generate a class for it (since the
>>>>> layout and fields are not known). That is all the the warning is trying to
>>>>> say. The other structs about which you get a warning seems to be the same
>>>>> in that they don't have a definition in that file. (FWIW this is one of the
>>>>> areas we put a lot of effort into, so it makes sense that you didn't seen a
>>>>> warning in JDK 21). I think we need to clarify the error message to say
>>>>> more than "is not supported" though.
>>>>>
>>>>>
>>>>> > /usr/include/linux/usbdevice_fs.h: The code generated for the struct
>>>>> usbdevfs_urb does not compile (it refers to a
>>>>> type usbdevfs_iso_packet_desc, which is missing).
>>>>>
>>>>> This is actually the situation we were trying to detect with the
>>>>> ERRORs you encountered. There should have been an error about
>>>>> usbdevfs_iso_packet_desc being excluded, but it seems like we are not
>>>>> scanning the element types of arrays at the moment (linked PR tries to fix
>>>>> this as well).
>>>>>
>>>>>
>>>>> And finally, yeah: we need to update the README on the master branch
>>>>> to reference JDK 22 instead of 21.
>>>>>
>>>>> Thanks for reporting back! While we try to test as much as possible,
>>>>> some issues slip through the cracks, and are only found through user
>>>>> feedback like yours.
>>>>> Jorn
>>>>>
>>>>> [1]: https://github.com/openjdk/jextract/pull/217
>>>>> [2]:
>>>>> https://github.com/systemd/systemd/blob/main/src/libudev/libudev.h#L20C1-L20C13
>>>>> On 17/02/2024 15:40, Manuel Bleichenbacher wrote:
>>>>>
>>>>> Hi Maurizio
>>>>>
>>>>> I have tried to upgrade the JavaDoesUSB library (
>>>>> https://github.com/manuelbl/JavaDoesUSB) from JDK 21 to 22, and it
>>>>> was very frustrating. It failed on macOS, Windows nor Linux. And the
>>>>> problem is jextract.
>>>>>
>>>>> (The header files I'm trying the process are operating system header
>>>>> files.)
>>>>>
>>>>>
>>>>> On macOS, it always crashes:
>>>>>
>>>>> FATAL: Unexpected exception java.lang.NullPointerException: Cannot
>>>>> invoke "org.openjdk.jextract.Declaration$Constant.name()" because
>>>>> "<parameter2>" is null occurred
>>>>>
>>>>> I've tried different branches (master, panama, jdk22) and the
>>>>> pre-built binaries. They all behave the same except the error message is
>>>>> sometimes shorter and starts at "Cannot invoke..."
>>>>>
>>>>> The full commands can be found here (remove "--source"):
>>>>> https://github.com/manuelbl/JavaDoesUSB/blob/main/java-does-usb/jextract/macos/gen_macos.sh
>>>>>
>>>>>
>>>>> On Linux, jextract worked partially. But for several files, it refused
>>>>> to generate anything or generated code that didn't compile:
>>>>>
>>>>> /usr/include/fcntl.h:
>>>>> ERROR: stat depends on timespec which has been excluded
>>>>> ERROR: stat depends on timespec which has been excluded
>>>>> ERROR: stat depends on timespec which has been excluded
>>>>>
>>>>> /usr/include/libudev.h:
>>>>> ERROR: __pthread_list_t depends on __pthread_internal_list which has
>>>>> been excluded
>>>>> ERROR: __pthread_slist_t depends on __pthread_internal_slist which has
>>>>> been excluded
>>>>> ERROR: __pthread_mutex_s depends on __pthread_internal_list which has
>>>>> been excluded
>>>>> WARNING: Skipping va_list (type <error: struct __va_list_tag> is not
>>>>> supported)
>>>>> WARNING: Skipping __gnuc_va_list (type <error: struct __va_list_tag>
>>>>> is not supported)
>>>>> WARNING: Skipping udev (type Declared(udev) is not supported)
>>>>> WARNING: Skipping udev_list_entry (type Declared(udev_list_entry) is
>>>>> not supported)
>>>>> WARNING: Skipping udev_device (type Declared(udev_device) is not
>>>>> supported)
>>>>> WARNING: Skipping udev_monitor (type Declared(udev_monitor) is not
>>>>> supported)
>>>>> WARNING: Skipping udev_enumerate (type Declared(udev_enumerate) is not
>>>>> supported)
>>>>> WARNING: Skipping udev_queue (type Declared(udev_queue) is not
>>>>> supported)
>>>>> WARNING: Skipping udev_hwdb (type Declared(udev_hwdb) is not supported)
>>>>>
>>>>> sys/epoll.h:
>>>>> ERROR: __pthread_list_t depends on __pthread_internal_list which has
>>>>> been excluded
>>>>> ERROR: __pthread_slist_t depends on __pthread_internal_slist which has
>>>>> been excluded
>>>>> ERROR: __pthread_mutex_s depends on __pthread_internal_list which has
>>>>> been excluded
>>>>> ERROR: epoll_data_t depends on epoll_data which has been excluded
>>>>> ERROR: epoll_event depends on epoll_data which has been excluded
>>>>>
>>>>> /usr/include/linux/usbdevice_fs.h:
>>>>> The code generated for the structusbdevfs_urb does not compile (it
>>>>> refers to a type usbdevfs_iso_packet_desc, which is missing).
>>>>>
>>>>> The full commands for jextract can be found here. I've only removed
>>>>> "--source" as it is no longer needed.
>>>>>
>>>>> https://github.com/manuelbl/JavaDoesUSB/blob/main/java-does-usb/jextract/linux/gen_linux.sh
>>>>>
>>>>> All these files could successfully be processed with jextract 21.
>>>>>
>>>>> I don't understand the error message. What does "xxx depends on yyy
>>>>> which has been excluded" mean? Have I used it incorrectly? The error
>>>>> message mostly mentions types (and possibly functions) that I don't use,
>>>>> neither directly nor indirectly.
>>>>>
>>>>> And why are udev, udev_list_entry, udev_device etc. skipped? They used
>>>>> to work with jextract 21.
>>>>>
>>>>>
>>>>> On Windows, it completely failed as well. jextract outputs more than
>>>>> 5000 lines of errors and warnings. Here is just a random selection:
>>>>>
>>>>> ERROR: mbstate_t depends on _Mbstatet which has been excluded
>>>>> ERROR: LIST_ENTRY depends on _LIST_ENTRY which has been excluded
>>>>> ERROR: depends on _M128A which has been excluded
>>>>> ERROR: WIN32_FIND_STREAM_DATA depends on _WIN32_FIND_STREAM_DATA which
>>>>> has been excluded
>>>>> ERROR: tagEXTLOGFONTW depends on tagLOGFONTW which has been excluded
>>>>> ERROR: MULTIKEYHELPW depends on tagMULTIKEYHELPW which has been
>>>>> excluded
>>>>> WARNING: Skipping MEM_EXTENDED_PARAMETER.Reserved (bitfields are not
>>>>> supported)
>>>>> WARNING: Skipping Flags.Reserved (bitfields are not supported)
>>>>>
>>>>> The full command can be found here (remove "--source"):
>>>>> https://github.com/manuelbl/JavaDoesUSB/blob/main/java-does-usb/jextract/windows/gen_win.cmd
>>>>>
>>>>>
>>>>>
>>>>> The REAMDE on https://github.com/openjdk/jextract could also profit
>>>>> from an update:
>>>>>
>>>>> The command line "$ sh ./gradlew -Pjdk21_home=<jdk21_home_dir> ..."
>>>>> sent me on search for the branch with the JDK 22 code. But it's just the
>>>>> README that's outdated.
>>>>>
>>>>> And the instruction "Run the Gradle build with a Java version
>>>>> appropriate for the Gradle version. For example, Gradle 7.5.1 supports JDK
>>>>> 21." sent me down another rabbit hole. I have the latest Gradle version
>>>>> installed, which nicely runs with JDK 21. But it failed anyway. Turns out
>>>>> that the installed gradle is irrelevant but the gradle wrapper relevant
>>>>> (see command line above). And the gradle wrapper is at version 7.3.3. So
>>>>> there is nothing to choose really. You must use JDK 17 (or even earlier).
>>>>> It has never worked with JDK 21.
>>>>>
>>>>>
>>>>> I would appreciate some feedback. Are these bugs? Or has jextract
>>>>> become so much more restrictive?
>>>>>
>>>>> Regards
>>>>> Manuel
>>>>>
>>>>>
>>>>>
>>>>> Am Fr., 16. Feb. 2024 um 16:50 Uhr schrieb Maurizio Cimadamore <
>>>>> maurizio.cimadamore at oracle.com>:
>>>>>
>>>>>> Hi,
>>>>>> with JDK 22 near us, we have spent some quality time with jextract,
>>>>>> to
>>>>>> make sure the code it generates is as good as it can be ahead of the
>>>>>> finalizaton of the FFM API. This resulted in several changes, both in
>>>>>> the implementation (so, invisible to jextract users) and in the
>>>>>> generated code, as we cleaned up the translation strategy to better
>>>>>> adhere with the core principles behind the jextract tool. These
>>>>>> changes
>>>>>> are captured in details in this document:
>>>>>>
>>>>>> https://cr.openjdk.org/~mcimadamore/panama/jextract_changes.html
>>>>>>
>>>>>> It might be a good time to take the latest jextract for a spin (using
>>>>>> your favourite C library!) and report back, in case we missed
>>>>>> anything.
>>>>>> You can find the latest sources in this branch:
>>>>>>
>>>>>> https://github.com/openjdk/jextract/tree/panama
>>>>>>
>>>>>> Binary snapshots of this newer version are also available here (note
>>>>>> that MacOS/Arm64 builds is also supported now):
>>>>>>
>>>>>> https://jdk.java.net/jextract/
>>>>>>
>>>>>> Cheers
>>>>>> Maurizio
>>>>>>
>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jextract-dev/attachments/20240218/2ff50da3/attachment-0001.htm>
More information about the jextract-dev
mailing list