Growing jextract
Manuel Bleichenbacher
manuel.bleichenbacher at gmail.com
Sun Feb 18 08:53:04 UTC 2024
Hi Jorn
Regarding the dependencies: Do I get it correctly? If a struct, e.g.
epoll_event, includes another struct, e.g. epoll_data, I have to explicitly
include both of them?
If this is the case, it's a bit annoying but doable. It should probably be
better documented (or did I miss the documentation of this fact)?
Anyhow, it leads to the next issue (likely a bug or limitation of jextract)
with the generated code for epoll_event and epoll_data (from
<sys/epoll.h>). When the generated class epoll_event is accessed for the
first time, it crashes with:
Caused by: java.lang.IllegalArgumentException: Invalid alignment constraint
for member layout: [a8(ptr):[*:b1]|i4(fd)|i4(u32)|j8(u64)](data)
at
java.base/jdk.internal.foreign.layout.StructLayoutImpl.of(StructLayoutImpl.java:49)
at
java.base/java.lang.foreign.MemoryLayout.lambda$structLayout$1(MemoryLayout.java:1063)
at java.base/jdk.internal.foreign.Utils.wrapOverflow(Utils.java:277)
at
java.base/java.lang.foreign.MemoryLayout.structLayout(MemoryLayout.java:1062)
at
net.codecrete.usb/net.codecrete.usb.linux.gen.epoll.epoll_event.<clinit>(epoll_event.java:29)
epoll_event is a tricky structure. It looks like so:
typedef union epoll_data
{
void *ptr;
int fd;
uint32_t u32;
uint64_t u64;
} epoll_data_t;
struct epoll_event
{
uint32_t events;
epoll_data_t data;
} __EPOLL_PACKED;
Because it's packed, the fields data.ptr and data.u64 are not aligned.
It can be made to work if epoll_data.ptr and epoll_data.u64 are set to a
byte alignment of 4. But that's not actually correct. epoll_data is not a
packed or unaligned struct. It's only unaligned if used within another
packed struct.
Regards
Manuel
Am So., 18. Feb. 2024 um 09:34 Uhr schrieb Manuel Bleichenbacher <
manuel.bleichenbacher at gmail.com>:
> Hi Jorn
>
> Thanks for the prompt response. Using the pre-built binaries, I've made
> some progress on Linux but run into new issues. I will provide the feedback
> one by one.
>
> So let's start with the macOS. Here's the output run with verbose.
>
> I also propose to suppress the unrelated warning that occurs on all
> platforms. It's just annoying. And it will send newcomers in the wrong
> direction when they actually have to investigate real warnings or errors.
>
> Regards
> Manuel
>
>
> WARNING: A restricted method in java.lang.foreign.AddressLayout has been
> called
> WARNING: java.lang.foreign.AddressLayout::withTargetLayout has been called
> by org.openjdk.jextract.clang.libclang.Index_h in module
> org.openjdk.jextract
> WARNING: Use --enable-native-access=org.openjdk.jextract to avoid a
> warning for callers in this module
> WARNING: Restricted methods will be blocked in a future release unless
> native access is enabled
>
> Cannot invoke "org.openjdk.jextract.Declaration$Constant.name()" because
> "<parameter2>" is null
> java.lang.NullPointerException: Cannot invoke
> "org.openjdk.jextract.Declaration$Constant.name()" because "<parameter2>"
> is null
> at org.openjdk.jextract at 22
> /org.openjdk.jextract.impl.TreeMaker.enumConstantString(TreeMaker.java:532)
> at org.openjdk.jextract at 22
> /org.openjdk.jextract.impl.TreeMaker.lambda$createEnum$7(TreeMaker.java:375)
> at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
> at org.openjdk.jextract at 22
> /org.openjdk.jextract.impl.TreeMaker.createEnum(TreeMaker.java:373)
> at org.openjdk.jextract at 22
> /org.openjdk.jextract.impl.TreeMaker.createTreeInternal(TreeMaker.java:133)
> at org.openjdk.jextract at 22
> /org.openjdk.jextract.impl.TreeMaker.createTree(TreeMaker.java:119)
> at org.openjdk.jextract at 22
> /org.openjdk.jextract.impl.Parser.lambda$parse$2(Parser.java:86)
> at org.openjdk.jextract at 22
> /org.openjdk.jextract.clang.Cursor$CursorChildren.lambda$forEach$1(Cursor.java:261)
> at org.openjdk.jextract at 22
> /org.openjdk.jextract.clang.Cursor$CursorChildren$Context.visit(Cursor.java:234)
> at org.openjdk.jextract at 22
> /org.openjdk.jextract.clang.Cursor$CursorChildren.lambda$static$0(Cursor.java:252)
> at org.openjdk.jextract at 22
> /org.openjdk.jextract.clang.libclang.Index_h.clang_visitChildren(Index_h.java:8275)
> at org.openjdk.jextract at 22
> /org.openjdk.jextract.clang.Cursor$CursorChildren.forEachShortCircuit(Cursor.java:271)
> at org.openjdk.jextract at 22
> /org.openjdk.jextract.clang.Cursor$CursorChildren.forEach(Cursor.java:260)
> at org.openjdk.jextract at 22
> /org.openjdk.jextract.clang.Cursor.forEach(Cursor.java:201)
> at org.openjdk.jextract at 22
> /org.openjdk.jextract.impl.Parser.parse(Parser.java:64)
> at org.openjdk.jextract at 22
> /org.openjdk.jextract.JextractTool.parse(JextractTool.java:119)
> at org.openjdk.jextract at 22
> /org.openjdk.jextract.JextractTool.run(JextractTool.java:516)
> at org.openjdk.jextract at 22
> /org.openjdk.jextract.JextractTool.main(JextractTool.java:218)
>
>
> Am Sa., 17. Feb. 2024 um 20:46 Uhr schrieb Jorn Vernee <
> jorn.vernee at oracle.com>:
>
>> One more thing I realized about the warnings you get for
>> /usr/include/libudev.h: the structs about which you get warnings are not
>> being included (looking at your script, there are no --include-struct for
>> them), so there shouldn't be any need for us to warn about them either.
>>
>> We currently check whether things are included first, and then mark
>> things that aren't as 'skipped'. However, in the later pass which checks
>> for unsupported types, we also issue warnings for those skipped elements. I
>> think we should just skip issuing warnings for things that are marked as
>> skipped by the earlier pass.
>>
>> Jorn
>> On 17/02/2024 17:59, Jorn Vernee wrote:
>>
>> Hey Manuel,
>>
>> Thanks for giving jextract 22 a try, and sorry for the frustrating
>> experience.
>>
>> - The NullPointerException is definitely a bug. Jextract should not just
>> report an exception. We'll have a closer look at this on Monday, and see if
>> we can find what code causes the issue. It would help if you could edit the
>> jextract launch script, and add `JLINK_VM_OPTIONS=-Djextract.debug=true`
>> (`JLINK_VM_OPTIONS=` should already be there), and then retry the failed
>> extraction. This should print out more information, including the entire
>> stack trace. that's the part I'm interested in.
>>
>>
>> > I don't understand the error message. What does "xxx depends on yyy
>> which has been excluded" mean? Have I used it incorrectly? The error
>> message mostly mentions types (and possibly functions) that I don't use,
>> neither directly nor indirectly.
>>
>> It is meant to tell the user that they've included something that depends
>> on something else which was not included. This would previously lead to
>> uncompileable code, since FunctionDescriptors and MemoryLayouts now refer
>> directly to the layouts defined in struct classes. For example, if I have a
>> function like `void foo(struct A)`, then the FunctionDescriptor for `foo`
>> will depend directly on the class we generate for `struct A`:
>>
>> FunctionDescriptor.ofVoid(
>> A.layout()
>> );
>>
>> If `A` itself is not included, this code would be uncompileable. So, in
>> the latest version we (try to) issue an error about that.
>>
>> - For the 'ERROR's due to missing dependencies you are seeing: this also
>> looks like a bug. We are currently issuing errors if a _skipped_ symbols
>> depends on a skipped symbol as well, which makes no sense. We can just
>> ignore missing dependencies of skipped things. I've tried to implement a
>> quick fix here [1]. Though, you might be better off using the pre-built
>> binaries for now, which don't have this dependency scanning behavior in the
>> first place (i.e the builds at jdk.java.net/jextract)
>>
>>
>> > And why are udev, udev_list_entry, udev_device etc. skipped? They used
>> to work with jextract 21.
>>
>> - I took a look at libudev.h, and the warnings about unsupported types
>> seem correct to me. e.g. `struct udev` does not have a definition in
>> libudev.h [2], so jextract can not generate a class for it (since the
>> layout and fields are not known). That is all the the warning is trying to
>> say. The other structs about which you get a warning seems to be the same
>> in that they don't have a definition in that file. (FWIW this is one of the
>> areas we put a lot of effort into, so it makes sense that you didn't seen a
>> warning in JDK 21). I think we need to clarify the error message to say
>> more than "is not supported" though.
>>
>>
>> > /usr/include/linux/usbdevice_fs.h: The code generated for the struct
>> usbdevfs_urb does not compile (it refers to a
>> type usbdevfs_iso_packet_desc, which is missing).
>>
>> This is actually the situation we were trying to detect with the ERRORs
>> you encountered. There should have been an error about
>> usbdevfs_iso_packet_desc being excluded, but it seems like we are not
>> scanning the element types of arrays at the moment (linked PR tries to fix
>> this as well).
>>
>>
>> And finally, yeah: we need to update the README on the master branch to
>> reference JDK 22 instead of 21.
>>
>> Thanks for reporting back! While we try to test as much as possible, some
>> issues slip through the cracks, and are only found through user feedback
>> like yours.
>> Jorn
>>
>> [1]: https://github.com/openjdk/jextract/pull/217
>> [2]:
>> https://github.com/systemd/systemd/blob/main/src/libudev/libudev.h#L20C1-L20C13
>> On 17/02/2024 15:40, Manuel Bleichenbacher wrote:
>>
>> Hi Maurizio
>>
>> I have tried to upgrade the JavaDoesUSB library (
>> https://github.com/manuelbl/JavaDoesUSB) from JDK 21 to 22, and it was
>> very frustrating. It failed on macOS, Windows nor Linux. And the problem is
>> jextract.
>>
>> (The header files I'm trying the process are operating system header
>> files.)
>>
>>
>> On macOS, it always crashes:
>>
>> FATAL: Unexpected exception java.lang.NullPointerException: Cannot invoke
>> "org.openjdk.jextract.Declaration$Constant.name()" because "<parameter2>"
>> is null occurred
>>
>> I've tried different branches (master, panama, jdk22) and the pre-built
>> binaries. They all behave the same except the error message is sometimes
>> shorter and starts at "Cannot invoke..."
>>
>> The full commands can be found here (remove "--source"):
>> https://github.com/manuelbl/JavaDoesUSB/blob/main/java-does-usb/jextract/macos/gen_macos.sh
>>
>>
>> On Linux, jextract worked partially. But for several files, it refused to
>> generate anything or generated code that didn't compile:
>>
>> /usr/include/fcntl.h:
>> ERROR: stat depends on timespec which has been excluded
>> ERROR: stat depends on timespec which has been excluded
>> ERROR: stat depends on timespec which has been excluded
>>
>> /usr/include/libudev.h:
>> ERROR: __pthread_list_t depends on __pthread_internal_list which has been
>> excluded
>> ERROR: __pthread_slist_t depends on __pthread_internal_slist which has
>> been excluded
>> ERROR: __pthread_mutex_s depends on __pthread_internal_list which has
>> been excluded
>> WARNING: Skipping va_list (type <error: struct __va_list_tag> is not
>> supported)
>> WARNING: Skipping __gnuc_va_list (type <error: struct __va_list_tag> is
>> not supported)
>> WARNING: Skipping udev (type Declared(udev) is not supported)
>> WARNING: Skipping udev_list_entry (type Declared(udev_list_entry) is not
>> supported)
>> WARNING: Skipping udev_device (type Declared(udev_device) is not
>> supported)
>> WARNING: Skipping udev_monitor (type Declared(udev_monitor) is not
>> supported)
>> WARNING: Skipping udev_enumerate (type Declared(udev_enumerate) is not
>> supported)
>> WARNING: Skipping udev_queue (type Declared(udev_queue) is not supported)
>> WARNING: Skipping udev_hwdb (type Declared(udev_hwdb) is not supported)
>>
>> sys/epoll.h:
>> ERROR: __pthread_list_t depends on __pthread_internal_list which has been
>> excluded
>> ERROR: __pthread_slist_t depends on __pthread_internal_slist which has
>> been excluded
>> ERROR: __pthread_mutex_s depends on __pthread_internal_list which has
>> been excluded
>> ERROR: epoll_data_t depends on epoll_data which has been excluded
>> ERROR: epoll_event depends on epoll_data which has been excluded
>>
>> /usr/include/linux/usbdevice_fs.h:
>> The code generated for the structusbdevfs_urb does not compile (it refers
>> to a type usbdevfs_iso_packet_desc, which is missing).
>>
>> The full commands for jextract can be found here. I've only removed
>> "--source" as it is no longer needed.
>>
>> https://github.com/manuelbl/JavaDoesUSB/blob/main/java-does-usb/jextract/linux/gen_linux.sh
>>
>> All these files could successfully be processed with jextract 21.
>>
>> I don't understand the error message. What does "xxx depends on yyy which
>> has been excluded" mean? Have I used it incorrectly? The error message
>> mostly mentions types (and possibly functions) that I don't use, neither
>> directly nor indirectly.
>>
>> And why are udev, udev_list_entry, udev_device etc. skipped? They used to
>> work with jextract 21.
>>
>>
>> On Windows, it completely failed as well. jextract outputs more than 5000
>> lines of errors and warnings. Here is just a random selection:
>>
>> ERROR: mbstate_t depends on _Mbstatet which has been excluded
>> ERROR: LIST_ENTRY depends on _LIST_ENTRY which has been excluded
>> ERROR: depends on _M128A which has been excluded
>> ERROR: WIN32_FIND_STREAM_DATA depends on _WIN32_FIND_STREAM_DATA which
>> has been excluded
>> ERROR: tagEXTLOGFONTW depends on tagLOGFONTW which has been excluded
>> ERROR: MULTIKEYHELPW depends on tagMULTIKEYHELPW which has been excluded
>> WARNING: Skipping MEM_EXTENDED_PARAMETER.Reserved (bitfields are not
>> supported)
>> WARNING: Skipping Flags.Reserved (bitfields are not supported)
>>
>> The full command can be found here (remove "--source"):
>> https://github.com/manuelbl/JavaDoesUSB/blob/main/java-does-usb/jextract/windows/gen_win.cmd
>>
>>
>>
>> The REAMDE on https://github.com/openjdk/jextract could also profit from
>> an update:
>>
>> The command line "$ sh ./gradlew -Pjdk21_home=<jdk21_home_dir> ..." sent
>> me on search for the branch with the JDK 22 code. But it's just the README
>> that's outdated.
>>
>> And the instruction "Run the Gradle build with a Java version appropriate
>> for the Gradle version. For example, Gradle 7.5.1 supports JDK 21." sent me
>> down another rabbit hole. I have the latest Gradle version installed, which
>> nicely runs with JDK 21. But it failed anyway. Turns out that the installed
>> gradle is irrelevant but the gradle wrapper relevant (see command line
>> above). And the gradle wrapper is at version 7.3.3. So there is nothing to
>> choose really. You must use JDK 17 (or even earlier). It has never worked
>> with JDK 21.
>>
>>
>> I would appreciate some feedback. Are these bugs? Or has jextract become
>> so much more restrictive?
>>
>> Regards
>> Manuel
>>
>>
>>
>> Am Fr., 16. Feb. 2024 um 16:50 Uhr schrieb Maurizio Cimadamore <
>> maurizio.cimadamore at oracle.com>:
>>
>>> Hi,
>>> with JDK 22 near us, we have spent some quality time with jextract, to
>>> make sure the code it generates is as good as it can be ahead of the
>>> finalizaton of the FFM API. This resulted in several changes, both in
>>> the implementation (so, invisible to jextract users) and in the
>>> generated code, as we cleaned up the translation strategy to better
>>> adhere with the core principles behind the jextract tool. These changes
>>> are captured in details in this document:
>>>
>>> https://cr.openjdk.org/~mcimadamore/panama/jextract_changes.html
>>>
>>> It might be a good time to take the latest jextract for a spin (using
>>> your favourite C library!) and report back, in case we missed anything.
>>> You can find the latest sources in this branch:
>>>
>>> https://github.com/openjdk/jextract/tree/panama
>>>
>>> Binary snapshots of this newer version are also available here (note
>>> that MacOS/Arm64 builds is also supported now):
>>>
>>> https://jdk.java.net/jextract/
>>>
>>> Cheers
>>> Maurizio
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jextract-dev/attachments/20240218/9d364d89/attachment-0001.htm>
More information about the jextract-dev
mailing list