Growing jextract
Jorn Vernee
jorn.vernee at oracle.com
Sat Feb 17 16:59:51 UTC 2024
Hey Manuel,
Thanks for giving jextract 22 a try, and sorry for the frustrating
experience.
- The NullPointerException is definitely a bug. Jextract should not just
report an exception. We'll have a closer look at this on Monday, and see
if we can find what code causes the issue. It would help if you could
edit the jextract launch script, and add
`JLINK_VM_OPTIONS=-Djextract.debug=true` (`JLINK_VM_OPTIONS=` should
already be there), and then retry the failed extraction. This should
print out more information, including the entire stack trace. that's the
part I'm interested in.
> I don't understand the error message. What does "xxx depends on yyy
which has been excluded" mean? Have I used it incorrectly? The error
message mostly mentions types (and possibly functions) that I don't use,
neither directly nor indirectly.
It is meant to tell the user that they've included something that
depends on something else which was not included. This would previously
lead to uncompileable code, since FunctionDescriptors and MemoryLayouts
now refer directly to the layouts defined in struct classes. For
example, if I have a function like `void foo(struct A)`, then the
FunctionDescriptor for `foo` will depend directly on the class we
generate for `struct A`:
FunctionDescriptor.ofVoid(
A.layout()
);
If `A` itself is not included, this code would be uncompileable. So, in
the latest version we (try to) issue an error about that.
- For the 'ERROR's due to missing dependencies you are seeing: this also
looks like a bug. We are currently issuing errors if a _skipped_ symbols
depends on a skipped symbol as well, which makes no sense. We can just
ignore missing dependencies of skipped things. I've tried to implement a
quick fix here [1]. Though, you might be better off using the pre-built
binaries for now, which don't have this dependency scanning behavior in
the first place (i.e the builds at jdk.java.net/jextract)
> And why are udev, udev_list_entry, udev_device etc. skipped? They
used to work with jextract 21.
- I took a look at libudev.h, and the warnings about unsupported types
seem correct to me. e.g. `struct udev` does not have a definition in
libudev.h [2], so jextract can not generate a class for it (since the
layout and fields are not known). That is all the the warning is trying
to say. The other structs about which you get a warning seems to be the
same in that they don't have a definition in that file. (FWIW this is
one of the areas we put a lot of effort into, so it makes sense that you
didn't seen a warning in JDK 21). I think we need to clarify the error
message to say more than "is not supported" though.
> /usr/include/linux/usbdevice_fs.h: The code generated for the struct
usbdevfs_urb does not compile (it refers to a
type usbdevfs_iso_packet_desc, which is missing).
This is actually the situation we were trying to detect with the ERRORs
you encountered. There should have been an error about
usbdevfs_iso_packet_desc being excluded, but it seems like we are not
scanning the element types of arrays at the moment (linked PR tries to
fix this as well).
And finally, yeah: we need to update the README on the master branch to
reference JDK 22 instead of 21.
Thanks for reporting back! While we try to test as much as possible,
some issues slip through the cracks, and are only found through user
feedback like yours.
Jorn
[1]: https://github.com/openjdk/jextract/pull/217
[2]:
https://github.com/systemd/systemd/blob/main/src/libudev/libudev.h#L20C1-L20C13
On 17/02/2024 15:40, Manuel Bleichenbacher wrote:
> Hi Maurizio
>
> I have tried to upgrade the JavaDoesUSB library
> (https://github.com/manuelbl/JavaDoesUSB) from JDK 21 to 22, and it
> was very frustrating. It failed on macOS, Windows nor Linux. And the
> problem is jextract.
>
> (The header files I'm trying the process are operating system header
> files.)
>
>
> On macOS, it always crashes:
>
> FATAL: Unexpected exception java.lang.NullPointerException: Cannot
> invoke "org.openjdk.jextract.Declaration$Constant.name()" because
> "<parameter2>" is null occurred
>
> I've tried different branches (master, panama, jdk22) and the
> pre-built binaries. They all behave the same except the error message
> is sometimes shorter and starts at "Cannot invoke..."
>
> The full commands can be found here (remove "--source"):
> https://github.com/manuelbl/JavaDoesUSB/blob/main/java-does-usb/jextract/macos/gen_macos.sh
>
>
> On Linux, jextract worked partially. But for several files, it refused
> to generate anything or generated code that didn't compile:
>
> /usr/include/fcntl.h:
> ERROR: stat depends on timespec which has been excluded
> ERROR: stat depends on timespec which has been excluded
> ERROR: stat depends on timespec which has been excluded
>
> /usr/include/libudev.h:
> ERROR: __pthread_list_t depends on __pthread_internal_list which has
> been excluded
> ERROR: __pthread_slist_t depends on __pthread_internal_slist which has
> been excluded
> ERROR: __pthread_mutex_s depends on __pthread_internal_list which has
> been excluded
> WARNING: Skipping va_list (type <error: struct __va_list_tag> is not
> supported)
> WARNING: Skipping __gnuc_va_list (type <error: struct __va_list_tag>
> is not supported)
> WARNING: Skipping udev (type Declared(udev) is not supported)
> WARNING: Skipping udev_list_entry (type Declared(udev_list_entry) is
> not supported)
> WARNING: Skipping udev_device (type Declared(udev_device) is not
> supported)
> WARNING: Skipping udev_monitor (type Declared(udev_monitor) is not
> supported)
> WARNING: Skipping udev_enumerate (type Declared(udev_enumerate) is not
> supported)
> WARNING: Skipping udev_queue (type Declared(udev_queue) is not supported)
> WARNING: Skipping udev_hwdb (type Declared(udev_hwdb) is not supported)
>
> sys/epoll.h:
> ERROR: __pthread_list_t depends on __pthread_internal_list which has
> been excluded
> ERROR: __pthread_slist_t depends on __pthread_internal_slist which has
> been excluded
> ERROR: __pthread_mutex_s depends on __pthread_internal_list which has
> been excluded
> ERROR: epoll_data_t depends on epoll_data which has been excluded
> ERROR: epoll_event depends on epoll_data which has been excluded
>
> /usr/include/linux/usbdevice_fs.h:
> The code generated for the structusbdevfs_urb does not compile (it
> refers to a type usbdevfs_iso_packet_desc, which is missing).
>
> The full commands for jextract can be found here. I've only removed
> "--source" as it is no longer needed.
> https://github.com/manuelbl/JavaDoesUSB/blob/main/java-does-usb/jextract/linux/gen_linux.sh
>
> All these files could successfully be processed with jextract 21.
>
> I don't understand the error message. What does "xxx depends on yyy
> which has been excluded" mean? Have I used it incorrectly? The error
> message mostly mentions types (and possibly functions) that I don't
> use, neither directly nor indirectly.
>
> And why are udev, udev_list_entry, udev_device etc. skipped? They used
> to work with jextract 21.
>
>
> On Windows, it completely failed as well. jextract outputs more than
> 5000 lines of errors and warnings. Here is just a random selection:
>
> ERROR: mbstate_t depends on _Mbstatet which has been excluded
> ERROR: LIST_ENTRY depends on _LIST_ENTRY which has been excluded
> ERROR: depends on _M128A which has been excluded
> ERROR: WIN32_FIND_STREAM_DATA depends on _WIN32_FIND_STREAM_DATA which
> has been excluded
> ERROR: tagEXTLOGFONTW depends on tagLOGFONTW which has been excluded
> ERROR: MULTIKEYHELPW depends on tagMULTIKEYHELPW which has been excluded
> WARNING: Skipping MEM_EXTENDED_PARAMETER.Reserved (bitfields are not
> supported)
> WARNING: Skipping Flags.Reserved (bitfields are not supported)
>
> The full command can be found here (remove "--source"):
> https://github.com/manuelbl/JavaDoesUSB/blob/main/java-does-usb/jextract/windows/gen_win.cmd
>
>
>
> The REAMDE on https://github.com/openjdk/jextract could also profit
> from an update:
>
> The command line "$ sh ./gradlew -Pjdk21_home=<jdk21_home_dir> ..."
> sent me on search for the branch with the JDK 22 code. But it's just
> the README that's outdated.
>
> And the instruction "Run the Gradle build with a Java version
> appropriate for the Gradle version. For example, Gradle 7.5.1 supports
> JDK 21." sent me down another rabbit hole. I have the latest Gradle
> version installed, which nicely runs with JDK 21. But it failed
> anyway. Turns out that the installed gradle is irrelevant but the
> gradle wrapper relevant (see command line above). And the gradle
> wrapper is at version 7.3.3. So there is nothing to choose really. You
> must use JDK 17 (or even earlier). It has never worked with JDK 21.
>
>
> I would appreciate some feedback. Are these bugs? Or has jextract
> become so much more restrictive?
>
> Regards
> Manuel
>
>
>
> Am Fr., 16. Feb. 2024 um 16:50 Uhr schrieb Maurizio Cimadamore
> <maurizio.cimadamore at oracle.com>:
>
> Hi,
> with JDK 22 near us, we have spent some quality time with
> jextract, to
> make sure the code it generates is as good as it can be ahead of the
> finalizaton of the FFM API. This resulted in several changes, both in
> the implementation (so, invisible to jextract users) and in the
> generated code, as we cleaned up the translation strategy to better
> adhere with the core principles behind the jextract tool. These
> changes
> are captured in details in this document:
>
> https://cr.openjdk.org/~mcimadamore/panama/jextract_changes.html
>
> It might be a good time to take the latest jextract for a spin (using
> your favourite C library!) and report back, in case we missed
> anything.
> You can find the latest sources in this branch:
>
> https://github.com/openjdk/jextract/tree/panama
>
> Binary snapshots of this newer version are also available here (note
> that MacOS/Arm64 builds is also supported now):
>
> https://jdk.java.net/jextract/
>
> Cheers
> Maurizio
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jextract-dev/attachments/20240217/9a365743/attachment-0001.htm>
More information about the jextract-dev
mailing list