Growing jextract

Sun Feb 18 13:46:44 UTC 2024

Here's feedback from the Windows front:

I've managed to get it running using the pre-built binaries.

Using the pre-built binaries, each run outputs more than 1000 warnings
related to bitfields even though I'm not using any of them. And many
warnings are output several times:

WARNING: skipping
_PROCESS_MITIGATION_REDIRECTION_TRUST_POLICY.EnforceRedirectionTrust: type
is bitfield

This makes it nearly impossible to spot any actual problems, and thus it's
not just an annoyance.

And just for completeness:

About every 30th run on Windows crashes with the below output. This error
has existed for as long as I have been using jextract.

libclang: crash detected during reparsing
libclang: crash detected during reparsing
libclang: crash detected during reparsing
libclang: crash detected during reparsing
libclang: crash detected during reparsing
libclang: crash detected during reparsing
libclang: crash detected during reparsing
libclang: crash detected during reparsing
libclang: crash detected during reparsing
libclang: crash detected during reparsing
Re-parsing failed: Crashed

Regards
Manuel

Am Sa., 17. Feb. 2024 um 20:46 Uhr schrieb Jorn Vernee <
jorn.vernee at oracle.com>:

> One more thing I realized about the warnings you get for
> /usr/include/libudev.h: the structs about which you get warnings are not
> being included (looking at your script, there are no --include-struct for
> them), so there shouldn't be any need for us to warn about them either.
>
> We currently check whether things are included first, and then mark things
> that aren't as 'skipped'. However, in the later pass which checks for
> unsupported types, we also issue warnings for those skipped elements. I
> think we should just skip issuing warnings for things that are marked as
> skipped by the earlier pass.
>
> Jorn
> On 17/02/2024 17:59, Jorn Vernee wrote:
>
> Hey Manuel,
>
> Thanks for giving jextract 22 a try, and sorry for the frustrating
> experience.
>
> - The NullPointerException is definitely a bug. Jextract should not just
> report an exception. We'll have a closer look at this on Monday, and see if
> we can find what code causes the issue. It would help if you could edit the
> jextract launch script, and add `JLINK_VM_OPTIONS=-Djextract.debug=true`
> (`JLINK_VM_OPTIONS=` should already be there), and then retry the failed
> extraction. This should print out more information, including the entire
> stack trace. that's the part I'm interested in.
>
>
> > I don't understand the error message. What does "xxx depends on yyy
> which has been excluded" mean? Have I used it incorrectly? The error
> message mostly mentions types (and possibly functions) that I don't use,
> neither directly nor indirectly.
>
> It is meant to tell the user that they've included something that depends
> on something else which was not included. This would previously lead to
> uncompileable code, since FunctionDescriptors and MemoryLayouts now refer
> directly to the layouts defined in struct classes. For example, if I have a
> function like `void foo(struct A)`, then the FunctionDescriptor for `foo`
> will depend directly on the class we generate for `struct A`:
>
>     FunctionDescriptor.ofVoid(
>         A.layout()
>     );
>
> If `A` itself is not included, this code would be uncompileable. So, in
> the latest version we (try to) issue an error about that.
>
> - For the 'ERROR's due to missing dependencies you are seeing: this also
> looks like a bug. We are currently issuing errors if a _skipped_ symbols
> depends on a skipped symbol as well, which makes no sense. We can just
> ignore missing dependencies of skipped things. I've tried to implement a
> quick fix here [1]. Though, you might be better off using the pre-built
> binaries for now, which don't have this dependency scanning behavior in the
> first place (i.e the builds at jdk.java.net/jextract)
>
>
> > And why are udev, udev_list_entry, udev_device etc. skipped? They used
> to work with jextract 21.
>
> - I took a look at libudev.h, and the warnings about unsupported types
> seem correct to me. e.g. `struct udev` does not have a definition in
> libudev.h [2], so jextract can not generate a class for it (since the
> layout and fields are not known). That is all the the warning is trying to
> say. The other structs about which you get a warning seems to be the same
> in that they don't have a definition in that file. (FWIW this is one of the
> areas we put a lot of effort into, so it makes sense that you didn't seen a
> warning in JDK 21). I think we need to clarify the error message to say
> more than "is not supported" though.
>
>
> > /usr/include/linux/usbdevice_fs.h: The code generated for the struct
> usbdevfs_urb does not compile (it refers to a
> type usbdevfs_iso_packet_desc, which is missing).
>
> This is actually the situation we were trying to detect with the ERRORs
> you encountered. There should have been an error about
> usbdevfs_iso_packet_desc being excluded, but it seems like we are not
> scanning the element types of arrays at the moment (linked PR tries to fix
> this as well).
>
>
> And finally, yeah: we need to update the README on the master branch to
> reference JDK 22 instead of 21.
>
> Thanks for reporting back! While we try to test as much as possible, some
> issues slip through the cracks, and are only found through user feedback
> like yours.
> Jorn
>
> [1]: https://github.com/openjdk/jextract/pull/217
> [2]:
> https://github.com/systemd/systemd/blob/main/src/libudev/libudev.h#L20C1-L20C13
> On 17/02/2024 15:40, Manuel Bleichenbacher wrote:
>
> Hi Maurizio
>
> I have tried to upgrade the JavaDoesUSB library (
> https://github.com/manuelbl/JavaDoesUSB) from JDK 21 to 22, and it was
> very frustrating. It failed on macOS, Windows nor Linux. And the problem is
> jextract.
>
> (The header files I'm trying the process are operating system header
> files.)
>
>
> On macOS, it always crashes:
>
> FATAL: Unexpected exception java.lang.NullPointerException: Cannot invoke
> "org.openjdk.jextract.Declaration$Constant.name()" because "<parameter2>"
> is null occurred
>
> I've tried different branches (master, panama, jdk22) and the pre-built
> binaries. They all behave the same except the error message is sometimes
> shorter and starts at "Cannot invoke..."
>
> The full commands can be found here (remove "--source"):
> https://github.com/manuelbl/JavaDoesUSB/blob/main/java-does-usb/jextract/macos/gen_macos.sh
>
>
> On Linux, jextract worked partially. But for several files, it refused to
> generate anything or generated code that didn't compile:
>
> /usr/include/fcntl.h:
> ERROR: stat depends on timespec which has been excluded
> ERROR: stat depends on timespec which has been excluded
> ERROR: stat depends on timespec which has been excluded
>
> /usr/include/libudev.h:
> ERROR: __pthread_list_t depends on __pthread_internal_list which has been
> excluded
> ERROR: __pthread_slist_t depends on __pthread_internal_slist which has
> been excluded
> ERROR: __pthread_mutex_s depends on __pthread_internal_list which has been
> excluded
> WARNING: Skipping va_list (type <error: struct __va_list_tag> is not
> supported)
> WARNING: Skipping __gnuc_va_list (type <error: struct __va_list_tag> is
> not supported)
> WARNING: Skipping udev (type Declared(udev) is not supported)
> WARNING: Skipping udev_list_entry (type Declared(udev_list_entry) is not
> supported)
> WARNING: Skipping udev_device (type Declared(udev_device) is not supported)
> WARNING: Skipping udev_monitor (type Declared(udev_monitor) is not
> supported)
> WARNING: Skipping udev_enumerate (type Declared(udev_enumerate) is not
> supported)
> WARNING: Skipping udev_queue (type Declared(udev_queue) is not supported)
> WARNING: Skipping udev_hwdb (type Declared(udev_hwdb) is not supported)
>
> sys/epoll.h:
> ERROR: __pthread_list_t depends on __pthread_internal_list which has been
> excluded
> ERROR: __pthread_slist_t depends on __pthread_internal_slist which has
> been excluded
> ERROR: __pthread_mutex_s depends on __pthread_internal_list which has been
> excluded
> ERROR: epoll_data_t depends on epoll_data which has been excluded
> ERROR: epoll_event depends on epoll_data which has been excluded
>
> /usr/include/linux/usbdevice_fs.h:
> The code generated for the structusbdevfs_urb does not compile (it refers
> to a type usbdevfs_iso_packet_desc, which is missing).
>
> The full commands for jextract can be found here. I've only removed
> "--source" as it is no longer needed.
>
> https://github.com/manuelbl/JavaDoesUSB/blob/main/java-does-usb/jextract/linux/gen_linux.sh
>
> All these files could successfully be processed with jextract 21.
>
> I don't understand the error message. What does "xxx depends on yyy which
> has been excluded" mean? Have I used it incorrectly? The error message
> mostly mentions types (and possibly functions) that I don't use, neither
> directly nor indirectly.
>
> And why are udev, udev_list_entry, udev_device etc. skipped? They used to
> work with jextract 21.
>
>
> On Windows, it completely failed as well. jextract outputs more than 5000
> lines of errors and warnings. Here is just a random selection:
>
> ERROR: mbstate_t depends on _Mbstatet which has been excluded
> ERROR: LIST_ENTRY depends on _LIST_ENTRY which has been excluded
> ERROR:  depends on _M128A which has been excluded
> ERROR: WIN32_FIND_STREAM_DATA depends on _WIN32_FIND_STREAM_DATA which has
> been excluded
> ERROR: tagEXTLOGFONTW depends on tagLOGFONTW which has been excluded
> ERROR: MULTIKEYHELPW depends on tagMULTIKEYHELPW which has been excluded
> WARNING: Skipping MEM_EXTENDED_PARAMETER.Reserved (bitfields are not
> supported)
> WARNING: Skipping Flags.Reserved (bitfields are not supported)
>
> The full command can be found here (remove "--source"):
> https://github.com/manuelbl/JavaDoesUSB/blob/main/java-does-usb/jextract/windows/gen_win.cmd
>
>
>
> The REAMDE on https://github.com/openjdk/jextract could also profit from
> an update:
>
> The command line "$ sh ./gradlew -Pjdk21_home=<jdk21_home_dir> ..." sent
> me on search for the branch with the JDK 22 code. But it's just the README
> that's outdated.
>
> And the instruction "Run the Gradle build with a Java version appropriate
> for the Gradle version. For example, Gradle 7.5.1 supports JDK 21." sent me
> down another rabbit hole. I have the latest Gradle version installed, which
> nicely runs with JDK 21. But it failed anyway. Turns out that the installed
> gradle is irrelevant but the gradle wrapper relevant (see command line
> above). And the gradle wrapper is at version 7.3.3. So there is nothing to
> choose really. You must use JDK 17 (or even earlier). It has never worked
> with JDK 21.
>
>
> I would appreciate some feedback. Are these bugs? Or has jextract become
> so much more restrictive?
>
> Regards
> Manuel
>
>
>
> Am Fr., 16. Feb. 2024 um 16:50 Uhr schrieb Maurizio Cimadamore <
> maurizio.cimadamore at oracle.com>:
>
>> Hi,
>> with JDK 22 near us, we have spent some quality time with jextract, to
>> make sure the code it generates is as good as it can be ahead of the
>> finalizaton of the FFM API. This resulted in several changes, both in
>> the implementation (so, invisible to jextract users) and in the
>> generated code, as we cleaned up the translation strategy to better
>> adhere with the core principles behind the jextract tool. These changes
>> are captured in details in this document:
>>
>> https://cr.openjdk.org/~mcimadamore/panama/jextract_changes.html
>>
>> It might be a good time to take the latest jextract for a spin (using
>> your favourite C library!) and report back, in case we missed anything.
>> You can find the latest sources in this branch:
>>
>> https://github.com/openjdk/jextract/tree/panama
>>
>> Binary snapshots of this newer version are also available here (note
>> that MacOS/Arm64 builds is also supported now):
>>
>> https://jdk.java.net/jextract/
>>
>> Cheers
>> Maurizio
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jextract-dev/attachments/20240218/25bc2fbe/attachment-0001.htm>