[External] : Re: Growing jextract
Jorn Vernee
jorn.vernee at oracle.com
Sun Feb 18 17:29:45 UTC 2024
> Using the pre-built binaries, each run outputs more than 1000
warnings related to bitfields even though I'm not using any of them.
This is a similar issue as the other warnings you got. We shouldn't be
issuing these warnings about things that are not included. Fixing one
should fix the other. (FWIW, we've discussed adding a flag to silence
these warnings altogether as well, but haven't gotten around to
implementing that yet)
> About every 30th run on Windows crashes with the below output. This
error has existed for as long as I have been using jextract.
This is caused by a bug in libclang [1]. I've looked into this, and it's
unfortunately buried deep in the compiler, so I couldn't fix it myself
(without a lot more time investment). I've brought this up on the LLVM
discord, but there didn't seem to be any interest in the issue
unfortunately.
Jorn
[1]: https://bugs.openjdk.org/browse/CODETOOLS-7903174
On 18/02/2024 14:46, Manuel Bleichenbacher wrote:
> Here's feedback from the Windows front:
>
> I've managed to get it running using the pre-built binaries.
>
> Using the pre-built binaries, each run outputs more than 1000 warnings
> related to bitfields even though I'm not using any of them. And many
> warnings are output several times:
>
> WARNING: skipping
> _PROCESS_MITIGATION_REDIRECTION_TRUST_POLICY.EnforceRedirectionTrust:
> type is bitfield
>
> This makes it nearly impossible to spot any actual problems, and thus
> it's not just an annoyance.
>
> And just for completeness:
>
> About every 30th run on Windows crashes with the below output. This
> error has existed for as long as I have been using jextract.
>
> libclang: crash detected during reparsing
> libclang: crash detected during reparsing
> libclang: crash detected during reparsing
> libclang: crash detected during reparsing
> libclang: crash detected during reparsing
> libclang: crash detected during reparsing
> libclang: crash detected during reparsing
> libclang: crash detected during reparsing
> libclang: crash detected during reparsing
> libclang: crash detected during reparsing
> Re-parsing failed: Crashed
> Regards
> Manuel
>
> Am Sa., 17. Feb. 2024 um 20:46 Uhr schrieb Jorn Vernee
> <jorn.vernee at oracle.com>:
>
> One more thing I realized about the warnings you get for
> /usr/include/libudev.h: the structs about which you get warnings
> are not being included (looking at your script, there are no
> --include-struct for them), so there shouldn't be any need for us
> to warn about them either.
>
> We currently check whether things are included first, and then
> mark things that aren't as 'skipped'. However, in the later pass
> which checks for unsupported types, we also issue warnings for
> those skipped elements. I think we should just skip issuing
> warnings for things that are marked as skipped by the earlier pass.
>
> Jorn
>
> On 17/02/2024 17:59, Jorn Vernee wrote:
>>
>> Hey Manuel,
>>
>> Thanks for giving jextract 22 a try, and sorry for the
>> frustrating experience.
>>
>> - The NullPointerException is definitely a bug. Jextract should
>> not just report an exception. We'll have a closer look at this on
>> Monday, and see if we can find what code causes the issue. It
>> would help if you could edit the jextract launch script, and add
>> `JLINK_VM_OPTIONS=-Djextract.debug=true` (`JLINK_VM_OPTIONS=`
>> should already be there), and then retry the failed extraction.
>> This should print out more information, including the entire
>> stack trace. that's the part I'm interested in.
>>
>>
>> > I don't understand the error message. What does "xxx depends on
>> yyy which has been excluded" mean? Have I used it incorrectly?
>> The error message mostly mentions types (and possibly functions)
>> that I don't use, neither directly nor indirectly.
>>
>> It is meant to tell the user that they've included something that
>> depends on something else which was not included. This would
>> previously lead to uncompileable code, since FunctionDescriptors
>> and MemoryLayouts now refer directly to the layouts defined in
>> struct classes. For example, if I have a function like `void
>> foo(struct A)`, then the FunctionDescriptor for `foo` will depend
>> directly on the class we generate for `struct A`:
>>
>> FunctionDescriptor.ofVoid(
>> A.layout()
>> );
>>
>> If `A` itself is not included, this code would be uncompileable.
>> So, in the latest version we (try to) issue an error about that.
>>
>> - For the 'ERROR's due to missing dependencies you are seeing:
>> this also looks like a bug. We are currently issuing errors if a
>> _skipped_ symbols depends on a skipped symbol as well, which
>> makes no sense. We can just ignore missing dependencies of
>> skipped things. I've tried to implement a quick fix here [1].
>> Though, you might be better off using the pre-built binaries for
>> now, which don't have this dependency scanning behavior in the
>> first place (i.e the builds at jdk.java.net/jextract
>> <https://urldefense.com/v3/__http://jdk.java.net/jextract__;!!ACWV5N9M2RV99hQ!OjZknHXPvBPucpDLIN4ldf4U5mweGvMOYTgt6BNtv5yjQqQNDGavbvWdUF6OilCQIe6imoLiGhflOUpZqKpv1drv01iI$>)
>>
>>
>> > And why are udev, udev_list_entry, udev_device etc. skipped?
>> They used to work with jextract 21.
>>
>> - I took a look at libudev.h, and the warnings about unsupported
>> types seem correct to me. e.g. `struct udev` does not have a
>> definition in libudev.h [2], so jextract can not generate a class
>> for it (since the layout and fields are not known). That is all
>> the the warning is trying to say. The other structs about which
>> you get a warning seems to be the same in that they don't have a
>> definition in that file. (FWIW this is one of the areas we put a
>> lot of effort into, so it makes sense that you didn't seen a
>> warning in JDK 21). I think we need to clarify the error message
>> to say more than "is not supported" though.
>>
>>
>> > /usr/include/linux/usbdevice_fs.h: The code generated for the
>> struct usbdevfs_urb does not compile (it refers to a
>> type usbdevfs_iso_packet_desc, which is missing).
>>
>> This is actually the situation we were trying to detect with the
>> ERRORs you encountered. There should have been an error about
>> usbdevfs_iso_packet_desc being excluded, but it seems like we are
>> not scanning the element types of arrays at the moment (linked PR
>> tries to fix this as well).
>>
>>
>> And finally, yeah: we need to update the README on the master
>> branch to reference JDK 22 instead of 21.
>>
>> Thanks for reporting back! While we try to test as much as
>> possible, some issues slip through the cracks, and are only found
>> through user feedback like yours.
>> Jorn
>>
>> [1]: https://github.com/openjdk/jextract/pull/217
>> <https://urldefense.com/v3/__https://github.com/openjdk/jextract/pull/217__;!!ACWV5N9M2RV99hQ!OjZknHXPvBPucpDLIN4ldf4U5mweGvMOYTgt6BNtv5yjQqQNDGavbvWdUF6OilCQIe6imoLiGhflOUpZqKpv1a8gEPJ9$>
>> [2]:
>> https://github.com/systemd/systemd/blob/main/src/libudev/libudev.h#L20C1-L20C13
>> <https://urldefense.com/v3/__https://github.com/systemd/systemd/blob/main/src/libudev/libudev.h*L20C1-L20C13__;Iw!!ACWV5N9M2RV99hQ!OjZknHXPvBPucpDLIN4ldf4U5mweGvMOYTgt6BNtv5yjQqQNDGavbvWdUF6OilCQIe6imoLiGhflOUpZqKpv1X17v_oJ$>
>>
>> On 17/02/2024 15:40, Manuel Bleichenbacher wrote:
>>> Hi Maurizio
>>>
>>> I have tried to upgrade the JavaDoesUSB library
>>> (https://github.com/manuelbl/JavaDoesUSB
>>> <https://urldefense.com/v3/__https://github.com/manuelbl/JavaDoesUSB__;!!ACWV5N9M2RV99hQ!OjZknHXPvBPucpDLIN4ldf4U5mweGvMOYTgt6BNtv5yjQqQNDGavbvWdUF6OilCQIe6imoLiGhflOUpZqKpv1RCDi_bU$>)
>>> from JDK 21 to 22, and it was very frustrating. It failed on
>>> macOS, Windows nor Linux. And the problem is jextract.
>>>
>>> (The header files I'm trying the process are operating system
>>> header files.)
>>>
>>>
>>> On macOS, it always crashes:
>>>
>>> FATAL: Unexpected exception java.lang.NullPointerException:
>>> Cannot invoke "org.openjdk.jextract.Declaration$Constant.name()"
>>> because "<parameter2>" is null occurred
>>>
>>> I've tried different branches (master, panama, jdk22) and the
>>> pre-built binaries. They all behave the same except the error
>>> message is sometimes shorter and starts at "Cannot invoke..."
>>>
>>> The full commands can be found here (remove "--source"):
>>> https://github.com/manuelbl/JavaDoesUSB/blob/main/java-does-usb/jextract/macos/gen_macos.sh
>>> <https://urldefense.com/v3/__https://github.com/manuelbl/JavaDoesUSB/blob/main/java-does-usb/jextract/macos/gen_macos.sh__;!!ACWV5N9M2RV99hQ!OjZknHXPvBPucpDLIN4ldf4U5mweGvMOYTgt6BNtv5yjQqQNDGavbvWdUF6OilCQIe6imoLiGhflOUpZqKpv1fE2fRNl$>
>>>
>>>
>>> On Linux, jextract worked partially. But for several files, it
>>> refused to generate anything or generated code that didn't compile:
>>>
>>> /usr/include/fcntl.h:
>>> ERROR: stat depends on timespec which has been excluded
>>> ERROR: stat depends on timespec which has been excluded
>>> ERROR: stat depends on timespec which has been excluded
>>>
>>> /usr/include/libudev.h:
>>> ERROR: __pthread_list_t depends on __pthread_internal_list which
>>> has been excluded
>>> ERROR: __pthread_slist_t depends on __pthread_internal_slist
>>> which has been excluded
>>> ERROR: __pthread_mutex_s depends on __pthread_internal_list
>>> which has been excluded
>>> WARNING: Skipping va_list (type <error: struct __va_list_tag> is
>>> not supported)
>>> WARNING: Skipping __gnuc_va_list (type <error: struct
>>> __va_list_tag> is not supported)
>>> WARNING: Skipping udev (type Declared(udev) is not supported)
>>> WARNING: Skipping udev_list_entry (type
>>> Declared(udev_list_entry) is not supported)
>>> WARNING: Skipping udev_device (type Declared(udev_device) is not
>>> supported)
>>> WARNING: Skipping udev_monitor (type Declared(udev_monitor) is
>>> not supported)
>>> WARNING: Skipping udev_enumerate (type Declared(udev_enumerate)
>>> is not supported)
>>> WARNING: Skipping udev_queue (type Declared(udev_queue) is not
>>> supported)
>>> WARNING: Skipping udev_hwdb (type Declared(udev_hwdb) is not
>>> supported)
>>>
>>> sys/epoll.h:
>>> ERROR: __pthread_list_t depends on __pthread_internal_list which
>>> has been excluded
>>> ERROR: __pthread_slist_t depends on __pthread_internal_slist
>>> which has been excluded
>>> ERROR: __pthread_mutex_s depends on __pthread_internal_list
>>> which has been excluded
>>> ERROR: epoll_data_t depends on epoll_data which has been excluded
>>> ERROR: epoll_event depends on epoll_data which has been excluded
>>>
>>> /usr/include/linux/usbdevice_fs.h:
>>> The code generated for the structusbdevfs_urb does not compile
>>> (it refers to a type usbdevfs_iso_packet_desc, which is missing).
>>>
>>> The full commands for jextract can be found here. I've only
>>> removed "--source" as it is no longer needed.
>>> https://github.com/manuelbl/JavaDoesUSB/blob/main/java-does-usb/jextract/linux/gen_linux.sh
>>> <https://urldefense.com/v3/__https://github.com/manuelbl/JavaDoesUSB/blob/main/java-does-usb/jextract/linux/gen_linux.sh__;!!ACWV5N9M2RV99hQ!OjZknHXPvBPucpDLIN4ldf4U5mweGvMOYTgt6BNtv5yjQqQNDGavbvWdUF6OilCQIe6imoLiGhflOUpZqKpv1RC2UlDa$>
>>>
>>> All these files could successfully be processed with jextract 21.
>>>
>>> I don't understand the error message. What does "xxx depends on
>>> yyy which has been excluded" mean? Have I used it incorrectly?
>>> The error message mostly mentions types (and possibly functions)
>>> that I don't use, neither directly nor indirectly.
>>>
>>> And why are udev, udev_list_entry, udev_device etc. skipped?
>>> They used to work with jextract 21.
>>>
>>>
>>> On Windows, it completely failed as well. jextract outputs more
>>> than 5000 lines of errors and warnings. Here is just a random
>>> selection:
>>>
>>> ERROR: mbstate_t depends on _Mbstatet which has been excluded
>>> ERROR: LIST_ENTRY depends on _LIST_ENTRY which has been excluded
>>> ERROR: depends on _M128A which has been excluded
>>> ERROR: WIN32_FIND_STREAM_DATA depends on _WIN32_FIND_STREAM_DATA
>>> which has been excluded
>>> ERROR: tagEXTLOGFONTW depends on tagLOGFONTW which has been excluded
>>> ERROR: MULTIKEYHELPW depends on tagMULTIKEYHELPW which has been
>>> excluded
>>> WARNING: Skipping MEM_EXTENDED_PARAMETER.Reserved (bitfields are
>>> not supported)
>>> WARNING: Skipping Flags.Reserved (bitfields are not supported)
>>>
>>> The full command can be found here (remove "--source"):
>>> https://github.com/manuelbl/JavaDoesUSB/blob/main/java-does-usb/jextract/windows/gen_win.cmd
>>> <https://urldefense.com/v3/__https://github.com/manuelbl/JavaDoesUSB/blob/main/java-does-usb/jextract/windows/gen_win.cmd__;!!ACWV5N9M2RV99hQ!OjZknHXPvBPucpDLIN4ldf4U5mweGvMOYTgt6BNtv5yjQqQNDGavbvWdUF6OilCQIe6imoLiGhflOUpZqKpv1Rk5Xxdt$>
>>>
>>>
>>>
>>> The REAMDE on https://github.com/openjdk/jextract
>>> <https://urldefense.com/v3/__https://github.com/openjdk/jextract__;!!ACWV5N9M2RV99hQ!OjZknHXPvBPucpDLIN4ldf4U5mweGvMOYTgt6BNtv5yjQqQNDGavbvWdUF6OilCQIe6imoLiGhflOUpZqKpv1TRit9Ny$>
>>> could also profit from an update:
>>>
>>> The command line "$ sh ./gradlew -Pjdk21_home=<jdk21_home_dir>
>>> ..." sent me on search for the branch with the JDK 22 code. But
>>> it's just the README that's outdated.
>>>
>>> And the instruction "Run the Gradle build with a Java version
>>> appropriate for the Gradle version. For example, Gradle 7.5.1
>>> supports JDK 21." sent me down another rabbit hole. I have the
>>> latest Gradle version installed, which nicely runs with JDK 21.
>>> But it failed anyway. Turns out that the installed gradle is
>>> irrelevant but the gradle wrapper relevant (see command line
>>> above). And the gradle wrapper is at version 7.3.3. So there is
>>> nothing to choose really. You must use JDK 17 (or even earlier).
>>> It has never worked with JDK 21.
>>>
>>>
>>> I would appreciate some feedback. Are these bugs? Or has
>>> jextract become so much more restrictive?
>>>
>>> Regards
>>> Manuel
>>>
>>>
>>>
>>> Am Fr., 16. Feb. 2024 um 16:50 Uhr schrieb Maurizio Cimadamore
>>> <maurizio.cimadamore at oracle.com>:
>>>
>>> Hi,
>>> with JDK 22 near us, we have spent some quality time with
>>> jextract, to
>>> make sure the code it generates is as good as it can be
>>> ahead of the
>>> finalizaton of the FFM API. This resulted in several
>>> changes, both in
>>> the implementation (so, invisible to jextract users) and in the
>>> generated code, as we cleaned up the translation strategy to
>>> better
>>> adhere with the core principles behind the jextract tool.
>>> These changes
>>> are captured in details in this document:
>>>
>>> https://cr.openjdk.org/~mcimadamore/panama/jextract_changes.html
>>>
>>> It might be a good time to take the latest jextract for a
>>> spin (using
>>> your favourite C library!) and report back, in case we
>>> missed anything.
>>> You can find the latest sources in this branch:
>>>
>>> https://github.com/openjdk/jextract/tree/panama
>>> <https://urldefense.com/v3/__https://github.com/openjdk/jextract/tree/panama__;!!ACWV5N9M2RV99hQ!OjZknHXPvBPucpDLIN4ldf4U5mweGvMOYTgt6BNtv5yjQqQNDGavbvWdUF6OilCQIe6imoLiGhflOUpZqKpv1ZVCerma$>
>>>
>>> Binary snapshots of this newer version are also available
>>> here (note
>>> that MacOS/Arm64 builds is also supported now):
>>>
>>> https://jdk.java.net/jextract/
>>> <https://urldefense.com/v3/__https://jdk.java.net/jextract/__;!!ACWV5N9M2RV99hQ!OjZknHXPvBPucpDLIN4ldf4U5mweGvMOYTgt6BNtv5yjQqQNDGavbvWdUF6OilCQIe6imoLiGhflOUpZqKpv1Yggg9CV$>
>>>
>>> Cheers
>>> Maurizio
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jextract-dev/attachments/20240218/3a6b8068/attachment-0001.htm>
More information about the jextract-dev
mailing list