August 2022 - jextract-dev

RFR: 7903239: ofAddress factory of function pointer type is wrong for struct returns
by Marko 02 Jan '24

02 Jan '24

When generating the lambda inside the `ofAddress` factory, it is missing the `SegmentAllocator` needed when the function returns a struct. This is [implemented for regular functions](https://github.com/openjdk/jextract/blob/0582eaf1b4cdba95f0ee8c2…, but not for function pointers. ------------- Commit messages: - 7903239: ofAddress factory of function pointer type is wrong for struct returns Changes: https://git.openjdk.org/jextract/pull/58/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=58&range=00 Issue: https://bugs.openjdk.org/browse/CODETOOLS-7903239 Stats: 47 lines in 3 files changed: 44 ins; 2 del; 1 mod Patch: https://git.openjdk.org/jextract/pull/58.diff Fetch: git fetch https://git.openjdk.org/jextract pull/58/head:pull/58 PR: https://git.openjdk.org/jextract/pull/58

3 11

RFR: Add path independence
by Per Minborg 02 Jan '24

02 Jan '24

This small PR incorporated feedback from the original PR: https://github.com/openjdk/jextract/pull/66 made after it was integrated. ------------- Commit messages: - Add path independence Changes: https://git.openjdk.org/jextract/pull/68/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=68&range=00 Stats: 10 lines in 4 files changed: 2 ins; 0 del; 8 mod Patch: https://git.openjdk.org/jextract/pull/68.diff Fetch: git fetch https://git.openjdk.org/jextract pull/68/head:pull/68 PR: https://git.openjdk.org/jextract/pull/68

5 7

RFR: 7903272: Make method handle methods private
by Per Minborg 20 Sep '22

20 Sep '22

xxxx$MH methods should be declared private to reduce cluttering in end users' IDEs and elsewhere. This PR also solves esthetic issues in the generated code. ------------- Commit messages: - Make method handle methods private Changes: https://git.openjdk.org/jextract/pull/67/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=67&range=00 Issue: https://bugs.openjdk.org/browse/CODETOOLS-7903272 Stats: 3 lines in 1 file changed: 0 ins; 0 del; 3 mod Patch: https://git.openjdk.org/jextract/pull/67.diff Fetch: git fetch https://git.openjdk.org/jextract pull/67/head:pull/67 PR: https://git.openjdk.org/jextract/pull/67

2 5

RFR: Make some classes/fields final and add privat ctr
by Per Minborg 20 Sep '22

20 Sep '22

This PR makes some of the classes and fields `final`. Also, a general cleanup of the `RuntimeHelper` class is proposed. There are some files that are "overhidden" (by attempting to "override"/hide static methods). This will be fixed in a separate PR and so these classes are not affected in this PR. ------------- Commit messages: - Make some classes/fields final and add privat ctr Changes: https://git.openjdk.org/jextract/pull/65/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=65&range=00 Stats: 99 lines in 6 files changed: 49 ins; 19 del; 31 mod Patch: https://git.openjdk.org/jextract/pull/65.diff Fetch: git fetch https://git.openjdk.org/jextract pull/65/head:pull/65 PR: https://git.openjdk.org/jextract/pull/65

2 9

jextract for operating system API
by Manuel Bleichenbacher 06 Sep '22

06 Sep '22

Hi everybody As promised, I've compiled my findings when using jextract for the Java Does USB project, which uses the foreign function & memory API to communicate with USB devices on Windows, macOS and Linux. After initial difficulties with jextract, I've written all the layouts, method and variable handles manually. But now I've given jextract another try. The commands I've used can be found in in the "jextract" branch of the project at https://github.com/manuelbl/JavaDoesUSB/tree/jextract/java-does-usb/jextract Please note that I think that automatically generating code for some of the old Windows and POSIX APIs is almost impossible. My use of jextract (for OS APIs) is probably far from the main use with third-party libraries. And I might also have made mistakes, or there might be a simple solution to my issues. See my conclusions below for areas where I think jextract has still potential. Linux: Overall, I was able to generate most of what I needed with jextract. The issues were: - Generating code for libsystemd (sd-device.h) failed with "unknown type name 'intmax_t'". I think this should be defined by inttypes.h, which is included. My guess is that clang headers and Linux headers are mixed up. - When generating code for usbdevice_fs.h, it didn't generate the constants for USBDEVFS_CLAIMINTERFACE and similar ones. The reason might be that the defines look like functions even though they evaluate to a constant value. macOS: Major parts of macOS are provided in the form of frameworks and they are treated in a special way, both when it comes to the header files and dynamic libraries. As an example, my library uses several functions from the CoreFoundation framework. In C/CC++, you can simply include them like so: #include <CoreFoundation/CoreFoundation.h> That's surprising as the file actually resides in /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/CoreFoundation.framework/Versions/A/Headers/CoreFoundation.h. Note that it is in a directory called "Headers" and not in "CoreFoundation" as the include statement suggests. The relative path "CoreFoundation/CoreFoundation.h" does not exist at all. This wouldn't be a big problem if this was just the entry point. But this pattern is used recursively. CoreFoundation.h includes about 30 other header files like this, all with a relative path that does not exist. Xcode magically resolves this. But jextract seems to be missing this magic. So it cannot be used for any macOS frameworks. In other words, it cannot be for the major part of the macOS APIs. My library uses a single non-framework functions. So using jextract for macOS is pointless without resolving this issue. Windows: On Windows I spent the most time -- with no success at all. I only partially understand the problems. Here is what I've encountered: The first simple goal was to have code for CloseHandle() generated. I tried it three different ways: - Generating code for handleapi.h generated 1000 classes. Code for CloseHandle() was generated in 2 classes, but none of them was publicly available. - Generating code for handleapi.h restricted to the function CloseHandle() generated a small number of class with a public static method for CloseHandle(). However, at run-time the method produces the error "java.lang.UnsatisfiedLinkError: unresolved symbol: CloseHandle" - Generating code for windows.h restricted to CloseHandle() also resulted in the UnsatisfiedLinkError. I then tried to generate code for Winusb.h. Unfortunately, this header file is not independent. It expects that you have already processed other header files before it. So it fails with "error: unknown type name 'LARGE_INTEGER'". This behavior with be reproduced in Visual Studio. Next up was Cfgmgr32.h. It also doesn't seem to be independent and fails with "error: unknown type name 'ULONG'". Setupapi.h and Usbioctl.h behave similarly. Windows was the only system where I had problems with preprocessor macros. The macro _M_AMD64=100 is certainly needed. That got me further than without it. And I've defined two Unicode macros, copied from the Visual Studio project. But the macros might still be the source of some problems. Conclusion: At the moment, jextract doesn't help much for accessing OS APIs. On Linux, the "unknown type name 'intmax_t'" is worth investigating. It could be helpful for many APIs. On macOS, it's all about the frameworks and the magic behind resolving the include path. If it can be made to work, it hopefully opens up the world of framework (plus some challenges that are still hidden). On Windows, more investigation is needed. Work on the required macros is needed, as well as more experiments with header files just built for code generation (to get beyond the problem of header files that are not independent). Regards Manuel

4 8

RFR: 7903271: Add sample of libzstd
by Per Minborg 06 Sep '22

06 Sep '22

This PR adds a sample using `libzstd`, a high-performance compression library. ------------- Commit messages: - Add sample of libzstd Changes: https://git.openjdk.org/jextract/pull/66/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=66&range=00 Issue: https://bugs.openjdk.org/browse/CODETOOLS-7903271 Stats: 158 lines in 8 files changed: 158 ins; 0 del; 0 mod Patch: https://git.openjdk.org/jextract/pull/66.diff Fetch: git fetch https://git.openjdk.org/jextract pull/66/head:pull/66 PR: https://git.openjdk.org/jextract/pull/66

3 8

Tool architecture vs build-time architeture vs run-time architecture
by Manuel Bleichenbacher 05 Sep '22

05 Sep '22

I'm migrating this detail question over from the panama-dev mailing list: can jextract built for macOS on Intel run on macOS for ARM generate correct code for macOS on ARM? Note that the architecture (Intel vs ARM) affects three separate things: 1. jextract (the JVM and jextract are built for a certain platform and architecture) 2. the architecture where jextract is run (macOS can run Intel code on ARM) 3. the architecture of the resulting / the architecture where the generated code will be run I understand that separate code is needed for different platforms and architectures. So typically you would run jextract on each platform separately. But what decides if jextract generates code for ARM or Intel? Is it (1) the jextract variant or is it (2) the operating system it is run on? Or does it even generate an unusable result if (1) and (2) are different?

2 1

SymbolLookup.loaderLookup() vs SymbolLookup.libraryLookup()
by Manuel Bleichenbacher 30 Aug '22

30 Aug '22

If code is created for libudev (on Linux) with the command: jextract --source -l udev --include-function udev_new /usr/include/libudev.h the resulting code fails at run-time with: java.lang.UnsatisfiedLinkError: no udev in java.library.path: /usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib That's of course correct as the libudev is in /usr/lib/x86_64-linux-gnu. But it isn't just a random application specific directory. It's a directory that is searched for libraries by default (depending on the architecture the name changes) and contains many basic libraries for Linux. The generated code can be fixed easily with a minor modification. Instead of the generated code (in RuntimeHelper.java): System.loadLibrary("udev"); SymbolLookup loaderLookup = SymbolLookup.loaderLookup(); one can just put: SymbolLookup loaderLookup = SymbolLookup.libraryLookup("libudev.so", MemorySession.openImplicit()); Would it be possible to add an option to jextract so it generates code using SymbolLookup.libraryLookup() instead of SymbolLookup.loaderLookup() and does not generate a call to System.loadLibrary()?

1 0

RFR: 7903269: jextract test failures after Layout updates change
by Athijegannathan Sundararajan 29 Aug '22

29 Aug '22

Missed isStruct, isUnion changes in tests ------------- Commit messages: - 7903269: jextract test failures after Layout updates change Changes: https://git.openjdk.org/jextract/pull/64/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=64&range=00 Issue: https://bugs.openjdk.org/browse/CODETOOLS-7903269 Stats: 11 lines in 3 files changed: 4 ins; 0 del; 7 mod Patch: https://git.openjdk.org/jextract/pull/64.diff Fetch: git fetch https://git.openjdk.org/jextract pull/64/head:pull/64 PR: https://git.openjdk.org/jextract/pull/64

2 2

Re: jextract for operating system API
by Manuel Bleichenbacher 29 Aug '22

29 Aug '22

In case you wonder how Xcode resolves the path to framework headers, you might want to start with the WWDC 18 presentation "Behind the Scenes of the Xcode Build Process" (https://developer.apple.com/videos/play/wwdc2018/415/) in particular from 20:16 to 22:06. The framework resolution is actually a standard clang feature (and Xcode uses clang). Have a look at the -F and -iframework command line option. Xcode also uses Clang modules. But as far as I understand, modules are only used to speed up compilation and are not required for find the header files. The WWDC notes can be found at https://www.wwdcnotes.com/notes/wwdc18/415/, the relevant part is the chapter "Clang compiler". -- Manuel On Wed, Aug 24, 2022 at 10:31 AM Manuel Bleichenbacher < manuel.bleichenbacher(a)gmail.com> wrote: > Thanks a lot. > > If you prioritize the work related to my experiments, please give the > missing support for macOS frameworks the highest priority. > > -- Manuel > > > On Wed, Aug 24, 2022 at 6:34 AM Sundararajan Athijegannathan < > sundararajan.athijegannathan(a)oracle.com> wrote: > >> Thanks for your report on progress. We'll try to files (I'm not on >> Windows so it'll take sometime before bug filing). >> >> Thanks >> -Sundar >> ------------------------------ >> *From:* jextract-dev <jextract-dev-retn(a)openjdk.org> on behalf of Manuel >> Bleichenbacher <manuel.bleichenbacher(a)gmail.com> >> *Sent:* 24 August 2022 02:01 >> *To:* jextract-dev(a)openjdk.org <jextract-dev(a)openjdk.org> >> *Subject:* Re: jextract for operating system API >> >> On Windows, I've made major progress, solved most of the issues and run >> into a few new ones. >> >> The main thing is: It's not possible to directly process a header file >> like winusb.h. Instead, a helper header file needs to be created: >> >> #include <windows.h> >> #include <winusb.h> >> >> A mistake I made yesterday was the omission of "-l Kernel32" (or whatever >> library is needed). >> >> With these two changes and the use of --include-xxx, more or less all >> functions, structs and macros can be created. (I didn't yet have time to >> look into GUID constants like GUID_DEVINTERFACE_USB_DEVICE; they probably >> don't work due to macro trickery.) >> >> There are 2 structs that do not work (with rather advanced issues): >> >> (1) The generated class _USB_NODE_CONNECTION_INFORMATION_EX cannot be >> instanciated. The constructor throws the exception: >> java.lang.UnsupportedOperationException: Invalid alignment requirements for >> layout s16(DeviceAddress) >> >> This struct (like many USB descriptors) is packed with 16-bit integer >> values on odd offsets. jextract insists on creating a layout with strict >> alignment rules. So the generated layout >> net.codecrete.usb.windows.gen.usbioctl._USB_NODE_CONNECTION_INFORMATION_EX#$struct$LAYOUT >> fails, while the manually created layout >> net.codecrete.usb.windows.USBIOCtl#USB_NODE_CONNECTION_INFORMATION_EX$Struct >> works. >> >> (2) The struct _SP_DEVICE_INTERFACE_DETAIL_DATA_W isn't usable. In C, it >> looks like this: >> >> typedef struct _SP_DEVICE_INTERFACE_DETAIL_DATA_W { >> DWORD cbSize; >> WCHAR DevicePath[ANYSIZE_ARRAY]; >> } SP_DEVICE_INTERFACE_DETAIL_DATA_W, *PSP_DEVICE_INTERFACE_DETAIL_DATA_W; >> >> It's basically a variable length struct. ANYSIZE_ARRAY is defined as 1. >> But you are supposed to allocate additional memory for the DevicePath. I >> guess this is old style trickery that cannot be solved with an automated >> tool. >> >> Minor annoyances are: >> >> - jextract on Windows is a batch file that never returns. So it's not >> possible to call jextract multiple times for a single batch file. >> - It would be nicer if structs used the nice typedef name instead of the >> ugly struct name, e.g. SP_DEVICE_INTERFACE_DATA_W instead of >> _SP_DEVICE_INTERFACE_DATA_W (note the leading underscore). >> >> Overall: success on Windows. >> >> Regards >> Manuel >> >> On Tue, Aug 23, 2022 at 5:48 AM Sundararajan Athijegannathan < >> sundararajan.athijegannathan(a)oracle.com> wrote: >> >> Thanks for your experiments and comments. I'll go over and file bug(s) as >> needed. >> >> -Sundar >> ------------------------------ >> *From:* jextract-dev <jextract-dev-retn(a)openjdk.org> on behalf of Manuel >> Bleichenbacher <manuel.bleichenbacher(a)gmail.com> >> *Sent:* 23 August 2022 02:24 >> *To:* jextract-dev(a)openjdk.org <jextract-dev(a)openjdk.org> >> *Subject:* jextract for operating system API >> >> Hi everybody >> >> As promised, I've compiled my findings when using jextract for the Java >> Does USB project, which uses the foreign function & memory API to >> communicate with USB devices on Windows, macOS and Linux. After initial >> difficulties with jextract, I've written all the layouts, method and >> variable handles manually. But now I've given jextract another try. The >> commands I've used can be found in in the "jextract" branch of the project >> at >> https://github.com/manuelbl/JavaDoesUSB/tree/jextract/java-does-usb/jextract >> >> Please note that I think that automatically generating code for some of >> the old Windows and POSIX APIs is almost impossible. My use of jextract >> (for OS APIs) is probably far from the main use with third-party libraries. >> And I might also have made mistakes, or there might be a simple solution to >> my issues. See my conclusions below for areas where I think jextract has >> still potential. >> >> >> Linux: >> >> Overall, I was able to generate most of what I needed with jextract. The >> issues were: >> >> - Generating code for libsystemd (sd-device.h) failed with "unknown type >> name 'intmax_t'". I think this should be defined by inttypes.h, which is >> included. My guess is that clang headers and Linux headers are mixed up. >> - When generating code for usbdevice_fs.h, it didn't generate the >> constants for USBDEVFS_CLAIMINTERFACE and similar ones. The reason might be >> that the defines look like functions even though they evaluate to a >> constant value. >> >> >> macOS: >> >> Major parts of macOS are provided in the form of frameworks and they are >> treated in a special way, both when it comes to the header files and >> dynamic libraries. As an example, my library uses several functions from >> the CoreFoundation framework. In C/CC++, you can simply include them like >> so: >> >> #include <CoreFoundation/CoreFoundation.h> >> >> That's surprising as the file actually resides in >> /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/CoreFoundation.framework/Versions/A/Headers/CoreFoundation.h. >> >> Note that it is in a directory called "Headers" and not in >> "CoreFoundation" as the include statement suggests. The relative path >> "CoreFoundation/CoreFoundation.h" does not exist at all. >> >> This wouldn't be a big problem if this was just the entry point. But this >> pattern is used recursively. CoreFoundation.h includes about 30 other >> header files like this, all with a relative path that does not exist. >> >> Xcode magically resolves this. But jextract seems to be missing this >> magic. So it cannot be used for any macOS frameworks. In other words, it >> cannot be for the major part of the macOS APIs. My library uses a single >> non-framework functions. So using jextract for macOS is pointless without >> resolving this issue. >> >> >> Windows: >> >> On Windows I spent the most time -- with no success at all. I only >> partially understand the problems. Here is what I've encountered: >> >> The first simple goal was to have code for CloseHandle() generated. I >> tried it three different ways: >> >> - Generating code for handleapi.h generated 1000 classes. Code for >> CloseHandle() was generated in 2 classes, but none of them was publicly >> available. >> - Generating code for handleapi.h restricted to the function >> CloseHandle() generated a small number of class with a public static method >> for CloseHandle(). However, at run-time the method produces the error >> "java.lang.UnsatisfiedLinkError: unresolved symbol: CloseHandle" >> - Generating code for windows.h restricted to CloseHandle() also >> resulted in the UnsatisfiedLinkError. >> >> I then tried to generate code for Winusb.h. Unfortunately, this header >> file is not independent. It expects that you have already processed other >> header files before it. So it fails with "error: unknown type name >> 'LARGE_INTEGER'". This behavior with be reproduced in Visual Studio. >> >> Next up was Cfgmgr32.h. It also doesn't seem to be independent and fails >> with "error: unknown type name 'ULONG'". Setupapi.h and Usbioctl.h behave >> similarly. >> >> Windows was the only system where I had problems with preprocessor >> macros. The macro _M_AMD64=100 is certainly needed. That got me further >> than without it. And I've defined two Unicode macros, copied from the >> Visual Studio project. But the macros might still be the source of some >> problems. >> >> >> >> Conclusion: >> >> At the moment, jextract doesn't help much for accessing OS APIs. >> >> On Linux, the "unknown type name 'intmax_t'" is worth investigating. It >> could be helpful for many APIs. >> >> On macOS, it's all about the frameworks and the magic behind resolving >> the include path. If it can be made to work, it hopefully opens up the >> world of framework (plus some challenges that are still hidden). >> >> On Windows, more investigation is needed. Work on the required macros is >> needed, as well as more experiments with header files just built for code >> generation (to get beyond the problem of header files that are not >> independent). >> >> >> Regards >> Manuel >> >> >>

1 0