jextract for operating system API

Mon Sep 5 14:03:09 UTC 2022

Hi Jorn,

Thanks for looking into this issues.

As a general remark. My code is publicly available on GitHub (
https://github.com/manuelbl/JavaDoesUSB) and I even maintain a list of the
jextract issues (
https://github.com/manuelbl/JavaDoesUSB/tree/main/java-does-usb/jextract).

On Mon, Sep 5, 2022 at 3:33 PM Jorn Vernee <jorn.vernee at oracle.com> wrote:

> Hello,
>
> The main thing is: It's not possible to directly process a header file
> like winusb.h. Instead, a helper header file needs to be created:
>
> #include <windows.h>
> #include <winusb.h>
>
> You've probably found this already, but a lot of Windows functions require
> including Windows.h instead of the header that defines the function. The
> header that should be included is listed in the table of requirements in
> the documentation of the function, for instance for CloseHandle [1], it
> lists "*Header* handleapi.h (include Windows.h)". The DLL is also listed
> there.
>
> I'm surprised that Windows.h is needed for Winusb.h, since e.g.
> WinUsb_Initialize [2] only lists Winusb.h
>
> [1]:
> https://docs.microsoft.com/en-us/windows/win32/api/handleapi/nf-handleapi-closehandle
> [2]:
> https://docs.microsoft.com/en-us/windows/win32/api/winusb/nf-winusb-winusb_initialize
>
Yes, I've figured this out and I have created a helper header file that I'm
using for jextract (I think I've communicated this before on the mailing
list).

>
> (1) The generated class _USB_NODE_CONNECTION_INFORMATION_EX cannot be
> instanciated. The constructor throws the exception:
> java.lang.UnsupportedOperationException: Invalid alignment requirements for
> layout s16(DeviceAddress)
>
> This struct (like many USB descriptors) is packed with 16-bit integer
> values on odd offsets. jextract insists on creating a layout with strict
> alignment rules. So the generated layout
> net.codecrete.usb.windows.gen.usbioctl._USB_NODE_CONNECTION_INFORMATION_EX#$struct$LAYOUT
> fails, while the manually created layout
> net.codecrete.usb.windows.USBIOCtl#USB_NODE_CONNECTION_INFORMATION_EX$Struct
> works.
>
> This seems like a bug in jextract.
>
> As a possible workaround, you could generate source code with `--source`
> and then manually drop the alignment from the layouts as well (using
> .withBitAlignment(8)).
>
My current workaround is to use my on struct definition as I often
regenerate the source to add more functions and structs.

> (2) The struct _SP_DEVICE_INTERFACE_DETAIL_DATA_W isn't usable. In C, it
> looks like this:
>
> typedef struct _SP_DEVICE_INTERFACE_DETAIL_DATA_W {
>   DWORD cbSize;
>   WCHAR DevicePath[ANYSIZE_ARRAY];
> } SP_DEVICE_INTERFACE_DETAIL_DATA_W, *PSP_DEVICE_INTERFACE_DETAIL_DATA_W;
>
> It's basically a variable length struct. ANYSIZE_ARRAY is defined as 1.
> But you are supposed to allocate additional memory for the DevicePath. I
> guess this is old style trickery that cannot be solved with an automated
> tool.
>
> Yes, I'm not sure there is much we could do automatically. Especially
> since this array is not defined without a size, which would make it a
> proper incomplete array type.
>
> To work with such arrays, I think the best approach would be to create an
> unchecked memory segment starting at the offset of the array to access the
> data (using MemorySegment::ofAddress).
>
I'm not sure I understand the approach. But I've figured out a workaround.
See
https://github.com/manuelbl/JavaDoesUSB/blob/main/java-does-usb/src/main/java/net/codecrete/usb/windows/DeviceProperty.java#L107

> Minor annoyances are:
>
> - jextract on Windows is a batch file that never returns. So it's not
> possible to call jextract multiple times for a single batch file.
>
> I don't see that issue when I run it on my machine. You mean e.g.
> `jextract --help` never returns to the command prompt?
>

The problem is different. If I have a batch file like this:

jextract header1.h
jextract header2.h

Then the second command is never executed since jextract never returns to
the calling batch file. If you have nested batch file, you would generally
need to use "call". But that doesn't help since the command.exe process is
replaced with the java process.

> - It would be nicer if structs used the nice typedef name instead of the
> ugly struct name, e.g. SP_DEVICE_INTERFACE_DATA_W instead of
> _SP_DEVICE_INTERFACE_DATA_W (note the leading underscore).
>
> If there is a typedef, jextract should also generate a class with the nice
> name which extends the ugly name. It should be possible to use the class
> with the nice name instead as well (through static method "inheritance").
>

This is partially correct. I've figured out that if I include both the
struct _SP_DEVICE_INTERFACE_DATA_W and the typedef
SP_DEVICE_INTERFACE_DATA_W, then I can use the nice name
SP_DEVICE_INTERFACE_DATA_W. But if I only include the typedef, it generates
any empty class that doesn't inherit from anything (except Object). That
looks like a bug to me.

Cheers,
> Jorn
>
>
> On Tue, Aug 23, 2022 at 5:48 AM Sundararajan Athijegannathan <
> sundararajan.athijegannathan at oracle.com> wrote:
>
>> Thanks for your experiments and comments. I'll go over and file bug(s) as
>> needed.
>>
>> -Sundar
>> ------------------------------
>> *From:* jextract-dev <jextract-dev-retn at openjdk.org> on behalf of Manuel
>> Bleichenbacher <manuel.bleichenbacher at gmail.com>
>> *Sent:* 23 August 2022 02:24
>> *To:* jextract-dev at openjdk.org <jextract-dev at openjdk.org>
>> *Subject:* jextract for operating system API
>>
>> Hi everybody
>>
>> As promised, I've compiled my findings when using jextract for the Java
>> Does USB project, which uses the foreign function & memory API to
>> communicate with USB devices on Windows, macOS and Linux. After initial
>> difficulties with jextract, I've written all the layouts, method and
>> variable handles manually. But now I've given jextract another try. The
>> commands I've used can be found in in the "jextract" branch of the project
>> at
>> https://github.com/manuelbl/JavaDoesUSB/tree/jextract/java-does-usb/jextract
>>
>> Please note that I think that automatically generating code for some of
>> the old Windows and POSIX APIs is almost impossible. My use of jextract
>> (for OS APIs) is probably far from the main use with third-party libraries.
>> And I might also have made mistakes, or there might be a simple solution to
>> my issues. See my conclusions below for areas where I think jextract has
>> still potential.
>>
>>
>> Linux:
>>
>> Overall, I was able to generate most of what I needed with jextract. The
>> issues were:
>>
>> - Generating code for libsystemd (sd-device.h) failed with "unknown type
>> name 'intmax_t'". I think this should be defined by inttypes.h, which is
>> included. My guess is that clang headers and Linux headers are mixed up.
>> - When generating code for usbdevice_fs.h, it didn't generate the
>> constants for USBDEVFS_CLAIMINTERFACE and similar ones. The reason might be
>> that the defines look like functions even though they evaluate to a
>> constant value.
>>
>>
>> macOS:
>>
>> Major parts of macOS are provided in the form of frameworks and they are
>> treated in a special way, both when it comes to the header files and
>> dynamic libraries. As an example, my library uses several functions from
>> the CoreFoundation framework. In C/CC++, you can simply include them like
>> so:
>>
>> #include <CoreFoundation/CoreFoundation.h>
>>
>> That's surprising as the file actually resides in
>> /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/CoreFoundation.framework/Versions/A/Headers/CoreFoundation.h.
>>
>> Note that it is in a directory called "Headers" and not in
>> "CoreFoundation" as the include statement suggests. The relative path
>> "CoreFoundation/CoreFoundation.h" does not exist at all.
>>
>> This wouldn't be a big problem if this was just the entry point. But this
>> pattern is used recursively. CoreFoundation.h includes about 30 other
>> header files like this, all with a relative path that does not exist.
>>
>> Xcode magically resolves this. But jextract seems to be missing this
>> magic. So it cannot be used for any macOS frameworks. In other words, it
>> cannot be for the major part of the macOS APIs. My library uses a single
>> non-framework functions. So using jextract for macOS is pointless without
>> resolving this issue.
>>
>>
>> Windows:
>>
>> On Windows I spent the most time -- with no success at all. I only
>> partially understand the problems. Here is what I've encountered:
>>
>> The first simple goal was to have code for CloseHandle() generated. I
>> tried it three different ways:
>>
>>    - Generating code for handleapi.h generated 1000 classes. Code for
>>    CloseHandle() was generated in 2 classes, but none of them was publicly
>>    available.
>>    - Generating code for handleapi.h restricted to the function
>>    CloseHandle() generated a small number of class with a public static method
>>    for CloseHandle(). However, at run-time the method produces the error
>>    "java.lang.UnsatisfiedLinkError: unresolved symbol: CloseHandle"
>>    - Generating code for windows.h restricted to CloseHandle() also
>>    resulted in the UnsatisfiedLinkError.
>>
>> I then tried to generate code for Winusb.h. Unfortunately, this header
>> file is not independent. It expects that you have already processed other
>> header files before it. So it fails with "error: unknown type name
>> 'LARGE_INTEGER'". This behavior with be reproduced in Visual Studio.
>>
>> Next up was Cfgmgr32.h. It also doesn't seem to be independent and fails
>> with "error: unknown type name 'ULONG'". Setupapi.h and Usbioctl.h behave
>> similarly.
>>
>> Windows was the only system where I had problems with preprocessor
>> macros. The macro _M_AMD64=100 is certainly needed. That got me further
>> than without it. And I've defined two Unicode macros, copied from the
>> Visual Studio project. But the macros might still be the source of some
>> problems.
>>
>>
>>
>> Conclusion:
>>
>> At the moment, jextract doesn't help much for accessing OS APIs.
>>
>> On Linux, the "unknown type name 'intmax_t'" is worth investigating. It
>> could be helpful for many APIs.
>>
>> On macOS, it's all about the frameworks and the magic behind resolving
>> the include path. If it can be made to work, it hopefully opens up the
>> world of framework (plus some challenges that are still hidden).
>>
>> On Windows, more investigation is needed. Work on the required macros is
>> needed, as well as more experiments with header files just built for code
>> generation (to get beyond the problem of header files that are not
>> independent).
>>
>>
>> Regards
>> Manuel
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jextract-dev/attachments/20220905/a4fda522/attachment-0001.htm>