jextract for operating system API

Tue Aug 23 20:31:09 UTC 2022

On Windows, I've made major progress, solved most of the issues and run
into a few new ones.

The main thing is: It's not possible to directly process a header file like
winusb.h. Instead, a helper header file needs to be created:

#include <windows.h>
#include <winusb.h>

A mistake I made yesterday was the omission of "-l Kernel32" (or whatever
library is needed).

With these two changes and the use of --include-xxx, more or less all
functions, structs and macros can be created. (I didn't yet have time to
look into GUID constants like GUID_DEVINTERFACE_USB_DEVICE; they probably
don't work due to macro trickery.)

There are 2 structs that do not work (with rather advanced issues):

(1) The generated class _USB_NODE_CONNECTION_INFORMATION_EX cannot be
instanciated. The constructor throws the exception:
java.lang.UnsupportedOperationException: Invalid alignment requirements for
layout s16(DeviceAddress)

This struct (like many USB descriptors) is packed with 16-bit integer
values on odd offsets. jextract insists on creating a layout with strict
alignment rules. So the generated layout
net.codecrete.usb.windows.gen.usbioctl._USB_NODE_CONNECTION_INFORMATION_EX#$struct$LAYOUT
fails, while the manually created layout
net.codecrete.usb.windows.USBIOCtl#USB_NODE_CONNECTION_INFORMATION_EX$Struct
works.

(2) The struct _SP_DEVICE_INTERFACE_DETAIL_DATA_W isn't usable. In C, it
looks like this:

typedef struct _SP_DEVICE_INTERFACE_DETAIL_DATA_W {
  DWORD cbSize;
  WCHAR DevicePath[ANYSIZE_ARRAY];
} SP_DEVICE_INTERFACE_DETAIL_DATA_W, *PSP_DEVICE_INTERFACE_DETAIL_DATA_W;

It's basically a variable length struct. ANYSIZE_ARRAY is defined as 1. But
you are supposed to allocate additional memory for the DevicePath. I guess
this is old style trickery that cannot be solved with an automated tool.

Minor annoyances are:

- jextract on Windows is a batch file that never returns. So it's not
possible to call jextract multiple times for a single batch file.
- It would be nicer if structs used the nice typedef name instead of the
ugly struct name, e.g. SP_DEVICE_INTERFACE_DATA_W instead of
_SP_DEVICE_INTERFACE_DATA_W (note the leading underscore).

Overall: success on Windows.

Regards
Manuel

On Tue, Aug 23, 2022 at 5:48 AM Sundararajan Athijegannathan <
sundararajan.athijegannathan at oracle.com> wrote:

> Thanks for your experiments and comments. I'll go over and file bug(s) as
> needed.
>
> -Sundar
> ------------------------------
> *From:* jextract-dev <jextract-dev-retn at openjdk.org> on behalf of Manuel
> Bleichenbacher <manuel.bleichenbacher at gmail.com>
> *Sent:* 23 August 2022 02:24
> *To:* jextract-dev at openjdk.org <jextract-dev at openjdk.org>
> *Subject:* jextract for operating system API
>
> Hi everybody
>
> As promised, I've compiled my findings when using jextract for the Java
> Does USB project, which uses the foreign function & memory API to
> communicate with USB devices on Windows, macOS and Linux. After initial
> difficulties with jextract, I've written all the layouts, method and
> variable handles manually. But now I've given jextract another try. The
> commands I've used can be found in in the "jextract" branch of the project
> at
> https://github.com/manuelbl/JavaDoesUSB/tree/jextract/java-does-usb/jextract
>
> Please note that I think that automatically generating code for some of
> the old Windows and POSIX APIs is almost impossible. My use of jextract
> (for OS APIs) is probably far from the main use with third-party libraries.
> And I might also have made mistakes, or there might be a simple solution to
> my issues. See my conclusions below for areas where I think jextract has
> still potential.
>
>
> Linux:
>
> Overall, I was able to generate most of what I needed with jextract. The
> issues were:
>
> - Generating code for libsystemd (sd-device.h) failed with "unknown type
> name 'intmax_t'". I think this should be defined by inttypes.h, which is
> included. My guess is that clang headers and Linux headers are mixed up.
> - When generating code for usbdevice_fs.h, it didn't generate the
> constants for USBDEVFS_CLAIMINTERFACE and similar ones. The reason might be
> that the defines look like functions even though they evaluate to a
> constant value.
>
>
> macOS:
>
> Major parts of macOS are provided in the form of frameworks and they are
> treated in a special way, both when it comes to the header files and
> dynamic libraries. As an example, my library uses several functions from
> the CoreFoundation framework. In C/CC++, you can simply include them like
> so:
>
> #include <CoreFoundation/CoreFoundation.h>
>
> That's surprising as the file actually resides in
> /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/CoreFoundation.framework/Versions/A/Headers/CoreFoundation.h.
>
> Note that it is in a directory called "Headers" and not in
> "CoreFoundation" as the include statement suggests. The relative path
> "CoreFoundation/CoreFoundation.h" does not exist at all.
>
> This wouldn't be a big problem if this was just the entry point. But this
> pattern is used recursively. CoreFoundation.h includes about 30 other
> header files like this, all with a relative path that does not exist.
>
> Xcode magically resolves this. But jextract seems to be missing this
> magic. So it cannot be used for any macOS frameworks. In other words, it
> cannot be for the major part of the macOS APIs. My library uses a single
> non-framework functions. So using jextract for macOS is pointless without
> resolving this issue.
>
>
> Windows:
>
> On Windows I spent the most time -- with no success at all. I only
> partially understand the problems. Here is what I've encountered:
>
> The first simple goal was to have code for CloseHandle() generated. I
> tried it three different ways:
>
>    - Generating code for handleapi.h generated 1000 classes. Code for
>    CloseHandle() was generated in 2 classes, but none of them was publicly
>    available.
>    - Generating code for handleapi.h restricted to the function
>    CloseHandle() generated a small number of class with a public static method
>    for CloseHandle(). However, at run-time the method produces the error
>    "java.lang.UnsatisfiedLinkError: unresolved symbol: CloseHandle"
>    - Generating code for windows.h restricted to CloseHandle() also
>    resulted in the UnsatisfiedLinkError.
>
> I then tried to generate code for Winusb.h. Unfortunately, this header
> file is not independent. It expects that you have already processed other
> header files before it. So it fails with "error: unknown type name
> 'LARGE_INTEGER'". This behavior with be reproduced in Visual Studio.
>
> Next up was Cfgmgr32.h. It also doesn't seem to be independent and fails
> with "error: unknown type name 'ULONG'". Setupapi.h and Usbioctl.h behave
> similarly.
>
> Windows was the only system where I had problems with preprocessor macros.
> The macro _M_AMD64=100 is certainly needed. That got me further than
> without it. And I've defined two Unicode macros, copied from the Visual
> Studio project. But the macros might still be the source of some problems.
>
>
>
> Conclusion:
>
> At the moment, jextract doesn't help much for accessing OS APIs.
>
> On Linux, the "unknown type name 'intmax_t'" is worth investigating. It
> could be helpful for many APIs.
>
> On macOS, it's all about the frameworks and the magic behind resolving the
> include path. If it can be made to work, it hopefully opens up the world of
> framework (plus some challenges that are still hidden).
>
> On Windows, more investigation is needed. Work on the required macros is
> needed, as well as more experiments with header files just built for code
> generation (to get beyond the problem of header files that are not
> independent).
>
>
> Regards
> Manuel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jextract-dev/attachments/20220823/9473f770/attachment-0001.htm>