jextract for operating system API

[Manuel Bleichenbacher]

Hi everybody

As promised, I've compiled my findings when using jextract for the Java
Does USB project, which uses the foreign function & memory API to
communicate with USB devices on Windows, macOS and Linux. After initial
difficulties with jextract, I've written all the layouts, method and
variable handles manually. But now I've given jextract another try. The
commands I've used can be found in in the "jextract" branch of the project
at
https://github.com/manuelbl/JavaDoesUSB/tree/jextract/java-does-usb/jextract

Please note that I think that automatically generating code for some of the
old Windows and POSIX APIs is almost impossible. My use of jextract (for OS
APIs) is probably far from the main use with third-party libraries. And I
might also have made mistakes, or there might be a simple solution to my
issues. See my conclusions below for areas where I think jextract has still
potential.


Linux:

Overall, I was able to generate most of what I needed with jextract. The
issues were:

- Generating code for libsystemd (sd-device.h) failed with "unknown type
name 'intmax_t'". I think this should be defined by inttypes.h, which is
included. My guess is that clang headers and Linux headers are mixed up.
- When generating code for usbdevice_fs.h, it didn't generate the constants
for USBDEVFS_CLAIMINTERFACE and similar ones. The reason might be that the
defines look like functions even though they evaluate to a constant value.


macOS:

Major parts of macOS are provided in the form of frameworks and they are
treated in a special way, both when it comes to the header files and
dynamic libraries. As an example, my library uses several functions from
the CoreFoundation framework. In C/CC++, you can simply include them like
so:

#include <CoreFoundation/CoreFoundation.h>

That's surprising as the file actually resides in
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/CoreFoundation.framework/Versions/A/Headers/CoreFoundation.h.

Note that it is in a directory called "Headers" and not in "CoreFoundation"
as the include statement suggests. The relative path
"CoreFoundation/CoreFoundation.h" does not exist at all.

This wouldn't be a big problem if this was just the entry point. But this
pattern is used recursively. CoreFoundation.h includes about 30 other
header files like this, all with a relative path that does not exist.

Xcode magically resolves this. But jextract seems to be missing this magic.
So it cannot be used for any macOS frameworks. In other words, it cannot be
for the major part of the macOS APIs. My library uses a single
non-framework functions. So using jextract for macOS is pointless without
resolving this issue.


Windows:

On Windows I spent the most time -- with no success at all. I only
partially understand the problems. Here is what I've encountered:

The first simple goal was to have code for CloseHandle() generated. I tried
it three different ways:

   - Generating code for handleapi.h generated 1000 classes. Code for
   CloseHandle() was generated in 2 classes, but none of them was publicly
   available.
   - Generating code for handleapi.h restricted to the function
   CloseHandle() generated a small number of class with a public static method
   for CloseHandle(). However, at run-time the method produces the error
   "java.lang.UnsatisfiedLinkError: unresolved symbol: CloseHandle"
   - Generating code for windows.h restricted to CloseHandle() also
   resulted in the UnsatisfiedLinkError.

I then tried to generate code for Winusb.h. Unfortunately, this header file
is not independent. It expects that you have already processed other header
files before it. So it fails with "error: unknown type name
'LARGE_INTEGER'". This behavior with be reproduced in Visual Studio.

Next up was Cfgmgr32.h. It also doesn't seem to be independent and fails
with "error: unknown type name 'ULONG'". Setupapi.h and Usbioctl.h behave
similarly.

Windows was the only system where I had problems with preprocessor macros.
The macro _M_AMD64=100 is certainly needed. That got me further than
without it. And I've defined two Unicode macros, copied from the Visual
Studio project. But the macros might still be the source of some problems.



Conclusion:

At the moment, jextract doesn't help much for accessing OS APIs.

On Linux, the "unknown type name 'intmax_t'" is worth investigating. It
could be helpful for many APIs.

On macOS, it's all about the frameworks and the magic behind resolving the
include path. If it can be made to work, it hopefully opens up the world of
framework (plus some challenges that are still hidden).

On Windows, more investigation is needed. Work on the required macros is
needed, as well as more experiments with header files just built for code
generation (to get beyond the problem of header files that are not
independent).


Regards
Manuel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jextract-dev/attachments/20220822/1de3ba10/attachment-0001.htm>