January 2023 - jextract-dev

RFR: 7903239: ofAddress factory of function pointer type is wrong for struct returns
by Marko 02 Jan '24

02 Jan '24

When generating the lambda inside the `ofAddress` factory, it is missing the `SegmentAllocator` needed when the function returns a struct. This is [implemented for regular functions](https://github.com/openjdk/jextract/blob/0582eaf1b4cdba95f0ee8c2…, but not for function pointers. ------------- Commit messages: - 7903239: ofAddress factory of function pointer type is wrong for struct returns Changes: https://git.openjdk.org/jextract/pull/58/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=58&range=00 Issue: https://bugs.openjdk.org/browse/CODETOOLS-7903239 Stats: 47 lines in 3 files changed: 44 ins; 2 del; 1 mod Patch: https://git.openjdk.org/jextract/pull/58.diff Fetch: git fetch https://git.openjdk.org/jextract pull/58/head:pull/58 PR: https://git.openjdk.org/jextract/pull/58

3 11

RFR: Add path independence
by Per Minborg 02 Jan '24

02 Jan '24

This small PR incorporated feedback from the original PR: https://github.com/openjdk/jextract/pull/66 made after it was integrated. ------------- Commit messages: - Add path independence Changes: https://git.openjdk.org/jextract/pull/68/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=68&range=00 Stats: 10 lines in 4 files changed: 2 ins; 0 del; 8 mod Patch: https://git.openjdk.org/jextract/pull/68.diff Fetch: git fetch https://git.openjdk.org/jextract pull/68/head:pull/68 PR: https://git.openjdk.org/jextract/pull/68

5 7

RFR: Add Dockerfile, zero-dep reproducible builds
by Gavin Ray 02 Jan '24

02 Jan '24

As title says, adds a Dockerfile that will fetch all necessary deps and build `jextract` for you on Ubuntu 22.10 It would be nice if some official JDK organization could put this on their Dockerhub, so that users could just do: $ docker run --rm -it openjdk/jextract:latest I've pushed it to my personal Dockerhub, so if anyone would like to test it out, you can use the README directions but prepend `gavinray/` before the image name: $ echo "struct Foo { int a; long b; }; int use_foo(struct Foo foo);" >> sample.h $ docker run --rm -it -v ${PWD}:/tmp/jextract \ gavinray/jextract:latest \ -I /tmp/jextract \ --source \ -l libfoo \ --output /tmp/jextract/generated \ --target-package com.example \ ------------- Commit messages: - Add Dockerfile, zero-dep reproducible builds Changes: https://git.openjdk.org/jextract/pull/74/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=74&range=00 Stats: 75 lines in 2 files changed: 75 ins; 0 del; 0 mod Patch: https://git.openjdk.org/jextract/pull/74.diff Fetch: git fetch https://git.openjdk.org/jextract pull/74/head:pull/74 PR: https://git.openjdk.org/jextract/pull/74

3 6

RFR: Add third-party sample list to README, populate with java-vulkan.
by Michael Ennen 02 Jan '24

02 Jan '24

I made a more complex example (1000 lines, mostly because setting up a boilerplate Vulkan program is super lengthy) of using jextract on Vulkan and Win32 APIs. I think it may be too lengthy and too much of a maintenance burden to commit to the `/samples` directory, so I thought maybe including a section on third-party samples would be better. If this is not something you want to entertain that's totally understandable, but thought I would put it out there a bigger sample that interfaces with a pretty complex API may be helpful to others. ------------- Commit messages: - Add third-party sample list to README, populate with java-vulkan. Changes: https://git.openjdk.org/jextract/pull/93/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=93&range=00 Stats: 3 lines in 1 file changed: 2 ins; 0 del; 1 mod Patch: https://git.openjdk.org/jextract/pull/93.diff Fetch: git fetch https://git.openjdk.org/jextract pull/93/head:pull/93 PR: https://git.openjdk.org/jextract/pull/93

3 3

RFR: Fix jextract after removal of support for sub-byte layouts
by Maurizio Cimadamore 01 Feb '23

01 Feb '23

This patch fixes some test failures which are caused by https://git.openjdk.org/panama-foreign/pull/766. As that patch removes support for sub-byte layouts, we can no longer represent bitfields using group layouts (because we can no longer model group elements whose size is not a multiple off 8). In reality, jextract isn't doing much with bitfields: it just generates a bunch of padding layouts of the right size and storing them into a group layout, but this layout, of course, cannot be used for access (because sub-byte layout doesn't work with VarHandles). To fix this, I have added a new subtype of `Declaration.Variable` - namely, `Declaration.Bitfield` - this is like a variable - it has a name (but no layout), and has a type. On top of this, a `Bitfield` also stores an `offset` (relative to the contained) and a `width` (the size in bits of the bitfield). Most of the changes in this patch are caused by the fact that the underlying IR for bitfields has changed (as described above). Since bitfields are still variables (whose kind is `BITFIELD`) and since variable of that kind are ignored by the jextract backend, nothing is going to change, really. That said, should we want to add bitfield accessors one day, all the info to generate the accessor should be available in the jextract IR. ------------- Commit messages: - Initial push Changes: https://git.openjdk.org/jextract/pull/101/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=101&range=00 Stats: 125 lines in 8 files changed: 80 ins; 21 del; 24 mod Patch: https://git.openjdk.org/jextract/pull/101.diff Fetch: git fetch https://git.openjdk.org/jextract pull/101/head:pull/101 PR: https://git.openjdk.org/jextract/pull/101

2 2

jextract and bit fields
by Rémy Maucherat 01 Feb '23

01 Feb '23

Hi, When processing certain bit-fields, jextract produces an error: "Invalid bitSize: 1" In addition to not being really helpful (it took me some time to find out what the problem was since no reference to a location is given, nor that this is about a bit-field). A typical example causing the error is something like: struct my_struct { unsigned int foo : 1; }; Obviously we don't have these sort of memory saving "smart" fancy tricks in Java (which is probably good since they can cause serious problems when misused ...). Is support for this planned eventually, or should bit-fields be avoided for Panama compatibility ? Thanks ! Rémy

2 2

RFR: Update code to use address target layouts
by Maurizio Cimadamore 30 Jan '23

30 Jan '23

This patch tweaks jextract to replace usages of `OfAddress.asUnbounded` with: `withTargetLayout(sequenceLayout(JAVA_BYTE))` ------------- Commit messages: - Update code to use target layouts Changes: https://git.openjdk.org/jextract/pull/100/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=100&range=00 Stats: 11 lines in 4 files changed: 7 ins; 0 del; 4 mod Patch: https://git.openjdk.org/jextract/pull/100.diff Fetch: git fetch https://git.openjdk.org/jextract pull/100/head:pull/100 PR: https://git.openjdk.org/jextract/pull/100

2 3

RFR: Drop VaList test from jextract
by Maurizio Cimadamore 09 Jan '23

09 Jan '23

Following https://git.openjdk.org/panama-foreign/pull/763, this patch removes a VaList test (which is now failing) from the jextract code. ------------- Commit messages: - Initial push Changes: https://git.openjdk.org/jextract/pull/99/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=99&range=00 Stats: 128 lines in 3 files changed: 0 ins; 128 del; 0 mod Patch: https://git.openjdk.org/jextract/pull/99.diff Fetch: git fetch https://git.openjdk.org/jextract pull/99/head:pull/99 PR: https://git.openjdk.org/jextract/pull/99

2 2

Re: Generating a common interface for multiple platform specific binding
by Maurizio Cimadamore 09 Jan '23

09 Jan '23

On 08/01/2023 18:37, Martin Pernollet wrote: > Thank you Duncan! > > I did something similar : take the binding generated for Ubuntu and > use them on macOS and it was working properly. > > I only add to comment loading of the library inside glut_h and do it > at my application startup so that it uses macOS path rather than the > linux one. Hi Martin, perhaps a better approach would be to not pass any "-l" (lower case "L") to jextract, so that no library name/path is hardwired in the generated bindings. Then your application logic could, inside a static initializer, set up the required libraries, via System::loadLibrary. But we know jextract needs to do more in this area, and have some way for developers to "plug in" their own symbol lookup object. We have a bug for this: https://bugs.openjdk.org/browse/CODETOOLS-7903186 In principle, we could "just" use a service loader - in practice this is more difficult, given that a service loader implies that there is some jextract "runtime" library (e.g. a jar) available when you run your application code. For now we have tried to avoid this (e.g. requiring dependencies on runtime libraries) so that the jextract experience could be simpler/more direct. Maurizio > > Regards, > > Martin > ------- Original Message ------- > Le dimanche 8 janvier 2023 à 19:24, Duncan Gittins > <duncan.gittins(a)gmail.com> a écrit : > >> For what its worth, I was interested to find that the simple Freeglut >> / OpenGL demo programs I've tried on Windows also work on my WSL >> Ubuntu with the Windows generated jextract classes via X server. >> >> All I changed was remove the "-l opengl32 -l freeglut" parameters >> from Windows jextract run so that it did not inject the Windows >> loadLibrary calls into RuntimeHelper.java, and substitute equivalent >> code into the demo instead, like this >> >> System.out.format("Initialise freeglut for os.name=%s%n", >> System.getProperty("os.name")); >> final String[] libs = System.getProperty("os.name").startsWith("Windows") >> ? new String[]{"opengl32", "freeglut"} : new String[]{"glut"}; >> for (String lib : libs) { >> System.out.format("loadLibrary(\"%s\")%n", lib); >> System.loadLibrary(lib); >> } >> >> I guess I am lucky the APIs and struct used by the simple code sample >> are consistent. >> >> Kind regards >> >> Duncan >> >> >> On 20/12/2022 15:51, Maurizio Cimadamore wrote: >>> >>> Hi Martin, >>> at the time of writing, jextract does not have a solution for this. >>> >>> I also believe that a general solution might not even exist - >>> sometimes the bindings can vary quite wildly across different >>> platforms, and there could be significant layout mismatches which >>> would be hard to reconcile. For instance on Windows, C_LONG is a >>> layout with type ValueLayout.OfInt, whereas on Linux x64 the type is >>> ValueLayout.OfLong. On top of that, some subset of native functions >>> might only be available on some platforms but not in others, etc. >>> >>> Considering all this, while some automated solution might be >>> possible for the use case you have in mind, most surely it would >>> fail to scale to the general case - so the general recommendation is >>> to generate different bindings (one per platform) and then come up >>> with some abstraction layer on top (which is sort of what you are >>> trying to do). Another approach that might work would be to work at >>> a lower level - and generate ad-hoc downcall method handles which >>> automagically fix up mismatches that can arise across platforms >>> (e.g. if a function is available in two platforms with a parameter >>> mismatch int vs. long, adapt the int-accepting method handle to take >>> a long, so that a uniform signature can be used). Note that jextract >>> should expose the method handles it generates, so perhaps this >>> adaptation can be done semi-automatically based on what comes out of >>> jextract (and based on what the mismatches really are in your use >>> case). >>> >>> I know that e.g. the JavaDoesUSB [1] project has faced similar >>> issues, so perhaps their authors might want to share some insights here. >>> >>> Cheers >>> Maurizio >>> >>> [1] - https://github.com/manuelbl/JavaDoesUSB >>> >>> On 20/12/2022 15:04, Martin Pernollet wrote: >>>> Hi everyone, >>>> >>>> I am back on track with OpenGL binding with Panama! >>>> >>>> I have one code design / tooling question related to JExtract : is >>>> it possible to *generate a common interface that would be >>>> implemented by all platform specific binding* generated by JExtract >>>> since JDK 19 or 20? >>>> >>>> Here's my use case in more detail : I've been advised to generate >>>> *different binding for different OS* (and maybe version). For >>>> OpenGL, this lead me to a glut_h binding for macOS, one for Windows >>>> and one for Linux. >>>> >>>> To let *the user/developer face a single entry point*, I manually >>>> write a GL interface >>>> <https://urldefense.com/v3/__https://github.com/jzy3d/panama-gl/blob/feature…>. >>>> I then define a GL_macOS_10_15_3 >>>> <https://urldefense.com/v3/__https://github.com/jzy3d/panama-gl/blob/feature…> >>>> class that wraps the binding (!). When I expand the prototype to >>>> Windows, I should copy paste this to GL_Windows_10 and modify the >>>> imports to reference the appropriate bindings. The goal is to write >>>> code like this >>>> >>>> GL gl = Platform.selectAmong(GL_macOS_10_15_3.class, >>>> GL_windows_10.class, ...) >>>> gl.glDoSomething() >>>> >>>> I don't think there would be another way to have java developer >>>> write *applications that ignore the target hardware*. However my >>>> approach is stupid : time consuming and error prone because manual. >>>> A real life case may have 10 implementations and 1000 functions. >>>> >>>> *Does JExtract provide a solution to this*? Should I create this >>>> tool by myself based on all generated bindings ? Would anyone >>>> *recommend something smarter*? >>>> >>>> Regards, >>>> >>>> Martin >>>> >> >

1 0

Re: Generating a common interface for multiple platform specific binding
by Duncan Gittins 08 Jan '23

08 Jan '23

For what its worth, I was interested to find that the simple Freeglut / OpenGL demo programs I've tried on Windows also work on my WSL Ubuntu with the Windows generated jextract classes via X server. All I changed was remove the "-l opengl32 -l freeglut" parameters from Windows jextract run so that it did not inject the Windows loadLibrary calls into RuntimeHelper.java, and substitute equivalent code into the demo instead, like this System.out.format("Initialise freeglut for os.name=%s%n", System.getProperty("os.name")); final String[] libs = System.getProperty("os.name").startsWith("Windows") ? new String[]{"opengl32", "freeglut"} : new String[]{"glut"}; for (String lib : libs) { System.out.format("loadLibrary(\"%s\")%n", lib); System.loadLibrary(lib); } I guess I am lucky the APIs and struct used by the simple code sample are consistent. Kind regards Duncan On 20/12/2022 15:51, Maurizio Cimadamore wrote: > > Hi Martin, > at the time of writing, jextract does not have a solution for this. > > I also believe that a general solution might not even exist - > sometimes the bindings can vary quite wildly across different > platforms, and there could be significant layout mismatches which > would be hard to reconcile. For instance on Windows, C_LONG is a > layout with type ValueLayout.OfInt, whereas on Linux x64 the type is > ValueLayout.OfLong. On top of that, some subset of native functions > might only be available on some platforms but not in others, etc. > > Considering all this, while some automated solution might be possible > for the use case you have in mind, most surely it would fail to scale > to the general case - so the general recommendation is to generate > different bindings (one per platform) and then come up with some > abstraction layer on top (which is sort of what you are trying to do). > Another approach that might work would be to work at a lower level - > and generate ad-hoc downcall method handles which automagically fix up > mismatches that can arise across platforms (e.g. if a function is > available in two platforms with a parameter mismatch int vs. long, > adapt the int-accepting method handle to take a long, so that a > uniform signature can be used). Note that jextract should expose the > method handles it generates, so perhaps this adaptation can be done > semi-automatically based on what comes out of jextract (and based on > what the mismatches really are in your use case). > > I know that e.g. the JavaDoesUSB [1] project has faced similar issues, > so perhaps their authors might want to share some insights here. > > Cheers > Maurizio > > [1] - https://github.com/manuelbl/JavaDoesUSB > > On 20/12/2022 15:04, Martin Pernollet wrote: >> Hi everyone, >> >> I am back on track with OpenGL binding with Panama! >> >> I have one code design / tooling question related to JExtract : is it >> possible to *generate a common interface that would be implemented by >> all platform specific binding* generated by JExtract since JDK 19 or 20? >> >> Here's my use case in more detail : I've been advised to generate >> *different binding for different OS* (and maybe version). For OpenGL, >> this lead me to a glut_h binding for macOS, one for Windows and one >> for Linux. >> >> To let *the user/developer face a single entry point*, I manually >> write a GL interface >> <https://github.com/jzy3d/panama-gl/blob/feature/fbo/src/main/java/opengl/GL…>. >> I then define a GL_macOS_10_15_3 >> <https://github.com/jzy3d/panama-gl/blob/feature/fbo/src/main/java/opengl/ma…> >> class that wraps the binding (!). When I expand the prototype to >> Windows, I should copy paste this to GL_Windows_10 and modify the >> imports to reference the appropriate bindings. The goal is to write >> code like this >> >> GL gl = Platform.selectAmong(GL_macOS_10_15_3.class, >> GL_windows_10.class, ...) >> gl.glDoSomething() >> >> I don't think there would be another way to have java developer write >> *applications that ignore the target hardware*. However my approach >> is stupid : time consuming and error prone because manual. A real >> life case may have 10 implementations and 1000 functions. >> >> *Does JExtract provide a solution to this*? Should I create this tool >> by myself based on all generated bindings ? Would anyone *recommend >> something smarter*? >> >> Regards, >> >> Martin >>

1 0