From maurizio.cimadamore at oracle.com Tue Jan 3 10:14:53 2023 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Tue, 3 Jan 2023 10:14:53 +0000 Subject: JExtract unknown type name 'intmax_t' for /usr/include/yara.h In-Reply-To: References: Message-ID: <7108763b-8e0d-6928-0344-1233aa676591@oracle.com> Hi, this issue seems to be related to the fact that the jextract gradle build did not fail, despite the LLVM path being used not containing all the bells and whistles that the jextract build requires. So, a possible quality of life improvement here would be to add some more sanity checks on the build, to check that the structure of the LLVM folder is as expected. IIRC, the structure of the apt-installed LLVM does not 100% match that of a LLVM binary snapshot. Cheers Maurizio On 28/12/2022 17:23, Maurizio Casciano wrote: > Thanks? to Jorn Vernee, who suggested me to try with the GitHub > LLVM+CLang build archive > I was able to generate the Java bindings for the YARA headers using > this build > https://github.com/llvm/llvm-project/releases/download/llvmorg-15.0.6/clang+llvm-15.0.6-x86_64-linux-gnu-ubuntu-18.04.tar.xz > > A more detailed answer is also available on SO > https://stackoverflow.com/questions/74931278/jextract-yara-headers-throws-unknown-type-name-intmax-t > > Best, > Maurizio > > Il giorno mar 27 dic 2022 alle ore 16:53 Maurizio Casciano > ha scritto: > > Dear All, > I'm trying to generate Java bindings for the yara library but I'm > getting > > |error: unknown type name 'intmax_t' when running the jextract > command:| > > |build/jextract/bin/jextract --source --output java-yara > --target-package com.virustotal.yara /usr/include/yara.h | > > even if I include the relevant additional paths: > > |-I /usr/include -I /usr/include/yara -I > /usr/lib/llvm-15/lib/clang/15.0.6/include| > > Here is my question on Stack-Overflow > https://stackoverflow.com/questions/74931278/jextract-usr-include-yara-h-throws-unknown-type-name-intmax-t > and my GitHub repository > https://github.com/MaurizioCasciano/jextract/tree/unknown_type_name_intmax_t > > What may be missing in the command? > > Best, > Maurizio > > > > -- > > Maurizio Casciano > -------------- next part -------------- An HTML attachment was scrubbed... URL: From duncan.gittins at gmail.com Sun Jan 8 18:24:26 2023 From: duncan.gittins at gmail.com (Duncan Gittins) Date: Sun, 8 Jan 2023 18:24:26 +0000 Subject: Generating a common interface for multiple platform specific binding In-Reply-To: <690e5e0f-4768-7477-b8ad-2ba3737cf580@oracle.com> References: <74cJVGa9pM5ITkWNNcQWqDEttFfBMlhbKsdBGH25SduiUtoPEsY2Parc7F9Ei4EEzNVd6LSIYgu_vUyjGKY_H8LxU7m4OhgQVZ7AW_VNjPg=@protonmail.com> <690e5e0f-4768-7477-b8ad-2ba3737cf580@oracle.com> Message-ID: <52b0f1aa-801a-2f88-6b81-ec210cb57bed@gmail.com> For what its worth, I was interested to find that the simple Freeglut / OpenGL demo programs I've tried on Windows also work on my WSL Ubuntu with the Windows generated jextract classes via X server. All I changed was remove the "-l opengl32 -l freeglut" parameters from Windows jextract run so that it did not inject the Windows loadLibrary calls into RuntimeHelper.java, and substitute equivalent code into the demo instead, like this ??????? System.out.format("Initialise freeglut for os.name=%s%n", System.getProperty("os.name")); ??????? final String[] libs = System.getProperty("os.name").startsWith("Windows") ??????????????????????????? ? new String[]{"opengl32", "freeglut"} :? new String[]{"glut"}; ??????? for (String lib : libs) { ??????????? System.out.format("loadLibrary(\"%s\")%n", lib); ??????????? System.loadLibrary(lib); ??????? } I guess I am lucky the APIs and struct used by the simple code sample are consistent. Kind regards Duncan On 20/12/2022 15:51, Maurizio Cimadamore wrote: > > Hi Martin, > at the time of writing, jextract does not have a solution for this. > > I also believe that a general solution might not even exist - > sometimes the bindings can vary quite wildly across different > platforms, and there could be significant layout mismatches which > would be hard to reconcile. For instance on Windows, C_LONG is a > layout with type ValueLayout.OfInt, whereas on Linux x64 the type is > ValueLayout.OfLong. On top of that, some subset of native functions > might only be available on some platforms but not in others, etc. > > Considering all this, while some automated solution might be possible > for the use case you have in mind, most surely it would fail to scale > to the general case - so the general recommendation is to generate > different bindings (one per platform) and then come up with some > abstraction layer on top (which is sort of what you are trying to do). > Another approach that might work would be to work at a lower level - > and generate ad-hoc downcall method handles which automagically fix up > mismatches that can arise across platforms (e.g. if a function is > available in two platforms with a parameter mismatch int vs. long, > adapt the int-accepting method handle to take a long, so that a > uniform signature can be used). Note that jextract should expose the > method handles it generates, so perhaps this adaptation can be done > semi-automatically based on what comes out of jextract (and based on > what the mismatches really are in your use case). > > I know that e.g. the JavaDoesUSB [1] project has faced similar issues, > so perhaps their authors might want to share some insights here. > > Cheers > Maurizio > > [1] - https://github.com/manuelbl/JavaDoesUSB > > On 20/12/2022 15:04, Martin Pernollet wrote: >> Hi everyone, >> >> I am back on track with OpenGL binding with Panama! >> >> I have one code design / tooling question related to JExtract : is it >> possible to *generate a common interface that would be implemented by >> all platform specific binding* generated by JExtract since JDK 19 or 20? >> >> Here's my use case in more detail : I've been advised to generate >> *different binding for different OS* (and maybe version). For OpenGL, >> this lead me to a glut_h binding for macOS, one for Windows and one >> for Linux. >> >> To let *the user/developer face a single entry point*, I manually >> write a GL interface >> . >> I then define a GL_macOS_10_15_3 >> >> class that wraps the binding (!). When I expand the prototype to >> Windows, I should copy paste this to GL_Windows_10 and modify the >> imports to reference the appropriate bindings. The goal is to write >> code like this >> >> GL gl = Platform.selectAmong(GL_macOS_10_15_3.class, >> GL_windows_10.class, ...) >> gl.glDoSomething() >> >> I don't think there would be another way to have java developer write >> *applications that ignore the target hardware*. However my approach >> is stupid : time consuming and error prone?because manual. A real >> life case may have 10 implementations and 1000 functions. >> >> *Does JExtract provide a solution to this*? Should I create this tool >> by myself based on all generated bindings ? Would anyone *recommend >> something smarter*? >> >> Regards, >> >> Martin >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From maurizio.cimadamore at oracle.com Mon Jan 9 10:18:02 2023 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Mon, 9 Jan 2023 10:18:02 +0000 Subject: Generating a common interface for multiple platform specific binding In-Reply-To: References: <74cJVGa9pM5ITkWNNcQWqDEttFfBMlhbKsdBGH25SduiUtoPEsY2Parc7F9Ei4EEzNVd6LSIYgu_vUyjGKY_H8LxU7m4OhgQVZ7AW_VNjPg=@protonmail.com> <690e5e0f-4768-7477-b8ad-2ba3737cf580@oracle.com> <52b0f1aa-801a-2f88-6b81-ec210cb57bed@gmail.com> Message-ID: On 08/01/2023 18:37, Martin Pernollet wrote: > Thank you Duncan! > > I did something similar : take the binding generated for Ubuntu and > use them on macOS and it was working properly. > > I only add to comment loading of the library inside glut_h and do it > at my application startup so that it uses macOS path rather than the > linux one. Hi Martin, perhaps a better approach would be to not pass any "-l" (lower case "L") to jextract, so that no library name/path is hardwired in the generated bindings. Then your application logic could, inside a static initializer, set up the required libraries, via System::loadLibrary. But we know jextract needs to do more in this area, and have some way for developers to "plug in" their own symbol lookup object. We have a bug for this: https://bugs.openjdk.org/browse/CODETOOLS-7903186 In principle, we could "just" use a service loader - in practice this is more difficult, given that a service loader implies that there is some jextract "runtime" library (e.g. a jar) available when you run your application code. For now we have tried to avoid this (e.g. requiring dependencies on runtime libraries) so that the jextract experience could be simpler/more direct. Maurizio > > Regards, > > Martin > ------- Original Message ------- > Le dimanche 8 janvier 2023 ? 19:24, Duncan Gittins > a ?crit?: > >> For what its worth, I was interested to find that the simple Freeglut >> / OpenGL demo programs I've tried on Windows also work on my WSL >> Ubuntu with the Windows generated jextract classes via X server. >> >> All I changed was remove the "-l opengl32 -l freeglut" parameters >> from Windows jextract run so that it did not inject the Windows >> loadLibrary calls into RuntimeHelper.java, and substitute equivalent >> code into the demo instead, like this >> >> System.out.format("Initialise freeglut for os.name=%s%n", >> System.getProperty("os.name")); >> final String[] libs = System.getProperty("os.name").startsWith("Windows") >> ? new String[]{"opengl32", "freeglut"} : new String[]{"glut"}; >> for (String lib : libs) { >> System.out.format("loadLibrary(\"%s\")%n", lib); >> System.loadLibrary(lib); >> } >> >> I guess I am lucky the APIs and struct used by the simple code sample >> are consistent. >> >> Kind regards >> >> Duncan >> >> >> On 20/12/2022 15:51, Maurizio Cimadamore wrote: >>> >>> Hi Martin, >>> at the time of writing, jextract does not have a solution for this. >>> >>> I also believe that a general solution might not even exist - >>> sometimes the bindings can vary quite wildly across different >>> platforms, and there could be significant layout mismatches which >>> would be hard to reconcile. For instance on Windows, C_LONG is a >>> layout with type ValueLayout.OfInt, whereas on Linux x64 the type is >>> ValueLayout.OfLong. On top of that, some subset of native functions >>> might only be available on some platforms but not in others, etc. >>> >>> Considering all this, while some automated solution might be >>> possible for the use case you have in mind, most surely it would >>> fail to scale to the general case - so the general recommendation is >>> to generate different bindings (one per platform) and then come up >>> with some abstraction layer on top (which is sort of what you are >>> trying to do). Another approach that might work would be to work at >>> a lower level - and generate ad-hoc downcall method handles which >>> automagically fix up mismatches that can arise across platforms >>> (e.g. if a function is available in two platforms with a parameter >>> mismatch int vs. long, adapt the int-accepting method handle to take >>> a long, so that a uniform signature can be used). Note that jextract >>> should expose the method handles it generates, so perhaps this >>> adaptation can be done semi-automatically based on what comes out of >>> jextract (and based on what the mismatches really are in your use >>> case). >>> >>> I know that e.g. the JavaDoesUSB [1] project has faced similar >>> issues, so perhaps their authors might want to share some insights here. >>> >>> Cheers >>> Maurizio >>> >>> [1] - https://github.com/manuelbl/JavaDoesUSB >>> >>> On 20/12/2022 15:04, Martin Pernollet wrote: >>>> Hi everyone, >>>> >>>> I am back on track with OpenGL binding with Panama! >>>> >>>> I have one code design / tooling question related to JExtract : is >>>> it possible to *generate a common interface that would be >>>> implemented by all platform specific binding* generated by JExtract >>>> since JDK 19 or 20? >>>> >>>> Here's my use case in more detail : I've been advised to generate >>>> *different binding for different OS* (and maybe version). For >>>> OpenGL, this lead me to a glut_h binding for macOS, one for Windows >>>> and one for Linux. >>>> >>>> To let *the user/developer face a single entry point*, I manually >>>> write a GL interface >>>> . >>>> I then define a GL_macOS_10_15_3 >>>> >>>> class that wraps the binding (!). When I expand the prototype to >>>> Windows, I should copy paste this to GL_Windows_10 and modify the >>>> imports to reference the appropriate bindings. The goal is to write >>>> code like this >>>> >>>> GL gl = Platform.selectAmong(GL_macOS_10_15_3.class, >>>> GL_windows_10.class, ...) >>>> gl.glDoSomething() >>>> >>>> I don't think there would be another way to have java developer >>>> write *applications that ignore the target hardware*. However my >>>> approach is stupid : time consuming and error prone because manual. >>>> A real life case may have 10 implementations and 1000 functions. >>>> >>>> *Does JExtract provide a solution to this*? Should I create this >>>> tool by myself based on all generated bindings ? Would anyone >>>> *recommend something smarter*? >>>> >>>> Regards, >>>> >>>> Martin >>>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mcimadamore at openjdk.org Mon Jan 9 14:23:00 2023 From: mcimadamore at openjdk.org (Maurizio Cimadamore) Date: Mon, 9 Jan 2023 14:23:00 GMT Subject: RFR: Drop VaList test from jextract Message-ID: Following https://git.openjdk.org/panama-foreign/pull/763, this patch removes a VaList test (which is now failing) from the jextract code. ------------- Commit messages: - Initial push Changes: https://git.openjdk.org/jextract/pull/99/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=99&range=00 Stats: 128 lines in 3 files changed: 0 ins; 128 del; 0 mod Patch: https://git.openjdk.org/jextract/pull/99.diff Fetch: git fetch https://git.openjdk.org/jextract pull/99/head:pull/99 PR: https://git.openjdk.org/jextract/pull/99 From jvernee at openjdk.org Mon Jan 9 14:23:01 2023 From: jvernee at openjdk.org (Jorn Vernee) Date: Mon, 9 Jan 2023 14:23:01 GMT Subject: RFR: Drop VaList test from jextract In-Reply-To: References: Message-ID: <_qBZ6Ah8F_YUjHTbyAEHaptzfLXeApYozzmzFFgn2Xc=.23633d0b-d536-4cdc-b7d6-c4907afc139d@github.com> On Mon, 9 Jan 2023 10:56:24 GMT, Maurizio Cimadamore wrote: > Following https://git.openjdk.org/panama-foreign/pull/763, this patch removes a VaList test (which is now failing) from the jextract code. Marked as reviewed by jvernee (no project role). ------------- PR: https://git.openjdk.org/jextract/pull/99 From mcimadamore at openjdk.org Mon Jan 9 15:00:22 2023 From: mcimadamore at openjdk.org (Maurizio Cimadamore) Date: Mon, 9 Jan 2023 15:00:22 GMT Subject: Integrated: Drop VaList test from jextract In-Reply-To: References: Message-ID: On Mon, 9 Jan 2023 10:56:24 GMT, Maurizio Cimadamore wrote: > Following https://git.openjdk.org/panama-foreign/pull/763, this patch removes a VaList test (which is now failing) from the jextract code. This pull request has now been integrated. Changeset: d30ee95c Author: Maurizio Cimadamore URL: https://git.openjdk.org/jextract/commit/d30ee95c04dff3487235cfe6e7e8383c277ed7ea Stats: 128 lines in 3 files changed: 0 ins; 128 del; 0 mod Drop VaList test from jextract Reviewed-by: jvernee ------------- PR: https://git.openjdk.org/jextract/pull/99 From remm at apache.org Mon Jan 30 13:25:57 2023 From: remm at apache.org (=?UTF-8?Q?R=C3=A9my_Maucherat?=) Date: Mon, 30 Jan 2023 14:25:57 +0100 Subject: jextract and bit fields Message-ID: Hi, When processing certain bit-fields, jextract produces an error: "Invalid bitSize: 1" In addition to not being really helpful (it took me some time to find out what the problem was since no reference to a location is given, nor that this is about a bit-field). A typical example causing the error is something like: struct my_struct { unsigned int foo : 1; }; Obviously we don't have these sort of memory saving "smart" fancy tricks in Java (which is probably good since they can cause serious problems when misused ...). Is support for this planned eventually, or should bit-fields be avoided for Panama compatibility ? Thanks ! R?my From maurizio.cimadamore at oracle.com Mon Jan 30 16:39:58 2023 From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore) Date: Mon, 30 Jan 2023 16:39:58 +0000 Subject: jextract and bit fields In-Reply-To: References: Message-ID: <0f263042-9186-50ae-18e3-50a00efd5496@oracle.com> Hey Remi, while jextract does not support bitfields, it should also not crash when seeing them :-) I have tried extracting a simple header containing the struct in your email using the jextract binary at [1]. I could successfully extract it, and obtain a |my_struct| Java wrapper, with this layout: |static final GroupLayout $struct$LAYOUT = MemoryLayout.structLayout( MemoryLayout.structLayout( MemoryLayout.paddingLayout(1).withName("foo"), MemoryLayout.paddingLayout(31) ) ).withName("my_struct"); | As expected, the contents of the struct are just padding. That said, I?m beginnin to suspect that you are seeing this issue with a more recent version of jextract, perhaps the one for the ?panama? branch? We have made recent changes to how padding layouts are defined - e.g. so that their size/alignment must be a multiple of 8. This would cause the issue you are seeing. Maurizio On 30/01/2023 13:25, R?my Maucherat wrote: > Hi, > > When processing certain bit-fields, jextract produces an error: > "Invalid bitSize: 1" > In addition to not being really helpful (it took me some time to find > out what the problem was since no reference to a location is given, > nor that this is about a bit-field). > > A typical example causing the error is something like: > struct my_struct { > unsigned int foo : 1; > }; > > Obviously we don't have these sort of memory saving "smart" fancy > tricks in Java (which is probably good since they can cause serious > problems when misused ...). > > Is support for this planned eventually, or should bit-fields be > avoided for Panama compatibility ? > > Thanks ! > > R?my ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From mcimadamore at openjdk.org Mon Jan 30 17:45:31 2023 From: mcimadamore at openjdk.org (Maurizio Cimadamore) Date: Mon, 30 Jan 2023 17:45:31 GMT Subject: RFR: Update code to use address target layouts Message-ID: This patch tweaks jextract to replace usages of `OfAddress.asUnbounded` with: `withTargetLayout(sequenceLayout(JAVA_BYTE))` ------------- Commit messages: - Update code to use target layouts Changes: https://git.openjdk.org/jextract/pull/100/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=100&range=00 Stats: 11 lines in 4 files changed: 7 ins; 0 del; 4 mod Patch: https://git.openjdk.org/jextract/pull/100.diff Fetch: git fetch https://git.openjdk.org/jextract pull/100/head:pull/100 PR: https://git.openjdk.org/jextract/pull/100 From mcimadamore at openjdk.org Mon Jan 30 17:45:32 2023 From: mcimadamore at openjdk.org (Maurizio Cimadamore) Date: Mon, 30 Jan 2023 17:45:32 GMT Subject: RFR: Update code to use address target layouts In-Reply-To: References: Message-ID: On Mon, 30 Jan 2023 17:38:54 GMT, Maurizio Cimadamore wrote: > This patch tweaks jextract to replace usages of `OfAddress.asUnbounded` with: > > `withTargetLayout(sequenceLayout(JAVA_BYTE))` Note: there are some failing tests (bitfields) due to the changes introduced by https://git.openjdk.org/panama-foreign/pull/766. We will deal with those in another PR. ------------- PR: https://git.openjdk.org/jextract/pull/100 From jvernee at openjdk.org Mon Jan 30 18:17:46 2023 From: jvernee at openjdk.org (Jorn Vernee) Date: Mon, 30 Jan 2023 18:17:46 GMT Subject: RFR: Update code to use address target layouts In-Reply-To: References: Message-ID: On Mon, 30 Jan 2023 17:38:54 GMT, Maurizio Cimadamore wrote: > This patch tweaks jextract to replace usages of `OfAddress.asUnbounded` with: > > `withTargetLayout(sequenceLayout(JAVA_BYTE))` Marked as reviewed by jvernee (no project role). ------------- PR: https://git.openjdk.org/jextract/pull/100 From mcimadamore at openjdk.org Mon Jan 30 18:52:33 2023 From: mcimadamore at openjdk.org (Maurizio Cimadamore) Date: Mon, 30 Jan 2023 18:52:33 GMT Subject: Integrated: Update code to use address target layouts In-Reply-To: References: Message-ID: On Mon, 30 Jan 2023 17:38:54 GMT, Maurizio Cimadamore wrote: > This patch tweaks jextract to replace usages of `OfAddress.asUnbounded` with: > > `withTargetLayout(sequenceLayout(JAVA_BYTE))` This pull request has now been integrated. Changeset: 2c5b10b3 Author: Maurizio Cimadamore URL: https://git.openjdk.org/jextract/commit/2c5b10b3b4793fcdf2fe98206c8d64d7e517bc0f Stats: 11 lines in 4 files changed: 7 ins; 0 del; 4 mod Update code to use address target layouts Reviewed-by: jvernee ------------- PR: https://git.openjdk.org/jextract/pull/100 From mcimadamore at openjdk.org Mon Jan 30 22:42:16 2023 From: mcimadamore at openjdk.org (Maurizio Cimadamore) Date: Mon, 30 Jan 2023 22:42:16 GMT Subject: RFR: Fix jextract after removal of support for sub-byte layouts Message-ID: This patch fixes some test failures which are caused by https://git.openjdk.org/panama-foreign/pull/766. As that patch removes support for sub-byte layouts, we can no longer represent bitfields using group layouts (because we can no longer model group elements whose size is not a multiple off 8). In reality, jextract isn't doing much with bitfields: it just generates a bunch of padding layouts of the right size and storing them into a group layout, but this layout, of course, cannot be used for access (because sub-byte layout doesn't work with VarHandles). To fix this, I have added a new subtype of `Declaration.Variable` - namely, `Declaration.Bitfield` - this is like a variable - it has a name (but no layout), and has a type. On top of this, a `Bitfield` also stores an `offset` (relative to the contained) and a `width` (the size in bits of the bitfield). Most of the changes in this patch are caused by the fact that the underlying IR for bitfields has changed (as described above). Since bitfields are still variables (whose kind is `BITFIELD`) and since variable of that kind are ignored by the jextract backend, nothing is going to change, really. That said, should we want to add bitfield accessors one day, all the info to generate the accessor should be available in the jextract IR. ------------- Commit messages: - Initial push Changes: https://git.openjdk.org/jextract/pull/101/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=101&range=00 Stats: 125 lines in 8 files changed: 80 ins; 21 del; 24 mod Patch: https://git.openjdk.org/jextract/pull/101.diff Fetch: git fetch https://git.openjdk.org/jextract pull/101/head:pull/101 PR: https://git.openjdk.org/jextract/pull/101