October 2023 - jextract-dev

RFR: 7903239: ofAddress factory of function pointer type is wrong for struct returns
by Marko 02 Jan '24

02 Jan '24

When generating the lambda inside the `ofAddress` factory, it is missing the `SegmentAllocator` needed when the function returns a struct. This is [implemented for regular functions](https://github.com/openjdk/jextract/blob/0582eaf1b4cdba95f0ee8c2…, but not for function pointers. ------------- Commit messages: - 7903239: ofAddress factory of function pointer type is wrong for struct returns Changes: https://git.openjdk.org/jextract/pull/58/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=58&range=00 Issue: https://bugs.openjdk.org/browse/CODETOOLS-7903239 Stats: 47 lines in 3 files changed: 44 ins; 2 del; 1 mod Patch: https://git.openjdk.org/jextract/pull/58.diff Fetch: git fetch https://git.openjdk.org/jextract pull/58/head:pull/58 PR: https://git.openjdk.org/jextract/pull/58

3 11

RFR: Add path independence
by Per Minborg 02 Jan '24

02 Jan '24

This small PR incorporated feedback from the original PR: https://github.com/openjdk/jextract/pull/66 made after it was integrated. ------------- Commit messages: - Add path independence Changes: https://git.openjdk.org/jextract/pull/68/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=68&range=00 Stats: 10 lines in 4 files changed: 2 ins; 0 del; 8 mod Patch: https://git.openjdk.org/jextract/pull/68.diff Fetch: git fetch https://git.openjdk.org/jextract pull/68/head:pull/68 PR: https://git.openjdk.org/jextract/pull/68

5 7

RFR: Add Dockerfile, zero-dep reproducible builds
by Gavin Ray 02 Jan '24

02 Jan '24

As title says, adds a Dockerfile that will fetch all necessary deps and build `jextract` for you on Ubuntu 22.10 It would be nice if some official JDK organization could put this on their Dockerhub, so that users could just do: $ docker run --rm -it openjdk/jextract:latest I've pushed it to my personal Dockerhub, so if anyone would like to test it out, you can use the README directions but prepend `gavinray/` before the image name: $ echo "struct Foo { int a; long b; }; int use_foo(struct Foo foo);" >> sample.h $ docker run --rm -it -v ${PWD}:/tmp/jextract \ gavinray/jextract:latest \ -I /tmp/jextract \ --source \ -l libfoo \ --output /tmp/jextract/generated \ --target-package com.example \ ------------- Commit messages: - Add Dockerfile, zero-dep reproducible builds Changes: https://git.openjdk.org/jextract/pull/74/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=74&range=00 Stats: 75 lines in 2 files changed: 75 ins; 0 del; 0 mod Patch: https://git.openjdk.org/jextract/pull/74.diff Fetch: git fetch https://git.openjdk.org/jextract pull/74/head:pull/74 PR: https://git.openjdk.org/jextract/pull/74

3 6

RFR: Add third-party sample list to README, populate with java-vulkan.
by Michael Ennen 02 Jan '24

02 Jan '24

I made a more complex example (1000 lines, mostly because setting up a boilerplate Vulkan program is super lengthy) of using jextract on Vulkan and Win32 APIs. I think it may be too lengthy and too much of a maintenance burden to commit to the `/samples` directory, so I thought maybe including a section on third-party samples would be better. If this is not something you want to entertain that's totally understandable, but thought I would put it out there a bigger sample that interfaces with a pretty complex API may be helpful to others. ------------- Commit messages: - Add third-party sample list to README, populate with java-vulkan. Changes: https://git.openjdk.org/jextract/pull/93/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=93&range=00 Stats: 3 lines in 1 file changed: 2 ins; 0 del; 1 mod Patch: https://git.openjdk.org/jextract/pull/93.diff Fetch: git fetch https://git.openjdk.org/jextract pull/93/head:pull/93 PR: https://git.openjdk.org/jextract/pull/93

3 3

debug logging
by Rel 29 Oct '23

29 Oct '23

Hi all, Developing with jextract is a bit difficult: - there is not too much debug logging, the "-Dlibclang.debug=true" enables certain logging but mostly for clang and not jextract itself. To find out what jextract is doing and trace it, is very difficult. It could be remidiated with running debug under IDE but it does not work well if you generate bindings for big "C" projects. Using "-Djextract.debug=true" as of today seems only prints more information during the exceptions but does not show what steps were taken before it - Sometime you receive "Failed to parse /tmp/jextract$1721579888733730008.h:" message, when you try to see contents of /tmp/jextract$1721579888733730008.h to understand why it is failed the file is already deleted. - no logging on what properties we pass to clang Please let me know if I all this make sense or I am doing something wrong. As part of my experiments on c++ side I integrated JUL to address these problems: https://github.com/enatai/jextractcpp/commit/531f1d6183a53b0a1da0c18d7411f1… If this seems useful for jextract I can send MR for it. Regards,

3 10

RFR: 7903574: jextract should add lang=c to snippets containing C sources
by Weijun Wang 23 Oct '23

23 Oct '23

add the tag when snippet is generated. ------------- Commit messages: - 7903574: jextract should add lang=c to snippets containing C sources Changes: https://git.openjdk.org/jextract/pull/137/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=137&range=00 Issue: https://bugs.openjdk.org/browse/CODETOOLS-7903574 Stats: 2 lines in 1 file changed: 0 ins; 0 del; 2 mod Patch: https://git.openjdk.org/jextract/pull/137.diff Fetch: git fetch https://git.openjdk.org/jextract.git pull/137/head:pull/137 PR: https://git.openjdk.org/jextract/pull/137

2 2

RFR: (jdk22 port) 7903435: TestDocComments fails on Windows
by Jorn Vernee 21 Oct '23

21 Oct '23

Port of https://github.com/openjdk/jextract/pull/106 for jdk 22 branch ------------- Commit messages: - 7903435: TestDocComments fails on Windows Changes: https://git.openjdk.org/jextract/pull/133/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=133&range=00 Stats: 3 lines in 3 files changed: 0 ins; 0 del; 3 mod Patch: https://git.openjdk.org/jextract/pull/133.diff Fetch: git fetch https://git.openjdk.org/jextract.git pull/133/head:pull/133 PR: https://git.openjdk.org/jextract/pull/133

2 2

RFR: Update GHA for jdk22
by Jorn Vernee 21 Oct '23

21 Oct '23

Update GHA to run on jdk 22. Also switched the gradle runner JDK to 17, since that is LTS with Oracle (which is where we get it from). ------------- Commit messages: - Fix mac JAVA path - Update for jdk 22 Changes: https://git.openjdk.org/jextract/pull/134/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=134&range=00 Stats: 16 lines in 1 file changed: 0 ins; 0 del; 16 mod Patch: https://git.openjdk.org/jextract/pull/134.diff Fetch: git fetch https://git.openjdk.org/jextract.git pull/134/head:pull/134 PR: https://git.openjdk.org/jextract/pull/134

2 4

RFR: 7903573: Jextract does not support atomic types
by Maurizio Cimadamore 20 Oct '23

20 Oct '23

This PR fixes a crash when jextract processes atomic types. These types are exposed in libclang in a weird way: their kind is set to `Atomic` but they do not reveal what the underlying type is (except for the type spelling). The underlying type has to be discovered using the `clang_Type_getValueType` function on the `Atomic` type. This PR does just that, and also adds a test for various atomic types. ------------- Commit messages: - Remove whitespaces - Add copyright - Initial push Changes: https://git.openjdk.org/jextract/pull/132/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=132&range=00 Issue: https://bugs.openjdk.org/browse/CODETOOLS-7903573 Stats: 158 lines in 6 files changed: 158 ins; 0 del; 0 mod Patch: https://git.openjdk.org/jextract/pull/132.diff Fetch: git fetch https://git.openjdk.org/jextract.git pull/132/head:pull/132 PR: https://git.openjdk.org/jextract/pull/132

2 2

Re: jextract fails to parse stdatomic.h
by Maurizio Cimadamore 20 Oct '23

20 Oct '23

Hi, A fix for this issue is being worked on here: https://github.com/openjdk/jextract/pull/132 This will be backported to the 21 branch (and we will respin new binaries for it). CheersMaurizio On Fri, 2023-10-20 at 09:51 -0700, Artem Ananev wrote: > Hi Maurizio, > > thank you for quick response. > > Is there a workaround to this issue? Will it be addressed in the > final version of FFI / jextract? > > From what I saw during debugging, your assumption about the > underlying problem looks right. In TypeImpl class there is a field > `layoutMaker`, which implements a type visitor. When this issue with > atomic_flag happens, this visitor throws an unsupported exception > from its visitType() method. > > Thanks, > Artem > > > On Oct 19, 2023, at 5:05 PM, Maurizio Cimadamore < > > maurizio.cimadamore(a)oracle.com> wrote: > > > > Hi Artem, > > thanks for the report. This issue is probably a latent one, which > > has been made manifest by a recent PR: > > > > https://git.openjdk.org/jextract/pull/121 > > > > I believe the underlying cause is that there is some type involved > > in this declaration is not supported (meaning we can't generate a > > layout for it). This causes the layout to be dropped from the > > struct. But now we have a size mismatch, as clang is reporting that > > the size of the struct is 1 byte, but, since we end up with an > > empty struct layout, our size is zero. > > > > If I had to guess (given the headers involved), I'd say that the > > Atomic type kind is not supported by TypeMaker. > > > > Probably related to this old issue: > > https://mail.openjdk.org/pipermail/panama-dev/2019-May/005387.html > > > > Maurizio > > > > On 20/10/2023 00:42, Artem Ananev wrote: > > > Hi, > > > > > > I use jextract version "JDK version 21+35-LTS-2513” on a Ubuntu > > > 22.04.3 box. It fails with the following error: > > > > > > ``` > > > java.lang.AssertionError: Unexpected size for layout []. Found 0 > > > ; expected 1 > > > at org.openjdk.jextract(a)21/org.openjdk.jextract.impl.RecordLayo > > > utComputer.checkSize(RecordLayoutComputer.java:198) > > > at org.openjdk.jextract(a)21/org.openjdk.jextract.impl.StructLayo > > > utComputer.finishRecord(StructLayoutComputer.java:142) > > > at org.openjdk.jextract(a)21/org.openjdk.jextract.impl.RecordLayo > > > utComputer.compute(RecordLayoutComputer.java:114) > > > at org.openjdk.jextract(a)21/org.openjdk.jextract.impl.RecordLayo > > > utComputer.computeInternal(RecordLayoutComputer.java:89) > > > at org.openjdk.jextract(a)21/org.openjdk.jextract.impl.RecordLayo > > > utComputer.compute(RecordLayoutComputer.java:71) > > > at org.openjdk.jextract(a)21/org.openjdk.jextract.impl.TreeMaker. > > > createRecord(TreeMaker.java:205) > > > at org.openjdk.jextract(a)21/org.openjdk.jextract.impl.TreeMaker. > > > createTreeInternal(TreeMaker.java:104) > > > at org.openjdk.jextract(a)21/org.openjdk.jextract.impl.TreeMaker. > > > createTree(TreeMaker.java:93) > > > at org.openjdk.jextract(a)21/org.openjdk.jextract.impl.Parser.lam > > > bda$parse$2(Parser.java:86) > > > at org.openjdk.jextract(a)21/org.openjdk.jextract.clang.Cursor$Cu > > > rsorChildren$Context.visit(Cursor.java:223) > > > at org.openjdk.jextract(a)21/org.openjdk.jextract.clang.Cursor$Cu > > > rsorChildren.lambda$static$0(Cursor.java:242) > > > at org.openjdk.jextract(a)21/org.openjdk.jextract.clang.libclang. > > > Index_h.clang_visitChildren(Index_h.java:5108) > > > at org.openjdk.jextract(a)21/org.openjdk.jextract.clang.Cursor$Cu > > > rsorChildren.forEach(Cursor.java:254) > > > at org.openjdk.jextract(a)21/org.openjdk.jextract.clang.Cursor.fo > > > rEach(Cursor.java:194) > > > at org.openjdk.jextract(a)21/org.openjdk.jextract.impl.Parser.par > > > se(Parser.java:64) > > > at org.openjdk.jextract(a)21/org.openjdk.jextract.JextractTool.pa > > > rse(JextractTool.java:116) > > > at org.openjdk.jextract(a)21/org.openjdk.jextract.JextractTool.ru > > > n(JextractTool.java:466) > > > at org.openjdk.jextract(a)21/org.openjdk.jextract.JextractTool.ma > > > in(JextractTool.java:174) > > > Unrecoverable uncaught exception encountered. The VM will now > > > exit > > > ``` > > > > > > There is no way to understand where exactly it fails, so I > > > downloaded jextract sources, ran it in debug mode, and found the > > > error happens when it’s parsing <stdatomic.h>, in particular this > > > line: > > > > > > ``` > > > typedef struct atomic_flag { atomic_bool _Value; } atomic_flag; > > > ``` > > > > > > If I comment out this line (and all other declarations in the > > > same header that depend on atomic_flag), jextract no longer > > > fails. Note that this header is a part of jextract distribution, > > > it’s in <jextract home>/conf/jextract/stdatomic.h > > > > > > Steps to reproduce: > > > > > > ``` > > > echo '#include <stdatomic.h>' >/tmp/test.h && jextract --source > > > /tmp/test.h > > > ``` > > > > > > Thanks, > > > Artem > > >

1 0

Re: jextract fails to parse stdatomic.h
by Maurizio Cimadamore 20 Oct '23

20 Oct '23

Hi Artem, thanks for the report. This issue is probably a latent one, which has been made manifest by a recent PR: https://git.openjdk.org/jextract/pull/121 I believe the underlying cause is that there is some type involved in this declaration is not supported (meaning we can't generate a layout for it). This causes the layout to be dropped from the struct. But now we have a size mismatch, as clang is reporting that the size of the struct is 1 byte, but, since we end up with an empty struct layout, our size is zero. If I had to guess (given the headers involved), I'd say that the Atomic type kind is not supported by TypeMaker. Probably related to this old issue: https://mail.openjdk.org/pipermail/panama-dev/2019-May/005387.html Maurizio On 20/10/2023 00:42, Artem Ananev wrote: > Hi, > > I use jextract version "JDK version 21+35-LTS-2513” on a Ubuntu 22.04.3 box. It fails with the following error: > > ``` > java.lang.AssertionError: Unexpected size for layout []. Found 0 ; expected 1 > at org.openjdk.jextract@21/org.openjdk.jextract.impl.RecordLayoutComputer.checkSize(RecordLayoutComputer.java:198) > at org.openjdk.jextract@21/org.openjdk.jextract.impl.StructLayoutComputer.finishRecord(StructLayoutComputer.java:142) > at org.openjdk.jextract@21/org.openjdk.jextract.impl.RecordLayoutComputer.compute(RecordLayoutComputer.java:114) > at org.openjdk.jextract@21/org.openjdk.jextract.impl.RecordLayoutComputer.computeInternal(RecordLayoutComputer.java:89) > at org.openjdk.jextract@21/org.openjdk.jextract.impl.RecordLayoutComputer.compute(RecordLayoutComputer.java:71) > at org.openjdk.jextract@21/org.openjdk.jextract.impl.TreeMaker.createRecord(TreeMaker.java:205) > at org.openjdk.jextract@21/org.openjdk.jextract.impl.TreeMaker.createTreeInternal(TreeMaker.java:104) > at org.openjdk.jextract@21/org.openjdk.jextract.impl.TreeMaker.createTree(TreeMaker.java:93) > at org.openjdk.jextract@21/org.openjdk.jextract.impl.Parser.lambda$parse$2(Parser.java:86) > at org.openjdk.jextract@21/org.openjdk.jextract.clang.Cursor$CursorChildren$Context.visit(Cursor.java:223) > at org.openjdk.jextract@21/org.openjdk.jextract.clang.Cursor$CursorChildren.lambda$static$0(Cursor.java:242) > at org.openjdk.jextract@21/org.openjdk.jextract.clang.libclang.Index_h.clang_visitChildren(Index_h.java:5108) > at org.openjdk.jextract@21/org.openjdk.jextract.clang.Cursor$CursorChildren.forEach(Cursor.java:254) > at org.openjdk.jextract@21/org.openjdk.jextract.clang.Cursor.forEach(Cursor.java:194) > at org.openjdk.jextract@21/org.openjdk.jextract.impl.Parser.parse(Parser.java:64) > at org.openjdk.jextract@21/org.openjdk.jextract.JextractTool.parse(JextractTool.java:116) > at org.openjdk.jextract@21/org.openjdk.jextract.JextractTool.run(JextractTool.java:466) > at org.openjdk.jextract@21/org.openjdk.jextract.JextractTool.main(JextractTool.java:174) > Unrecoverable uncaught exception encountered. The VM will now exit > ``` > > There is no way to understand where exactly it fails, so I downloaded jextract sources, ran it in debug mode, and found the error happens when it’s parsing <stdatomic.h>, in particular this line: > > ``` > typedef struct atomic_flag { atomic_bool _Value; } atomic_flag; > ``` > > If I comment out this line (and all other declarations in the same header that depend on atomic_flag), jextract no longer fails. Note that this header is a part of jextract distribution, it’s in <jextract home>/conf/jextract/stdatomic.h > > Steps to reproduce: > > ``` > echo '#include <stdatomic.h>' >/tmp/test.h && jextract --source /tmp/test.h > ``` > > Thanks, > Artem >

1 0

Re: [External] : Re: Jextract failure on llvm 17.0.1 with sdl
by Jorn Vernee 17 Oct '23

17 Oct '23

> I can update my branch with those changes and make an mr out of it if you'd like to use it as reference? Thanks, that could be useful for reference. I think we might want to change the error handling strategy of unexposed types in general, and instead filter them out, rather than crashing with an exception. Jorn On 17/10/2023 02:15, Clayton Walker wrote: > Thanks for the additional details, other than working around it I was > pretty lost when I found out that its kind was (as you mentioned) > unexposed. > > To get around that locally (pretty sure nothing in SDL requires > __bf16) I ended up adding a case to canonical.equalType(t) where if > t.spelling().equals("__bf16") I return > Type.primitive(Primitive.Kind.BFloat16). (Which itself is added to the > Type.Primitive.Kind class as an unsupported layout size 2). > > I can update my branch with those changes and make an mr out of it if > you'd like to use it as reference? > > On Mon, Oct 16, 2023 at 3:06 PM Jorn Vernee <jorn.vernee(a)oracle.com> > wrote: > > Hi, > > This exception comes from the following section in > TypeMaker::makeTypeInternal:" > > case Unexposed: > case Elaborated: > org.openjdk.jextract.clang.Type canonical = > t.canonicalType(); > if (canonical.equalType(t)) { > throw new TypeException("Unknown type with same > canonical type: " + t.spelling()); > } > return makeType(canonical); > > I think in this case we're looking at an 'Unexposed' type, which > means > that libclang (the C interface of clang) does not support this > type [1]. > Looking at the implementation of `GetTypeKind` in libclang [2], it > delegates to `GetBuiltinTypeKind` [3] for builtin types, and this > doesn't have a case for BFloat16, so instead it returns > `CXType_Unexposed`. i.e. this seems to be an issue that needs to be > fixed in libclang itself. There might be other changes that are > needed > to fully support this type, but I can't say off the top of my head. > > As a workaround, we might be able to specifically detect __bf16 > based on > its name/spelling, and then return an instance of Type.Primitive from > TypeMaker::makeTypeInternal, that is later filtered out in > `UnsupportedLayouts`. That would allow jextract to continue and > generate > something, but wouldn't work for cases where the __bf16 type is > actually > required since we would just omit any function/variable that used > this > type in its declaration. > > Jorn > > [1]: > https://clang.llvm.org/doxygen/group__CINDEX__TYPES.html#gaad39de597b13a188… > <https://urldefense.com/v3/__https://clang.llvm.org/doxygen/group__CINDEX__T…> > [2]: > https://github.com/llvm/llvm-project/blob/llvmorg-17.0.1/clang/tools/libcla… > <https://urldefense.com/v3/__https://github.com/llvm/llvm-project/blob/llvmo…> > [3]: > https://github.com/llvm/llvm-project/blob/llvmorg-17.0.1/clang/tools/libcla… > <https://urldefense.com/v3/__https://github.com/llvm/llvm-project/blob/llvmo…> > > On 16/10/2023 20:15, Clayton Walker wrote: > > Hello, > > > > I've been working on creating a set of bindings against sdl2, > and have > > encountered the following issue: > > > > Unknown type with same canonical type: __bf16 > > > > I wasn't sure what needed to be fixed, perhaps a missing > declaration > > or mapping in jextract, but after a bit of digging I tried > adding the > > clang BFloat16 data type but had no luck. > > > https://github.com/openjdk/jextract/compare/jdk21...Sineaggi:jextract:jdk-2… > <https://urldefense.com/v3/__https://github.com/openjdk/jextract/compare/jdk…> > > >

1 0

Why are java and javac included in the prebuilt binaries?
by Oliver Weiler 02 Oct '23

02 Oct '23

Hi everyone! A few users of SDKMAN! have requested to add jextract as a new candidate to SDKMAN!. After adding some of the prebuilt versions, we realized that the bin/ directory contains java and javac in addition to jextract itself. The problem is that SDKMAN! adds the path of jextract's bin/ directory to the PATH, which collides the java and javac versions SDKMAN!s respective Java candidate. Would it be possible to remove java/javac from the prebuilt binaries? Best regards, Oliver

2 1