November 2023 - jextract-dev

RFR: 7903239: ofAddress factory of function pointer type is wrong for struct returns
by Marko 02 Jan '24

02 Jan '24

When generating the lambda inside the `ofAddress` factory, it is missing the `SegmentAllocator` needed when the function returns a struct. This is [implemented for regular functions](https://github.com/openjdk/jextract/blob/0582eaf1b4cdba95f0ee8c2…, but not for function pointers. ------------- Commit messages: - 7903239: ofAddress factory of function pointer type is wrong for struct returns Changes: https://git.openjdk.org/jextract/pull/58/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=58&range=00 Issue: https://bugs.openjdk.org/browse/CODETOOLS-7903239 Stats: 47 lines in 3 files changed: 44 ins; 2 del; 1 mod Patch: https://git.openjdk.org/jextract/pull/58.diff Fetch: git fetch https://git.openjdk.org/jextract pull/58/head:pull/58 PR: https://git.openjdk.org/jextract/pull/58

3 11

RFR: Add path independence
by Per Minborg 02 Jan '24

02 Jan '24

This small PR incorporated feedback from the original PR: https://github.com/openjdk/jextract/pull/66 made after it was integrated. ------------- Commit messages: - Add path independence Changes: https://git.openjdk.org/jextract/pull/68/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=68&range=00 Stats: 10 lines in 4 files changed: 2 ins; 0 del; 8 mod Patch: https://git.openjdk.org/jextract/pull/68.diff Fetch: git fetch https://git.openjdk.org/jextract pull/68/head:pull/68 PR: https://git.openjdk.org/jextract/pull/68

5 7

RFR: Add Dockerfile, zero-dep reproducible builds
by Gavin Ray 02 Jan '24

02 Jan '24

As title says, adds a Dockerfile that will fetch all necessary deps and build `jextract` for you on Ubuntu 22.10 It would be nice if some official JDK organization could put this on their Dockerhub, so that users could just do: $ docker run --rm -it openjdk/jextract:latest I've pushed it to my personal Dockerhub, so if anyone would like to test it out, you can use the README directions but prepend `gavinray/` before the image name: $ echo "struct Foo { int a; long b; }; int use_foo(struct Foo foo);" >> sample.h $ docker run --rm -it -v ${PWD}:/tmp/jextract \ gavinray/jextract:latest \ -I /tmp/jextract \ --source \ -l libfoo \ --output /tmp/jextract/generated \ --target-package com.example \ ------------- Commit messages: - Add Dockerfile, zero-dep reproducible builds Changes: https://git.openjdk.org/jextract/pull/74/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=74&range=00 Stats: 75 lines in 2 files changed: 75 ins; 0 del; 0 mod Patch: https://git.openjdk.org/jextract/pull/74.diff Fetch: git fetch https://git.openjdk.org/jextract pull/74/head:pull/74 PR: https://git.openjdk.org/jextract/pull/74

3 6

RFR: Add third-party sample list to README, populate with java-vulkan.
by Michael Ennen 02 Jan '24

02 Jan '24

I made a more complex example (1000 lines, mostly because setting up a boilerplate Vulkan program is super lengthy) of using jextract on Vulkan and Win32 APIs. I think it may be too lengthy and too much of a maintenance burden to commit to the `/samples` directory, so I thought maybe including a section on third-party samples would be better. If this is not something you want to entertain that's totally understandable, but thought I would put it out there a bigger sample that interfaces with a pretty complex API may be helpful to others. ------------- Commit messages: - Add third-party sample list to README, populate with java-vulkan. Changes: https://git.openjdk.org/jextract/pull/93/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=93&range=00 Stats: 3 lines in 1 file changed: 2 ins; 0 del; 1 mod Patch: https://git.openjdk.org/jextract/pull/93.diff Fetch: git fetch https://git.openjdk.org/jextract pull/93/head:pull/93 PR: https://git.openjdk.org/jextract/pull/93

3 3

RFR: Simplify jextract backend pipeline
by Maurizio Cimadamore 01 Dec '23

01 Dec '23

Now that we have mutable attributes, we can make the jextract pipeline a little simpler, by making sure that each step in the jextract pipeline doesn't destroy the information contained in the jextract model and, additionally, by making sure that each step can only add attributes to the IR (which are acted upon later on). To do this, I had to do some foundational changes to TreeMaker, to make sure that all declaration nodes were being correctly de-duplicated: since declarations are referenced back by types, if a declaration is re-created multiple times, the types pointing to that declaration end up being out of sync, and that's not desirable (because that could lead to a loss of some of the attributes). I also removed the `Transformer` interface: altering the model is not something the jextract pipeline should attempt to do. This meant that existing transformers have been reimplemented as side-effecting visitors, as follows: * `EnumConstantLifter` adds an attribute to constants which belong to an enum (but no longer drops the enum on the ground); * `IncludeFilter` marks several declaration with a `Skip` attribute, to signal `outputFactory` not to generate any code for them; * `DuplicateFilter` also uses the `Skip` attribute to drop redundant declarations. Additionally, the `NameMangler` has been simplified quite a bit, to the point that now it is just a simple visitor which ends up attaching attributes to declarations (`JavaName`, `JavaParameterNames`, `JavaFunctionalInterfaceName`). These attributes are later inspected (from `OutputFactory`). Since now the attribute is added directly to the relevant declaration, there is no need to have complex keying logic inside `NameMangler`. ------------- Commit messages: - Initial push Changes: https://git.openjdk.org/jextract/pull/148/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=148&range=00 Stats: 318 lines in 10 files changed: 52 ins; 181 del; 85 mod Patch: https://git.openjdk.org/jextract/pull/148.diff Fetch: git fetch https://git.openjdk.org/jextract.git pull/148/head:pull/148 PR: https://git.openjdk.org/jextract/pull/148

2 11

RFR: Make attributes mutable
by Maurizio Cimadamore 30 Nov '23

30 Nov '23

This PR makes the attribute list associated with a declaration (and a type) mutable. This means that we can now tweak a declaration/type after the fact. While the declaration/type is still immutable, the information attached to it can grow over time. This should be useful to reduce coupling between different steps in the extraction process. I plan to use this to simplify the way we deal with name mangling. ------------- Commit messages: - Cleanup AttributedImpl - Add copyright - Make attributes mutable Changes: https://git.openjdk.org/jextract/pull/146/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=146&range=00 Stats: 371 lines in 12 files changed: 159 ins; 189 del; 23 mod Patch: https://git.openjdk.org/jextract/pull/146.diff Fetch: git fetch https://git.openjdk.org/jextract.git pull/146/head:pull/146 PR: https://git.openjdk.org/jextract/pull/146

2 5

RFR: Update GHA for panama branch
by Jorn Vernee 30 Nov '23

30 Nov '23

Port of https://github.com/openjdk/jextract/pull/134 + some polish such as using the latest jtreg version, using the latest version of the setup-java and checkout actions, and avoiding use of explicit version numbers in variable names, to avoid having to update all of them again in the future. Historically we've had the github tests disabled on the panama branch, since we technically require a JDK from that panama-foreign repo. However, right now there is no difference between the Panama repo and JDK22, so we can use the JDK22 EA binary temporarily, and get some extra GHA testing. ------------- Commit messages: - Missed renaming of one variable - Polish - Update GHA for jdk22 Changes: https://git.openjdk.org/jextract/pull/147/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=147&range=00 Stats: 20 lines in 1 file changed: 0 ins; 0 del; 20 mod Patch: https://git.openjdk.org/jextract/pull/147.diff Fetch: git fetch https://git.openjdk.org/jextract.git pull/147/head:pull/147 PR: https://git.openjdk.org/jextract/pull/147

2 2

RFR: 7903586: Revisit jextract constant classes
by Maurizio Cimadamore 29 Nov '23

29 Nov '23

This PR removes the use of jextract constant classes, and falls back to a simpler code generation strategies which move the various constants closer to where they belong: * for structs and callbacks, the constants are added to the class/interface being generated; * for the header class, we use a local holder class per function to hold the method handle and function descriptor constants. This scheme results in much less generated code (as no constant class is emitted), but with the same performance and laziness as the old scheme. When working with the code, I've decided to simplify the handling of constant generation, in the spirit of what was recently done with the introduction of string templates. That is, instead of using a magic `Constant` class, which has methods to generate getters/setters for a given constant, I converted much of the code into "static" string templates, so that it's easier to see which code gets generated when staring at the jextract code. As a result, the `Constant` class has been removed. ------------- Commit messages: - Drop unused imports - Revert build changes - Improve indentation code - Re-add proper indentation to layouts and descriptors - Simplify code further - Simplify code - Simplify code - Initial push Changes: https://git.openjdk.org/jextract/pull/145/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=145&range=00 Issue: https://bugs.openjdk.org/browse/CODETOOLS-7903586 Stats: 918 lines in 7 files changed: 287 ins; 577 del; 54 mod Patch: https://git.openjdk.org/jextract/pull/145.diff Fetch: git fetch https://git.openjdk.org/jextract.git pull/145/head:pull/145 PR: https://git.openjdk.org/jextract/pull/145

2 10

RFR: 7903595: Leverage String templates
by Jorn Vernee 28 Nov '23

28 Nov '23

Leverage string templates for source code generation. This makes it much easier to see at a glance what the code that is being generated looks like. It's also a very convenient way of writing this kind of code going forward, and is a good test for the new language feature. Note that I didn't touch the `Constants::emitLayoutField` and `Constants::emitFunctionDescField`, since they do some pretty complex inline generation to render layouts. I didn't feel like this was worth messing with since we are planning to replace the separate constant classes nay way. Though, I did re-write some of the other methods in `Constants` since they were low-hanging fruits. ------------- Commit messages: - leverage string templates in Constants.java in some cases - renamed appendIndented - WIP - use string templates Changes: https://git.openjdk.org/jextract/pull/144/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=144&range=00 Issue: https://bugs.openjdk.org/browse/CODETOOLS-7903595 Stats: 513 lines in 6 files changed: 102 ins; 244 del; 167 mod Patch: https://git.openjdk.org/jextract/pull/144.diff Fetch: git fetch https://git.openjdk.org/jextract.git pull/144/head:pull/144 PR: https://git.openjdk.org/jextract/pull/144

2 5

RFR: 7903590: Refactor jextract source code generation
by Jorn Vernee 27 Nov '23

27 Nov '23

We currently use `ClassSourceBuilder` both for top level classes and nested classes. That means for instance that we need to check whether to generate package declarations and import statements, and for nested classes we also emit code into a separate intermediate buffer that is copied into the enclosing buffer when we end the nested class. This PR proposes splitting part of the logic in `ClassSourceBuilder` into a separate `SourceFileBuilder`, which handles package decl/imports, and holds the underling `StringBuilder` which allows a nested class to share the `StringBuilder` with its enclosing class. Each `SourceFileBuilder` produces exactly one `JavaFileObject`. `TopLevelBuilder` and `Constants` now aggregate over `SourceFileBuilder`s instead of `JavaSourceBuilder`s. This also allows reducing the number of classes that implement `JavaSourceBuilder` to just `TopLevelBuilder` and `StructBuilder`, and allows dropping several methods from the interface. Other builders now wrap `SourceFi leBuilder`, and emit code into its buffer. Jextract source code generation related code currently relies on method overriding to allow sub classes to inject behavior into the generation code in a super class. This, however, makes it hard to follow what method implementation is actually being called when reading code. And, any method call may become 'suspicious' because it might do something non-obvious. For example, `emitConstructor` is overridden to do nothing in several sub-classes. But, this is not obvious when seeing the `emitConstructor()` call in `ClassSourceBuilder::begin`. Furthermore, we use method overriding for instance to specify the class modifiers for a `ClassSourceBuilder` subclass, which means that control flow will flow through several class levels when generating code. This, and other cases, could be replaced by a simple (`String`) value passed to `ClassSourceBuilder` constructor. Using a direct field value in the generation code makes it clear that there is nothing special going on (as the method call might imply). This PR proposes to remove most uses of method overriding, to make it easier to follow control flow when reading the code. This has a few implications: - Direct field access is used where possible, and data is passed to *Builder constructors instead of through a virtual method call to a sub class implementation - `emitConstructor` was not needed in all cases. So, I've moved it from `ClassSourceBuilder::classBegin` to callers of that method instead. This allows the call to be omitted in some cases, without the need to define a sub class that overrides `emitConstructor` to do nothing. (Also renamed to `emitPrivateDefaultConstructor` for clarity). - Doc comments for methods need to be indented. This was previously handled by overriding the `emitDocComment` method in `HeaderFileBuilder` to add the indentation. Instead, I've moved the call to `emitDocComment` to the code generation `emit*` methods, which now handle the indentation for both the doc comment and the code together. - The `TopLevelBuilder.SplitHeader` and `FirstHeader` sub classes are no longer needed. They can be replaced by `HeaderFileBuilder`. I've create a factory method that takes over part of the code gen logic that `FirstHeader` was injecting by overriding `classBegin`. - Sub classes that need to inject extra code gen, now have their own generation methods which compose calls to `classBegin`/`classEnd` with other code gen methods. `classBegin` and `classEnd` now do the dumb/obvious thing. A good example of this is the new `generate` method in `FunctionalInterfaceBuilder`: void generate() { emitDocComment(funcType, className()); classBegin(); emitFunctionalInterfaceMethod(); emitFunctionalFactories(); emitFunctionalFactoryForPointer(); classEnd(); } I quite like how all the code here is in one place. ------------- Commit messages: - don't emit constructor for first header + inline FIB modifiers - remove extra changes - add newSourceFile factory method - use isNested() predicate - fix header comment indentation - reduce inheritance - SourceFileBuilder Changes: https://git.openjdk.org/jextract/pull/141/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=141&range=00 Issue: https://bugs.openjdk.org/browse/CODETOOLS-7903590 Stats: 667 lines in 10 files changed: 236 ins; 258 del; 173 mod Patch: https://git.openjdk.org/jextract/pull/141.diff Fetch: git fetch https://git.openjdk.org/jextract.git pull/141/head:pull/141 PR: https://git.openjdk.org/jextract/pull/141

2 19