From im at lostillusion.net  Mon Aug  1 18:56:42 2022
From: im at lostillusion.net (Marko)
Date: Mon, 1 Aug 2022 14:56:42 -0400
Subject: BUG: function pointers inside a struct do not generate
 SegmentAllocators when needed
In-Reply-To: <f3de9911-5014-8bde-52b1-cc36428b9134@oracle.com>
References: <CAHFdwNboGw===xxZ3_3czsi7sf=SHqBsWNEq0nSsj-rzMtUGkA@mail.gmail.com>
 <18e2cc45-c1fc-4ff7-4aa1-2fad60d40187@oracle.com>
 <CAHFdwNb0O6nwu_ajLHy9VfTrv4OdfvJqWFZ5LqTd7MB03WXAQQ@mail.gmail.com>
 <CAHFdwNaVTWJdqkf=bidiJN-iQeyyi8vyF0Xe__bOeP0Q9Ncx8g@mail.gmail.com>
 <f3de9911-5014-8bde-52b1-cc36428b9134@oracle.com>
Message-ID: <CAHFdwNabm-Yq6knHNuwkH+RyGwrfaAa3kZsp1QMN-diyUZSc-g@mail.gmail.com>

Hey Maurizio,
Thanks for the explanation, it helped a lot with understanding with
what's going on. I'm not quite sure how to cause a problem with the
two use cases you described, though.

Scenario 1 would be equivalent to using the allocate method in the FI I believe.
Scenario 2 would be calling ofAddress on the memory segment/address we
got from 1.

This is what I tried, and it seemed to work, though. The linker does
not seem to add a SegmentAllocator when creating the upcall, like you
said. However, wrapping the returned memory segment/address, from
scenario 1, inside the FI using ofAddress and calling it works...? I'm
assuming somewhere my logic is flawed, and the method handle should
not accept the additional SegmentAllocator when invoking. I just don't
quite understand why it doesn't fail. Anyway, I think your solution is
quite reasonable and that it should use the implicit allocator by
default as well, as that seems to be the behavior in other places of
jextract.

Thanks
Marko

On Fri, Jul 29, 2022 at 5:41 PM Maurizio Cimadamore
<maurizio.cimadamore at oracle.com> wrote:
>
> Hi Marko,
> I think adding the missing SegmentAllocator parameter can be possible
> fix, but the reality here is that there is an asymmetry between two use
> cases:
>
> 1. when you create an upcall from a lambda expression, the segment
> allocator parameter is not required (in fact the upcall machinery will
> _not_ provide one to the upcall, see below)
> 2. when you want to wrap a functional interface around an upcall stub
> address, the struct returned by the stub will be copied into a _fresh_
> segment allocated using the provided segment allocator
>
> Because of this, I think that just prepending a SegmentAllocator
> parameter in front of the functional descriptor is not gonna cut it.
> That is, if you have a C function pointer like:
>
> ```
> struct Foo (*f)(int)
> ```
>
> The Linker will expect an upcall MethodHandle with signature
> `(int)MemorySegment`, and no SegmentAllocator parameter. If you add an
> extra SegmentAllocator parameter, then the method handle we derive from
> the lambda expression in (1) is going to have the wrong type (unless we
> insert a dummy SegmentAllocator argument).
>
> When chatting with Jorn offline about this, we were discussing the
> possibility of emitting, for the above signature, a functional interface
> like the one below:
>
> ```
> interface FI$f {
>
>     MemorySegment apply(int i);
>     default MemorySegment(SegmentAllocator allocator, int i) { throw new
> UnsupportedOperationException(); }
>
>     ...
> }
> ```
>
> This has some advantages:
>
> * we can still provide a lambda-friendly factory for (1) - note that the
> functional interface signature would still be the "correct" one for the
> upcall (e.g. the one w/o the SegmentAllocator), so nothing changes here
> * we can also provide the `ofAddress` method for (2), which builds a new
> FI$f given a memory address; to do that, we create an instance as follows:
>
> ```
> FI$f ofAddress(MemoryAddress address) {
>     ...
>
>     return new FI$f() {
>        MemorySegment apply(int i) { return
> apply(SegmentAllocator.implicitAllocator(), i); }
>        MemorySegment apply(SegmentAllocator, int i) { <call downcall
> handle> }
>     };
> }
> ```
>
> in other words, the returned functional interface instance supports two
> invocation modes, w/ and w/o a SegmentAllocator. If no SegmentAllocator
> is provided, we just use the implicit allocator (another option would be
> to throw, but perhaps that would be too harsh?)
>
> Thoughts?
>
> Cheers
> Maurizio
>
>
>
>
>
>
> On 29/07/2022 00:42, Marko wrote:
> > Perhaps the additional parameter was not as weird as I originally thought.
> >> It seems a bit odd though IMHO, as then
> >> HeaderFileBuilder#addVar and ToplevelBuilder#addVar are also forced to
> >> have this parameter even though it goes unused there.
> > I just tried it out and the getters are also missing when generated as
> > a part of the header file.
> > I'll polish my fix and send a PR for a better opinion.
> >
> > Thanks
> > Marko
> >
> >
> > On Thu, Jul 28, 2022 at 7:25 PM Marko <im at lostillusion.net> wrote:
> >> My apologies, gmail seems to have only replied directly to Jorn
> >> previously. Here is the original message.
> >>
> >> Hey Jorn,
> >> Thanks for filing the report. I agree that the problem is where the
> >> `ofAddress` function is generated. It does not check to see if a
> >> segment allocator is needed. It seems quite trivial to add though as
> >> it is already implemented for regular functions,
> >> https://github.com/openjdk/jextract/blob/0582eaf1b4cdba95f0ee8c2480767433bb647d0d/src/main/java/org/openjdk/jextract/impl/HeaderFileBuilder.java#L122-L127.
> >>
> >> However, when this is fixed, the getter must also be updated to
> >> reflect the addition of the SegmentAllocator parameter.
> >> I believe this constitutes an additional parameter here
> >> https://github.com/openjdk/jextract/blob/0582eaf1b4cdba95f0ee8c2480767433bb647d0d/src/main/java/org/openjdk/jextract/impl/StructBuilder.java#L128=
> >> similar to fiName, that represents whether a segment allocator is
> >> needed or not. It seems a bit odd though IMHO, as then
> >> HeaderFileBuilder#addVar and ToplevelBuilder#addVar are also forced to
> >> have this parameter even though it goes unused there.
> >>
> >> I implemented the fix and the proper getter generation as well, and it
> >> seems to all work on my end. I wonder if this boolean parameter is the
> >> way to go before submitting a PR though.
> >>
> >> Thanks,
> >> Marko
> >>
> >> On Thu, Jul 28, 2022 at 5:20 PM Jorn Vernee <jorn.vernee at oracle.com> wrote:
> >>> Hello Marko,
> >>>
> >>> Thanks for the report! I can reproduce it here as well. I think the
> >>> problem is the lambda in the `ofAddress` factory that is generated in
> >>> the interface for the callback type. Note that the getter will just end
> >>> up calling that factory. (you can see that also in your stack trace: `at
> >>> Foo$bar.lambda$ofAddress$0(Foo.java:32)`).
> >>>
> >>> I've filed: https://bugs.openjdk.org/browse/CODETOOLS-7903239
> >>>
> >>> Cheers,
> >>> Jorn
> >>>
> >>> On 28/07/2022 02:00, Marko wrote:
> >>>> Hello again!
> >>>> When trying to call a function pointer inside a struct which returns a
> >>>> MemorySegment, I receive the following error:
> >>>> ```
> >>>> Exception in thread "main" java.lang.AssertionError: should not reach here
> >>>> at Foo$bar.lambda$ofAddress$0(Foo.java:34)
> >>>> at Main.main(Main.java:8)
> >>>> Caused by: java.lang.invoke.WrongMethodTypeException: expected
> >>>> (NativeSymbol,SegmentAllocator)MemorySegment but found
> >>>> (NativeSymbol)MemorySegment
> >>>> at java.base/java.lang.invoke.Invokers.newWrongMethodTypeException(Invokers.java:523)
> >>>> at java.base/java.lang.invoke.Invokers.checkExactType(Invokers.java:532)
> >>>> at Foo$bar.lambda$ofAddress$0(Foo.java:32)
> >>>> ... 1 more
> >>>> ```
> >>>> I believe this is due to
> >>>> `FunctionalInterfaceBuilder#emitFunctionalFactoryForPointer` not
> >>>> checking to see if the returnType is a MemorySegment, as it does in
> >>>> `HeaderFileBuilder` for regular functions, see
> >>>> https://github.com/openjdk/jextract/blob/0582eaf1b4cdba95f0ee8c2480767433bb647d0d/src/main/java/org/openjdk/jextract/impl/HeaderFileBuilder.java#L122-L127.
> >>>>
> >>>> I began implementing a fix by copying this functionality over, however
> >>>> there is a bit of an issue I came across. When jextract emits the
> >>>> getter for the functional interface in a struct (StructBuilder), only
> >>>> the name of the interface is known. AFAIK, it does not know whether it
> >>>> needs a SegmentAllocator and cannot currently generate the correct
> >>>> code.
> >>>>
> >>>> I thought about adding another field to the method
> >>>> `emitFunctionalFactoryForPointer`, similar to fiName, like a boolean
> >>>> for whether a segment allocator is needed or not, however, I can't help but
> >>>> feel it is a bit hacky. I wonder what your opinions are on this.
> >>>>
> >>>> A repro can be found here:
> >>>> https://github.com/lost-illusi0n/jextract-func-ptr-seg-alloc-repro
> >>>> Just run the run script if you are on Linux. You will have to change the
> >>>> library output in the script if you are on another os.

From im at lostillusion.net  Mon Aug  1 19:06:37 2022
From: im at lostillusion.net (Marko)
Date: Mon, 1 Aug 2022 15:06:37 -0400
Subject: BUG: function pointers inside a struct do not generate
 SegmentAllocators when needed
In-Reply-To: <CAHFdwNabm-Yq6knHNuwkH+RyGwrfaAa3kZsp1QMN-diyUZSc-g@mail.gmail.com>
References: <CAHFdwNboGw===xxZ3_3czsi7sf=SHqBsWNEq0nSsj-rzMtUGkA@mail.gmail.com>
 <18e2cc45-c1fc-4ff7-4aa1-2fad60d40187@oracle.com>
 <CAHFdwNb0O6nwu_ajLHy9VfTrv4OdfvJqWFZ5LqTd7MB03WXAQQ@mail.gmail.com>
 <CAHFdwNaVTWJdqkf=bidiJN-iQeyyi8vyF0Xe__bOeP0Q9Ncx8g@mail.gmail.com>
 <f3de9911-5014-8bde-52b1-cc36428b9134@oracle.com>
 <CAHFdwNabm-Yq6knHNuwkH+RyGwrfaAa3kZsp1QMN-diyUZSc-g@mail.gmail.com>
Message-ID: <CAHFdwNakBxMgiDdb12gybAtvbWADzuw=81U=9uGKQVSY1f2XDA@mail.gmail.com>

Disregard.

On Mon, Aug 1, 2022 at 2:56 PM Marko <im at lostillusion.net> wrote:
>
> Hey Maurizio,
> Thanks for the explanation, it helped a lot with understanding with
> what's going on. I'm not quite sure how to cause a problem with the
> two use cases you described, though.
>
> Scenario 1 would be equivalent to using the allocate method in the FI I believe.
> Scenario 2 would be calling ofAddress on the memory segment/address we
> got from 1.
>
> This is what I tried, and it seemed to work, though. The linker does
> not seem to add a SegmentAllocator when creating the upcall, like you
> said. However, wrapping the returned memory segment/address, from
> scenario 1, inside the FI using ofAddress and calling it works...? I'm
> assuming somewhere my logic is flawed, and the method handle should
> not accept the additional SegmentAllocator when invoking. I just don't
> quite understand why it doesn't fail. Anyway, I think your solution is
> quite reasonable and that it should use the implicit allocator by
> default as well, as that seems to be the behavior in other places of
> jextract.
>
> Thanks
> Marko
>
> On Fri, Jul 29, 2022 at 5:41 PM Maurizio Cimadamore
> <maurizio.cimadamore at oracle.com> wrote:
> >
> > Hi Marko,
> > I think adding the missing SegmentAllocator parameter can be possible
> > fix, but the reality here is that there is an asymmetry between two use
> > cases:
> >
> > 1. when you create an upcall from a lambda expression, the segment
> > allocator parameter is not required (in fact the upcall machinery will
> > _not_ provide one to the upcall, see below)
> > 2. when you want to wrap a functional interface around an upcall stub
> > address, the struct returned by the stub will be copied into a _fresh_
> > segment allocated using the provided segment allocator
> >
> > Because of this, I think that just prepending a SegmentAllocator
> > parameter in front of the functional descriptor is not gonna cut it.
> > That is, if you have a C function pointer like:
> >
> > ```
> > struct Foo (*f)(int)
> > ```
> >
> > The Linker will expect an upcall MethodHandle with signature
> > `(int)MemorySegment`, and no SegmentAllocator parameter. If you add an
> > extra SegmentAllocator parameter, then the method handle we derive from
> > the lambda expression in (1) is going to have the wrong type (unless we
> > insert a dummy SegmentAllocator argument).
> >
> > When chatting with Jorn offline about this, we were discussing the
> > possibility of emitting, for the above signature, a functional interface
> > like the one below:
> >
> > ```
> > interface FI$f {
> >
> >     MemorySegment apply(int i);
> >     default MemorySegment(SegmentAllocator allocator, int i) { throw new
> > UnsupportedOperationException(); }
> >
> >     ...
> > }
> > ```
> >
> > This has some advantages:
> >
> > * we can still provide a lambda-friendly factory for (1) - note that the
> > functional interface signature would still be the "correct" one for the
> > upcall (e.g. the one w/o the SegmentAllocator), so nothing changes here
> > * we can also provide the `ofAddress` method for (2), which builds a new
> > FI$f given a memory address; to do that, we create an instance as follows:
> >
> > ```
> > FI$f ofAddress(MemoryAddress address) {
> >     ...
> >
> >     return new FI$f() {
> >        MemorySegment apply(int i) { return
> > apply(SegmentAllocator.implicitAllocator(), i); }
> >        MemorySegment apply(SegmentAllocator, int i) { <call downcall
> > handle> }
> >     };
> > }
> > ```
> >
> > in other words, the returned functional interface instance supports two
> > invocation modes, w/ and w/o a SegmentAllocator. If no SegmentAllocator
> > is provided, we just use the implicit allocator (another option would be
> > to throw, but perhaps that would be too harsh?)
> >
> > Thoughts?
> >
> > Cheers
> > Maurizio
> >
> >
> >
> >
> >
> >
> > On 29/07/2022 00:42, Marko wrote:
> > > Perhaps the additional parameter was not as weird as I originally thought.
> > >> It seems a bit odd though IMHO, as then
> > >> HeaderFileBuilder#addVar and ToplevelBuilder#addVar are also forced to
> > >> have this parameter even though it goes unused there.
> > > I just tried it out and the getters are also missing when generated as
> > > a part of the header file.
> > > I'll polish my fix and send a PR for a better opinion.
> > >
> > > Thanks
> > > Marko
> > >
> > >
> > > On Thu, Jul 28, 2022 at 7:25 PM Marko <im at lostillusion.net> wrote:
> > >> My apologies, gmail seems to have only replied directly to Jorn
> > >> previously. Here is the original message.
> > >>
> > >> Hey Jorn,
> > >> Thanks for filing the report. I agree that the problem is where the
> > >> `ofAddress` function is generated. It does not check to see if a
> > >> segment allocator is needed. It seems quite trivial to add though as
> > >> it is already implemented for regular functions,
> > >> https://github.com/openjdk/jextract/blob/0582eaf1b4cdba95f0ee8c2480767433bb647d0d/src/main/java/org/openjdk/jextract/impl/HeaderFileBuilder.java#L122-L127.
> > >>
> > >> However, when this is fixed, the getter must also be updated to
> > >> reflect the addition of the SegmentAllocator parameter.
> > >> I believe this constitutes an additional parameter here
> > >> https://github.com/openjdk/jextract/blob/0582eaf1b4cdba95f0ee8c2480767433bb647d0d/src/main/java/org/openjdk/jextract/impl/StructBuilder.java#L128=
> > >> similar to fiName, that represents whether a segment allocator is
> > >> needed or not. It seems a bit odd though IMHO, as then
> > >> HeaderFileBuilder#addVar and ToplevelBuilder#addVar are also forced to
> > >> have this parameter even though it goes unused there.
> > >>
> > >> I implemented the fix and the proper getter generation as well, and it
> > >> seems to all work on my end. I wonder if this boolean parameter is the
> > >> way to go before submitting a PR though.
> > >>
> > >> Thanks,
> > >> Marko
> > >>
> > >> On Thu, Jul 28, 2022 at 5:20 PM Jorn Vernee <jorn.vernee at oracle.com> wrote:
> > >>> Hello Marko,
> > >>>
> > >>> Thanks for the report! I can reproduce it here as well. I think the
> > >>> problem is the lambda in the `ofAddress` factory that is generated in
> > >>> the interface for the callback type. Note that the getter will just end
> > >>> up calling that factory. (you can see that also in your stack trace: `at
> > >>> Foo$bar.lambda$ofAddress$0(Foo.java:32)`).
> > >>>
> > >>> I've filed: https://bugs.openjdk.org/browse/CODETOOLS-7903239
> > >>>
> > >>> Cheers,
> > >>> Jorn
> > >>>
> > >>> On 28/07/2022 02:00, Marko wrote:
> > >>>> Hello again!
> > >>>> When trying to call a function pointer inside a struct which returns a
> > >>>> MemorySegment, I receive the following error:
> > >>>> ```
> > >>>> Exception in thread "main" java.lang.AssertionError: should not reach here
> > >>>> at Foo$bar.lambda$ofAddress$0(Foo.java:34)
> > >>>> at Main.main(Main.java:8)
> > >>>> Caused by: java.lang.invoke.WrongMethodTypeException: expected
> > >>>> (NativeSymbol,SegmentAllocator)MemorySegment but found
> > >>>> (NativeSymbol)MemorySegment
> > >>>> at java.base/java.lang.invoke.Invokers.newWrongMethodTypeException(Invokers.java:523)
> > >>>> at java.base/java.lang.invoke.Invokers.checkExactType(Invokers.java:532)
> > >>>> at Foo$bar.lambda$ofAddress$0(Foo.java:32)
> > >>>> ... 1 more
> > >>>> ```
> > >>>> I believe this is due to
> > >>>> `FunctionalInterfaceBuilder#emitFunctionalFactoryForPointer` not
> > >>>> checking to see if the returnType is a MemorySegment, as it does in
> > >>>> `HeaderFileBuilder` for regular functions, see
> > >>>> https://github.com/openjdk/jextract/blob/0582eaf1b4cdba95f0ee8c2480767433bb647d0d/src/main/java/org/openjdk/jextract/impl/HeaderFileBuilder.java#L122-L127.
> > >>>>
> > >>>> I began implementing a fix by copying this functionality over, however
> > >>>> there is a bit of an issue I came across. When jextract emits the
> > >>>> getter for the functional interface in a struct (StructBuilder), only
> > >>>> the name of the interface is known. AFAIK, it does not know whether it
> > >>>> needs a SegmentAllocator and cannot currently generate the correct
> > >>>> code.
> > >>>>
> > >>>> I thought about adding another field to the method
> > >>>> `emitFunctionalFactoryForPointer`, similar to fiName, like a boolean
> > >>>> for whether a segment allocator is needed or not, however, I can't help but
> > >>>> feel it is a bit hacky. I wonder what your opinions are on this.
> > >>>>
> > >>>> A repro can be found here:
> > >>>> https://github.com/lost-illusi0n/jextract-func-ptr-seg-alloc-repro
> > >>>> Just run the run script if you are on Linux. You will have to change the
> > >>>> library output in the script if you are on another os.

From im at lostillusion.net  Mon Aug  1 19:08:34 2022
From: im at lostillusion.net (Marko)
Date: Mon, 1 Aug 2022 15:08:34 -0400
Subject: BUG: function pointers inside a struct do not generate
 SegmentAllocators when needed
In-Reply-To: <f3de9911-5014-8bde-52b1-cc36428b9134@oracle.com>
References: <CAHFdwNboGw===xxZ3_3czsi7sf=SHqBsWNEq0nSsj-rzMtUGkA@mail.gmail.com>
 <18e2cc45-c1fc-4ff7-4aa1-2fad60d40187@oracle.com>
 <CAHFdwNb0O6nwu_ajLHy9VfTrv4OdfvJqWFZ5LqTd7MB03WXAQQ@mail.gmail.com>
 <CAHFdwNaVTWJdqkf=bidiJN-iQeyyi8vyF0Xe__bOeP0Q9Ncx8g@mail.gmail.com>
 <f3de9911-5014-8bde-52b1-cc36428b9134@oracle.com>
Message-ID: <CAHFdwNYCaULQR3hVcmtb+dxTt03USX5KYA8oPcCCHaFkPQ9H5Q@mail.gmail.com>

Further email difficulties/default settings are odd, apologies!

Hey Maurizio,
Thanks for the explanation, it helped a lot with understanding with
what's going on. I'm not quite sure how to cause a problem with the
two use cases you described, though.

Scenario 1 would be equivalent to using the allocate method in the FI I believe.
Scenario 2 would be calling ofAddress on the memory segment/address we
got from 1.

This is what I tried, and it seemed to work, though. The linker does
not seem to add a SegmentAllocator when creating the upcall, like you
said. However, wrapping the returned memory segment/address, from
scenario 1, inside the FI using ofAddress and calling it works...? I'm
assuming somewhere my logic is flawed, and the method handle should
not accept the additional SegmentAllocator when invoking. I just don't
quite understand why it doesn't fail. Anyway, I think your solution is
quite reasonable and that it should use the implicit allocator by
default as well, as that seems to be the behavior in other places of
jextract.

Thanks
Marko

On Fri, Jul 29, 2022 at 5:41 PM Maurizio Cimadamore
<maurizio.cimadamore at oracle.com> wrote:
>
> Hi Marko,
> I think adding the missing SegmentAllocator parameter can be possible
> fix, but the reality here is that there is an asymmetry between two use
> cases:
>
> 1. when you create an upcall from a lambda expression, the segment
> allocator parameter is not required (in fact the upcall machinery will
> _not_ provide one to the upcall, see below)
> 2. when you want to wrap a functional interface around an upcall stub
> address, the struct returned by the stub will be copied into a _fresh_
> segment allocated using the provided segment allocator
>
> Because of this, I think that just prepending a SegmentAllocator
> parameter in front of the functional descriptor is not gonna cut it.
> That is, if you have a C function pointer like:
>
> ```
> struct Foo (*f)(int)
> ```
>
> The Linker will expect an upcall MethodHandle with signature
> `(int)MemorySegment`, and no SegmentAllocator parameter. If you add an
> extra SegmentAllocator parameter, then the method handle we derive from
> the lambda expression in (1) is going to have the wrong type (unless we
> insert a dummy SegmentAllocator argument).
>
> When chatting with Jorn offline about this, we were discussing the
> possibility of emitting, for the above signature, a functional interface
> like the one below:
>
> ```
> interface FI$f {
>
>     MemorySegment apply(int i);
>     default MemorySegment(SegmentAllocator allocator, int i) { throw new
> UnsupportedOperationException(); }
>
>     ...
> }
> ```
>
> This has some advantages:
>
> * we can still provide a lambda-friendly factory for (1) - note that the
> functional interface signature would still be the "correct" one for the
> upcall (e.g. the one w/o the SegmentAllocator), so nothing changes here
> * we can also provide the `ofAddress` method for (2), which builds a new
> FI$f given a memory address; to do that, we create an instance as follows:
>
> ```
> FI$f ofAddress(MemoryAddress address) {
>     ...
>
>     return new FI$f() {
>        MemorySegment apply(int i) { return
> apply(SegmentAllocator.implicitAllocator(), i); }
>        MemorySegment apply(SegmentAllocator, int i) { <call downcall
> handle> }
>     };
> }
> ```
>
> in other words, the returned functional interface instance supports two
> invocation modes, w/ and w/o a SegmentAllocator. If no SegmentAllocator
> is provided, we just use the implicit allocator (another option would be
> to throw, but perhaps that would be too harsh?)
>
> Thoughts?
>
> Cheers
> Maurizio
>
>
>
>
>
>
> On 29/07/2022 00:42, Marko wrote:
> > Perhaps the additional parameter was not as weird as I originally thought.
> >> It seems a bit odd though IMHO, as then
> >> HeaderFileBuilder#addVar and ToplevelBuilder#addVar are also forced to
> >> have this parameter even though it goes unused there.
> > I just tried it out and the getters are also missing when generated as
> > a part of the header file.
> > I'll polish my fix and send a PR for a better opinion.
> >
> > Thanks
> > Marko
> >
> >
> > On Thu, Jul 28, 2022 at 7:25 PM Marko <im at lostillusion.net> wrote:
> >> My apologies, gmail seems to have only replied directly to Jorn
> >> previously. Here is the original message.
> >>
> >> Hey Jorn,
> >> Thanks for filing the report. I agree that the problem is where the
> >> `ofAddress` function is generated. It does not check to see if a
> >> segment allocator is needed. It seems quite trivial to add though as
> >> it is already implemented for regular functions,
> >> https://github.com/openjdk/jextract/blob/0582eaf1b4cdba95f0ee8c2480767433bb647d0d/src/main/java/org/openjdk/jextract/impl/HeaderFileBuilder.java#L122-L127.
> >>
> >> However, when this is fixed, the getter must also be updated to
> >> reflect the addition of the SegmentAllocator parameter.
> >> I believe this constitutes an additional parameter here
> >> https://github.com/openjdk/jextract/blob/0582eaf1b4cdba95f0ee8c2480767433bb647d0d/src/main/java/org/openjdk/jextract/impl/StructBuilder.java#L128=
> >> similar to fiName, that represents whether a segment allocator is
> >> needed or not. It seems a bit odd though IMHO, as then
> >> HeaderFileBuilder#addVar and ToplevelBuilder#addVar are also forced to
> >> have this parameter even though it goes unused there.
> >>
> >> I implemented the fix and the proper getter generation as well, and it
> >> seems to all work on my end. I wonder if this boolean parameter is the
> >> way to go before submitting a PR though.
> >>
> >> Thanks,
> >> Marko
> >>
> >> On Thu, Jul 28, 2022 at 5:20 PM Jorn Vernee <jorn.vernee at oracle.com> wrote:
> >>> Hello Marko,
> >>>
> >>> Thanks for the report! I can reproduce it here as well. I think the
> >>> problem is the lambda in the `ofAddress` factory that is generated in
> >>> the interface for the callback type. Note that the getter will just end
> >>> up calling that factory. (you can see that also in your stack trace: `at
> >>> Foo$bar.lambda$ofAddress$0(Foo.java:32)`).
> >>>
> >>> I've filed: https://bugs.openjdk.org/browse/CODETOOLS-7903239
> >>>
> >>> Cheers,
> >>> Jorn
> >>>
> >>> On 28/07/2022 02:00, Marko wrote:
> >>>> Hello again!
> >>>> When trying to call a function pointer inside a struct which returns a
> >>>> MemorySegment, I receive the following error:
> >>>> ```
> >>>> Exception in thread "main" java.lang.AssertionError: should not reach here
> >>>> at Foo$bar.lambda$ofAddress$0(Foo.java:34)
> >>>> at Main.main(Main.java:8)
> >>>> Caused by: java.lang.invoke.WrongMethodTypeException: expected
> >>>> (NativeSymbol,SegmentAllocator)MemorySegment but found
> >>>> (NativeSymbol)MemorySegment
> >>>> at java.base/java.lang.invoke.Invokers.newWrongMethodTypeException(Invokers.java:523)
> >>>> at java.base/java.lang.invoke.Invokers.checkExactType(Invokers.java:532)
> >>>> at Foo$bar.lambda$ofAddress$0(Foo.java:32)
> >>>> ... 1 more
> >>>> ```
> >>>> I believe this is due to
> >>>> `FunctionalInterfaceBuilder#emitFunctionalFactoryForPointer` not
> >>>> checking to see if the returnType is a MemorySegment, as it does in
> >>>> `HeaderFileBuilder` for regular functions, see
> >>>> https://github.com/openjdk/jextract/blob/0582eaf1b4cdba95f0ee8c2480767433bb647d0d/src/main/java/org/openjdk/jextract/impl/HeaderFileBuilder.java#L122-L127.
> >>>>
> >>>> I began implementing a fix by copying this functionality over, however
> >>>> there is a bit of an issue I came across. When jextract emits the
> >>>> getter for the functional interface in a struct (StructBuilder), only
> >>>> the name of the interface is known. AFAIK, it does not know whether it
> >>>> needs a SegmentAllocator and cannot currently generate the correct
> >>>> code.
> >>>>
> >>>> I thought about adding another field to the method
> >>>> `emitFunctionalFactoryForPointer`, similar to fiName, like a boolean
> >>>> for whether a segment allocator is needed or not, however, I can't help but
> >>>> feel it is a bit hacky. I wonder what your opinions are on this.
> >>>>
> >>>> A repro can be found here:
> >>>> https://github.com/lost-illusi0n/jextract-func-ptr-seg-alloc-repro
> >>>> Just run the run script if you are on Linux. You will have to change the
> >>>> library output in the script if you are on another os.

From maurizio.cimadamore at oracle.com  Mon Aug  1 21:09:04 2022
From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore)
Date: Mon, 1 Aug 2022 22:09:04 +0100
Subject: BUG: function pointers inside a struct do not generate
 SegmentAllocators when needed
In-Reply-To: <CAHFdwNYCaULQR3hVcmtb+dxTt03USX5KYA8oPcCCHaFkPQ9H5Q@mail.gmail.com>
References: <CAHFdwNboGw===xxZ3_3czsi7sf=SHqBsWNEq0nSsj-rzMtUGkA@mail.gmail.com>
 <18e2cc45-c1fc-4ff7-4aa1-2fad60d40187@oracle.com>
 <CAHFdwNb0O6nwu_ajLHy9VfTrv4OdfvJqWFZ5LqTd7MB03WXAQQ@mail.gmail.com>
 <CAHFdwNaVTWJdqkf=bidiJN-iQeyyi8vyF0Xe__bOeP0Q9Ncx8g@mail.gmail.com>
 <f3de9911-5014-8bde-52b1-cc36428b9134@oracle.com>
 <CAHFdwNYCaULQR3hVcmtb+dxTt03USX5KYA8oPcCCHaFkPQ9H5Q@mail.gmail.com>
Message-ID: <cdcc8b2d-947b-1e2c-e88f-11e5280de150@oracle.com>

Hi Marko,
I took a look at your PR:

https://github.com/openjdk/jextract/pull/58/files

What you do there works because, even though the downcall that 
`ofAddress` creates accepts an additional SegmentAllocator parameter, 
you provide such extra parameter in here:

```
 ???? if(fiType.returnType().equals(MemorySegment.class)) {
 ??????????????? append(", (SegmentAllocator)session");
 ??????????? }
```

E.g. you take the session passed to `ofAddress` and turn that into a 
`SegmentAllocator` (which works, since MemorySession implements 
SegmentAllocator), and use that to allocate.

By doing that, you effectively inject a "default" allocator into the 
downcall method handle, which makes sure that the returned MemorySegment 
belongs to the provided session.

This is a very simple, yet effective fix (well done!), and I think one 
that could be good enough, at least for now.

Where this fix type of fix leaves a bit to be desired is that it doesn't 
provide the degree of flexibility that a client might want. Say you have 
a more efficient SegmentAllocator, and you want to use _that_ to 
allocate the returned struct; that is not possible with the approach in 
your PR.

That said, I'm not sure how big of a problem that is, given that in 
special situations clients can still wrap function pointers on their 
own, if they are not happy with what jextract provides. So I think your 
simple solution probably goes a long way (and we can always come back 
later and tweak this again).

Thanks!
Maurizio

On 01/08/2022 20:08, Marko wrote:
> Further email difficulties/default settings are odd, apologies!
>
> Hey Maurizio,
> Thanks for the explanation, it helped a lot with understanding with
> what's going on. I'm not quite sure how to cause a problem with the
> two use cases you described, though.
>
> Scenario 1 would be equivalent to using the allocate method in the FI I believe.
> Scenario 2 would be calling ofAddress on the memory segment/address we
> got from 1.
>
> This is what I tried, and it seemed to work, though. The linker does
> not seem to add a SegmentAllocator when creating the upcall, like you
> said. However, wrapping the returned memory segment/address, from
> scenario 1, inside the FI using ofAddress and calling it works...? I'm
> assuming somewhere my logic is flawed, and the method handle should
> not accept the additional SegmentAllocator when invoking. I just don't
> quite understand why it doesn't fail. Anyway, I think your solution is
> quite reasonable and that it should use the implicit allocator by
> default as well, as that seems to be the behavior in other places of
> jextract.
>
> Thanks
> Marko
>
> On Fri, Jul 29, 2022 at 5:41 PM Maurizio Cimadamore
> <maurizio.cimadamore at oracle.com> wrote:
>> Hi Marko,
>> I think adding the missing SegmentAllocator parameter can be possible
>> fix, but the reality here is that there is an asymmetry between two use
>> cases:
>>
>> 1. when you create an upcall from a lambda expression, the segment
>> allocator parameter is not required (in fact the upcall machinery will
>> _not_ provide one to the upcall, see below)
>> 2. when you want to wrap a functional interface around an upcall stub
>> address, the struct returned by the stub will be copied into a _fresh_
>> segment allocated using the provided segment allocator
>>
>> Because of this, I think that just prepending a SegmentAllocator
>> parameter in front of the functional descriptor is not gonna cut it.
>> That is, if you have a C function pointer like:
>>
>> ```
>> struct Foo (*f)(int)
>> ```
>>
>> The Linker will expect an upcall MethodHandle with signature
>> `(int)MemorySegment`, and no SegmentAllocator parameter. If you add an
>> extra SegmentAllocator parameter, then the method handle we derive from
>> the lambda expression in (1) is going to have the wrong type (unless we
>> insert a dummy SegmentAllocator argument).
>>
>> When chatting with Jorn offline about this, we were discussing the
>> possibility of emitting, for the above signature, a functional interface
>> like the one below:
>>
>> ```
>> interface FI$f {
>>
>>      MemorySegment apply(int i);
>>      default MemorySegment(SegmentAllocator allocator, int i) { throw new
>> UnsupportedOperationException(); }
>>
>>      ...
>> }
>> ```
>>
>> This has some advantages:
>>
>> * we can still provide a lambda-friendly factory for (1) - note that the
>> functional interface signature would still be the "correct" one for the
>> upcall (e.g. the one w/o the SegmentAllocator), so nothing changes here
>> * we can also provide the `ofAddress` method for (2), which builds a new
>> FI$f given a memory address; to do that, we create an instance as follows:
>>
>> ```
>> FI$f ofAddress(MemoryAddress address) {
>>      ...
>>
>>      return new FI$f() {
>>         MemorySegment apply(int i) { return
>> apply(SegmentAllocator.implicitAllocator(), i); }
>>         MemorySegment apply(SegmentAllocator, int i) { <call downcall
>> handle> }
>>      };
>> }
>> ```
>>
>> in other words, the returned functional interface instance supports two
>> invocation modes, w/ and w/o a SegmentAllocator. If no SegmentAllocator
>> is provided, we just use the implicit allocator (another option would be
>> to throw, but perhaps that would be too harsh?)
>>
>> Thoughts?
>>
>> Cheers
>> Maurizio
>>
>>
>>
>>
>>
>>
>> On 29/07/2022 00:42, Marko wrote:
>>> Perhaps the additional parameter was not as weird as I originally thought.
>>>> It seems a bit odd though IMHO, as then
>>>> HeaderFileBuilder#addVar and ToplevelBuilder#addVar are also forced to
>>>> have this parameter even though it goes unused there.
>>> I just tried it out and the getters are also missing when generated as
>>> a part of the header file.
>>> I'll polish my fix and send a PR for a better opinion.
>>>
>>> Thanks
>>> Marko
>>>
>>>
>>> On Thu, Jul 28, 2022 at 7:25 PM Marko <im at lostillusion.net> wrote:
>>>> My apologies, gmail seems to have only replied directly to Jorn
>>>> previously. Here is the original message.
>>>>
>>>> Hey Jorn,
>>>> Thanks for filing the report. I agree that the problem is where the
>>>> `ofAddress` function is generated. It does not check to see if a
>>>> segment allocator is needed. It seems quite trivial to add though as
>>>> it is already implemented for regular functions,
>>>> https://urldefense.com/v3/__https://github.com/openjdk/jextract/blob/0582eaf1b4cdba95f0ee8c2480767433bb647d0d/src/main/java/org/openjdk/jextract/impl/HeaderFileBuilder.java*L122-L127__;Iw!!ACWV5N9M2RV99hQ!Ju4E0gBlSpNpLkrFM97aohppFrNxuYCupnykO3jWsUarbLQ7KY33yrIGCfiv5-bwrlqaHWkcFleOyftmGi4snw$ .
>>>>
>>>> However, when this is fixed, the getter must also be updated to
>>>> reflect the addition of the SegmentAllocator parameter.
>>>> I believe this constitutes an additional parameter here
>>>> https://urldefense.com/v3/__https://github.com/openjdk/jextract/blob/0582eaf1b4cdba95f0ee8c2480767433bb647d0d/src/main/java/org/openjdk/jextract/impl/StructBuilder.java*L128=__;Iw!!ACWV5N9M2RV99hQ!Ju4E0gBlSpNpLkrFM97aohppFrNxuYCupnykO3jWsUarbLQ7KY33yrIGCfiv5-bwrlqaHWkcFleOyftu-dxHNA$
>>>> similar to fiName, that represents whether a segment allocator is
>>>> needed or not. It seems a bit odd though IMHO, as then
>>>> HeaderFileBuilder#addVar and ToplevelBuilder#addVar are also forced to
>>>> have this parameter even though it goes unused there.
>>>>
>>>> I implemented the fix and the proper getter generation as well, and it
>>>> seems to all work on my end. I wonder if this boolean parameter is the
>>>> way to go before submitting a PR though.
>>>>
>>>> Thanks,
>>>> Marko
>>>>
>>>> On Thu, Jul 28, 2022 at 5:20 PM Jorn Vernee <jorn.vernee at oracle.com> wrote:
>>>>> Hello Marko,
>>>>>
>>>>> Thanks for the report! I can reproduce it here as well. I think the
>>>>> problem is the lambda in the `ofAddress` factory that is generated in
>>>>> the interface for the callback type. Note that the getter will just end
>>>>> up calling that factory. (you can see that also in your stack trace: `at
>>>>> Foo$bar.lambda$ofAddress$0(Foo.java:32)`).
>>>>>
>>>>> I've filed: https://bugs.openjdk.org/browse/CODETOOLS-7903239
>>>>>
>>>>> Cheers,
>>>>> Jorn
>>>>>
>>>>> On 28/07/2022 02:00, Marko wrote:
>>>>>> Hello again!
>>>>>> When trying to call a function pointer inside a struct which returns a
>>>>>> MemorySegment, I receive the following error:
>>>>>> ```
>>>>>> Exception in thread "main" java.lang.AssertionError: should not reach here
>>>>>> at Foo$bar.lambda$ofAddress$0(Foo.java:34)
>>>>>> at Main.main(Main.java:8)
>>>>>> Caused by: java.lang.invoke.WrongMethodTypeException: expected
>>>>>> (NativeSymbol,SegmentAllocator)MemorySegment but found
>>>>>> (NativeSymbol)MemorySegment
>>>>>> at java.base/java.lang.invoke.Invokers.newWrongMethodTypeException(Invokers.java:523)
>>>>>> at java.base/java.lang.invoke.Invokers.checkExactType(Invokers.java:532)
>>>>>> at Foo$bar.lambda$ofAddress$0(Foo.java:32)
>>>>>> ... 1 more
>>>>>> ```
>>>>>> I believe this is due to
>>>>>> `FunctionalInterfaceBuilder#emitFunctionalFactoryForPointer` not
>>>>>> checking to see if the returnType is a MemorySegment, as it does in
>>>>>> `HeaderFileBuilder` for regular functions, see
>>>>>> https://urldefense.com/v3/__https://github.com/openjdk/jextract/blob/0582eaf1b4cdba95f0ee8c2480767433bb647d0d/src/main/java/org/openjdk/jextract/impl/HeaderFileBuilder.java*L122-L127__;Iw!!ACWV5N9M2RV99hQ!Ju4E0gBlSpNpLkrFM97aohppFrNxuYCupnykO3jWsUarbLQ7KY33yrIGCfiv5-bwrlqaHWkcFleOyftmGi4snw$ .
>>>>>>
>>>>>> I began implementing a fix by copying this functionality over, however
>>>>>> there is a bit of an issue I came across. When jextract emits the
>>>>>> getter for the functional interface in a struct (StructBuilder), only
>>>>>> the name of the interface is known. AFAIK, it does not know whether it
>>>>>> needs a SegmentAllocator and cannot currently generate the correct
>>>>>> code.
>>>>>>
>>>>>> I thought about adding another field to the method
>>>>>> `emitFunctionalFactoryForPointer`, similar to fiName, like a boolean
>>>>>> for whether a segment allocator is needed or not, however, I can't help but
>>>>>> feel it is a bit hacky. I wonder what your opinions are on this.
>>>>>>
>>>>>> A repro can be found here:
>>>>>> https://urldefense.com/v3/__https://github.com/lost-illusi0n/jextract-func-ptr-seg-alloc-repro__;!!ACWV5N9M2RV99hQ!Ju4E0gBlSpNpLkrFM97aohppFrNxuYCupnykO3jWsUarbLQ7KY33yrIGCfiv5-bwrlqaHWkcFleOyfuzpLEnvg$
>>>>>> Just run the run script if you are on Linux. You will have to change the
>>>>>> library output in the script if you are on another os.

From mcimadamore at openjdk.org  Mon Aug  1 21:17:08 2022
From: mcimadamore at openjdk.org (Maurizio Cimadamore)
Date: Mon, 1 Aug 2022 21:17:08 GMT
Subject: RFR: 7903239: ofAddress factory of function pointer type is wrong
 for struct returns
In-Reply-To: <VOQBzvWupTMIQsAccTVhA-t5RFqY36XSmpTRgqbYTnY=.ec8ee007-22ae-41bb-abbe-eeb291c899a0@github.com>
References: <VOQBzvWupTMIQsAccTVhA-t5RFqY36XSmpTRgqbYTnY=.ec8ee007-22ae-41bb-abbe-eeb291c899a0@github.com>
Message-ID: <6R2rAubAfCIVIdhDDG83frOeKVuv4BXTTfV-skwa9Vw=.cb1ab556-f665-4469-875a-ec04b3fe1c7d@github.com>

On Fri, 29 Jul 2022 04:47:22 GMT, Marko <duke at openjdk.org> wrote:

> When generating the lambda inside the `ofAddress` factory, it is missing the `SegmentAllocator` needed when the function returns a struct. This is [implemented for regular functions](https://github.com/openjdk/jextract/blob/0582eaf1b4cdba95f0ee8c2480767433bb647d0d/src/main/java/org/openjdk/jextract/impl/HeaderFileBuilder.java#L122-L127), but not for function pointers.

test/testng/org/openjdk/jextract/test/toolprovider/Test7903239.java line 34:

> 32: public class Test7903239 extends JextractToolRunner {
> 33:     @Test
> 34:     public void testFunctionPointerReturningStruct() {

While this test in itself is good, I feel it's perhaps the wrong kind of test, in the sense that it only checks that jextract completes normally. Maybe a test that uses the jextract jtreg extension would be better? For inspiration look here:

https://github.com/openjdk/jextract/blob/master/test/jtreg/generator/funcPointerInvokers/TestFuncPointerInvokers.java

You can basically run jextract as part of a jtreg test action, as here:

https://github.com/openjdk/jextract/blob/master/test/jtreg/generator/funcPointerInvokers/TestFuncPointerInvokers.java#L39

Then you can actually compile (against the jextracted bits) and even run:

https://github.com/openjdk/jextract/blob/master/test/jtreg/generator/funcPointerInvokers/TestFuncPointerInvokers.java#L40

So you can check that the generated code actually behaves as expected at runtime.

-------------

PR: https://git.openjdk.org/jextract/pull/58

From duke at openjdk.org  Tue Aug  2 18:01:38 2022
From: duke at openjdk.org (Marko)
Date: Tue, 2 Aug 2022 18:01:38 GMT
Subject: RFR: 7903239: ofAddress factory of function pointer type is wrong
 for struct returns [v2]
In-Reply-To: <VOQBzvWupTMIQsAccTVhA-t5RFqY36XSmpTRgqbYTnY=.ec8ee007-22ae-41bb-abbe-eeb291c899a0@github.com>
References: <VOQBzvWupTMIQsAccTVhA-t5RFqY36XSmpTRgqbYTnY=.ec8ee007-22ae-41bb-abbe-eeb291c899a0@github.com>
Message-ID: <u-9qNi-acwwA28cJNyKMFKce79MNpHH71uPoGNSfhqE=.0347dc0f-55fa-45d8-9f9f-a5f2f650f4dd@github.com>

> When generating the lambda inside the `ofAddress` factory, it is missing the `SegmentAllocator` needed when the function returns a struct. This is [implemented for regular functions](https://github.com/openjdk/jextract/blob/0582eaf1b4cdba95f0ee8c2480767433bb647d0d/src/main/java/org/openjdk/jextract/impl/HeaderFileBuilder.java#L122-L127), but not for function pointers.

Marko has updated the pull request incrementally with one additional commit since the last revision:

  add proper tests

-------------

Changes:
  - all: https://git.openjdk.org/jextract/pull/58/files
  - new: https://git.openjdk.org/jextract/pull/58/files/85c002ae..896aeabe

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jextract&pr=58&range=01
 - incr: https://webrevs.openjdk.org/?repo=jextract&pr=58&range=00-01

  Stats: 162 lines in 5 files changed: 95 ins; 65 del; 2 mod
  Patch: https://git.openjdk.org/jextract/pull/58.diff
  Fetch: git fetch https://git.openjdk.org/jextract pull/58/head:pull/58

PR: https://git.openjdk.org/jextract/pull/58

From im at lostillusion.net  Tue Aug  2 19:02:13 2022
From: im at lostillusion.net (Marko)
Date: Tue, 2 Aug 2022 15:02:13 -0400
Subject: BUG: function pointers inside a struct do not generate
 SegmentAllocators when needed
In-Reply-To: <cdcc8b2d-947b-1e2c-e88f-11e5280de150@oracle.com>
References: <CAHFdwNboGw===xxZ3_3czsi7sf=SHqBsWNEq0nSsj-rzMtUGkA@mail.gmail.com>
 <18e2cc45-c1fc-4ff7-4aa1-2fad60d40187@oracle.com>
 <CAHFdwNb0O6nwu_ajLHy9VfTrv4OdfvJqWFZ5LqTd7MB03WXAQQ@mail.gmail.com>
 <CAHFdwNaVTWJdqkf=bidiJN-iQeyyi8vyF0Xe__bOeP0Q9Ncx8g@mail.gmail.com>
 <f3de9911-5014-8bde-52b1-cc36428b9134@oracle.com>
 <CAHFdwNYCaULQR3hVcmtb+dxTt03USX5KYA8oPcCCHaFkPQ9H5Q@mail.gmail.com>
 <cdcc8b2d-947b-1e2c-e88f-11e5280de150@oracle.com>
Message-ID: <CAHFdwNYBzRDe4PS+9V7DEmHnC1iKi_s=mzaoizfu0yP24BNcSA@mail.gmail.com>

Hey Maurizio,
Thanks for taking a look at the PR and the guidance so far. It is
great news that it properly injects the "default" allocator as I
imagined it to. It would be nice to allow the use of a custom
allocator though, however my knowledge of the project was not quite
there yet to know how that would look like. I committed a new test and
I think the PR is ready to be looked at.
Thanks,
Marko.


On Mon, Aug 1, 2022 at 5:09 PM Maurizio Cimadamore
<maurizio.cimadamore at oracle.com> wrote:
>
> Hi Marko,
> I took a look at your PR:
>
> https://github.com/openjdk/jextract/pull/58/files
>
> What you do there works because, even though the downcall that
> `ofAddress` creates accepts an additional SegmentAllocator parameter,
> you provide such extra parameter in here:
>
> ```
>       if(fiType.returnType().equals(MemorySegment.class)) {
>                  append(", (SegmentAllocator)session");
>              }
> ```
>
> E.g. you take the session passed to `ofAddress` and turn that into a
> `SegmentAllocator` (which works, since MemorySession implements
> SegmentAllocator), and use that to allocate.
>
> By doing that, you effectively inject a "default" allocator into the
> downcall method handle, which makes sure that the returned MemorySegment
> belongs to the provided session.
>
> This is a very simple, yet effective fix (well done!), and I think one
> that could be good enough, at least for now.
>
> Where this fix type of fix leaves a bit to be desired is that it doesn't
> provide the degree of flexibility that a client might want. Say you have
> a more efficient SegmentAllocator, and you want to use _that_ to
> allocate the returned struct; that is not possible with the approach in
> your PR.
>
> That said, I'm not sure how big of a problem that is, given that in
> special situations clients can still wrap function pointers on their
> own, if they are not happy with what jextract provides. So I think your
> simple solution probably goes a long way (and we can always come back
> later and tweak this again).
>
> Thanks!
> Maurizio
>
> On 01/08/2022 20:08, Marko wrote:
> > Further email difficulties/default settings are odd, apologies!
> >
> > Hey Maurizio,
> > Thanks for the explanation, it helped a lot with understanding with
> > what's going on. I'm not quite sure how to cause a problem with the
> > two use cases you described, though.
> >
> > Scenario 1 would be equivalent to using the allocate method in the FI I believe.
> > Scenario 2 would be calling ofAddress on the memory segment/address we
> > got from 1.
> >
> > This is what I tried, and it seemed to work, though. The linker does
> > not seem to add a SegmentAllocator when creating the upcall, like you
> > said. However, wrapping the returned memory segment/address, from
> > scenario 1, inside the FI using ofAddress and calling it works...? I'm
> > assuming somewhere my logic is flawed, and the method handle should
> > not accept the additional SegmentAllocator when invoking. I just don't
> > quite understand why it doesn't fail. Anyway, I think your solution is
> > quite reasonable and that it should use the implicit allocator by
> > default as well, as that seems to be the behavior in other places of
> > jextract.
> >
> > Thanks
> > Marko
> >
> > On Fri, Jul 29, 2022 at 5:41 PM Maurizio Cimadamore
> > <maurizio.cimadamore at oracle.com> wrote:
> >> Hi Marko,
> >> I think adding the missing SegmentAllocator parameter can be possible
> >> fix, but the reality here is that there is an asymmetry between two use
> >> cases:
> >>
> >> 1. when you create an upcall from a lambda expression, the segment
> >> allocator parameter is not required (in fact the upcall machinery will
> >> _not_ provide one to the upcall, see below)
> >> 2. when you want to wrap a functional interface around an upcall stub
> >> address, the struct returned by the stub will be copied into a _fresh_
> >> segment allocated using the provided segment allocator
> >>
> >> Because of this, I think that just prepending a SegmentAllocator
> >> parameter in front of the functional descriptor is not gonna cut it.
> >> That is, if you have a C function pointer like:
> >>
> >> ```
> >> struct Foo (*f)(int)
> >> ```
> >>
> >> The Linker will expect an upcall MethodHandle with signature
> >> `(int)MemorySegment`, and no SegmentAllocator parameter. If you add an
> >> extra SegmentAllocator parameter, then the method handle we derive from
> >> the lambda expression in (1) is going to have the wrong type (unless we
> >> insert a dummy SegmentAllocator argument).
> >>
> >> When chatting with Jorn offline about this, we were discussing the
> >> possibility of emitting, for the above signature, a functional interface
> >> like the one below:
> >>
> >> ```
> >> interface FI$f {
> >>
> >>      MemorySegment apply(int i);
> >>      default MemorySegment(SegmentAllocator allocator, int i) { throw new
> >> UnsupportedOperationException(); }
> >>
> >>      ...
> >> }
> >> ```
> >>
> >> This has some advantages:
> >>
> >> * we can still provide a lambda-friendly factory for (1) - note that the
> >> functional interface signature would still be the "correct" one for the
> >> upcall (e.g. the one w/o the SegmentAllocator), so nothing changes here
> >> * we can also provide the `ofAddress` method for (2), which builds a new
> >> FI$f given a memory address; to do that, we create an instance as follows:
> >>
> >> ```
> >> FI$f ofAddress(MemoryAddress address) {
> >>      ...
> >>
> >>      return new FI$f() {
> >>         MemorySegment apply(int i) { return
> >> apply(SegmentAllocator.implicitAllocator(), i); }
> >>         MemorySegment apply(SegmentAllocator, int i) { <call downcall
> >> handle> }
> >>      };
> >> }
> >> ```
> >>
> >> in other words, the returned functional interface instance supports two
> >> invocation modes, w/ and w/o a SegmentAllocator. If no SegmentAllocator
> >> is provided, we just use the implicit allocator (another option would be
> >> to throw, but perhaps that would be too harsh?)
> >>
> >> Thoughts?
> >>
> >> Cheers
> >> Maurizio
> >>
> >>
> >>
> >>
> >>
> >>
> >> On 29/07/2022 00:42, Marko wrote:
> >>> Perhaps the additional parameter was not as weird as I originally thought.
> >>>> It seems a bit odd though IMHO, as then
> >>>> HeaderFileBuilder#addVar and ToplevelBuilder#addVar are also forced to
> >>>> have this parameter even though it goes unused there.
> >>> I just tried it out and the getters are also missing when generated as
> >>> a part of the header file.
> >>> I'll polish my fix and send a PR for a better opinion.
> >>>
> >>> Thanks
> >>> Marko
> >>>
> >>>
> >>> On Thu, Jul 28, 2022 at 7:25 PM Marko <im at lostillusion.net> wrote:
> >>>> My apologies, gmail seems to have only replied directly to Jorn
> >>>> previously. Here is the original message.
> >>>>
> >>>> Hey Jorn,
> >>>> Thanks for filing the report. I agree that the problem is where the
> >>>> `ofAddress` function is generated. It does not check to see if a
> >>>> segment allocator is needed. It seems quite trivial to add though as
> >>>> it is already implemented for regular functions,
> >>>> https://urldefense.com/v3/__https://github.com/openjdk/jextract/blob/0582eaf1b4cdba95f0ee8c2480767433bb647d0d/src/main/java/org/openjdk/jextract/impl/HeaderFileBuilder.java*L122-L127__;Iw!!ACWV5N9M2RV99hQ!Ju4E0gBlSpNpLkrFM97aohppFrNxuYCupnykO3jWsUarbLQ7KY33yrIGCfiv5-bwrlqaHWkcFleOyftmGi4snw$ .
> >>>>
> >>>> However, when this is fixed, the getter must also be updated to
> >>>> reflect the addition of the SegmentAllocator parameter.
> >>>> I believe this constitutes an additional parameter here
> >>>> https://urldefense.com/v3/__https://github.com/openjdk/jextract/blob/0582eaf1b4cdba95f0ee8c2480767433bb647d0d/src/main/java/org/openjdk/jextract/impl/StructBuilder.java*L128=__;Iw!!ACWV5N9M2RV99hQ!Ju4E0gBlSpNpLkrFM97aohppFrNxuYCupnykO3jWsUarbLQ7KY33yrIGCfiv5-bwrlqaHWkcFleOyftu-dxHNA$
> >>>> similar to fiName, that represents whether a segment allocator is
> >>>> needed or not. It seems a bit odd though IMHO, as then
> >>>> HeaderFileBuilder#addVar and ToplevelBuilder#addVar are also forced to
> >>>> have this parameter even though it goes unused there.
> >>>>
> >>>> I implemented the fix and the proper getter generation as well, and it
> >>>> seems to all work on my end. I wonder if this boolean parameter is the
> >>>> way to go before submitting a PR though.
> >>>>
> >>>> Thanks,
> >>>> Marko
> >>>>
> >>>> On Thu, Jul 28, 2022 at 5:20 PM Jorn Vernee <jorn.vernee at oracle.com> wrote:
> >>>>> Hello Marko,
> >>>>>
> >>>>> Thanks for the report! I can reproduce it here as well. I think the
> >>>>> problem is the lambda in the `ofAddress` factory that is generated in
> >>>>> the interface for the callback type. Note that the getter will just end
> >>>>> up calling that factory. (you can see that also in your stack trace: `at
> >>>>> Foo$bar.lambda$ofAddress$0(Foo.java:32)`).
> >>>>>
> >>>>> I've filed: https://bugs.openjdk.org/browse/CODETOOLS-7903239
> >>>>>
> >>>>> Cheers,
> >>>>> Jorn
> >>>>>
> >>>>> On 28/07/2022 02:00, Marko wrote:
> >>>>>> Hello again!
> >>>>>> When trying to call a function pointer inside a struct which returns a
> >>>>>> MemorySegment, I receive the following error:
> >>>>>> ```
> >>>>>> Exception in thread "main" java.lang.AssertionError: should not reach here
> >>>>>> at Foo$bar.lambda$ofAddress$0(Foo.java:34)
> >>>>>> at Main.main(Main.java:8)
> >>>>>> Caused by: java.lang.invoke.WrongMethodTypeException: expected
> >>>>>> (NativeSymbol,SegmentAllocator)MemorySegment but found
> >>>>>> (NativeSymbol)MemorySegment
> >>>>>> at java.base/java.lang.invoke.Invokers.newWrongMethodTypeException(Invokers.java:523)
> >>>>>> at java.base/java.lang.invoke.Invokers.checkExactType(Invokers.java:532)
> >>>>>> at Foo$bar.lambda$ofAddress$0(Foo.java:32)
> >>>>>> ... 1 more
> >>>>>> ```
> >>>>>> I believe this is due to
> >>>>>> `FunctionalInterfaceBuilder#emitFunctionalFactoryForPointer` not
> >>>>>> checking to see if the returnType is a MemorySegment, as it does in
> >>>>>> `HeaderFileBuilder` for regular functions, see
> >>>>>> https://urldefense.com/v3/__https://github.com/openjdk/jextract/blob/0582eaf1b4cdba95f0ee8c2480767433bb647d0d/src/main/java/org/openjdk/jextract/impl/HeaderFileBuilder.java*L122-L127__;Iw!!ACWV5N9M2RV99hQ!Ju4E0gBlSpNpLkrFM97aohppFrNxuYCupnykO3jWsUarbLQ7KY33yrIGCfiv5-bwrlqaHWkcFleOyftmGi4snw$ .
> >>>>>>
> >>>>>> I began implementing a fix by copying this functionality over, however
> >>>>>> there is a bit of an issue I came across. When jextract emits the
> >>>>>> getter for the functional interface in a struct (StructBuilder), only
> >>>>>> the name of the interface is known. AFAIK, it does not know whether it
> >>>>>> needs a SegmentAllocator and cannot currently generate the correct
> >>>>>> code.
> >>>>>>
> >>>>>> I thought about adding another field to the method
> >>>>>> `emitFunctionalFactoryForPointer`, similar to fiName, like a boolean
> >>>>>> for whether a segment allocator is needed or not, however, I can't help but
> >>>>>> feel it is a bit hacky. I wonder what your opinions are on this.
> >>>>>>
> >>>>>> A repro can be found here:
> >>>>>> https://urldefense.com/v3/__https://github.com/lost-illusi0n/jextract-func-ptr-seg-alloc-repro__;!!ACWV5N9M2RV99hQ!Ju4E0gBlSpNpLkrFM97aohppFrNxuYCupnykO3jWsUarbLQ7KY33yrIGCfiv5-bwrlqaHWkcFleOyfuzpLEnvg$
> >>>>>> Just run the run script if you are on Linux. You will have to change the
> >>>>>> library output in the script if you are on another os.

From mcimadamore at openjdk.org  Wed Aug  3 21:12:25 2022
From: mcimadamore at openjdk.org (Maurizio Cimadamore)
Date: Wed, 3 Aug 2022 21:12:25 GMT
Subject: RFR: 7903239: ofAddress factory of function pointer type is wrong
 for struct returns [v2]
In-Reply-To: <u-9qNi-acwwA28cJNyKMFKce79MNpHH71uPoGNSfhqE=.0347dc0f-55fa-45d8-9f9f-a5f2f650f4dd@github.com>
References: <VOQBzvWupTMIQsAccTVhA-t5RFqY36XSmpTRgqbYTnY=.ec8ee007-22ae-41bb-abbe-eeb291c899a0@github.com>
 <u-9qNi-acwwA28cJNyKMFKce79MNpHH71uPoGNSfhqE=.0347dc0f-55fa-45d8-9f9f-a5f2f650f4dd@github.com>
Message-ID: <AmwrSaxXzM586hN6F2SaiXazs-eB2dl9dcbTMBP8hB0=.75d54479-a170-4d8a-944e-9795e41ffc10@github.com>

On Tue, 2 Aug 2022 18:01:38 GMT, Marko <duke at openjdk.org> wrote:

>> When generating the lambda inside the `ofAddress` factory, it is missing the `SegmentAllocator` needed when the function returns a struct. This is [implemented for regular functions](https://github.com/openjdk/jextract/blob/0582eaf1b4cdba95f0ee8c2480767433bb647d0d/src/main/java/org/openjdk/jextract/impl/HeaderFileBuilder.java#L122-L127), but not for function pointers.
>
> Marko has updated the pull request incrementally with one additional commit since the last revision:
> 
>   add proper tests

test/jtreg/generator/test7903239/Test7903239.java line 51:

> 49:             var foo = test7903239_h.foo$SEGMENT();
> 50: 
> 51:             var barA = Foo.a(foo, session).apply();

I think the limitation of the strategy are clearer in this example :-)
The allocator is decided once and for all when the functional interface instance is obtained. Any other call to `apply` will use the same segment allocator specified on creation. To solve this, I believe the only way is to use the default method setup I described earlier. But the patch is good enough as is for now.

test/jtreg/generator/test7903239/libTest7903239.c line 2:

> 1: /*
> 2:  * Copyright (c) 2020, Oracle and/or its affiliates. All rights reserved.

If the test is new, should it have 2022 in the copyright year? (also in c file)

-------------

PR: https://git.openjdk.org/jextract/pull/58

From duke at openjdk.org  Wed Aug  3 22:16:27 2022
From: duke at openjdk.org (Marko)
Date: Wed, 3 Aug 2022 22:16:27 GMT
Subject: RFR: 7903239: ofAddress factory of function pointer type is wrong
 for struct returns [v3]
In-Reply-To: <VOQBzvWupTMIQsAccTVhA-t5RFqY36XSmpTRgqbYTnY=.ec8ee007-22ae-41bb-abbe-eeb291c899a0@github.com>
References: <VOQBzvWupTMIQsAccTVhA-t5RFqY36XSmpTRgqbYTnY=.ec8ee007-22ae-41bb-abbe-eeb291c899a0@github.com>
Message-ID: <QhhHJmkETgWqwnXGWsjmlIcDg374PxNGSkTZsDGOabI=.0d094c66-6959-4865-9852-e59d8a403480@github.com>

> When generating the lambda inside the `ofAddress` factory, it is missing the `SegmentAllocator` needed when the function returns a struct. This is [implemented for regular functions](https://github.com/openjdk/jextract/blob/0582eaf1b4cdba95f0ee8c2480767433bb647d0d/src/main/java/org/openjdk/jextract/impl/HeaderFileBuilder.java#L122-L127), but not for function pointers.

Marko has updated the pull request incrementally with one additional commit since the last revision:

  update copyright

-------------

Changes:
  - all: https://git.openjdk.org/jextract/pull/58/files
  - new: https://git.openjdk.org/jextract/pull/58/files/896aeabe..851d4b33

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jextract&pr=58&range=02
 - incr: https://webrevs.openjdk.org/?repo=jextract&pr=58&range=01-02

  Stats: 3 lines in 3 files changed: 0 ins; 0 del; 3 mod
  Patch: https://git.openjdk.org/jextract/pull/58.diff
  Fetch: git fetch https://git.openjdk.org/jextract pull/58/head:pull/58

PR: https://git.openjdk.org/jextract/pull/58

From duke at openjdk.org  Wed Aug  3 22:16:31 2022
From: duke at openjdk.org (Marko)
Date: Wed, 3 Aug 2022 22:16:31 GMT
Subject: RFR: 7903239: ofAddress factory of function pointer type is wrong
 for struct returns [v2]
In-Reply-To: <AmwrSaxXzM586hN6F2SaiXazs-eB2dl9dcbTMBP8hB0=.75d54479-a170-4d8a-944e-9795e41ffc10@github.com>
References: <VOQBzvWupTMIQsAccTVhA-t5RFqY36XSmpTRgqbYTnY=.ec8ee007-22ae-41bb-abbe-eeb291c899a0@github.com>
 <u-9qNi-acwwA28cJNyKMFKce79MNpHH71uPoGNSfhqE=.0347dc0f-55fa-45d8-9f9f-a5f2f650f4dd@github.com>
 <AmwrSaxXzM586hN6F2SaiXazs-eB2dl9dcbTMBP8hB0=.75d54479-a170-4d8a-944e-9795e41ffc10@github.com>
Message-ID: <bzXbA9Dh6sFVaLGGswO7IGMZjI-miDU-OusJGVwzHfU=.f61f268f-d8e4-4cb4-977f-f0255d17ec6d@github.com>

On Wed, 3 Aug 2022 21:08:33 GMT, Maurizio Cimadamore <mcimadamore at openjdk.org> wrote:

>> Marko has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   add proper tests
>
> test/jtreg/generator/test7903239/Test7903239.java line 51:
> 
>> 49:             var foo = test7903239_h.foo$SEGMENT();
>> 50: 
>> 51:             var barA = Foo.a(foo, session).apply();
> 
> I think the limitation of the strategy are clearer in this example :-)
> The allocator is decided once and for all when the functional interface instance is obtained. Any other call to `apply` will use the same segment allocator specified on creation. To solve this, I believe the only way is to use the default method setup I described earlier. But the patch is good enough as is for now.

Yes, the tradeoff is quite apparent, but I suppose reasonable to an extent. Your solution is probably the best road to go down when playing around with it though ?.

-------------

PR: https://git.openjdk.org/jextract/pull/58

From mcimadamore at openjdk.org  Thu Aug  4 10:23:32 2022
From: mcimadamore at openjdk.org (Maurizio Cimadamore)
Date: Thu, 4 Aug 2022 10:23:32 GMT
Subject: RFR: 7903239: ofAddress factory of function pointer type is wrong
 for struct returns [v3]
In-Reply-To: <QhhHJmkETgWqwnXGWsjmlIcDg374PxNGSkTZsDGOabI=.0d094c66-6959-4865-9852-e59d8a403480@github.com>
References: <VOQBzvWupTMIQsAccTVhA-t5RFqY36XSmpTRgqbYTnY=.ec8ee007-22ae-41bb-abbe-eeb291c899a0@github.com>
 <QhhHJmkETgWqwnXGWsjmlIcDg374PxNGSkTZsDGOabI=.0d094c66-6959-4865-9852-e59d8a403480@github.com>
Message-ID: <7608OM3JyEElki4wyfnzcq1OzQi1fC8Owgilkdc-PTE=.9278cbcb-b9ef-46b7-be4b-6c05354ec736@github.com>

On Wed, 3 Aug 2022 22:16:27 GMT, Marko <duke at openjdk.org> wrote:

>> When generating the lambda inside the `ofAddress` factory, it is missing the `SegmentAllocator` needed when the function returns a struct. This is [implemented for regular functions](https://github.com/openjdk/jextract/blob/0582eaf1b4cdba95f0ee8c2480767433bb647d0d/src/main/java/org/openjdk/jextract/impl/HeaderFileBuilder.java#L122-L127), but not for function pointers.
>
> Marko has updated the pull request incrementally with one additional commit since the last revision:
> 
>   update copyright

Marked as reviewed by mcimadamore (Committer).

test/jtreg/generator/test7903239/libTest7903239.c line 34:

> 32: };
> 33: 
> 34: struct Foo foo = { a, b };

Watch out: newline needed at the end

-------------

PR: https://git.openjdk.org/jextract/pull/58

From filip.krakowski at hhu.de  Thu Aug  4 13:47:02 2022
From: filip.krakowski at hhu.de (Filip Krakowski)
Date: Thu, 4 Aug 2022 15:47:02 +0200
Subject: Code changes required after JDK-8291473
Message-ID: <c8debf61-8251-16db-b6a5-6303475f9d35@hhu.de>

Hi,

since jextract does not work with the latest changes introduced in 
https://git.openjdk.org/panama-foreign/pull/694, I started to work on 
adjusting the sources in [1]. I took the following steps so far:

 ? 1. Find & replace all occurrences and imports of "Addressable" and 
"MemoryAddress" and replace it with "MemorySegment"
 ? 2. Change all occurences of "Linker.downcallType" and 
"Linker.upcallType" with "Linker.methodType"
 ? 3. Remove duplicate if-statements resulting from Step 1
 ? 4. Make C_POINTER$LAYOUT unbounded (in 
org.openjdk.jextract.impl.ToplevelBuilder#primitiveLayoutString)
 ? 5. Use MemorySegment#equals instead of reference comparison inside 
org.openjdk.jextract.clang.org.openjdk.jextract.clang#getLocation
 ? 6. Tweak build.gradle so that the user does not have to specify the 
jdk home directory. Gradle will automatically search for the specified 
JDK version. It can also be instructed to search inside custom locations 
using the org.gradle.java.installations.paths property, which can be set 
inside ${HOME}/.gradle/gradle.properties.

Since my approach is relatively naive ("find & replace" + fixing 
compiler and runtime errors), important parts of the code may be 
incorrect at the moment. I tested it with my Gradle plugin's demo 
project [2], which works. But again, this is only a simple program just 
calling printf and doing nothing fancy. In the next step I will try to 
rewrite some jtreg tests and see if they pass. Hopefully, this should 
point out all incorrect parts.

If you want to try it out, you need to compile [1] and add the resulting 
"bin"-Folder ("$PROJECT_DIR/build/jextract/bin") to your PATH. 
Afterwards [2] should just work by executing "./gradlew run". If 
successful, the program should print "Hello World" (using the native 
printf function).

I would greatly appreciate feedback regarding my general approach to 
code changes. In particular, I am unsure about changing the C_POINTER 
layout to be unbounded.

Best regards,
Filip

[1] - https://github.com/krakowski/jextract/tree/jdk20
[2] - https://github.com/krakowski/jextract-demo

From filip.krakowski at hhu.de  Thu Aug  4 15:39:58 2022
From: filip.krakowski at hhu.de (Filip Krakowski)
Date: Thu, 4 Aug 2022 17:39:58 +0200
Subject: Code changes required after JDK-8291473
In-Reply-To: <c8debf61-8251-16db-b6a5-6303475f9d35@hhu.de>
References: <c8debf61-8251-16db-b6a5-6303475f9d35@hhu.de>
Message-ID: <e58d4e1c-e414-5555-38f3-00cfb4965bc7@hhu.de>

Hi,

Small update:

I looked through all jtreg tests and made the same replacements as 
before [1]. On my system 91 out of 97 tests pass, although I made an 
error somewhere during replacing all "MemoryAddress" and "Addressable" 
occurrences, because the signature belonging to the generated method of 
native functions returning a pointer now expects a SegmentAllocator as 
the first parameter, where the underlying MethodHandle does not. Calling 
such methods results in an WrongMethodTypeException since the 
MethodHandle is not expecting a SegmentAllocator.

I will have to find where jextract decides whether to add a 
SegmentAllocator to the signature or not, as in case of pointers as the 
return type, it should not be there.

Best regards,
Filip

[1] - https://github.com/krakowski/jextract/tree/jdk20

On 04.08.22 15:47, Filip Krakowski wrote:
> Hi,
>
> since jextract does not work with the latest changes introduced in 
> https://git.openjdk.org/panama-foreign/pull/694, I started to work on 
> adjusting the sources in [1]. I took the following steps so far:
>
> ? 1. Find & replace all occurrences and imports of "Addressable" and 
> "MemoryAddress" and replace it with "MemorySegment"
> ? 2. Change all occurences of "Linker.downcallType" and 
> "Linker.upcallType" with "Linker.methodType"
> ? 3. Remove duplicate if-statements resulting from Step 1
> ? 4. Make C_POINTER$LAYOUT unbounded (in 
> org.openjdk.jextract.impl.ToplevelBuilder#primitiveLayoutString)
> ? 5. Use MemorySegment#equals instead of reference comparison inside 
> org.openjdk.jextract.clang.org.openjdk.jextract.clang#getLocation
> ? 6. Tweak build.gradle so that the user does not have to specify the 
> jdk home directory. Gradle will automatically search for the specified 
> JDK version. It can also be instructed to search inside custom 
> locations using the org.gradle.java.installations.paths property, 
> which can be set inside ${HOME}/.gradle/gradle.properties.
>
> Since my approach is relatively naive ("find & replace" + fixing 
> compiler and runtime errors), important parts of the code may be 
> incorrect at the moment. I tested it with my Gradle plugin's demo 
> project [2], which works. But again, this is only a simple program 
> just calling printf and doing nothing fancy. In the next step I will 
> try to rewrite some jtreg tests and see if they pass. Hopefully, this 
> should point out all incorrect parts.
>
> If you want to try it out, you need to compile [1] and add the 
> resulting "bin"-Folder ("$PROJECT_DIR/build/jextract/bin") to your 
> PATH. Afterwards [2] should just work by executing "./gradlew run". If 
> successful, the program should print "Hello World" (using the native 
> printf function).
>
> I would greatly appreciate feedback regarding my general approach to 
> code changes. In particular, I am unsure about changing the C_POINTER 
> layout to be unbounded.
>
> Best regards,
> Filip
>
> [1] - https://github.com/krakowski/jextract/tree/jdk20
> [2] - https://github.com/krakowski/jextract-demo


From maurizio.cimadamore at oracle.com  Thu Aug  4 17:06:40 2022
From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore)
Date: Thu, 4 Aug 2022 18:06:40 +0100
Subject: Code changes required after JDK-8291473
In-Reply-To: <c8debf61-8251-16db-b6a5-6303475f9d35@hhu.de>
References: <c8debf61-8251-16db-b6a5-6303475f9d35@hhu.de>
Message-ID: <0f31b101-53e7-a8fc-e202-ff662a876f09@oracle.com>

Hi Filip,
thanks for jumping into this. We have some internal version of jextract 
that works against the latest panama changes, but we need to find a 
place for it in the jextract repo. The main branch is not it, as we use 
that branch to generate the binaries for the current release (at this 
time Java 19).

The main idea we've been discussing (but would be nice to have feedback 
on it) would be to have two branches:

* main: this is where changes for JDK N live
* next: this is where changes for JDK N+1 live

When JDK N is GA'ed, we do two things:

* we "freeze" the contents of the "main" branch into a branch called jdkN-1
* we move "next" into "main" (so we'll start generating binary snapshot 
that refer to the latest GA'ed JDK release).

Right now we are a bit in an inconsistent state because (a) main points 
to JDK 19 bits even though JDK 19 is not GA'ed yet and (b) we have no 
place where to put the JDK 20 bits. But I think that, if we structure 
the repo as described above, we'll get in a saner place (once JDK 19 is 
GA'ed, that is :-) ).

Thoughts?
Maurizio



On 04/08/2022 14:47, Filip Krakowski wrote:
> Hi,
>
> since jextract does not work with the latest changes introduced in 
> https://git.openjdk.org/panama-foreign/pull/694, I started to work on 
> adjusting the sources in [1]. I took the following steps so far:
>
> ? 1. Find & replace all occurrences and imports of "Addressable" and 
> "MemoryAddress" and replace it with "MemorySegment"
> ? 2. Change all occurences of "Linker.downcallType" and 
> "Linker.upcallType" with "Linker.methodType"
> ? 3. Remove duplicate if-statements resulting from Step 1
> ? 4. Make C_POINTER$LAYOUT unbounded (in 
> org.openjdk.jextract.impl.ToplevelBuilder#primitiveLayoutString)
> ? 5. Use MemorySegment#equals instead of reference comparison inside 
> org.openjdk.jextract.clang.org.openjdk.jextract.clang#getLocation
> ? 6. Tweak build.gradle so that the user does not have to specify the 
> jdk home directory. Gradle will automatically search for the specified 
> JDK version. It can also be instructed to search inside custom 
> locations using the org.gradle.java.installations.paths property, 
> which can be set inside ${HOME}/.gradle/gradle.properties.
>
> Since my approach is relatively naive ("find & replace" + fixing 
> compiler and runtime errors), important parts of the code may be 
> incorrect at the moment. I tested it with my Gradle plugin's demo 
> project [2], which works. But again, this is only a simple program 
> just calling printf and doing nothing fancy. In the next step I will 
> try to rewrite some jtreg tests and see if they pass. Hopefully, this 
> should point out all incorrect parts.
>
> If you want to try it out, you need to compile [1] and add the 
> resulting "bin"-Folder ("$PROJECT_DIR/build/jextract/bin") to your 
> PATH. Afterwards [2] should just work by executing "./gradlew run". If 
> successful, the program should print "Hello World" (using the native 
> printf function).
>
> I would greatly appreciate feedback regarding my general approach to 
> code changes. In particular, I am unsure about changing the C_POINTER 
> layout to be unbounded.
>
> Best regards,
> Filip
>
> [1] - https://github.com/krakowski/jextract/tree/jdk20
> [2] - https://github.com/krakowski/jextract-demo

From filip.krakowski at hhu.de  Thu Aug  4 18:17:33 2022
From: filip.krakowski at hhu.de (Filip Krakowski)
Date: Thu, 4 Aug 2022 20:17:33 +0200
Subject: Code changes required after JDK-8291473
In-Reply-To: <0f31b101-53e7-a8fc-e202-ff662a876f09@oracle.com>
References: <c8debf61-8251-16db-b6a5-6303475f9d35@hhu.de>
 <0f31b101-53e7-a8fc-e202-ff662a876f09@oracle.com>
Message-ID: <9a02ed6b-3022-9bd9-ae8b-5d684ecb8c42@hhu.de>

Hi Maurizio,

> When JDK N is GA'ed, we do two things:
>
> * we "freeze" the contents of the "main" branch into a branch called 
> jdkN-1
> * we move "next" into "main" (so we'll start generating binary 
> snapshot that refer to the latest GA'ed JDK release).
if I understand it correctly, the "next" branch would be the branch for 
the next JDK waiting to be GA'ed and "main" would hold the currently 
GA'ed JDK (generating binary snapshots)? If yes, wouldn't the same 
problem reappear once development for JDK 21 starts? "next" would be 
pointing to JDK 20 (the next version to be GA'ed) and there would be no 
place for JDK 21.

Best regards,
Filip



From maurizio.cimadamore at oracle.com  Thu Aug  4 18:40:01 2022
From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore)
Date: Thu, 4 Aug 2022 19:40:01 +0100
Subject: Code changes required after JDK-8291473
In-Reply-To: <9a02ed6b-3022-9bd9-ae8b-5d684ecb8c42@hhu.de>
References: <c8debf61-8251-16db-b6a5-6303475f9d35@hhu.de>
 <0f31b101-53e7-a8fc-e202-ff662a876f09@oracle.com>
 <9a02ed6b-3022-9bd9-ae8b-5d684ecb8c42@hhu.de>
Message-ID: <751a627f-8fd4-6210-eb45-2fcdc1dd13bb@oracle.com>


On 04/08/2022 19:17, Filip Krakowski wrote:
> Hi Maurizio,
>
>> When JDK N is GA'ed, we do two things:
>>
>> * we "freeze" the contents of the "main" branch into a branch called 
>> jdkN-1
>> * we move "next" into "main" (so we'll start generating binary 
>> snapshot that refer to the latest GA'ed JDK release).
> if I understand it correctly, the "next" branch would be the branch 
> for the next JDK waiting to be GA'ed and "main" would hold the 
> currently GA'ed JDK (generating binary snapshots)? If yes, wouldn't 
> the same problem reappear once development for JDK 21 starts? "next" 
> would be pointing to JDK 20 (the next version to be GA'ed) and there 
> would be no place for JDK 21.

Let's say the last GA'ed release is 26.

Then, the "main" branch will point at jextract for 26.

There will also be branches for 25, 24, 23, ....

And the "next" branch will at an experimental version of jextract which 
works with 27.

What you say is true, though - there is a lag between when JDK N + 1 is 
created and worked upon and when JDK N is GA'ed. In this temporal 
interval (which is where we are now, and lasts for about 3 months).

Another option, which is perhaps simpler, is just to have "main" and 
"jdkN" branches.

When JDK N is created, we also create corresponding branch on jextract.

The "main" branch is only updated when a new JDK N is GA'ed (by copying 
contents from the already existing JDK N branch).

Maurizio

>
> Best regards,
> Filip
>
>

From filip.krakowski at hhu.de  Thu Aug  4 19:08:22 2022
From: filip.krakowski at hhu.de (Filip Krakowski)
Date: Thu, 4 Aug 2022 21:08:22 +0200
Subject: Code changes required after JDK-8291473
In-Reply-To: <751a627f-8fd4-6210-eb45-2fcdc1dd13bb@oracle.com>
References: <c8debf61-8251-16db-b6a5-6303475f9d35@hhu.de>
 <0f31b101-53e7-a8fc-e202-ff662a876f09@oracle.com>
 <9a02ed6b-3022-9bd9-ae8b-5d684ecb8c42@hhu.de>
 <751a627f-8fd4-6210-eb45-2fcdc1dd13bb@oracle.com>
Message-ID: <68433e50-e717-db0c-eaa3-7c0ab4fca796@hhu.de>

Hi Maurizio,

I like the idea of having just "main" and "jdkN" as branches. Another 
idea I could think of would be to drop "main" as well and just use 
"jdkN" branches (making the current GA'ed JDK or the latest JDK the 
default branch at GitHub) for each available JDK and create binaries 
based on tags. For example, once "jdk19" gets GA'ed it would receive the 
tag "19.0.0" and if a patch has to be introduced in "jdk18" and "jdk19", 
both branches could be updated and tagged with "18.0.1" and "19.0.1" 
respectively. This in turn would trigger a GitHub Action (or some 
external pipeline), which would create both binaries "jextract-18.0.1" 
and "jextract-19.0.1". Each branch would therefore only produce binaries 
when it receives a new tag. Could this work?

Best regards,
Filip

On 04.08.22 20:40, Maurizio Cimadamore wrote:
>
> On 04/08/2022 19:17, Filip Krakowski wrote:
>> Hi Maurizio,
>>
>>> When JDK N is GA'ed, we do two things:
>>>
>>> * we "freeze" the contents of the "main" branch into a branch called 
>>> jdkN-1
>>> * we move "next" into "main" (so we'll start generating binary 
>>> snapshot that refer to the latest GA'ed JDK release).
>> if I understand it correctly, the "next" branch would be the branch 
>> for the next JDK waiting to be GA'ed and "main" would hold the 
>> currently GA'ed JDK (generating binary snapshots)? If yes, wouldn't 
>> the same problem reappear once development for JDK 21 starts? "next" 
>> would be pointing to JDK 20 (the next version to be GA'ed) and there 
>> would be no place for JDK 21.
>
> Let's say the last GA'ed release is 26.
>
> Then, the "main" branch will point at jextract for 26.
>
> There will also be branches for 25, 24, 23, ....
>
> And the "next" branch will at an experimental version of jextract 
> which works with 27.
>
> What you say is true, though - there is a lag between when JDK N + 1 
> is created and worked upon and when JDK N is GA'ed. In this temporal 
> interval (which is where we are now, and lasts for about 3 months).
>
> Another option, which is perhaps simpler, is just to have "main" and 
> "jdkN" branches.
>
> When JDK N is created, we also create corresponding branch on jextract.
>
> The "main" branch is only updated when a new JDK N is GA'ed (by 
> copying contents from the already existing JDK N branch).
>
> Maurizio
>
>>
>> Best regards,
>> Filip
>>
>>


From maurizio.cimadamore at oracle.com  Thu Aug  4 19:14:28 2022
From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore)
Date: Thu, 4 Aug 2022 20:14:28 +0100
Subject: Code changes required after JDK-8291473
In-Reply-To: <68433e50-e717-db0c-eaa3-7c0ab4fca796@hhu.de>
References: <c8debf61-8251-16db-b6a5-6303475f9d35@hhu.de>
 <0f31b101-53e7-a8fc-e202-ff662a876f09@oracle.com>
 <9a02ed6b-3022-9bd9-ae8b-5d684ecb8c42@hhu.de>
 <751a627f-8fd4-6210-eb45-2fcdc1dd13bb@oracle.com>
 <68433e50-e717-db0c-eaa3-7c0ab4fca796@hhu.de>
Message-ID: <e40f2dd7-3471-9cb4-48c7-b6500416b7b0@oracle.com>


On 04/08/2022 20:08, Filip Krakowski wrote:
> Each branch would therefore only produce binaries when it receives a 
> new tag. Could this work? 

I'd have to check - my feeling is that, no that would not be possible.

Another issue is that we'd need to keep updating the main github branch 
every few months (not a big issue).

Maurizio



From maurizio.cimadamore at oracle.com  Thu Aug  4 21:23:46 2022
From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore)
Date: Thu, 4 Aug 2022 22:23:46 +0100
Subject: Code changes required after JDK-8291473
In-Reply-To: <e58d4e1c-e414-5555-38f3-00cfb4965bc7@hhu.de>
References: <c8debf61-8251-16db-b6a5-6303475f9d35@hhu.de>
 <e58d4e1c-e414-5555-38f3-00cfb4965bc7@hhu.de>
Message-ID: <c0256484-7104-2e1f-0b91-92ac6e4709de@oracle.com>

Hi Filip,
coming back to your original questions:

> I am unsure about changing the C_POINTER layout to be unbounded. 
I think this is how we intended the API to be used. Currently, with JDK 
19 jextract, you get back a MemoryAddress and you can dereference that. 
So it seems logical to preserve same behavior (using the unbounded 
address layout).


>
> I will have to find where jextract decides whether to add a 
> SegmentAllocator to the signature or not, as in case of pointers as 
> the return type, it should not be there.

Yes, I recall having to fight with same issue - in a much of places we 
do some "instanceof MemorySegment" and then assume that, if true, we 
need an allocator. All those places need to be updated to test on the 
return _layout_. Example:

https://github.com/openjdk/jextract/blob/master/src/main/java/org/openjdk/jextract/impl/HeaderFileBuilder.java#L122

Cheers
Maurizio


>
> Best regards,
> Filip
>
> [1] - https://github.com/krakowski/jextract/tree/jdk20
>
> On 04.08.22 15:47, Filip Krakowski wrote:
>> Hi,
>>
>> since jextract does not work with the latest changes introduced in 
>> https://git.openjdk.org/panama-foreign/pull/694, I started to work on 
>> adjusting the sources in [1]. I took the following steps so far:
>>
>> ? 1. Find & replace all occurrences and imports of "Addressable" and 
>> "MemoryAddress" and replace it with "MemorySegment"
>> ? 2. Change all occurences of "Linker.downcallType" and 
>> "Linker.upcallType" with "Linker.methodType"
>> ? 3. Remove duplicate if-statements resulting from Step 1
>> ? 4. Make C_POINTER$LAYOUT unbounded (in 
>> org.openjdk.jextract.impl.ToplevelBuilder#primitiveLayoutString)
>> ? 5. Use MemorySegment#equals instead of reference comparison inside 
>> org.openjdk.jextract.clang.org.openjdk.jextract.clang#getLocation
>> ? 6. Tweak build.gradle so that the user does not have to specify the 
>> jdk home directory. Gradle will automatically search for the 
>> specified JDK version. It can also be instructed to search inside 
>> custom locations using the org.gradle.java.installations.paths 
>> property, which can be set inside ${HOME}/.gradle/gradle.properties.
>>
>> Since my approach is relatively naive ("find & replace" + fixing 
>> compiler and runtime errors), important parts of the code may be 
>> incorrect at the moment. I tested it with my Gradle plugin's demo 
>> project [2], which works. But again, this is only a simple program 
>> just calling printf and doing nothing fancy. In the next step I will 
>> try to rewrite some jtreg tests and see if they pass. Hopefully, this 
>> should point out all incorrect parts.
>>
>> If you want to try it out, you need to compile [1] and add the 
>> resulting "bin"-Folder ("$PROJECT_DIR/build/jextract/bin") to your 
>> PATH. Afterwards [2] should just work by executing "./gradlew run". 
>> If successful, the program should print "Hello World" (using the 
>> native printf function).
>>
>> I would greatly appreciate feedback regarding my general approach to 
>> code changes. In particular, I am unsure about changing the C_POINTER 
>> layout to be unbounded.
>>
>> Best regards,
>> Filip
>>
>> [1] - https://github.com/krakowski/jextract/tree/jdk20
>> [2] - https://github.com/krakowski/jextract-demo
>

From duke at openjdk.org  Fri Aug  5 04:32:04 2022
From: duke at openjdk.org (Marko)
Date: Fri, 5 Aug 2022 04:32:04 GMT
Subject: RFR: 7903239: ofAddress factory of function pointer type is wrong
 for struct returns [v4]
In-Reply-To: <VOQBzvWupTMIQsAccTVhA-t5RFqY36XSmpTRgqbYTnY=.ec8ee007-22ae-41bb-abbe-eeb291c899a0@github.com>
References: <VOQBzvWupTMIQsAccTVhA-t5RFqY36XSmpTRgqbYTnY=.ec8ee007-22ae-41bb-abbe-eeb291c899a0@github.com>
Message-ID: <cTlhwM18VT-Zv9UzdC3_EmI6J_JoQ7smDsQZvW5t9Vo=.ce71ceda-21f7-45b4-a3eb-9b124411c1f0@github.com>

> When generating the lambda inside the `ofAddress` factory, it is missing the `SegmentAllocator` needed when the function returns a struct. This is [implemented for regular functions](https://github.com/openjdk/jextract/blob/0582eaf1b4cdba95f0ee8c2480767433bb647d0d/src/main/java/org/openjdk/jextract/impl/HeaderFileBuilder.java#L122-L127), but not for function pointers.

Marko has updated the pull request incrementally with one additional commit since the last revision:

  newline

-------------

Changes:
  - all: https://git.openjdk.org/jextract/pull/58/files
  - new: https://git.openjdk.org/jextract/pull/58/files/851d4b33..c23b05b5

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jextract&pr=58&range=03
 - incr: https://webrevs.openjdk.org/?repo=jextract&pr=58&range=02-03

  Stats: 2 lines in 1 file changed: 1 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/jextract/pull/58.diff
  Fetch: git fetch https://git.openjdk.org/jextract pull/58/head:pull/58

PR: https://git.openjdk.org/jextract/pull/58

From filip.krakowski at hhu.de  Fri Aug  5 10:23:38 2022
From: filip.krakowski at hhu.de (Filip Krakowski)
Date: Fri, 5 Aug 2022 12:23:38 +0200
Subject: Code changes required after JDK-8291473
In-Reply-To: <e40f2dd7-3471-9cb4-48c7-b6500416b7b0@oracle.com>
References: <c8debf61-8251-16db-b6a5-6303475f9d35@hhu.de>
 <0f31b101-53e7-a8fc-e202-ff662a876f09@oracle.com>
 <9a02ed6b-3022-9bd9-ae8b-5d684ecb8c42@hhu.de>
 <751a627f-8fd4-6210-eb45-2fcdc1dd13bb@oracle.com>
 <68433e50-e717-db0c-eaa3-7c0ab4fca796@hhu.de>
 <e40f2dd7-3471-9cb4-48c7-b6500416b7b0@oracle.com>
Message-ID: <35555875-787c-b587-4723-0b67f541e3ba@hhu.de>

Hi,

is it a question of maintainability (diverging branches in case of 
patches, etc.), which would stand in the way of this, or a question of 
the infrastructure that would be required? In the latter case, I could 
try to set something up to accomplish the job of automatically creating 
releases from each pushed tag matching a version number.

Since the project is hosted on GitHub, I could implement a GitHub 
Action, which would be triggered on each pushed tag. This Action would 
then compile the code for different platforms (Linux, macOS, Windows - 
GitHub provides images for each), create a release inside the repository 
and attach the created binaries to it. The repository already contains a 
GitHub Action for building and testing jextract, which I could use as a 
starting point.

Best regards,
Filip

On 04.08.22 21:14, Maurizio Cimadamore wrote:
>
> On 04/08/2022 20:08, Filip Krakowski wrote:
>> Each branch would therefore only produce binaries when it receives a 
>> new tag. Could this work? 
>
> I'd have to check - my feeling is that, no that would not be possible.
>
> Another issue is that we'd need to keep updating the main github 
> branch every few months (not a big issue).
>
> Maurizio
>
>


From maurizio.cimadamore at oracle.com  Fri Aug  5 11:31:49 2022
From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore)
Date: Fri, 5 Aug 2022 12:31:49 +0100
Subject: Code changes required after JDK-8291473
In-Reply-To: <35555875-787c-b587-4723-0b67f541e3ba@hhu.de>
References: <c8debf61-8251-16db-b6a5-6303475f9d35@hhu.de>
 <0f31b101-53e7-a8fc-e202-ff662a876f09@oracle.com>
 <9a02ed6b-3022-9bd9-ae8b-5d684ecb8c42@hhu.de>
 <751a627f-8fd4-6210-eb45-2fcdc1dd13bb@oracle.com>
 <68433e50-e717-db0c-eaa3-7c0ab4fca796@hhu.de>
 <e40f2dd7-3471-9cb4-48c7-b6500416b7b0@oracle.com>
 <35555875-787c-b587-4723-0b67f541e3ba@hhu.de>
Message-ID: <8494784f-f77f-fe73-a092-4e58888b9a76@oracle.com>

Hi Filip,
due to various complex reasons, it is currently not possible for us to 
release binary snapshots using GitHub actions. All the binaries we 
publish come from an internal CI pipeline, which builds and test using 
predefined, well-known configurations, so that we can ensure repeatable 
results. While this is not set in stone, and OpenJDK is relatively new 
to GitHub workflows, I think that, for the time being, we have to work 
with what we've got.

That said, nothing prevents us from creating several internal pipelines, 
one per jextract version, and add a new one every 6 months or so, 
perhaps with some policy to "garbage collect" old pipelines (e.g. maybe 
providing binaries up to JDK N-3 would be a good starting point?) - 
although I'd have to check with the infra team and see what solution 
they'd be most comfortable with.

P.S. While it would be possible, in principle, for 3rd parties to build 
snapshots of jextract, I find that solution less ideal. Plugins such as 
yours need a stable link to work with. A 3rd party build might be a 
great place to get started (or where to get more frequent jextract 
binary updates, for example), but doesn't provide (by design) that kind 
of long-term commitment that some of the tools in the ecosystem would 
require, at least IMHO.

Thanks
Maurizio

On 05/08/2022 11:23, Filip Krakowski wrote:
> Hi,
>
> is it a question of maintainability (diverging branches in case of 
> patches, etc.), which would stand in the way of this, or a question of 
> the infrastructure that would be required? In the latter case, I could 
> try to set something up to accomplish the job of automatically 
> creating releases from each pushed tag matching a version number.
>
> Since the project is hosted on GitHub, I could implement a GitHub 
> Action, which would be triggered on each pushed tag. This Action would 
> then compile the code for different platforms (Linux, macOS, Windows - 
> GitHub provides images for each), create a release inside the 
> repository and attach the created binaries to it. The repository 
> already contains a GitHub Action for building and testing jextract, 
> which I could use as a starting point.
>
> Best regards,
> Filip
>
> On 04.08.22 21:14, Maurizio Cimadamore wrote:
>>
>> On 04/08/2022 20:08, Filip Krakowski wrote:
>>> Each branch would therefore only produce binaries when it receives a 
>>> new tag. Could this work? 
>>
>> I'd have to check - my feeling is that, no that would not be possible.
>>
>> Another issue is that we'd need to keep updating the main github 
>> branch every few months (not a big issue).
>>
>> Maurizio
>>
>>
>

From maurizio.cimadamore at oracle.com  Mon Aug  8 17:44:31 2022
From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore)
Date: Mon, 8 Aug 2022 18:44:31 +0100
Subject: Code changes required after JDK-8291473
In-Reply-To: <8494784f-f77f-fe73-a092-4e58888b9a76@oracle.com>
References: <c8debf61-8251-16db-b6a5-6303475f9d35@hhu.de>
 <0f31b101-53e7-a8fc-e202-ff662a876f09@oracle.com>
 <9a02ed6b-3022-9bd9-ae8b-5d684ecb8c42@hhu.de>
 <751a627f-8fd4-6210-eb45-2fcdc1dd13bb@oracle.com>
 <68433e50-e717-db0c-eaa3-7c0ab4fca796@hhu.de>
 <e40f2dd7-3471-9cb4-48c7-b6500416b7b0@oracle.com>
 <35555875-787c-b587-4723-0b67f541e3ba@hhu.de>
 <8494784f-f77f-fe73-a092-4e58888b9a76@oracle.com>
Message-ID: <5115a250-0fc9-9add-186b-e8df4a6f650d@oracle.com>

After some internal discussion, we are going to do the following:

* create a "panama" branch, which contains the changes needed to run 
jextract with the API contained in the Panama repo
* have (as now) a set of versioned branches, for jdk18 and jdk19
* when 20 becomes GA, we will create a new jdk20 branch, copy the 
contents of the "panama" branch into it and make it the default branch
* we will _not_ provide binary snapshots for the "panama" branch, as 
that's a "bleeding edge" sort of scenario (where you have to build the 
panama repo manually to go with it as well)

Cheers
Maurizio


On 05/08/2022 12:31, Maurizio Cimadamore wrote:
> Hi Filip,
> due to various complex reasons, it is currently not possible for us to 
> release binary snapshots using GitHub actions. All the binaries we 
> publish come from an internal CI pipeline, which builds and test using 
> predefined, well-known configurations, so that we can ensure 
> repeatable results. While this is not set in stone, and OpenJDK is 
> relatively new to GitHub workflows, I think that, for the time being, 
> we have to work with what we've got.
>
> That said, nothing prevents us from creating several internal 
> pipelines, one per jextract version, and add a new one every 6 months 
> or so, perhaps with some policy to "garbage collect" old pipelines 
> (e.g. maybe providing binaries up to JDK N-3 would be a good starting 
> point?) - although I'd have to check with the infra team and see what 
> solution they'd be most comfortable with.
>
> P.S. While it would be possible, in principle, for 3rd parties to 
> build snapshots of jextract, I find that solution less ideal. Plugins 
> such as yours need a stable link to work with. A 3rd party build might 
> be a great place to get started (or where to get more frequent 
> jextract binary updates, for example), but doesn't provide (by design) 
> that kind of long-term commitment that some of the tools in the 
> ecosystem would require, at least IMHO.
>
> Thanks
> Maurizio
>
> On 05/08/2022 11:23, Filip Krakowski wrote:
>> Hi,
>>
>> is it a question of maintainability (diverging branches in case of 
>> patches, etc.), which would stand in the way of this, or a question 
>> of the infrastructure that would be required? In the latter case, I 
>> could try to set something up to accomplish the job of automatically 
>> creating releases from each pushed tag matching a version number.
>>
>> Since the project is hosted on GitHub, I could implement a GitHub 
>> Action, which would be triggered on each pushed tag. This Action 
>> would then compile the code for different platforms (Linux, macOS, 
>> Windows - GitHub provides images for each), create a release inside 
>> the repository and attach the created binaries to it. The repository 
>> already contains a GitHub Action for building and testing jextract, 
>> which I could use as a starting point.
>>
>> Best regards,
>> Filip
>>
>> On 04.08.22 21:14, Maurizio Cimadamore wrote:
>>>
>>> On 04/08/2022 20:08, Filip Krakowski wrote:
>>>> Each branch would therefore only produce binaries when it receives 
>>>> a new tag. Could this work? 
>>>
>>> I'd have to check - my feeling is that, no that would not be possible.
>>>
>>> Another issue is that we'd need to keep updating the main github 
>>> branch every few months (not a big issue).
>>>
>>> Maurizio
>>>
>>>
>>

From filip.krakowski at hhu.de  Tue Aug  9 08:29:04 2022
From: filip.krakowski at hhu.de (Filip Krakowski)
Date: Tue, 9 Aug 2022 10:29:04 +0200
Subject: Code changes required after JDK-8291473
In-Reply-To: <5115a250-0fc9-9add-186b-e8df4a6f650d@oracle.com>
References: <c8debf61-8251-16db-b6a5-6303475f9d35@hhu.de>
 <0f31b101-53e7-a8fc-e202-ff662a876f09@oracle.com>
 <9a02ed6b-3022-9bd9-ae8b-5d684ecb8c42@hhu.de>
 <751a627f-8fd4-6210-eb45-2fcdc1dd13bb@oracle.com>
 <68433e50-e717-db0c-eaa3-7c0ab4fca796@hhu.de>
 <e40f2dd7-3471-9cb4-48c7-b6500416b7b0@oracle.com>
 <35555875-787c-b587-4723-0b67f541e3ba@hhu.de>
 <8494784f-f77f-fe73-a092-4e58888b9a76@oracle.com>
 <5115a250-0fc9-9add-186b-e8df4a6f650d@oracle.com>
Message-ID: <3fde1696-b9f2-09a9-17e0-9d42667b47f8@hhu.de>

Hi,

this sounds great! Once the changes get pushed I will include jextract's 
"panama" branch into our CI pipeline, so that we always have a version 
that is in sync with the latest "foreign-memaccess+abi" branch.

Thanks for the quick response! Looking forward to trying out it out :)

Best regards,
Filip

On 08.08.22 19:44, Maurizio Cimadamore wrote:
> After some internal discussion, we are going to do the following:
>
> * create a "panama" branch, which contains the changes needed to run 
> jextract with the API contained in the Panama repo
> * have (as now) a set of versioned branches, for jdk18 and jdk19
> * when 20 becomes GA, we will create a new jdk20 branch, copy the 
> contents of the "panama" branch into it and make it the default branch
> * we will _not_ provide binary snapshots for the "panama" branch, as 
> that's a "bleeding edge" sort of scenario (where you have to build the 
> panama repo manually to go with it as well)
>
> Cheers
> Maurizio
>
>
> On 05/08/2022 12:31, Maurizio Cimadamore wrote:
>> Hi Filip,
>> due to various complex reasons, it is currently not possible for us 
>> to release binary snapshots using GitHub actions. All the binaries we 
>> publish come from an internal CI pipeline, which builds and test 
>> using predefined, well-known configurations, so that we can ensure 
>> repeatable results. While this is not set in stone, and OpenJDK is 
>> relatively new to GitHub workflows, I think that, for the time being, 
>> we have to work with what we've got.
>>
>> That said, nothing prevents us from creating several internal 
>> pipelines, one per jextract version, and add a new one every 6 months 
>> or so, perhaps with some policy to "garbage collect" old pipelines 
>> (e.g. maybe providing binaries up to JDK N-3 would be a good starting 
>> point?) - although I'd have to check with the infra team and see what 
>> solution they'd be most comfortable with.
>>
>> P.S. While it would be possible, in principle, for 3rd parties to 
>> build snapshots of jextract, I find that solution less ideal. Plugins 
>> such as yours need a stable link to work with. A 3rd party build 
>> might be a great place to get started (or where to get more frequent 
>> jextract binary updates, for example), but doesn't provide (by 
>> design) that kind of long-term commitment that some of the tools in 
>> the ecosystem would require, at least IMHO.
>>
>> Thanks
>> Maurizio
>>
>> On 05/08/2022 11:23, Filip Krakowski wrote:
>>> Hi,
>>>
>>> is it a question of maintainability (diverging branches in case of 
>>> patches, etc.), which would stand in the way of this, or a question 
>>> of the infrastructure that would be required? In the latter case, I 
>>> could try to set something up to accomplish the job of automatically 
>>> creating releases from each pushed tag matching a version number.
>>>
>>> Since the project is hosted on GitHub, I could implement a GitHub 
>>> Action, which would be triggered on each pushed tag. This Action 
>>> would then compile the code for different platforms (Linux, macOS, 
>>> Windows - GitHub provides images for each), create a release inside 
>>> the repository and attach the created binaries to it. The repository 
>>> already contains a GitHub Action for building and testing jextract, 
>>> which I could use as a starting point.
>>>
>>> Best regards,
>>> Filip
>>>
>>> On 04.08.22 21:14, Maurizio Cimadamore wrote:
>>>>
>>>> On 04/08/2022 20:08, Filip Krakowski wrote:
>>>>> Each branch would therefore only produce binaries when it receives 
>>>>> a new tag. Could this work? 
>>>>
>>>> I'd have to check - my feeling is that, no that would not be possible.
>>>>
>>>> Another issue is that we'd need to keep updating the main github 
>>>> branch every few months (not a big issue).
>>>>
>>>> Maurizio
>>>>
>>>>
>>>


From sundar at openjdk.org  Thu Aug 11 10:01:39 2022
From: sundar at openjdk.org (Athijegannathan Sundararajan)
Date: Thu, 11 Aug 2022 10:01:39 GMT
Subject: RFR: making jextract to work with foreign-memaccess+abi branch of
 panama-foreign
Message-ID: <VCTGRh4pTTfzdJKcsXHiPImokwUbzWsiCCedF5sxjU4=.836800c3-aa3d-4e2e-a374-7bcdb40ae4e7@github.com>

making jextract to work with foreign-memaccess+abi branch of panama-foreign

-------------

Commit messages:
 - fixed libffmpeg for library version and other merge issues
 - cherry-pick brought more changes than needed. fixed.
 - sync jextract with panama-foreign mcimadamore's "ffm_next" branch
 - changes for MemorySession.addCloseAction, null Runnable argument removal in MemorySegment.ofAddress calls,
 - using NULL instead of 0 address in tensorflow
 - Fixed samples for API changes and scripts for enable-preview source version.
 - Fix allocator issues
 - Tweak method names
 - Fix samples
 - Drop struct unsafe builder
 - ... and 3 more: https://git.openjdk.org/jextract/compare/0582eaf1...32b7cec3

Changes: https://git.openjdk.org/jextract/pull/59/files
 Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=59&range=00
  Stats: 750 lines in 114 files changed: 34 ins; 156 del; 560 mod
  Patch: https://git.openjdk.org/jextract/pull/59.diff
  Fetch: git fetch https://git.openjdk.org/jextract pull/59/head:pull/59

PR: https://git.openjdk.org/jextract/pull/59

From sundar at openjdk.org  Thu Aug 11 10:05:47 2022
From: sundar at openjdk.org (Athijegannathan Sundararajan)
Date: Thu, 11 Aug 2022 10:05:47 GMT
Subject: RFR: making jextract to work with foreign-memaccess+abi branch of
 panama-foreign [v2]
In-Reply-To: <VCTGRh4pTTfzdJKcsXHiPImokwUbzWsiCCedF5sxjU4=.836800c3-aa3d-4e2e-a374-7bcdb40ae4e7@github.com>
References: <VCTGRh4pTTfzdJKcsXHiPImokwUbzWsiCCedF5sxjU4=.836800c3-aa3d-4e2e-a374-7bcdb40ae4e7@github.com>
Message-ID: <P7IgSSx5izwSZeu2laxJDjzGTyQhm6D5xvCvoxeYTXU=.2efe5446-f86f-43b9-a29d-fde4bc84747a@github.com>

> making jextract to work with foreign-memaccess+abi branch of panama-foreign

Athijegannathan Sundararajan has updated the pull request incrementally with one additional commit since the last revision:

  fixed README.md to mention about jdk20_home var name and panama-foreign jdk build requirement.

-------------

Changes:
  - all: https://git.openjdk.org/jextract/pull/59/files
  - new: https://git.openjdk.org/jextract/pull/59/files/32b7cec3..69e8f079

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jextract&pr=59&range=01
 - incr: https://webrevs.openjdk.org/?repo=jextract&pr=59&range=00-01

  Stats: 3 lines in 1 file changed: 0 ins; 0 del; 3 mod
  Patch: https://git.openjdk.org/jextract/pull/59.diff
  Fetch: git fetch https://git.openjdk.org/jextract pull/59/head:pull/59

PR: https://git.openjdk.org/jextract/pull/59

From sundar at openjdk.org  Thu Aug 11 13:18:00 2022
From: sundar at openjdk.org (Athijegannathan Sundararajan)
Date: Thu, 11 Aug 2022 13:18:00 GMT
Subject: RFR: making jextract to work with foreign-memaccess+abi branch of
 panama-foreign [v3]
In-Reply-To: <VCTGRh4pTTfzdJKcsXHiPImokwUbzWsiCCedF5sxjU4=.836800c3-aa3d-4e2e-a374-7bcdb40ae4e7@github.com>
References: <VCTGRh4pTTfzdJKcsXHiPImokwUbzWsiCCedF5sxjU4=.836800c3-aa3d-4e2e-a374-7bcdb40ae4e7@github.com>
Message-ID: <ulKyZoXwkes7r1J9f8NHGJCN6Q1-di3QNohqjUvWPV8=.a3e38aca-6c06-49e6-98d2-c5490b604b95@github.com>

> making jextract to work with foreign-memaccess+abi branch of panama-foreign

Athijegannathan Sundararajan has updated the pull request incrementally with one additional commit since the last revision:

  adding 'panama' branch to ignore list

-------------

Changes:
  - all: https://git.openjdk.org/jextract/pull/59/files
  - new: https://git.openjdk.org/jextract/pull/59/files/69e8f079..610da996

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jextract&pr=59&range=02
 - incr: https://webrevs.openjdk.org/?repo=jextract&pr=59&range=01-02

  Stats: 1 line in 1 file changed: 1 ins; 0 del; 0 mod
  Patch: https://git.openjdk.org/jextract/pull/59.diff
  Fetch: git fetch https://git.openjdk.org/jextract pull/59/head:pull/59

PR: https://git.openjdk.org/jextract/pull/59

From sundar at openjdk.org  Thu Aug 11 15:06:46 2022
From: sundar at openjdk.org (Athijegannathan Sundararajan)
Date: Thu, 11 Aug 2022 15:06:46 GMT
Subject: Withdrawn: making jextract to work with foreign-memaccess+abi branch
 of panama-foreign
In-Reply-To: <VCTGRh4pTTfzdJKcsXHiPImokwUbzWsiCCedF5sxjU4=.836800c3-aa3d-4e2e-a374-7bcdb40ae4e7@github.com>
References: <VCTGRh4pTTfzdJKcsXHiPImokwUbzWsiCCedF5sxjU4=.836800c3-aa3d-4e2e-a374-7bcdb40ae4e7@github.com>
Message-ID: <pDIPFIzAP9C9TRbQXpr56VVEm-WVZLwUiK2lTH5QlZw=.e94920d2-f00e-4069-af0d-bc7dd98cb489@github.com>

On Thu, 11 Aug 2022 09:55:36 GMT, Athijegannathan Sundararajan <sundar at openjdk.org> wrote:

> making jextract to work with foreign-memaccess+abi branch of panama-foreign

This pull request has been closed without being integrated.

-------------

PR: https://git.openjdk.org/jextract/pull/59

From sundar at openjdk.org  Fri Aug 12 10:06:21 2022
From: sundar at openjdk.org (Athijegannathan Sundararajan)
Date: Fri, 12 Aug 2022 10:06:21 GMT
Subject: RFR: making jextract work with latest panama-foreign repo targetting
 jdk 20
Message-ID: <_H-CjVNElRokT6DWTzNxRXkQPrVc1YbKtZ7kchQiYig=.ce686061-df93-485b-8375-b38d5a02492f@github.com>

making jextract work with latest panama-foreign repo targetting jdk 20

-------------

Commit messages:
 - making jextract work with latest panama-foreign repo targetting jdk 20

Changes: https://git.openjdk.org/jextract/pull/60/files
 Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=60&range=00
  Stats: 483 lines in 118 files changed: 31 ins; 44 del; 408 mod
  Patch: https://git.openjdk.org/jextract/pull/60.diff
  Fetch: git fetch https://git.openjdk.org/jextract pull/60/head:pull/60

PR: https://git.openjdk.org/jextract/pull/60

From sundar at openjdk.org  Fri Aug 12 10:15:56 2022
From: sundar at openjdk.org (Athijegannathan Sundararajan)
Date: Fri, 12 Aug 2022 10:15:56 GMT
Subject: RFR: making jextract work with latest panama-foreign repo
 targetting jdk 20 [v2]
In-Reply-To: <_H-CjVNElRokT6DWTzNxRXkQPrVc1YbKtZ7kchQiYig=.ce686061-df93-485b-8375-b38d5a02492f@github.com>
References: <_H-CjVNElRokT6DWTzNxRXkQPrVc1YbKtZ7kchQiYig=.ce686061-df93-485b-8375-b38d5a02492f@github.com>
Message-ID: <FA3ZSyGvwu76gdG89A8rd9Y3Urj_EFEc_Jbrcq2Ev3o=.3120bc1a-1d44-45a6-ac26-2c6adcef580a@github.com>

> making jextract work with latest panama-foreign repo targetting jdk 20

Athijegannathan Sundararajan has updated the pull request incrementally with one additional commit since the last revision:

  version info changed in README.md

-------------

Changes:
  - all: https://git.openjdk.org/jextract/pull/60/files
  - new: https://git.openjdk.org/jextract/pull/60/files/0af0593d..efaaa1c9

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jextract&pr=60&range=01
 - incr: https://webrevs.openjdk.org/?repo=jextract&pr=60&range=00-01

  Stats: 3 lines in 1 file changed: 0 ins; 0 del; 3 mod
  Patch: https://git.openjdk.org/jextract/pull/60.diff
  Fetch: git fetch https://git.openjdk.org/jextract pull/60/head:pull/60

PR: https://git.openjdk.org/jextract/pull/60

From mcimadamore at openjdk.org  Fri Aug 12 11:02:36 2022
From: mcimadamore at openjdk.org (Maurizio Cimadamore)
Date: Fri, 12 Aug 2022 11:02:36 GMT
Subject: RFR: making jextract work with latest panama-foreign repo
 targetting jdk 20 [v2]
In-Reply-To: <FA3ZSyGvwu76gdG89A8rd9Y3Urj_EFEc_Jbrcq2Ev3o=.3120bc1a-1d44-45a6-ac26-2c6adcef580a@github.com>
References: <_H-CjVNElRokT6DWTzNxRXkQPrVc1YbKtZ7kchQiYig=.ce686061-df93-485b-8375-b38d5a02492f@github.com>
 <FA3ZSyGvwu76gdG89A8rd9Y3Urj_EFEc_Jbrcq2Ev3o=.3120bc1a-1d44-45a6-ac26-2c6adcef580a@github.com>
Message-ID: <Z9SHYvnRUJwJNs6lA0uP-kHBHGKvKMT-sCkwZ-fOHxU=.1d4dc90b-8b45-4f57-b463-833f81c6e60c@github.com>

On Fri, 12 Aug 2022 10:15:56 GMT, Athijegannathan Sundararajan <sundar at openjdk.org> wrote:

>> making jextract work with latest panama-foreign repo targetting jdk 20
>
> Athijegannathan Sundararajan has updated the pull request incrementally with one additional commit since the last revision:
> 
>   version info changed in README.md

Good job.
I like how the tests (and samples) are simpler than they used to be, as a lot of address -> segment conversion is now gone.

README.md line 7:

> 5: ### Getting started
> 6: 
> 7: `jextract` depends on the [C libclang API](https://clang.llvm.org/doxygen/group__CINDEX.html). To build the jextract sources, the easiest option is to download LLVM binaries for your platform, which can be found [here](https://releases.llvm.org/download.html) (a version >= 9 is required). Both the `jextract` tool and the bindings it generates depend heavily on the [Foreign Function & Memory API](https://openjdk.java.net/jeps/424), so a suitable jdk build from panama-foreign repo is also required.

Suggestion:

`jextract` depends on the [C libclang API](https://clang.llvm.org/doxygen/group__CINDEX.html). To build the jextract sources, the easiest option is to download LLVM binaries for your platform, which can be found [here](https://releases.llvm.org/download.html) (a version >= 9 is required). Both the `jextract` tool and the bindings it generates depend heavily on the [Foreign Function & Memory API](https://openjdk.java.net/jeps/424), so a suitable build of the [panama/foreign repository](https://github.com/openjdk/panama-foreign) is also required.

samples/libffmpeg/LibffmpegMain.java line 123:

> 121:                 var pStream = streamsArray.getAtIndex(C_POINTER, i);
> 122:                 // AVStream stream;
> 123:                 var stream = MemorySegment.ofAddress(pStream.address(), AVStream.sizeof(), session);

I understand this is a straight port, but I don't think this is needed any longer. E.g. you can just use `pStream` and pass it to the subsequent getter, I think? (same in other snippets in this sample)

src/main/java/org/openjdk/jextract/impl/Utils.java line 117:

> 115:                 "VarHandle", "ByteOrder",
> 116:                 "FunctionDescriptor", "LibraryLookup",
> 117:                 "MemoryLayout", "MemorySession",

Is this correct? Where does MemorySesison comes from?

src/main/resources/org/openjdk/jextract/impl/resources/RuntimeHelper.java.template line 34:

> 32: 
> 33:     final static SegmentAllocator CONSTANT_ALLOCATOR =
> 34:             (size, align) -> MemorySegment.allocateNative(size, align, MemorySession.global());

These changes are probably not needed. E.g. implicit session seems good.

src/main/resources/org/openjdk/jextract/impl/resources/RuntimeHelper.java.template line 52:

> 50: 
> 51:     static final MemorySegment lookupGlobalVariable(String name, MemoryLayout layout) {
> 52:         return SYMBOL_LOOKUP.lookup(name).map(symbol -> MemorySegment.ofAddress(symbol.address(), layout.byteSize(), MemorySession.global())).orElse(null);

this is bogus: note that `symbol` already has a session. So the above should just use that.

src/main/resources/org/openjdk/jextract/impl/resources/RuntimeHelper.java.template line 52:

> 50: 
> 51:     static final MemorySegment lookupGlobalVariable(String name, MemoryLayout layout) {
> 52:         return SYMBOL_LOOKUP.lookup(name).map(symbol -> MemorySegment.ofAddress(symbol.address(), layout.byteSize(), MemorySession.global())).orElse(null);

Suggestion:

        return SYMBOL_LOOKUP.lookup(name).map(symbol -> MemorySegment.ofAddress(symbol.address(), layout.byteSize(), symbol.session())).orElse(null);

test/jtreg/generator/funcPointerInvokers/TestFuncPointerInvokers.java line 137:

> 135:         try (MemorySession session = MemorySession.openConfined()) {
> 136:             fp_addr$set(fp_addr.allocate((addr) -> MemorySegment.ofAddress(addr.address() + 1), session));
> 137:             assertEquals(fp_addr.ofAddress(fp_addr$get(), session).apply(MemorySegment.ofAddress(42)).address(), MemorySegment.ofAddress(43).address());

no need for calling `address` before assertEquals

test/jtreg/generator/funcPointerInvokers/TestFuncPointerInvokers.java line 137:

> 135:         try (MemorySession session = MemorySession.openConfined()) {
> 136:             fp_addr$set(fp_addr.allocate((addr) -> MemorySegment.ofAddress(addr.address() + 1), session));
> 137:             assertEquals(fp_addr.ofAddress(fp_addr$get(), session).apply(MemorySegment.ofAddress(42)).address(), MemorySegment.ofAddress(43).address());

Suggestion:

            assertEquals(fp_addr.ofAddress(fp_addr$get(), session).apply(MemorySegment.ofAddress(42)), MemorySegment.ofAddress(43));

-------------

PR: https://git.openjdk.org/jextract/pull/60

From sundar at openjdk.org  Fri Aug 12 11:38:39 2022
From: sundar at openjdk.org (Athijegannathan Sundararajan)
Date: Fri, 12 Aug 2022 11:38:39 GMT
Subject: RFR: making jextract work with latest panama-foreign repo
 targetting jdk 20 [v3]
In-Reply-To: <_H-CjVNElRokT6DWTzNxRXkQPrVc1YbKtZ7kchQiYig=.ce686061-df93-485b-8375-b38d5a02492f@github.com>
References: <_H-CjVNElRokT6DWTzNxRXkQPrVc1YbKtZ7kchQiYig=.ce686061-df93-485b-8375-b38d5a02492f@github.com>
Message-ID: <m6AIVyeknGOCCw4c8lEsQxLdn4u0P2K2fys5gU14vrg=.e866c853-4e88-4fd2-9afc-3b8ac2facd79@github.com>

> making jextract work with latest panama-foreign repo targetting jdk 20

Athijegannathan Sundararajan has updated the pull request incrementally with one additional commit since the last revision:

  changes suggested in review. Missed MemoryAddress->MemorySegment change in 'tcl' sample.

-------------

Changes:
  - all: https://git.openjdk.org/jextract/pull/60/files
  - new: https://git.openjdk.org/jextract/pull/60/files/efaaa1c9..b5eccfca

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jextract&pr=60&range=02
 - incr: https://webrevs.openjdk.org/?repo=jextract&pr=60&range=01-02

  Stats: 33 lines in 8 files changed: 0 ins; 9 del; 24 mod
  Patch: https://git.openjdk.org/jextract/pull/60.diff
  Fetch: git fetch https://git.openjdk.org/jextract pull/60/head:pull/60

PR: https://git.openjdk.org/jextract/pull/60

From mcimadamore at openjdk.org  Fri Aug 12 12:16:35 2022
From: mcimadamore at openjdk.org (Maurizio Cimadamore)
Date: Fri, 12 Aug 2022 12:16:35 GMT
Subject: RFR: making jextract work with latest panama-foreign repo
 targetting jdk 20 [v3]
In-Reply-To: <m6AIVyeknGOCCw4c8lEsQxLdn4u0P2K2fys5gU14vrg=.e866c853-4e88-4fd2-9afc-3b8ac2facd79@github.com>
References: <_H-CjVNElRokT6DWTzNxRXkQPrVc1YbKtZ7kchQiYig=.ce686061-df93-485b-8375-b38d5a02492f@github.com>
 <m6AIVyeknGOCCw4c8lEsQxLdn4u0P2K2fys5gU14vrg=.e866c853-4e88-4fd2-9afc-3b8ac2facd79@github.com>
Message-ID: <XT8x3h5h_aP60gEo04x4d9O5mHnu_b38REoI_ZaKMMo=.c9d3e653-ccb5-4aeb-9a11-d53619085b72@github.com>

On Fri, 12 Aug 2022 11:38:39 GMT, Athijegannathan Sundararajan <sundar at openjdk.org> wrote:

>> making jextract work with latest panama-foreign repo targetting jdk 20
>
> Athijegannathan Sundararajan has updated the pull request incrementally with one additional commit since the last revision:
> 
>   changes suggested in review. Missed MemoryAddress->MemorySegment change in 'tcl' sample.

Looks (very) good

samples/libffmpeg/LibffmpegMain.java line 108:

> 106:             // AVFrameContext formatCtx;
> 107:             // formatCtx.nb_streams
> 108:             int nb_streams = AVFormatContext.nb_streams$get(pFormatCtx);

nice!!

-------------

Marked as reviewed by mcimadamore (Committer).

PR: https://git.openjdk.org/jextract/pull/60

From sundar at openjdk.org  Fri Aug 12 13:18:40 2022
From: sundar at openjdk.org (Athijegannathan Sundararajan)
Date: Fri, 12 Aug 2022 13:18:40 GMT
Subject: Integrated: making jextract work with latest panama-foreign repo
 targetting jdk 20
In-Reply-To: <_H-CjVNElRokT6DWTzNxRXkQPrVc1YbKtZ7kchQiYig=.ce686061-df93-485b-8375-b38d5a02492f@github.com>
References: <_H-CjVNElRokT6DWTzNxRXkQPrVc1YbKtZ7kchQiYig=.ce686061-df93-485b-8375-b38d5a02492f@github.com>
Message-ID: <AbQfRAzMJ4awkysokoYvyrQcb7spNxCy4AYzp6HylD0=.5d6e010c-5b0b-4658-994c-85bcec4a57eb@github.com>

On Fri, 12 Aug 2022 09:59:58 GMT, Athijegannathan Sundararajan <sundar at openjdk.org> wrote:

> making jextract work with latest panama-foreign repo targetting jdk 20

This pull request has now been integrated.

Changeset: 5e7d2327
Author:    Athijegannathan Sundararajan <sundar at openjdk.org>
URL:       https://git.openjdk.org/jextract/commit/5e7d2327d124d1ce443aac8e515d67be4319574c
Stats:     500 lines in 120 files changed: 31 ins; 53 del; 416 mod

making jextract work with latest panama-foreign repo targetting jdk 20

-------------

PR: https://git.openjdk.org/jextract/pull/60

From maurizio.cimadamore at oracle.com  Fri Aug 12 14:51:52 2022
From: maurizio.cimadamore at oracle.com (Maurizio Cimadamore)
Date: Fri, 12 Aug 2022 15:51:52 +0100
Subject: Code changes required after JDK-8291473
In-Reply-To: <3fde1696-b9f2-09a9-17e0-9d42667b47f8@hhu.de>
References: <c8debf61-8251-16db-b6a5-6303475f9d35@hhu.de>
 <0f31b101-53e7-a8fc-e202-ff662a876f09@oracle.com>
 <9a02ed6b-3022-9bd9-ae8b-5d684ecb8c42@hhu.de>
 <751a627f-8fd4-6210-eb45-2fcdc1dd13bb@oracle.com>
 <68433e50-e717-db0c-eaa3-7c0ab4fca796@hhu.de>
 <e40f2dd7-3471-9cb4-48c7-b6500416b7b0@oracle.com>
 <35555875-787c-b587-4723-0b67f541e3ba@hhu.de>
 <8494784f-f77f-fe73-a092-4e58888b9a76@oracle.com>
 <5115a250-0fc9-9add-186b-e8df4a6f650d@oracle.com>
 <3fde1696-b9f2-09a9-17e0-9d42667b47f8@hhu.de>
Message-ID: <6e92f586-f553-00e8-3f67-52da934f5802@oracle.com>

Should be fixed now. Look at the panama branch here:

https://github.com/openjdk/jextract/tree/panama

Maurizio

On 09/08/2022 09:29, Filip Krakowski wrote:
> Hi,
>
> this sounds great! Once the changes get pushed I will include 
> jextract's "panama" branch into our CI pipeline, so that we always 
> have a version that is in sync with the latest "foreign-memaccess+abi" 
> branch.
>
> Thanks for the quick response! Looking forward to trying out it out :)
>
> Best regards,
> Filip
>
> On 08.08.22 19:44, Maurizio Cimadamore wrote:
>> After some internal discussion, we are going to do the following:
>>
>> * create a "panama" branch, which contains the changes needed to run 
>> jextract with the API contained in the Panama repo
>> * have (as now) a set of versioned branches, for jdk18 and jdk19
>> * when 20 becomes GA, we will create a new jdk20 branch, copy the 
>> contents of the "panama" branch into it and make it the default branch
>> * we will _not_ provide binary snapshots for the "panama" branch, as 
>> that's a "bleeding edge" sort of scenario (where you have to build 
>> the panama repo manually to go with it as well)
>>
>> Cheers
>> Maurizio
>>
>>
>> On 05/08/2022 12:31, Maurizio Cimadamore wrote:
>>> Hi Filip,
>>> due to various complex reasons, it is currently not possible for us 
>>> to release binary snapshots using GitHub actions. All the binaries 
>>> we publish come from an internal CI pipeline, which builds and test 
>>> using predefined, well-known configurations, so that we can ensure 
>>> repeatable results. While this is not set in stone, and OpenJDK is 
>>> relatively new to GitHub workflows, I think that, for the time 
>>> being, we have to work with what we've got.
>>>
>>> That said, nothing prevents us from creating several internal 
>>> pipelines, one per jextract version, and add a new one every 6 
>>> months or so, perhaps with some policy to "garbage collect" old 
>>> pipelines (e.g. maybe providing binaries up to JDK N-3 would be a 
>>> good starting point?) - although I'd have to check with the infra 
>>> team and see what solution they'd be most comfortable with.
>>>
>>> P.S. While it would be possible, in principle, for 3rd parties to 
>>> build snapshots of jextract, I find that solution less ideal. 
>>> Plugins such as yours need a stable link to work with. A 3rd party 
>>> build might be a great place to get started (or where to get more 
>>> frequent jextract binary updates, for example), but doesn't provide 
>>> (by design) that kind of long-term commitment that some of the tools 
>>> in the ecosystem would require, at least IMHO.
>>>
>>> Thanks
>>> Maurizio
>>>
>>> On 05/08/2022 11:23, Filip Krakowski wrote:
>>>> Hi,
>>>>
>>>> is it a question of maintainability (diverging branches in case of 
>>>> patches, etc.), which would stand in the way of this, or a question 
>>>> of the infrastructure that would be required? In the latter case, I 
>>>> could try to set something up to accomplish the job of 
>>>> automatically creating releases from each pushed tag matching a 
>>>> version number.
>>>>
>>>> Since the project is hosted on GitHub, I could implement a GitHub 
>>>> Action, which would be triggered on each pushed tag. This Action 
>>>> would then compile the code for different platforms (Linux, macOS, 
>>>> Windows - GitHub provides images for each), create a release inside 
>>>> the repository and attach the created binaries to it. The 
>>>> repository already contains a GitHub Action for building and 
>>>> testing jextract, which I could use as a starting point.
>>>>
>>>> Best regards,
>>>> Filip
>>>>
>>>> On 04.08.22 21:14, Maurizio Cimadamore wrote:
>>>>>
>>>>> On 04/08/2022 20:08, Filip Krakowski wrote:
>>>>>> Each branch would therefore only produce binaries when it 
>>>>>> receives a new tag. Could this work? 
>>>>>
>>>>> I'd have to check - my feeling is that, no that would not be 
>>>>> possible.
>>>>>
>>>>> Another issue is that we'd need to keep updating the main github 
>>>>> branch every few months (not a big issue).
>>>>>
>>>>> Maurizio
>>>>>
>>>>>
>>>>
>

From sundar at openjdk.org  Tue Aug 16 13:31:19 2022
From: sundar at openjdk.org (Athijegannathan Sundararajan)
Date: Tue, 16 Aug 2022 13:31:19 GMT
Subject: RFR: 7903253: sync jextract tests for sequence layout factory split
Message-ID: <SlIM1kr7eYFxaKUHUCNCyyLWO2oeqFMpDjc3T9YBpNw=.f646e796-4e96-4dc6-9cb3-68867809d8ef@github.com>

Avoid -1 elementCount for Sequence layout factory call.

-------------

Commit messages:
 - 7903253: sync jextract tests for sequence layout factory split

Changes: https://git.openjdk.org/jextract/pull/61/files
 Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=61&range=00
  Issue: https://bugs.openjdk.org/browse/CODETOOLS-7903253
  Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/jextract/pull/61.diff
  Fetch: git fetch https://git.openjdk.org/jextract pull/61/head:pull/61

PR: https://git.openjdk.org/jextract/pull/61

From mcimadamore at openjdk.org  Tue Aug 16 15:25:39 2022
From: mcimadamore at openjdk.org (Maurizio Cimadamore)
Date: Tue, 16 Aug 2022 15:25:39 GMT
Subject: RFR: 7903253: sync jextract tests for sequence layout factory
 split
In-Reply-To: <SlIM1kr7eYFxaKUHUCNCyyLWO2oeqFMpDjc3T9YBpNw=.f646e796-4e96-4dc6-9cb3-68867809d8ef@github.com>
References: <SlIM1kr7eYFxaKUHUCNCyyLWO2oeqFMpDjc3T9YBpNw=.f646e796-4e96-4dc6-9cb3-68867809d8ef@github.com>
Message-ID: <Iyytx7xmWmCtaaKhp2oZ-wxuPnqvhnDWsLypp5p8g4o=.34471aee-00fc-4d5f-a553-48e5968e1b57@github.com>

On Tue, 16 Aug 2022 13:26:21 GMT, Athijegannathan Sundararajan <sundar at openjdk.org> wrote:

> Avoid -1 elementCount for Sequence layout factory call.

Looks good

-------------

Marked as reviewed by mcimadamore (Committer).

PR: https://git.openjdk.org/jextract/pull/61

From sundar at openjdk.org  Tue Aug 16 15:29:35 2022
From: sundar at openjdk.org (Athijegannathan Sundararajan)
Date: Tue, 16 Aug 2022 15:29:35 GMT
Subject: Integrated: 7903253: sync jextract tests for sequence layout factory
 split
In-Reply-To: <SlIM1kr7eYFxaKUHUCNCyyLWO2oeqFMpDjc3T9YBpNw=.f646e796-4e96-4dc6-9cb3-68867809d8ef@github.com>
References: <SlIM1kr7eYFxaKUHUCNCyyLWO2oeqFMpDjc3T9YBpNw=.f646e796-4e96-4dc6-9cb3-68867809d8ef@github.com>
Message-ID: <qdNY2kwgKJ5waUq00W1k-2V3AOuaUAXwaomQUEw1Z1s=.80ac8929-9fb1-4ec1-a0d4-850bdb800f97@github.com>

On Tue, 16 Aug 2022 13:26:21 GMT, Athijegannathan Sundararajan <sundar at openjdk.org> wrote:

> Avoid -1 elementCount for Sequence layout factory call.

This pull request has now been integrated.

Changeset: f7704415
Author:    Athijegannathan Sundararajan <sundar at openjdk.org>
URL:       https://git.openjdk.org/jextract/commit/f7704415fb802711b82ddde4aa378d822b181939
Stats:     1 line in 1 file changed: 0 ins; 0 del; 1 mod

7903253: sync jextract tests for sequence layout factory split

Reviewed-by: mcimadamore

-------------

PR: https://git.openjdk.org/jextract/pull/61

From manuel.bleichenbacher at gmail.com  Mon Aug 22 10:39:58 2022
From: manuel.bleichenbacher at gmail.com (Manuel Bleichenbacher)
Date: Mon, 22 Aug 2022 12:39:58 +0200
Subject: Tool architecture vs build-time architeture vs run-time architecture
Message-ID: <CAA7F5j+5pyUu80TBJnh+ukUa0Pwwk7sf+buG+1yD=jeYu7i1OA@mail.gmail.com>

I'm migrating this detail question over from the panama-dev mailing list:
can jextract built for macOS on Intel run on macOS for ARM generate correct
code for macOS on ARM?

Note that the architecture (Intel vs ARM) affects three separate things:

   1. jextract (the JVM and jextract are built for a certain platform and
   architecture)
   2. the architecture where jextract is run (macOS can run Intel code on
   ARM)
   3. the architecture of the resulting / the architecture where the
   generated code will be run

I understand that separate code is needed for different platforms and
architectures. So typically you would run jextract on each platform
separately. But what decides if jextract generates code for ARM or Intel?
Is it (1) the jextract variant or is it (2) the operating system it is run
on? Or does it even generate an unusable result if (1) and (2) are
different?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jextract-dev/attachments/20220822/efadda09/attachment.htm>

From manuel.bleichenbacher at gmail.com  Mon Aug 22 20:54:02 2022
From: manuel.bleichenbacher at gmail.com (Manuel Bleichenbacher)
Date: Mon, 22 Aug 2022 22:54:02 +0200
Subject: jextract for operating system API
Message-ID: <CAA7F5jLH3Mk8ffwT_eWNAy1UywNrSAXgRYp8NJTdt42JhwGkdg@mail.gmail.com>

Hi everybody

As promised, I've compiled my findings when using jextract for the Java
Does USB project, which uses the foreign function & memory API to
communicate with USB devices on Windows, macOS and Linux. After initial
difficulties with jextract, I've written all the layouts, method and
variable handles manually. But now I've given jextract another try. The
commands I've used can be found in in the "jextract" branch of the project
at
https://github.com/manuelbl/JavaDoesUSB/tree/jextract/java-does-usb/jextract

Please note that I think that automatically generating code for some of the
old Windows and POSIX APIs is almost impossible. My use of jextract (for OS
APIs) is probably far from the main use with third-party libraries. And I
might also have made mistakes, or there might be a simple solution to my
issues. See my conclusions below for areas where I think jextract has still
potential.


Linux:

Overall, I was able to generate most of what I needed with jextract. The
issues were:

- Generating code for libsystemd (sd-device.h) failed with "unknown type
name 'intmax_t'". I think this should be defined by inttypes.h, which is
included. My guess is that clang headers and Linux headers are mixed up.
- When generating code for usbdevice_fs.h, it didn't generate the constants
for USBDEVFS_CLAIMINTERFACE and similar ones. The reason might be that the
defines look like functions even though they evaluate to a constant value.


macOS:

Major parts of macOS are provided in the form of frameworks and they are
treated in a special way, both when it comes to the header files and
dynamic libraries. As an example, my library uses several functions from
the CoreFoundation framework. In C/CC++, you can simply include them like
so:

#include <CoreFoundation/CoreFoundation.h>

That's surprising as the file actually resides in
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/CoreFoundation.framework/Versions/A/Headers/CoreFoundation.h.

Note that it is in a directory called "Headers" and not in "CoreFoundation"
as the include statement suggests. The relative path
"CoreFoundation/CoreFoundation.h" does not exist at all.

This wouldn't be a big problem if this was just the entry point. But this
pattern is used recursively. CoreFoundation.h includes about 30 other
header files like this, all with a relative path that does not exist.

Xcode magically resolves this. But jextract seems to be missing this magic.
So it cannot be used for any macOS frameworks. In other words, it cannot be
for the major part of the macOS APIs. My library uses a single
non-framework functions. So using jextract for macOS is pointless without
resolving this issue.


Windows:

On Windows I spent the most time -- with no success at all. I only
partially understand the problems. Here is what I've encountered:

The first simple goal was to have code for CloseHandle() generated. I tried
it three different ways:

   - Generating code for handleapi.h generated 1000 classes. Code for
   CloseHandle() was generated in 2 classes, but none of them was publicly
   available.
   - Generating code for handleapi.h restricted to the function
   CloseHandle() generated a small number of class with a public static method
   for CloseHandle(). However, at run-time the method produces the error
   "java.lang.UnsatisfiedLinkError: unresolved symbol: CloseHandle"
   - Generating code for windows.h restricted to CloseHandle() also
   resulted in the UnsatisfiedLinkError.

I then tried to generate code for Winusb.h. Unfortunately, this header file
is not independent. It expects that you have already processed other header
files before it. So it fails with "error: unknown type name
'LARGE_INTEGER'". This behavior with be reproduced in Visual Studio.

Next up was Cfgmgr32.h. It also doesn't seem to be independent and fails
with "error: unknown type name 'ULONG'". Setupapi.h and Usbioctl.h behave
similarly.

Windows was the only system where I had problems with preprocessor macros.
The macro _M_AMD64=100 is certainly needed. That got me further than
without it. And I've defined two Unicode macros, copied from the Visual
Studio project. But the macros might still be the source of some problems.



Conclusion:

At the moment, jextract doesn't help much for accessing OS APIs.

On Linux, the "unknown type name 'intmax_t'" is worth investigating. It
could be helpful for many APIs.

On macOS, it's all about the frameworks and the magic behind resolving the
include path. If it can be made to work, it hopefully opens up the world of
framework (plus some challenges that are still hidden).

On Windows, more investigation is needed. Work on the required macros is
needed, as well as more experiments with header files just built for code
generation (to get beyond the problem of header files that are not
independent).


Regards
Manuel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jextract-dev/attachments/20220822/1de3ba10/attachment-0001.htm>

From sundararajan.athijegannathan at oracle.com  Tue Aug 23 03:48:42 2022
From: sundararajan.athijegannathan at oracle.com (Sundararajan Athijegannathan)
Date: Tue, 23 Aug 2022 03:48:42 +0000
Subject: jextract for operating system API
In-Reply-To: <CAA7F5jLH3Mk8ffwT_eWNAy1UywNrSAXgRYp8NJTdt42JhwGkdg@mail.gmail.com>
References: <CAA7F5jLH3Mk8ffwT_eWNAy1UywNrSAXgRYp8NJTdt42JhwGkdg@mail.gmail.com>
Message-ID: <DM6PR10MB2954EC3FC86C45525019878DEA709@DM6PR10MB2954.namprd10.prod.outlook.com>

Thanks for your experiments and comments. I'll go over and file bug(s) as needed.

-Sundar
________________________________
From: jextract-dev <jextract-dev-retn at openjdk.org> on behalf of Manuel Bleichenbacher <manuel.bleichenbacher at gmail.com>
Sent: 23 August 2022 02:24
To: jextract-dev at openjdk.org <jextract-dev at openjdk.org>
Subject: jextract for operating system API

Hi everybody

As promised, I've compiled my findings when using jextract for the Java Does USB project, which uses the foreign function & memory API to communicate with USB devices on Windows, macOS and Linux. After initial difficulties with jextract, I've written all the layouts, method and variable handles manually. But now I've given jextract another try. The commands I've used can be found in in the "jextract" branch of the project at https://github.com/manuelbl/JavaDoesUSB/tree/jextract/java-does-usb/jextract

Please note that I think that automatically generating code for some of the old Windows and POSIX APIs is almost impossible. My use of jextract (for OS APIs) is probably far from the main use with third-party libraries. And I might also have made mistakes, or there might be a simple solution to my issues. See my conclusions below for areas where I think jextract has still potential.


Linux:

Overall, I was able to generate most of what I needed with jextract. The issues were:

- Generating code for libsystemd (sd-device.h) failed with "unknown type name 'intmax_t'". I think this should be defined by inttypes.h, which is included. My guess is that clang headers and Linux headers are mixed up.
- When generating code for usbdevice_fs.h, it didn't generate the constants for USBDEVFS_CLAIMINTERFACE and similar ones. The reason might be that the defines look like functions even though they evaluate to a constant value.


macOS:

Major parts of macOS are provided in the form of frameworks and they are treated in a special way, both when it comes to the header files and dynamic libraries. As an example, my library uses several functions from the CoreFoundation framework. In C/CC++, you can simply include them like so:

#include <CoreFoundation/CoreFoundation.h>

That's surprising as the file actually resides in /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/CoreFoundation.framework/Versions/A/Headers/CoreFoundation.h.

Note that it is in a directory called "Headers" and not in "CoreFoundation" as the include statement suggests. The relative path "CoreFoundation/CoreFoundation.h" does not exist at all.

This wouldn't be a big problem if this was just the entry point. But this pattern is used recursively. CoreFoundation.h includes about 30 other header files like this, all with a relative path that does not exist.

Xcode magically resolves this. But jextract seems to be missing this magic. So it cannot be used for any macOS frameworks. In other words, it cannot be for the major part of the macOS APIs. My library uses a single non-framework functions. So using jextract for macOS is pointless without resolving this issue.


Windows:

On Windows I spent the most time -- with no success at all. I only partially understand the problems. Here is what I've encountered:

The first simple goal was to have code for CloseHandle() generated. I tried it three different ways:

  *   Generating code for handleapi.h generated 1000 classes. Code for CloseHandle() was generated in 2 classes, but none of them was publicly available.
  *   Generating code for handleapi.h restricted to the function CloseHandle() generated a small number of class with a public static method for CloseHandle(). However, at run-time the method produces the error "java.lang.UnsatisfiedLinkError: unresolved symbol: CloseHandle"
  *   Generating code for windows.h restricted to CloseHandle() also resulted in the UnsatisfiedLinkError.

I then tried to generate code for Winusb.h. Unfortunately, this header file is not independent. It expects that you have already processed other header files before it. So it fails with "error: unknown type name 'LARGE_INTEGER'". This behavior with be reproduced in Visual Studio.

Next up was Cfgmgr32.h. It also doesn't seem to be independent and fails with "error: unknown type name 'ULONG'". Setupapi.h and Usbioctl.h behave similarly.

Windows was the only system where I had problems with preprocessor macros. The macro _M_AMD64=100 is certainly needed. That got me further than without it. And I've defined two Unicode macros, copied from the Visual Studio project. But the macros might still be the source of some problems.



Conclusion:

At the moment, jextract doesn't help much for accessing OS APIs.

On Linux, the "unknown type name 'intmax_t'" is worth investigating. It could be helpful for many APIs.

On macOS, it's all about the frameworks and the magic behind resolving the include path. If it can be made to work, it hopefully opens up the world of framework (plus some challenges that are still hidden).

On Windows, more investigation is needed. Work on the required macros is needed, as well as more experiments with header files just built for code generation (to get beyond the problem of header files that are not independent).


Regards
Manuel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jextract-dev/attachments/20220823/9d289547/attachment.htm>

From manuel.bleichenbacher at gmail.com  Tue Aug 23 20:31:09 2022
From: manuel.bleichenbacher at gmail.com (Manuel Bleichenbacher)
Date: Tue, 23 Aug 2022 22:31:09 +0200
Subject: jextract for operating system API
In-Reply-To: <DM6PR10MB2954EC3FC86C45525019878DEA709@DM6PR10MB2954.namprd10.prod.outlook.com>
References: <CAA7F5jLH3Mk8ffwT_eWNAy1UywNrSAXgRYp8NJTdt42JhwGkdg@mail.gmail.com>
 <DM6PR10MB2954EC3FC86C45525019878DEA709@DM6PR10MB2954.namprd10.prod.outlook.com>
Message-ID: <CAA7F5jK+_1J7K2fhdox+1Y6SKvx4JCSW0947Njmzy7ztm_Qt9w@mail.gmail.com>

On Windows, I've made major progress, solved most of the issues and run
into a few new ones.

The main thing is: It's not possible to directly process a header file like
winusb.h. Instead, a helper header file needs to be created:

#include <windows.h>
#include <winusb.h>

A mistake I made yesterday was the omission of "-l Kernel32" (or whatever
library is needed).

With these two changes and the use of --include-xxx, more or less all
functions, structs and macros can be created. (I didn't yet have time to
look into GUID constants like GUID_DEVINTERFACE_USB_DEVICE; they probably
don't work due to macro trickery.)

There are 2 structs that do not work (with rather advanced issues):

(1) The generated class _USB_NODE_CONNECTION_INFORMATION_EX cannot be
instanciated. The constructor throws the exception:
java.lang.UnsupportedOperationException: Invalid alignment requirements for
layout s16(DeviceAddress)

This struct (like many USB descriptors) is packed with 16-bit integer
values on odd offsets. jextract insists on creating a layout with strict
alignment rules. So the generated layout
net.codecrete.usb.windows.gen.usbioctl._USB_NODE_CONNECTION_INFORMATION_EX#$struct$LAYOUT
fails, while the manually created layout
net.codecrete.usb.windows.USBIOCtl#USB_NODE_CONNECTION_INFORMATION_EX$Struct
works.

(2) The struct _SP_DEVICE_INTERFACE_DETAIL_DATA_W isn't usable. In C, it
looks like this:

typedef struct _SP_DEVICE_INTERFACE_DETAIL_DATA_W {
  DWORD cbSize;
  WCHAR DevicePath[ANYSIZE_ARRAY];
} SP_DEVICE_INTERFACE_DETAIL_DATA_W, *PSP_DEVICE_INTERFACE_DETAIL_DATA_W;

It's basically a variable length struct. ANYSIZE_ARRAY is defined as 1. But
you are supposed to allocate additional memory for the DevicePath. I guess
this is old style trickery that cannot be solved with an automated tool.

Minor annoyances are:

- jextract on Windows is a batch file that never returns. So it's not
possible to call jextract multiple times for a single batch file.
- It would be nicer if structs used the nice typedef name instead of the
ugly struct name, e.g. SP_DEVICE_INTERFACE_DATA_W instead of
_SP_DEVICE_INTERFACE_DATA_W (note the leading underscore).

Overall: success on Windows.

Regards
Manuel

On Tue, Aug 23, 2022 at 5:48 AM Sundararajan Athijegannathan <
sundararajan.athijegannathan at oracle.com> wrote:

> Thanks for your experiments and comments. I'll go over and file bug(s) as
> needed.
>
> -Sundar
> ------------------------------
> *From:* jextract-dev <jextract-dev-retn at openjdk.org> on behalf of Manuel
> Bleichenbacher <manuel.bleichenbacher at gmail.com>
> *Sent:* 23 August 2022 02:24
> *To:* jextract-dev at openjdk.org <jextract-dev at openjdk.org>
> *Subject:* jextract for operating system API
>
> Hi everybody
>
> As promised, I've compiled my findings when using jextract for the Java
> Does USB project, which uses the foreign function & memory API to
> communicate with USB devices on Windows, macOS and Linux. After initial
> difficulties with jextract, I've written all the layouts, method and
> variable handles manually. But now I've given jextract another try. The
> commands I've used can be found in in the "jextract" branch of the project
> at
> https://github.com/manuelbl/JavaDoesUSB/tree/jextract/java-does-usb/jextract
>
> Please note that I think that automatically generating code for some of
> the old Windows and POSIX APIs is almost impossible. My use of jextract
> (for OS APIs) is probably far from the main use with third-party libraries.
> And I might also have made mistakes, or there might be a simple solution to
> my issues. See my conclusions below for areas where I think jextract has
> still potential.
>
>
> Linux:
>
> Overall, I was able to generate most of what I needed with jextract. The
> issues were:
>
> - Generating code for libsystemd (sd-device.h) failed with "unknown type
> name 'intmax_t'". I think this should be defined by inttypes.h, which is
> included. My guess is that clang headers and Linux headers are mixed up.
> - When generating code for usbdevice_fs.h, it didn't generate the
> constants for USBDEVFS_CLAIMINTERFACE and similar ones. The reason might be
> that the defines look like functions even though they evaluate to a
> constant value.
>
>
> macOS:
>
> Major parts of macOS are provided in the form of frameworks and they are
> treated in a special way, both when it comes to the header files and
> dynamic libraries. As an example, my library uses several functions from
> the CoreFoundation framework. In C/CC++, you can simply include them like
> so:
>
> #include <CoreFoundation/CoreFoundation.h>
>
> That's surprising as the file actually resides in
> /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/CoreFoundation.framework/Versions/A/Headers/CoreFoundation.h.
>
> Note that it is in a directory called "Headers" and not in
> "CoreFoundation" as the include statement suggests. The relative path
> "CoreFoundation/CoreFoundation.h" does not exist at all.
>
> This wouldn't be a big problem if this was just the entry point. But this
> pattern is used recursively. CoreFoundation.h includes about 30 other
> header files like this, all with a relative path that does not exist.
>
> Xcode magically resolves this. But jextract seems to be missing this
> magic. So it cannot be used for any macOS frameworks. In other words, it
> cannot be for the major part of the macOS APIs. My library uses a single
> non-framework functions. So using jextract for macOS is pointless without
> resolving this issue.
>
>
> Windows:
>
> On Windows I spent the most time -- with no success at all. I only
> partially understand the problems. Here is what I've encountered:
>
> The first simple goal was to have code for CloseHandle() generated. I
> tried it three different ways:
>
>    - Generating code for handleapi.h generated 1000 classes. Code for
>    CloseHandle() was generated in 2 classes, but none of them was publicly
>    available.
>    - Generating code for handleapi.h restricted to the function
>    CloseHandle() generated a small number of class with a public static method
>    for CloseHandle(). However, at run-time the method produces the error
>    "java.lang.UnsatisfiedLinkError: unresolved symbol: CloseHandle"
>    - Generating code for windows.h restricted to CloseHandle() also
>    resulted in the UnsatisfiedLinkError.
>
> I then tried to generate code for Winusb.h. Unfortunately, this header
> file is not independent. It expects that you have already processed other
> header files before it. So it fails with "error: unknown type name
> 'LARGE_INTEGER'". This behavior with be reproduced in Visual Studio.
>
> Next up was Cfgmgr32.h. It also doesn't seem to be independent and fails
> with "error: unknown type name 'ULONG'". Setupapi.h and Usbioctl.h behave
> similarly.
>
> Windows was the only system where I had problems with preprocessor macros.
> The macro _M_AMD64=100 is certainly needed. That got me further than
> without it. And I've defined two Unicode macros, copied from the Visual
> Studio project. But the macros might still be the source of some problems.
>
>
>
> Conclusion:
>
> At the moment, jextract doesn't help much for accessing OS APIs.
>
> On Linux, the "unknown type name 'intmax_t'" is worth investigating. It
> could be helpful for many APIs.
>
> On macOS, it's all about the frameworks and the magic behind resolving the
> include path. If it can be made to work, it hopefully opens up the world of
> framework (plus some challenges that are still hidden).
>
> On Windows, more investigation is needed. Work on the required macros is
> needed, as well as more experiments with header files just built for code
> generation (to get beyond the problem of header files that are not
> independent).
>
>
> Regards
> Manuel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jextract-dev/attachments/20220823/9473f770/attachment-0001.htm>

From sundararajan.athijegannathan at oracle.com  Wed Aug 24 04:34:53 2022
From: sundararajan.athijegannathan at oracle.com (Sundararajan Athijegannathan)
Date: Wed, 24 Aug 2022 04:34:53 +0000
Subject: jextract for operating system API
In-Reply-To: <CAA7F5jK+_1J7K2fhdox+1Y6SKvx4JCSW0947Njmzy7ztm_Qt9w@mail.gmail.com>
References: <CAA7F5jLH3Mk8ffwT_eWNAy1UywNrSAXgRYp8NJTdt42JhwGkdg@mail.gmail.com>
 <DM6PR10MB2954EC3FC86C45525019878DEA709@DM6PR10MB2954.namprd10.prod.outlook.com>
 <CAA7F5jK+_1J7K2fhdox+1Y6SKvx4JCSW0947Njmzy7ztm_Qt9w@mail.gmail.com>
Message-ID: <DM6PR10MB29546BD30EC971027E2C87D9EA739@DM6PR10MB2954.namprd10.prod.outlook.com>

Thanks for your report on progress. We'll try to files (I'm not on Windows so it'll take sometime before bug filing).

Thanks
-Sundar
________________________________
From: jextract-dev <jextract-dev-retn at openjdk.org> on behalf of Manuel Bleichenbacher <manuel.bleichenbacher at gmail.com>
Sent: 24 August 2022 02:01
To: jextract-dev at openjdk.org <jextract-dev at openjdk.org>
Subject: Re: jextract for operating system API

On Windows, I've made major progress, solved most of the issues and run into a few new ones.

The main thing is: It's not possible to directly process a header file like winusb.h. Instead, a helper header file needs to be created:

#include <windows.h>
#include <winusb.h>

A mistake I made yesterday was the omission of "-l Kernel32" (or whatever library is needed).

With these two changes and the use of --include-xxx, more or less all functions, structs and macros can be created. (I didn't yet have time to look into GUID constants like GUID_DEVINTERFACE_USB_DEVICE; they probably don't work due to macro trickery.)

There are 2 structs that do not work (with rather advanced issues):

(1) The generated class _USB_NODE_CONNECTION_INFORMATION_EX cannot be instanciated. The constructor throws the exception: java.lang.UnsupportedOperationException: Invalid alignment requirements for layout s16(DeviceAddress)

This struct (like many USB descriptors) is packed with 16-bit integer values on odd offsets. jextract insists on creating a layout with strict alignment rules. So the generated layout net.codecrete.usb.windows.gen.usbioctl._USB_NODE_CONNECTION_INFORMATION_EX#$struct$LAYOUT fails, while the manually created layout net.codecrete.usb.windows.USBIOCtl#USB_NODE_CONNECTION_INFORMATION_EX$Struct works.

(2) The struct _SP_DEVICE_INTERFACE_DETAIL_DATA_W isn't usable. In C, it looks like this:

typedef struct _SP_DEVICE_INTERFACE_DETAIL_DATA_W {
  DWORD cbSize;
  WCHAR DevicePath[ANYSIZE_ARRAY];
} SP_DEVICE_INTERFACE_DETAIL_DATA_W, *PSP_DEVICE_INTERFACE_DETAIL_DATA_W;

It's basically a variable length struct. ANYSIZE_ARRAY is defined as 1. But you are supposed to allocate additional memory for the DevicePath. I guess this is old style trickery that cannot be solved with an automated tool.

Minor annoyances are:

- jextract on Windows is a batch file that never returns. So it's not possible to call jextract multiple times for a single batch file.
- It would be nicer if structs used the nice typedef name instead of the ugly struct name, e.g. SP_DEVICE_INTERFACE_DATA_W instead of _SP_DEVICE_INTERFACE_DATA_W (note the leading underscore).

Overall: success on Windows.

Regards
Manuel

On Tue, Aug 23, 2022 at 5:48 AM Sundararajan Athijegannathan <sundararajan.athijegannathan at oracle.com<mailto:sundararajan.athijegannathan at oracle.com>> wrote:
Thanks for your experiments and comments. I'll go over and file bug(s) as needed.

-Sundar
________________________________
From: jextract-dev <jextract-dev-retn at openjdk.org<mailto:jextract-dev-retn at openjdk.org>> on behalf of Manuel Bleichenbacher <manuel.bleichenbacher at gmail.com<mailto:manuel.bleichenbacher at gmail.com>>
Sent: 23 August 2022 02:24
To: jextract-dev at openjdk.org<mailto:jextract-dev at openjdk.org> <jextract-dev at openjdk.org<mailto:jextract-dev at openjdk.org>>
Subject: jextract for operating system API

Hi everybody

As promised, I've compiled my findings when using jextract for the Java Does USB project, which uses the foreign function & memory API to communicate with USB devices on Windows, macOS and Linux. After initial difficulties with jextract, I've written all the layouts, method and variable handles manually. But now I've given jextract another try. The commands I've used can be found in in the "jextract" branch of the project at https://github.com/manuelbl/JavaDoesUSB/tree/jextract/java-does-usb/jextract

Please note that I think that automatically generating code for some of the old Windows and POSIX APIs is almost impossible. My use of jextract (for OS APIs) is probably far from the main use with third-party libraries. And I might also have made mistakes, or there might be a simple solution to my issues. See my conclusions below for areas where I think jextract has still potential.


Linux:

Overall, I was able to generate most of what I needed with jextract. The issues were:

- Generating code for libsystemd (sd-device.h) failed with "unknown type name 'intmax_t'". I think this should be defined by inttypes.h, which is included. My guess is that clang headers and Linux headers are mixed up.
- When generating code for usbdevice_fs.h, it didn't generate the constants for USBDEVFS_CLAIMINTERFACE and similar ones. The reason might be that the defines look like functions even though they evaluate to a constant value.


macOS:

Major parts of macOS are provided in the form of frameworks and they are treated in a special way, both when it comes to the header files and dynamic libraries. As an example, my library uses several functions from the CoreFoundation framework. In C/CC++, you can simply include them like so:

#include <CoreFoundation/CoreFoundation.h>

That's surprising as the file actually resides in /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/CoreFoundation.framework/Versions/A/Headers/CoreFoundation.h.

Note that it is in a directory called "Headers" and not in "CoreFoundation" as the include statement suggests. The relative path "CoreFoundation/CoreFoundation.h" does not exist at all.

This wouldn't be a big problem if this was just the entry point. But this pattern is used recursively. CoreFoundation.h includes about 30 other header files like this, all with a relative path that does not exist.

Xcode magically resolves this. But jextract seems to be missing this magic. So it cannot be used for any macOS frameworks. In other words, it cannot be for the major part of the macOS APIs. My library uses a single non-framework functions. So using jextract for macOS is pointless without resolving this issue.


Windows:

On Windows I spent the most time -- with no success at all. I only partially understand the problems. Here is what I've encountered:

The first simple goal was to have code for CloseHandle() generated. I tried it three different ways:

  *   Generating code for handleapi.h generated 1000 classes. Code for CloseHandle() was generated in 2 classes, but none of them was publicly available.
  *   Generating code for handleapi.h restricted to the function CloseHandle() generated a small number of class with a public static method for CloseHandle(). However, at run-time the method produces the error "java.lang.UnsatisfiedLinkError: unresolved symbol: CloseHandle"
  *   Generating code for windows.h restricted to CloseHandle() also resulted in the UnsatisfiedLinkError.

I then tried to generate code for Winusb.h. Unfortunately, this header file is not independent. It expects that you have already processed other header files before it. So it fails with "error: unknown type name 'LARGE_INTEGER'". This behavior with be reproduced in Visual Studio.

Next up was Cfgmgr32.h. It also doesn't seem to be independent and fails with "error: unknown type name 'ULONG'". Setupapi.h and Usbioctl.h behave similarly.

Windows was the only system where I had problems with preprocessor macros. The macro _M_AMD64=100 is certainly needed. That got me further than without it. And I've defined two Unicode macros, copied from the Visual Studio project. But the macros might still be the source of some problems.



Conclusion:

At the moment, jextract doesn't help much for accessing OS APIs.

On Linux, the "unknown type name 'intmax_t'" is worth investigating. It could be helpful for many APIs.

On macOS, it's all about the frameworks and the magic behind resolving the include path. If it can be made to work, it hopefully opens up the world of framework (plus some challenges that are still hidden).

On Windows, more investigation is needed. Work on the required macros is needed, as well as more experiments with header files just built for code generation (to get beyond the problem of header files that are not independent).


Regards
Manuel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jextract-dev/attachments/20220824/a4f059ef/attachment-0001.htm>

From duke at openjdk.org  Mon Aug 29 08:39:01 2022
From: duke at openjdk.org (Per Minborg)
Date: Mon, 29 Aug 2022 08:39:01 GMT
Subject: RFR: Layout updates
Message-ID: <FpjfqhZ0MMy5wxTuzc2iT-wgTK2eH2i99slPXMcXu80=.198407aa-f6e6-4f58-9d9c-83dc49716396@github.com>

This PR updates jextract to work with the recent changes in Panama.

-------------

Commit messages:
 - Update to reflect recent changes in Panama

Changes: https://git.openjdk.org/jextract/pull/62/files
 Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=62&range=00
  Stats: 18 lines in 4 files changed: 14 ins; 0 del; 4 mod
  Patch: https://git.openjdk.org/jextract/pull/62.diff
  Fetch: git fetch https://git.openjdk.org/jextract pull/62/head:pull/62

PR: https://git.openjdk.org/jextract/pull/62

From sundar at openjdk.org  Mon Aug 29 08:59:17 2022
From: sundar at openjdk.org (Athijegannathan Sundararajan)
Date: Mon, 29 Aug 2022 08:59:17 GMT
Subject: RFR: Layout updates
In-Reply-To: <FpjfqhZ0MMy5wxTuzc2iT-wgTK2eH2i99slPXMcXu80=.198407aa-f6e6-4f58-9d9c-83dc49716396@github.com>
References: <FpjfqhZ0MMy5wxTuzc2iT-wgTK2eH2i99slPXMcXu80=.198407aa-f6e6-4f58-9d9c-83dc49716396@github.com>
Message-ID: <ky6wyoqD3-IbPrpqOCIT6MFPOGGWsmE5sD7hXescXm0=.fcff3300-30ed-42ef-af4e-91316b691466@github.com>

On Mon, 29 Aug 2022 08:31:27 GMT, Per Minborg <duke at openjdk.org> wrote:

> This PR updates jextract to work with the recent changes in Panama.

src/main/java/org/openjdk/jextract/impl/Utils.java line 322:

> 320:     }
> 321: 
> 322:     static String layoutClassNameInDeclaration(MemoryLayout layout) {

Could add a comment here on an internal class and public API interface class?

src/main/java/org/openjdk/jextract/impl/Utils.java line 323:

> 321: 
> 322:     static String layoutClassNameInDeclaration(MemoryLayout layout) {
> 323:         if (layout.getClass().getName().contains(".internal")) {

Could the check be based on interface vs class itself rather than ".internal" in package name of internal class?

-------------

PR: https://git.openjdk.org/jextract/pull/62

From duke at openjdk.org  Mon Aug 29 09:23:42 2022
From: duke at openjdk.org (Per Minborg)
Date: Mon, 29 Aug 2022 09:23:42 GMT
Subject: RFR: Layout updates [v2]
In-Reply-To: <FpjfqhZ0MMy5wxTuzc2iT-wgTK2eH2i99slPXMcXu80=.198407aa-f6e6-4f58-9d9c-83dc49716396@github.com>
References: <FpjfqhZ0MMy5wxTuzc2iT-wgTK2eH2i99slPXMcXu80=.198407aa-f6e6-4f58-9d9c-83dc49716396@github.com>
Message-ID: <vTTDUsqRG37Ubf_5h_vI7b5Qx5iQ0YGpLmJhjQlXmpw=.c14fcbac-d8ff-4f20-929e-7c673766802e@github.com>

> This PR updates jextract to work with the recent changes in Panama.

Per Minborg has updated the pull request incrementally with two additional commits since the last revision:

 - Fix javadoc
 - Update after comments

-------------

Changes:
  - all: https://git.openjdk.org/jextract/pull/62/files
  - new: https://git.openjdk.org/jextract/pull/62/files/1d57d212..56682b5b

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jextract&pr=62&range=01
 - incr: https://webrevs.openjdk.org/?repo=jextract&pr=62&range=00-01

  Stats: 16 lines in 3 files changed: 10 ins; 0 del; 6 mod
  Patch: https://git.openjdk.org/jextract/pull/62.diff
  Fetch: git fetch https://git.openjdk.org/jextract pull/62/head:pull/62

PR: https://git.openjdk.org/jextract/pull/62

From sundar at openjdk.org  Mon Aug 29 09:23:42 2022
From: sundar at openjdk.org (Athijegannathan Sundararajan)
Date: Mon, 29 Aug 2022 09:23:42 GMT
Subject: RFR: Layout updates [v2]
In-Reply-To: <vTTDUsqRG37Ubf_5h_vI7b5Qx5iQ0YGpLmJhjQlXmpw=.c14fcbac-d8ff-4f20-929e-7c673766802e@github.com>
References: <FpjfqhZ0MMy5wxTuzc2iT-wgTK2eH2i99slPXMcXu80=.198407aa-f6e6-4f58-9d9c-83dc49716396@github.com>
 <vTTDUsqRG37Ubf_5h_vI7b5Qx5iQ0YGpLmJhjQlXmpw=.c14fcbac-d8ff-4f20-929e-7c673766802e@github.com>
Message-ID: <apl8JSSmjjZhe8PABoMMvg2BWGDN2QGMNo4UehD-oVc=.bfb497b5-22fa-4242-85b5-730c8e35d61d@github.com>

On Mon, 29 Aug 2022 09:19:54 GMT, Per Minborg <duke at openjdk.org> wrote:

>> This PR updates jextract to work with the recent changes in Panama.
>
> Per Minborg has updated the pull request incrementally with two additional commits since the last revision:
> 
>  - Fix javadoc
>  - Update after comments

LGTM

-------------

Marked as reviewed by sundar (Committer).

PR: https://git.openjdk.org/jextract/pull/62

From duke at openjdk.org  Mon Aug 29 09:23:42 2022
From: duke at openjdk.org (Per Minborg)
Date: Mon, 29 Aug 2022 09:23:42 GMT
Subject: Integrated: Layout updates
In-Reply-To: <FpjfqhZ0MMy5wxTuzc2iT-wgTK2eH2i99slPXMcXu80=.198407aa-f6e6-4f58-9d9c-83dc49716396@github.com>
References: <FpjfqhZ0MMy5wxTuzc2iT-wgTK2eH2i99slPXMcXu80=.198407aa-f6e6-4f58-9d9c-83dc49716396@github.com>
Message-ID: <FG9pf1DASrjqwAj0fd1ck0uT3FGJq2UJf42jXhdHlzU=.c1e2c040-dfb6-4239-8224-bf4daa90b4db@github.com>

On Mon, 29 Aug 2022 08:31:27 GMT, Per Minborg <duke at openjdk.org> wrote:

> This PR updates jextract to work with the recent changes in Panama.

This pull request has now been integrated.

Changeset: d1d3c2c2
Author:    Per Minborg <pminborg at gmail.com>
Committer: Athijegannathan Sundararajan <sundar at openjdk.org>
URL:       https://git.openjdk.org/jextract/commit/d1d3c2c28ff09d9f0347ee2090aff9b4f6577a56
Stats:     28 lines in 4 files changed: 24 ins; 0 del; 4 mod

Layout updates

Reviewed-by: sundar

-------------

PR: https://git.openjdk.org/jextract/pull/62

From sundar at openjdk.org  Mon Aug 29 10:26:39 2022
From: sundar at openjdk.org (Athijegannathan Sundararajan)
Date: Mon, 29 Aug 2022 10:26:39 GMT
Subject: RFR: 7903269: jextract test failures after Layout updates change
Message-ID: <6p_j_Y_URHmth0cdLyUfZbgsHewXHXBmVxt9tVPL9JQ=.92820957-07c4-4417-8d63-dfdb87dbd22c@github.com>

Missed isStruct, isUnion changes in tests

-------------

Commit messages:
 - 7903269: jextract test failures after Layout updates change

Changes: https://git.openjdk.org/jextract/pull/64/files
 Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=64&range=00
  Issue: https://bugs.openjdk.org/browse/CODETOOLS-7903269
  Stats: 11 lines in 3 files changed: 4 ins; 0 del; 7 mod
  Patch: https://git.openjdk.org/jextract/pull/64.diff
  Fetch: git fetch https://git.openjdk.org/jextract pull/64/head:pull/64

PR: https://git.openjdk.org/jextract/pull/64

From manuel.bleichenbacher at gmail.com  Mon Aug 29 11:45:59 2022
From: manuel.bleichenbacher at gmail.com (Manuel Bleichenbacher)
Date: Mon, 29 Aug 2022 13:45:59 +0200
Subject: jextract for operating system API
In-Reply-To: <CAA7F5jJhCQ3pQcmdaR7BX=0XzCR=6n-eSCgEpSjZjJftMrL+5w@mail.gmail.com>
References: <CAA7F5jLH3Mk8ffwT_eWNAy1UywNrSAXgRYp8NJTdt42JhwGkdg@mail.gmail.com>
 <DM6PR10MB2954EC3FC86C45525019878DEA709@DM6PR10MB2954.namprd10.prod.outlook.com>
 <CAA7F5jK+_1J7K2fhdox+1Y6SKvx4JCSW0947Njmzy7ztm_Qt9w@mail.gmail.com>
 <DM6PR10MB29546BD30EC971027E2C87D9EA739@DM6PR10MB2954.namprd10.prod.outlook.com>
 <CAA7F5jJhCQ3pQcmdaR7BX=0XzCR=6n-eSCgEpSjZjJftMrL+5w@mail.gmail.com>
Message-ID: <CAA7F5j+8McXGHndX+_aNmvToa76BveYuCeroTjax=mL+FoPxSg@mail.gmail.com>

 In case you wonder how Xcode resolves the path to framework headers, you
might want to start with the WWDC 18 presentation "Behind the Scenes of the
Xcode Build Process" (https://developer.apple.com/videos/play/wwdc2018/415/),
in particular from 20:16 to 22:06.

The framework resolution is actually a standard clang feature (and Xcode
uses clang). Have a look at the -F and -iframework command line option.

Xcode also uses Clang modules. But as far as I understand, modules are only
used to speed up compilation and are not required for find the header files.

The WWDC notes can be found at https://www.wwdcnotes.com/notes/wwdc18/415/,
the relevant part is the chapter "Clang compiler".

-- Manuel

On Wed, Aug 24, 2022 at 10:31 AM Manuel Bleichenbacher <
manuel.bleichenbacher at gmail.com> wrote:

> Thanks a lot.
>
> If you prioritize the work related to my experiments, please give the
> missing support for macOS frameworks the highest priority.
>
> -- Manuel
>
>
> On Wed, Aug 24, 2022 at 6:34 AM Sundararajan Athijegannathan <
> sundararajan.athijegannathan at oracle.com> wrote:
>
>> Thanks for your report on progress. We'll try to files (I'm not on
>> Windows so it'll take sometime before bug filing).
>>
>> Thanks
>> -Sundar
>> ------------------------------
>> *From:* jextract-dev <jextract-dev-retn at openjdk.org> on behalf of Manuel
>> Bleichenbacher <manuel.bleichenbacher at gmail.com>
>> *Sent:* 24 August 2022 02:01
>> *To:* jextract-dev at openjdk.org <jextract-dev at openjdk.org>
>> *Subject:* Re: jextract for operating system API
>>
>> On Windows, I've made major progress, solved most of the issues and run
>> into a few new ones.
>>
>> The main thing is: It's not possible to directly process a header file
>> like winusb.h. Instead, a helper header file needs to be created:
>>
>> #include <windows.h>
>> #include <winusb.h>
>>
>> A mistake I made yesterday was the omission of "-l Kernel32" (or whatever
>> library is needed).
>>
>> With these two changes and the use of --include-xxx, more or less all
>> functions, structs and macros can be created. (I didn't yet have time to
>> look into GUID constants like GUID_DEVINTERFACE_USB_DEVICE; they probably
>> don't work due to macro trickery.)
>>
>> There are 2 structs that do not work (with rather advanced issues):
>>
>> (1) The generated class _USB_NODE_CONNECTION_INFORMATION_EX cannot be
>> instanciated. The constructor throws the exception:
>> java.lang.UnsupportedOperationException: Invalid alignment requirements for
>> layout s16(DeviceAddress)
>>
>> This struct (like many USB descriptors) is packed with 16-bit integer
>> values on odd offsets. jextract insists on creating a layout with strict
>> alignment rules. So the generated layout
>> net.codecrete.usb.windows.gen.usbioctl._USB_NODE_CONNECTION_INFORMATION_EX#$struct$LAYOUT
>> fails, while the manually created layout
>> net.codecrete.usb.windows.USBIOCtl#USB_NODE_CONNECTION_INFORMATION_EX$Struct
>> works.
>>
>> (2) The struct _SP_DEVICE_INTERFACE_DETAIL_DATA_W isn't usable. In C, it
>> looks like this:
>>
>> typedef struct _SP_DEVICE_INTERFACE_DETAIL_DATA_W {
>>   DWORD cbSize;
>>   WCHAR DevicePath[ANYSIZE_ARRAY];
>> } SP_DEVICE_INTERFACE_DETAIL_DATA_W, *PSP_DEVICE_INTERFACE_DETAIL_DATA_W;
>>
>> It's basically a variable length struct. ANYSIZE_ARRAY is defined as 1.
>> But you are supposed to allocate additional memory for the DevicePath. I
>> guess this is old style trickery that cannot be solved with an automated
>> tool.
>>
>> Minor annoyances are:
>>
>> - jextract on Windows is a batch file that never returns. So it's not
>> possible to call jextract multiple times for a single batch file.
>> - It would be nicer if structs used the nice typedef name instead of the
>> ugly struct name, e.g. SP_DEVICE_INTERFACE_DATA_W instead of
>> _SP_DEVICE_INTERFACE_DATA_W (note the leading underscore).
>>
>> Overall: success on Windows.
>>
>> Regards
>> Manuel
>>
>> On Tue, Aug 23, 2022 at 5:48 AM Sundararajan Athijegannathan <
>> sundararajan.athijegannathan at oracle.com> wrote:
>>
>> Thanks for your experiments and comments. I'll go over and file bug(s) as
>> needed.
>>
>> -Sundar
>> ------------------------------
>> *From:* jextract-dev <jextract-dev-retn at openjdk.org> on behalf of Manuel
>> Bleichenbacher <manuel.bleichenbacher at gmail.com>
>> *Sent:* 23 August 2022 02:24
>> *To:* jextract-dev at openjdk.org <jextract-dev at openjdk.org>
>> *Subject:* jextract for operating system API
>>
>> Hi everybody
>>
>> As promised, I've compiled my findings when using jextract for the Java
>> Does USB project, which uses the foreign function & memory API to
>> communicate with USB devices on Windows, macOS and Linux. After initial
>> difficulties with jextract, I've written all the layouts, method and
>> variable handles manually. But now I've given jextract another try. The
>> commands I've used can be found in in the "jextract" branch of the project
>> at
>> https://github.com/manuelbl/JavaDoesUSB/tree/jextract/java-does-usb/jextract
>>
>> Please note that I think that automatically generating code for some of
>> the old Windows and POSIX APIs is almost impossible. My use of jextract
>> (for OS APIs) is probably far from the main use with third-party libraries.
>> And I might also have made mistakes, or there might be a simple solution to
>> my issues. See my conclusions below for areas where I think jextract has
>> still potential.
>>
>>
>> Linux:
>>
>> Overall, I was able to generate most of what I needed with jextract. The
>> issues were:
>>
>> - Generating code for libsystemd (sd-device.h) failed with "unknown type
>> name 'intmax_t'". I think this should be defined by inttypes.h, which is
>> included. My guess is that clang headers and Linux headers are mixed up.
>> - When generating code for usbdevice_fs.h, it didn't generate the
>> constants for USBDEVFS_CLAIMINTERFACE and similar ones. The reason might be
>> that the defines look like functions even though they evaluate to a
>> constant value.
>>
>>
>> macOS:
>>
>> Major parts of macOS are provided in the form of frameworks and they are
>> treated in a special way, both when it comes to the header files and
>> dynamic libraries. As an example, my library uses several functions from
>> the CoreFoundation framework. In C/CC++, you can simply include them like
>> so:
>>
>> #include <CoreFoundation/CoreFoundation.h>
>>
>> That's surprising as the file actually resides in
>> /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/CoreFoundation.framework/Versions/A/Headers/CoreFoundation.h.
>>
>> Note that it is in a directory called "Headers" and not in
>> "CoreFoundation" as the include statement suggests. The relative path
>> "CoreFoundation/CoreFoundation.h" does not exist at all.
>>
>> This wouldn't be a big problem if this was just the entry point. But this
>> pattern is used recursively. CoreFoundation.h includes about 30 other
>> header files like this, all with a relative path that does not exist.
>>
>> Xcode magically resolves this. But jextract seems to be missing this
>> magic. So it cannot be used for any macOS frameworks. In other words, it
>> cannot be for the major part of the macOS APIs. My library uses a single
>> non-framework functions. So using jextract for macOS is pointless without
>> resolving this issue.
>>
>>
>> Windows:
>>
>> On Windows I spent the most time -- with no success at all. I only
>> partially understand the problems. Here is what I've encountered:
>>
>> The first simple goal was to have code for CloseHandle() generated. I
>> tried it three different ways:
>>
>>    - Generating code for handleapi.h generated 1000 classes. Code for
>>    CloseHandle() was generated in 2 classes, but none of them was publicly
>>    available.
>>    - Generating code for handleapi.h restricted to the function
>>    CloseHandle() generated a small number of class with a public static method
>>    for CloseHandle(). However, at run-time the method produces the error
>>    "java.lang.UnsatisfiedLinkError: unresolved symbol: CloseHandle"
>>    - Generating code for windows.h restricted to CloseHandle() also
>>    resulted in the UnsatisfiedLinkError.
>>
>> I then tried to generate code for Winusb.h. Unfortunately, this header
>> file is not independent. It expects that you have already processed other
>> header files before it. So it fails with "error: unknown type name
>> 'LARGE_INTEGER'". This behavior with be reproduced in Visual Studio.
>>
>> Next up was Cfgmgr32.h. It also doesn't seem to be independent and fails
>> with "error: unknown type name 'ULONG'". Setupapi.h and Usbioctl.h behave
>> similarly.
>>
>> Windows was the only system where I had problems with preprocessor
>> macros. The macro _M_AMD64=100 is certainly needed. That got me further
>> than without it. And I've defined two Unicode macros, copied from the
>> Visual Studio project. But the macros might still be the source of some
>> problems.
>>
>>
>>
>> Conclusion:
>>
>> At the moment, jextract doesn't help much for accessing OS APIs.
>>
>> On Linux, the "unknown type name 'intmax_t'" is worth investigating. It
>> could be helpful for many APIs.
>>
>> On macOS, it's all about the frameworks and the magic behind resolving
>> the include path. If it can be made to work, it hopefully opens up the
>> world of framework (plus some challenges that are still hidden).
>>
>> On Windows, more investigation is needed. Work on the required macros is
>> needed, as well as more experiments with header files just built for code
>> generation (to get beyond the problem of header files that are not
>> independent).
>>
>>
>> Regards
>> Manuel
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jextract-dev/attachments/20220829/136dbfe9/attachment-0001.htm>

From duke at openjdk.org  Mon Aug 29 11:46:15 2022
From: duke at openjdk.org (Per Minborg)
Date: Mon, 29 Aug 2022 11:46:15 GMT
Subject: RFR: 7903269: jextract test failures after Layout updates change
In-Reply-To: <6p_j_Y_URHmth0cdLyUfZbgsHewXHXBmVxt9tVPL9JQ=.92820957-07c4-4417-8d63-dfdb87dbd22c@github.com>
References: <6p_j_Y_URHmth0cdLyUfZbgsHewXHXBmVxt9tVPL9JQ=.92820957-07c4-4417-8d63-dfdb87dbd22c@github.com>
Message-ID: <9QnQpc35zzUtCFtvByywJgbBFGlYmR3Blqkq9WfnbqM=.89df63c6-bca4-42ba-9417-287707a66420@github.com>

On Mon, 29 Aug 2022 10:20:36 GMT, Athijegannathan Sundararajan <sundar at openjdk.org> wrote:

> Missed isStruct, isUnion changes in tests

LGTM

-------------

PR: https://git.openjdk.org/jextract/pull/64

From sundar at openjdk.org  Mon Aug 29 11:48:28 2022
From: sundar at openjdk.org (Athijegannathan Sundararajan)
Date: Mon, 29 Aug 2022 11:48:28 GMT
Subject: Integrated: 7903269: jextract test failures after Layout updates
 change
In-Reply-To: <6p_j_Y_URHmth0cdLyUfZbgsHewXHXBmVxt9tVPL9JQ=.92820957-07c4-4417-8d63-dfdb87dbd22c@github.com>
References: <6p_j_Y_URHmth0cdLyUfZbgsHewXHXBmVxt9tVPL9JQ=.92820957-07c4-4417-8d63-dfdb87dbd22c@github.com>
Message-ID: <7hixRSCs75SZ47YQvY4nz83KjFXfxXQ43WooE8algYU=.c515314c-29e8-45ee-9690-219da52347ed@github.com>

On Mon, 29 Aug 2022 10:20:36 GMT, Athijegannathan Sundararajan <sundar at openjdk.org> wrote:

> Missed isStruct, isUnion changes in tests

This pull request has now been integrated.

Changeset: 297fed99
Author:    Athijegannathan Sundararajan <sundar at openjdk.org>
URL:       https://git.openjdk.org/jextract/commit/297fed990f778be98c77c493477cba5a2f47abc3
Stats:     11 lines in 3 files changed: 4 ins; 0 del; 7 mod

7903269: jextract test failures after Layout updates change

-------------

PR: https://git.openjdk.org/jextract/pull/64

From duke at openjdk.org  Tue Aug 30 07:35:33 2022
From: duke at openjdk.org (Per Minborg)
Date: Tue, 30 Aug 2022 07:35:33 GMT
Subject: RFR: Make some classes/fields final and add privat ctr
Message-ID: <8BLTnXd3IDnf1HHyAPrrtIxYqT5fpPopMXuN8l8vnqI=.124a61a2-418b-4c82-9497-69ee89935b90@github.com>

This PR makes some of the classes and fields `final`. Also, a general cleanup of the `RuntimeHelper` class is proposed.

There are some files that are "overhidden" (by attempting to "override"/hide static methods). This will be fixed in a separate PR and so these classes are not affected in this PR.

-------------

Commit messages:
 - Make some classes/fields final and add privat ctr

Changes: https://git.openjdk.org/jextract/pull/65/files
 Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=65&range=00
  Stats: 99 lines in 6 files changed: 49 ins; 19 del; 31 mod
  Patch: https://git.openjdk.org/jextract/pull/65.diff
  Fetch: git fetch https://git.openjdk.org/jextract pull/65/head:pull/65

PR: https://git.openjdk.org/jextract/pull/65

From duke at openjdk.org  Tue Aug 30 07:56:28 2022
From: duke at openjdk.org (Per Minborg)
Date: Tue, 30 Aug 2022 07:56:28 GMT
Subject: RFR: 7903270: Util classes should be final and have a private
 constructor [v2]
In-Reply-To: <8BLTnXd3IDnf1HHyAPrrtIxYqT5fpPopMXuN8l8vnqI=.124a61a2-418b-4c82-9497-69ee89935b90@github.com>
References: <8BLTnXd3IDnf1HHyAPrrtIxYqT5fpPopMXuN8l8vnqI=.124a61a2-418b-4c82-9497-69ee89935b90@github.com>
Message-ID: <QL8D3c0FaLR2IvoLKtTpxEiVnAz1kYaA7a4p60CaKGs=.ab67a94c-6247-4d0f-b65b-d01b54d084f2@github.com>

> This PR makes some of the classes and fields `final`. Also, a general cleanup of the `RuntimeHelper` class is proposed.
> 
> There are some files that are "overhidden" (by attempting to "override"/hide static methods). This will be fixed in a separate PR and so these classes are not affected in this PR.

Per Minborg has updated the pull request incrementally with two additional commits since the last revision:

 - Make constants package private
 - Remove debug output

-------------

Changes:
  - all: https://git.openjdk.org/jextract/pull/65/files
  - new: https://git.openjdk.org/jextract/pull/65/files/9dcd3e3a..2a7fe21f

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jextract&pr=65&range=01
 - incr: https://webrevs.openjdk.org/?repo=jextract&pr=65&range=00-01

  Stats: 9 lines in 2 files changed: 5 ins; 3 del; 1 mod
  Patch: https://git.openjdk.org/jextract/pull/65.diff
  Fetch: git fetch https://git.openjdk.org/jextract pull/65/head:pull/65

PR: https://git.openjdk.org/jextract/pull/65

From duke at openjdk.org  Tue Aug 30 08:05:18 2022
From: duke at openjdk.org (Per Minborg)
Date: Tue, 30 Aug 2022 08:05:18 GMT
Subject: RFR: 7903270: Util classes should be final and have a private
 constructor [v3]
In-Reply-To: <8BLTnXd3IDnf1HHyAPrrtIxYqT5fpPopMXuN8l8vnqI=.124a61a2-418b-4c82-9497-69ee89935b90@github.com>
References: <8BLTnXd3IDnf1HHyAPrrtIxYqT5fpPopMXuN8l8vnqI=.124a61a2-418b-4c82-9497-69ee89935b90@github.com>
Message-ID: <E0CFrINqbdBSE2KXNf7hPN9V8Blr180FownQixgUhAc=.b4c14372-368d-496d-8d49-3453c6a42d38@github.com>

> This PR makes some of the classes and fields `final`. Also, a general cleanup of the `RuntimeHelper` class is proposed.
> 
> There are some files that are "overhidden" (by attempting to "override"/hide static methods). This will be fixed in a separate PR and so these classes are not affected in this PR.

Per Minborg has updated the pull request incrementally with one additional commit since the last revision:

  Make Constants package private

-------------

Changes:
  - all: https://git.openjdk.org/jextract/pull/65/files
  - new: https://git.openjdk.org/jextract/pull/65/files/2a7fe21f..5a070770

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jextract&pr=65&range=02
 - incr: https://webrevs.openjdk.org/?repo=jextract&pr=65&range=01-02

  Stats: 5 lines in 1 file changed: 5 ins; 0 del; 0 mod
  Patch: https://git.openjdk.org/jextract/pull/65.diff
  Fetch: git fetch https://git.openjdk.org/jextract pull/65/head:pull/65

PR: https://git.openjdk.org/jextract/pull/65

From duke at openjdk.org  Tue Aug 30 08:55:44 2022
From: duke at openjdk.org (Per Minborg)
Date: Tue, 30 Aug 2022 08:55:44 GMT
Subject: RFR: 7903270: Util classes should be final and have a private
 constructor [v4]
In-Reply-To: <8BLTnXd3IDnf1HHyAPrrtIxYqT5fpPopMXuN8l8vnqI=.124a61a2-418b-4c82-9497-69ee89935b90@github.com>
References: <8BLTnXd3IDnf1HHyAPrrtIxYqT5fpPopMXuN8l8vnqI=.124a61a2-418b-4c82-9497-69ee89935b90@github.com>
Message-ID: <TuJE-R3kddX9oK6ITItNH0UjwVFho3dC-4yZP9MBuqs=.93ae091c-ca5c-494c-b8f4-6874536325b2@github.com>

> This PR makes some of the classes and fields `final`. Also, a general cleanup of the `RuntimeHelper` class is proposed.
> 
> There are some files that are "overhidden" (by attempting to "override"/hide static methods). This will be fixed in a separate PR and so these classes are not affected in this PR.

Per Minborg has updated the pull request incrementally with one additional commit since the last revision:

  Add option to show generating class

-------------

Changes:
  - all: https://git.openjdk.org/jextract/pull/65/files
  - new: https://git.openjdk.org/jextract/pull/65/files/5a070770..29ce89c2

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jextract&pr=65&range=03
 - incr: https://webrevs.openjdk.org/?repo=jextract&pr=65&range=02-03

  Stats: 30 lines in 2 files changed: 21 ins; 7 del; 2 mod
  Patch: https://git.openjdk.org/jextract/pull/65.diff
  Fetch: git fetch https://git.openjdk.org/jextract pull/65/head:pull/65

PR: https://git.openjdk.org/jextract/pull/65

From duke at openjdk.org  Tue Aug 30 09:00:30 2022
From: duke at openjdk.org (Per Minborg)
Date: Tue, 30 Aug 2022 09:00:30 GMT
Subject: RFR: 7903270: Util classes should be final and have a private
 constructor [v4]
In-Reply-To: <TuJE-R3kddX9oK6ITItNH0UjwVFho3dC-4yZP9MBuqs=.93ae091c-ca5c-494c-b8f4-6874536325b2@github.com>
References: <8BLTnXd3IDnf1HHyAPrrtIxYqT5fpPopMXuN8l8vnqI=.124a61a2-418b-4c82-9497-69ee89935b90@github.com>
 <TuJE-R3kddX9oK6ITItNH0UjwVFho3dC-4yZP9MBuqs=.93ae091c-ca5c-494c-b8f4-6874536325b2@github.com>
Message-ID: <m2H_MY1QiKCfckglerRchi6mZTvcJep8xNr6tCAzYv8=.41300adf-5d6a-4433-8f8d-74a0f839f690@github.com>

On Tue, 30 Aug 2022 08:55:44 GMT, Per Minborg <duke at openjdk.org> wrote:

>> This PR makes some of the classes and fields `final`. Also, a general cleanup of the `RuntimeHelper` class is proposed.
>> 
>> There are some files that are "overhidden" (by attempting to "override"/hide static methods). This will be fixed in a separate PR and so these classes are not affected in this PR.
>
> Per Minborg has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Add option to show generating class

src/main/java/org/openjdk/jextract/impl/ClassSourceBuilder.java line 202:

> 200:             assert packageName().indexOf('/') == -1 : "package name invalid: " + packageName();
> 201:             append("// Generated by jextract");
> 202:             if (SHOW_GENERATING_CLASS) {

This is very useful when figuring out how a class is generated.

-------------

PR: https://git.openjdk.org/jextract/pull/65

From duke at openjdk.org  Tue Aug 30 13:30:40 2022
From: duke at openjdk.org (Per Minborg)
Date: Tue, 30 Aug 2022 13:30:40 GMT
Subject: RFR: 7903271: Add sample of libzstd
Message-ID: <KWwMcU_Zv1WJ1d_Jey5PZ67GhsNhvIv2F4AL1m3juWU=.9d7860af-77e5-42a9-9b6b-e59836235d74@github.com>

This PR adds a sample using `libzstd`, a high-performance compression library.

-------------

Commit messages:
 - Add sample of libzstd

Changes: https://git.openjdk.org/jextract/pull/66/files
 Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=66&range=00
  Issue: https://bugs.openjdk.org/browse/CODETOOLS-7903271
  Stats: 158 lines in 8 files changed: 158 ins; 0 del; 0 mod
  Patch: https://git.openjdk.org/jextract/pull/66.diff
  Fetch: git fetch https://git.openjdk.org/jextract pull/66/head:pull/66

PR: https://git.openjdk.org/jextract/pull/66

From duke at openjdk.org  Tue Aug 30 13:35:33 2022
From: duke at openjdk.org (Per Minborg)
Date: Tue, 30 Aug 2022 13:35:33 GMT
Subject: RFR: 7903271: Add sample of libzstd [v2]
In-Reply-To: <KWwMcU_Zv1WJ1d_Jey5PZ67GhsNhvIv2F4AL1m3juWU=.9d7860af-77e5-42a9-9b6b-e59836235d74@github.com>
References: <KWwMcU_Zv1WJ1d_Jey5PZ67GhsNhvIv2F4AL1m3juWU=.9d7860af-77e5-42a9-9b6b-e59836235d74@github.com>
Message-ID: <8EVLvTTBTeHTKCOuPGVKy9BLAZ3qiyKQwXLbhKeodM0=.c2df2202-c2d0-4054-a525-b18e74550f1e@github.com>

> This PR adds a sample using `libzstd`, a high-performance compression library.

Per Minborg has updated the pull request incrementally with one additional commit since the last revision:

  Simplify compile

-------------

Changes:
  - all: https://git.openjdk.org/jextract/pull/66/files
  - new: https://git.openjdk.org/jextract/pull/66/files/4aae1ab3..1b1e3bc7

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jextract&pr=66&range=01
 - incr: https://webrevs.openjdk.org/?repo=jextract&pr=66&range=00-01

  Stats: 3 lines in 2 files changed: 0 ins; 2 del; 1 mod
  Patch: https://git.openjdk.org/jextract/pull/66.diff
  Fetch: git fetch https://git.openjdk.org/jextract pull/66/head:pull/66

PR: https://git.openjdk.org/jextract/pull/66

From duke at openjdk.org  Tue Aug 30 13:49:24 2022
From: duke at openjdk.org (Per Minborg)
Date: Tue, 30 Aug 2022 13:49:24 GMT
Subject: RFR: 7903271: Add sample of libzstd [v3]
In-Reply-To: <KWwMcU_Zv1WJ1d_Jey5PZ67GhsNhvIv2F4AL1m3juWU=.9d7860af-77e5-42a9-9b6b-e59836235d74@github.com>
References: <KWwMcU_Zv1WJ1d_Jey5PZ67GhsNhvIv2F4AL1m3juWU=.9d7860af-77e5-42a9-9b6b-e59836235d74@github.com>
Message-ID: <gg5hxzG0a8bd0a1PjWlAtDtC5JEjTW3Mx_jb-hD9NuE=.b62e2953-46ce-4bd6-a31e-5a2228026259@github.com>

> This PR adds a sample using `libzstd`, a high-performance compression library.

Per Minborg has updated the pull request incrementally with one additional commit since the last revision:

  Update copyright year

-------------

Changes:
  - all: https://git.openjdk.org/jextract/pull/66/files
  - new: https://git.openjdk.org/jextract/pull/66/files/1b1e3bc7..e0715993

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jextract&pr=66&range=02
 - incr: https://webrevs.openjdk.org/?repo=jextract&pr=66&range=01-02

  Stats: 1 line in 1 file changed: 0 ins; 0 del; 1 mod
  Patch: https://git.openjdk.org/jextract/pull/66.diff
  Fetch: git fetch https://git.openjdk.org/jextract pull/66/head:pull/66

PR: https://git.openjdk.org/jextract/pull/66

From sundar at openjdk.org  Tue Aug 30 13:49:24 2022
From: sundar at openjdk.org (Athijegannathan Sundararajan)
Date: Tue, 30 Aug 2022 13:49:24 GMT
Subject: RFR: 7903271: Add sample of libzstd [v3]
In-Reply-To: <gg5hxzG0a8bd0a1PjWlAtDtC5JEjTW3Mx_jb-hD9NuE=.b62e2953-46ce-4bd6-a31e-5a2228026259@github.com>
References: <KWwMcU_Zv1WJ1d_Jey5PZ67GhsNhvIv2F4AL1m3juWU=.9d7860af-77e5-42a9-9b6b-e59836235d74@github.com>
 <gg5hxzG0a8bd0a1PjWlAtDtC5JEjTW3Mx_jb-hD9NuE=.b62e2953-46ce-4bd6-a31e-5a2228026259@github.com>
Message-ID: <kSzeq8UWhX4FYPaZ7TCwKb6vk4Ziir25cVw8jT4kGfY=.66a1c67f-db35-495f-89ed-583a7a42dc0b@github.com>

On Tue, 30 Aug 2022 13:46:05 GMT, Per Minborg <duke at openjdk.org> wrote:

>> This PR adds a sample using `libzstd`, a high-performance compression library.
>
> Per Minborg has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Update copyright year

This is cool! LGTM

-------------

Marked as reviewed by sundar (Committer).

PR: https://git.openjdk.org/jextract/pull/66

From duke at openjdk.org  Tue Aug 30 13:50:36 2022
From: duke at openjdk.org (Per Minborg)
Date: Tue, 30 Aug 2022 13:50:36 GMT
Subject: Integrated: 7903271: Add sample of libzstd
In-Reply-To: <KWwMcU_Zv1WJ1d_Jey5PZ67GhsNhvIv2F4AL1m3juWU=.9d7860af-77e5-42a9-9b6b-e59836235d74@github.com>
References: <KWwMcU_Zv1WJ1d_Jey5PZ67GhsNhvIv2F4AL1m3juWU=.9d7860af-77e5-42a9-9b6b-e59836235d74@github.com>
Message-ID: <090m6ejeovqSgoKHao-DWsVJYvL0P2yteX16jey-NO8=.f3321eb8-2809-4c82-9bc9-6f2f7f6b4df6@github.com>

On Tue, 30 Aug 2022 13:25:08 GMT, Per Minborg <duke at openjdk.org> wrote:

> This PR adds a sample using `libzstd`, a high-performance compression library.

This pull request has now been integrated.

Changeset: dae7fac5
Author:    Per Minborg <pminborg at gmail.com>
Committer: Athijegannathan Sundararajan <sundar at openjdk.org>
URL:       https://git.openjdk.org/jextract/commit/dae7fac5c049c006a42f9427be3038dddcb41e39
Stats:     156 lines in 8 files changed: 156 ins; 0 del; 0 mod

7903271: Add sample of libzstd

Reviewed-by: sundar

-------------

PR: https://git.openjdk.org/jextract/pull/66

From duke at openjdk.org  Tue Aug 30 14:48:02 2022
From: duke at openjdk.org (Per Minborg)
Date: Tue, 30 Aug 2022 14:48:02 GMT
Subject: RFR: 7903272: Make method handle methods private
Message-ID: <mYUq9zR5STIXSxNp5l6FZdqdCMIhoxbWnY0A7p7mO_I=.de91e0a2-d87e-43be-9287-91449ea34571@github.com>

xxxx$MH methods should be declared private to reduce cluttering in end users' IDEs and elsewhere.
 
 This PR also solves esthetic issues in the generated code.

-------------

Commit messages:
 - Make method handle methods private

Changes: https://git.openjdk.org/jextract/pull/67/files
 Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=67&range=00
  Issue: https://bugs.openjdk.org/browse/CODETOOLS-7903272
  Stats: 3 lines in 1 file changed: 0 ins; 0 del; 3 mod
  Patch: https://git.openjdk.org/jextract/pull/67.diff
  Fetch: git fetch https://git.openjdk.org/jextract pull/67/head:pull/67

PR: https://git.openjdk.org/jextract/pull/67

From mduigou at openjdk.org  Tue Aug 30 17:08:23 2022
From: mduigou at openjdk.org (Mike Duigou)
Date: Tue, 30 Aug 2022 17:08:23 GMT
Subject: RFR: 7903271: Add sample of libzstd [v3]
In-Reply-To: <gg5hxzG0a8bd0a1PjWlAtDtC5JEjTW3Mx_jb-hD9NuE=.b62e2953-46ce-4bd6-a31e-5a2228026259@github.com>
References: <KWwMcU_Zv1WJ1d_Jey5PZ67GhsNhvIv2F4AL1m3juWU=.9d7860af-77e5-42a9-9b6b-e59836235d74@github.com>
 <gg5hxzG0a8bd0a1PjWlAtDtC5JEjTW3Mx_jb-hD9NuE=.b62e2953-46ce-4bd6-a31e-5a2228026259@github.com>
Message-ID: <h1DyNQ4JWZ8tGwYI0mzZqchSS7yQgAvN4H1K0VuFD3Q=.ac461ba5-fcf8-45f8-b0c0-f4d776b1c57d@github.com>

On Tue, 30 Aug 2022 13:49:24 GMT, Per Minborg <duke at openjdk.org> wrote:

>> This PR adds a sample using `libzstd`, a high-performance compression library.
>
> Per Minborg has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Update copyright year

Thank you for contributing this example!

-------------

PR: https://git.openjdk.org/jextract/pull/66

From mduigou at openjdk.org  Tue Aug 30 17:23:34 2022
From: mduigou at openjdk.org (Mike Duigou)
Date: Tue, 30 Aug 2022 17:23:34 GMT
Subject: RFR: 7903271: Add sample of libzstd [v3]
In-Reply-To: <gg5hxzG0a8bd0a1PjWlAtDtC5JEjTW3Mx_jb-hD9NuE=.b62e2953-46ce-4bd6-a31e-5a2228026259@github.com>
References: <KWwMcU_Zv1WJ1d_Jey5PZ67GhsNhvIv2F4AL1m3juWU=.9d7860af-77e5-42a9-9b6b-e59836235d74@github.com>
 <gg5hxzG0a8bd0a1PjWlAtDtC5JEjTW3Mx_jb-hD9NuE=.b62e2953-46ce-4bd6-a31e-5a2228026259@github.com>
Message-ID: <CIOIuQkKrtOM_9Mlrem7-4Sf599x90tHXe7iWhia2XU=.0c0162d8-bbe0-4864-9aff-a440721d1d8a@github.com>

On Tue, 30 Aug 2022 13:49:24 GMT, Per Minborg <duke at openjdk.org> wrote:

>> This PR adds a sample using `libzstd`, a high-performance compression library.
>
> Per Minborg has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Update copyright year

samples/libzstd/README line 1:

> 1: * Install libffmpeg on Mac using homebrew

This refers to libffmpeg rather than zstd.

samples/libzstd/README line 1:

> 1: * Install libffmpeg on Mac using homebrew

You could also suggest installing with system package manager (yum, apt, ?)

samples/libzstd/compile.sh line 2:

> 1: jextract -t libzstd \
> 2:   -I /usr/local/Cellar/zstd/1.5.2/include \

Consider replacing the path `/usr/local/Cellar/zstd/1.5.2` with `$(brew --prefix zstd)` or `$(pkg-config --variable=prefix libzstd)`

samples/libzstd/compile.sh line 5:

> 3:   -l zstd \
> 4:   --header-class-name Libzstd \
> 5:     /usr/local/Cellar/zstd/1.5.2/include/zstd.h

Same comment about using non-static prefix path.

samples/libzstd/compilesource.sh line 2:

> 1: jextract --source -t libzstd \
> 2:   -I /usr/local/Cellar/zstd/1.5.2/include \

Same comments for zstd prefix path.

-------------

PR: https://git.openjdk.org/jextract/pull/66

From manuel.bleichenbacher at gmail.com  Tue Aug 30 18:47:06 2022
From: manuel.bleichenbacher at gmail.com (Manuel Bleichenbacher)
Date: Tue, 30 Aug 2022 20:47:06 +0200
Subject: SymbolLookup.loaderLookup() vs SymbolLookup.libraryLookup()
Message-ID: <CAA7F5jJ73k1L4igZ4pBMYgpsnwyv2qRh=8GQgrKJfuvyeX=Mng@mail.gmail.com>

If code is created for libudev (on Linux) with the  command:

    jextract --source -l udev --include-function udev_new
/usr/include/libudev.h

the resulting code fails at run-time with:

java.lang.UnsatisfiedLinkError: no udev in java.library.path:
/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib

That's of course correct as the libudev is in /usr/lib/x86_64-linux-gnu.
But it isn't just a random application specific directory. It's a directory
that is searched for libraries by default (depending on the architecture
the name changes) and contains many basic libraries for Linux.

The generated code can be fixed easily with a minor modification. Instead
of the generated code (in RuntimeHelper.java):

        System.loadLibrary("udev");
        SymbolLookup loaderLookup = SymbolLookup.loaderLookup();

one can just put:

        SymbolLookup loaderLookup =
SymbolLookup.libraryLookup("libudev.so", MemorySession.openImplicit());

Would it be possible to add an option to jextract so it generates code
using SymbolLookup.libraryLookup() instead of SymbolLookup.loaderLookup()
and does not generate a call to System.loadLibrary()?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/jextract-dev/attachments/20220830/c6a8a5ad/attachment.htm>

From duke at openjdk.org  Wed Aug 31 07:20:29 2022
From: duke at openjdk.org (Per Minborg)
Date: Wed, 31 Aug 2022 07:20:29 GMT
Subject: RFR: 7903271: Add sample of libzstd [v3]
In-Reply-To: <h1DyNQ4JWZ8tGwYI0mzZqchSS7yQgAvN4H1K0VuFD3Q=.ac461ba5-fcf8-45f8-b0c0-f4d776b1c57d@github.com>
References: <KWwMcU_Zv1WJ1d_Jey5PZ67GhsNhvIv2F4AL1m3juWU=.9d7860af-77e5-42a9-9b6b-e59836235d74@github.com>
 <gg5hxzG0a8bd0a1PjWlAtDtC5JEjTW3Mx_jb-hD9NuE=.b62e2953-46ce-4bd6-a31e-5a2228026259@github.com>
 <h1DyNQ4JWZ8tGwYI0mzZqchSS7yQgAvN4H1K0VuFD3Q=.ac461ba5-fcf8-45f8-b0c0-f4d776b1c57d@github.com>
Message-ID: <EJPZEtVi6leWuw-7owIa5zrBBknl0KHyRQaW6uYIz60=.eabc4691-06e6-4233-81ff-78c172791951@github.com>

On Tue, 30 Aug 2022 17:04:54 GMT, Mike Duigou <mduigou at openjdk.org> wrote:

>> Per Minborg has updated the pull request incrementally with one additional commit since the last revision:
>> 
>>   Update copyright year
>
> Thank you for contributing this example!

Thanks @bondolo for your feedback. Please review https://github.com/openjdk/jextract/pull/68 where I have incorporated your suggestions above.

-------------

PR: https://git.openjdk.org/jextract/pull/66

From duke at openjdk.org  Wed Aug 31 07:23:06 2022
From: duke at openjdk.org (Per Minborg)
Date: Wed, 31 Aug 2022 07:23:06 GMT
Subject: RFR: Add path independence
Message-ID: <sF8Wi82un7AW3micwsMGbspGDZyXEFQpgMiyTav5XkM=.05edda59-c34a-49cf-a51e-4448c9002cc7@github.com>

This small PR incorporated feedback from the original PR: https://github.com/openjdk/jextract/pull/66 made after it was integrated.

-------------

Commit messages:
 - Add path independence

Changes: https://git.openjdk.org/jextract/pull/68/files
 Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=68&range=00
  Stats: 10 lines in 4 files changed: 2 ins; 0 del; 8 mod
  Patch: https://git.openjdk.org/jextract/pull/68.diff
  Fetch: git fetch https://git.openjdk.org/jextract pull/68/head:pull/68

PR: https://git.openjdk.org/jextract/pull/68