From mcimadamore at openjdk.org Thu Jun 1 15:58:37 2023 From: mcimadamore at openjdk.org (Maurizio Cimadamore) Date: Thu, 1 Jun 2023 15:58:37 GMT Subject: RFR: 7903485 Windows.h fails to extract on jextract/panama Message-ID: This patch fixes a number of issues I discovered while trying to extract Windows.h on a recent jextract: * the size of `long double` layout is wrong. This means that if a struct with a field of that type is declared, StructLayoutComputer will fail (as it will see a size != expected size reported by clang) * There are cases where, also for unsupported layouts, we end up creating function descriptors with things containing padding. This is caused by the fact that the check for unsupported layouts in functions is performed *after* the creation of the function descriptor. * Sometimes libclang reports field cursors in an out-of-order fashion. I have not been able to pinpoint the exact cause, as all my attempts to create a reduced test case failed. When this behavior occurs, jextract ends up generating extra fields and padding. I believe this behavior has always been there, but now brought to the fore by the new eager checks. * A change in behavior of recent libclang causes `cursor.spelling()` to return non-empty strings (see https://github.com/llvm/llvm-project/issues/58429) * A quality of life fix, to generate "sensible" warnings when variadic callbacks are found ------------- Commit messages: - Strengthen TypeImpl - Make TypeImpl return empty optional for non-sensical layouts - Initial push Changes: https://git.openjdk.org/jextract/pull/122/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=122&range=00 Issue: https://bugs.openjdk.org/browse/CODETOOLS-7903485 Stats: 65 lines in 8 files changed: 31 ins; 19 del; 15 mod Patch: https://git.openjdk.org/jextract/pull/122.diff Fetch: git fetch https://git.openjdk.org/jextract.git pull/122/head:pull/122 PR: https://git.openjdk.org/jextract/pull/122 From mcimadamore at openjdk.org Thu Jun 1 15:58:37 2023 From: mcimadamore at openjdk.org (Maurizio Cimadamore) Date: Thu, 1 Jun 2023 15:58:37 GMT Subject: RFR: 7903485 Windows.h fails to extract on jextract/panama In-Reply-To: References: Message-ID: <8mSWfMtFInFFgLHgIGIyAfXxADxRj_JQJYtcQIizoS0=.afb46343-ca7e-4751-80d0-fd661efd1859@github.com> On Thu, 1 Jun 2023 15:15:56 GMT, Maurizio Cimadamore wrote: > This patch fixes a number of issues I discovered while trying to extract Windows.h on a recent jextract: > > * the size of `long double` layout is wrong. This means that if a struct with a field of that type is declared, StructLayoutComputer will fail (as it will see a size != expected size reported by clang) > * There are cases where, also for unsupported layouts, we end up creating function descriptors with things containing padding. This is caused by the fact that the check for unsupported layouts in functions is performed *after* the creation of the function descriptor. > * Sometimes libclang reports field cursors in an out-of-order fashion. I have not been able to pinpoint the exact cause, as all my attempts to create a reduced test case failed. When this behavior occurs, jextract ends up generating extra fields and padding. I believe this behavior has always been there, but now brought to the fore by the new eager checks. > * A change in behavior of recent libclang causes `cursor.spelling()` to return non-empty strings (see https://github.com/llvm/llvm-project/issues/58429) > * A quality of life fix, to generate "sensible" warnings when variadic callbacks are found Note - I did not add new tests. For some of the stuff I could reproduce (e.g. failure when creating function descriptor, or issues with `long double` size) we already have tests (see LibUnsupportedTest), and I suspect these tests fail w/o this patch (but I have not tried). Similarly, I believe that some tests will fail when running against a recent version of libclang, due to the cursor.spelling() changes. There is no new test for the out of order issue with record fields (as I have no reproducer, unfortunately). src/main/java/org/openjdk/jextract/impl/OutputFactory.java line 225: > 223: //generate functional interface > 224: if (func.varargs() && !func.argumentTypes().isEmpty()) { > 225: warn("varargs in callbacks is not supported: " + CDeclarationPrinter.declaration(func, javaName)); This is a quality of life fix - on Windows, a very very big message is reported as we print out the function descriptor of the corresponding C function. By turning that into a C type (using same logic we use for javadoc generation) the warning messages is more amenable. ------------- PR Comment: https://git.openjdk.org/jextract/pull/122#issuecomment-1572312222 PR Review Comment: https://git.openjdk.org/jextract/pull/122#discussion_r1213343668 From jvernee at openjdk.org Wed Jun 7 14:34:23 2023 From: jvernee at openjdk.org (Jorn Vernee) Date: Wed, 7 Jun 2023 14:34:23 GMT Subject: RFR: 7903485 Windows.h fails to extract on jextract/panama In-Reply-To: References: Message-ID: On Thu, 1 Jun 2023 15:15:56 GMT, Maurizio Cimadamore wrote: > This patch fixes a number of issues I discovered while trying to extract Windows.h on a recent jextract: > > * the size of `long double` layout is wrong. This means that if a struct with a field of that type is declared, StructLayoutComputer will fail (as it will see a size != expected size reported by clang) > * There are cases where, also for unsupported layouts, we end up creating function descriptors with things containing padding. This is caused by the fact that the check for unsupported layouts in functions is performed *after* the creation of the function descriptor. > * Sometimes libclang reports field cursors in an out-of-order fashion. I have not been able to pinpoint the exact cause, as all my attempts to create a reduced test case failed. When this behavior occurs, jextract ends up generating extra fields and padding. I believe this behavior has always been there, but now brought to the fore by the new eager checks. > * A change in behavior of recent libclang causes `cursor.spelling()` to return non-empty strings (see https://github.com/llvm/llvm-project/issues/58429) > * A quality of life fix, to generate "sensible" warnings when variadic callbacks are found I gave this a try here as well. Windows.h extracts successfully, nice work! There is still one failing test due to https://github.com/openjdk/jextract/pull/106 missing from the panama branch. It is a really small fix, so maybe you could fold it into this PR? src/main/java/org/openjdk/jextract/impl/StructLayoutComputer.java line 91: > 89: if (offset > expectedOffset) { > 90: // out-of-order field, skip > 91: return; So, I think as a result of returning here, we will essentially just emit padding instead of the out of order field, right? I think it would be nice to log a message as well to let the user know that something went wrong, and the resulting layout might not be usable (similar to how we do that for unsupported layouts) src/main/java/org/openjdk/jextract/impl/TreeMaker.java line 217: > 215: return null; > 216: } > 217: return Declaration.scoped(scopeKind, CursorPosition.of(c), c.spelling()); Could you explain this change? I guess we're assuming that there are no decls here since this is not a definition? src/main/java/org/openjdk/jextract/impl/TypeImpl.java line 376: > 374: try { > 375: return Optional.of(getLayoutInternal(t)); > 376: } catch (Throwable ex) { This seems to be reverting the changes from: https://github.com/openjdk/jextract/pull/121/files#r1210157420 Not sure if that's intended? src/main/java/org/openjdk/jextract/impl/UnsupportedLayouts.java line 44: > 42: public static final MemoryLayout __INT128 = makeUnsupportedLayout(16, "__int128"); > 43: > 44: public static final MemoryLayout LONG_DOUBLE = makeUnsupportedLayout(IS_WINDOWS ? 8 : 16, "long double"); We could use the `ValueLayout.JAVA_DOUBLE` layout in `Type.java` for the `long double` type (we are already selecting the layout based on the platform for `long` there as well). If we do that `long double` would just work on Windows. FWIW, I gave this a try, and it looks like we'd just have to skip `LibUnsupportedTest::testIgnoredMethods` on Windows in that case. Windows.h still extracts successfully as well: diff --git a/src/main/java/org/openjdk/jextract/Type.java b/src/main/java/org/openjdk/jextract/Type.java index 65e0fed..291ec0c 100644 --- a/src/main/java/org/openjdk/jextract/Type.java +++ b/src/main/java/org/openjdk/jextract/Type.java @@ -137,7 +137,9 @@ public interface Type { /** * {@code long double} type. */ - LongDouble("long double", UnsupportedLayouts.LONG_DOUBLE), + LongDouble("long double", TypeImpl.IS_WINDOWS ? + ValueLayout.JAVA_DOUBLE : + UnsupportedLayouts.LONG_DOUBLE), /** * {@code float128} type. */ diff --git a/src/main/java/org/openjdk/jextract/impl/UnsupportedLayouts.java b/src/main/java/org/openjdk/jextract/impl/UnsupportedLayouts.java index a6b1aec..0797368 100644 --- a/src/main/java/org/openjdk/jextract/impl/UnsupportedLayouts.java +++ b/src/main/java/org/openjdk/jextract/impl/UnsupportedLayouts.java @@ -37,11 +37,9 @@ import java.nio.ByteOrder; public final class UnsupportedLayouts { private UnsupportedLayouts() {} - private static final boolean IS_WINDOWS = System.getProperty("os.name").startsWith("Windows"); - public static final MemoryLayout __INT128 = makeUnsupportedLayout(16, "__int128"); - public static final MemoryLayout LONG_DOUBLE = makeUnsupportedLayout(IS_WINDOWS ? 8 : 16, "long double"); + public static final MemoryLayout LONG_DOUBLE = makeUnsupportedLayout(16, "long double"); public static final MemoryLayout _FLOAT128 = makeUnsupportedLayout(16, "_float128"); diff --git a/test/jtreg/generator/test8257892/LibUnsupportedTest.java b/test/jtreg/generator/test8257892/LibUnsupportedTest.java index 7605826..86f09d6 100644 --- a/test/jtreg/generator/test8257892/LibUnsupportedTest.java +++ b/test/jtreg/generator/test8257892/LibUnsupportedTest.java @@ -28,6 +28,7 @@ import java.lang.foreign.MemoryLayout; import java.lang.foreign.MemoryLayout.PathElement; import java.lang.foreign.MemorySegment; import org.testng.annotations.Test; +import org.testng.SkipException; import test.jextract.unsupported.unsupported_h; import static org.testng.Assert.assertEquals; @@ -51,6 +52,8 @@ import test.jextract.unsupported.*; */ public class LibUnsupportedTest { + private static final boolean IS_WINDOWS = System.getProperty("os.name").startsWith("Windows"); + @Test public void testAllocateFoo() { try (Arena arena = Arena.ofConfined()) { @@ -86,6 +89,9 @@ public class LibUnsupportedTest { @Test public void testIgnoredMethods() { + if (IS_WINDOWS) { + throw new SkipException("long double works on Windows"); + } assertNull(findMethod(unsupported_h.class, "func")); assertNull(findMethod(unsupported_h.class, "func2")); assertNull(findMethod(unsupported_h.class, "func3")); ------------- PR Review: https://git.openjdk.org/jextract/pull/122#pullrequestreview-1467763096 PR Review Comment: https://git.openjdk.org/jextract/pull/122#discussion_r1221655651 PR Review Comment: https://git.openjdk.org/jextract/pull/122#discussion_r1221668473 PR Review Comment: https://git.openjdk.org/jextract/pull/122#discussion_r1221671068 PR Review Comment: https://git.openjdk.org/jextract/pull/122#discussion_r1221682039 From mcimadamore at openjdk.org Mon Jun 12 09:45:08 2023 From: mcimadamore at openjdk.org (Maurizio Cimadamore) Date: Mon, 12 Jun 2023 09:45:08 GMT Subject: RFR: 7903485 Windows.h fails to extract on jextract/panama In-Reply-To: References: Message-ID: On Wed, 7 Jun 2023 14:11:58 GMT, Jorn Vernee wrote: >> This patch fixes a number of issues I discovered while trying to extract Windows.h on a recent jextract: >> >> * the size of `long double` layout is wrong. This means that if a struct with a field of that type is declared, StructLayoutComputer will fail (as it will see a size != expected size reported by clang) >> * There are cases where, also for unsupported layouts, we end up creating function descriptors with things containing padding. This is caused by the fact that the check for unsupported layouts in functions is performed *after* the creation of the function descriptor. >> * Sometimes libclang reports field cursors in an out-of-order fashion. I have not been able to pinpoint the exact cause, as all my attempts to create a reduced test case failed. When this behavior occurs, jextract ends up generating extra fields and padding. I believe this behavior has always been there, but now brought to the fore by the new eager checks. >> * A change in behavior of recent libclang causes `cursor.spelling()` to return non-empty strings (see https://github.com/llvm/llvm-project/issues/58429) >> * A quality of life fix, to generate "sensible" warnings when variadic callbacks are found > > src/main/java/org/openjdk/jextract/impl/TypeImpl.java line 376: > >> 374: try { >> 375: return Optional.of(getLayoutInternal(t)); >> 376: } catch (Throwable ex) { > > This seems to be reverting the changes from: https://github.com/openjdk/jextract/pull/121/files#r1210157420 > > Not sure if that's intended? I found other cases where the created layout was problematic and this fix was causing other jextract crashes. I think if we want to do this change we need better investigation of the implications, so I've reverted it for now. ------------- PR Review Comment: https://git.openjdk.org/jextract/pull/122#discussion_r1226381281 From mcimadamore at openjdk.org Mon Jun 12 10:00:09 2023 From: mcimadamore at openjdk.org (Maurizio Cimadamore) Date: Mon, 12 Jun 2023 10:00:09 GMT Subject: RFR: 7903485 Windows.h fails to extract on jextract/panama In-Reply-To: References: Message-ID: On Wed, 7 Jun 2023 14:10:19 GMT, Jorn Vernee wrote: >> This patch fixes a number of issues I discovered while trying to extract Windows.h on a recent jextract: >> >> * the size of `long double` layout is wrong. This means that if a struct with a field of that type is declared, StructLayoutComputer will fail (as it will see a size != expected size reported by clang) >> * There are cases where, also for unsupported layouts, we end up creating function descriptors with things containing padding. This is caused by the fact that the check for unsupported layouts in functions is performed *after* the creation of the function descriptor. >> * Sometimes libclang reports field cursors in an out-of-order fashion. I have not been able to pinpoint the exact cause, as all my attempts to create a reduced test case failed. When this behavior occurs, jextract ends up generating extra fields and padding. I believe this behavior has always been there, but now brought to the fore by the new eager checks. >> * A change in behavior of recent libclang causes `cursor.spelling()` to return non-empty strings (see https://github.com/llvm/llvm-project/issues/58429) >> * A quality of life fix, to generate "sensible" warnings when variadic callbacks are found > > src/main/java/org/openjdk/jextract/impl/TreeMaker.java line 217: > >> 215: return null; >> 216: } >> 217: return Declaration.scoped(scopeKind, CursorPosition.of(c), c.spelling()); > > Could you explain this change? I guess we're assuming that there are no decls here since this is not a definition? To be 100% fair, I don't understand which situation is the `return` in the `else` supposed to cover. Even the old code seems off. E.g. `RecordLayoutComputer` seems to filter out cases where `c.getDefinition().isInvalid()`. But for everything else there should be a layout. But the layout was thrown away in the old impl as well. So, my goal here was to have a cleaner separation between "supported" cases and "unsupported" ones. Note that if we create a scoped decl w/o a layout (as we do in the old code), that thing ends up being skipped anyway by `OutputFactory`: if (d.layout().isEmpty() || structDefinitionSeen(d)) { //skip decl return null; } ``` Anyway, the main problem that was fixed here was to make sure that the _name_ of the type returned by `RecordLayoutComputer.compute` was reused, at least in the "supported" path. IMHO, we could just return `null` in the else, and we would probably not change semantics. > src/main/java/org/openjdk/jextract/impl/UnsupportedLayouts.java line 44: > >> 42: public static final MemoryLayout __INT128 = makeUnsupportedLayout(16, "__int128"); >> 43: >> 44: public static final MemoryLayout LONG_DOUBLE = makeUnsupportedLayout(IS_WINDOWS ? 8 : 16, "long double"); > > We could use the `ValueLayout.JAVA_DOUBLE` layout in `Type.java` for the `long double` type (we are already selecting the layout based on the platform for `long` there as well). If we do that `long double` would just work on Windows. > > FWIW, I gave this a try, and it looks like we'd just have to skip `LibUnsupportedTest::testIgnoredMethods` on Windows in that case. Windows.h still extracts successfully as well: > > > diff --git a/src/main/java/org/openjdk/jextract/Type.java b/src/main/java/org/openjdk/jextract/Type.java > index 65e0fed..291ec0c 100644 > --- a/src/main/java/org/openjdk/jextract/Type.java > +++ b/src/main/java/org/openjdk/jextract/Type.java > @@ -137,7 +137,9 @@ public interface Type { > /** > * {@code long double} type. > */ > - LongDouble("long double", UnsupportedLayouts.LONG_DOUBLE), > + LongDouble("long double", TypeImpl.IS_WINDOWS ? > + ValueLayout.JAVA_DOUBLE : > + UnsupportedLayouts.LONG_DOUBLE), > /** > * {@code float128} type. > */ > diff --git a/src/main/java/org/openjdk/jextract/impl/UnsupportedLayouts.java b/src/main/java/org/openjdk/jextract/impl/UnsupportedLayouts.java > index a6b1aec..0797368 100644 > --- a/src/main/java/org/openjdk/jextract/impl/UnsupportedLayouts.java > +++ b/src/main/java/org/openjdk/jextract/impl/UnsupportedLayouts.java > @@ -37,11 +37,9 @@ import java.nio.ByteOrder; > public final class UnsupportedLayouts { > private UnsupportedLayouts() {} > > - private static final boolean IS_WINDOWS = System.getProperty("os.name").startsWith("Windows"); > - > public static final MemoryLayout __INT128 = makeUnsupportedLayout(16, "__int128"); > > - public static final MemoryLayout LONG_DOUBLE = makeUnsupportedLayout(IS_WINDOWS ? 8 : 16, "long double"); > + public static final MemoryLayout LONG_DOUBLE = makeUnsupportedLayout(16, "long double"); > > public static final MemoryLayout _FLOAT128 = makeUnsupportedLayout(16, "_float128"); > > diff --git a/test/jtreg/generator/test8257892/LibUnsupportedTest.java b/test/jtreg/generator/test8257892/LibUnsupportedTest.java > index 7605826..86f09d6 100644 > --- a/test/jtreg/generator/test8257892/LibUnsuppor... I can do that if that's preferred. (it's mostly a choice on whether we want the set of supported types to be stable across platforms or not). ------------- PR Review Comment: https://git.openjdk.org/jextract/pull/122#discussion_r1226398225 PR Review Comment: https://git.openjdk.org/jextract/pull/122#discussion_r1226399671 From mcimadamore at openjdk.org Mon Jun 12 11:12:12 2023 From: mcimadamore at openjdk.org (Maurizio Cimadamore) Date: Mon, 12 Jun 2023 11:12:12 GMT Subject: RFR: 7903485 Windows.h fails to extract on jextract/panama In-Reply-To: References: Message-ID: <4V8D3e_m70QDvhudP_TfqIsko72Ak-wbDuXrB_NpWJ0=.e85e7e86-b7e7-4e38-8d34-217a33811c5f@github.com> On Mon, 12 Jun 2023 09:42:10 GMT, Maurizio Cimadamore wrote: >> src/main/java/org/openjdk/jextract/impl/TypeImpl.java line 376: >> >>> 374: try { >>> 375: return Optional.of(getLayoutInternal(t)); >>> 376: } catch (Throwable ex) { >> >> This seems to be reverting the changes from: https://github.com/openjdk/jextract/pull/121/files#r1210157420 >> >> Not sure if that's intended? > > I found other cases where the created layout was problematic and this fix was causing other jextract crashes. I think if we want to do this change we need better investigation of the implications, so I've reverted it for now. I did another test, just to be sure, and now this seems no longer necessary. I believe the main fix was the changes related to the timing when function descriptors were created - e.g. now if a function contains unsupported layouts we no longer attempt to get a descriptor for it. Before this PR we did, and that caused the function descriptor factory to fail (as unsupported layouts end up with padding). ------------- PR Review Comment: https://git.openjdk.org/jextract/pull/122#discussion_r1226487526 From mcimadamore at openjdk.org Mon Jun 12 11:34:05 2023 From: mcimadamore at openjdk.org (Maurizio Cimadamore) Date: Mon, 12 Jun 2023 11:34:05 GMT Subject: RFR: 7903485 Windows.h fails to extract on jextract/panama In-Reply-To: References: Message-ID: On Wed, 7 Jun 2023 14:01:32 GMT, Jorn Vernee wrote: >> This patch fixes a number of issues I discovered while trying to extract Windows.h on a recent jextract: >> >> * the size of `long double` layout is wrong. This means that if a struct with a field of that type is declared, StructLayoutComputer will fail (as it will see a size != expected size reported by clang) >> * There are cases where, also for unsupported layouts, we end up creating function descriptors with things containing padding. This is caused by the fact that the check for unsupported layouts in functions is performed *after* the creation of the function descriptor. >> * Sometimes libclang reports field cursors in an out-of-order fashion. I have not been able to pinpoint the exact cause, as all my attempts to create a reduced test case failed. When this behavior occurs, jextract ends up generating extra fields and padding. I believe this behavior has always been there, but now brought to the fore by the new eager checks. >> * A change in behavior of recent libclang causes `cursor.spelling()` to return non-empty strings (see https://github.com/llvm/llvm-project/issues/58429) >> * A quality of life fix, to generate "sensible" warnings when variadic callbacks are found > > src/main/java/org/openjdk/jextract/impl/StructLayoutComputer.java line 91: > >> 89: if (offset > expectedOffset) { >> 90: // out-of-order field, skip >> 91: return; > > So, I think as a result of returning here, we will essentially just emit padding instead of the out of order field, right? > > I think it would be nice to log a message as well to let the user know that something went wrong, and the resulting layout might not be usable (similar to how we do that for unsupported layouts) What happens here is that, typically, we already have generated a padding _before_ (e.g. because we jumped into a struct, and saw that first offset is e.g. 42). So we later go back and see offset 0, so we try to emit something for which we already have emitted padding. The layout will be usable (e.g it has right size and alignmment), but some fields will be skipped. ------------- PR Review Comment: https://git.openjdk.org/jextract/pull/122#discussion_r1226515168 From mcimadamore at openjdk.org Mon Jun 12 11:34:06 2023 From: mcimadamore at openjdk.org (Maurizio Cimadamore) Date: Mon, 12 Jun 2023 11:34:06 GMT Subject: RFR: 7903485 Windows.h fails to extract on jextract/panama In-Reply-To: References: Message-ID: <68Pg5eD65NN_PPxhPXexd3Fn9Rb8qDeC4Ww8uxFsRiI=.373cd10e-4c0c-4359-92f2-4ce8b1b0644c@github.com> On Mon, 12 Jun 2023 09:55:34 GMT, Maurizio Cimadamore wrote: >> src/main/java/org/openjdk/jextract/impl/TreeMaker.java line 217: >> >>> 215: return null; >>> 216: } >>> 217: return Declaration.scoped(scopeKind, CursorPosition.of(c), c.spelling()); >> >> Could you explain this change? I guess we're assuming that there are no decls here since this is not a definition? > > To be 100% fair, I don't understand which situation is the `return` in the `else` supposed to cover. Even the old code seems off. E.g. `RecordLayoutComputer` seems to filter out cases where `c.getDefinition().isInvalid()`. But for everything else there should be a layout. But the layout was thrown away in the old impl as well. So, my goal here was to have a cleaner separation between "supported" cases and "unsupported" ones. Note that if we create a scoped decl w/o a layout (as we do in the old code), that thing ends up being skipped anyway by `OutputFactory`: > > > if (d.layout().isEmpty() || structDefinitionSeen(d)) { > //skip decl > return null; > } > ``` > > Anyway, the main problem that was fixed here was to make sure that the _name_ of the type returned by `RecordLayoutComputer.compute` was reused, at least in the "supported" path. IMHO, we could just return `null` in the else, and we would probably not change semantics. Did some more analysis, which refreshed my memory. When the cursor is not a definition, record layout computer will just return a fake declared type, backed by a padding layout. It will **not** look at any of the sub-cursors (as there's no definition for clang to see), so there will be no members attached to the scoped tree of the declared type. In other words, `decls` is guaranteed to be empty here. ------------- PR Review Comment: https://git.openjdk.org/jextract/pull/122#discussion_r1226511952 From mcimadamore at openjdk.org Mon Jun 12 12:12:47 2023 From: mcimadamore at openjdk.org (Maurizio Cimadamore) Date: Mon, 12 Jun 2023 12:12:47 GMT Subject: RFR: 7903485 Windows.h fails to extract on jextract/panama [v2] In-Reply-To: References: Message-ID: > This patch fixes a number of issues I discovered while trying to extract Windows.h on a recent jextract: > > * the size of `long double` layout is wrong. This means that if a struct with a field of that type is declared, StructLayoutComputer will fail (as it will see a size != expected size reported by clang) > * There are cases where, also for unsupported layouts, we end up creating function descriptors with things containing padding. This is caused by the fact that the check for unsupported layouts in functions is performed *after* the creation of the function descriptor. > * Sometimes libclang reports field cursors in an out-of-order fashion. I have not been able to pinpoint the exact cause, as all my attempts to create a reduced test case failed. When this behavior occurs, jextract ends up generating extra fields and padding. I believe this behavior has always been there, but now brought to the fore by the new eager checks. > * A change in behavior of recent libclang causes `cursor.spelling()` to return non-empty strings (see https://github.com/llvm/llvm-project/issues/58429) > * A quality of life fix, to generate "sensible" warnings when variadic callbacks are found Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: Address review comments ------------- Changes: - all: https://git.openjdk.org/jextract/pull/122/files - new: https://git.openjdk.org/jextract/pull/122/files/91c47be7..469d8df7 Webrevs: - full: https://webrevs.openjdk.org/?repo=jextract&pr=122&range=01 - incr: https://webrevs.openjdk.org/?repo=jextract&pr=122&range=00-01 Stats: 23 lines in 6 files changed: 10 ins; 8 del; 5 mod Patch: https://git.openjdk.org/jextract/pull/122.diff Fetch: git fetch https://git.openjdk.org/jextract.git pull/122/head:pull/122 PR: https://git.openjdk.org/jextract/pull/122 From jvernee at openjdk.org Mon Jun 12 12:41:17 2023 From: jvernee at openjdk.org (Jorn Vernee) Date: Mon, 12 Jun 2023 12:41:17 GMT Subject: RFR: 7903485 Windows.h fails to extract on jextract/panama [v2] In-Reply-To: References: Message-ID: <0hOMsmxvP_GT0tLY4fe5gXQxy_ZPp3gat5qS7rfysTw=.c192755e-7815-4011-9ab6-9a1c60bef7d3@github.com> On Mon, 12 Jun 2023 09:56:40 GMT, Maurizio Cimadamore wrote: >> src/main/java/org/openjdk/jextract/impl/UnsupportedLayouts.java line 44: >> >>> 42: public static final MemoryLayout __INT128 = makeUnsupportedLayout(16, "__int128"); >>> 43: >>> 44: public static final MemoryLayout LONG_DOUBLE = makeUnsupportedLayout(IS_WINDOWS ? 8 : 16, "long double"); >> >> We could use the `ValueLayout.JAVA_DOUBLE` layout in `Type.java` for the `long double` type (we are already selecting the layout based on the platform for `long` there as well). If we do that `long double` would just work on Windows. >> >> FWIW, I gave this a try, and it looks like we'd just have to skip `LibUnsupportedTest::testIgnoredMethods` on Windows in that case. Windows.h still extracts successfully as well: >> >> >> diff --git a/src/main/java/org/openjdk/jextract/Type.java b/src/main/java/org/openjdk/jextract/Type.java >> index 65e0fed..291ec0c 100644 >> --- a/src/main/java/org/openjdk/jextract/Type.java >> +++ b/src/main/java/org/openjdk/jextract/Type.java >> @@ -137,7 +137,9 @@ public interface Type { >> /** >> * {@code long double} type. >> */ >> - LongDouble("long double", UnsupportedLayouts.LONG_DOUBLE), >> + LongDouble("long double", TypeImpl.IS_WINDOWS ? >> + ValueLayout.JAVA_DOUBLE : >> + UnsupportedLayouts.LONG_DOUBLE), >> /** >> * {@code float128} type. >> */ >> diff --git a/src/main/java/org/openjdk/jextract/impl/UnsupportedLayouts.java b/src/main/java/org/openjdk/jextract/impl/UnsupportedLayouts.java >> index a6b1aec..0797368 100644 >> --- a/src/main/java/org/openjdk/jextract/impl/UnsupportedLayouts.java >> +++ b/src/main/java/org/openjdk/jextract/impl/UnsupportedLayouts.java >> @@ -37,11 +37,9 @@ import java.nio.ByteOrder; >> public final class UnsupportedLayouts { >> private UnsupportedLayouts() {} >> >> - private static final boolean IS_WINDOWS = System.getProperty("os.name").startsWith("Windows"); >> - >> public static final MemoryLayout __INT128 = makeUnsupportedLayout(16, "__int128"); >> >> - public static final MemoryLayout LONG_DOUBLE = makeUnsupportedLayout(IS_WINDOWS ? 8 : 16, "long double"); >> + public static final MemoryLayout LONG_DOUBLE = makeUnsupportedLayout(16, "long double"); >> >> public static final MemoryLayout _FLOAT128 = makeUnsupportedLayout(16, "_float128"); >> >> diff --git a/test/jtreg/generator/test8257892/LibUnsupportedTest.java b/test/jtreg/generator/test8257892/LibUnsupportedTest.... > > I can do that if that's preferred. (it's mostly a choice on whether we want the set of supported types to be stable across platforms or not). I'm not sure I see the value in being consistent across platforms, tbh. Given also that the jextract output is already platform specific. And, after all, if we did support 128 bit `long double` in the linker, there would still be a difference between the platforms because they use different layouts for that type. ------------- PR Review Comment: https://git.openjdk.org/jextract/pull/122#discussion_r1226604156 From jvernee at openjdk.org Mon Jun 12 12:45:07 2023 From: jvernee at openjdk.org (Jorn Vernee) Date: Mon, 12 Jun 2023 12:45:07 GMT Subject: RFR: 7903485 Windows.h fails to extract on jextract/panama [v2] In-Reply-To: <68Pg5eD65NN_PPxhPXexd3Fn9Rb8qDeC4Ww8uxFsRiI=.373cd10e-4c0c-4359-92f2-4ce8b1b0644c@github.com> References: <68Pg5eD65NN_PPxhPXexd3Fn9Rb8qDeC4Ww8uxFsRiI=.373cd10e-4c0c-4359-92f2-4ce8b1b0644c@github.com> Message-ID: On Mon, 12 Jun 2023 11:29:29 GMT, Maurizio Cimadamore wrote: >> To be 100% fair, I don't understand which situation is the `return` in the `else` supposed to cover. Even the old code seems off. E.g. `RecordLayoutComputer` seems to filter out cases where `c.getDefinition().isInvalid()`. But for everything else there should be a layout. But the layout was thrown away in the old impl as well. So, my goal here was to have a cleaner separation between "supported" cases and "unsupported" ones. Note that if we create a scoped decl w/o a layout (as we do in the old code), that thing ends up being skipped anyway by `OutputFactory`: >> >> >> if (d.layout().isEmpty() || structDefinitionSeen(d)) { >> //skip decl >> return null; >> } >> ``` >> >> Anyway, the main problem that was fixed here was to make sure that the _name_ of the type returned by `RecordLayoutComputer.compute` was reused, at least in the "supported" path. IMHO, we could just return `null` in the else, and we would probably not change semantics. > > Did some more analysis, which refreshed my memory. When the cursor is not a definition, record layout computer will just return a fake declared type, backed by a padding layout. It will **not** look at any of the sub-cursors (as there's no definition for clang to see), so there will be no members attached to the scoped tree of the declared type. In other words, `decls` is guaranteed to be empty here. I thought perhaps the `else` was covering opaque types. i.e. in a case where we just have `typedef struct Opaque* opaque_t`, there would still be a `Declaration.Scoped` in the tree without a layout. ------------- PR Review Comment: https://git.openjdk.org/jextract/pull/122#discussion_r1226609539 From jvernee at openjdk.org Mon Jun 12 12:51:09 2023 From: jvernee at openjdk.org (Jorn Vernee) Date: Mon, 12 Jun 2023 12:51:09 GMT Subject: RFR: 7903485 Windows.h fails to extract on jextract/panama [v2] In-Reply-To: References: Message-ID: <-E4itOGxQoVA4TCHCJb3A1cNzIfVVdCGE9gaaWZ1puE=.5be279c1-1fe0-4ff1-b0c4-91c3f1373073@github.com> On Mon, 12 Jun 2023 12:12:47 GMT, Maurizio Cimadamore wrote: >> This patch fixes a number of issues I discovered while trying to extract Windows.h on a recent jextract: >> >> * the size of `long double` layout is wrong. This means that if a struct with a field of that type is declared, StructLayoutComputer will fail (as it will see a size != expected size reported by clang) >> * There are cases where, also for unsupported layouts, we end up creating function descriptors with things containing padding. This is caused by the fact that the check for unsupported layouts in functions is performed *after* the creation of the function descriptor. >> * Sometimes libclang reports field cursors in an out-of-order fashion. I have not been able to pinpoint the exact cause, as all my attempts to create a reduced test case failed. When this behavior occurs, jextract ends up generating extra fields and padding. I believe this behavior has always been there, but now brought to the fore by the new eager checks. >> * A change in behavior of recent libclang causes `cursor.spelling()` to return non-empty strings (see https://github.com/llvm/llvm-project/issues/58429) >> * A quality of life fix, to generate "sensible" warnings when variadic callbacks are found > > Maurizio Cimadamore has updated the pull request incrementally with one additional commit since the last revision: > > Address review comments Marked as reviewed by jvernee (Committer). ------------- PR Review: https://git.openjdk.org/jextract/pull/122#pullrequestreview-1474850337 From mcimadamore at openjdk.org Mon Jun 12 13:42:08 2023 From: mcimadamore at openjdk.org (Maurizio Cimadamore) Date: Mon, 12 Jun 2023 13:42:08 GMT Subject: RFR: 7903485 Windows.h fails to extract on jextract/panama [v2] In-Reply-To: References: <68Pg5eD65NN_PPxhPXexd3Fn9Rb8qDeC4Ww8uxFsRiI=.373cd10e-4c0c-4359-92f2-4ce8b1b0644c@github.com> Message-ID: On Mon, 12 Jun 2023 12:42:45 GMT, Jorn Vernee wrote: > I thought perhaps the `else` was covering opaque types. i.e. in a case where we just have `typedef struct Opaque* opaque_t`, there would still be a `Declaration.Scoped` in the tree without a layout. Yes and no - the main difference is that we tolerate opaque types when they are nested somewhere else. E.g. a pointer to an opaque type is a valid type for a struct field, or function parameter. Since the routine that create types ends up calling TreeMaker when it sees a struct/union, we have to do something for these types (other than returning `null`). But in such cases there' never a nested declaration attached to such types (because they are... opaque). ------------- PR Review Comment: https://git.openjdk.org/jextract/pull/122#discussion_r1226683757 From mcimadamore at openjdk.org Mon Jun 12 14:32:09 2023 From: mcimadamore at openjdk.org (Maurizio Cimadamore) Date: Mon, 12 Jun 2023 14:32:09 GMT Subject: Integrated: 7903485 Windows.h fails to extract on jextract/panama In-Reply-To: References: Message-ID: On Thu, 1 Jun 2023 15:15:56 GMT, Maurizio Cimadamore wrote: > This patch fixes a number of issues I discovered while trying to extract Windows.h on a recent jextract: > > * the size of `long double` layout is wrong. This means that if a struct with a field of that type is declared, StructLayoutComputer will fail (as it will see a size != expected size reported by clang) > * There are cases where, also for unsupported layouts, we end up creating function descriptors with things containing padding. This is caused by the fact that the check for unsupported layouts in functions is performed *after* the creation of the function descriptor. > * Sometimes libclang reports field cursors in an out-of-order fashion. I have not been able to pinpoint the exact cause, as all my attempts to create a reduced test case failed. When this behavior occurs, jextract ends up generating extra fields and padding. I believe this behavior has always been there, but now brought to the fore by the new eager checks. > * A change in behavior of recent libclang causes `cursor.spelling()` to return non-empty strings (see https://github.com/llvm/llvm-project/issues/58429) > * A quality of life fix, to generate "sensible" warnings when variadic callbacks are found This pull request has now been integrated. Changeset: ff8ce046 Author: Maurizio Cimadamore URL: https://git.openjdk.org/jextract/commit/ff8ce046eb7a5813fc76cac227f9a42ebb82bd62 Stats: 78 lines in 9 files changed: 39 ins; 25 del; 14 mod 7903485: Windows.h fails to extract on jextract/panama Reviewed-by: jvernee ------------- PR: https://git.openjdk.org/jextract/pull/122 From mcimadamore at openjdk.org Thu Jun 15 09:52:44 2023 From: mcimadamore at openjdk.org (Maurizio Cimadamore) Date: Thu, 15 Jun 2023 09:52:44 GMT Subject: RFR: Update jextract/panama branch to track JDK 22 In-Reply-To: References: Message-ID: On Thu, 15 Jun 2023 09:47:30 GMT, Maurizio Cimadamore wrote: > This PR contains some initial changes to enable jextract support for the latest Panama repo (and JDK 22 in general). > > It contains the usual updates to source, build and readme files (e.g. to replace jdk21 with jdk22). > > On top of that there are some changes to support the latest API changes in the Panama repo such as: > * removal of implicit sequence layout factory > * renaming of XYZUtf8String -> XYZString src/main/resources/org/openjdk/jextract/impl/resources/RuntimeHelper.java.template line 27: > 25: import static java.lang.foreign.Linker.*; > 26: import static java.lang.foreign.ValueLayout.*; > 27: import static java.lang.Long.MAX_VALUE; In principle, this is not required. But one test was failing as it was declaring a struct called "Long" :-) ------------- PR Review Comment: https://git.openjdk.org/jextract/pull/123#discussion_r1230750669 From mcimadamore at openjdk.org Thu Jun 15 09:52:41 2023 From: mcimadamore at openjdk.org (Maurizio Cimadamore) Date: Thu, 15 Jun 2023 09:52:41 GMT Subject: RFR: Update jextract/panama branch to track JDK 22 Message-ID: This PR contains some initial changes to enable jextract support for the latest Panama repo (and JDK 22 in general). It contains the usual updates to source, build and readme files (e.g. to replace jdk21 with jdk22). On top of that there are some changes to support the latest API changes in the Panama repo such as: * removal of implicit sequence layout factory * renaming of XYZUtf8String -> XYZString ------------- Commit messages: - Initial push Changes: https://git.openjdk.org/jextract/pull/123/files Webrev: https://webrevs.openjdk.org/?repo=jextract&pr=123&range=00 Stats: 39 lines in 19 files changed: 1 ins; 0 del; 38 mod Patch: https://git.openjdk.org/jextract/pull/123.diff Fetch: git fetch https://git.openjdk.org/jextract.git pull/123/head:pull/123 PR: https://git.openjdk.org/jextract/pull/123 From jvernee at openjdk.org Thu Jun 15 11:55:21 2023 From: jvernee at openjdk.org (Jorn Vernee) Date: Thu, 15 Jun 2023 11:55:21 GMT Subject: RFR: Update jextract/panama branch to track JDK 22 In-Reply-To: References: Message-ID: On Thu, 15 Jun 2023 09:47:30 GMT, Maurizio Cimadamore wrote: > This PR contains some initial changes to enable jextract support for the latest Panama repo (and JDK 22 in general). > > It contains the usual updates to source, build and readme files (e.g. to replace jdk21 with jdk22). > > On top of that there are some changes to support the latest API changes in the Panama repo such as: > * removal of implicit sequence layout factory > * renaming of XYZUtf8String -> XYZString Marked as reviewed by jvernee (Committer). ------------- PR Review: https://git.openjdk.org/jextract/pull/123#pullrequestreview-1481388993 From mcimadamore at openjdk.org Fri Jun 16 00:37:21 2023 From: mcimadamore at openjdk.org (Maurizio Cimadamore) Date: Fri, 16 Jun 2023 00:37:21 GMT Subject: Integrated: Update jextract/panama branch to track JDK 22 In-Reply-To: References: Message-ID: <3_CgE8GQv2pzP2TX0DiW-ZcAAoJjceB9QOgyb74gHRY=.4fd47b96-d866-498d-909d-0c7fe4ca81b8@github.com> On Thu, 15 Jun 2023 09:47:30 GMT, Maurizio Cimadamore wrote: > This PR contains some initial changes to enable jextract support for the latest Panama repo (and JDK 22 in general). > > It contains the usual updates to source, build and readme files (e.g. to replace jdk21 with jdk22). > > On top of that there are some changes to support the latest API changes in the Panama repo such as: > * removal of implicit sequence layout factory > * renaming of XYZUtf8String -> XYZString This pull request has now been integrated. Changeset: ca8942c3 Author: Maurizio Cimadamore URL: https://git.openjdk.org/jextract/commit/ca8942c38894040e76edf77a111ad7cd5f06052f Stats: 39 lines in 19 files changed: 1 ins; 0 del; 38 mod Update jextract/panama branch to track JDK 22 Reviewed-by: jvernee ------------- PR: https://git.openjdk.org/jextract/pull/123