From Tyler.Steele at ibm.com Fri Jun 9 17:02:30 2023 From: Tyler.Steele at ibm.com (Tyler Steele) Date: Fri, 9 Jun 2023 17:02:30 +0000 Subject: Making the foreign liner a required API Message-ID: Hello Jorn and fellow porters, I am following up with this message from Jorn from a few months ago [1]. I am generally supportive of their proposal, but I am not certain how to implement it. I have successfully built and bundled libffi on AIX, but I didn't use the createLibffiBundle.sh script provided by the PR. What I'm feeling confused about is how libffi will be incorporated into the jvm. Does this createLibffiBundle.sh get called programmatically during the build process (I can't seem to find where that would happen currently), or does each platform check in a bundle with a compiled version of the library to source control (and the build phase assumes it's present for those platforms that support it)? At present I'm getting an error at build time after enabling the fallback linker on aix. src/java.base/share/native/libfallbackLinker/fallbackLinker.c:66:10: warning: implicit declaration of function 'ffi_get_struct_offsets' is invalid in C99 [-Wimplicit-function-declaration] return ffi_get_struct_offsets((ffi_abi) abi, jlong_to_ptr(type), jlong_to_ptr(offsets)); ^ Which is not totally surprising given that I haven't figured out how to link it yet. Thanks in advance, Tyler [1] https://mail.openjdk.org/pipermail/porters-dev/2023-March/000753.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From jorn.vernee at oracle.com Fri Jun 9 22:27:46 2023 From: jorn.vernee at oracle.com (Jorn Vernee) Date: Sat, 10 Jun 2023 00:27:46 +0200 Subject: Making the foreign liner a required API In-Reply-To: References: Message-ID: <14c37e14-66c2-3457-dbe8-fdc5b078b36d@oracle.com> Hello Tyler, First of all, thanks for trying this out! To answer some of your questions: > Does this createLibffiBundle.sh get called programmatically during the build process (I can't seem to find where that would happen currently), or does each platform check in a bundle with a compiled version of the library to source control (and the build phase assumes it's present for those platforms that support it)? I pointed to the createLibffiBundle.sh script as an example, so you could get a rough idea of the steps I took. The script takes a devkit as input, which is an artifact we use internally for our CI builds. These bundles (devkit and libffi) are not distributed. The libffi library is either automatically detected by the build system if it is installed in a well-known system location, or it can be provided manually using the --with-libffi configure option (I've used the latter myself). I you'd like to know how the build works, I suggest taking a look at the ./make/autoconf/lib-ffi.m4 file, which is responsible for pulling the contents of the --with-libffi option into the build system. It basically sets up several make variables with compiler and linker options. You could grep through the rest of the ./make folder for some of the LIBFFI_* variables that are 'exported' at the end of the lib-ffi.m4 file using AC_SUBST. For the fallback linker you should find the relevant usages in the ./make/modules/java.base/lib.gmk file (near the end). To incorporate the library (.dll/.so/.dylib) into the JDK, it is essentially just copied into the right folder of the built JDK image. Automatic copying can be turned on using the --enable-libffi-bundling configuration flag. In practice, I'm using `--with-libffi= --enable-libffi-bundling --enable-fallback-linker` to create a working JDK image. > At present I'm getting an error at build time after enabling the fallback linker on aix. This is likely caused by an outdated version of libffi. The ffi_get_struct_offsets function was added in version 3.3, so that one or a later version should work. I've done all my testing with version 3.4.2, so I recommend going with that. Also, note that we decided to postpone making the linker API required for now. Given the deprecation of the 32-bit Windows x86 port in JDK 21, making the linker API required after that port is removed would avoid the need of having to amend the public API and implementation to allow selecting one of the 2 major ABIs on that platform (cdecl or stdcall). HTH, Jorn On 09/06/2023 19:02, Tyler Steele wrote: > Hello Jorn and fellow porters, > > I am following up with this message from Jorn from a few months ago > [1]. I am generally supportive of their proposal, but I am not certain > how to implement it. I have successfully built and bundled libffi on > AIX, but I didn't use the createLibffiBundle.sh script provided by the PR. > > What I'm feeling confused about is how libffi will be incorporated > into the jvm. Does this createLibffiBundle.sh get called > programmatically during the build process (I can't seem to find where > that would happen currently), or does each platform check in a bundle > with a compiled version of the library to source control (and the > build phase assumes it's present for those platforms that support it)? > At present I'm getting an error at build time after enabling the > fallback linker on aix. > > src/java.base/share/native/libfallbackLinker/fallbackLinker.c:66:10: > warning: implicit declaration of function 'ffi_get_struct_offsets' is > invalid in C99 [-Wimplicit-function-declaration] > ? return ffi_get_struct_offsets((ffi_abi) abi, jlong_to_ptr(type), > jlong_to_ptr(offsets)); > ? ? ? ? ?^ > Which is not totally surprising given that I haven't figured out how > to link it yet. > > Thanks in advance, > Tyler > > [1] https://mail.openjdk.org/pipermail/porters-dev/2023-March/000753.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From Tyler.Steele at ibm.com Thu Jun 22 23:32:15 2023 From: Tyler.Steele at ibm.com (Tyler Steele) Date: Thu, 22 Jun 2023 23:32:15 +0000 Subject: Making the foreign liner a required API In-Reply-To: <14c37e14-66c2-3457-dbe8-fdc5b078b36d@oracle.com> References: <14c37e14-66c2-3457-dbe8-fdc5b078b36d@oracle.com> Message-ID: Hi Jorn, Thanks for you email a few weeks back. That was very helpful in getting started. Now that I've debugged my libffi build on AIX, I have been able to configure OpenJDK successfully with the options you mentioned in your previous email. --enable-fallback-linker --with-libffi=libffi # a symbolicly linked dir containing lib and include --enable-libffi-bundling I am now vexed by the following error which occurs during make images (for OpenJDK): gmake[3]: *** No rule to make target 'libffi/lib/libffi.so.8', needed by '/home/hotspot/openjdk/jdk-tyler/build/aix-ppc64-server-fastdebug/support/modules_libs/java.base/libffi.so.8'. Stop. I see java.base/Copy.gmk:238 ought to be bringing that file into the build. I can also see by the error message that it has found the location of the correct lib. Did you face any similar issue in getting your bundle to work? Thanks Tyler ________________________________ From: Jorn Vernee Sent: June 9, 2023 16:27 To: Tyler Steele Cc: porters-dev at openjdk.org Subject: [EXTERNAL] Re: Making the foreign liner a required API Hello Tyler, First of all, thanks for trying this out! To answer some of your questions: > Does this createLibffiBundle.?sh get called programmatically during the build process (I can't seem to find where that would happen currently), or ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello Tyler, First of all, thanks for trying this out! To answer some of your questions: > Does this createLibffiBundle.sh get called programmatically during the build process (I can't seem to find where that would happen currently), or does each platform check in a bundle with a compiled version of the library to source control (and the build phase assumes it's present for those platforms that support it)? I pointed to the createLibffiBundle.sh script as an example, so you could get a rough idea of the steps I took. The script takes a devkit as input, which is an artifact we use internally for our CI builds. These bundles (devkit and libffi) are not distributed. The libffi library is either automatically detected by the build system if it is installed in a well-known system location, or it can be provided manually using the --with-libffi configure option (I've used the latter myself). I you'd like to know how the build works, I suggest taking a look at the ./make/autoconf/lib-ffi.m4 file, which is responsible for pulling the contents of the --with-libffi option into the build system. It basically sets up several make variables with compiler and linker options. You could grep through the rest of the ./make folder for some of the LIBFFI_* variables that are 'exported' at the end of the lib-ffi.m4 file using AC_SUBST. For the fallback linker you should find the relevant usages in the ./make/modules/java.base/lib.gmk file (near the end). To incorporate the library (.dll/.so/.dylib) into the JDK, it is essentially just copied into the right folder of the built JDK image. Automatic copying can be turned on using the --enable-libffi-bundling configuration flag. In practice, I'm using `--with-libffi= --enable-libffi-bundling --enable-fallback-linker` to create a working JDK image. > At present I'm getting an error at build time after enabling the fallback linker on aix. This is likely caused by an outdated version of libffi. The ffi_get_struct_offsets function was added in version 3.3, so that one or a later version should work. I've done all my testing with version 3.4.2, so I recommend going with that. Also, note that we decided to postpone making the linker API required for now. Given the deprecation of the 32-bit Windows x86 port in JDK 21, making the linker API required after that port is removed would avoid the need of having to amend the public API and implementation to allow selecting one of the 2 major ABIs on that platform (cdecl or stdcall). HTH, Jorn On 09/06/2023 19:02, Tyler Steele wrote: Hello Jorn and fellow porters, I am following up with this message from Jorn from a few months ago [1]. I am generally supportive of their proposal, but I am not certain how to implement it. I have successfully built and bundled libffi on AIX, but I didn't use the createLibffiBundle.sh script provided by the PR. What I'm feeling confused about is how libffi will be incorporated into the jvm. Does this createLibffiBundle.sh get called programmatically during the build process (I can't seem to find where that would happen currently), or does each platform check in a bundle with a compiled version of the library to source control (and the build phase assumes it's present for those platforms that support it)? At present I'm getting an error at build time after enabling the fallback linker on aix. src/java.base/share/native/libfallbackLinker/fallbackLinker.c:66:10: warning: implicit declaration of function 'ffi_get_struct_offsets' is invalid in C99 [-Wimplicit-function-declaration] return ffi_get_struct_offsets((ffi_abi) abi, jlong_to_ptr(type), jlong_to_ptr(offsets)); ^ Which is not totally surprising given that I haven't figured out how to link it yet. Thanks in advance, Tyler [1] https://mail.openjdk.org/pipermail/porters-dev/2023-March/000753.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.tribble at gmail.com Fri Jun 23 11:00:11 2023 From: peter.tribble at gmail.com (Peter Tribble) Date: Fri, 23 Jun 2023 12:00:11 +0100 Subject: Making the foreign liner a required API In-Reply-To: <14c37e14-66c2-3457-dbe8-fdc5b078b36d@oracle.com> References: <14c37e14-66c2-3457-dbe8-fdc5b078b36d@oracle.com> Message-ID: Thanks Jorn, On Fri, Jun 9, 2023 at 11:28?PM Jorn Vernee wrote: > To incorporate the library (.dll/.so/.dylib) into the JDK, it is > essentially just copied into the right folder of the built JDK image. > Automatic copying can be turned on using the --enable-libffi-bundling > configuration flag. In practice, I'm using `--with-libffi= extracted bundle> --enable-libffi-bundling --enable-fallback-linker` to > create a working JDK image. > I've been trying this on illumos/Solaris, and simply adding --enable-fallback-linker gets me a clean build against the system libffi. Bundling is a little trickier. I need to be explicit, because the Solaris 64-bit library location isn't one that's in the default search list, so --with-libffi-lib=/usr/lib/amd64 would be the correct way to specify where to look. However, in practice that fails because my system has multiple versions of libffi.so installed for binary compatibility (I'm not sure how common this may be on other platforms), and configure fails like so checking for libffi lib file location... /var/tmp/ud/jdk21-jdk-21-28/build/.configure-support/generated-configure.sh: line 144232: test: too many arguments configure: error: Could not locate libffi.so.? for bundling in /usr/lib/amd64 I think the way round this, for me, is to create a sacrificial bundle area and point the build at that, but I wonder if the correct approach might be to resolve what libffi.so points to rather than using the libffi.so.? pattern, which might match multiple things, and will break if the FFI shared library version ever grows to double digits. Thanks, -- -Peter Tribble http://www.petertribble.co.uk/ - http://ptribble.blogspot.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jorn.vernee at oracle.com Fri Jun 23 14:02:49 2023 From: jorn.vernee at oracle.com (Jorn Vernee) Date: Fri, 23 Jun 2023 16:02:49 +0200 Subject: [External] : RE: Making the foreign liner a required API In-Reply-To: References: <14c37e14-66c2-3457-dbe8-fdc5b078b36d@oracle.com> Message-ID: Hi Tyler, This looks like the file libffi/lib/libffi.so.8 is missing, so make tries to create it, and then finds there's no rule to do that (but, I've not run into this error myself, so it might be something else). You could try an absolute path for --with-libffi. Jorn On 23/06/2023 01:32, Tyler Steele wrote: > Hi Jorn, > > Thanks for you email a few weeks back. That was very helpful in > getting started. > > Now that I've debugged my libffi build on AIX, I have been able to > configure OpenJDK successfully with the options you mentioned in your > previous email. > > --enable-fallback-linker > --with-libffi=libffi # a symbolicly linked dir containing lib and > include > --enable-libffi-bundling > > I am now vexed by the following error which occurs during make images > (for OpenJDK): > > gmake[3]: *** No rule to make target 'libffi/lib/libffi.so.8', needed > by > '/home/hotspot/openjdk/jdk-tyler/build/aix-ppc64-server-fastdebug/support/modules_libs/java.base/libffi.so.8'. > ?Stop. > > I see java.base/Copy.gmk:238 ought to be bringing that file into the > build. I can also see by the error message that it has found the > location of the correct lib. Did you face any similar issue in getting > your bundle to work? > > Thanks > Tyler > ------------------------------------------------------------------------ > *From:* Jorn Vernee > *Sent:* June 9, 2023 16:27 > *To:* Tyler Steele > *Cc:* porters-dev at openjdk.org > *Subject:* [EXTERNAL] Re: Making the foreign liner a required API > Hello Tyler, First of all, thanks for trying this out! To answer some > of your questions: > Does this createLibffiBundle.?sh get called > programmatically during the build process (I can't seem to find where > that would happen currently), or > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > ZjQcmQRYFpfptBannerEnd > > Hello Tyler, > > > First of all, thanks for trying this out! > > > To answer some of your questions: > > > > Does this createLibffiBundle.sh get called programmatically during > the build process (I can't seem to find where that would happen > currently), or does each platform check in a bundle with a compiled > version of the library to source control (and the build phase assumes > it's present for those platforms that support it)? > > I pointed to the createLibffiBundle.sh script as an example, so you > could get a rough idea of the steps I took. The script takes a devkit > as input, which is an artifact we use internally for our CI builds. > These bundles (devkit and libffi) are not distributed. > > > The libffi library is either automatically detected by the build > system if it is installed in a well-known system location, or it can > be provided manually using the --with-libffi configure option (I've > used the latter myself). I you'd like to know how the build works, I > suggest taking a look at the ./make/autoconf/lib-ffi.m4 file, which is > responsible for pulling the contents of the --with-libffi option into > the build system. It basically sets up several make variables with > compiler and linker options. You could grep through the rest of the > ./make folder for some of the LIBFFI_* variables that are 'exported' > at the end of the lib-ffi.m4 file using AC_SUBST. For the fallback > linker you should find the relevant usages in the > ./make/modules/java.base/lib.gmk file (near the end). > > > To incorporate the library (.dll/.so/.dylib) into the JDK, it is > essentially just copied into the right folder of the built JDK image. > Automatic copying can be turned on using the --enable-libffi-bundling > configuration flag. In practice, I'm using `--with-libffi= extracted bundle> --enable-libffi-bundling --enable-fallback-linker` > to create a working JDK image. > > > > At present I'm getting an error at build time after enabling the > fallback linker on aix. > > This is likely caused by an outdated version of libffi. The > ffi_get_struct_offsets function was added in version 3.3, so that one > or a later version should work. I've done all my testing with version > 3.4.2, so I recommend going with that. > > > Also, note that we decided to postpone making the linker API required > for now. Given the deprecation of the 32-bit Windows x86 port in JDK > 21, making the linker API required after that port is removed would > avoid the need of having to amend the public API and implementation to > allow selecting one of the 2 major ABIs on that platform (cdecl or > stdcall). > > > HTH, > Jorn > > > On 09/06/2023 19:02, Tyler Steele wrote: >> Hello Jorn and fellow porters, >> >> I am following up with this message from Jorn from a few months ago >> [1]. I am generally supportive of their proposal, but I am not >> certain how to implement it. I have successfully built and bundled >> libffi on AIX, but I didn't use the createLibffiBundle.sh script >> provided by the PR. >> >> What I'm feeling confused about is how libffi will be incorporated >> into the jvm. Does this createLibffiBundle.sh get called >> programmatically during the build process (I can't seem to find where >> that would happen currently), or does each platform check in a bundle >> with a compiled version of the library to source control (and the >> build phase assumes it's present for those platforms that support >> it)? At present I'm getting an error at build time after enabling the >> fallback linker on aix. >> >> src/java.base/share/native/libfallbackLinker/fallbackLinker.c:66:10: >> warning: implicit declaration of function 'ffi_get_struct_offsets' is >> invalid in C99 [-Wimplicit-function-declaration] >> ? return ffi_get_struct_offsets((ffi_abi) abi, jlong_to_ptr(type), >> jlong_to_ptr(offsets)); >> ? ? ? ? ?^ >> Which is not totally surprising given that I haven't figured out how >> to link it yet. >> >> Thanks in advance, >> Tyler >> >> [1] >> https://mail.openjdk.org/pipermail/porters-dev/2023-March/000753.html >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From jorn.vernee at oracle.com Fri Jun 23 16:33:37 2023 From: jorn.vernee at oracle.com (Jorn Vernee) Date: Fri, 23 Jun 2023 18:33:37 +0200 Subject: [External] : Re: Making the foreign liner a required API In-Reply-To: References: <14c37e14-66c2-3457-dbe8-fdc5b078b36d@oracle.com> Message-ID: Hi Peter, > the Solaris 64-bit library location isn't one that's in the default search list It might be worth it to enhance the make/autoconf/lib-ffi.m4 script to look in the alternative library location as well. I recently added support for mac and windows as well [1], but prior to that, it really only seemed to be made for linux. So, there's probably still room for improvement. > I wonder if the correct approach might be to resolve what libffi.so points to rather than using the libffi.so.? pattern This might work, but libffi.so doesn't always seem to be a link. I suggest trying to work out some improvements to the lib-ffi.m4 script, and then we can update the script. Perhaps we need a --with-libffi-libfile option that points to the exact library file. Jorn [1]: https://github.com/openjdk/jdk/pull/14446 On 23/06/2023 13:00, Peter Tribble wrote: > Thanks Jorn, > > On Fri, Jun 9, 2023 at 11:28?PM Jorn Vernee > wrote: > > To incorporate the library (.dll/.so/.dylib) into the JDK, it is > essentially just copied into the right folder of the built JDK > image. Automatic copying can be turned on using the > --enable-libffi-bundling configuration flag. In practice, I'm > using `--with-libffi= > --enable-libffi-bundling --enable-fallback-linker` to create a > working JDK image. > > > I've been trying this on illumos/Solaris, and simply adding > > --enable-fallback-linker > > gets me a clean build against the system libffi. > > Bundling is a little trickier. I need to be explicit, because the > Solaris 64-bit library location isn't > one that's in the default search list, so > > --with-libffi-lib=/usr/lib/amd64 > > would be the correct way to specify where to look. However, in > practice that fails because my system > has multiple versions of libffi.so installed for binary compatibility > (I'm not sure how common this may > be on other platforms), and configure fails like so > > checking for libffi lib file location... > /var/tmp/ud/jdk21-jdk-21-28/build/.configure-support/generated-configure.sh: > line 144232: test: too many arguments > configure: error: Could not locate libffi.so.? for bundling in > /usr/lib/amd64 > > I think the way round this, for me, is to create a sacrificial bundle > area and point the build at that, > but I wonder if the correct approach might be to resolve what > libffi.so points to rather than using > the libffi.so.? pattern, which might match multiple things, and will > break if the FFI shared library > version ever grows to double digits. > > Thanks, > > -- > -Peter Tribble > http://www.petertribble.co.uk/ > > - http://ptribble.blogspot.com/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: