RFR: 8277489: Rewrite JAAS UnixLoginModule with FFM [v5]

[Martin Doerr]

On Sat, 3 Jan 2026 18:45:40 GMT, Weijun Wang <weijun at openjdk.org> wrote:

>> No, I don't have `jextract` for AIX, but I think @varada1110 does.
>> The signature is `static int getpwuid_r(uid_t, struct passwd *, char *, size_t, struct passwd **)` which essentially matches. (A possible problem may be that `uid_t` is unsigned, but we're treating it as `int`. However, the sign bit is 0 in my experiments. All parameters are passed in 64-bit registers in which `int` gets sign extended.)
>> I have found `extern int _posix_getpwuid_r(uid_t, struct passwd *, char *, int, struct passwd **)` which takes an `int` for the buffer length and that functions works if I call it directly from Java (instead of the one above)!
>> Calling it through my C wrapper which uses the `size_t` also works as already stated above.
>> I think the AIX specific investigation could be done in a separate issue if the discussion is getting too long, here.
>
> Or I can call this `_posix_getpwuid_r` function if `isAix()` is true. Is this enough?
> 
>      private static final MethodHandle getpwuid_r = LINKER
> -            .downcallHandle(SYMBOL_LOOKUP.findOrThrow("getpwuid_r"),
> +            .downcallHandle(SYMBOL_LOOKUP.findOrThrow(
> +                            OperatingSystem.isAix() ? "_posix_getpwuid_r" : "getpwuid_r"),
>                      FunctionDescriptor.of(C_INT, C_INT, C_POINTER, C_POINTER,
> -                            C_SIZE_T, C_POINTER));
> +                            OperatingSystem.isAix() ? C_INT : C_SIZE_T,
> +                            C_POINTER));

Thank you for taking care of it! I appreciate it. This has worked:

diff --git a/src/jdk.security.auth/share/classes/com/sun/security/auth/module/UnixSystem.java b/src/jdk.security.auth/share/classes/com/sun/security/auth/module/UnixSystem.java
index ee2c5effcf0..3afab6f7974 100644
--- a/src/jdk.security.auth/share/classes/com/sun/security/auth/module/UnixSystem.java
+++ b/src/jdk.security.auth/share/classes/com/sun/security/auth/module/UnixSystem.java
@@ -38,6 +38,8 @@
 import java.lang.invoke.MethodHandle;
 import java.lang.invoke.VarHandle;
 
+import jdk.internal.util.OperatingSystem;
+
 import static java.lang.foreign.MemoryLayout.PathElement.groupElement;
 
 /**
@@ -106,9 +108,9 @@ public class UnixSystem {
             .downcallHandle(SYMBOL_LOOKUP.findOrThrow("getgid"),
                     FunctionDescriptor.of(C_INT));
     private static final MethodHandle getpwuid_r = LINKER
-            .downcallHandle(SYMBOL_LOOKUP.findOrThrow("getpwuid_r"),
+            .downcallHandle(SYMBOL_LOOKUP.findOrThrow(OperatingSystem.isAix() ? "_posix_getpwuid_r" : "getpwuid_r"),
                     FunctionDescriptor.of(C_INT, C_INT, C_POINTER, C_POINTER,
-                            C_SIZE_T, C_POINTER));
+                            OperatingSystem.isAix() ? C_INT : C_SIZE_T, C_POINTER));
 
     private static final GroupLayout passwd_layout = MemoryLayout.structLayout(
             C_POINTER.withName("pw_name"),
@@ -136,7 +138,7 @@ public class UnixSystem {
     // sysconf(_SC_GETPW_R_SIZE_MAX) on macOS is 4096 and 1024 on Linux.
     // Not calling sysconf() here because _SC_GETPW_R_SIZE_MAX is different
     // on different platforms.
-    private static final long GETPW_R_SIZE_MAX = 4096L;
+    private static final int GETPW_R_SIZE_MAX = 4096;
 
     /**
      * Instantiate a {@code UnixSystem} and load

Note that `GETPW_R_SIZE_MAX` needs to be an `int` because `long` to `int` conversion would require an explicit cast (probably also for arm32). The other way round works implicitly.

I'd like to have feedback from the AIX experts as well. They may still be on vacation for a couple of days.

We still have a potential problem with the UID parameter. We are using an `int`, but the C code expects an `uint32_t`. Some platforms pass 32 bit values in 64 bit registers and expect a proper extension (zero for unsigned and sign for signed). If the UID becomes larger than INT_MAX, we use sign extend which is wrong.
Not sure of anyone uses so large UIDs and if we could restrict them.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/28931#discussion_r2661215966