RFR: 8277489: Rewrite JAAS UnixLoginModule with FFM [v5]

[David Lloyd]

On Mon, 5 Jan 2026 18:00:09 GMT, Martin Doerr <mdoerr at openjdk.org> wrote:

>> I created fake account and group on my linux-x64 with numbers bigger than `Integer.MAX_VALUE` and call `getgroups` and `getpwuid_r`.  The results always look good after a `Integer.toUnsignedLong()` conversion.
>> 
>> I would think it's safe because it's only called after the C functions.
>
> What you have done is fine. Thanks! However, there is one potential problem left:
> We are passing `tmpUid` to `getpwuid_r` as an `int`. That results in the following sequence (example from AIX):
> 
> [2.537s][trace][foreign,downcall]  ;; { argument shuffle
> [2.537s][trace][foreign,downcall]   0x0a0001000747d744:   mr      r12,r3
> [2.537s][trace][foreign,downcall]   0x0a0001000747d748:   extsw   r3,r4
> [2.537s][trace][foreign,downcall]   0x0a0001000747d74c:   mr      r4,r5
> [2.537s][trace][foreign,downcall]   0x0a0001000747d750:   mr      r5,r6
> [2.537s][trace][foreign,downcall]   0x0a0001000747d754:   extsw   r6,r7
> [2.537s][trace][foreign,downcall]   0x0a0001000747d758:   mr      r7,r8
> [2.537s][trace][foreign,downcall]  ;; } argument shuffle
> 
> The 4 Byte value for `tmpUid` is taken from Register r4, sign extended to 8 Byte long and put into the first argument register r3. The sign extend is wrong because `uid` is an `uint32_t`. That violates the calling convention. We have no way to tell the FFM that we need zero extend.
> 
> A possible workaround would be to do the conversion in Java and passing it as long:
> 
> diff --git a/src/jdk.security.auth/share/classes/com/sun/security/auth/module/UnixSystem.java b/src/jdk.security.auth/share/classes/com/sun/security/auth/module/UnixSystem.java
> index ed520529ac8..573513b7bef 100644
> --- a/src/jdk.security.auth/share/classes/com/sun/security/auth/module/UnixSystem.java
> +++ b/src/jdk.security.auth/share/classes/com/sun/security/auth/module/UnixSystem.java
> @@ -25,6 +25,7 @@
>  
>  package com.sun.security.auth.module;
>  
> +import jdk.internal.util.Architecture;
>  import jdk.internal.util.OperatingSystem;
>  
>  import java.lang.foreign.AddressLayout;
> @@ -83,6 +84,8 @@ public class UnixSystem {
>              = (ValueLayout.OfByte) LINKER.canonicalLayouts().get("char");
>      private static final ValueLayout.OfInt C_INT
>              = (ValueLayout.OfInt) LINKER.canonicalLayouts().get("int");
> +    private static final ValueLayout.OfLong C_LONG
> +            = (ValueLayout.OfLong) LINKER.canonicalLayouts().get("long");
>      private static final AddressLayout C_POINTER
>              = ((AddressLayout) LINKER.canonicalLayouts().get("void*"))
>              .withTargetLayout(MemoryLayout.sequenceLayout(java.lang.Long.MAX_VALUE, C_CHAR));
> @@ -110,10 +113,14 @@ public class UnixSystem {
>  
>      // getpwuid_r does not work on AIX, instead we use another similar function
>      // extern int _posix_getpwuid_r(uid_t, struct passwd *, char *, int, struct passwd **)
> +  ...

FWIW there is an issue and ML discussion relating to extension of arguments with respect to `ValueLayout`: https://bugs.openjdk.org/browse/JDK-8336664

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/28931#discussion_r2662391104