RFR: 8277489: Rewrite JAAS UnixLoginModule with FFM [v5]

[Martin Doerr]

On Sat, 3 Jan 2026 02:37:07 GMT, Weijun Wang <weijun at openjdk.org> wrote:

>> src/jdk.security.auth/share/classes/com/sun/security/auth/module/UnixLoginModule.java line 154:
>> 
>>> 152:         if (ss.getUsername() != null) {
>>> 153:             // When getpwuid_r fails, username will not be available.
>>> 154:             userPrincipal = new UnixPrincipal(ss.getUsername());
>> 
>> With this change, the test passes on AIX (though the `getpwuid_r` call does not work as expected).
>> @JoKern65, @varada1110: The `getpwuid_r` FFM call doesn't set the `result`. The strange thing is that it works when I call it through a C wrapper (same FFM call):
>> 
>> #include <pwd.h>
>> 
>> __attribute__((visibility("default"))) int call_getpwuid_r(int uid, struct passwd *pwd,
>>                                              char *buf, size_t buflen, struct passwd **result)
>> {
>>   return getpwuid_r(uid, pwd, buf, buflen, result);
>> }
>> 
>> Any idea?
>
> Have you tried `jextract` on AIX? Does the generated code have the same `FunctionDescriptor`?

No, I don't have `jextract` for AIX, but I think @varada1110 does.
The signature is `static int getpwuid_r(uid_t, struct passwd *, char *, size_t, struct passwd **)` which essentially matches. (A possible problem may be that `uid_t` is unsigned, but we're treating it as `int`. However, the sign bit is 0 in my experiments. All parameters are passed in 64-bit registers in which `int` gets sign extended.)
I have found `extern int _posix_getpwuid_r(uid_t, struct passwd *, char *, int, struct passwd **)` which takes an `int` for the buffer length and that functions works if I call it directly from Java (instead of the one above)!
Calling it through my C wrapper which uses the `size_t` also works as already stated above.
I think the AIX specific investigation could be done in a separate issue if the discussion is getting too long, here.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/28931#discussion_r2659065739