Inquiry: Handling Unix sockets created by glibc/NSS during checkpoint

Tue Sep 16 14:48:54 UTC 2025

Hi Ma Zhen,

I have to admit that I mistook sockets for named pipes when I was 
thinking about this the first time. The problem seems to be mentioned in 
CRIU docs [1]. Since you describe a connection to daemon (not a named 
unix socket) I guess there would be a path forward if the socket is idle 
& stateless: open another connection to the daemon and inherit this 
(note that you would have to use `-XX:CRaCIgnoredFileDescriptors` to 
prevent automatically closed inherited FDs).

However, while a technical solution might exist, this is really not up 
to CRaC philosophy. The process *should* isolate itself from the rest of 
the system; from CRaC POV the inability to do that in glibc is a 
deficiency (if there was API for that the native part of CRaC in JVM 
would call it). It might be possible to lookup & call 
`__nss_disable_nscd` early on during boot, but given that this is not a 
public API I don't think this belongs to the general JVM code.

The application has cooperate with CRaC a bit. Hopefully it is a 
reasonable requirement to deploy in an environment that does not inject 
FDs to the application like NSCD does. Googling around it doesn't seem 
to have the best reputation.

In these cases, I would love to provide more guidance in the first error 
the user gets, but IIUC this socket is not easy to classify as NSCD 
socket without tracing its origin.

I am glad that the containerized solution works for you.

Radim

[1] https://criu.org/External_UNIX_socket

On 9/16/25 11:42, ma zhen wrote:
>
> Hi Radim,
>
> Thank you so much for the detailed and insightful response.
>
> I followed your advice and set the parameter 
> `-XX:CRaCAllowedOpenFilePrefixes=socket:`. This successfully bypassed 
> CRaC's own validation check, and the log confirmed this with the 
> following message:
>
> `JVM: FD fd=4 type=socket path="socket:[11012921],port=29295" OK: 
> allowed in -XX:CRaCAllowedOpenFilePrefixes`
>
> However, this then revealed an underlying issue, seemingly within 
> CRIU. During the dump process, CRIU first reported:
>
> `Error (criu/sk-unix.c:865): unix: External socket is used. Consider 
> using --ext-unix-sk option.`
>
> Following CRIU's advice, I passed this option using the 
> `CRAC_CRIU_OPTS` environment variable 
> (`CRAC_CRIU_OPTS="--ext-unix-sk"`). Unfortunately, this led to a 
> different error from CRIU:
>
> `Error (criu/sk-unix.c:871): unix: Can't dump half of stream unix 
> connection.`
>
> My initial interpretation of this final error is that CRIU may have a 
> limitation in handling a process that holds only one end of an 
> external, stream-oriented (`SOCK_STREAM`) Unix socket connection. 
> However, I'm not entirely certain about this, and I plan to look into 
> CRIU's documentation and code to confirm this behavior.
>
> In the meantime, I was wondering if this aligns with your experience? 
> Perhaps you've encountered this specific CRIU error before.
>
> This experience certainly reinforces your other recommendation of 
> running the application in a minimal container to avoid creating the 
> socket in the first place. It seems like the most reliable path 
> forward for now.
>
> Thank you again for your guidance. It's been extremely valuable.
>
> Best regards,
>
> mazhen
>
> Radim Vansa <rvansa at azul.com> 于2025年9月15日周一 23:12写道：
>
>     Hi Ma Zhen,
>
>     we are aware of similar issue where an application has
>     `/var/cache/nscd/passwd` mapped despite not having the priviledge to
>     open() this file - the application can receive a file descriptor
>     through
>     a socket and then is able to mmap it. Another case are files under
>     `/var/lib/sss/mc/` opened by getpwuid_r, getpwname_r, getgrgid_r,
>     getgrname_r or similar functions.
>
>     You're right that File Descriptor Policies cannot be applied here,
>     these
>     work on a Java level (the FD must have an associated Java object).
>
>     There is a VM option `-XX:CRaCAllowedOpenFilePrefixes` that lets the
>     checkpoint to proceed if a file from this path is opened; in most
>     cases
>     CRIU can reopen a regular file without issues (and it should be
>     able to
>     handle sockets as well). I have not tested if the path matching works
>     with sockets, but shouldn't be too difficult to fix up for Unix
>     sockets.
>
>     Besides this there's no Resource-handling on native level (you cannot
>     register a native hook), though it might be possible to find an
>     open FD
>     and close it from Java - I wouldn't recommend such hacky way.
>
>     To be honest on systems where we've encountered this issue we rather
>     disabled NSCD service completely. If you can't control the
>     environment,
>     you can run the application in a container that won't be
>     configured with
>     these services.
>
>     Cheers,
>
>     Radim
>
>     On 9/15/25 11:29, ma zhen wrote:
>     >
>     > Hi CRaC developers,
>     >
>     > I am currently working on adapting a Java application to support
>     CRaC.
>     > I've encountered a specific challenge related to a Unix socket
>     that is
>     > preventing successful checkpoint creation.
>     >
>     > During the checkpoint process, I consistently receive a
>     > CheckpointOpenSocketException for a specific file descriptor, which
>     > lsof identifies as a Unix socket.
>     >
>     > I have conducted a detailed investigation to trace the origin of
>     this
>     > socket and found that it is not created directly by my Java
>     > application code. Instead, it is created by the underlying glibc
>     > library as part of the Name Service Switch (NSS) framework. The
>     call
>     > stack, captured using BCC, clearly shows that the socket() call
>     > originates from glibc's __nscd_* functions. This happens when
>     the JVM
>     > or application triggers a name service lookup (e.g., resolving a
>     user
>     > ID). In my specific environment, this results in a Unix socket
>     > connection from the Java process to the lwsmd daemon for
>     authentication.
>     >
>     > Because this socket is created and managed within the native C
>     > library, the standard approach of implementing a Java-level
>     > org.crac.Resource to close and restore it doesn't seem
>     applicable, as
>     > my application code has no direct handle or control over its
>     lifecycle.
>     >
>     > I have documented the full analysis, including the error, lsof
>     output,
>     > and BCC stack traces, in a detailed write-up which you can find
>     here:
>     >
>     https://github.com/mz1999/blog/blob/master/docs/trace_java_socket_creation-en.md
>     >
>     > My question is: What is the recommended approach for handling such
>     > file descriptors that are opened by underlying native libraries
>     > without direct control from the Java application?
>     >
>     > Are there any existing mechanisms, perhaps through advanced file
>     > descriptor policies, or any planned features that might address
>     this
>     > common scenario? Or is there another workaround that the team would
>     > suggest?
>     >
>     > Thank you for your time and for developing this fantastic
>     project. Any
>     > guidance you can provide would be greatly appreciated.
>     >
>     > Best regards,
>     > mazhen
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/crac-dev/attachments/20250916/70fb4399/attachment-0001.htm>