From mz1999 at gmail.com Mon Sep 15 09:29:03 2025 From: mz1999 at gmail.com (ma zhen) Date: Mon, 15 Sep 2025 17:29:03 +0800 Subject: Inquiry: Handling Unix sockets created by glibc/NSS during checkpoint Message-ID: Hi CRaC developers, I am currently working on adapting a Java application to support CRaC. I've encountered a specific challenge related to a Unix socket that is preventing successful checkpoint creation. During the checkpoint process, I consistently receive a CheckpointOpenSocketException for a specific file descriptor, which lsof identifies as a Unix socket. I have conducted a detailed investigation to trace the origin of this socket and found that it is not created directly by my Java application code. Instead, it is created by the underlying glibc library as part of the Name Service Switch (NSS) framework. The call stack, captured using BCC, clearly shows that the socket() call originates from glibc's __nscd_* functions. This happens when the JVM or application triggers a name service lookup (e.g., resolving a user ID). In my specific environment, this results in a Unix socket connection from the Java process to the lwsmd daemon for authentication. Because this socket is created and managed within the native C library, the standard approach of implementing a Java-level org.crac.Resource to close and restore it doesn't seem applicable, as my application code has no direct handle or control over its lifecycle. I have documented the full analysis, including the error, lsof output, and BCC stack traces, in a detailed write-up which you can find here: https://github.com/mz1999/blog/blob/master/docs/trace_java_socket_creation-en.md My question is: What is the recommended approach for handling such file descriptors that are opened by underlying native libraries without direct control from the Java application? Are there any existing mechanisms, perhaps through advanced file descriptor policies, or any planned features that might address this common scenario? Or is there another workaround that the team would suggest? Thank you for your time and for developing this fantastic project. Any guidance you can provide would be greatly appreciated. Best regards, mazhen -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvansa at azul.com Mon Sep 15 15:12:08 2025 From: rvansa at azul.com (Radim Vansa) Date: Mon, 15 Sep 2025 17:12:08 +0200 Subject: Inquiry: Handling Unix sockets created by glibc/NSS during checkpoint In-Reply-To: References: Message-ID: <40425d19-68cc-4a80-8976-1b8937ca2a2d@azul.com> Hi Ma Zhen, we are aware of similar issue where an application has `/var/cache/nscd/passwd` mapped despite not having the priviledge to open() this file - the application can receive a file descriptor through a socket and then is able to mmap it. Another case are files under `/var/lib/sss/mc/` opened by getpwuid_r, getpwname_r, getgrgid_r, getgrname_r or similar functions. You're right that File Descriptor Policies cannot be applied here, these work on a Java level (the FD must have an associated Java object). There is a VM option `-XX:CRaCAllowedOpenFilePrefixes` that lets the checkpoint to proceed if a file from this path is opened; in most cases CRIU can reopen a regular file without issues (and it should be able to handle sockets as well). I have not tested if the path matching works with sockets, but shouldn't be too difficult to fix up for Unix sockets. Besides this there's no Resource-handling on native level (you cannot register a native hook), though it might be possible to find an open FD and close it from Java - I wouldn't recommend such hacky way. To be honest on systems where we've encountered this issue we rather disabled NSCD service completely. If you can't control the environment, you can run the application in a container that won't be configured with these services. Cheers, Radim On 9/15/25 11:29, ma zhen wrote: > > Hi CRaC developers, > > I am currently working on adapting a Java application to support CRaC. > I've encountered a specific challenge related to a Unix socket that is > preventing successful checkpoint creation. > > During the checkpoint process, I consistently receive a > CheckpointOpenSocketException for a specific file descriptor, which > lsof identifies as a Unix socket. > > I have conducted a detailed investigation to trace the origin of this > socket and found that it is not created directly by my Java > application code. Instead, it is created by the underlying glibc > library as part of the Name Service Switch (NSS) framework. The call > stack, captured using BCC, clearly shows that the socket() call > originates from glibc's __nscd_* functions. This happens when the JVM > or application triggers a name service lookup (e.g., resolving a user > ID). In my specific environment, this results in a Unix socket > connection from the Java process to the lwsmd daemon for authentication. > > Because this socket is created and managed within the native C > library, the standard approach of implementing a Java-level > org.crac.Resource to close and restore it doesn't seem applicable, as > my application code has no direct handle or control over its lifecycle. > > I have documented the full analysis, including the error, lsof output, > and BCC stack traces, in a detailed write-up which you can find here: > https://github.com/mz1999/blog/blob/master/docs/trace_java_socket_creation-en.md > > My question is: What is the recommended approach for handling such > file descriptors that are opened by underlying native libraries > without direct control from the Java application? > > Are there any existing mechanisms, perhaps through advanced file > descriptor policies, or any planned features that might address this > common scenario? Or is there another workaround that the team would > suggest? > > Thank you for your time and for developing this fantastic project. Any > guidance you can provide would be greatly appreciated. > > Best regards, > mazhen From mz1999 at gmail.com Tue Sep 16 09:42:02 2025 From: mz1999 at gmail.com (ma zhen) Date: Tue, 16 Sep 2025 17:42:02 +0800 Subject: Inquiry: Handling Unix sockets created by glibc/NSS during checkpoint In-Reply-To: <40425d19-68cc-4a80-8976-1b8937ca2a2d@azul.com> References: <40425d19-68cc-4a80-8976-1b8937ca2a2d@azul.com> Message-ID: Hi Radim, Thank you so much for the detailed and insightful response. I followed your advice and set the parameter `-XX:CRaCAllowedOpenFilePrefixes=socket:`. This successfully bypassed CRaC's own validation check, and the log confirmed this with the following message: `JVM: FD fd=4 type=socket path="socket:[11012921],port=29295" OK: allowed in -XX:CRaCAllowedOpenFilePrefixes` However, this then revealed an underlying issue, seemingly within CRIU. During the dump process, CRIU first reported: `Error (criu/sk-unix.c:865): unix: External socket is used. Consider using --ext-unix-sk option.` Following CRIU's advice, I passed this option using the `CRAC_CRIU_OPTS` environment variable (`CRAC_CRIU_OPTS="--ext-unix-sk"`). Unfortunately, this led to a different error from CRIU: `Error (criu/sk-unix.c:871): unix: Can't dump half of stream unix connection.` My initial interpretation of this final error is that CRIU may have a limitation in handling a process that holds only one end of an external, stream-oriented (`SOCK_STREAM`) Unix socket connection. However, I'm not entirely certain about this, and I plan to look into CRIU's documentation and code to confirm this behavior. In the meantime, I was wondering if this aligns with your experience? Perhaps you've encountered this specific CRIU error before. This experience certainly reinforces your other recommendation of running the application in a minimal container to avoid creating the socket in the first place. It seems like the most reliable path forward for now. Thank you again for your guidance. It's been extremely valuable. Best regards, mazhen Radim Vansa ?2025?9?15??? 23:12??? > Hi Ma Zhen, > > we are aware of similar issue where an application has > `/var/cache/nscd/passwd` mapped despite not having the priviledge to > open() this file - the application can receive a file descriptor through > a socket and then is able to mmap it. Another case are files under > `/var/lib/sss/mc/` opened by getpwuid_r, getpwname_r, getgrgid_r, > getgrname_r or similar functions. > > You're right that File Descriptor Policies cannot be applied here, these > work on a Java level (the FD must have an associated Java object). > > There is a VM option `-XX:CRaCAllowedOpenFilePrefixes` that lets the > checkpoint to proceed if a file from this path is opened; in most cases > CRIU can reopen a regular file without issues (and it should be able to > handle sockets as well). I have not tested if the path matching works > with sockets, but shouldn't be too difficult to fix up for Unix sockets. > > Besides this there's no Resource-handling on native level (you cannot > register a native hook), though it might be possible to find an open FD > and close it from Java - I wouldn't recommend such hacky way. > > To be honest on systems where we've encountered this issue we rather > disabled NSCD service completely. If you can't control the environment, > you can run the application in a container that won't be configured with > these services. > > Cheers, > > Radim > > On 9/15/25 11:29, ma zhen wrote: > > > > Hi CRaC developers, > > > > I am currently working on adapting a Java application to support CRaC. > > I've encountered a specific challenge related to a Unix socket that is > > preventing successful checkpoint creation. > > > > During the checkpoint process, I consistently receive a > > CheckpointOpenSocketException for a specific file descriptor, which > > lsof identifies as a Unix socket. > > > > I have conducted a detailed investigation to trace the origin of this > > socket and found that it is not created directly by my Java > > application code. Instead, it is created by the underlying glibc > > library as part of the Name Service Switch (NSS) framework. The call > > stack, captured using BCC, clearly shows that the socket() call > > originates from glibc's __nscd_* functions. This happens when the JVM > > or application triggers a name service lookup (e.g., resolving a user > > ID). In my specific environment, this results in a Unix socket > > connection from the Java process to the lwsmd daemon for authentication. > > > > Because this socket is created and managed within the native C > > library, the standard approach of implementing a Java-level > > org.crac.Resource to close and restore it doesn't seem applicable, as > > my application code has no direct handle or control over its lifecycle. > > > > I have documented the full analysis, including the error, lsof output, > > and BCC stack traces, in a detailed write-up which you can find here: > > > https://github.com/mz1999/blog/blob/master/docs/trace_java_socket_creation-en.md > > > > My question is: What is the recommended approach for handling such > > file descriptors that are opened by underlying native libraries > > without direct control from the Java application? > > > > Are there any existing mechanisms, perhaps through advanced file > > descriptor policies, or any planned features that might address this > > common scenario? Or is there another workaround that the team would > > suggest? > > > > Thank you for your time and for developing this fantastic project. Any > > guidance you can provide would be greatly appreciated. > > > > Best regards, > > mazhen > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rvansa at azul.com Tue Sep 16 14:48:54 2025 From: rvansa at azul.com (Radim Vansa) Date: Tue, 16 Sep 2025 16:48:54 +0200 Subject: Inquiry: Handling Unix sockets created by glibc/NSS during checkpoint In-Reply-To: References: <40425d19-68cc-4a80-8976-1b8937ca2a2d@azul.com> Message-ID: <8c72e932-a929-4121-850a-437f491814b8@azul.com> Hi Ma Zhen, I have to admit that I mistook sockets for named pipes when I was thinking about this the first time. The problem seems to be mentioned in CRIU docs [1]. Since you describe a connection to daemon (not a named unix socket) I guess there would be a path forward if the socket is idle & stateless: open another connection to the daemon and inherit this (note that you would have to use `-XX:CRaCIgnoredFileDescriptors` to prevent automatically closed inherited FDs). However, while a technical solution might exist, this is really not up to CRaC philosophy. The process *should* isolate itself from the rest of the system; from CRaC POV the inability to do that in glibc is a deficiency (if there was API for that the native part of CRaC in JVM would call it). It might be possible to lookup & call `__nss_disable_nscd` early on during boot, but given that this is not a public API I don't think this belongs to the general JVM code. The application has cooperate with CRaC a bit. Hopefully it is a reasonable requirement to deploy in an environment that does not inject FDs to the application like NSCD does. Googling around it doesn't seem to have the best reputation. In these cases, I would love to provide more guidance in the first error the user gets, but IIUC this socket is not easy to classify as NSCD socket without tracing its origin. I am glad that the containerized solution works for you. Radim [1] https://criu.org/External_UNIX_socket On 9/16/25 11:42, ma zhen wrote: > > Hi Radim, > > Thank you so much for the detailed and insightful response. > > I followed your advice and set the parameter > `-XX:CRaCAllowedOpenFilePrefixes=socket:`. This successfully bypassed > CRaC's own validation check, and the log confirmed this with the > following message: > > `JVM: FD fd=4 type=socket path="socket:[11012921],port=29295" OK: > allowed in -XX:CRaCAllowedOpenFilePrefixes` > > However, this then revealed an underlying issue, seemingly within > CRIU. During the dump process, CRIU first reported: > > `Error (criu/sk-unix.c:865): unix: External socket is used. Consider > using --ext-unix-sk option.` > > Following CRIU's advice, I passed this option using the > `CRAC_CRIU_OPTS` environment variable > (`CRAC_CRIU_OPTS="--ext-unix-sk"`). Unfortunately, this led to a > different error from CRIU: > > `Error (criu/sk-unix.c:871): unix: Can't dump half of stream unix > connection.` > > My initial interpretation of this final error is that CRIU may have a > limitation in handling a process that holds only one end of an > external, stream-oriented (`SOCK_STREAM`) Unix socket connection. > However, I'm not entirely certain about this, and I plan to look into > CRIU's documentation and code to confirm this behavior. > > In the meantime, I was wondering if this aligns with your experience? > Perhaps you've encountered this specific CRIU error before. > > This experience certainly reinforces your other recommendation of > running the application in a minimal container to avoid creating the > socket in the first place. It seems like the most reliable path > forward for now. > > Thank you again for your guidance. It's been extremely valuable. > > Best regards, > > mazhen > > Radim Vansa ?2025?9?15??? 23:12??? > > Hi Ma Zhen, > > we are aware of similar issue where an application has > `/var/cache/nscd/passwd` mapped despite not having the priviledge to > open() this file - the application can receive a file descriptor > through > a socket and then is able to mmap it. Another case are files under > `/var/lib/sss/mc/` opened by getpwuid_r, getpwname_r, getgrgid_r, > getgrname_r or similar functions. > > You're right that File Descriptor Policies cannot be applied here, > these > work on a Java level (the FD must have an associated Java object). > > There is a VM option `-XX:CRaCAllowedOpenFilePrefixes` that lets the > checkpoint to proceed if a file from this path is opened; in most > cases > CRIU can reopen a regular file without issues (and it should be > able to > handle sockets as well). I have not tested if the path matching works > with sockets, but shouldn't be too difficult to fix up for Unix > sockets. > > Besides this there's no Resource-handling on native level (you cannot > register a native hook), though it might be possible to find an > open FD > and close it from Java - I wouldn't recommend such hacky way. > > To be honest on systems where we've encountered this issue we rather > disabled NSCD service completely. If you can't control the > environment, > you can run the application in a container that won't be > configured with > these services. > > Cheers, > > Radim > > On 9/15/25 11:29, ma zhen wrote: > > > > Hi CRaC developers, > > > > I am currently working on adapting a Java application to support > CRaC. > > I've encountered a specific challenge related to a Unix socket > that is > > preventing successful checkpoint creation. > > > > During the checkpoint process, I consistently receive a > > CheckpointOpenSocketException for a specific file descriptor, which > > lsof identifies as a Unix socket. > > > > I have conducted a detailed investigation to trace the origin of > this > > socket and found that it is not created directly by my Java > > application code. Instead, it is created by the underlying glibc > > library as part of the Name Service Switch (NSS) framework. The > call > > stack, captured using BCC, clearly shows that the socket() call > > originates from glibc's __nscd_* functions. This happens when > the JVM > > or application triggers a name service lookup (e.g., resolving a > user > > ID). In my specific environment, this results in a Unix socket > > connection from the Java process to the lwsmd daemon for > authentication. > > > > Because this socket is created and managed within the native C > > library, the standard approach of implementing a Java-level > > org.crac.Resource to close and restore it doesn't seem > applicable, as > > my application code has no direct handle or control over its > lifecycle. > > > > I have documented the full analysis, including the error, lsof > output, > > and BCC stack traces, in a detailed write-up which you can find > here: > > > https://github.com/mz1999/blog/blob/master/docs/trace_java_socket_creation-en.md > > > > My question is: What is the recommended approach for handling such > > file descriptors that are opened by underlying native libraries > > without direct control from the Java application? > > > > Are there any existing mechanisms, perhaps through advanced file > > descriptor policies, or any planned features that might address > this > > common scenario? Or is there another workaround that the team would > > suggest? > > > > Thank you for your time and for developing this fantastic > project. Any > > guidance you can provide would be greatly appreciated. > > > > Best regards, > > mazhen > -------------- next part -------------- An HTML attachment was scrubbed... URL: