Replacing syscall(_NR_fork) on linux with syscall(_NR_clone)

[Martin Buchholz]

At Google, we also had trouble with syscall(__NR_fork).  We switched to
simply calling fork despite the ominous warnings and have had no trouble
since.  We avoid calling clone - see my old failures with it in
UNIXProcess_md.c.  Although if we follow the successful precedent of
UNIXProcess_md.c we should call vfork instead, but we haven't yet had the
need.

@@ -6012,8 +6689,15 @@
   // separate process to execve. Make a direct syscall to fork process.
   // On IA64 there's no fork syscall, we have to use fork() and hope for
   // the best...
-  pid_t pid = NOT_IA64(syscall(__NR_fork);)
-              IA64_ONLY(fork();)
+  // pid_t pid = NOT_IA64(syscall(__NR_fork);)
+  //            IA64_ONLY(fork();)
+
+  // Google Specific: Changing the above code to use the fork glibc call in
+  // NOT_IA64. This allows the successful execution of the fork in a case
where
+  // the previous syscall hung  and hence allows successful OOM exit.
+  // We believe that fork() is now async-safe and suspect the above note
about
+  // pthread_atfork handlers and resetting the pthread library is outdated.
+  pid_t pid = fork();

   if (pid < 0) {
     // fork failed
@@ -6029,10 +6713,16 @@
     // in the new process, so make a system call directly.
     // IA64 should use normal execve() from glibc to match the glibc fork()
     // above.
-    NOT_IA64(syscall(__NR_execve, "/bin/sh", argv, environ);)
-    IA64_ONLY(execve("/bin/sh", (char* const*)argv, environ);)
+    // NOT_IA64(syscall(__NR_execve, "/bin/sh", argv, environ);)
+    // IA64_ONLY(execve("/bin/sh", (char* const*)argv, environ);)
+
+    // Google Specific: Based on fork case above changed this code to use
execve
+    // glibc call in NOT_IA64 case.
+    execve("/bin/sh", (char* const*)argv, environ);

     // execve failed
+    tty->print_cr("execve of OnError command \"/bin/sh -c %s\" failed with
errno %d: %s",
+                  cmd, errno, strerror(errno));
     _exit(-1);

   } else  {


On Fri, Apr 10, 2015 at 12:47 AM, David Holmes <david.holmes at oracle.com>
wrote:

> In os::fork_and_exec we provide a per platform fork/exec implementation
> for use by the OnError and OnOutOfMemory JVM options.
>
> On linux this is implemented with direct syscalls to _NR_fork (sys_fork)
> and _NR_execve.
>
> We already encountered a problem on linux-sparc:
>
> http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/
> 2015-February/013811.html
>
> due to a different register usage for the return value, which is so far
> restricted to the 7u train but will need addressing in 8u and 9.
>
> Further it seems that the fork syscall has actually been deprecated for
> quite some time and we are now seeing kernels on some platforms which are
> no longer providing any of the deprecated syscalls - we simply get an error
> ENOSYS.
>
> The fork() library was updated to use clone a long time ago, but as I
> understand it we can't use the fork() library call because we might be
> executing in a signal-handling context within the error handler.
>
> So I'm considering switching to use the clone syscall for all linux
> systems.
>
> Thoughts/comments?
>
> David
>
>