[jdk17] RFR: JDK-8265369 [macos-aarch64] java/net/MulticastSocket/Promiscuous.java failed with "SocketException: Cannot allocate memory"

[Chris Hegarty]

On Mon, 14 Jun 2021 15:28:01 GMT, Mark Sheppard <msheppar at openjdk.org> wrote:

> JDK-8265369 [macos-aarch64] java/net/MulticastSocket/Promiscuous.java failed with "SocketException: Cannot allocate memory"
> 
> The test java/net/MulticastSocket/Promiscuous.java has been observed to fail on a regular basis on macosx-aarch.
> This is typically under heavy test load on a test machine. Analysis of the problem have
> shown that the setsockopt for joining a multicast group will intermittently fail with ENOMEM.
> 
> While analysis of test environment shows significant memory usage and some memory pressure, it is
> not excessive and as such it is deemed transition or temporary condition, such that a retry of the
> setsockopt system call, has been seen to mitigate the issue. This adds to the stability of the
> Promiscuous.java test and reduces test failure noise.
> 
> The proposed fix is in open/src/java.base/unix/native/libnet/PlainDatagramSocketImpl.c
> in the mcast_join_leave function.  That is, if setsockopt to join an mcast group fails, and the errno == ENOMEM, 
> then re-invoke the setsockopt system call for joining a mcast group.
> The change has been applied as a conditional compilation.
> Additionally this change result in the Promiscuous.java test being removed from the
> ProblemList.txt.
> 
> Please oblige and review the changes for a fix of the issue JDK-8265369

src/java.base/unix/native/libnet/PlainDatagramSocketImpl.c line 2017:

> 2015:                 }
> 2016:             } else {
> 2017: #endif

The handling of ENOMEM here is consistent with how it is handled in the NIO area - good.

I wonder if a little restructuring may simplify the call flow? For example, something similar to:

  int n;
  ...
  /*
   * Join the multicast group.
   */
   n = setsockopt(fd, IPPROTO_IP, (join ? IP_ADD_MEMBERSHIP:IP_DROP_MEMBERSHIP),
                          (char *) &mname, mname_len);
#ifdef __APPLE__
   // workaround macOS bug where IP_ADD/DROP_MEMBERSHIP fails intermittently
   if (n < 0 && errno == ENOMEM) {
      n = setsockopt(fd, IPPROTO_IP, (join ? IP_ADD_MEMBERSHIP:IP_DROP_MEMBERSHIP),
                                          (char *) &mname, mname_len);
   }
#endif

  if (n < 0) {  ...

-------------

PR: https://git.openjdk.java.net/jdk17/pull/44