RFR: 8303509: Socket setTrafficClass does not work for IPv4 connections when IPv6 enabled

[Alan Bateman]

This is a small update to the underlying implementation of Socket.setTrafficClass so that it attempts to set the IP_TOS socket option when the socket is IPv6 and the operating system allows this option to be set on IPv6 sockets. The change is deliberately limited to the SocketImpl and SocketChannel implementation to avoid impacting DatagramChannel.

Socket.setTrafficClass is specified as a hint. For TCP connections it is possible on some platforms to set IP_TOS on an unbound socket to the ToS value to use in the SYN packet when establishing a connection. Setting it after a connection is established is possible on some platforms too. The JDK uses IPv6 sockets since JDK 1.4 so it gets a bit more complicated due to patchy support for IP_TOS.

- Linux allows IP_TOS to be set on an IPv6 socket at any time. If set on an unbound socket then the ToS will have the expected value when establishing a connection and it can be changed any time after connecting.
- macOS doesn't allow IP_TOS to be set on an IPv6 socket and it doesn't use the IPV6_TCLASS value as the IPv4 ToS. So on macOS, you have to use an IPv4 socket and set IP_TOS before connecting the socket.
- Windows hasn't allowed applications change the ToS for a long time. It allows the IP_TOS to be set on an unbound IPv6 socket but the value is not used.

The bug report report is Socket.setTrafficClass hasn't worked on Linux since JDK 13 when the SocketImpl was replaced. The workaround is to run with-Djava.net.preferIPv4=true. 

The proposed change is to just attempt to set both IPV6_TCLASS and IP_TOS when the socket is IPv6 and the operating system allows IPPROTO level sockets to be setup on IPv6 sockets. At some point I'd like to replace the underlying socket options implementation as it no longer needs to be in libnet, and that would be the limit to re-visit the equivalent in the channel implementations.

I had to use tcpdump and strace to test the changes, I don't think it's possible to create an automated test for this change.

-------------

Commit messages:
 - Extend to SocketChannel socket adaptor
 - Initial commit

Changes: https://git.openjdk.org/jdk/pull/12872/files
 Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=12872&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8303509
  Stats: 27 lines in 3 files changed: 17 ins; 1 del; 9 mod
  Patch: https://git.openjdk.org/jdk/pull/12872.diff
  Fetch: git fetch https://git.openjdk.org/jdk pull/12872/head:pull/12872

PR: https://git.openjdk.org/jdk/pull/12872