RFR: 8328723: IP Address error when client enables HTTPS endpoint check on server socket

[Prajwal Kumaraswamy]

On Sat, 6 Jul 2024 00:37:51 GMT, Bradford Wetmore <wetmore at openjdk.org> wrote:

>> The client identity checks when "HTTPS" endpoint identification algorithm is set on SSL server throws "java.security.cert.CertificateException: No subject alternative names present" when client certificate's SubjectAltName extension does not match its IP address
>> 
>> Since the server has no external knowledge of what the client's identity ought to be,  HTTPS identity checks must be disabled on the server side.
>> The exception message has been fixed to indicate the same.
>> 
>> I have performed the test both on SSL Server Engine and SSL Server Socket and attached are logs and snapshot for reference, also I have ran the changes against external test suite and test runs are green.
>
> src/java.base/share/classes/sun/security/ssl/X509TrustManagerImpl.java line 432:
> 
>> 430:         if (!identifiable) {
>> 431:             try {
>> 432:                 checkIdentity(peerHost,
> 
> You might do the direct check for the `checkClientTrusted`/`HTTPS` and immediately throw.  If that doesn't fail, then do the fallback `checkIdentity`.  Doing so would save the overhead of calling `checkIdentity` only to find out there was an error.
> 
> 
> if (!identifiable) {
>     // Clients can't identify themselves via SNI/hostnames in HTTPS.
>     if (checkClientTrusted && "HTTPS".equalsIgnoreCase(algorithm)) {
>         throw...
>     }
>     checkIdentity(peerHost, trustedChain[0], algorithm, chainsToPublicCA);
> }

Initially I had the same thought, however if an existing app has a ceritficate with it's IP address entry in SubjectAltName extension and endpoint set to HTTPS it is expected handshake will pass without any issues.

If I do this check first, then it'll break existing application where certificate has been created to match this criteria.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/20048#discussion_r1669722255