RFR: 8311596: Add separate system properties for TLS server and client for maximum chain length [v8]

[Sean Mullan]

On Fri, 10 Nov 2023 06:49:04 GMT, Xue-Lei Andrew Fan <xuelei at openjdk.org> wrote:

>> Good point - the CSR and RN could have been a bit more specific about the compatibility effect of changing the default from 10 to 8, so we will update that. Note that the CertPathBuilder default max path length [is 5 non-self-issued intermediate CA certificates](https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/security/cert/PKIXBuilderParameters.html#setMaxPathLength(int)), so even a lower value of 8 should have low risk as rigid TLS cert chains greater than 6 certs (where the 6th cert is the end entity cert which is not affected by the CertPathBuilder limit) on the wire will already be rejected. The additional certs permitted in the wire format is more for including a few more additional certs that might help build a valid path when there is more than one possible chain that a server or client might accept, which sometimes happens.
>
> Good point about the setMaxPathLength() limitation.  It looks like there might be an issue when the interactivities of setMaxPathLength(and its default value) and the properties defined here are not considered.  For example, what if the property is set to 8, while the setMaxPathLength is of value 5?  For the "PKIX" trust manager, per your description, if the property is set to 8, but 5 is the limit actually.  It looks like a weird behavior to me.  If I remember correctly, the "SunX509" trust manager does not use PKIXBuilderParameters, while the "PKIX" trust manager does.  It might be not the behavior we'd like to have that property 8 work for "SunX509" but not for "PKIX" trust manager.
> 
> Anyway, it might be better to look into the interactive behaviors among the properties and setMaxPathLength/default value.
> 
>> Good point - the CSR and RN could have been a bit more specific about the compatibility effect of changing the default from 10 to 8, so we will update that. Note that the CertPathBuilder default max path length [is 5 non-self-issued intermediate CA certificates](https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/security/cert/PKIXBuilderParameters.html#setMaxPathLength(int)), so even a lower value of 8 should have low risk as rigid TLS cert chains greater than 6 certs (where the 6th cert is the end entity cert which is not affected by the CertPathBuilder limit) on the wire will already be rejected. The additional certs permitted in the wire format is more for including a few more additional certs that might help build a valid path when there is more than one possible chain that a server or client might accept, which sometimes happens.

PKIX has been the default SunJSSE TrustManager since JDK 5 and AFAIK we have not had any reports of compatibility issues. This leads me to believe that valid TLS chains longer than 6 certificates are extremely rare.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/15163#discussion_r1391249509