[JDK-8146293] - Proposal to fix RSASSA-PSS AlgorithmChecker constraints for TLS 1.2

[Christopher Fox]

Hello,


We have been looking into supporting RSASSA-PSS signature algorithms within the chain of an end-entity certificate used for TLS 1.2. The EE certificate itself is not signed with RSASSA-PSS.


As mentioned in JDK-8146293<https://bugs.openjdk.java.net/browse/JDK-8146293>, we run into the exception: java.security.cert.CertificateException: Certificates does not conform to algorithm constraints


Upon closer inspection we believe there are 2 workarounds for this issue:


1) Update sun.security.provider.certpath.AlgorithmChecker#check(java.security.cert.Certificate, java.util.Collection<java.lang.String>) to call getSigAlgName from the provided certificate (var1), instead of the converted sun.security.x509.X509CertImpl (var3).


Looking at the code in question:


    public void check(Certificate var1, Collection<String> var2) throws CertPathValidatorException {
        if(var1 instanceof X509Certificate && this.constraints != null) {
            X509CertImpl var3 = null;

            try {
                var3 = X509CertImpl.toImpl((X509Certificate)var1);
            } catch (CertificateException var15) {
                throw new CertPathValidatorException(var15);
            }

            PublicKey var4 = var3.getPublicKey();
            String var5 = var3.getSigAlgName();
            AlgorithmId var6 = null;

            try {
                var6 = (AlgorithmId)var3.get("x509.algorithm");
            } catch (CertificateException var14) {
                throw new CertPathValidatorException(var14);
            }

            AlgorithmParameters var7 = var6.getParameters();
            if(!this.constraints.permits(SIGNATURE_PRIMITIVE_SET, var5, var7)) {
                throw new CertPathValidatorException("Algorithm constraints check failed: " + var5, (Throwable)null, (CertPath)null, -1, BasicReason.ALGORITHM_CONSTRAINED);
            } else {
            ....

The problem is that the sun.security.x509.X509CertImpl cannot convert the RSASSA-PSS algorithm OID to its friendly name when var3.getSigAlgName() is called:

[cid:6a0141b3-a283-46ca-9db8-115cafc77a07]
NOTE: In this case var1 is a instance of org.bouncycastle.jce.provider.X509CertificateObject

In our tests, making this change results in a successful TLS connection without further changes:

- String var5 = var3.getSigAlgName();
+ String var5 = ((X509Certificate)var1).getSigAlgName();


2) Update sun.security.x509.AlgorithmId to properly map the RSASSA-PSS algorithm OID to its friendly name. We have not experimented with this option, but believe it would have the same outcome, but with more code to change.


Any thoughts from the community on which approach would be accepted into the JDK, or alternative suggestions not mentioned here, are appreciated.


Thanks,

Chris Fox

Senior Software Engineer @ MobileIron
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/security-dev/attachments/20170210/e17c4245/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pastedImage.png
Type: image/png
Size: 11906 bytes
Desc: pastedImage.png
URL: <https://mail.openjdk.org/pipermail/security-dev/attachments/20170210/e17c4245/pastedImage.png>