[JDK-8146293] - Proposal to fix RSASSA-PSS AlgorithmChecker constraints for TLS 1.2

[Sean Mullan]

Hi Chris,

Comments inline ...

On 2/10/17 4:41 PM, Christopher Fox wrote:
> We have been looking into supporting RSASSA-PSS signature algorithms
> within the chain of an end-entity certificate used for TLS 1.2. The EE
> certificate itself is not signed with RSASSA-PSS.
>
> As mentioned in JDK-8146293
> <https://bugs.openjdk.java.net/browse/JDK-8146293>, we run into the
> exception: java.security.cert.CertificateException: Certificates does
> not conform to algorithm constraints
>
> Upon closer inspection we believe there are 2 workarounds for this issue:
>
> 1)
> Update sun.security.provider.certpath.AlgorithmChecker#check(java.security.cert.Certificate,
> java.util.Collection<java.lang.String>) to call getSigAlgName from the
> provided certificate (var1), instead of the
> converted sun.security.x509.X509CertImpl (var3).
>
>     Looking at the code in question:
>
>         public void check(Certificate var1, Collection<String> var2) throws CertPathValidatorException {
>             if(var1 instanceof X509Certificate && this.constraints != null) {
>                 X509CertImpl var3 = null;
>
>                 try {
>                     var3 = X509CertImpl.toImpl((X509Certificate)var1);
>                 } catch (CertificateException var15) {
>                     throw new CertPathValidatorException(var15);
>                 }
>
>                 PublicKey var4 = var3.getPublicKey();
>                 String var5 = var3.getSigAlgName();
>                 AlgorithmId var6 = null;
>
>                 try {
>                     var6 = (AlgorithmId)var3.get("x509.algorithm");
>                 } catch (CertificateException var14) {
>                     throw new CertPathValidatorException(var14);
>                 }
>
>                 AlgorithmParameters var7 = var6.getParameters();
>                 if(!this.constraints.permits(SIGNATURE_PRIMITIVE_SET, var5, var7)) {
>                     throw new CertPathValidatorException("Algorithm constraints check failed: " + var5, (Throwable)null, (CertPath)null, -1, BasicReason.ALGORITHM_CONSTRAINED);
>                 } else {
>                 ....
>
>     The problem is that the sun.security.x509.X509CertImpl cannot
>     convert the RSASSA-PSS algorithm OID to its friendly name when
>     var3.getSigAlgName() is called:
>
>     NOTE: In this case var1 is a instance
>     of org.bouncycastle.jce.provider.X509CertificateObject
>
>     In our tests, making this change results in a successful TLS
>     connection without further changes:
>
>     - Stringvar5 = var3.getSigAlgName();
>     + Stringvar5 = ((X509Certificate)var1).getSigAlgName();

We have just fixed this in JDK 9: 
http://hg.openjdk.java.net/jdk9/dev/jdk/rev/d911fe42d2da

> 2) Update sun.security.x509.AlgorithmId to properly map the RSASSA-PSS
> algorithm OID to its friendly name. We have not experimented with this
> option, but believe it would have the same outcome, but with more code
> to change.

I think that's a more involved changes that will be addressed by 
JDK-8146293.

Thanks,
Sean