RFR 8222805: sun/security/pkcs11/tls/tls12/TestTLS12.java fails with Unsupported signature algorithm: rsa_pss_rsae_sha256

[Sean Mullan]

Hi Martin,

On 4/23/19 3:26 PM, Martin Balao wrote:
> Hi,
> 
> I'd like to propose a fix for 8222805 [1].
> 
> Webrev.00:
> 
>   * http://cr.openjdk.java.net/~mbalao/webrevs/8222805/8222805.webrev.00/
> 
> This issue is similar to 8221271 [2]: jdk.tls.disabledAlgorithms
> property is only read when SSLAlgorithmConstraints class is initialized
> and if this happens before TestTLS12.initialize method is executed,
> RSASSA-PSS algorithm is not disabled. This only reproduces on some
> environments. There are no more properties used in TestTLS12 so I don't
> expect a similar failure to occur.

I don't think this will work the way you think. The reason is that there 
is no system property for jdk.tls.disabledAlgorithms. It is only a 
security property so java -Djdk.tls.disabledAlgorithms has no effect.

(You could verify that by printing out the value of the 
jdk.tls.disabledAlgorithms security property in the test).

A workaround is to specify -Djava.security.properties=<file> where file 
contains the single line:

jdk.tls.disabledAlgorithms=RSASSA-PSS

This will override the value of the jdk.tls.disabledAlgorithms property 
in the java.security file.

However, I think there may be a more subtle bug or configuration issue 
underlying this. It seems like this could come up in real scenarios. You 
should never disable a strong algorithm, even if it is unsupported, in 
order to establish a TLS session. It should be able to negotiate a 
session using a different algorithm.

We have seen a similar issue with RSA-PSS like this with the SunMSCAPI 
provider but I think that was a bit different. Tony, or Xuelei, does 
this seem familiar?

So, unless you have a good explanation for that, on the outset, I don't 
think the fix is appropriate and we should spend more time looking at 
this. In the interests of time, I would ProblemList this test and open a 
separate bug for this issue.

Thanks,
Sean


> 
> Thanks,
> Martin.-
> 
> --
> [1] - https://bugs.openjdk.java.net/browse/JDK-8222805
> [2] - https://bugs.openjdk.java.net/browse/JDK-8221271
>