RFR 8205445: Add RSASSA-PSS Signature support to SunMSCAPI

[Xuelei Fan]

Hi Weijun,

The release note and the following notes look reasonable to me.

For the implementation part, could it be a little bit more 
straightforward if wrapping the new attributes 
(pss/pssParams/fallbackSignature) and codes (if pss/fallbackSignature, 
etc) in the PSS subclass?

Did you want to remove the debug code in the security.cpp?  It seems 
that they are not used any more.

Xuelei

On 6/21/2018 4:12 AM, Weijun Wang wrote:
> Please take a review on this change
> 
>    http://cr.openjdk.java.net/~weijun/8205445/webrev.00/
>     
> and the release note at
> 
>    https://bugs.openjdk.java.net/browse/JDK-8205471
> 
> The code change adds RSASSA-PSS signature support to the SunMSCAPI provider.
> 
> Several notes:
> 
> 1. CryptoAPI (which SunMSCAPI is based on and now a deprecated technology) does not support RSASSA-PSS. In fact, CNG [1] is used to perform the signing and verification. This is certainly not a perfect solution and we are thinking of support CNG in a more sophisticated way in future releases of JDK.
> 
> 2. For unknown reason, the newly added verification code for RSASSA-PSS does not work correctly (precisely, ::NCryptTranslateHandle returns NTE_INVALID_PARAMETER). A fallback mechanism is added into mscapi/RSASignature.java. A SunRsaSign Signature object is actually used when a SunMSCAPI Signature is initialized to verify an RSASSA-PSS signature.
> 
> 3. It looks like CNG only supports PSSParamterSpec with the same message hash algorithm and MGF1 hash algorithm, because there is only one algorithm field in BCRYPT_PSS_PADDING_INFO [2]. This is checked when setting the parameter.
> 
> 4. It looks like CNG only supports RSASSA-PSS using these hash algorithms: SHA-1, SHA-256, SHA-384, and SHA-512. This is not checked at parameter setting but sign() will throw a SignatureException saying "Unrecognised hash algorithm". Since the verify() side uses a fallback SunRsaSign signature, other hash algorithms are supported.
> 
> Thanks
> Max
> 
> [1] https://msdn.microsoft.com/en-us/library/windows/desktop/aa376210(v=vs.85).aspx
> [2] https://msdn.microsoft.com/en-us/library/windows/desktop/aa375529(v=vs.85).aspx
> [3] https://msdn.microsoft.com/en-us/library/windows/desktop/aa375534(v=vs.85).aspx
>