[jdk11u-dev] RFR: 8271199: Mutual TLS handshake fails signing client certificate with custom sensitive PKCS11 key

[Alexey Bakhtin]

Please review backport of JDK-8271199 to 11u
Backport is not clean because of changes introduced by JDK-8023980 and JDK-8172366. These changes are not related to functionality of JDK-8271199 so merge is trivial.
Also, there are simple changes related to instanceof pattern not implemented in JDK11.

sun/security/rsa jtreg and custom test with IAIK provider are passed

-------------

Commit messages:
 - Backport f6232982b91cb2314e96ddbde3984836a810a556

Changes: https://git.openjdk.java.net/jdk11u-dev/pull/1036/files
 Webrev: https://webrevs.openjdk.java.net/?repo=jdk11u-dev&pr=1036&range=00
  Issue: https://bugs.openjdk.java.net/browse/JDK-8271199
  Stats: 94 lines in 2 files changed: 72 ins; 15 del; 7 mod
  Patch: https://git.openjdk.java.net/jdk11u-dev/pull/1036.diff
  Fetch: git fetch https://git.openjdk.java.net/jdk11u-dev pull/1036/head:pull/1036

PR: https://git.openjdk.java.net/jdk11u-dev/pull/1036