Unable to use custom SSLEngine with default TrustManagerFactory after updating to ea20 (and later)

[Norman Maurer]

Will do tomorrow latest.

Thanks for the quick reply.

Bye
Norman 

> Am 10.07.2018 um 18:53 schrieb Xuelei Fan <xuelei.fan at oracle.com>:
> 
> Hi Norman,
> 
> It's an interesting user case of the TrustManagerFactory.  Please file a bug.
> 
> Thanks,
> Xuelei
> 
>> On 7/10/2018 9:57 AM, Alan Bateman wrote:
>> Forwarding to security-dev.
>>> On 10/07/2018 17:47, Norman Maurer wrote:
>>> Hi all,
>>> 
>>> I just tried to run netty[1] testsuite with the latest jdk11 EA release (21) and saw some class-cast-exception with our custom SSLEngine implementation
>>> 
>>> 
>>> Caused by: java.lang.ClassCastException: class io.netty.handler.ssl.OpenSslEngine cannot be cast to class sun.security.ssl.SSLEngineImpl (io.netty.handler.ssl.OpenSslEngine is in unnamed module of loader 'app'; sun.security.ssl.SSLEngineImpl is in module java.base of loader 'bootstrap')
>>> at java.base/sun.security.ssl.SSLAlgorithmConstraints.<init>(SSLAlgorithmConstraints.java:93)
>>> at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:270)
>>> at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141)
>>> at io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:237)
>>> at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:621)
>>> ... 27 more
>>> 
>>> 
>>> This change seems to be related to:
>>> http://hg.openjdk.java.net/jdk/jdk11/rev/68fa3d4026ea
>>> 
>>> I think you miss an instanceof check here in SSLAlgorithmConstraints before try to cast to SSLEngineImpl, as otherwise it will be impossible to use custom implementations of SSLEngine (which we have in netty) with the default TrustManagerFactory.
>>> 
>>> Does this sound correct ? Should I open a bug-report ?
>>> 
>>> Bye
>>> Norman
>>> 
>>> 
>>>