Unable to use custom SSLEngine with default TrustManagerFactory after updating to ea20 (and later)

[Xuelei Fan]

Hi Norman,

It's an interesting user case of the TrustManagerFactory.  Please file a 
bug.

Thanks,
Xuelei

On 7/10/2018 9:57 AM, Alan Bateman wrote:
> Forwarding to security-dev.
> 
> On 10/07/2018 17:47, Norman Maurer wrote:
>> Hi all,
>>
>> I just tried to run netty[1] testsuite with the latest jdk11 EA 
>> release (21) and saw some class-cast-exception with our custom 
>> SSLEngine implementation
>>
>>
>> Caused by: java.lang.ClassCastException: class 
>> io.netty.handler.ssl.OpenSslEngine cannot be cast to class 
>> sun.security.ssl.SSLEngineImpl (io.netty.handler.ssl.OpenSslEngine is 
>> in unnamed module of loader 'app'; sun.security.ssl.SSLEngineImpl is 
>> in module java.base of loader 'bootstrap')
>> at 
>> java.base/sun.security.ssl.SSLAlgorithmConstraints.<init>(SSLAlgorithmConstraints.java:93)
>> at 
>> java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:270)
>> at 
>> java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141)
>> at 
>> io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:237)
>> at 
>> io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:621)
>> ... 27 more
>>
>>
>> This change seems to be related to:
>> http://hg.openjdk.java.net/jdk/jdk11/rev/68fa3d4026ea
>>
>> I think you miss an instanceof check here in SSLAlgorithmConstraints 
>> before try to cast to SSLEngineImpl, as otherwise it will be 
>> impossible to use custom implementations of SSLEngine (which we have 
>> in netty) with the default TrustManagerFactory.
>>
>> Does this sound correct ? Should I open a bug-report ?
>>
>> Bye
>> Norman
>>
>>
>>
>