Unable to use custom SSLEngine with default TrustManagerFactory after updating to ea20 (and later)

[Alan Bateman]

Forwarding to security-dev.

On 10/07/2018 17:47, Norman Maurer wrote:
> Hi all,
>
> I just tried to run netty[1] testsuite with the latest jdk11 EA 
> release (21) and saw some class-cast-exception with our custom 
> SSLEngine implementation
>
>
> Caused by: java.lang.ClassCastException: class 
> io.netty.handler.ssl.OpenSslEngine cannot be cast to class 
> sun.security.ssl.SSLEngineImpl (io.netty.handler.ssl.OpenSslEngine is 
> in unnamed module of loader 'app'; sun.security.ssl.SSLEngineImpl is 
> in module java.base of loader 'bootstrap')
> at 
> java.base/sun.security.ssl.SSLAlgorithmConstraints.<init>(SSLAlgorithmConstraints.java:93)
> at 
> java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:270)
> at 
> java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141)
> at 
> io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:237)
> at 
> io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:621)
> ... 27 more
>
>
> This change seems to be related to:
> http://hg.openjdk.java.net/jdk/jdk11/rev/68fa3d4026ea
>
> I think you miss an instanceof check here in SSLAlgorithmConstraints 
> before try to cast to SSLEngineImpl, as otherwise it will be 
> impossible to use custom implementations of SSLEngine (which we have 
> in netty) with the default TrustManagerFactory.
>
> Does this sound correct ? Should I open a bug-report ?
>
> Bye
> Norman
>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/security-dev/attachments/20180710/fccd352e/attachment.htm>