Unable to use custom SSLEngine with default TrustManagerFactory after updating to ea20 (and later)

[Norman Maurer]

Hey Pallavi,

Thanks :)

I just noticed you will also need to do the instanceof before you cast to SocketImpl in this class. Unfortunately I can not add a comment this issue directly (it seems), so hopefully mention it here is good enough.

Bye
Norman



> On 11. Jul 2018, at 08:54, Pallavi Sonal <pallavi.sonal at oracle.com> wrote:
> 
> Hi Norman ,
> Please refer https://bugs.openjdk.java.net/browse/JDK-8207029 to view your report, it has been moved to JDK.
> 
> Thanks,
> Pallavi Sonal
> 
> Message: 5
> Date: Wed, 11 Jul 2018 08:21:44 +0200
> From: Norman Maurer <norman.maurer at googlemail.com>
> To: Xuelei Fan <xuelei.fan at oracle.com>
> Cc: OpenJDK Dev list <security-dev at openjdk.java.net>
> Subject: Re: Unable to use custom SSLEngine with default
> 	TrustManagerFactory after updating to ea20 (and later)
> Message-ID: <BA805BE1-887F-444F-932D-BAFE5A3DF74F at googlemail.com>
> Content-Type: text/plain; charset="utf-8"
> 
> Submitted it via https://bugreport.java.com <https://bugreport.java.com/>.
> 
> Please let me know once it ?transferred? to https://bugs.openjdk.java.net <https://bugs.openjdk.java.net/>
> 
> 
> Bye
> Norman
> 
> 
> 
>> On 10. Jul 2018, at 20:26, Norman Maurer <norman.maurer at googlemail.com> wrote:
>> 
>> Will do tomorrow latest.
>> 
>> Thanks for the quick reply.
>> 
>> Bye
>> Norman
>> 
>>> Am 10.07.2018 um 18:53 schrieb Xuelei Fan <xuelei.fan at oracle.com>:
>>> 
>>> Hi Norman,
>>> 
>>> It's an interesting user case of the TrustManagerFactory.  Please file a bug.
>>> 
>>> Thanks,
>>> Xuelei
>>> 
>>>> On 7/10/2018 9:57 AM, Alan Bateman wrote:
>>>> Forwarding to security-dev.
>>>>> On 10/07/2018 17:47, Norman Maurer wrote:
>>>>> Hi all,
>>>>> 
>>>>> I just tried to run netty[1] testsuite with the latest jdk11 EA 
>>>>> release (21) and saw some class-cast-exception with our custom 
>>>>> SSLEngine implementation
>>>>> 
>>>>> 
>>>>> Caused by: java.lang.ClassCastException: class 
>>>>> io.netty.handler.ssl.OpenSslEngine cannot be cast to class 
>>>>> sun.security.ssl.SSLEngineImpl (io.netty.handler.ssl.OpenSslEngine 
>>>>> is in unnamed module of loader 'app'; 
>>>>> sun.security.ssl.SSLEngineImpl is in module java.base of loader 
>>>>> 'bootstrap') at 
>>>>> java.base/sun.security.ssl.SSLAlgorithmConstraints.<init>(SSLAlgori
>>>>> thmConstraints.java:93) at 
>>>>> java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509Tr
>>>>> ustManagerImpl.java:270) at 
>>>>> java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(
>>>>> X509TrustManagerImpl.java:141) at 
>>>>> io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedT
>>>>> rustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientConte
>>>>> xt.java:237) at 
>>>>> io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertifi
>>>>> cateVerifier.verify(ReferenceCountedOpenSslContext.java:621)
>>>>> ... 27 more
>>>>> 
>>>>> 
>>>>> This change seems to be related to:
>>>>> http://hg.openjdk.java.net/jdk/jdk11/rev/68fa3d4026ea
>>>>> 
>>>>> I think you miss an instanceof check here in SSLAlgorithmConstraints before try to cast to SSLEngineImpl, as otherwise it will be impossible to use custom implementations of SSLEngine (which we have in netty) with the default TrustManagerFactory.
>>>>> 
>>>>> Does this sound correct ? Should I open a bug-report ?
>>>>> 
>>>>> Bye
>>>>> Norman
>>>>> 
>>>>> 
>>>>> 
> 
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://mail.openjdk.java.net/pipermail/security-dev/attachments/20180711/c207ae06/attachment.html>
> 
> End of security-dev Digest, Vol 133, Issue 12
> *********************************************