SSLEngine.unwrap on read-only input ByteBuffer

[Chris Vest]

Hi,

In Netty we've been trying to design some safer APIs, and attempted to make
more use of read-only ByteBuffers.

We discovered that SSLEngine.unwrap does not like read-only input buffers,
even though the input buffers should in theory only be read from. We
obviously make sure that the output buffers are writable.

By my reading of the javadoc, and the code, I believe this was intended to
work - or at least not intended to not work - but probably wasn't tested
directly.

When we try we get this stack trace on adopt-openjdk-11.0.7:

javax.net.ssl.SSLProtocolException: null
> at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:129)
> at
> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:326)
> at
> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:269)
> at
> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264)
> at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:118)
> at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:668)
> at
> java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:623)
> at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:441)
> at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:420)
> at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:674)
> at io.netty5.handler.ssl.EngineWrapper.unwrap(EngineWrapper.java:100)
> at io.netty5.handler.ssl.SslHandler.unwrap(SslHandler.java:1227)
> at
> io.netty5.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1105)
> at io.netty5.handler.ssl.SslHandler.decode(SslHandler.java:1165)
> at
> io.netty5.handler.codec.ByteToMessageDecoderForBuffer.decodeRemovalReentryProtection(ByteToMessageDecoderForBuffer.java:384)
> at
> io.netty5.handler.codec.ByteToMessageDecoderForBuffer.callDecode(ByteToMessageDecoderForBuffer.java:327)
> ... 20 common frames omitted
> Caused by: java.nio.ReadOnlyBufferException: null
> at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2493)
> at
> java.base/sun.security.ssl.SSLCipher$T12GcmReadCipherGenerator$GcmReadCipher.decrypt(SSLCipher.java:1629)
> at
> java.base/sun.security.ssl.SSLEngineInputRecord.decodeInputRecord(SSLEngineInputRecord.java:240)
> at
> java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:197)
> at
> java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:160)
> at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:108)
> ... 31 common frames omitted


I also tried this on a panama-preview snapshot JDK I have, and got a
similar stack trace:

% java -version
> openjdk version "19-internal" 2022-09-20
> OpenJDK Runtime Environment (fastdebug build
> 19-internal-adhoc.chris.panama-foreign)
> OpenJDK 64-Bit Server VM (fastdebug build
> 19-internal-adhoc.chris.panama-foreign, mixed mode)


% git show
> commit 144af9f43cd2d6f88b675b8c85e4034e5b9d6695 (HEAD -> foreign-preview,
> origin/foreign-preview)


javax.net.ssl.SSLProtocolException: null
> at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:129)
> at
> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:371)
> at
> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:314)
> at
> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:309)
> at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:121)
> at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736)
> at
> java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691)
> at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506)
> at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482)
> at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:719)
> at io.netty5.handler.ssl.EngineWrapper.unwrap(EngineWrapper.java:100)
> at io.netty5.handler.ssl.SslHandler.unwrap(SslHandler.java:1227)
> at
> io.netty5.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1105)
> at io.netty5.handler.ssl.SslHandler.decode(SslHandler.java:1165)
> at
> io.netty5.handler.codec.ByteToMessageDecoderForBuffer.decodeRemovalReentryProtection(ByteToMessageDecoderForBuffer.java:384)
> at
> io.netty5.handler.codec.ByteToMessageDecoderForBuffer.callDecode(ByteToMessageDecoderForBuffer.java:327)
> ... 20 common frames omitted
> Caused by: java.nio.ReadOnlyBufferException: null
> at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2497)
> at
> java.base/sun.security.ssl.SSLCipher$T13GcmReadCipherGenerator$GcmReadCipher.decrypt(SSLCipher.java:1933)
> at
> java.base/sun.security.ssl.SSLEngineInputRecord.decodeInputRecord(SSLEngineInputRecord.java:239)
> at
> java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:196)
> at
> java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:159)
> at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
> ... 31 common frames omitted


We can work around this in Netty since we need to support JDK versions that
has this issue anyway, but I think it's a bug that should be fixed at some
point.

Thanks,
Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/security-dev/attachments/20220323/1904d0bc/attachment.htm>