Issues with ALPN implementation in JDK 9

[Simone Bordet]

Hi,

I gave a shot at implementing ALPN in JDK 9 in Jetty.

TLDR: I could not find a way to make it work. This email is to discuss
whether I am off road or discuss possible solutions.

Below my feedback.

* Lack of facilities to convert TLS protocol bytes to protocol strings.
This class already exist in JDK, sun.security.ssl.ProtocolVersion, it
would just need to be exposed in javax.net.ssl.

* Lack of facilities to convert TLS cipher bytes to cipher name strings.
As above, sun.security.ssl.CipherSuite exists, needs to be exposed publicly.

Note that for the 2 bullets above, a recent message from Mark Reinhold
to jdk9-dev confirmed that JDK 9 is *not yet* feature complete, so I
hope they can be considered for inclusion.

* Server-side Implementation
I followed the guidelines reported here:
http://mail.openjdk.java.net/pipermail/security-dev/2015-December/013132.html,
namely:

1) Read network bytes after initial connection.
2) Parse network bytes, expecting TLS ClientHello message.
3) Extract from ClientHello the TLS protocol version, the TLS ciphers,
the ALPN protocols.

At this point, I should negotiate the application protocol, and it
must be only one.

Assuming the ClientHello TLS protocol is spoken by both peers, the
server logic can create pairs (cipher, app_proto) for each of the
ciphers in common between client and server, and discard the ciphers
that are not good for any protocol.

At this point, among all the valid pairs, I need to choose a protocol.
Let's make an example; the pairs are:

(TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, h3)
(TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, h2)
(TLS_WHATEVER, http/1.1)

Here "h3" is the future HTTP/3 protocol which I picked as an example
to show the problem I will encounter.

Because at this point the server logic must choose one protocol only
(so that it can be returned in the ServerHello), it picks h3, which
goes along with a ECDSA cipher, so:

sslParams.setApplicationProtocols(new String[]{"h3"});
sslParams.setCipherSuites(new
String[]{"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", ...);
sslEngine.setSSLParameters(sslParams);

At this point, the server logic is good to let SSLEngine do the
unwrap(), and pass the original network bytes to unwrap().

During the unwrap(), the JDK implementation picks a cipher based on
the JDK logic.
In particular, in my case, I had a keystore with a certificate that
was *not* ECDSA.
If, in the snippet above, I set more than one cipher on the
SSLParameters, then perhaps a weaker cipher could be negotiated that
is not good for h3.
Otherwise, if I set only one cipher, there are no ciphers in common
and the TLS handshake is terminated with an error.

Bottom line, no negotiation is possible with this approach.

Next attempt I made was that before calling unwrap(), the server code
opens the keystore, and verifies if the certificate is ECDSA, handles
SNI, etc.

However, this means duplicating all the JDK logic to make sure that
the server logic *before* calling unwrap() is the same of the JDK so
that when unwrap() is called there will be no failures.

I don't think this is maintainable; the JDK is entitled to change the
logic following CVEs, optimizations and what not, and each such change
risks to break existing server code.

I then tried another approach.

In the server code, before calling unwrap(), I would remember the
pairs (cipher, app_proto), but *not* calling
SSLParameters.setApplicationProtocol().

I would then call unwrap(), where the JDK would choose the cipher.
The cipher is chosen in the NEED_TASK step of the unwrap(), so after
the task is run, the cipher chosen by the JDK is now available to the
server logic.

At this point I called again the server logic and, given the exact
cipher, choose the right protocol among the pairs that I have
previously stored.
In the example above, the JDK would have chose the RSA cipher because
the certificate was not ECDSA, and the server logic would have chosen
h2 as the application protocol:

sslParams.setApplicationProtocol(new String[]{"h2"});

Then let the unwrap()/wrap() code to finish the TLS handshake (in
particular, generate the ServerHello).

This approach has the benefit of cipher pre-selection done by the
server logic (it will retain not the intersection of ciphers between
client and server, but a possibly more restricted set that is valid
for the application protocols that are supported - imagine when
http/1.1 is not supported), coupled with JDK logic to interact with
SNI and certificates, coupled with a "late" selection of the
application protocol based on the cipher selected by the JDK logic.

Unfortunately, it does not work.

It does not work because the JDK implementation of
SSLEngine.setSSLParameters() is (more or less):

if (!handshake.started()) {
  handshaker.setApplicationProtocols(applicationProtocols);
  ...
}

By the time the task is run in the NEED_TASK step,
handshaker.started==true, so the application protocols are not copied
into the handshaker and are not used to generate the ServerHello.

Conclusions.

I could not make a reliable ALPN implementation with the current JDK 9.
If I am off road, then that's good news, and I will be all ears for
alternative approaches.

If I am correct, I would like to discuss whether it would be possible
to delay handshaker.started=true to a later time, so that
SSLParameters can be changed just after the NEED_TASK step, so that
server applications will be able to interact with the JDK for what
pertains TLS protocols, ciphers, SNI, etc. without duplicating the
logic.

Comments welcome.

Thanks !

-- 
Simone Bordet
http://bordet.blogspot.com
---
Finally, no matter how good the architecture and design are,
to deliver bug-free software with optimal performance and reliability,
the implementation technique must be flawless.   Victoria Livschitz