Optional client authentication throws SSLHandshakeException

[ramidzkh]

Optional client authentication, such as enabled by QuicSslContextBuilder#clientAuth(ClientAuth.OPTIONAL), causes Netty QUIC clients to fail connecting if no sufficient key material is found, such as with QuicSslContextBuilder#keyManager. This raises a cryptic exception with a limited stack trace

Caused by: javax.net.ssl.SSLHandshakeException: QUICHE_ERR_TLS_FAIL: error:1000007e:SSL routines:OPENSSL_internal:CERT_CB_ERROR
	at io.netty.incubator.codec.quic.Quiche.newException(Quiche.java:758)
	at io.netty.incubator.codec.quic.Quiche.throwIfError(Quiche.java:777)
	at io.netty.incubator.codec.quic.QuicheQuicChannel.connectionSendSimple(QuicheQuicChannel.java:1161)
	at io.netty.incubator.codec.quic.QuicheQuicChannel.connectionSend(QuicheQuicChannel.java:1250)

My guess is that BoringSSLCertificateCallback#handle removes the engine and ends up yielding a failure to BoringSSL if keying material is not found, without checking if it's mandatory with the server (which I don't think is possible to determine within the Java portion at this time)

Example server

QuicSslContext context = QuicSslContextBuilder
        .forServer(new File("server_key.pem"), null, new File("server_certificate.pem"))
        .trustManager(new File("ca_certificate.pem"))
        .applicationProtocols("echo/0.0.1")
        .clientAuth(ClientAuth.OPTIONAL) // try toggling me
        .build();
ChannelHandler codec = new QuicServerCodecBuilder()
        .sslContext(context)
        .maxIdleTimeout(5, TimeUnit.SECONDS)
        // Configure some limits for the maximal number of streams (and the data) that we want to handle.
        .initialMaxData(10000000)
        .initialMaxStreamDataBidirectionalLocal(1000000)
        .initialMaxStreamDataBidirectionalRemote(1000000)
        .initialMaxStreamsBidirectional(100)
        .initialMaxStreamsUnidirectional(100)
        .tokenHandler(InsecureQuicTokenHandler.INSTANCE)
        .handler(new ChannelInboundHandlerAdapter() {
            @Override
            public void userEventTriggered(ChannelHandlerContext ctx, Object evt) {
                if (evt == SslHandshakeCompletionEvent.SUCCESS) {
                    try {
                        X509Certificate peer = (X509Certificate) ((QuicChannel) ctx.channel()).sslEngine().getSession().getPeerCertificates()[0];
                        System.err.println("Verification... success!");
                    } catch (Exception ignored) {
                        System.err.println("Verification... failure!");
                    }
                }

                ctx.fireUserEventTriggered(evt);
            }

            @Override
            public boolean isSharable() {
                return true;
            }
        })
        .streamHandler(new ChannelInboundHandlerAdapter() {
            @Override
            public void channelRead(ChannelHandlerContext ctx, Object msg) {
                System.out.println(((ByteBuf) msg).toString(StandardCharsets.UTF_8));
                ctx.writeAndFlush(msg);
            }

            @Override
            public boolean isSharable() {
                return true;
            }
        })
        .build();

NioEventLoopGroup group = new NioEventLoopGroup(1);

try {
    Channel channel = new Bootstrap()
            .group(group)
            .channel(NioDatagramChannel.class)
            .handler(codec)
            .bind(8737)
            .sync().channel();

    while (System.in.read() != 'q') {
        Thread.onSpinWait();
    }

    System.err.println("closing");
    channel.close();
} finally {
    group.shutdownGracefully();
}

Example client

QuicSslContext context = QuicSslContextBuilder.forClient()
        .keyManager(new File("client_key.pem"), null, new File("client_certificate.pem")) // try toggling me
        .trustManager(new File("ca_certificate.pem"))
        .applicationProtocols("echo/0.0.1")
        .build();
NioEventLoopGroup group = new NioEventLoopGroup(1);

try {
    ChannelHandler codec = new QuicClientCodecBuilder()
            .sslContext(context)
            .maxIdleTimeout(5, TimeUnit.SECONDS)
            .initialMaxData(10000000)
            .initialMaxStreamDataBidirectionalLocal(1000000)
            .build();

    Channel channel = new Bootstrap()
            .group(group)
            .channel(NioDatagramChannel.class)
            .handler(codec)
            .bind(0)
            .sync()
            .channel();

    QuicChannel.newBootstrap(channel)
            .streamHandler(new ChannelInboundHandlerAdapter())
            .remoteAddress(new InetSocketAddress("localhost", 8737))
            .connect().get()
            .createStream(QuicStreamType.BIDIRECTIONAL, new ChannelInboundHandlerAdapter() {
                @Override
                public void channelActive(ChannelHandlerContext ctx) {
                    ctx.fireChannelActive();

                    ctx.channel().eventLoop().scheduleAtFixedRate(() -> {
                        ctx.channel().writeAndFlush(Unpooled.copiedBuffer("Ping pong", StandardCharsets.UTF_8));
                    }, 0, 1, TimeUnit.SECONDS);
                }

                @Override
                public void channelRead(ChannelHandlerContext ctx, Object msg) {
                    ByteBuf byteBuf = (ByteBuf) msg;
                    System.out.println(byteBuf.toString(StandardCharsets.UTF_8));
                    byteBuf.release();
                }
            })
            .get();

    while (System.in.read() != 'q') {
        Thread.onSpinWait();
    }

    System.err.println("closing");
    channel.close();
} finally {
    group.shutdownGracefully();
}

Sample keying material

ca_certificate.pem

-----BEGIN CERTIFICATE-----
MIIBnjCCAUWgAwIBAgIUF++wS6VuO6SKbDuvxS+BeIrOD3cwCgYIKoZIzj0EAwIw
IDEeMBwGA1UEAwwVQ2VydGlmaWNhdGUgQXV0aG9yaXR5MB4XDTIzMDgyMTEwNTk0
NloXDTI0MDgyMDEwNTk0NlowIDEeMBwGA1UEAwwVQ2VydGlmaWNhdGUgQXV0aG9y
aXR5MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAErQ4ubJWsng+cKdKMypQujxqG
3vZ05oT2s6706JgCmc6hXt9qj0JHQjf2HWTNeI9U+0AJ0pSEEglYAn46W5w2UqNd
MFswHQYDVR0OBBYEFLAgTkykPYhOHZXIlLKsbhUmJqt3MB8GA1UdIwQYMBaAFLAg
TkykPYhOHZXIlLKsbhUmJqt3MAwGA1UdEwQFMAMBAf8wCwYDVR0PBAQDAgGGMAoG
CCqGSM49BAMCA0cAMEQCIBDPAQv0vABoliztfhFkWCIeOHng3HQv08mKao+D6rsR
AiA5+ByWfgUp9dm4HaR6n4jHgtd8eqjmmP1cAqQyBrIXHg==
-----END CERTIFICATE-----

ca_key.pem

-----BEGIN PRIVATE KEY-----
MIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQgpjhdBoWstqSNxyDy
7+rAhy7RTGVm8zQYCOt8EIDP0tShRANCAAStDi5slayeD5wp0ozKlC6PGobe9nTm
hPazrvTomAKZzqFe32qPQkdCN/YdZM14j1T7QAnSlIQSCVgCfjpbnDZS
-----END PRIVATE KEY-----

client_certificate.pem

-----BEGIN CERTIFICATE-----
MIIBmDCCAT2gAwIBAgIULd9kweftX/tneUmkihfohL8dN9kwCgYIKoZIzj0EAwIw
IDEeMBwGA1UEAwwVQ2VydGlmaWNhdGUgQXV0aG9yaXR5MB4XDTIzMDgyMTExMDAw
M1oXDTI0MDgyMDExMDAwM1owETEPMA0GA1UEAwwGQ2xpZW50MFkwEwYHKoZIzj0C
AQYIKoZIzj0DAQcDQgAExTql2B3ucr4DkF5XO8OdO2OHzvzNZOuEFgdm47FdnPjS
kawdz4JFGcY/EvMNWRCcS/vk2bPTI5A9LlhfdbvLkaNkMGIwCwYDVR0PBAQDAgeA
MBMGA1UdJQQMMAoGCCsGAQUFBwMCMB0GA1UdDgQWBBQ1yB4RluY31V4GD/itpc/j
OxOyBjAfBgNVHSMEGDAWgBSwIE5MpD2ITh2VyJSyrG4VJiardzAKBggqhkjOPQQD
AgNJADBGAiEA3nu5IgfbvBnrBRceCpiVR1MFBRRv4ZoZ6PSH9RLaj00CIQDzN9LZ
GJCBS4aZf1p1giOqYJiaOP/+lfL0h3LzUj+UHw==
-----END CERTIFICATE-----

client_key.pem

-----BEGIN PRIVATE KEY-----
MIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQgXotYYkBAqR+CyjxR
LkqkMgLrGJpRg8hzgNYca77908KhRANCAATFOqXYHe5yvgOQXlc7w507Y4fO/M1k
64QWB2bjsV2c+NKRrB3PgkUZxj8S8w1ZEJxL++TZs9MjkD0uWF91u8uR
-----END PRIVATE KEY-----

server_certificate.pem

-----BEGIN CERTIFICATE-----
MIIBljCCAT2gAwIBAgIUZAIrbx4izLQKx30m9a9tN59jw+MwCgYIKoZIzj0EAwIw
IDEeMBwGA1UEAwwVQ2VydGlmaWNhdGUgQXV0aG9yaXR5MB4XDTIzMDgyMTEwNTk1
NVoXDTI0MDgyMDEwNTk1NVowETEPMA0GA1UEAwwGU2VydmVyMFkwEwYHKoZIzj0C
AQYIKoZIzj0DAQcDQgAEE+XH4GL4uvMdsP11Aax89W4uKDGXhb1L9rQzxU75LikH
a9+7FChfpDYi+gKnTgHSHY/lXK8+go27QQiOgk/SO6NkMGIwCwYDVR0PBAQDAgeA
MBMGA1UdJQQMMAoGCCsGAQUFBwMBMB0GA1UdDgQWBBS92IDN4DGO6ka/byXH4YOX
4Lc46DAfBgNVHSMEGDAWgBSwIE5MpD2ITh2VyJSyrG4VJiardzAKBggqhkjOPQQD
AgNHADBEAiB6407JaAZn/H9jsDbNpcCLwwxU45jvWVSNwYVuRpAA2AIgeEdYIH+K
/5fFqqp9U1AN7SxkmjN4MjtJJ279n+2PVU8=
-----END CERTIFICATE-----

server_key.pem

-----BEGIN PRIVATE KEY-----
MIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQgKwRLm4q4RvoefreL
M+yvpHXnxCnB0yshwDebsJriAWOhRANCAAQT5cfgYvi68x2w/XUBrHz1bi4oMZeF
vUv2tDPFTvkuKQdr37sUKF+kNiL6AqdOAdIdj+Vcrz6CjbtBCI6CT9I7
-----END PRIVATE KEY-----

[normanmaurer]

See #567