Warnings at startup when running with --client-cert-auth and --client-cert-allowed-hostname

[eest]

What happened?

When setting up etcd with client certificate authentication there is a warning logged at startup, initially this can be seen:

{"level":"info","ts":"2023-04-21T17:10:23.947694Z","caller":"embed/serve.go:250","msg":"serving client traffic securely","traffic":"grpc+http","address":"[::]:2379"}
{"level":"warn","ts":"2023-04-21T17:10:23.958967Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:49588","server-name":"","error":"tls: failed to verify client certificate: x509: certificate specifies an incompatible key usage"}

After some web crawling I found this issue was related to not having TLS Web Client Authentication on the server certificate: #9785 (comment). However, after fixing this error by updating the server cert I then proceeded to add --client-cert-allowed-hostname=localhost and this resulted in a new warning:

{"level":"info","ts":"2023-04-21T17:19:21.792091Z","caller":"embed/serve.go:250","msg":"serving client traffic securely","traffic":"grpc+http","address":"[::]:2379"}
WARNING: 2023/04/21 17:19:21 [core] grpc: addrConn.createTransport failed to connect to {0.0.0.0:2379 0.0.0.0:2379 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: client certificate authentication failed"
{"level":"warn","ts":"2023-04-21T17:19:21.79548Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:48846","server-name":"","error":"remote error: tls: bad certificate"}

What did you expect to happen?

No errors should be logged during normal operation. At least it should be documented what extensions are expected on the server cert.

How can we reproduce it (as minimally and precisely as possible)?

I will start with reproducing the initial problem, and then updating things to get the second warning.

• Install etcd 3.5.8 as described at https://github.com/etcd-io/etcd/releases/tag/v3.5.8:

ETCD_VER=v3.5.8

# choose either URL
GOOGLE_URL=https://storage.googleapis.com/etcd
GITHUB_URL=https://github.com/etcd-io/etcd/releases/download
DOWNLOAD_URL=${GOOGLE_URL}

rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
rm -rf /tmp/etcd-download-test && mkdir -p /tmp/etcd-download-test

curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
tar xzvf /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /tmp/etcd-download-test --strip-components=1
rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz

/tmp/etcd-download-test/etcd --version
/tmp/etcd-download-test/etcdctl version
/tmp/etcd-download-test/etcdutl version

• Install cfssl and cfssljson

curl -L https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl_1.6.4_linux_amd64 -o /usr/local/bin/cfssl
curl -L https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssljson_1.6.4_linux_amd64 -o /usr/local/bin/cfssljson
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson

• Create a directory for storing stuff

mkdir etcd-lab
cd etcd-lab

• Define CA settings

cat << 'EOF' > ca.json
{
    "ca": {
        "expiry": "87600h"
    },
    "cn": "My Root CA",
    "key": {
        "algo": "ecdsa",
        "size": 384
    }
}
EOF

• Define cfssl config

cat << 'EOF' > cfssl.json
{
    "signing": {
        "profiles": {
            "server": {
                "expiry": "87600h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth"
                ]
            },
            "client": {
                "expiry": "87600h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth"
                ]
            }
        }
    }
}
EOF

• Define server cert

cat << 'EOF' > etcd.json
{
    "cn": "localhost",
    "hosts": [
        "localhost"
    ],
    "key": {
        "algo": "ecdsa",
        "size": 384
    }
}
EOF

• Create CA cert

cfssl gencert -initca ca.json | cfssljson -bare ca

• Create server cert

cfssl gencert -config cfssl.json -profile server -ca ca.pem -ca-key ca-key.pem etcd.json | cfssljson -bare etcd

• Start etcd:

/tmp/etcd-download-test/etcd --cert-file=etcd.pem --key-file=etcd-key.pem --listen-client-urls=https://0.0.0.0:2379 --advertise-client-urls=https://0.0.0.0:2379 --log-outputs=stderr --log-level=info --client-cert-auth=true --trusted-ca-file=ca.pem
[...]
{"level":"info","ts":"2023-04-21T18:31:11.146954Z","caller":"embed/serve.go:250","msg":"serving client traffic securely","traffic":"grpc+http","address":"[::]:2379"}
{"level":"warn","ts":"2023-04-21T18:31:11.156387Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:35898","server-name":"","error":"tls: failed to verify client certificate: x509: certificate specifies an incompatible key usage"}

• Then, to fix this error, add the client authentication capability like this diff shows:

diff -u cfssl.json.orig cfssl.json
--- cfssl.json.orig	2023-04-21 17:29:35.401056421 +0000
+++ cfssl.json	2023-04-21 17:17:43.791443864 +0000
@@ -6,7 +6,8 @@
                 "usages": [
                     "signing",
                     "key encipherment",
-                    "server auth"
+                    "server auth",
+                    "client auth"
                 ]
             },
             "client": {

• Regenerate the cert with the client auth capability:

cfssl gencert -config cfssl.json -profile server -ca ca.pem -ca-key ca-key.pem etcd.json | cfssljson -bare etcd

• Run etcd again, now it does not complain:

/tmp/etcd-download-test/etcd --cert-file=etcd.pem --key-file=etcd-key.pem --listen-client-urls=https://0.0.0.0:2379 --advertise-client-urls=https://0.0.0.0:2379 --log-outputs=stderr --log-level=info --client-cert-auth=true --trusted-ca-file=ca.pem
[...]
{"level":"info","ts":"2023-04-21T17:47:27.712281Z","caller":"embed/serve.go:250","msg":"serving client traffic securely","traffic":"grpc+http","address":"[::]:2379"}

• The above issue seems known based on the mentioned ticket, but after fixing this I have noticed another warning: if i now add --client-cert-allowed-hostname=localhost:

/tmp/etcd-download-test/etcd --cert-file=etcd.pem --key-file=etcd-key.pem --listen-client-urls=https://0.0.0.0:2379 --advertise-client-urls=https://0.0.0.0:2379 --log-outputs=stderr --log-level=info --client-cert-auth=true --trusted-ca-file=ca.pem --client-cert-allowed-hostname=localhost
[...]
{"level":"info","ts":"2023-04-21T18:33:03.454957Z","caller":"embed/serve.go:250","msg":"serving client traffic securely","traffic":"grpc+http","address":"[::]:2379"}
{"level":"warn","ts":"2023-04-21T18:33:03.457835Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:48002","server-name":"","error":"remote error: tls: bad certificate"}
WARNING: 2023/04/21 18:33:03 [core] grpc: addrConn.createTransport failed to connect to {0.0.0.0:2379 0.0.0.0:2379 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: client certificate authentication failed"

• Even with this warning external client cert auth still works, we can verify by creating a client cert like so:

cat << 'EOF' > root.json
{
    "cn": "root",
    "hosts": [
        "localhost"
    ],
    "key": {
        "algo": "ecdsa",
        "size": 384
    }
}
EOF

• Generate the client cert and key:

cfssl gencert -config cfssl.json -profile client -ca ca.pem -ca-key ca-key.pem root.json | cfssljson -bare root

• We can now authenticate using the cert and key:

/tmp/etcd-download-test/etcdctl --cacert ca.pem --cert root.pem --key root-key.pem --endpoints=https://localhost:2379 endpoint health
https://localhost:2379 is healthy: successfully committed proposal: took = 23.619248ms

• For reference: we can verify that the allowed hostname filter works by breaking the root client cert:

diff -u root.json.orig root.json
--- root.json.orig	2023-04-21 18:39:00.475548298 +0000
+++ root.json	2023-04-21 18:39:07.095749546 +0000
@@ -1,7 +1,7 @@
 {
     "cn": "root",
     "hosts": [
-        "localhost"
+        "localhostt"
     ],
     "key": {
         "algo": "ecdsa",

• Generate a new broken cert:

cfssl gencert -config cfssl.json -profile client -ca ca.pem -ca-key ca-key.pem root.json | cfssljson -bare root

• Now the filter kicks in as expected:

/tmp/etcd-download-test/etcdctl --cacert ca.pem --cert root.pem --key root-key.pem --endpoints=https://localhost:2379 endpoint health
{"level":"warn","ts":"2023-04-21T18:40:30.836335Z","logger":"client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000588380/localhost:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection closed before server preface received"}
https://localhost:2379 is unhealthy: failed to commit proposal: context deadline exceeded
Error: unhealthy cluster

• And the server logs:

{"level":"warn","ts":"2023-04-21T18:40:25.852991Z","caller":"embed/config_logging.go:160","msg":"rejected connection","remote-addr":"127.0.0.1:48750","server-name":"localhost","ip-addresses":[],"dns-names":["localhostt"],"error":"client certificate authentication failed"}

... so the filter seems to work as expected for external clients, but for some reason the server always logs that warning message on startup and it is unclear how bad it is, or how I should fix it.

Anything else we need to know?

No response

Etcd version (please run commands below)

/tmp/etcd-download-test/etcd --version
etcd Version: 3.5.8
Git SHA: 217d183e5
Go Version: go1.19.8
Go OS/Arch: linux/amd64

/tmp/etcd-download-test/etcdctl version
etcdctl version: 3.5.8
API version: 3.5

Etcd configuration (command line flags or environment variables)

No response

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

No response

Relevant log output

No response

[ahrtr]

• The above issue seems known based on the mentioned ticket, but after fixing this I have noticed another warning: if i now add --client-cert-allowed-hostname=localhost:

/tmp/etcd-download-test/etcd --cert-file=etcd.pem --key-file=etcd-key.pem --listen-client-urls=https://0.0.0.0:2379 --advertise-client-urls=https://0.0.0.0:2379 --log-outputs=stderr --log-level=info --client-cert-auth=true --trusted-ca-file=ca.pem --client-cert-allowed-hostname=localhost
[...]
{"level":"info","ts":"2023-04-21T18:33:03.454957Z","caller":"embed/serve.go:250","msg":"serving client traffic securely","traffic":"grpc+http","address":"[::]:2379"}
{"level":"warn","ts":"2023-04-21T18:33:03.457835Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:48002","server-name":"","error":"remote error: tls: bad certificate"}
WARNING: 2023/04/21 18:33:03 [core] grpc: addrConn.createTransport failed to connect to {0.0.0.0:2379 0.0.0.0:2379 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: client certificate authentication failed"

Could you disable gRPC gateway and try again? See command below,

 /tmp/etcd-download-test/etcd --cert-file=etcd.pem --key-file=etcd-key.pem --listen-client-urls=https://0.0.0.0:2379 --advertise-client-urls=https://0.0.0.0:2379 --log-outputs=stderr --log-level=info --client-cert-auth=true --trusted-ca-file=ca.pem --client-cert-allowed-hostname=localhost --enable-grpc-gateway=false

[ahrtr]

Also please try command below in two cases: (1) with gRPC gateway enabled, (2) with gRPC gateway disabled.

curl -v --cacert ca.pem --cert root.pem --key root-key.pem -L https://localhost:2379/v3/kv/range -X POST -d '{"key":"L3B1Yi9hYWFh"}'

[eest]

Thank you for looking into this!

With --enable-grpc-gateway=false the startup warning disappears:

/tmp/etcd-download-test/etcd --cert-file=etcd.pem --key-file=etcd-key.pem --listen-client-urls=https://0.0.0.0:2379 --advertise-client-urls=https://0.0.0.0:2379 --log-outputs=stderr --log-level=info --client-cert-auth --trusted-ca-file=ca.pem --client-cert-allowed-hostname=localhost --enable-grpc-gateway=false
[...]
{"level":"info","ts":"2023-04-22T06:14:05.370924Z","caller":"embed/serve.go:250","msg":"serving client traffic securely","traffic":"grpc+http","address":"[::]:2379"}

The requested curl commands, first with gRPC enabled, starting the server:

/tmp/etcd-download-test/etcd --cert-file=etcd.pem --key-file=etcd-key.pem --listen-client-urls=https://0.0.0.0:2379 --advertise-client-urls=https://0.0.0.0:2379 --log-outputs=stderr --log-level=info --client-cert-auth --trusted-ca-file=ca.pem --client-cert-allowed-hostname=localhost
[...]
{"level":"info","ts":"2023-04-22T06:14:46.12944Z","caller":"embed/serve.go:250","msg":"serving client traffic securely","traffic":"grpc+http","address":"[::]:2379"}
WARNING: 2023/04/22 06:14:46 [core] grpc: addrConn.createTransport failed to connect to {0.0.0.0:2379 0.0.0.0:2379 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: client certificate authentication failed"
{"level":"warn","ts":"2023-04-22T06:14:46.131939Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:50658","server-name":"","error":"remote error: tls: bad certificate"}

The curl fails:

curl -v --cacert ca.pem --cert root.pem --key root-key.pem -L https://localhost:2379/v3/kv/range -X POST -d '{"key":"L3B1Yi9hYWFh"}'
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 127.0.0.1:2379...
* Connected to localhost (127.0.0.1) port 2379 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: ca.pem
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=localhost
*  start date: Apr 21 18:27:00 2023 GMT
*  expire date: Apr 18 18:27:00 2033 GMT
*  subjectAltName: host "localhost" matched cert's "localhost"
*  issuer: CN=My Root CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x55b410ce4560)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> POST /v3/kv/range HTTP/2
> Host: localhost:2379
> user-agent: curl/7.81.0
> accept: */*
> content-length: 22
> content-type: application/x-www-form-urlencoded
>
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* We are completely uploaded and fine
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 4294967295)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 503
< access-control-allow-headers: accept, content-type, authorization
< access-control-allow-methods: POST, GET, OPTIONS, PUT, DELETE
< access-control-allow-origin: *
< content-type: application/json
< content-length: 261
< date: Sat, 22 Apr 2023 06:16:41 GMT
<
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection #0 to host localhost left intact
{"error":"connection error: desc = \"transport: authentication handshake failed: client certificate authentication failed\"","code":14,"message":"connection error: desc = \"transport: authentication handshake failed: client certificate authentication failed\""}

Any time I run the failing curl the server logs this, so it seems like it might be triggering the same problem:

{"level":"warn","ts":"2023-04-22T06:17:47.330868Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:43630","server-name":"","error":"remote error: tls: bad certificate"}
WARNING: 2023/04/22 06:17:47 [core] grpc: addrConn.createTransport failed to connect to {0.0.0.0:2379 0.0.0.0:2379 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: client certificate authentication failed"

Restarting the server with gRPC gateway disabled:

/tmp/etcd-download-test/etcd --cert-file=etcd.pem --key-file=etcd-key.pem --listen-client-urls=https://0.0.0.0:2379 --advertise-client-urls=https://0.0.0.0:2379 --log-outputs=stderr --log-level=info --client-cert-auth --trusted-ca-file=ca.pem --client-cert-allowed-hostname=localhost --enable-grpc-gateway=false
[...]
{"level":"info","ts":"2023-04-22T06:25:00.455775Z","caller":"embed/serve.go:250","msg":"serving client traffic securely","traffic":"grpc+http","address":"[::]:2379"}

The curl now results in a 404 instead:

curl -v --cacert ca.pem --cert root.pem --key root-key.pem -L '
Note: Unnecessary use of -X or --request, POST is already inferred.}'
*   Trying 127.0.0.1:2379...
* Connected to localhost (127.0.0.1) port 2379 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: ca.pem
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=localhost
*  start date: Apr 21 18:27:00 2023 GMT
*  expire date: Apr 18 18:27:00 2033 GMT
*  subjectAltName: host "localhost" matched cert's "localhost"
*  issuer: CN=My Root CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x5589674b6560)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> POST /v3/kv/range HTTP/2
> Host: localhost:2379
> user-agent: curl/7.81.0
> accept: */*
> content-length: 22
> content-type: application/x-www-form-urlencoded
>
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* We are completely uploaded and fine
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 4294967295)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 404
< access-control-allow-headers: accept, content-type, authorization
< access-control-allow-methods: POST, GET, OPTIONS, PUT, DELETE
< access-control-allow-origin: *
< content-type: text/plain; charset=utf-8
< x-content-type-options: nosniff
< content-length: 19
< date: Sat, 22 Apr 2023 06:25:14 GMT
<
* TLSv1.2 (IN), TLS header, Supplemental data (23):
404 page not found
* Connection #0 to host localhost left intact

The server logs nothing for those 404 requests.

[ahrtr]

Thanks for the feedback. @eest

The gRPC gateway is used to support REST requests, the rough workflow is something like below. The etcd server's server side certificate is also used by the gRPC gateway when connecting to the backend gRPC server. Most likely the gRPC gateway uses a different hostname (e.g. the output of hostname or a IP address) when connecting to the gRPC server.

client ----> gRPC gateway  ---> gRPC server

for some reason the server always logs that warning message on startup and it is unclear how bad it is, or how I should fix it.

It only affects the gRPC gateway. If you don't need to submit any REST requests, then it's fine. Regarding how to fix it, could you just remove the --client-cert-allowed-hostname?

[eest]

I could remove --client-cert-allowed-hostname, I was just thinking it would be an interesting ACL-like way of stating that only client certs created with a matching SAN name (not localhost in that case, more like etcd-01.example.com) was valid for that given server. I guess you can get the same functionality by using a separate CA per server instead.

Since I only use etcdctl and the go library https://pkg.go.dev/go.etcd.io/etcd/client/v3 for interacting with the server maybe I am better off just using --enable-grpc-gateway=false since neither should need it.

I did notice that 3.5.8 specifically now mentions "traffic":"grpc+http" in its startup message, but this does not change when disabling the gRPC gateway, does this http refer to something else?

[ahrtr]

I guess you can get the same functionality by using a separate CA per server instead.

It isn't a priority for now. We might also change the behavior in future.

I did notice that 3.5.8 specifically now mentions "traffic":"grpc+http" in its startup message, but this does not change when disabling the gRPC gateway, does this http refer to something else?

Please read #15402 (comment). etcd supports specifying separate endpoint for http starting from 3.5.8.

Can we close this ticket?

[eest]

Thanks for taking some time to explain the details of what is going on! I'll close this ticket now.