Proxy doesn't respect peer-auto-tls or auto-tls

[micahhausler]

I've got a 3-node cluster running with --auto-tls and --peer-auto-tls, but my proxies configured with --auto-tls and --peer-auto-tls are still trying to validate the cluster's certs.

Environment

Host OS: Container Linux by CoreOS 1353.7.0
Etcd version: 3.1.7
Cloud: AWS

Peer Setup

coreos:
  units:
    - name: etcd3-peers.service
      command: start
      content: |
          [Unit]
          Description=Write a file with the etcd peers that we should bootstrap to\n",
          Requires=docker.socket
          After=docker.socket
          [Service]
          Restart=on-failure
          RestartSec=10
          ExecStartPre=-/usr/bin/systemctl stop etcd-member
          ExecStartPre=/usr/bin/docker pull monsantoco/etcd-aws-cluster:latest
          ExecStartPre=/usr/bin/docker run \
              --rm=true \
              --net host \
              -v /etc/sysconfig/:/etc/sysconfig/ \
              -e ETCD_CLIENT_SCHEME=https \
              -e ETCD_PEER_SCHEME=https \
              -e PROXY_ASG=${etcd_asg_name} \
              monsantoco/etcd-aws-cluster:latest
          ExecStart=/usr/bin/systemctl start etcd-member.service
    - name: etcd-member.service
      command: stop
      enable: true
      drop-ins:
        - name: 10-etcd-member.conf
          content: |
            [Service]
            EnvironmentFile=/etc/sysconfig/etcd-peers
            Environment="ETCD_IMAGE_TAG=v3.1.7"
            Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://$private_ipv4:2380"
            Environment="ETCD_LISTEN_PEER_URLS=https://$private_ipv4:2380"
            Environment="ETCD_ADVERTISE_CLIENT_URLS=https://$private_ipv4:2379,https://127.0.0.1:2379"
            Environment="ETCD_LISTEN_CLIENT_URLS=https://$private_ipv4:2379,https://127.0.0.1:2379"
            Environment="ETCD_PEER_AUTO_TLS=true"
            Environment="ETCD_AUTO_TLS=true"
            Environment="ETCD_DATA_DIR=/opt/etcd/data"
            Environment="ETCD_PROXY=on"
            EnvironmentFile=/etc/sysconfig/etcd-peers 
            ExecStartPre=/usr/bin/mkdir -p /opt/etcd/data
            ExecStartPre=/usr/bin/chown -R etcd:etcd /opt/etcd/data

/etc/sysconfig/etcd-peers

ETCD_INITIAL_CLUSTER_STATE=new
ETCD_NAME=i-055cc1e60a4435bda
ETCD_INITIAL_CLUSTER="i-0a54cf28ed2ad93d6=https://172.16.80.96:2380,i-0535ae47f59b26423=https://172.16.48.66:2380,i-0eb2c20a097c0367e=https://172.16.70.222:2380"

Logs

etcd-wrapper[2208]: 2017-05-15 19:14:33.433485 I | pkg/flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=https://172.16.91.110:2379,https://127.0.0.1:2379
etcd-wrapper[2208]: 2017-05-15 19:14:33.433562 I | pkg/flags: recognized and used environment variable ETCD_AUTO_TLS=true
etcd-wrapper[2208]: 2017-05-15 19:14:33.433585 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd
etcd-wrapper[2208]: 2017-05-15 19:14:33.433612 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=https://172.16.91.110:2380
etcd-wrapper[2208]: 2017-05-15 19:14:33.433622 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=i-0a54cf28ed2ad93d6=https://172.16.80.96:2380,i-0535ae47f59b26423=https://172.16.48.66:2380,i-0eb2c20a097c0367e=https://172.16.70.222:2380
etcd-wrapper[2208]: 2017-05-15 19:14:33.433636 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new
etcd-wrapper[2208]: 2017-05-15 19:14:33.433659 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=https://172.16.91.110:2379,https://127.0.0.1:2379
etcd-wrapper[2208]: 2017-05-15 19:14:33.433673 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=https://172.16.91.110:2380
etcd-wrapper[2208]: 2017-05-15 19:14:33.433693 I | pkg/flags: recognized and used environment variable ETCD_NAME=i-055cc1e60a4435bda
etcd-wrapper[2208]: 2017-05-15 19:14:33.433702 I | pkg/flags: recognized and used environment variable ETCD_PEER_AUTO_TLS=true
etcd-wrapper[2208]: 2017-05-15 19:14:33.433721 I | pkg/flags: recognized and used environment variable ETCD_PROXY=on
etcd-wrapper[2208]: 2017-05-15 19:14:33.433759 W | pkg/flags: unrecognized environment variable ETCD_USER=etcd
etcd-wrapper[2208]: 2017-05-15 19:14:33.433768 W | pkg/flags: unrecognized environment variable ETCD_IMAGE_TAG=v3.1.7
etcd-wrapper[2208]: 2017-05-15 19:14:33.433794 I | etcdmain: etcd Version: 3.1.7
etcd-wrapper[2208]: 2017-05-15 19:14:33.433801 I | etcdmain: Git SHA: 43b7507
etcd-wrapper[2208]: 2017-05-15 19:14:33.433806 I | etcdmain: Go Version: go1.7.5
etcd-wrapper[2208]: 2017-05-15 19:14:33.433811 I | etcdmain: Go OS/Arch: linux/amd64
etcd-wrapper[2208]: 2017-05-15 19:14:33.433817 I | etcdmain: setting maximum number of CPUs to 2, total number of available CPUs is 2
etcd-wrapper[2208]: 2017-05-15 19:14:33.433873 N | etcdmain: proxy: this proxy supports v2 API only!
etcd-wrapper[2208]: 2017-05-15 19:14:33.433932 I | etcdmain: proxy: using peer urls [https://172.16.48.66:2380 https://172.16.70.222:2380 https://172.16.80.96:2380]
etcd-wrapper[2208]: 2017-05-15 19:14:33.468377 W | etcdserver: could not get cluster response from https://172.16.48.66:2380: Get https://172.16.48.66:2380/members: x509: certificate signed by unknown authority
etcd-wrapper[2208]: 2017-05-15 19:14:33.483373 W | etcdserver: could not get cluster response from https://172.16.70.222:2380: Get https://172.16.70.222:2380/members: x509: certificate signed by unknown authority
etcd-wrapper[2208]: 2017-05-15 19:14:33.498446 W | etcdserver: could not get cluster response from https://172.16.80.96:2380: Get https://172.16.80.96:2380/members: x509: certificate signed by unknown authority
etcd-wrapper[2208]: 2017-05-15 19:14:33.498640 W | etcdmain: proxy: could not retrieve cluster information from the given urls
etcd-wrapper[2208]: 2017-05-15 19:14:33.498715 C | etcdmain: cannot listen on TLS for https://127.0.0.1:2379: KeyFile and CertFile are not presented
systemd[1]: etcd-member.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: Failed to start etcd (System Application Container).
systemd[1]: etcd-member.service: Unit entered failed state.
systemctl[2167]: Job for etcd-member.service failed because the control process exited with error code.
systemctl[2167]: See "systemctl  status etcd-member.service" and "journalctl  -xe" for details.
systemd[1]: etcd-member.service: Failed with result 'exit-code'.
systemd[1]: etcd3-peers.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: etcd3-peers.service: Unit entered failed state.
systemd[1]: etcd3-peers.service: Failed with result 'exit-code'.

Curl response (from the proxy)

$ curl -s https://172.16.70.222:2380/members -k | jq
[
  {
    "id": 12081281743585325000,
    "peerURLs": [
      "https://172.16.48.66:2380"
    ],
    "name": "i-0535ae47f59b26423",
    "clientURLs": [
      "https://127.0.0.1:2379",
      "https://172.16.48.66:2379"
    ]
  },
  {
    "id": 15733603985652122000,
    "peerURLs": [
      "https://172.16.80.96:2380"
    ],
    "name": "i-0a54cf28ed2ad93d6",
    "clientURLs": [
      "https://127.0.0.1:2379",
      "https://172.16.80.96:2379"
    ]
  },
  {
    "id": 16092623858802352000,
    "peerURLs": [
      "https://172.16.70.222:2380"
    ],
    "name": "i-0eb2c20a097c0367e",
    "clientURLs": [
      "https://127.0.0.1:2379",
      "https://172.16.70.222:2379"
    ]
  }
]

[micahhausler]

It looks to me like part of the problem is in the proxy listener: https://github.com/coreos/etcd/blob/release-3.1/etcdmain/etcd.go#L312-L318.

I'd like to take a stab at this, should I make a branch off of the release-3.1 branch?

[gyuho]

but my proxies configured with --auto-tls and --peer-auto-tls are still trying to validate the cluster's certs.

This is expected? v2 proxy just forwards request to cluster. And if the cluster enabled client TLS, v2 proxy would require the certs of the cluster, to boot.

etcdmain: cannot listen on TLS for https://127.0.0.1:2379: KeyFile and CertFile are not presented

Auto TLS was added in v3, and the httpproxy is from v2, so seems like we are missing documentation on this.

Try starting the proxy process with cluster's peer certs?
Auto TLS just auto generates but does not do any auto configuration.

[micahhausler]

OK, maybe my understanding is off.

I've got a 3-node cluster with auto-tls and peer-auto-tls, so I don't have certs I can hand over to my proxies.

It seems to me like there are two issues:

1. A proxy started with --peer-auto-tls should be able to make requests to the masters and not worry about the certificates.

2. A proxy started with --auto-tls should generate its own certs, which its clearly not doing:

   etcdmain: cannot listen on TLS for https://127.0.0.1:2379: KeyFile and CertFile are not presented
   

It sounds like the proxy is v2 only, and I'm guessing that v2 requests cannot be upgraded to v3, so the above points are moot?

Is there any plan for a v3 proxy? Or do v3 clients have to talk to the master?

[heyitsanthony]

I think this needs something like --insecure-skip-verify-tls if passing in --auto-tls to the proxy?

[xiang90]

@heyitsanthony Agree.

[xiang90]

@micahhausler would you like to send a pr to get this fixed?

[micahhausler]

@xiang90 Yea, based on the previous comment I understood that it was expected behavior. I'd be happy to make v2 requests respect a --insecure-skip-verify-tls with --auto-tls.

[YeruchamB]

Hey I'm having similar problems..

I'm currently just trying to run a simple single-node cluster with one proxy as a POC.
I need the traffic between the proxy and server to be encrypted but not between the client and the proxy.

Cluster configuration:
#!/usr/bin/env bash

THIS_IP="$1"
THIS_NAME=infra-${THIS_IP}

TOKEN=token-02
CLUSTER_STATE=new
etcd --data-dir=data.etcd --name ${THIS_NAME}
--auto-tls --peer-auto-tls
--initial-advertise-peer-urls https://${THIS_IP}:2380 --listen-peer-urls https://${THIS_IP}:2380
--advertise-client-urls https://${THIS_IP}:2379 --listen-client-urls https://${THIS_IP}:2379
--discovery https://discovery.etcd.io/48f750c4f2254d71e7726b45abef4379
--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

Proxy Configuration:
etcd --proxy on
--listen-client-urls http://127.0.0.1:2379
--peer-auto-tls
--discovery https://discovery.etcd.io/48f750c4f2254d71e7726b45abef4379 \

Trying to run a simple etcdctl command results in the following error:
ETCDCTL_API=2 ./etcdctl member list
2018-02-04 09:52:57.871465 I | proxy/httpproxy: failed to direct request to https://10.40.4.112:2379: x509: certificate signed by unknown authority
2018-02-04 09:52:57.871492 I | proxy/httpproxy: marked endpoint https://10.40.4.112:2379 unavailable
2018-02-04 09:52:57.871513 I | proxy/httpproxy: unable to get response from 1 endpoint(s)
2018-02-04 09:52:57.872778 I | proxy/httpproxy: zero endpoints currently available
client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 127.0.0.1:4001: getsockopt: connection refused
; error #1: client: etcd member http://127.0.0.1:2379 has no leader