consul-k8s Connect inject does not support consul with TLS enabled and HTTP disabled.

[bhufsmith]

I spent a bunch of time getting TLS set up by modifying the helm chart to suite my needs, and everything seems to be working with consul itself.

I would like to take advantage of connect-inject, because it is a cool feature, and makes my life a lot easier. Just specify an annotation, and get service level TLS without additional config... that's awesome.

However, after getting consul servers and agents talking with TLS, and gossip encryption this fails to work anymore.

{
  "verify_incoming": true,
  "verify_outgoing": true,
  "verify_server_hostname": true,
  "ca_file": "consul-agent-ca.pem",
  "cert_file": "dc1-client-consul-0.pem",
  "key_file": "dc1-client-consul-0-key.pem",
  "ports": {
    "http": -1,
    "https": 8501
  }
}

Using this config for my clients to enable tls, and disable the HTTP endpoint breaks the connect-inject sidecar.

1. The injection for the sidecar assumes HTTP, as seen below:

export CONSUL_HTTP_ADDR="${HOST_IP}:8500"
export CONSUL_GRPC_ADDR="${HOST_IP}:8502"

# Register the service. The HCL is stored in the volume so that
# the preStop hook can access it to deregister the service.
cat <<EOF >/consul/connect-inject/service.hcl
services {
  id   = "${POD_NAME}-{{ .ServiceName }}-proxy"
  name = "{{ .ServiceName }}-proxy"
  kind = "connect-proxy"
  address = "${POD_IP}"
  port = 20000
  proxy {
    destination_service_name = "{{ .ServiceName }}"
    destination_service_id = "{{ .ServiceName}}"
    {{ if (gt .ServicePort 0) -}}
    local_service_address = "127.0.0.1"
    local_service_port = {{ .ServicePort }}
    {{ end -}}
    {{ range .Upstreams -}}
    upstreams {
      {{- if .Name }}
      destination_type = "service" 
      destination_name = "{{ .Name }}"
      {{- end}}
      {{- if .Query }}
      destination_type = "prepared_query" 
      destination_name = "{{ .Query}}"
      {{- end}}
      local_bind_port = {{ .LocalPort }}
      {{- if .Datacenter }}
      datacenter = "{{ .Datacenter }}"
      {{- end}}
    }
    {{ end }}
  }
  checks {
    name = "Proxy Public Listener"
    tcp = "${POD_IP}:20000"
    interval = "10s"
    deregister_critical_service_after = "10m"
  }
  checks {
    name = "Destination Alias"
    alias_service = "{{ .ServiceName }}"
  }
}
EOF
/bin/consul services register /consul/connect-inject/service.hcl
# Generate the envoy bootstrap code
/bin/consul connect envoy \
  -proxy-id="${POD_NAME}-{{ .ServiceName }}-proxy" \
  -bootstrap > /consul/connect-inject/envoy-bootstrap.yaml
# Copy the Consul binary
cp /bin/consul /consul/connect-inject/consul

2. There is no way for me to override that configuration, or modify it without just writing my own init container.

See error below:

NAME                            READY   STATUS                  RESTARTS   AGE
sample-nginx-68c7d9bdd7-p6fxt   0/2     Init:CrashLoopBackOff   9          25m

$ kubectl logs sample-nginx-68c7d9bdd7-p6fxt consul-connect-inject-init
Error registering service "nginx-proxy": Put http://<ip-redacted>:8500/v1/agent/service/register: dial tcp <ip redacted>:8500: connect: connection refused

I would like the ability to tell connect inject that it needs to use a cert, key, and ca from a secret, like I can do for the consul servers, and clients. That way, I can set up some certificates that the init container can use when registering the service, without having to enable HTTP connections on my consul agents.

At minimum, it would be more secure if all I had to do was disable "verify_incoming_https" on my consul agents, and specify that I want an HTTPS connection, and which port to use.

Maybe there is a better solution, but I figured I would put this here so it can be discussed.

Edit:

If it was not obvious from above, I am running consul in Kubernetes, using the helm chart to deploy.
I am running - kubernetes 1.11.
Consul 1.4.2
Consul-k8s 0.6

The above init config can be seen hardcoded here : https://github.com/hashicorp/consul-k8s/blob/master/connect-inject/container_init.go

[bhufsmith]

Looks like there is a pull request that attempts to solve this issue: #30

[stormmore]

It isn't just an issue with connect. It also stops the ACL init job from working even if I set CONSUL_HTTP_SSL=true. I believe the issue is with:

consul-k8s/subcommand/server-acl-init/command.go

Line 144 in bccf2e2

Scheme: "http",

I suspect that it might be hard coded else where too.

[soniagupta9]

I am facing the same issue, I have the following configuration:

{
      "verify_incoming": true,
      "verify_outgoing": true,
      "verify_server_hostname": false,
      "ca_file": "/consul/certs/ca.pem",
      "cert_file": "/consul/certs/cert.pem",
      "key_file": "/consul/certs/key.pem",
      "ports": {
        "http": 8500,
        "https": 8501
      }
    }

but I still see following error from the consul-connect sidecar:

2019/08/21 22:33:01 [WARN] agent: Check "service:test-serivce-sidecar-proxy:1" socket connection failed: dial tcp 172.24.0.211:20000: connect: connection refused
    2019/08/21 22:33:02 [WARN] grpc: Server.Serve failed to complete security handshake from "172.24.0.207:54798": tls: first record does not look like a TLS handshake

Is #30 going to be merged anytime soon? Is there any alternative solution to the above problem?