Please provide an example of extraConfig so TLS can work #74

vanderhoofen · 2018-11-30T01:14:29Z

Lines 70 to 73 in ec0de41

    
             # extraConfig is a raw string of extra configuration to set with the 
        
             # server. This should be JSON. 
        
             extraConfig: | 
        
               {}

How exactly should client.extraConfig and client.extraVolumes be set so that we can have TLS enabled from Consul client to a Consul cluster outside of kubernetes?

Could you provide examples?

Trial and error formatting this 'raw JSON payload' via "helm --set client.extraConfig" isn't working out

We want to not have to customize values.yaml by hand but rather just use the chart

woz5999 · 2018-12-06T19:12:19Z

Can you provide an example of the whole helm --set command? It should be sufficient to set that value as a json string wrapped in single quotes.

vanderhoofen · 2018-12-06T20:41:01Z

Hi Jeff,

Sure.

There's a couple challenges with this consul-helm chart.

Missing an easy way to import / utilize existing TLS certs to join a k8s instance with existing consul cluster outside of k8s (must be created manually)
The chart requires significant manual clone + edit operations to get started with anything missing in the chart (passed in via extraConfig + extraVolumes)

Example helm install:

First manually create Kubernetes Secret (consul-certs) containing the 3 TLS Certs + Gossip Key required for "in-k8s" Consul to speak with external Consul cluster. (not shown)

`

git clone https://github.com/hashicorp/consul-helm.git && cd consul-helm

helm install --name consul .
--namespace consul
--set "global.enabled=false"
--set "global.datacenter=noc"
--set "global.image=consul:1.4.0"
--set "server.enabled=false"
--set "dns.enabled=true"
--set "syncCatalog.enabled=true"
--set "client.enabled=true"
--set 'client.join[0]=192.168.250.102'
--set 'client.join[1]=192.168.250.103'
--set 'client.join[2]=192.168.250.104'
--set 'client.join[3]=192.168.250.105'
--set 'client.join[4]=192.168.250.106'
--set "client.extraVolumes[0].type=secret"
--set "client.extraVolumes[0].name=consul-certs"
--set client.extraConfig='{
"ca_path": "/consul/userconfig/consul-certs",
"ca_file": "/consul/userconfig/consul-certs/ca.pem",
"cert_file": "/consul/userconfig/consul-certs/consul-client.pem",
"key_file": "/consul/userconfig/consul-certs/consul-client-key.pem"
}'
`

Helm fails with:
Error: render error in "consul/templates/client-config-configmap.yaml": template: consul/templates/client-config-configmap.yaml:15:14: executing "consul/templates/client-config-configmap.yaml" at <.Values.client.extra...>: wrong type for value; expected string; got []interface {}

This is probably because Helm (go) insists this value in "--set" parameters must be a raw string (albeit JSON). Single quoting it does not protect Helm from choking on the string. I've tried various quoting and escaping methods to no avail.

Bottom line: There seems to be no way to configure TLS or anything else in extraConfig w/o doing a bunch of smelly manual operations (clone, edit, check, install?)

Could you provide an example where this works via helm install w/o unsightly values.yaml edits?

Thanks,

woz5999 · 2018-12-06T21:41:31Z

You might try --set-string or --set-file for client.extraConfig instead of --set.
https://github.com/technosophos/k8s-helm/blob/master/docs/using_helm.md

vanderhoofen · 2018-12-07T18:56:17Z

--set-string fails with the same error:
Error: render error in "consul/templates/client-config-configmap.yaml": template: consul/templates/client-config-configmap.yaml:15:14: executing "consul/templates/client-config-configmap.yaml" at <.Values.client.extra...>: wrong type for value; expected string; got []interface {}

--set-file requires more custom file operations instead of just tuning --set params

adilyse · 2019-03-11T19:00:22Z

Hi @vanderhoofen,

I've just merged a fix for how we interpret extraConfig which should enable setting it from the command line. From that PR description:

This enables users to specify extraConfig values using Helm's --set CLI
flag while installing the Helm chart. The format for using this command is:

--set 'client.extraConfig="{"log_level": "DEBUG"}"'

Please feel free to open a new issue if this doesn't solve the issue for you.

pgbezerra · 2019-04-05T14:46:00Z

Well, I solved this in a easier way.
You don't need to write down your config with --set client.extraConfig.
This chart deploys a configmap named consul-client-config.

Just change the key extra-from-values.json in consul-client-config config map with your file and voilá!

my-values.yml:

---
  client: 
    enabled: "true"
    extraVolumes: 
      - name: "consul-secret"
        load: "false"
        type: "secret"
    join: 
      - "xxx.com"
      - "yyy.com"
  dns: 
    enabled: "true"
  global: 
    datacenter: "datacenter1"
    domain: "mydomain"
    enabled: "false"

config map consul-client-config key: extra-from-values.json:

{
    "verify_outgoing": true,
    "verify_incoming": true,
    "verify_server_hostname": true,
    "ca_file": "/consul/userconfig/consul-secret/ca",
    "key_file": "/consul/userconfig/consul-secret/key",
    "cert_file": "/consul/userconfig/consul-secret/cert",
    "primary_datacenter": "datacenter1",
    "acl_down_policy": "extend-cache",
    "acl_default_policy": "allow",
    "acl_agent_token": "14253987253985329853298",
    "acl_agent_master_token": "14267473457345738"
}
--

vanderhoofen · 2019-04-05T14:51:18Z

Hi @vanderhoofen,

I've just merged a fix for how we interpret extraConfig which should enable setting it from the command line. From that PR description:

This enables users to specify extraConfig values using Helm's --set CLI
flag while installing the Helm chart. The format for using this command is:
--set 'client.extraConfig="{"log_level": "DEBUG"}"'
Please feel free to open a new issue if this doesn't solve the issue for you.

This is great. Thanks

svlady · 2019-05-13T13:06:44Z

There is a bit more needed than just providing the extraConfig section. The following steps working for me:

Generating certificates and placing them to k8s secrets. There are multiple ways for doing this and I have opted to use own consul helpers:

$ consul tls ca create
$ consul tls cert create -server -additional-dnsname=*.c01-consul-server.devops.svc

$ kubectl create secret generic consul-server-certs \
  --from-file=ca.pem=consul-agent-ca.pem \
  --from-file=consul.pem=dc1-server-consul-0.pem \
  --from-file=consul-key.pem=dc1-server-consul-0-key.pem

$ consul tls cert create -client

$ kubectl create secret generic consul-client-certs \
  --from-file=ca.pem=consul-agent-ca.pem \
  --from-file=consul.pem=dc1-client-consul-0.pem \
  --from-file=consul-key.pem=dc1-client-consul-0-key.pem

Adjusting health probes. It's possible to get curl commands to work either, but using consul own probes is a bit more straightforward thanks to env variables. For example in server-statefulset.yaml

...
          livenessProbe:
            exec:
              command:
                - consul
                - operator
                - raft
                - list-peers
            failureThreshold: 2
            initialDelaySeconds: 5
            periodSeconds: 3
            successThreshold: 1
            timeoutSeconds: 5
          readinessProbe:
            exec:
              command:
                - consul
                - members
            failureThreshold: 2
            initialDelaySeconds: 5
            periodSeconds: 3
            successThreshold: 1
            timeoutSeconds: 5
...

and the client-daemonset.yaml can use a similar check.

Now, the following additional values will fill the gaps and glue it all together:

global:
  enabled: true
  domain: consul
  datacenter: dc1
  gossipEncryption:
    secretName: consul
    secretKey: gossip-encryption-key

server:
  enabled: true
  replicas: 3
  bootstrapExpect: 3

  affinity: |
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app: {{ template "consul.name" . }}
              release: "{{ .Release.Name }}"
              component: server
          topologyKey: failure-domain.beta.kubernetes.io/zone

  extraConfig: |
    {
      "ui": true,
      "client_addr": "0.0.0.0",
      "enable_script_checks": false,
      "disable_remote_exec": true,
      "verify_incoming": true,
      "verify_outgoing": true,
      "verify_server_hostname": true,
      "ca_file": "/consul/userconfig/consul-server-certs/ca.pem",
      "cert_file": "/consul/userconfig/consul-server-certs/consul.pem",
      "key_file": "/consul/userconfig/consul-server-certs/consul-key.pem",
      "ports": {
        "http": -1,
        "https": 8500
      }
    }

  extraEnvironmentVars: 
    CONSUL_CACERT: /consul/userconfig/consul-server-certs/ca.pem
    CONSUL_CLIENT_CERT: /consul/userconfig/consul-server-certs/consul.pem
    CONSUL_CLIENT_KEY: /consul/userconfig/consul-server-certs/consul-key.pem
    CONSUL_HTTP_SSL: true
    # CONSUL_HTTP_ADDR: https://localhost:8501

  extraVolumes:
    - type: secret
      name: consul-server-certs
      load: false

client:
  enabled: true
  grpc: true

  extraConfig: |
    {
      "verify_incoming": true,
      "verify_outgoing": true,
      "verify_server_hostname": true,
      "ca_file": "/consul/userconfig/consul-client-certs/ca.pem",
      "cert_file": "/consul/userconfig/consul-client-certs/consul.pem",
      "key_file": "/consul/userconfig/consul-client-certs/consul-key.pem",
      "ports": {
        "http": -1,
        "https": 8500
      }
    }

  extraEnvironmentVars: 
    CONSUL_CACERT: /consul/userconfig/consul-client-certs/ca.pem
    CONSUL_CLIENT_CERT: /consul/userconfig/consul-client-certs/consul.pem
    CONSUL_CLIENT_KEY: /consul/userconfig/consul-client-certs/consul-key.pem
    CONSUL_HTTP_SSL: true
    # CONSUL_HTTP_ADDR: https://localhost:8501

  extraVolumes:
    - type: secret
      name: consul-client-certs
      load: false

dns:
  enabled: true

ui:
  enabled: true
  service:
    enabled: true
    type: NodePort

NB: the affinity rules are not essential here, I have had them in place due to multi-zone cluster setup.

Keep in mind that this configuration is running HTTPS on the same 8500 port, instead of HTTP. Normally, in production you would want to use HTTPS, so this adjustment is fine. If for some reason you need to keep the both HTTP and HTTPS ports available, you can use 8501 for HTTPS and uncomment CONSUL_HTTP_ADDR env variable to point all the tools to the proper API endpoint.

bsakweson · 2019-12-02T15:21:24Z

I know this thread is old and has been closed but this is the only thread in here that deals with TLS for this chart. It seems as if the consul syncCatalog is designed to work without tls. Once tls is enabled on consul server and client (Agent), syncCatalog seems to stop working. Has anyone been able to get it to work correctly?

HofmannZ · 2021-01-25T14:28:03Z

Is this still the prefered way for consul/vault on k8s with tls?

lkysow · 2021-01-25T17:20:12Z

@HofmannZ this ticket is super old so perhaps opening a new one with your specific use-case would be better. Quickly, turning on TLS can be done with:

global:
  tls:
    enabled: true

adilyse added the documentation Request for further documentation label Dec 4, 2018

adilyse mentioned this issue Mar 6, 2019

Remove quotes from extraConfig inputs #125

Merged

adilyse closed this as completed in #125 Mar 11, 2019

lkysow mentioned this issue Dec 2, 2019

Automatic Secure Pod Communication #294

Closed

bsakweson mentioned this issue Dec 5, 2019

Chart Documentation #300

Closed

HofmannZ mentioned this issue Jan 25, 2021

Consul and Vault with TLS on Kubernetes #784

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Please provide an example of extraConfig so TLS can work #74

Please provide an example of extraConfig so TLS can work #74

vanderhoofen commented Nov 30, 2018 •

edited

Loading

woz5999 commented Dec 6, 2018

vanderhoofen commented Dec 6, 2018 •

edited

Loading

woz5999 commented Dec 6, 2018

vanderhoofen commented Dec 7, 2018

adilyse commented Mar 11, 2019

pgbezerra commented Apr 5, 2019

vanderhoofen commented Apr 5, 2019

svlady commented May 13, 2019

bsakweson commented Dec 2, 2019

HofmannZ commented Jan 25, 2021

lkysow commented Jan 25, 2021

Please provide an example of extraConfig so TLS can work #74

Please provide an example of extraConfig so TLS can work #74

Comments

vanderhoofen commented Nov 30, 2018 • edited Loading

woz5999 commented Dec 6, 2018

vanderhoofen commented Dec 6, 2018 • edited Loading

woz5999 commented Dec 6, 2018

vanderhoofen commented Dec 7, 2018

adilyse commented Mar 11, 2019

pgbezerra commented Apr 5, 2019

vanderhoofen commented Apr 5, 2019

svlady commented May 13, 2019

bsakweson commented Dec 2, 2019

HofmannZ commented Jan 25, 2021

lkysow commented Jan 25, 2021

vanderhoofen commented Nov 30, 2018 •

edited

Loading

vanderhoofen commented Dec 6, 2018 •

edited

Loading