Automate DNS redirection to Consul DNS #833

ndhanushkodi · 2021-11-03T07:38:45Z

Changes proposed in this PR:

When DNS is enabled, get the DNS Service IP and redirect DNS requests to the Consul DNS service.
Add kubedns as a recursor for Consul DNS

Corresponding consul PR: hashicorp/consul#11480

How I've tested this PR:

helm install -n consul consul ../charts/consul --set connectInject.enabled=true --set global.image=ashwinvenkatesh/consul@sha256:d02bf8201499a0a6be82fdf279bb1c5f9d2bc1eda2c5fac47ca832c879444292 --set global.imageK8S=ashwinvenkatesh/consul-k8s@sha256:ce817f69f109e7dcb27ba01f1cad29fcfedeff5ed424db21fa095e0e3d582fe9 --set dns.enableRedirection=true --set controller.enabled=true

k apply -f demo.yaml #included in this PR for testing, remove before merging

Then, exec onto the frontend pod, and

curl backend

nslookup backend.service.consul

and verify it returns a pod ip for backend. We can't curl backend.service.consul because envoy only knows to direct traffic based on VIPs not pod ips

How I expect reviewers to test this PR:

Code Review
Re-try above steps

Checklist:

Tests added
CHANGELOG entry added

HashiCorp engineers only, community PRs should not add a changelog entry.
Entries should use present tense (e.g. Add support for...)

charts/consul/templates/client-dns-clusterrole.yaml

charts/consul/templates/client-dns-clusterrolebinding.yaml

charts/consul/templates/client-daemonset.yaml

charts/consul/templates/connect-inject-deployment.yaml

ndhanushkodi · 2021-11-03T07:58:05Z

demo.yaml

+        app: frontend
+      annotations:
+        "consul.hashicorp.com/connect-inject": "true"
+        "consul.hashicorp.com/transparent-proxy-exclude-outbound-cidrs": "151.101.2.133,151.101.66.133,151.101.130.133,151.101.194.133"


will let us apk update && apk add stuff

Force delete pods that are still running after `helm uninstall` so the next installation can begin immediately.

ishustava

great work Ashwin and Nitya! I've left some comments/questions inline.

charts/consul/templates/client-daemonset.yaml

charts/consul/templates/server-statefulset.yaml

charts/consul/test/unit/connect-inject-deployment.bats

charts/consul/values.yaml

demo.yaml

lkysow

Still looking but wanted to get feedback on my question about namespaces.

charts/consul/templates/_helpers.tpl

charts/consul/templates/client-daemonset.yaml

charts/consul/templates/connect-inject-deployment.yaml

control-plane/connect-inject/container_init.go

charts/consul/values.yaml

lkysow

Looks mostly good but wondering about how we're finding the ClusterIP.

Also for acceptance tests what are we thinking? Is this gonna be a separate PR?

control-plane/connect-inject/container_init.go

lkysow · 2021-11-09T23:41:58Z

control-plane/connect-inject/container_init.go

+
+	var consulDNSIP string
+	if dnsEnabled {
+		for _, e := range os.Environ() {


I found this kinda confusing. I think it could use a comment.

control-plane/connect-inject/container_init.go

thisisnotashwin · 2021-11-10T01:03:06Z

Also for acceptance tests what are we thinking? Is this gonna be a separate PR?

@lkysow Acceptance tests are a great question. The approach I was considering for acceptance testing this change was indirectly using the changes the will be added for transparent proxy for x-partition communication. I will update the fixtures to support transparent proxying the upstream which should test the usage of Consul DNS we ultimately care about.

lkysow

I think we can clean up the env var things now?

control-plane/connect-inject/container_init.go

lkysow · 2021-11-10T17:25:59Z

control-plane/connect-inject/handler.go

@@ -138,6 +138,10 @@ type Handler struct {
 	// from mesh services.
 	EnableConsulDNS bool

+	// ReleasePrefix is the prefix used for the installation which is used to determine the Service
+	// name of the Consul DNS service.
+	ReleasePrefix string


Can you use ResourcePrefix to match server-acl-init, create-federation-job and others

control-plane/subcommand/inject-connect/command.go

control-plane/connect-inject/container_init.go

charts/consul/templates/connect-inject-deployment.yaml

lkysow · 2021-11-10T17:54:16Z

charts/consul/test/unit/connect-inject-deployment.bats

+@test "connectInject/Deployment: -release-prefix unset by default" {
+  cd `chart_dir`
+  local actual=$(helm template \
+      -s templates/connect-inject-deployment.yaml \
+      --set 'connectInject.enabled=true' \
+      . | tee /dev/stderr |
+      yq -c -r '.spec.template.spec.containers[0].command | join(" ") | contains("-release-prefix=RELEASE-NAME-consul")' | tee /dev/stderr)
+  [ "${actual}" = "false" ]
+}
+
+@test "connectInject/Deployment: -release-prefix is true if dns.enabled=true and dns.enableRedirection=true" {
+  cd `chart_dir`
+  local actual=$(helm template \
+      -s templates/connect-inject-deployment.yaml \
+      --set 'connectInject.enabled=true' \
+      --set 'dns.enableRedirection=true' \
+      . | tee /dev/stderr |
+      yq -c -r '.spec.template.spec.containers[0].command | join(" ") | contains("-release-prefix=RELEASE-NAME-consul")' | tee /dev/stderr)
+  [ "${actual}" = "true" ]
+}


don't need these now

lkysow · 2021-11-10T17:55:11Z

control-plane/connect-inject/container_init.go

+		// If Consul DNS is enabled, we find the environment variable that has the value
+		// of the ClusterIP of the Consul DNS Service. constructDNSServiceHostName returns
+		// the name of the env variable whose value is the ClusterIP of the Consul DNS Service.
+		consulDNSClusterIP = os.Getenv(h.constructDNSServiceHostName())


this should error out if the env var is not found

control-plane/connect-inject/container_init.go

Tested with this branch and consul branch `dns-redirect-to-consul`.

Co-authored-by: Luke Kysow <[email protected]>

ndhanushkodi commented Nov 3, 2021

View reviewed changes

charts/consul/templates/client-dns-clusterrole.yaml Outdated Show resolved Hide resolved

ndhanushkodi commented Nov 3, 2021

View reviewed changes

charts/consul/templates/client-dns-clusterrolebinding.yaml Outdated Show resolved Hide resolved

ndhanushkodi marked this pull request as draft November 3, 2021 07:52

ndhanushkodi commented Nov 3, 2021

View reviewed changes

charts/consul/templates/client-daemonset.yaml Outdated Show resolved Hide resolved

ndhanushkodi commented Nov 3, 2021

View reviewed changes

charts/consul/templates/connect-inject-deployment.yaml Outdated Show resolved Hide resolved

ndhanushkodi commented Nov 3, 2021

View reviewed changes

ottaviohartman pushed a commit to ottaviohartman/consul-k8s that referenced this pull request Nov 3, 2021

accep tests: force delete pods after uninstall (hashicorp#833)

b09de16

Force delete pods that are still running after `helm uninstall` so the next installation can begin immediately.

thisisnotashwin force-pushed the automate-dns branch 4 times, most recently from 5475539 to 462098b Compare November 9, 2021 16:03

thisisnotashwin requested review from ishustava, a team and lkysow and removed request for a team November 9, 2021 17:01

thisisnotashwin marked this pull request as ready for review November 9, 2021 17:01

ishustava reviewed Nov 9, 2021

View reviewed changes

thisisnotashwin requested a review from ishustava November 9, 2021 20:48

thisisnotashwin force-pushed the automate-dns branch from bf6b45a to f29938f Compare November 9, 2021 21:03

lkysow reviewed Nov 9, 2021

View reviewed changes

ishustava approved these changes Nov 9, 2021

View reviewed changes

charts/consul/values.yaml Outdated Show resolved Hide resolved

lkysow reviewed Nov 9, 2021

View reviewed changes

thisisnotashwin requested review from lkysow and thisisnotashwin and removed request for thisisnotashwin November 10, 2021 01:00

lkysow reviewed Nov 10, 2021

View reviewed changes

thisisnotashwin force-pushed the automate-dns branch from 258eb41 to ac6faa4 Compare November 10, 2021 17:43

thisisnotashwin requested a review from lkysow November 10, 2021 17:44

lkysow approved these changes Nov 10, 2021

View reviewed changes

thisisnotashwin force-pushed the automate-dns branch 4 times, most recently from 705b8b9 to 7a4ab00 Compare November 10, 2021 18:17

lkysow reviewed Nov 10, 2021

View reviewed changes

control-plane/connect-inject/container_init.go Outdated Show resolved Hide resolved

lkysow approved these changes Nov 10, 2021

View reviewed changes

thisisnotashwin force-pushed the automate-dns branch from f5dc220 to 71b60f1 Compare November 10, 2021 18:44

ndhanushkodi and others added 8 commits November 10, 2021 14:26

template kube dns service IP using api request

93e26ab

when dns is enabled, dns queries are directed to consul.

8962ece

Tested with this branch and consul branch `dns-redirect-to-consul`.

adding demo.yaml for testing. remove before merging

7b66928

remove client clusterrole/rolbinding since they're not currently used

b372c9d

Add unit tests for the changes made.

13bf120

Add feedback from Iryna's review

44abc9b

Add suggestions from Luke.

75db8e9

Update control-plane/connect-inject/container_init.go

9d16c5a

Co-authored-by: Luke Kysow <[email protected]>

thisisnotashwin force-pushed the automate-dns branch from afaf55f to ff61a47 Compare November 10, 2021 19:28

Add CHANGELOG

521fb79

thisisnotashwin force-pushed the automate-dns branch from ff61a47 to 521fb79 Compare November 10, 2021 19:30

thisisnotashwin merged commit 44f81db into main Nov 10, 2021

thisisnotashwin deleted the automate-dns branch November 10, 2021 19:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Automate DNS redirection to Consul DNS #833

Automate DNS redirection to Consul DNS #833

ndhanushkodi commented Nov 3, 2021 •

edited by thisisnotashwin

Loading

ndhanushkodi Nov 3, 2021

ishustava left a comment

lkysow left a comment

lkysow left a comment

lkysow Nov 9, 2021

thisisnotashwin Nov 10, 2021

thisisnotashwin commented Nov 10, 2021

lkysow left a comment

lkysow Nov 10, 2021

lkysow Nov 10, 2021

lkysow Nov 10, 2021

Automate DNS redirection to Consul DNS #833

Automate DNS redirection to Consul DNS #833

Conversation

ndhanushkodi commented Nov 3, 2021 • edited by thisisnotashwin Loading

ndhanushkodi Nov 3, 2021

Choose a reason for hiding this comment

ishustava left a comment

Choose a reason for hiding this comment

lkysow left a comment

Choose a reason for hiding this comment

lkysow left a comment

Choose a reason for hiding this comment

lkysow Nov 9, 2021

Choose a reason for hiding this comment

thisisnotashwin Nov 10, 2021

Choose a reason for hiding this comment

thisisnotashwin commented Nov 10, 2021

lkysow left a comment

Choose a reason for hiding this comment

lkysow Nov 10, 2021

Choose a reason for hiding this comment

lkysow Nov 10, 2021

Choose a reason for hiding this comment

lkysow Nov 10, 2021

Choose a reason for hiding this comment

ndhanushkodi commented Nov 3, 2021 •

edited by thisisnotashwin

Loading