ingress-nginx: 'connect-ignore' label not working

[byrneo]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

I'm attempting to configure ingress-nginx as the k8s ingress for services which are part of a newly deployed consul-connect service mesh. I understand that there is an existing issue wherein consul won't support multiple services that have selectors for the same pod(s).

To try overcome that, i've used the consul.hashicorp.com/connect-ignore=true label as suggested by #592 , However, the two services I attempt to exclude still appear to be (attempting to) register to consul.

Reproduction Steps

On EKS v1.18

1. Install consul-connect with the following deploy/config.yml:

helm upgrade --install -f deploy/config.yaml consul hashicorp/consul --create-namespace -n consul --version "0.37.0"

global:
  name: consul
  datacenter: dcloud-dev-aws
  metrics:
    enabled: true
    enableAgentMetrics: true
  acls:
    manageSystemACLs: true
server:
  replicas: 1
ui:
  enabled: true
connectInject:
  enabled: true
  default: false
controller:
  enabled: true
prometheus:
  enabled: true

versions deployed by above chart:

• hashicorp/consul-k8s-control-plane:0.37.0
• hashicorp/consul:1.10.4

2. Install ingress-nginx via helm chart.

3. Add consul.hashicorp.com/connect-ignore=true label to the ingress-nginx-consul-controller-admission and ingress-nginx-consul-controller-metrics services

kubectl get svc -n ingress-nginx-consul
NAME                                        TYPE           CLUSTER-IP      EXTERNAL-IP                                                                     PORT(S)                      AGE
ingress-nginx-consul-controller             LoadBalancer   10.100.211.64   <redacted>.elb.us-east-1.amazonaws.com   80:30986/TCP,443:31505/TCP   20h
ingress-nginx-consul-controller-admission   ClusterIP      10.100.245.8    <none>                                                                          443/TCP                      20h
ingress-nginx-consul-controller-metrics     ClusterIP      10.100.8.80     <none>                                                                          9913/TCP                     20h

kubectl describe svc ingress-nginx-consul-controller-admission -n ingress-nginx-consul

Name:              ingress-nginx-consul-controller-admission
Namespace:         ingress-nginx-consul
Labels:            app.kubernetes.io/component=controller
                   app.kubernetes.io/instance=ingress-nginx-consul
                   app.kubernetes.io/managed-by=Helm
                   app.kubernetes.io/name=ingress-nginx
                   app.kubernetes.io/version=0.41.2
                   consul.hashicorp.com/connect-ignore=true
                   helm.sh/chart=ingress-nginx-3.10.1
Annotations:       meta.helm.sh/release-name: ingress-nginx-consul
                   meta.helm.sh/release-namespace: ingress-nginx-consul

 kubectl describe svc ingress-nginx-consul-controller-admission -n ingress-nginx-consul
Name:              ingress-nginx-consul-controller-admission
Namespace:         ingress-nginx-consul
Labels:            app.kubernetes.io/component=controller
                   app.kubernetes.io/instance=ingress-nginx-consul
                   app.kubernetes.io/managed-by=Helm
                   app.kubernetes.io/name=ingress-nginx
                   app.kubernetes.io/version=0.41.2
                   consul.hashicorp.com/connect-ignore=true
                   helm.sh/chart=ingress-nginx-3.10.1

Logs

Ingress controller pods won't start:

kubectl get pod -n ingress-nginx-consul
NAME                                               READY   STATUS                  RESTARTS   AGE
ingress-nginx-consul-controller-7f5997bd74-29l4w   0/2     Init:CrashLoopBackOff   173        20h
ingress-nginx-consul-controller-7f5997bd74-bqlqh   0/2     Init:CrashLoopBackOff   173        20h

kubectl logs -f ingress-nginx-consul-controller-7f5997bd74-29l4w -n ingress-nginx-consul  -c consul-connect-inject-init

2021-12-08T11:22:43.177Z [INFO]  Consul login complete
2021-12-08T11:22:43.180Z [INFO]  Unable to find registered services; retrying
<snip>
2021-12-08T11:22:52.201Z [INFO]  Check to ensure a Kubernetes service has been created for this application. If your pod is not starting also check the connect-inject deployment logs.

<snip>

2021-12-08T11:24:43.452Z [INFO]  Unable to find registered services; retrying
2021-12-08T11:24:43.452Z [ERROR] Timed out waiting for service registration: error="did not find correct number of services: 3"

Logs from consul connect injector suggest that it is not ignoring the two services ingress-nginx-consul-controller-admission and ingress-nginx-consul-controller-metrics:

kubectl logs -f consul-connect-injector-webhook-deployment-69744dbd8c-6pphl -n consul


2021-12-08T08:19:39.277Z	INFO	controller.endpoints	registering service with Consul	{"name": "ingress-nginx-consul-controller-metrics", "id": "ingress-nginx-consul-controller-7f5997bd74-bqlqh-ingress-nginx-consul-controller-metrics", "agentIP": "100.96.132.75"}
2021-12-08T08:19:39.354Z	INFO	controller.endpoints	registering proxy service with Consul	{"name": "ingress-nginx-consul-controller-metrics-sidecar-proxy"}
2021-12-08T08:19:39.374Z	INFO	controller.endpoints	updating health check status for service	{"name": "ingress-nginx-consul-controller-metrics", "reason": "Pod \"ingress-nginx-consul/ingress-nginx-consul-controller-7f5997bd74-bqlqh\" is not ready", "status": "critical"}
2021-12-08T08:19:39.381Z	INFO	controller.endpoints	registering service with Consul	{"name": "ingress-nginx-consul-controller-metrics", "id": "ingress-nginx-consul-controller-7f5997bd74-29l4w-ingress-nginx-consul-controller-metrics", "agentIP": "100.96.140.174"}
2021-12-08T08:19:39.386Z	INFO	controller.endpoints	registering proxy service with Consul	{"name": "ingress-nginx-consul-controller-metrics-sidecar-proxy"}
2021-12-08T08:19:39.410Z	INFO	controller.endpoints	updating health check status for service	{"name": "ingress-nginx-consul-controller-metrics", "reason": "Pod \"ingress-nginx-consul/ingress-nginx-consul-controller-7f5997bd74-29l4w\" is not ready", "status": "critical"}
2021-12-08T08:19:39.459Z	INFO	controller.endpoints	retrieved	{"name": "ingress-nginx-consul-controller-admission", "ns": "ingress-nginx-consul"}
2021-12-08T08:19:39.459Z	INFO	controller.endpoints	registering service with Consul	{"name": "ingress-nginx-consul-controller-admission", "id": "ingress-nginx-consul-controller-7f5997bd74-29l4w-ingress-nginx-consul-controller-admission", "agentIP": "100.96.140.174"}
2021-12-08T08:19:39.476Z	INFO	controller.endpoints	registering proxy service with Consul	{"name": "ingress-nginx-consul-controller-admission-sidecar-proxy"}
2021-12-08T08:19:39.496Z	INFO	controller.endpoints	updating health check status for service	{"name": "ingress-nginx-consul-controller-admission", "reason": "Pod \"ingress-nginx-consul/ingress-nginx-consul-controller-7f5997bd74-29l4w\" is not ready", "status": "critical"}
2021-12-08T08:19:39.557Z	INFO	controller.endpoints	registering service with Consul	{"name": "ingress-nginx-consul-controller-admission", "id": "ingress-nginx-consul-controller-7f5997bd74-bqlqh-ingress-nginx-consul-controller-admission", "agentIP": "100.96.132.75"}
2021-12-08T08:19:39.561Z	INFO	controller.endpoints	registering proxy service with Consul	{"name": "ingress-nginx-consul-controller-admission-sidecar-proxy"}
2021-12-08T08:19:39.580Z	INFO	controller.endpoints	updating health check status for service	{"name": "ingress-nginx-consul-controller-admission", "reason": "Pod \"ingress-nginx-consul/ingress-nginx-consul-controller-7f5997bd74-bqlqh\" is not ready", "status": "critical"}
<snip>
2021-12-08T08:19:40.075Z	INFO	controller.endpoints	retrieved	{"name": "ingress-nginx-consul-controller", "ns": "ingress-nginx-consul"}
2021-12-08T08:19:40.076Z	INFO	controller.endpoints	registering service with Consul	{"name": "ingress-nginx-consul-controller", "id": "ingress-nginx-consul-controller-7f5997bd74-bqlqh-ingress-nginx-consul-controller", "agentIP": "100.96.132.75"}
2021-12-08T08:19:40.248Z	INFO	controller.endpoints	registering proxy service with Consul	{"name": "ingress-nginx-consul-controller-sidecar-proxy"}
2021-12-08T08:19:40.292Z	INFO	controller.endpoints	updating health check status for service	{"name": "ingress-nginx-consul-controller", "reason": "Pod \"ingress-nginx-consul/ingress-nginx-consul-controller-7f5997bd74-bqlqh\" is not ready", "status": "critical"}
2021-12-08T08:19:40.295Z	INFO	controller.endpoints	registering service with Consul	{"name": "ingress-nginx-consul-controller", "id": "ingress-nginx-consul-controller-7f5997bd74-29l4w-ingress-nginx-consul-controller", "agentIP": "100.96.140.174"}
2021-12-08T08:19:40.297Z	INFO	controller.endpoints	registering proxy service with Consul	{"name": "ingress-nginx-consul-controller-sidecar-proxy"}
2021-12-08T08:19:40.309Z	INFO	controller.endpoints	updating health check status for service	{"name": "ingress-nginx-consul-controller", "reason": "Pod \"ingress-nginx-consul/ingress-nginx-consul-controller-7f5997bd74-29l4w\" is not ready", "status": "critical"}

Expected behavior

I was hoping that by excluding the two ingress-nginx services (ingress-nginx-consul-controller-admission and ingress-nginx-consul-controller-metrics) that consul would successfully enable the ingress-controller pods to start and inject a transparent envoy sidecar as expected.

Environment details

as above.

Additional Context

[byrneo]

Ah....upon re-reading #592 and having just found: #858 i think perhaps this functionality may not have been released yet!

[byrneo]

I built a docker image for the control-plane from the main branch and updated my environment. After applying the consul.hashicorp.com/service-ignore: "true" label to the two services, I could see them being de-registered in the consul logs and now the pods start up ok:

kubectl get pod -n ingress-nginx-consul
NAME                                              READY   STATUS    RESTARTS   AGE
ingress-nginx-consul-controller-ddb6988b8-kgn9b   2/2     Running   0          8m2s
ingress-nginx-consul-controller-ddb6988b8-xpg6b   2/2     Running   0          8m2s

Sorry for the noise - this unblocks me for now. I'll close the issue!

[t-eckert]

@byrneo,

Glad that unblocked you. The feature will be released this month.