Delete services synced to external Consul server when running helm delete

[jasonliu747]

Edited by @lkysow: After investigation, this request is about deleting services synced from Kubernetes to an external Consul server when the Consul helm release is deleted. This only applies for external Consul servers because if they were being deployed on Kubernetes, they would be deleted when helm delete is run.

The service synced to consul by annotation won't deregister after the service is deleted. And won't deregister after changing its annotation to consul.hashicorp.com/service-sync: "false" either.

Here is my values.yaml

[lkysow]

Hi @jasonliu747 I think this was fixed by #153 however we haven't got a release out yet. We'll get on that ASAP.

(possible duplicate of #76)

[jasonliu747]

FYI, if the helm release was deleted, services didn't deregister either. Cheers!

[lkysow]

@jasonliu747 can you test with the latest version (v0.9.4) please. I'm not sure what you mean by deleting the helm release? Do you mean Consul or a service you're syncing into Consul that is also managed by helm?

[jasonliu747]

For example, I have three local VMs as consul server. And deploy consul clients(server disabled in values.yaml) in K8S using helm. Latest version did fix #76. But when I executed helm delete --purge consul, services synced from K8S didn't deregister automatically, and a node called k8s-sync was still in the Nodes page.

[lkysow]

Okay I understand. That's expected behaviour. When the sync catalog exits, it doesn't delete services. If it did, then when you did a deploy of it or its Pod died, your services would be deleted.

[jasonliu747]

But when someone deletes consul from K8S using helm delete --purge consul, user might expect all the services/nodes synced from K8S should be deleted as well.

[lkysow]

Yes, they might expect that. We could build it using a helm hook I suppose. If that's what you're asking for I'll reword this issue.

[jasonliu747]

Awesome buddy. Thanks for ur hard working.

[till]

@lkysow is there a manual way to clear services registered through this or is my best bet to delete consul and try again?

[lkysow]

Hi, you can use the Consul HTTP API to delete services.

[till]

@lkysow that doesn't seem to work. A quick google told me that I have to fetch a service ID in order to unregister the service. At least that is what I am hoping you suggested. So I tried it on a test setup, I had previously "synced" the haproxy-ingress-metrics-ingress-controller using consul (or this chart).

cURL: http://localhost:57947/v1/catalog/service/haproxy-ingress-metrics-ingress-controller

[
  {
    "ID": "",
    "Node": "k8s-sync",
    "Address": "127.0.0.1",
    "Datacenter": "testing",
    "TaggedAddresses": null,
    "NodeMeta": {
      "external-source": "kubernetes"
    },
    "ServiceKind": "",
    "ServiceID": "haproxy-ingress-metrics-ingress-controller-4a7b801e5ecc",
    "ServiceName": "haproxy-ingress-metrics-ingress-controller",
    "ServiceTags": [
      "k8s"
    ],
    "ServiceAddress": "10.244.196.215",
    "ServiceWeights": {
      "Passing": 1,
      "Warning": 1
    },
    "ServiceMeta": {
      "external-k8s-ns": "ingress-controller",
      "external-source": "kubernetes",
      "port-ctrl-metrics": "10254",
      "port-metrics": "9101"
    },
    "ServicePort": 10254,
    "ServiceSocketPath": "",
    "ServiceEnableTagOverride": false,
    "ServiceProxy": {
      "Mode": "",
      "MeshGateway": {},
      "Expose": {}
    },
    "ServiceConnect": {},
    "CreateIndex": 8006511,
    "ModifyIndex": 8006511
  },
  {
    "ID": "",
    "Node": "k8s-sync",
    "Address": "127.0.0.1",
    "Datacenter": "testing",
    "TaggedAddresses": null,
    "NodeMeta": {
      "external-source": "kubernetes"
    },
    "ServiceKind": "",
    "ServiceID": "haproxy-ingress-metrics-ingress-controller-775a174389f8",
    "ServiceName": "haproxy-ingress-metrics-ingress-controller",
    "ServiceTags": [
      "k8s"
    ],
    "ServiceAddress": "10.244.36.138",
    "ServiceWeights": {
      "Passing": 1,
      "Warning": 1
    },
    "ServiceMeta": {
      "external-k8s-ns": "ingress-controller",
      "external-source": "kubernetes",
      "port-ctrl-metrics": "10254",
      "port-metrics": "9101"
    },
    "ServicePort": 10254,
    "ServiceSocketPath": "",
    "ServiceEnableTagOverride": false,
    "ServiceProxy": {
      "Mode": "",
      "MeshGateway": {},
      "Expose": {}
    },
    "ServiceConnect": {},
    "CreateIndex": 8006522,
    "ModifyIndex": 8006522
  },
  {
    "ID": "",
    "Node": "k8s-sync",
    "Address": "127.0.0.1",
    "Datacenter": "testing",
    "TaggedAddresses": null,
    "NodeMeta": {
      "external-source": "kubernetes"
    },
    "ServiceKind": "",
    "ServiceID": "haproxy-ingress-metrics-ingress-controller-fff5b8d95794",
    "ServiceName": "haproxy-ingress-metrics-ingress-controller",
    "ServiceTags": [
      "k8s"
    ],
    "ServiceAddress": "10.244.147.40",
    "ServiceWeights": {
      "Passing": 1,
      "Warning": 1
    },
    "ServiceMeta": {
      "external-k8s-ns": "ingress-controller",
      "external-source": "kubernetes",
      "port-ctrl-metrics": "10254",
      "port-metrics": "9101"
    },
    "ServicePort": 10254,
    "ServiceSocketPath": "",
    "ServiceEnableTagOverride": false,
    "ServiceProxy": {
      "Mode": "",
      "MeshGateway": {},
      "Expose": {}
    },
    "ServiceConnect": {},
    "CreateIndex": 8006514,
    "ModifyIndex": 8006514
  }
]

When I use either of the ServiceID I get:

curl --request PUT http://localhost:57947/v1/agent/service/deregister/haproxy-ingress-metrics-ingress-controller-fff5b8d95794
Unknown service ID "haproxy-ingress-metrics-ingress-controller-fff5b8d95794". Ensure that the service ID is passed, not the service name.

Any clue which API to use to clean this up?

[lkysow]

Hey I think you need to use the catalog deregister endpoints. Not the agent endpoints. Sorry for not being exact, I'm not at my laptop.

[till]

Hey I think you need to use the catalog deregister endpoints. Not the agent endpoints. Sorry for not being exact, I'm not at my laptop.

Thank you, much appreciated! :)

So for the record, or for anyone looking to delete all of what the catalog sync added:

curl --request PUT -d '{"Node":"k8s-sync"}' http://consul/v1/catalog/deregister