Consul Service Advertisement Refactor

[schmichael]

There have been enough issues related to this work I wanted to create an issue to link them all *
together.

Branch: https://github.com/hashicorp/nomad/tree/f-consul-refactor

The existing (0.5.5 and before) Consul service advertisement code has the following properties:

• Runs in the task executor so checks are updated even if the Nomad agent is down.
• Designed as a reconciliation loop to ensure Consul is consistent with Nomad (eg reaps orphaned Consul entries periodically)

This has the following issues:

• Reconciliation loop is expensive and must read all services out of Consul periodically even if nothing has changed.
• Since each task executor runs its own reconciliation loop this resource utilization gets far worse for users with many tasks. See Every Nomad executor queries all consul services and health checks every 5 seconds. #1995
• Reconciliation logic is complex and error prone. See Using a task name that is a prefix of another task's name can cause consul service flapping #2474

The goal of the Consul refactor is to address these issues by doing the following:

• Add/remove services/checks as appropriate. No periodic reconciliation.
• Move all Consul interaction into the Nomad client agent - not the task executor.
  • The one downside to this is that if the Nomad agent is down for longer than a script check's interval, the script check will not be run and the service's health in Consul will not be accurate.

[schmichael]

Reopening until #2554 lands.

[schmichael]

#2554 landed ages ago; not sure why I haven't closed this yet. :)

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.