Nomad Job Consul Service Registration Flapping in v0.8.3

[nathanwebsterdotme]

Nomad version

consul: v1.1.0
nomad: v0.8.3

Operating system and Environment details

ubuntu 16.04

Issue

When testing a simple local stack of Consul and Nomad on Vagrant, I am seeing the service registered in consul via a nomad job flapping (registers then deregisters constantly). This is not health check related, but the service is constantly being deregistered.

This seems to have been introduced in 0.8.3 as the issue doesn't happen in 0.8.1

Same issue as #4256

Reproduction steps

• 3 x Consul Server Nodes running v1.1.0
• 2 x Nomad nodes running Nomad Server, Nomad Client and Consul agent as a client.
• Deploy "simpleweb" job on Nomad
• "simpleweb" service registers then deregisters in Consul despite running in a stable manner and being constantly available.

Changing the Nomad version from v0.8.3 to v0.8.1 fixes the problem and the service stays registered in consul correctly.

Broken: https://github.com/nathanwebsterdotme/hashicorp_stack/tree/612b4375d6dc2dbceaa7d685e02ad6027eb46347 . (you can clone, run vagrant up then run the nomad job from nomad1 if you want to re-create).

Working: (version of Nomad changed to v.0.8.1)
https://github.com/nathanwebsterdotme/hashicorp_stack/tree/9a37c4183c04c881bf36bbd0845fa72e1fdd020a

Job file

https://github.com/nathanwebsterdotme/hashicorp_stack/blob/master/nomad/jobs/simpleweb.nomad

[nathanwebsterdotme]

I ensured the fix that @apenney put in to ensure the Nomad client is connecting to consul over 127.0.0.1 before raising this

[preetapan]

@nathanwebsterdotme I wasn't able to reproduce this in a single node consul cluster running 1.1.0. Had some ansible errors trying your setup that I need to work through. When your Consul cluster is up, is it stable? I am wondering if there are other issues with the consul cluster in your test setup like a serf health failure that is causing the local agent to fail its checks, and the only reason that Nomad v0.8.1 works is because it did not sync services correctly until PR #4170

[nathanwebsterdotme]

Consul cluster is stable and the service has been registered successfully for over 24 hours now using Nomad v0.8.1.

[preetapan]

Test binary nomad.zip

[preetapan]

@nathanwebsterdotme this should be fixed in the 0.8.4 release candidate. Thanks for reporting this.

[far-blue]

Just FYI, I ran into the same problem yesterday when upgrading our cluster. We run both server and client on the same nodes and reverting to 0.8.1 did the trick. Thanks for @nathanwebsterdotme for providing all the info needed to debug and @preetapan for the patch and the info on when the fix will be available :)

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.