Nomad 1.5.1 client unable to start with IPv6 server addresses

[protochron]

Nomad version

Nomad v1.5.1
BuildDate 2023-03-10T22:05:57Z
Revision 6c118dd

Operating system and Environment details

arm64 LInux

Issue

An agent attempting to join a cluster that only advertises addresses with IPv6 with Consul is broken because the IP address is improperly formatted. This may be related to #16550.

Reverting to nomad 1.5.0 fixes the issue.

Here's the output I'm getting in the logs when the agent starts (truncated for brevity):

Mar 21 14:20:00 ip-10-10-20-173 nomad[76334]:     2023-03-21T14:20:00.118Z [WARN]  client.fingerprint_mgr.network: unable to parse speed: path=/usr/sbin/e>
Mar 21 14:20:00 ip-10-10-20-173 nomad[76334]:     2023-03-21T14:20:00.200Z [INFO]  client.fingerprint_mgr.vault: Vault is available
Mar 21 14:20:00 ip-10-10-20-173 nomad[76334]:     2023-03-21T14:20:00.215Z [INFO]  client.plugin: starting plugin manager: plugin-type=csi
Mar 21 14:20:00 ip-10-10-20-173 nomad[76334]:     2023-03-21T14:20:00.215Z [INFO]  client.plugin: starting plugin manager: plugin-type=driver
Mar 21 14:20:00 ip-10-10-20-173 nomad[76334]:     2023-03-21T14:20:00.215Z [INFO]  client.plugin: starting plugin manager: plugin-type=device
Mar 21 14:20:00 ip-10-10-20-173 nomad[76334]:     2023-03-21T14:20:00.227Z [ERROR] client: error discovering nomad servers:
Mar 21 14:20:00 ip-10-10-20-173 nomad[76334]:   error=
Mar 21 14:20:00 ip-10-10-20-173 nomad[76334]:   | 9 errors occurred:
Mar 21 14:20:00 ip-10-10-20-173 nomad[76334]:   | \t* address 2600:(redacted):7629:4647: too many colons in address
Mar 21 14:20:00 ip-10-10-20-173 nomad[76334]:   | \t* address 2600(redacted):1943:4647: too many colons in address

Note the lack of square brackets around the ipv6 addresses.

Reproduction steps

• Configure a Nomad cluster to advertise ipv6 addresses with a consul block
• Install Nomad v1.5.1 on a linux server, and configure it to autodiscover the cluster with Consul
• Watch the agent fail to join the cluster with the errors pasted above

[tgross]

Hi @protochron! I'm pretty sure this was broken by #16217 which we shipped in Nomad 1.5.1. There's a line where we parsed out the serf tags like this addr, err := net.ResolveTCPAddr("tcp", fmt.Sprintf("%s:%s", addrTag, portTag)) which is incorrect for IPv6 🤦 We should have done this: addr, err := net.ResolveTCPAddr("tcp", net.JoinHostPort(addrstr, port)) instead. (Well, I should have done that instead. 😊 )

Fortunately that entire new bit of code was removed in #16490 which will be shipping very soon in Nomad 1.5.2.

[protochron]

I'm glad you're already on it @tgross! I was wondering if that PR was related, since I spotted it after filing this issue. I'll just hold off on upgrading until 1.5.2 😄

[schmichael]

I'm starting to cut the 1.5.2 release and this didn't make it in. Sorry for delaying it!

[tgross]

Bit of miscommunication here on my part. I should have closed this issue because it'll be fixed by #16490, which is going out in 1.5.2