consul-template exits with "invalid memory address or nil pointer dereference" after vault is sealed and unsealed

[gulavanir]

Consul Template version

v0.25.1

Configuration

vault {
  address      = "http://vault.service.consul:8200"

  token        = "s.########"

  unwrap_token = false

  renew_token  = true
}

Command

consul-template -config=/etc/consul-template.d/config/

Debug output

consul-template consul-templatew[723]: 2021/02/02 15:36:41.836894 [WARN] vault.token: failed to renew: Put http://vault.service.consul:8200/v1/auth/token/renew-self: dial tcp: lookup vault.service.consul on 127.0.0.53:53: no such host
consul-template consul-templatew[723]: 2021/02/02 15:36:41.836937 [WARN] vault.token: renewer done (maybe the lease expired)
consul-template consul-templatew[723]: 2021/02/02 15:36:41.836959 [WARN] (view) lease expired or is not renewable (retry attempt 8 after "32s")
consul-template consul-templatew[723]: panic: runtime error: invalid memory address or nil pointer dereference
consul-template consul-templatew[723]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x85a05c]
consul-template consul-templatew[723]: goroutine 832 [running]:
consul-template consul-templatew[723]: github.com/hashicorp/consul-template/dependency.isKVv2(0xc000127650, 0xc0000aa490, 0x18, 0x0, 0x0, 0xc000025b00, 0x0, 0x0)
consul-template consul-templatew[723]: /go/src/github.com/hashicorp/consul-template/dependency/vault_common.go:305 +0x35c
consul-template consul-templatew[723]: github.com/hashicorp/consul-template/dependency.(*VaultWriteQuery).writeSecret(0xc00020a410, 0xc0001159b0, 0xc000470410, 0xc000096eb8, 0x40bd06, 0xc0004703c0)
consul-template consul-templatew[723]: /go/src/github.com/hashicorp/consul-template/dependency/vault_write.go:162 +0x1be
consul-template consul-templatew[723]: github.com/hashicorp/consul-template/dependency.(*VaultWriteQuery).Fetch(0xc00020a410, 0xc0001159b0, 0xc0004703c0, 0x1, 0x1, 0x0, 0x0, 0x0)
consul-template consul-templatew[723]: /go/src/github.com/hashicorp/consul-template/dependency/vault_write.go:77 +0x142
consul-template consul-templatew[723]: github.com/hashicorp/consul-template/watch.(*View).fetch(0xc00024c700, 0xc000233c20, 0xc000233c80, 0xc00024af60)
consul-template consul-templatew[723]: /go/src/github.com/hashicorp/consul-template/watch/view.go:204 +0x15e
consul-template consul-templatew[723]: created by github.com/hashicorp/consul-template/watch.(*View).poll
consul-template consul-templatew[723]: /go/src/github.com/hashicorp/consul-template/watch/view.go:117 +0x445
consul-template systemd[1]: consul-template.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
consul-template systemd[1]: consul-template.service: Failed with result 'exit-code'.

Expected behavior

consul-template should be started successfully

Actual behavior

What actually happened?
consul-template failed with segmentation error.

Steps to reproduce

1. A 3 node cluster with consul-template, consul, vault, nomad is running.
2. Reboot all nodes, vault will be sealed after that. Consul-template is running at this point.
3. Run vault unseal, the older vault token in consul-template will not be usable anymore. consul-template starts throwing "vault.token: failed to renew" warning and stops with segmentation error above, unable to start for sometime. Although the vault integration will not work till the token is renewed, consul-template should not ideally exit.

[gulavanir]

If I am not mistaken https://github.com/hashicorp/consul-template/blob/master/dependency/vault_common.go#L306 this seems to have the fix for this issue. Please confirm.

Also, considering that this is still part of 0.25.1 version tag, what is a typical timeline for the fix to published?

[eikenb]

Hey @gulavanir, thanks for the report and followup.

That fix will be included in 0.25.2 which we're planning on getting out before the end of the month.

I don't have time to reproduce this issue and confirm that fixes it at this time. I've allocated some time in the next few weeks to review everything and see what I might have time to address for the release. I'll try to verify that this is fixed at that point. If you test your case with the fix please add a note here if you've verified it fixes it or not. Thanks.

[eikenb]

It does look like this fixes this issue. So the fix will be in the next release.