vault unwrap is failing in redirect HA mode

[silverbadge]

We're running a cluster of Vault 0.7.3 behind a load balancer that points to unsealed vault nodes. The load balancer is also excluding the leader from the backend servers pool. The clients are redirected to the leader which will then be able to see the real client IP address for app role cidr validation purpose. The problem we're currently experience is:
When unwrapping a token, we get error validating wrapping token: Vault is in standby mode

Vault status:

vault status
Sealed: false
Key Shares: 5
Key Threshold: 3
Unseal Progress: 0
Unseal Nonce:
Version: 0.7.3
Cluster Name: vault-cluster-e05b5033
Cluster ID: 30dbfa53-e2ca-444c-dc51-5f53ff744348

High-Availability Enabled: true
	Mode: standby
	Leader: https://realleadervaultaddress.com

Create wrapped secret-id

vault write -wrap-ttl=10m -f auth/approle/role/test/secret-id
Key                          	Value
---                          	-----
wrapping_token:              	<REDACTED>
wrapping_token_ttl:          	10m0s
wrapping_token_creation_time:	2017-07-27 12:54:13.288156401 +0000 UTC

Unwrap

vault unwrap <REDACTED>
Error making API request.

URL: PUT https://<REDACTED>/v1/sys/wrapping/unwrap
Code: 400. Errors:

* error validating wrapping token: Vault is in standby mode

[ctlajoie]

I also ran into this when using https://vault.service.consul:8200 as the vault endpoint for my nomad cluster. Simply using https://active.vault.service.consul:8200 resolved it, but this does seem like a bug since vault is supposed to forward requests to the active node.

[jefferai]

@ctlajoie OP indicates that they're using redirect mode, but actually I think this was a bug affecting both that has been fixed. It should behave better in 0.8.0-rc1.

[silverbadge]

I've just tested using vault unwrap command and it seems to be still failing after the upgrade to 0.8.1:

$ vault unwrap XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
Error making API request.

URL: PUT https://vault.services.internal/v1/sys/wrapping/unwrap
Code: 400. Errors:

* error validating wrapping token: Vault is in standby mode
$ vault version
Vault v0.8.1

$ vault status | grep -i version
Version: 0.8.1

[jefferai]

It's working for me:

jeff@snicketysnack:/tmp/vault-test-cluster-736589108$ VAULT_ADDR=https://127.0.0.1:41117
jeff@snicketysnack:/tmp/vault-test-cluster-736589108$ vault unwrap -tls-skip-verify dfbd0dbf-6ee4-3a63-ffc4-5a85e0693b0b
Key             	Value
---             	-----
refresh_interval	768h0m0s
asdf            	asdf

jeff@snicketysnack:/tmp/vault-test-cluster-736589108$ vault status -tls-skip-verify
Sealed: false
Key Shares: 3
Key Threshold: 3
Unseal Progress: 0
Unseal Nonce: 
Version: 0.8.3+pro
Cluster Name: vault-cluster-498d416a
Cluster ID: ec6f2573-1df3-038e-201c-ade0d78233fb

High-Availability Enabled: true
	Mode: standby
	Leader Cluster Address: https://127.0.0.1:44260

[tarpanpathak]

@jefferai we are having the same issue in 0.7.3. Are you saying we need to upgrade? If so, is the upgrade as simple as updating the binary and restarting the agent?

[jefferai]

We publish upgrade guides on our site -- go to the guides area and you'll then see it on the left.

[tarpanpathak]

Understood. Which version should we upgrade to? We are currently on 0.7.3.

[jefferai]

0.8.3.

[tarpanpathak]

Got it. Thx. One more question for you: is it a good idea to run multiple versions of Vault as part of the same cluster for canary deployments? e.g we are facing this issue on 0.7.3 and we want to bring up 3 more nodes on 0.8.3 and once they join/are unsealed, take the 0.7.3 nodes offline (or perhaps keep both the versions running).

[jefferai]

See here for our guidance around that: https://www.vaultproject.io/guides/upgrading/index.html