vault_github_team token_policies breaks policy mapping

[adamnoll]

Terraform Version

Terraform v0.11.13

• provider.vault v2.2.0

Affected Resource(s)

vault_github_team

Terraform Configuration Files

Before:

resource "vault_github_team" "sre" {
  backend  = "${vault_github_auth_backend.github.id}"
  team     = "sre"
  policies = ["admin"]
}

After:

resource "vault_github_team" "sre" {
  backend  = "${vault_github_auth_backend.github.id}"
  team     = "sre"
  token_policies = ["admin"]
}

Expected Behavior

When logging in with the correct github personal access token,

token_policies         ["default" "admin"]
policies                    ["default" "admin"]

Actual Behavior

Once the above code it applied to a vault instance, the "admin" policy is no longer bound to the token

token_policies         ["default"]
policies                    ["default"]

Reverting back from "token_policies" to "policies" will correct the behavior.

[lawliet89]

I cannot seem to reproduce your issue.

I added a test in 3078e3b to try and reproduce your issue but the tests passed.

Can you paste your Terraform Plan output when you change your Before to After Configuration?

Test Debug Output

UPDATE: vault_github_team.team
  backend:                 "github-8546346915128891810" => "github-8546346915128891810"
  id:                      "auth/github-8546346915128891810/map/teams/my-team-slugified" => "auth/github-8546346915128891810/map/teams/my-team-slugified"
  policies.#:              "2" => "0"
  policies.0:              "admin" => ""
  policies.1:              "security" => ""
  team:                    "my-team-slugified" => "my-team-slugified"
  token_bound_cidrs.#:     "0" => "0"
  token_explicit_max_ttl:  "0" => "0"
  token_max_ttl:           "0" => "1800"
  token_no_default_policy: "false" => "false"
  token_num_uses:          "0" => "0"
  token_period:            "0" => "0"
  token_policies.#:        "0" => "2"
  token_policies.0:        "" => "admin"
  token_policies.1:        "" => "security"
  token_ttl:               "0" => "300"
  token_type:              "default" => "default"



STATE:

vault_github_auth_backend.gh:
  ID = github-8546346915128891810
  provider = provider.vault
  accessor = auth_github_f0053a52
  base_url = 
  description = 
  max_ttl = 0s
  organization = vault
  path = github-8546346915128891810
  ttl = 0s
  tune.# = 1
  tune.0.default_lease_ttl = 768h
  tune.0.listing_visibility = 
  tune.0.max_lease_ttl = 768h
vault_github_team.team:
  ID = auth/github-8546346915128891810/map/teams/my-team-slugified
  provider = provider.vault
  backend = github-8546346915128891810
  policies.# = 2
  policies.0 = admin
  policies.1 = security
  team = my-team-slugified
  token_explicit_max_ttl = 0
  token_max_ttl = 0
  token_no_default_policy = false
  token_num_uses = 0
  token_period = 0
  token_ttl = 0
  token_type = default

  Dependencies:
    vault_github_auth_backend.gh
2019/08/14 16:55:30 [INFO] terraform: building graph: GraphTypeApply
2019/08/14 16:55:30 [DEBUG] adding implicit provider configuration provider.vault, implied first by vault_github_auth_backend.gh (prepare state)
2019/08/14 16:55:30 [DEBUG] ProviderTransformer: "vault_github_team.team" (*terraform.NodeApplyableResourceInstance) needs provider.vault
2019/08/14 16:55:30 [DEBUG] ProviderTransformer: "vault_github_auth_backend.gh (prepare state)" (*terraform.NodeApplyableResource) needs provider.vault
2019/08/14 16:55:30 [DEBUG] ProviderTransformer: "vault_github_team.team (prepare state)" (*terraform.NodeApplyableResource) needs provider.vault
2019/08/14 16:55:30 [DEBUG] ReferenceTransformer: "provider.vault" references: []
2019/08/14 16:55:30 [DEBUG] ReferenceTransformer: "vault_github_auth_backend.gh (prepare state)" references: []
2019/08/14 16:55:30 [DEBUG] ReferenceTransformer: "vault_github_team.team (prepare state)" references: []
2019/08/14 16:55:30 [DEBUG] ReferenceTransformer: "vault_github_team.team" references: [vault_github_auth_backend.gh (prepare state)]
2019/08/14 16:55:30 [DEBUG] Starting graph walk: walkApply
2019/08/14 16:55:30 [DEBUG] Vault API Request Details:
---[ REQUEST ]---------------------------------------
GET /v1/auth/token/lookup-self HTTP/1.1
Host: 127.0.0.1:8200
User-Agent: Go-http-client/1.1
X-Vault-Token: 12345
Accept-Encoding: gzip


-----------------------------------------------------
2019/08/14 16:55:30 [DEBUG] Vault API Response Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Content-Length: 512
Cache-Control: no-store
Content-Type: application/json
Date: Wed, 14 Aug 2019 08:55:30 GMT

{
 "request_id": "ed2d8579-844f-90c5-a9fb-7f8825345355",
 "lease_id": "",
 "renewable": false,
 "lease_duration": 0,
 "data": {
  "accessor": "lNmyxl9fOGAamTR2R5dPzGHy",
  "creation_time": 1565771232,
  "creation_ttl": 0,
  "display_name": "token",
  "entity_id": "",
  "expire_time": null,
  "explicit_max_ttl": 0,
  "id": "12345",
  "issue_time": "2019-08-14T16:27:12.970001367+08:00",
  "meta": null,
  "num_uses": 0,
  "orphan": true,
  "path": "auth/token/create",
  "policies": [
   "root"
  ],
  "renewable": false,
  "ttl": 0,
  "type": "service"
 },
 "wrap_info": null,
 "warnings": null,
 "auth": null
}

-----------------------------------------------------
2019/08/14 16:55:30 [DEBUG] Vault API Request Details:
---[ REQUEST ]---------------------------------------
POST /v1/auth/token/create HTTP/1.1
Host: 127.0.0.1:8200
User-Agent: Go-http-client/1.1
Content-Length: 128
X-Vault-Token: 12345
Accept-Encoding: gzip

{
 "ttl": "1200s",
 "explicit_max_ttl": "1200s",
 "display_name": "terraform",
 "num_uses": 0,
 "renewable": false,
 "type": "",
 "entity_alias": ""
}
-----------------------------------------------------
2019/08/14 16:55:30 [DEBUG] Vault API Response Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Content-Length: 396
Cache-Control: no-store
Content-Type: application/json
Date: Wed, 14 Aug 2019 08:55:30 GMT

{
 "request_id": "9f286e7f-233a-d796-149f-3b882ddf0508",
 "lease_id": "",
 "renewable": false,
 "lease_duration": 0,
 "data": null,
 "wrap_info": null,
 "warnings": null,
 "auth": {
  "client_token": "s.IVXP3LtVR4rd8bEaExGobPjf",
  "accessor": "b5YUW5zLSkGSH2LLSVwnSNQ2",
  "policies": [
   "root"
  ],
  "token_policies": [
   "root"
  ],
  "metadata": null,
  "lease_duration": 1200,
  "renewable": false,
  "entity_id": "",
  "token_type": "service",
  "orphan": false
 }
}

-----------------------------------------------------
2019/08/14 16:55:30 [INFO] Using Vault token with the following policies: root
2019/08/14 16:55:30 [WARN] Provider "vault" produced an invalid plan for vault_github_team.team, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .policies: planned value cty.ListValEmpty(cty.String) does not match config value cty.NullVal(cty.List(cty.String)) nor prior value cty.ListVal([]cty.Value{cty.StringVal("admin"), cty.StringVal("security")})
2019/08/14 16:55:30 [DEBUG] vault_github_team.team: applying the planned Update change
2019/08/14 16:55:30 [DEBUG] Vault API Request Details:
---[ REQUEST ]---------------------------------------
PUT /v1/auth/github-8546346915128891810/map/teams/my-team-slugified HTTP/1.1
Host: 127.0.0.1:8200
User-Agent: Go-http-client/1.1
Content-Length: 243
X-Vault-Token: s.IVXP3LtVR4rd8bEaExGobPjf
Accept-Encoding: gzip

{
 "key": "my-team-slugified",
 "token_bound_cidrs": [],
 "token_explicit_max_ttl": 0,
 "token_max_ttl": 1800,
 "token_no_default_policy": false,
 "token_num_uses": 0,
 "token_period": 0,
 "token_policies": [
  "security",
  "admin"
 ],
 "token_ttl": 300,
 "token_type": "default"
}
-----------------------------------------------------
2019/08/14 16:55:30 [DEBUG] Vault API Response Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 204 No Content
Content-Length: 0
Cache-Control: no-store
Content-Type: application/json
Date: Wed, 14 Aug 2019 08:55:30 GMT


-----------------------------------------------------
2019/08/14 16:55:30 [INFO] Saved github team map at 'auth/github-8546346915128891810/map/teams/my-team-slugified'
2019/08/14 16:55:30 [DEBUG] Vault API Request Details:
---[ REQUEST ]---------------------------------------
GET /v1/auth/github-8546346915128891810/map/teams/my-team-slugified HTTP/1.1
Host: 127.0.0.1:8200
User-Agent: Go-http-client/1.1
X-Vault-Token: s.IVXP3LtVR4rd8bEaExGobPjf
Accept-Encoding: gzip


-----------------------------------------------------
2019/08/14 16:55:30 [DEBUG] Vault API Response Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Content-Length: 401
Cache-Control: no-store
Content-Type: application/json
Date: Wed, 14 Aug 2019 08:55:30 GMT

{
 "request_id": "8cb00f80-2fe0-3535-b92b-a929c6f9ef81",
 "lease_id": "",
 "renewable": false,
 "lease_duration": 0,
 "data": {
  "key": "my-team-slugified",
  "token_bound_cidrs": [],
  "token_explicit_max_ttl": 0,
  "token_max_ttl": 1800,
  "token_no_default_policy": false,
  "token_num_uses": 0,
  "token_period": 0,
  "token_policies": [
   "security",
   "admin"
  ],
  "token_ttl": 300,
  "token_type": "default"
 },
 "wrap_info": null,
 "warnings": null,
 "auth": null
}

[adamnoll]

@lawliet89 Thanks for taking a look. Here is extra information:
Slightly different code:

Code before:

resource "vault_github_team" "eng_shared" {
  backend  = "${vault_github_auth_backend.github.id}"
  team     = "engineering"
  policies = ["eng_shared"]
}

Terraform Apply:

  + module.auth-backends.vault_github_team.eng_shared
      id:         <computed>
      backend:    "github"
      policies.#: "1"
      policies.0: "eng_shared"
      team:       "engineering"
      token_type: "default"

module.auth-backends.vault_github_team.eng_shared: Creating...
  backend:    "" => "github"
  policies.#: "" => "1"
  policies.0: "" => "eng_shared"
  team:       "" => "engineering"
  token_type: "" => "default"

Login:

vault login -method=github token='asdfasdf'

Key                    Value
---                    -----
token                  s.dbOe2sMOEEkNsTzsbBQhpyRc
token_accessor         vnzQWD5Jnoxmfc8vDpS40iD0
token_duration         1h
token_renewable        true
token_policies         ["admin" "default" "eng_shared"]
identity_policies      []
policies               ["admin" "default" "eng_shared"]
token_meta_org         xxx
token_meta_username    adamnoll

Code After:

resource "vault_github_team" "eng_shared" {
  backend        = "${vault_github_auth_backend.github.id}"
  team           = "engineering"
  token_policies = ["eng_shared"]
}

Terraform Apply:

  ~ module.auth-backends.vault_github_team.eng_shared
      policies.#:                "1" => "0"
      policies.0:                "eng_shared" => ""
      token_policies.#:          "0" => "1"
      token_policies.2481687925: "" => "eng_shared"

module.auth-backends.vault_github_team.eng_shared: Modifying... (ID: auth/github/map/teams/engineering)
  policies.#:                "1" => "0"
  policies.0:                "eng_shared" => ""
  token_policies.#:          "0" => "1"
  token_policies.2481687925: "" => "eng_shared"

Login:

vault login -method=github token='asdfasdf'

Key                    Value
---                    -----
token                  s.4rj0ZvNtfBNPOvKp9xHYaHNq
token_accessor         ffhrRCDfspc58C09G9P2FP9Z
token_duration         1h
token_renewable        true
token_policies         ["admin" "default"]
identity_policies      []
policies               ["admin" "default"]
token_meta_org         xxxx
token_meta_username    adamnoll

[adamnoll]

Also,
Vault version:
1.1.2

[lawliet89]

The token_x fields should only be used with Vault 1.2 since they are newly added to various roles in that version.

[acesir]

I have the same issue running vault 1.2.1.

token_policies applied for kubernetes_auth work fine while applying token_policies to github_auth does not link the policy correctly and applies the default policy.

on vault 1.2.1 this is not working correctly and policy needs to be used as well

[lawliet89]

This might be a problem with Vault itself.

After applying, say, vault_github_auth_backend using token_policies, what do you get when you do a vault read on the path auth/<GITHUB_AUTH_MOUNT>/map/teams/<TEAM>? Do you get the token_policies you have set?

If so, then logging in with the right team mapping and not getting the right policies is probably a bug with Vault. Otherwise, can you post a debug log for the whole Terraform apply procedure (and make sure to redact sensitive information like Vault tokens and Vault addresses).

[zx8]

Is this possibly related (or caused by a similar underlying issue) to:

• OIDC: Setting value token_policy does not set policy in token vault-plugin-auth-jwt#69
• hashicorp/vault@2eba341#diff-4ac32a78649ca5bdd8e0ba38b7006a1e

[lawliet89]

I just did a test and I think there's a bug with Vault:

resource "vault_github_auth_backend" "example" {
  organization = "<snip>"
}

resource "vault_policy" "example" {
  name = "test"

  policy = <<EOT
path "secret/my_app" {
  policy = "write"
}
EOT
}


resource "vault_github_team" "tf_devs" {
  backend        = vault_github_auth_backend.example.path
  team           = "<snip>"
  token_policies = [vault_policy.example.name]
}

After applying, I can confirm that Terraform does not show a diff.

Reading auth/github/map/teams/<snip> gives the right set of policies.

$ vault read -format json auth/github/map/teams/<snip> 
{
  "request_id": "fee98f43-304d-b974-39d2-f64a385f2f3d",
  "lease_id": "",
  "lease_duration": 0,
  "renewable": false,
  "data": {
    "key": "<snip>",
    "token_bound_cidrs": [],
    "token_explicit_max_ttl": 0,
    "token_max_ttl": 0,
    "token_no_default_policy": false,
    "token_num_uses": 0,
    "token_period": 0,
    "token_policies": [
      "test"
    ],
    "token_ttl": 0,
    "token_type": "default"
  },
  "warnings": null
}

Logging in with a token that belongs to the team above, however:

$ vault login -address http://127.0.0.1:8200 -method github
GitHub Personal Access Token (will be hidden): 
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                    Value
---                    -----
token                  s.sM2CCIxVMFZya8iVVHv5qt5B
token_accessor         VqHNFvnWVKdgaeElgQN3xrlM
token_duration         768h
token_renewable        true
token_policies         ["default"]
identity_policies      []
policies               ["default"]
token_meta_org         <snip>
token_meta_username    lawliet89

[kalafut]

@lawliet89 I think this is an issue with the provider. The github resource is old and uses some uncommon approaches, like the team/user policy mappings. While the final token should pick up the new token fields, the internal policy maps have not changed. You can see the difference in what's stored running the command manually vs with the provider:

➜  ~ vault write auth/github/map/teams/vault vault=test-policy

➜  ~ vault read auth/github/map/teams/vault
Key      Value
---      -----
key      vault
vault    test-policy

but after TF apply:

➜  ~ vault read auth/github/map/teams/vault
Key                        Value
---                        -----
key                        vault
token_bound_cidrs          []
token_explicit_max_ttl     0s
token_max_ttl              0s
token_no_default_policy    false
token_num_uses             0
token_period               0s
token_policies             [test-policy]
token_ttl                  0s
token_type                 default

The specialized Github code that parses this map doesn't know about "token_policies", and I don't think we want to change Vault for that. These maps don't store any of the other fields either. I think reverting the team/user map updates that were part of the token_ update commit will probably fix this.

[hamishforbes]

Same issue here, I don't get the correct policy applied when Terraform manages the team/policy mapping, but if I manually create a mapping following the Vault docs it works:

~> vault read  auth/github/map/teams/vault-operators
Key                        Value
---                        -----
key                        vault-operators
token_bound_cidrs          []
token_explicit_max_ttl     0s
token_max_ttl              0s
token_no_default_policy    false
token_num_uses             0
token_period               0s
token_policies             [admin, default]
token_ttl                  0s
token_type                 default

~> vault read  auth/github/map/teams/test
Key      Value
---      -----
key      test
value    admin,default

Users in the test team receive the admin policy and users in the vault-operators team do not.

If I rollback to v2.1.0 of this provider I get a crash, I'm guessing this is because it's getting a bunch of unexpected parameters back when refreshing the state?

edit: For anyone else with this issue, workaround is to set the deprecated policies parameter on the resource instead of token_policies and just ignore the deprecation message

[pgiles]

Since no one has yet to mention that it affects other auth methods, I'll throw in that AppRole (vault_approle_auth_backend_role) is also affected and the workaround @hamishforbes mentioned worked for us. If there is a way to suppress the deprecation messages, please share.

[hamishforbes]

token_policies is working on vault_approle_auth_backend_role for me, using Vault 1.2.2.

[joe-bowman]

I came here with the same problem, but the cause was this:

The token_x fields should only be used with Vault 1.2 since they are newly added to various roles in that version.

Can we update the deprecation message to state the version requirement; at the moment it does no such thing, just complains that the field is deprecated and token_* should be used instead.

[joemiller]

Using v2.3.0 of the tf-vault plugin, and vault 1.2.2.

token_policies does not seem to be working as expected with the vault_token_auth_backend_role resource.

It is listed in the documentation - https://www.terraform.io/docs/providers/vault/r/token_auth_backend_role.html

However, if it is used it will appear in the terraform diff on every TF run.

It might be an issue in Vault 1.2.2 as I don't see token_policies listed in the vault read output, eg:

$ vault read auth/token/roles/gke-foobar
Key                       Value
---                       -----
allowed_entity_aliases    <nil>
allowed_policies          [gke-foobar]
disallowed_policies       []
explicit_max_ttl          0s
name                      gke-foobar
orphan                    false
path_suffix               n/a
period                    0s
renewable                 true
token_explicit_max_ttl    0s
token_period              24h
token_type                default-service

TF resource:

resource "vault_token_auth_backend_role" "gke-foobar" {
  role_name        = "gke-foobar"
  token_policies   = ["gke-foobar"]
  allowed_policies = ["gke-foobar"]
  token_period     = 86400
  renewable        = true
}

[yermulnik]

Are these related issues?
#533