[Bug]: aws_grafana_workspace_api_key doesn't renew

[algojack]

Terraform Core Version

1.3.0

AWS Provider Version

4.32.0

Affected Resource(s)

• aws_grafana_workspace_api_key
• grafana provider

Expected Behavior

New key is created and terraform shows plan

Actual Behavior

Error: Invalid provider configuration
│ 
│ Provider "registry.terraform.io/grafana/grafana" requires explicit configuration. Add a provider block to the root module and configure the provider's required arguments as described in the provider
│ documentation.
│ 
╵
╷
│ Error: Missing required argument
│ 
│   with provider["registry.terraform.io/grafana/grafana"],
│   on <empty> line 0:
│   (source code not available)
│ 
│ "auth": one of `auth,cloud_api_key,oncall_access_token,sm_access_token` must be specified

Or if i have the grafana_data_source block also:

│ Error: the Grafana client is required for `grafana_data_source`. Set the auth and url provider attributes
│ 
│   with grafana_data_source.prometheus,
│   on main.tf line 50, in resource "grafana_data_source" "prometheus":
│   50: resource "grafana_data_source" "prometheus" {
│ 

Relevant Error/Panic Output Snippet

No response

Terraform Configuration Files

provider "aws" {
  region = "us-west-1"
}

provider "grafana" {
  alias = "my_grafana"
  url   = "https://${aws_grafana_workspace.my_aws_grafana.endpoint}"
  auth  = aws_grafana_workspace_api_key.my_aws_grafana_key.key
}

resource "aws_grafana_workspace" "my_aws_grafana" {
  // timestamp needed to work around duplicate workspace bug in aws managed grafana
  // https://stackoverflow.com/a/73411011/20103507
  description              = "My grafana ${formatdate("YYYYMMDDhhmmss", timestamp())}"
  account_access_type      = "CURRENT_ACCOUNT"
  authentication_providers = ["SAML"]
  permission_type          = "SERVICE_MANAGED"
  role_arn                 = "<replaced role arn here>"
  data_sources             = ["PROMETHEUS"]
}

resource "aws_grafana_workspace_api_key" "my_aws_grafana_key" {
  key_name        = "config-key"
  key_role        = "ADMIN"
  seconds_to_live = 3600
  workspace_id    = aws_grafana_workspace.my_aws_grafana.id
}

# Optionally also
resource "grafana_data_source" "prometheus" {
...
}

Steps to Reproduce

terraform apply
wait for the key to expire
terraform plan

Debug Output

No response

Panic Output

No response

Important Factoids

I'm using terraform to create aws_managed_grafana, and then creating an api key there that is required to have an expiration date. Once it expires, the grafana provider that i'm trying to use to configure this aws_managed_grafana instance, stops working. The whole plan stops working.

References

No response

Would you like to implement a fix?

No

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[algojack]

Apologies if this is posted in the wrong place, please let me know where to post if so.

Summary

When aws_grafana_workspace_api_key expires (required field seconds_to_live) it stops grafana provider from working completely. Can't even terraform destroy.

WorkAround

The only work around I found is to remove the old expired key, rename, and create new one:

terraform state rm 'aws_grafana_workspace_api_key.my_aws_grafana_key'

change name of the key and

terraform plan -out=tfplan -target=aws_grafana_workspace_api_key.my_aws_grafana_key
terraform apply "tfplan"

[algojack]

In Grafanas terraform example, the api key doesn't require expiration date:
https://registry.terraform.io/providers/grafana/grafana/latest/docs#creating-a-grafana-cloud-stack-provider
Therefore maybe that seconds_to_live field should be optional? (or at least the key should renew if expired)

[EricBizet]

In Grafanas terraform example, the api key doesn't require expiration date: https://registry.terraform.io/providers/grafana/grafana/latest/docs#creating-a-grafana-cloud-stack-provider Therefore maybe that seconds_to_live field should be optional? (or at least the key should renew if expired)

Seems like should be the case, maybe AWS made it so for security reasons 🤔 It would be great to lift this constraint

[justinretzolk]

Related #27043

[brianmaresca]

a workaround i've resorted to is an external data source that runs a script to create a key before each run. (deleting a key with the same name first, if one exists)

resource "aws_grafana_workspace" "grafana-workspace" {
  count                    = var.create_grafana_workspace
  name                     = "example-grafana-workspace"
  account_access_type      = "CURRENT_ACCOUNT"
  authentication_providers = ["SAML"]
  data_sources   = ["CLOUDWATCH"]
  permission_type          = "CUSTOMER_MANAGED"
  role_arn                 = aws_iam_role.grafana_cloudwatch_role.arn
}

# get the aws_grafana_workspace data if the above workspace resource is available 
# OR if a grafana_workspace_id output exists in the remote state data
data "aws_grafana_workspace" "optional_grafana_workspace" {
  count = length(aws_grafana_workspace.grafana-workspace) > 0 || try(data.terraform_remote_state.state.outputs.grafana_workspace_id, null) != null ? 1 : 0
  workspace_id = try(aws_grafana_workspace.grafana-workspace.id, try(data.terraform_remote_state.state.outputs.grafana_workspace_id, null))
}

data "external" "grafana_credentials" {
  count = length(data.aws_grafana_workspace.optional_grafana_workspace)
  program    = ["python3", "./scripts/api_key.py"]
  query      = {
    workspace_id          = data.aws_grafana_workspace.optional_grafana_workspace[0].workspace_id
    workspace_endpoint    = format("https://%s", data.aws_grafana_workspace.optional_grafana_workspace[0].endpoint)
    api_key_name          = "terraform"
    aws_region            = var.aws_region
    seconds_to_live       = 3600
  }
}

provider grafana {
  url  = try(data.external.grafana_credentials[0].result.endpoint, "https://unused")
  auth = try(data.external.grafana_credentials[0].result.key, "unused")
}

additionally, in the above example, if var.create_grafana_workspace = true, the plan sets an output value of the grafana workspace id, which can then be referenced in a future plan that may want to destroy the workspace by setting var.create_grafana_workspace = false. in this scenario, the workspace information isn't available from the previously created resource because the count is now 0. and we need the workspace id in order to create a new key to initialize the grafana provider, so that it can read and destroy any resources that use this provider.

so this works around expiring keys, and the case where you want to destroy a workspace that also had additional things added to the workspace by the grafana provider. but it's definitely not ideal

[wilstdu]

Any progress on the issue?

[dievri]

program = ["python3", "./scripts/api_key.py"]

@brianmaresca, where this script can be found?

[brianmaresca]

@dievri it's just a custom python script takes in the inputs defined in query to make boto3 calls to delete the existing key and create + return a fresh one with the same name

[mateusz-rutski-red]

I found a quite easy workaround. You just have to make sure one of the resource attributes' value changes on each run. I used timestamp built-in function and append it to key_name:

resource "aws_grafana_workspace_api_key" "grafana_key" {
  key_name = "terraform-${timestamp()}"
  key_role = "ADMIN"
  seconds_to_live = 180
  workspace_id = aws_grafana_workspace.grafana_workspace.id
}

[mrsufgi]

I found a quite easy workaround. You just have to make sure one of the resource attributes' value changes on each run. I used timestamp built-in function and append it to key_name:

resource "aws_grafana_workspace_api_key" "grafana_key" {
  key_name = "terraform-${timestamp()}"
  key_role = "ADMIN"
  seconds_to_live = 180
  workspace_id = aws_grafana_workspace.grafana_workspace.id
}

Not working, the plan stage fails with:
the Grafana client is required for 'grafana_data_source'. Set the auth and url provider attributes

btw not even removing the keys from the state change anything, only deleting the datasource state and datasource from grafana..

EDIT:

key_name = "terraform-${timestamp()}" is not enough; this will work only if your key wasn't already expired.
changing to a fixed value, "terraform-1" will work.

undesired. I wonder how so many people use it in production without this affecting their workflow.

[dannygoulder]

watch out with these workarounds - as mentioned here, AWS will charge you a monthly fee for each API key, so $8 per terraform run.

[brianmaresca]

@dannygoulder i think you are billed for active keys. so if you're creating a new key each terraform run without cleaning up the keys made from previous runs, then i think you would be getting hit with a charge per key. but if you delete and recreate the key each time i don't think there would be an additional charge. im still not 100% sure of this, not only is the api key pricing unclear and buried in the pricing docs, in the aws user guides and the aws terraform docs there are detailed instructions on the various ways to create an api key. of course they both conveniently have no reminder that you will be charged every single time you follow those steps.

[timurv-da]

@brianmaresca @dannygoulder unfortunately, they will charge you for every created API key (I'm talking about NOT enterprise version). Any key which was used at least one time is active for 30 days.

[brianmaresca]

@timurv-da i spoke with an AWS representative and asked point blank "are you charged every time you create an api key". response:

From what I'm seeing on the pricing page, it's just based on user licenses and creating/deleting API keys shouldn't impact pricing.

There are no charges to create, delete or edit.

You don't get charged for api keys you just have a quota limit per workspace. 

I also shared this github issue with the AWS rep, and she said that she will try to have someone respond here with the pricing clarification. 🤞

[timurv-da]

@brianmaresca I also had a conversation with AWS representative =)
It is strange, but they say about some users, their tokens.. But API keys are not linked to any user, obviously. Maybe it is about 9.* versions "service accounts" which can have many tokens.

[PedroOrona]

The service accounts could solve the problem but there isn't a service_account resource provided by AWS provider. This leads to solving this with a lambda function (or any other script) to automatically renew the API key every time it reaches the time limit.

This solution can be the fix for now but we also know that Grafana is about to deprecate API keys which is a big problem. That's why having a service account resource would be the final fix for these problems.