Having trouble using vault creds with datadog provider

[nitrocode]

I'm unsure if this is a limitation in terraform, an issue with the datadog provider, or an issue with the vault provider. If it's not with the vault provider, I'll close it and reopen for a different provider. Thank you.

Terraform Version

0.11.8

Affected Resource(s)

N/A

Terraform Configuration Files

provider "vault" {
  address = "https://vault.mycompany.com"
}

data "vault_generic_secret" "datadog_api" {
  path = "secret/datadog/provisioning/api"
}

data "vault_generic_secret" "datadog_app" {
  path = "secret/datadog/provisioning/app"
}

provider "datadog" {
  api_key = "${data.vault_generic_secret.datadog_api.data.value}"
  app_key = "${data.vault_generic_secret.datadog_app.data.value}"
  #api_key = "${data.vault_generic_secret.datadog_api.data["value"]}"
  #app_key = "${data.vault_generic_secret.datadog_app.data["value"]}"
}

# example
resource "datadog_monitor" "disk_above_90" {
  name               = "Disk Usage Above 90%"
  type               = "metric alert"
  message            = "message"
  escalation_message = "message"
  query = "avg(last_5m):avg:system.disk.in_use by {host} > 0.95"
  thresholds {
    ok       = 0
    warning  = 0.9
    critical = 0.95
  }
  notify_no_data    = false
  renotify_interval = 240
}

Debug Output

Error: Error refreshing state: 1 error(s) occurred:

* provider.datadog: No valid credential sources found for Datadog Provider. Please see https://terraform.io/docs/providers/datadog/index.html for more information on providing credentials for the Datadog Provider

Expected Behavior

It should have gotten the credentials from vault and supplied them to datadog.

Actual Behavior

It got the credentials from vault according to terraform show but did not feed them into the datadog provider.

Steps to Reproduce

1. terraform init && terraform apply

Important Factoids

Inserting the same values that I retrieved from vault directly into the terraform allowed it to work.

References

None

[cvbarros]

@nitrocode I think your problem lies within a limitation of Terraform itself, not related to Vault Provider. AFAIK, you can't depend on resource/unknown values for configuring provider blocks. Please see:
https://www.terraform.io/docs/configuration/providers.html#interpolation

[nitrocode]

@cvbarros interesting. thank you for responding. Would you happen to know what the correct way to pass in credentials to providers? do we need to hit vault manually to get values, then pass in those values using tfvars or input vars?

any idea if better integration is on a future roadmap?

[coryflucas]

Per https://www.terraform.io/docs/providers/vault/index.html#using-vault-credentials-in-terraform-configuration this seems like it is a supported use case for the Vault provider. The Terraform documents also mention that using data sources should work:

It is always valid to use input variables and data sources whose configurations do not in turn depend on as-yet-unknown values.

@nitrocode is it possible the credentials in vault are not valid? You get that error if they credentials provided are not valid.

[nitrocode]

@coryflucas Thanks for that link. Then this should work correctly. I'm using the exact same credentials. If I take the values directly from vault and pass them in as an input variable they work. If I use the vault provider, the datadog provider doesn't recognize the credentials.

Sounds more like a vault provider or terraform bug.

[cvbarros]

@coryflucas From my understanding, that paragraph refers that "other kinds of providers use credentials...", whilst the Vault provider works differently: It issues credentials for itself by creating a child token which doesn't need any prior configuration in the provider block.

A secret retrieved from Vault would fall into as-yet-unknown value classification, IMO, that can only figured out at "apply" time, when resolving the resource graph.

[cvbarros]

But indeed I wish this would work as @nitrocode posted, it would be very helpful.

[cvbarros]

If you go through this issue in Terraform, that appears to be the case:
hashicorp/terraform#4149

Which has been motivated by:
hashicorp/terraform#12393

[coryflucas]

I think this is an issue with the Datadog provider. I tried the AWS provider with credentials from a data source and it works correctly.

This fails:

data "null_data_source" "values" {
  inputs = {
    datadog_api_key = "..."
    datadog_app_key = "..."
  }
}

provider "datadog" {
  api_key = "${data.null_data_source.values.outputs["datadog_api_key"]}"
  app_key = "${data.null_data_source.values.outputs["datadog_app_key"]}"
}

resource "datadog_screenboard" "dummy" {
  title     = "asdf123"
  read_only = true

  widget {
    ...
  }
}

But this works:

data "null_data_source" "values" {
  inputs = {
    aws_access_key = "..."
    aws_secret_key = "..."
  }
}

provider "aws" {
  access_key = "${data.null_data_source.values.outputs["aws_access_key"]}"
  secret_key = "${data.null_data_source.values.outputs["aws_secret_key"]}"
  region     = "us-east-1"
}

resource "aws_eip" "dummy" {}

[cvbarros]

@coryflucas This would only be explained due to differences in provider schemas for AWS and Datadog providers, then.
While the Datadog provider has some DefaultFunc and Required: true for some attributes in it's configuration, the AWS providers defers this reading to for providerConfigure func / config.Client().

When you have tested this, did you have a shared credentials AWS file in your machine, or anything in your environment that may be used to configure the AWS Provider?

[nitrocode]

@cvbarros I use aws-vault to store my credentials in environment variables then call terraform so the aws provider definitely picked that up but why would that conflict with datadog?

[cvbarros]

@nitrocode I was referring to the test performed by Cory above, not to your specific problem.
In that context, we need to understand why interpolated resource dependencies work for AWS Provider but is not working for the Datadog provider. I'll set up some tests tomorrow with different providers to see if this is the case.

[coryflucas]

@cvbarros I can confirm I ran my test with no environmental AWS credentials. In addition I've done something like this with the Kubernetes and Helm providers without issue.

I believe the issue is that the Datadog providers tries to verify it has working API credentials in it's intialization code. The other providers seems to wait untill the resources are accessed to do so, so the call is delayed until the dependencies have been created and thus the values are there.

[nitrocode]

@coryflucas thanks for confirming. I rewrote this ticket on the repo terraform-provider-datadog.

[nitrocode]

Hi @coryflucas and @cvbarros . You guys were right and the current implementation forces initializing the provider prior to all the data sources finishing. I'm taking a whack at updating the provider to allow it to use vault as a source for its credentials. Here's what I have so far.

Here is my allow-vault branch and here is the comparison with master.

I've only attempted at modifying a single resource to see if it would work. It grabs the vault credentials but there is a race condition where GetClient() is run at the same time as the datadog resource creation which makes datadog think there aren't any keys associated with the provider and so the rest call returns a 400. When I run this I can see the vault creds are retrieved successfully in the debug output.

rm -rf .terraform/ && terraform init > /dev/null 2>&1
TF_LOG=trace terraform apply -auto-approve

I tried to follow the kubernetes and helm provider plugins but I must be missing something... Do I need a waiter?

If anyone has an idea, I'm very interested in continuing to learn go and contribute.