Hitting timing issue when trying to integrate with aws provider's aws_iam_access_key resource

[mattt416]

Hi Team,

I've got the following Terraform config:

provider "mongodbatlas" {
  version = "= 0.3.1"
}

provider "aws" {
  region     = "us-east-1"
  access_key = "redacted"
  secret_key = "redacted"
}

variable "organization_id" {
  type        = string
  default     = "redacted"
}

resource "mongodbatlas_project" "terraform_project" {
    name   = "Terraform Project"
    org_id = var.organization_id
}

resource "aws_iam_access_key" "mongodb_atlas" {
  user    = "atlas"
}

resource "mongodbatlas_encryption_at_rest" "encryption" {
  project_id = mongodbatlas_project.terraform_project.id

  aws_kms = {
    enabled = true
    access_key_id = aws_iam_access_key.mongodb_atlas.id
    secret_access_key = aws_iam_access_key.mongodb_atlas.secret
    customer_master_key_id = "redacted"
    region = "US_EAST_1"
  }
}

When applied, I'm hitting the following error:

Error: error creating Encryption at Rest: PATCH https://cloud.mongodb.com/api/atlas/v1.0/groups/redacted/encryptionAtRest: 400 (request "Bad Request") Invalid AWS credentials.

  on main.tf line 34, in resource "mongodbatlas_encryption_at_rest" "encryption":
  34: resource "mongodbatlas_encryption_at_rest" "encryption" {

If I immediately re-apply, the resource is created as expected, which implies the access_key_id and secret_access_key being passed to /encryptionAtRest are correct.

Further, if I add a sleep to the "mongodb_atlas" resource, the resource is created the first time without issue:

resource "aws_iam_access_key" "mongodb_atlas" {
  user    = "atlas"

  provisioner "local-exec" {
    command = "sleep 30;"
  }
}

This looks like a timing issue in the aws provider. Would there be value in handling this in the MongoDB Atlas provider, or should a work-around be implemented in the config as was demonstrated above?

Thanks,
Matt

[PacoDw]

Hi! @mattt416 I just made a PR #128 with new changes for Encryption at Rest resource fixing some issues, could you test your configuration in that branch to check if all is okay?

Thank you so much! if you have another comment or concern just let us know 👍

[mattt416]

Hey @PacoDw ! I've just built binary from #128 but am still hitting the same issue it seems.

[PacoDw]

Thank you so much! Let me check it out to resolve that, so I will try making the change in ASAP.

[mattt416]

Thanks @PacoDw ! I think just discussing whether this sort of "work-around" should be added to the Atlas provider would be highly appreciated.

[themantissa]

My .02 is that trying to bring this into the Atlas provider as a work around might be problematic. Users who already have the access key shouldn't run into this. It seems to happen when one tries to create the key right before setting up encryption at rest. If we had an easy way to check if hte key really exists or not we could od that but I think that could be challenging. The other option might be a standard x amount of sleep before trying to enable encryption at rest in case the key is not there yet but we'd need to figure out a good time amount and no idea what that should be.

@mattt416 I think this could be, potentially a good thing to note in our doc. Something like "ensure you give sufficient timing between the creation of an encryption key and applying it to ensure the key exists". Thoughts?

[mattt416]

@themantissa This is the first time I've come across this issue, so it might be worth leaving this issue open for a while so we can collect more data from other users who are seeing similar issues.

[jlforester]

I agree that this really isn't a mongodbatlas provider issue.

I've encountered this issue as well and my workaround was a little Go program that waits until the IAM user is successfully able to do a describe-keys using the IAM user and access keys created by the module. I have that program called from a provider block on the aws_iam_access_key resource. I'm finding that it's only a 10-20 second wait until after the access keys are created that they become usable.

[themantissa]

Verified simply issue from AWS side so going to document the workaround and then close this issue. Until documented we'll leave this open for others to discover workaround options.

[msandu62]

Having the same issue. Any solutions to this?
400 (request "INVALID_AWS_CREDENTIALS") Invalid AWS credentials.

[nikhil-mongo]

@themantissa Since we have moved to the Role based authentication for Encryption at Rest, this has solved the problem and I do not face it anymore.
Please see if we can close this now.