Terraform remote state backends: Dynamic configuration implementation and storage

[sgibson91]

Background

I have been trying to work with terraform across two authentications for a while now. The current workflow looks like the below.

• I need to be logged into 2i2c to be able to access our remote state backend on two-eye-two-see
• My 2i2c account doesn't have access to the project where the infrastructure will be deployed, so...
• Print out an access token while logged into my 2i2c account and add it to the backend block in main.tf. Always be vigilant not to check this into version control.
• Log into another account and rerun terraform init, terraform plan, etc...

This was working fine until my token expired mid-way through a terraform action and then state lock got me into a whole heap of trouble! Thankfully I was the only one using that workspace and the infrastructure is definitely not ready for production deployment yet, so not much is lost.

It would be a lot easier in this case if the remote backend was stored in the same project I was trying to deploy to so I didn't have to worry about multi-authentication. So...

How do we dynamically configure the remote backend?

First off, I tried to parameterise this problem. Have a variable such as backend that would point towards two-eye-two-see when set to default, and then set it to something else to point to another bucket somewhere else. I tried every trick I could think of, but it was all quite hacky and didn't really work.

But, terraform init has an optional flag -backend-config that accepts a path to a backend.hcl file. This file is formatted similarly to a tfvars file with key-value pairs. This seems like the solution to the problem!

First off, I need to test to see if providing a backend config file on the command line will automatically overwrite what is provided in main.tf. If yes, then that's great! Behaviour won't change for the majority of the team.

Actions

• Create a storage bucket in Columbia project for remote state storage
• Create a backend.hcl file pointing to Columbia bucket
• Test this correctly overwrites the config in main.tf as expected

[sgibson91]

Can confirm that passing -backend-config overwrites what's in main.tf! 🎉

[sgibson91]

PR: #561