Add backend configs for terraform remote state storage

docs/howto/operate/new-tf-cluster.md

@@ -31,7 +31,7 @@ Once you have created this file, open a Pull Request to the `pilot-hubs` repo fo
 
 ## Initialising Terraform
 
-The terraform state is located centrally in our `two-eye-two-see-org` GCP project, therefore you must authenticate `gcloud` to your `@2i2c.org` account before initialising terraform.
+Our default terraform state is located centrally in our `two-eye-two-see-org` GCP project, therefore you must authenticate `gcloud` to your `@2i2c.org` account before initialising terraform.
 
 ```bash
 gcloud auth application-default login
@@ -41,26 +41,18 @@ Then you can change into the terraform directory and initialise
 
 ```bash
 cd terraform
-terraform init
+terraform init -backend-config=backends/default-backend.hcl -reconfigure
 ```
 
 ````{note}
-If you are deploying a cluster to a project you have access to via a different account, such as a university-affliated account, there are a few extra steps to take.
-Hopefully, this will be a rare scenario.
+If you are working on a project which you cannot access with your 2i2c account, there are other backend config files stored in `terraform/backends` that will configure a different storage bucket to read/write the remote terraform state.
+This saves us the pain of having to handle multiple authentications as these storage buckets are within the project we are trying to deploy to.
 
-First, you will need to provide terraform with an [access token](https://www.terraform.io/docs/language/settings/backends/gcs.html#configuration-variables) to use the state files.
-You can generate one using the below commands and logging in with your 2i2c.org account.
+For example, to work with Pangeo you would initialise terraform like so:
 
 ```bash
-gcloud auth application-default login
-gcloud auth application-default print-access-token
+terraform init -backend-config=pangeo-backend.hcl -reconfigure
 ```
-
-Add the access token to the [terraform backend block](https://github.com/2i2c-org/pilot-hubs/blob/2ef8a4bf35bb5ee9bf04ab3db1218b8c183c5da2/terraform/main.tf#L2-L5) in `main.tf`.
-**DO NOT COMMIT THIS CHANGE.**
-Then run `terraform init` or `terraform init -reconfigure`.
-
-You can now login to your other gcloud account and proceed with the guide.
 ````
 
 ## Creating a new terraform workspace
@@ -72,6 +64,11 @@ Create a new workspace with the below command, and again give it the same name a
 terraform workspace new WORKSPACE_NAME
 ```
 
+```{note}
+Workspaces are defined **per backend**.
+If you can't find the workspace you're looking for, double check you've enabled the correct backend.
+```
+
 ## Plan and Apply Changes
 
 ```{note}

docs/topic/terraform.md

@@ -13,4 +13,67 @@ file with variable definitions for that cluster.
 
 Workspaces are stored centrally in the `two-eye-two-see-org` GCP project, even
 when we use Terraform for projects running on AWS / Azure. You must have
-access to this project before you can use terraform for our infrastructure.
+access to this project before you can use terraform for our infrastructure.
+
+You can initialise using the following command
+
+```bash
+terraform init -backend-config=backends/default-backend.hcl -reconfigure
+```
+
+```{note}
+Workspaces are defined **per backend**.
+If you can't find the workspace you're looking for, double check you've enabled the correct backend.
+```
+
+## Other remote state storage
+
+For some projects where we don't have access to using our 2i2c accounts, e.g. universities that require us to have specific university-affiliated identities, we can configure different backends to access the terraform state stored in those projects.
+Working this way saves us the pain of trying to work with terraform using two different authentications.
+The backend configs are stored in [`terraform/backends`](https://github.com/2i2c-org/pilot-hubs/tree/master/terraform/backends) and can be used by running `terraform init -backend-config=backends/NAME_OF_CHOSEN_BACKEND -reconfigure`.
+For example, for our Pangeo projects, run:
+
+```bash
+terraform init -backend-config=backends/pangeo-backend.hcl -reconfigure
+```
+
+## How to switch Terraform workspaces
+
+### If the new workspace is stored in the same backend as the current workspace
+
+If you want to switch to a different terraform workspace that is stored in the same backend that you initialised with, you can simply run:
+
+```bash
+terraform workspace switch WORKSPACE_NAME
+```
+
+For example, if you were working in the `pilot-hubs` workspace but want to switch to `justiceinnovationlab`, these are both stored in the same backend and so the command looks like:
+
+```bash
+terraform workspace switch justiceinnovationlab
+```
+
+````{note}
+For the majority of day-to-day work, this will be the prevalent workflow provided you have initialised terraform with
+
+```bash
+terraform init -backend-config=backends/default-backend.hcl -reconfigure
+```
+````
+
+### If the new workspace is stored in a different backend to the current workspace
+
+To switch between workspaces that are stored in _different_ backends, terraform will need to be reinitialised in order to pick up the new backend.
+The commands, therefore, are:
+
+```bash
+terraform init -backend-config=backends/<REQUIRED_CONFIG>.hcl -reconfigure
+terraform workspace select WORKSPACE_NAME
+```
+
+For example, if you were working on our `pilot-hubs`, with our default backend initialised, but wanted to switch to working on our Pangeo deployments, the commands would look as follows:
+
+```bash
+terraform init -backend-config=backends/pangeo-backend.hcl -reconfigure
+terraform workspace select pangeo-hubs
+```

terraform/backends/default-backend.hcl

@@ -0,0 +1,2 @@
+bucket = "two-eye-two-see-org-terraform-state"
+prefix = "terraform/state/pilot-hubs"

terraform/backends/pangeo-backend.hcl

@@ -0,0 +1,2 @@
+bucket = "pangeo-terraform-state"
+prefix = "terraform/state"

terraform/main.tf

@@ -1,8 +1,5 @@
 terraform {
-  backend "gcs" {
-    bucket = "two-eye-two-see-org-terraform-state"
-    prefix = "terraform/state/pilot-hubs"
-  }
+  backend "gcs" {}
 }
 
 // Service account used by all the nodes and pods in our cluster