Unify cluster state generation and saving of assets

[robszumski]

Today we have diverged slightly between installs via GUI or CLI for how the state and assets are saved after a cluster is brought up. Below are the differences and ways to solve this:

Installer GUI

• If user saves state manually, they have an assets.zip where ever they downloaded it
• If user doesn't save manually, the state is still accessible at <installer-root>/<os>/clusters/<name>_<date>

CLI with Terraform

• State is stored at <installer-root> in the default location (and managed with terraform env)
• Generated assets are stored at <installer-root>/generated
• TF vars are stored at <installer-root>/build/<name>

Problems

1. CLI instructions don't hint that you should use terraform env
2. CLI generated assets are not namespaced under the cluster name, ensuring that you overwrite your important secrets if you ever run another cluster
3. The filename of the GUI state doesn't match the CLI flow, which makes it hard for our destroy to be accurate for both.
4. The automatically saved assets folder is in a really weird location, right beside the binary you ran.
5. Absolute paths are generated based on this really weird location. Moving the assets will break future operations.

If you follow the GUI installer, here's the high level output, namespaced by cluster:

$ tree my-cluster-name_2017-07-11_10-29-54
├── config.tf
├── generated
├── license.txt
├── logs
├── modules
├── platforms
├── pull_secret.json
├── terraform.tfstate
└── terraform.tfvars

The generated folder contains all of the secrets and other data. This is also included/generated by Terraform CLI flows, but not in a namespaced location, as mentioned above.

$ tree tectonic/generated
├── auth
├── bootkube.sh
├── bootstrap-experimental
├── bootstrap-manifests
├── etcd
├── experimental
├── manifests
├── tectonic
├── tectonic-rkt.sh
├── tectonic.sh
└── tls

And the state is in the default location:

$ tree tectonic
├── build
├── build/clustername
...
├── generated
...
├── terraform.tfstate

Proposal

No matter how you install, generate assets that are as similar as possible in structure. For example, match folder names when possible, and add on additional things like TF state.

1. We store content in a single location, namespaced by cluster. tectonic/clusters/<name>/
2. Relative paths are used over absolute paths
3. Store generated content in this directory
4. Use explicit state files in this directory
5. Update our plan commands to send their output here
6. Pass the plan output to apply via flag. This is more proper and repeatable.
7. Update the destroy commands to point to the new state location
8. Update destroy guide to use the new format
9. This consistent directory structure will help folks destroying via UI (GUI: Make destroying clusters way easier #462)

[robszumski]

@kbrwn This is related to the issue that you had destroying

@kans Related to the destroy workflow we were talking about the other day

[so0k]

just had this issue when trying to create multiple clusters on AWS using Terraform and the CLI.

I tried using -state=build/${CLUSTER}/terraform.tfstate platforms/aws only to notice my assets of my first cluster were all overwritten.

I didn't know about the terraform env feature.

Apart from the docs update, also need to add tectonic_cluster_name everywhere generated is referenced?

[robszumski]

@so0k yes, those generated assets should be at clusters/<tectonic_cluster_name>/generated

[robszumski]

@sym3tri I'd like to bump priority of this. You can't destroy from just the state downloaded from the UI:

1. it doesn't have the original version of terraform
2. it's not the complete set of files (compare the copy saved next to the binary)

Here's what I see:

$ cd ~/Downloads/clustername_2017-07-12_13-24-20
$ export TERRAFORM_CONFIG=$(pwd)/.terraformrc
$ export AWS_ACCESS_KEY_ID=******
$ export AWS_SECRET_ACCESS_KEY= ******
$ /path/to/random-installer-i-happened-to-have/tectonic-installer/darwin/terraform destroy -state=terraform.tfstate
...stuff...
Error refreshing state: 10 error(s) occurred:

* module.bootkube.template_dir.bootstrap-experimental: template_dir.bootstrap-experimental: could not generate output checksum: lstat /path/to-where-happened-to-run-the-installer/tectonic-installer/darwin/clusters/clustername_2017-07-12_13-24-20/.terraform/modules/0391b25f8df845582a6b6a555d42296b/resources/experimental/bootstrap-manifests: no such file or directory
* module.bootkube.template_dir.bootkube-bootstrap: template_dir.bootkube-bootstrap: could not generate output checksum: lstat /path/to-where-happened-to-run-the-installer/tectonic-installer/darwin/clusters/clustername_2017-07-12_13-24-20/.terraform/modules/0391b25f8df845582a6b6a555d42296b/resources/bootstrap-manifests: no such file or directory
* module.bootkube.template_dir.etcd-experimental: template_dir.etcd-experimental: could not generate output checksum: lstat /path/to-where-happened-to-run-the-installer/tectonic-installer/darwin/clusters/clustername_2017-07-12_13-24-20/.terraform/modules/0391b25f8df845582a6b6a555d42296b/resources/experimental/etcd: no such file or directory
* module.bootkube.template_dir.experimental: template_dir.experimental: could not generate output checksum: lstat /path/to-where-happened-to-run-the-installer/tectonic-installer/darwin/clusters/clustername_2017-07-12_13-24-20/.terraform/modules/0391b25f8df845582a6b6a555d42296b/resources/experimental/manifests: no such file or directory
* module.tectonic.template_dir.tectonic: template_dir.tectonic: could not generate output checksum: lstat /path/to-where-happened-to-run-the-installer/tectonic-installer/darwin/clusters/clustername_2017-07-12_13-24-20/.terraform/modules/7f25271fe38c01e56cf0f5ea6a745446/resources/manifests: no such file or directory
* module.bootkube.template_dir.bootkube: template_dir.bootkube: could not generate output checksum: lstat /path/to-where-happened-to-run-the-installer/tectonic-installer/darwin/clusters/clustername_2017-07-12_13-24-20/.terraform/modules/0391b25f8df845582a6b6a555d42296b/resources/manifests: no such file or directory

[so0k]

I believe this is an upstream bug in the terraform template and local providers and created a ticket for it here hashicorp/terraform-provider-template#19

seems the yelp gitfile provider had similar issue - Yelp/terraform-provider-gitfile#2

[so0k]

@sym3tri - was I added to CoreOSdev atlassian jira by mistake? Wasn't sure where to contact you

[sym3tri]

@so0k sorry about that. There was an issue with our issue import. Should be resolved now.