e2e: improve provisioning defaults and documentation

This changeset improves the ergonomics of running the Nomad e2e test
provisioning process by defaulting to a blank `nomad_sha` in the
Terraform configuration. By default, a user will now need to pass in
one of the Nomad version flags. But they won't have to manually edit
the `provisioning.json` file for the common case of deploying a
released version of Nomad, and won't need to put dummy values for
`nomad_sha`.

Includes general documentation improvements.

e2e/README.md

@@ -34,39 +34,32 @@ provisioning configuration file for the e2e framework.
 terraform output provisioning | jq . > ../provisioning.json
 ```
 
-By default the `provisioning.json` will include a `nomad_sha` field
-for each node. You can edit this file to change the version of Nomad
-you want to deploy. Because each node has its own value, you can
-create cluster of mixed versions. The provisioning framework accepts
-any of the following options:
-
-- `nomad_sha`: This is a Nomad binary identified by its full commit SHA that's
-  stored in a shared s3 bucket that Nomad team developers can access. That
-  commit SHA can be from any branch that's pushed to remote.  (Ex.
-  `"nomad_sha": "0b6b475e7da77fed25727ea9f01f155a58481b6c"`)
-- `nomad_local_binary`: This is a path to a Nomad binary on your own host.
-  (Ex. `"nomad_local_binary": "/home/me/nomad"`)
-- `nomad_version`: This is a version number of Nomad that's been released to
-  HashiCorp. (Ex. `"nomad_version": "0.10.2"`)
-
-You can pass the following flags to `go test` to override the values
-in `provisioning.json` for all nodes:
-
-- `-nomad.local_file=string`: provision this specific local binary of Nomad
-- `-nomad.sha=string`: provision this specific sha from S3
+By default the `provisioning.json` will not include the Nomad version
+that will be deployed to each node. You can pass the following flags
+to `go test` to set the version for all nodes:
+
+- `-nomad.local_file=string`: provision this specific local binary of
+  Nomad. This is a path to a Nomad binary on your own
+  host. Ex. `-nomad.local_file=/home/me/nomad`
+- `-nomad.sha=string`: provision this specific sha from S3. This is a
+  Nomad binary identified by its full commit SHA that's stored in a
+  shared s3 bucket that Nomad team developers can access. That commit
+  SHA can be from any branch that's pushed to
+  remote. Ex. `-nomad.sha=0b6b475e7da77fed25727ea9f01f155a58481b6c`
 - `-nomad.version=string`: provision this version from
-  [releases.hashicorp.com](https://releases.hashicorp.com/nomad)
+  [releases.hashicorp.com](https://releases.hashicorp.com/nomad). Ex. `-nomad.version=0.10.2`
 
-Deploy Nomad to the cluster:
+Then deploy Nomad to the cluster by passing `-provision.terraform`
+without a Nomad version flag:
 
 ```sh
-# from the ./e2e/terraform directory, set your client environment
-$(terraform output environment)
-
-cd ..
-go test -v . -provision.terraform ./provisioning.json -skipTests
+go test -v . -nomad.version=0.10.2 -provision.terraform ./provisioning.json -skipTests
 ```
 
+Because it can take a little while for the cluster to settle, it's
+recommended to run this provisioning step (with `-skipTests`) first,
+and then run tests as separate step.
+
 ## Running
 
 After completing the provisioning step above, you can set the client
@@ -84,6 +77,7 @@ go test -v .
 If you want to run a specific suite, you can specify the `-suite` flag as
 shown below. Only the suite with a matching `Framework.TestSuite.Component`
 will be run, and all others will be skipped.
+
 ```sh
 go test -v -suite=Consul .
 ```
@@ -95,3 +89,63 @@ names in the full name of the tests:
 ```sh
 go test -v . -run 'TestE2E/Consul/\*consul\.ScriptChecksE2ETest/TestGroup'
 ```
+
+## I Want To...
+
+### ...SSH Into One Of The Test Machines
+
+You can use the Terraform output to find the IP address. The keys will
+in the `./terraform/keys/` directory.
+
+```sh
+ssh -i keys/nomad-e2e-*.pem ubuntu@${EC2_IP_ADDR}
+```
+
+### ...Deploy a Cluster of Mixed Nomad Versions
+
+The `provisioning.json` file output by Terraform has a blank field for
+`nomad_sha` for each node of the cluster (server and client). You can
+manually edit the file to replace this value with a `nomad_sha`,
+`nomad_local_binary`, or `nomad_version` for each node to create a
+cluster of mixed versions. The provisioning framework accepts any of
+the following options for those fields:
+
+- `nomad_sha`: This is a Nomad binary identified by its full commit
+  SHA that's stored in a shared s3 bucket that Nomad team developers
+  can access. That commit SHA can be from any branch that's pushed to
+  remote.  (Ex.  `"nomad_sha":
+  "0b6b475e7da77fed25727ea9f01f155a58481b6c"`)
+- `nomad_local_binary`: This is a path to a Nomad binary on your own
+  host.  (Ex. `"nomad_local_binary": "/home/me/nomad"`)
+- `nomad_version`: This is a version number of Nomad that's been
+  released to HashiCorp. (Ex. `"nomad_version": "0.10.2"`)
+
+Then deploy Nomad to the cluster by passing `-provision.terraform`
+without a Nomad version flag:
+
+```sh
+go test -v . -provision.terraform ./provisioning.json -skipTests
+```
+
+### ...Deploy Custom Configuration Files
+
+The `provisioning.json` file includes a `bundles` section for each
+node of the cluster (server and client). You can manually edit this
+file to add, remove, or replace
+
+```json
+"bundles": [
+  {
+    "destination": "/ops/shared/nomad/base.hcl",
+    "source": "/home/me/custom.hcl"
+  }
+]
+```
+
+### ...Deploy More Than 4 Linux Clients
+
+Right now the framework doesn't support this out-of-the-box because of
+the way the provisioning script adds specific client configurations to
+each client node (for constraint testing). You'll need to add
+additional configuration files to
+`./e2e/terraform/shared/nomad/indexed`.

e2e/terraform/README.md

@@ -10,7 +10,7 @@ that can be written to a JSON file used by the e2e framework's
 provisioning step.
 
 You can use Terraform to output the provisioning parameter JSON file the e2e
-framework uses by setting the `nomad_sha` variable.
+framework uses.
 
 ## Setup
 
@@ -19,27 +19,21 @@ and `AWS_SECRET_ACCESS_KEY`) to create the Nomad cluster. Use
 [envchain](https://github.com/sorah/envchain) to store your AWS credentials.
 
 Optionally, edit the `terraform.tfvars` file to change the number of
-Linux clients, Windows clients, or the Nomad build. You'll usually
-want to have the `nomad_sha` variable set here (or via the
-`TF_VAR_nomad_sha` env var) so that `terraform output provisioning` is
-populated with the build you want.
+Linux clients or Windows clients.
 
 ```hcl
 region               = "us-east-1"
 instance_type        = "t2.medium"
 server_count         = "3"
 client_count         = "4"
 windows_client_count = "1"
-
-# alternatively, set this via env var: TF_VAR_nomad_sha
-# nomad_sha            = ""
 ```
 
 Run Terraform apply to deploy the infrastructure:
 
 ```sh
 cd e2e/terraform/
-TF_VAR_nomad_sha=<nomad_sha> envchain nomadaws terraform apply
+envchain nomadaws terraform apply
 ```
 
 ## Outputs
@@ -75,10 +69,9 @@ ssh -i keys/nomad-e2e-*.pem Administrator@${EC2_IP_ADDR}
 
 ## Teardown
 
-The terraform state file stores all the info, so the `nomad_sha`
-doesn't need to be valid during teardown.
+The terraform state file stores all the info.
 
 ```sh
 cd e2e/terraform/
-envchain nomadaws TF_VAR_nomad_sha=yyyzzz terraform destroy
+envchain nomadaws terraform destroy
 ```

e2e/terraform/main.tf

@@ -34,7 +34,8 @@ variable "windows_client_count" {
 }
 
 variable "nomad_sha" {
-  description = "The sha of Nomad to run"
+  description = "The sha of Nomad to write to provisioning output"
+  default     = ""
 }
 
 provider "aws" {

e2e/terraform/terraform.tfvars

@@ -3,6 +3,3 @@ instance_type        = "t2.medium"
 server_count         = "3"
 client_count         = "4"
 windows_client_count = "1"
-
-# alternatively, set this via env var: TF_VAR_nomad_sha
-# nomad_sha            = ""