NOTE: We are actively experimenting with this in the open. Consider this ALPHA software and subject to change.
Terraform-controller - This is a low level tool to run Git controlled Terraform modules in Kubernetes. The controller manages the TF state file using Kubernetes as a remote statefile backend (requires Terraform 0.13.4)! You can have changes auto-applied or wait for an explicit "OK" before running.
There are two parts to the stack, the controller and the executor.
The controller creates three CRDs and runs controllers for modules and executions. A module is the building block and is the same as a terraform module. This is referenced from an execution which is used to combine all information needed to run Terraform. The execution combines Terraform variables and environment variables from secrets and/or config maps to provide to the executor.
The executor is a job that runs Terraform. Taking input from the execution run CRD the executor runs terraform init, terraform plan and terraform create/destroy depending on the context.
Executions have a 1-to-many relationship with execution runs, as updates or changes are made in the module or execution additional runs are created to update the terraform resources.
Use provided manifests kubectl create -f ./manifests to deploy to an existing k8s cluster. Manifests will create all CRDs necessary and a Deployment with the rancher/terraform-controller image.
~ kubectl get all -n terraform-controller
NAME READY STATUS RESTARTS AGE
pod/terraform-controller-8494cf85c5-x97sn 1/1 Running 0 17s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/terraform-controller 1/1 1 1 18s
NAME DESIRED CURRENT READY AGE
replicaset.apps/terraform-controller-8494cf85c5 1 1 1 18s
Everything is put in the terraform-controller namespace with these provided manifests. Edit metadata.namespace in files to change name space or remove to run in default. You will need to update the args for the command in the deployment to update or remove --namespace argument for the executable. Passing in the flag limits the controller to only watching CRD objects in it's namespace, remove this param to let the terraform-controller see all CRD objects in any namespace.
Use (k3d)[https://github.com/rancher/k3d/releases] to spin up small (k3s)[https://github.com/rancher/k3s] clusters for a quick start for using the Terraform Controller. The appliance image comes pre-built with the deployment manifests and will auto-create verything the Terraform Controller needs when they boot.
~ k3d create --name terraform-controller --image rancher/terraform-controller-appliance
~ export KUBECONFIG="$(k3d get-kubeconfig --name='terraform-controller')"
~ kubectl get all -n terraform-controller
NAME READY STATUS RESTARTS AGE
pod/terraform-controller-d774bbd44-w4mzk 0/1 Pending 0 1s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/terraform-controller 0/1 1 0 14s
NAME DESIRED CURRENT READY AGE
replicaset.apps/terraform-controller-d774bbd44 1 1 0 1s
~ kubectl get crd | grep terraformcontroller
executionruns.terraformcontroller.cattle.io 2019-05-22T17:19:01Z
executions.terraformcontroller.cattle.io 2019-05-22T17:19:01Z
modules.terraformcontroller.cattle.io 2019-05-22T17:19:01Z
~ k3d delete --name terraformWith the controller running you can run the provided example in the ./example directory which shows you how to run a basic Digital Ocean Terraform module which takes a do_token and do_name and creates a droplet.
Modify ./example/00-secret.yaml with your Digital Ocean API token and desired droplet name.
Run kubectl create -f ./example -n terraform-controller to create all envvars/secrets/module and the execution which will automatically run. The controller creates Jobs for the Terraform runs so to access logs check the pod logs for the executor create and destroy jobs. This example is setup to auto-confirm and auto-delete when the CRD object is destroyed.
Delete the droplet by deleting the CRD kubectl delete -f ./example/20-deployment.yaml -n terraform-controller.
In ./example/20-state.yaml its pre-configured to auto-approve and auto-delete when you make the execution CRD. You can turn off spec.destroyOnDelete and spec.autoConfirm and do these by hand doing the following.
To get the plan check logs of the pods used to run the job.
kubectl logs [executer-pod-name] -n terraform-controller
Assuming the action Terraform is going to perform is correct annotate the Execution Run to approve the changes:
kubectl annotate executionruns.terraform-controller.cattle.io [execution-run-name] -n terraform-controller approved="yes" --overwrite
Once the job completes, you can see the outputs from Terraform by checking the Execution Run:
kubectl get executionruns.terraform-controller.cattle.io [execution-run-name] -n terraform-controller -o yaml
With destroyOnDelete turned off you will have to delete the Droplet by hand as a destroy job will not kick off.
Create a Dockerfile
FROM rancher/terraform-controller-executor:v0.0.3 #Or whatever the release is
RUN curl https://myurl.com/get-some-binary
Build that image and push to a registry.
When creating the execution define the image:
apiVersion: terraformcontroller.cattle.io/v1
kind: Execution
metadata:
name: cluster-create
spec:
moduleName: cluster-modules
destroyOnDelete: true
autoConfirm: false
image: cloudnautique/tf-executor-rancher2-provider:v0.0.3 # Custom IMAGE
variables:
SecretNames:
- my-secret
envConfigNames:
- env-config
If you already have an execution, edit the CR via kubectl and add the image field.
make
Use ./bin/terraform-controller
docker run -d -v "/Path/To/Kubeconfig:/root/.kube/config" -e "KUBECONFIG=/root/.kube/config" -e "EXECUTOR_RUN_NAME=RUN_NAME" -e "EXECUTOR_ACTION=create" rancher/terraform-controller-executor:dev
Copyright (c) 2019 Rancher Labs, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.