Terraform Modules

Overview

A monorepo containing custom Terraform child modules. The modules were developed either wholely or primarily by me in a professional capacity over the past few years. They've been modified to allow them to be made public. The Git history has been squashed for the same reason. The entries in the modules' CHANGELOG files have not been squashed to preserve some form of history. The module versions mentioned in the CHANGELOG files, and their corresponding Git tags, no longer exist due to the Git history squash.

The modules are primarily for managing AWS resources and Kubernetes resources (running on EKS) with some Hashicorp Vault resources thrown in for good measure. The modules resources were used to manage applications stored in a self-hosted private Gitlab instance. The applications were deployed using Gitlab CI/CD pipelines. The pipelines were a heavily customized version of Gitlab's built-in Auto DevOps pipeline. The Gitlab pipeline definition for this project has been preserved for posterity. I hope to convert it to Github actions in the near future.

When some of these modules were created, they had to be designed to accommodate existing resources that were created outside of Terraform. As a result, their naming conventions and design decisions don't always make sense anymore. Those resources were often created by custom Ansible roles and/or playbooks. The roles and/or playbooks were used as a model for the initial implementation of the modules.

Every section in this README file after this one is from the project's original README file.

Using a Module

The modules in this project can be called by specifying the module's source as a Git URL. Terraform supports both SSH and HTTPS Git URLs. Every time a version of a module is released, a Git tag is created in this repository. A tag should always be included in the Git URL to pin the module to a specific version. Doing so will ensure consistent, predictable, and repeatable Terraform runs. The tags follow the naming convention of <module name>-v<version> where the version is a semantic version number controlled by the VERSION file in every module. Every new version of a module also creates a Gitlab release.

Module Development

Project Setup

Review the following sections prior to making any changes to the project to ensure a consistent development environment among developers.

Editor

An EditorConfig file is included in this template to help maintain consistent formatting. Be sure to install EditorConfig if you haven't already done so.

Pre-commit hooks

Included in this template is a configuration file for Git pre-commit framework. It contains hooks maintained by Gruntworks that are relevant to Terraform development.

1. Install Terraform 1.6
2. Install tflint
3. Install terraform-docs
4. Run pre-commit install to install the hooks
5. (Optional) verify the hooks are working by running pre-commit run --all-files

Creating a New Module

If you haven't already done so, review Hashicorp's documentation on module creation and development.

• Module Development.
• Module Creation Tutorial
• Module Creation Patterns
• Best Practices for Provider Version Constraints
• Input Variable Documentation
• Output Value Documentation

The workflow for creating a new module is as follows.

1. Create a new Git branch.

2. Create a new directory in the project. The name of the directory corresponds to the name of the module. Modules names must be kebab case to be consistent with the naming convention of public Terraform modules.

3. Create a terraform-docs pre-commit hook for the module in .pre-commit-config.yaml to automatically update the module's README.md file on commit. To create the hook, add a new entry to the list of hooks for the https://github.com/terraform-docs/terraform-docs repo. For example, if the new module is named my-new-module, then the new hook will look like the following YAML snippet.

     - id: terraform-docs-system
       name: "Run terraform-docs for my-new-module"
       files: "^my-new-module/.+"
       args: ["my-new-module"]

4. Copy the contents of the templates/module directory into the new directory. The template contains all of the necessary files for the standard module structure recommended by Hashicorp.

5. Implement and test the new module.

6. Populate the module's README.md file with useful information. Additionally, run terraform-docs to add generated documentation to the README.md file. For example, if the new module is named my-new-module, then the terraform-docs command to run from the project's root directory is the following.

   terraform-docs my-new-module

7. Add the new module to the .gitlab-ci.yml file by creating a new job that extends the .module-build job. For example, if the new module is named my-new-module, then the new job will look like the following.

   my-new-module:
       extends: .module-build

8. Create a merge request

9. Once the merge request has been reviewed and the CI/CD pipeline success, merge the branch.

10. A new pipeline will trigger once the branch has been merged. The new pipeline generates a new tag for the module as well as a Gitlab release.

11. Use the new module in other projects.

Modifying a Module

The workflow for modifying a module is as follows.

1. Create a new Git branch.
2. Update the module and test the changes.
3. Update the module's README.md file, if necessary.
4. Add a new entry to the module's CHANGELOG.md file under the Unreleased header. Refer to the Keep A Changelog documentation to determine what to add to the changelog.
5. Create a merge request
6. Add a link to the merge request to the entries added to the CHANGELOG.md file.
7. Once the merge request has been approved and the CI/CD pipeline success, merge the branch.

Releasing a New Module Version

1. Increment the module's version number in its VERSION file if code changes were made. Refer to the Semantic Versioning documentation to determine which component of the version number to increment.
2. Add a new section to the module's CHANGELOG file for the new version. All content that is currently under the Unreleased header must be moved to the new section.
3. Create a merge request
4. Once the merge request has been approved and the CI/CD pipeline success, merge the branch.
5. A new pipeline will trigger once the branch has been merged. The new pipeline generates a new tag for the module as well as a Gitlab release.
6. Upgrade the module in dependent projects by changing the value of the Git tag in the source attribute of every module call.

Module Testing

Testing a module can be done locally on a developer's machine by using a second Terraform project. In the second project, set the source of the attribute in the module call to the absolute file path of the module's directory in this project. If the second project already calls the module and it is under version control, the source attribute can be overridden by using an override file. Override files are ignored by Git but the values in them take precedence over the values defined in other Terraform files. This ensures that changes made for testing are not accidentally committed to Git.

The workflow for this testing strategy is as follows.

1. Add the file override.tf to the root of the second project.
2. Add override.tf to the second project's .gitignore file.
3. In the override file, make a module call with the same name as the existing module call.
4. Set the source attribute of the module call in the override file to the absolute file path of the module's directory in this project.
5. Run terraform get in the second project to install the new module source.
6. Add or modify module variables as necessary in the override file.
7. Run terraform plan in the second project and make changes to the module source if necessary. Repeat this step until the results of the plan are correct.