feat: Terraform AWS Provider Version 4 Upgrade (#460)

* feat: Terraform AWS Provider Version 4 Upgrade
BREAKING CHANGE: The module is upgraded to Terraform AWS provider 4.x. All new development will only support the new AWS Terraform provider. We keep a branch `terraform-aws-provider-3` to witch we welcome backports to AWS Terraform 3.x provider. Besides reviewing PR's we will do not any active checking on maintance on this branch. We strongly advise to update your deployment to the new provider version. For more details about upgrading see the [upgrade guide](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/guides/version-4-upgrade).

* feat: Add metadata options for docker mahcine (#490)
BREAKING CHANGE:  By default, AWS metadata service ((IMDSv2)[https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html]) is enabled and required for both the agent instance and the docker machine instance. For docker machine this require the GitLab managed docker machines distribution is used. Which the module usages by default.


Co-authored-by: Matthias Kay <[email protected]>
Co-authored-by: Mustafa Abdul-Kader <[email protected]>
Co-authored-by: Steve Wilson <[email protected]>

.github/workflows/ci.yml

@@ -25,7 +25,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        terraform: [0.15.0, 1.0.11, latest]
+        terraform: [1.0.11, latest]
         example:
           [
             "runner-default",

.pre-commit-config.yaml

@@ -6,6 +6,7 @@ repos:
         args:
           - --args=-recursive
       - id: terraform_tflint
+      - id: terraform_docs
   - repo: https://github.com/pre-commit/pre-commit-hooks
     rev: v4.2.0
     hooks:

examples/runner-default/main.tf

@@ -43,7 +43,6 @@ module "runner" {
 
   gitlab_runner_security_group_ids = [data.aws_security_group.default.id]
 
-  docker_machine_download_url   = "https://gitlab-docker-machine-downloads.s3.amazonaws.com/v0.16.2-gitlab.2/docker-machine"
   docker_machine_spot_price_bid = "on-demand-price"
 
   gitlab_runner_registration_config = {

examples/runner-default/versions.tf

@@ -1,22 +1,22 @@
 
 terraform {
-  required_version = ">= 0.15"
+  required_version = ">= 1"
   required_providers {
     aws = {
       source  = "hashicorp/aws"
-      version = "~> 3.74"
+      version = "~> 4.7"
     }
     local = {
       source  = "hashicorp/local"
-      version = "~> 2.2"
+      version = "~> 2"
     }
     null = {
       source  = "hashicorp/null"
       version = "~> 3.0"
     }
     tls = {
       source  = "hashicorp/tls"
-      version = "~> 3.3"
+      version = "~> 3"
     }
     random = {
       source  = "hashicorp/random"

examples/runner-docker/versions.tf

@@ -1,22 +1,22 @@
 
 terraform {
-  required_version = ">= 0.15"
+  required_version = ">= 1"
   required_providers {
     aws = {
       source  = "hashicorp/aws"
-      version = "~> 3.74"
+      version = "~> 4.7"
     }
     local = {
       source  = "hashicorp/local"
-      version = "~> 2.2"
+      version = "~> 2"
     }
     null = {
       source  = "hashicorp/null"
       version = "~> 3.0"
     }
     tls = {
       source  = "hashicorp/tls"
-      version = "~> 3.3"
+      version = "~> 3"
     }
     random = {
       source  = "hashicorp/random"

examples/runner-multi-region/README.md

@@ -130,4 +130,4 @@ No output.
 ## Outputs
 
 No outputs.
-<!-- END_TF_DOCS -->
+<!-- END_TF_DOCS -->

examples/runner-multi-region/_docs/README.md

@@ -0,0 +1,18 @@
+# Example - Spot Runner - Public subnets
+
+In this scenario the multiple runner agents can be created with different configuration by instantiating the module multiple times. Runners will scale automatically based on configuration. The S3 cache can be shared cross runners by managing the cache outside the module.
+
+![runners-cache](https://github.com/npalm/assets/raw/master/images/terraform-aws-gitlab-runner/runner-cache.png)
+
+This examples shows:
+- Usages of public subnets.
+- Usages of multiple runner instances sharing a common cache.
+- Overrides for tag naming.
+- Registration via GitLab token.
+- Auto scaling using `docker+machine` executor.
+- Register runner as [protected](https://docs.gitlab.com/ee/ci/runners/configure_runners.html#prevent-runners-from-revealing-sensitive-information). 
+
+
+## Prerequisite
+
+The Terraform version is managed using [tfenv](https://github.com/Zordrak/tfenv). If you are not using `tfenv` please check `.terraform-version` for the tested version.

examples/runner-multi-region/_docs/TF_MODULE.md

@@ -0,0 +1,25 @@
+## Providers
+
+| Name | Version |
+|------|---------|
+| aws | 2.56 |
+| local | 1.4 |
+| null | 2.1.2 |
+| tls | 2.1.1 |
+
+## Inputs
+
+| Name | Description | Type | Default | Required |
+|------|-------------|------|---------|:-----:|
+| aws\_region | AWS region. | `string` | `"eu-west-1"` | no |
+| environment | A name that identifies the environment, will used as prefix and for tagging. | `string` | `"runner-public"` | no |
+| gitlab\_url | URL of the gitlab instance to connect to. | `string` | `"https://gitlab.com"` | no |
+| private\_ssh\_key\_filename | n/a | `string` | `"generated/id_rsa"` | no |
+| public\_ssh\_key\_filename | n/a | `string` | `"generated/id_rsa.pub"` | no |
+| registration\_token | n/a | `any` | n/a | yes |
+| runner\_name | Name of the runner, will be used in the runner config.toml | `string` | `"public-auto"` | no |
+
+## Outputs
+
+No output.
+

examples/runner-multi-region/versions.tf

@@ -1,22 +1,22 @@
 
 terraform {
-  required_version = ">= 0.15"
+  required_version = ">= 1"
   required_providers {
     aws = {
       source  = "hashicorp/aws"
-      version = "~> 3.74"
+      version = "~> 4.7"
     }
     local = {
       source  = "hashicorp/local"
-      version = "~> 2.2"
+      version = "~> 2"
     }
     null = {
       source  = "hashicorp/null"
       version = "~> 3.0"
     }
     tls = {
       source  = "hashicorp/tls"
-      version = "~> 3.3"
+      version = "~> 3"
     }
     random = {
       source  = "hashicorp/random"

examples/runner-pre-registered/versions.tf

@@ -1,22 +1,22 @@
 
 terraform {
-  required_version = ">= 0.15"
+  required_version = ">= 1"
   required_providers {
     aws = {
       source  = "hashicorp/aws"
-      version = "~> 3.74"
+      version = "~> 4.7"
     }
     local = {
       source  = "hashicorp/local"
-      version = "~> 2.2"
+      version = "~> 2"
     }
     null = {
       source  = "hashicorp/null"
       version = "~> 3.0"
     }
     tls = {
       source  = "hashicorp/tls"
-      version = "~> 3.3"
+      version = "~> 3"
     }
     random = {
       source  = "hashicorp/random"

examples/runner-public/versions.tf

@@ -1,22 +1,22 @@
 
 terraform {
-  required_version = ">= 0.15"
+  required_version = ">= 1"
   required_providers {
     aws = {
       source  = "hashicorp/aws"
-      version = "~> 3.74"
+      version = "~> 4.7"
     }
     local = {
       source  = "hashicorp/local"
-      version = "~> 2.2"
+      version = "~> 2"
     }
     null = {
       source  = "hashicorp/null"
       version = "~> 3.0"
     }
     tls = {
       source  = "hashicorp/tls"
-      version = "~> 3.3"
+      version = "~> 3"
     }
     random = {
       source  = "hashicorp/random"

locals.tf

@@ -1,7 +1,7 @@
 locals {
   // Convert list to a string separated and prepend by a comma
   docker_machine_options_string = format(
-    ",%s",
+    ",\"amazonec2-metadata-token=${var.docker_machine_instance_metadata_options.http_tokens}\", \"amazonec2-metadata-token-response-hop-limit=${var.docker_machine_instance_metadata_options.http_put_response_hop_limit}\",%s",
     join(",", formatlist("%q", concat(var.docker_machine_options, local.runners_docker_registry_mirror_option))),
   )
 

main.tf

@@ -160,7 +160,16 @@ resource "aws_autoscaling_group" "gitlab_runner_instance" {
   health_check_grace_period = 0
   max_instance_lifetime     = var.asg_max_instance_lifetime
   enabled_metrics           = var.metrics_autoscaling
-  tags                      = local.agent_tags_propagated
+
+  dynamic "tag" {
+    for_each = local.agent_tags
+
+    content {
+      key                 = tag.key
+      value               = tag.value
+      propagate_at_launch = true
+    }
+  }
 
   launch_template {
     id      = aws_launch_template.gitlab_runner_instance.id
@@ -277,8 +286,10 @@ resource "aws_launch_template" "gitlab_runner_instance" {
   tags = local.tags
 
   metadata_options {
-    http_endpoint = var.runner_instance_metadata_options_http_endpoint
-    http_tokens   = var.runner_instance_metadata_options_http_tokens
+    http_endpoint               = var.runner_instance_metadata_options.http_endpoint
+    http_tokens                 = var.runner_instance_metadata_options.http_tokens
+    http_put_response_hop_limit = var.runner_instance_metadata_options.http_put_response_hop_limit
+    instance_metadata_tags      = var.runner_instance_metadata_options.instance_metadata_tags
   }
 
   lifecycle {

modules/cache/README.md

@@ -25,109 +25,3 @@ module "runner" {
 }
 ```
 <!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
-## Requirements
-
-| Name | Version |
-|------|---------|
-| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.15 |
-
-## Providers
-
-| Name | Version |
-|------|---------|
-| <a name="provider_aws"></a> [aws](#provider\_aws) | n/a |
-| <a name="provider_random"></a> [random](#provider\_random) | n/a |
-
-## Modules
-
-No modules.
-
-## Resources
-
-| Name | Type |
-|------|------|
-| [aws_iam_policy.docker_machine_cache](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_policy) | resource |
-| [aws_s3_bucket.build_cache](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket) | resource |
-| [aws_s3_bucket_public_access_block.build_cache_policy](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_public_access_block) | resource |
-| [random_string.s3_suffix](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/string) | resource |
-| [aws_caller_identity.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/caller_identity) | data source |
-
-## Inputs
-
-| Name | Description | Type | Default | Required |
-|------|-------------|------|---------|:--------:|
-| <a name="input_arn_format"></a> [arn\_format](#input\_arn\_format) | ARN format to be used. May be changed to support deployment in GovCloud/China regions. | `string` | `"arn:aws"` | no |
-| <a name="input_cache_bucket_name_include_account_id"></a> [cache\_bucket\_name\_include\_account\_id](#input\_cache\_bucket\_name\_include\_account\_id) | Boolean to add current account ID to cache bucket name. | `bool` | `true` | no |
-| <a name="input_cache_bucket_prefix"></a> [cache\_bucket\_prefix](#input\_cache\_bucket\_prefix) | Prefix for s3 cache bucket name. | `string` | `""` | no |
-| <a name="input_cache_bucket_set_random_suffix"></a> [cache\_bucket\_set\_random\_suffix](#input\_cache\_bucket\_set\_random\_suffix) | Random string suffix for s3 cache bucket | `bool` | `false` | no |
-| <a name="input_cache_bucket_versioning"></a> [cache\_bucket\_versioning](#input\_cache\_bucket\_versioning) | Boolean used to enable versioning on the cache bucket, false by default. | `string` | `"false"` | no |
-| <a name="input_cache_expiration_days"></a> [cache\_expiration\_days](#input\_cache\_expiration\_days) | Number of days before cache objects expires. | `number` | `1` | no |
-| <a name="input_cache_lifecycle_clear"></a> [cache\_lifecycle\_clear](#input\_cache\_lifecycle\_clear) | Enable the rule to cleanup the cache for expired objects. | `bool` | `true` | no |
-| <a name="input_cache_lifecycle_prefix"></a> [cache\_lifecycle\_prefix](#input\_cache\_lifecycle\_prefix) | Object key prefix identifying one or more objects to which the clean up rule applies. | `string` | `"runner/"` | no |
-| <a name="input_create_cache_bucket"></a> [create\_cache\_bucket](#input\_create\_cache\_bucket) | This module is by default included in the runner module. To disable the creation of the bucket this parameter can be disabled. | `bool` | `true` | no |
-| <a name="input_environment"></a> [environment](#input\_environment) | A name that identifies the environment, used as prefix and for tagging. | `string` | n/a | yes |
-| <a name="input_name_iam_objects"></a> [name\_iam\_objects](#input\_name\_iam\_objects) | Set the name prefix of all AWS IAM resources created by this module | `string` | `""` | no |
-| <a name="input_tags"></a> [tags](#input\_tags) | Map of tags that will be added to created resources. By default resources will be tagged with name and environment. | `map(string)` | `{}` | no |
-
-## Outputs
-
-| Name | Description |
-|------|-------------|
-| <a name="output_arn"></a> [arn](#output\_arn) | The ARN of the created bucket. |
-| <a name="output_bucket"></a> [bucket](#output\_bucket) | Name of the created bucket. |
-| <a name="output_policy_arn"></a> [policy\_arn](#output\_policy\_arn) | Policy for users of the cache (bucket). |
-<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
-
-<!-- BEGIN_TF_DOCS -->
-## Requirements
-
-| Name | Version |
-|------|---------|
-| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.15 |
-
-## Providers
-
-| Name | Version |
-|------|---------|
-| <a name="provider_aws"></a> [aws](#provider\_aws) | n/a |
-| <a name="provider_random"></a> [random](#provider\_random) | n/a |
-
-## Modules
-
-No modules.
-
-## Resources
-
-| Name | Type |
-|------|------|
-| [aws_iam_policy.docker_machine_cache](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_policy) | resource |
-| [aws_s3_bucket.build_cache](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket) | resource |
-| [aws_s3_bucket_public_access_block.build_cache_policy](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_public_access_block) | resource |
-| [random_string.s3_suffix](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/string) | resource |
-| [aws_caller_identity.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/caller_identity) | data source |
-
-## Inputs
-
-| Name | Description | Type | Default | Required |
-|------|-------------|------|---------|:--------:|
-| <a name="input_arn_format"></a> [arn\_format](#input\_arn\_format) | ARN format to be used. May be changed to support deployment in GovCloud/China regions. | `string` | `"arn:aws"` | no |
-| <a name="input_cache_bucket_name_include_account_id"></a> [cache\_bucket\_name\_include\_account\_id](#input\_cache\_bucket\_name\_include\_account\_id) | Boolean to add current account ID to cache bucket name. | `bool` | `true` | no |
-| <a name="input_cache_bucket_prefix"></a> [cache\_bucket\_prefix](#input\_cache\_bucket\_prefix) | Prefix for s3 cache bucket name. | `string` | `""` | no |
-| <a name="input_cache_bucket_set_random_suffix"></a> [cache\_bucket\_set\_random\_suffix](#input\_cache\_bucket\_set\_random\_suffix) | Random string suffix for s3 cache bucket | `bool` | `false` | no |
-| <a name="input_cache_bucket_versioning"></a> [cache\_bucket\_versioning](#input\_cache\_bucket\_versioning) | Boolean used to enable versioning on the cache bucket, false by default. | `string` | `"false"` | no |
-| <a name="input_cache_expiration_days"></a> [cache\_expiration\_days](#input\_cache\_expiration\_days) | Number of days before cache objects expires. | `number` | `1` | no |
-| <a name="input_cache_lifecycle_clear"></a> [cache\_lifecycle\_clear](#input\_cache\_lifecycle\_clear) | Enable the rule to cleanup the cache for expired objects. | `bool` | `true` | no |
-| <a name="input_cache_lifecycle_prefix"></a> [cache\_lifecycle\_prefix](#input\_cache\_lifecycle\_prefix) | Object key prefix identifying one or more objects to which the clean up rule applies. | `string` | `"runner/"` | no |
-| <a name="input_create_cache_bucket"></a> [create\_cache\_bucket](#input\_create\_cache\_bucket) | This module is by default included in the runner module. To disable the creation of the bucket this parameter can be disabled. | `bool` | `true` | no |
-| <a name="input_environment"></a> [environment](#input\_environment) | A name that identifies the environment, used as prefix and for tagging. | `string` | n/a | yes |
-| <a name="input_name_iam_objects"></a> [name\_iam\_objects](#input\_name\_iam\_objects) | Set the name prefix of all AWS IAM resources created by this module | `string` | `""` | no |
-| <a name="input_tags"></a> [tags](#input\_tags) | Map of tags that will be added to created resources. By default resources will be tagged with name and environment. | `map(string)` | `{}` | no |
-
-## Outputs
-
-| Name | Description |
-|------|-------------|
-| <a name="output_arn"></a> [arn](#output\_arn) | The ARN of the created bucket. |
-| <a name="output_bucket"></a> [bucket](#output\_bucket) | Name of the created bucket. |
-| <a name="output_policy_arn"></a> [policy\_arn](#output\_policy\_arn) | Policy for users of the cache (bucket). |
-<!-- END_TF_DOCS -->