Enhancement | Check and Update layers to use terraform, modules and provider resources in the latest versions

[rodriguez-matias]

What?

• Keep Updated all Terraform config on every layer.
• Keep all versions update changes registered in one place.

How?

• Check and Update versions of Terraform Core, Providers, and Modules.
• Get the latest release version from Terraform Registry.
• Update version constraints.
• Test layers with new versions and report potentials issues and parameters changes.

Why?

• Keeping Leverage Reference Architecture up to date.

---

Versions to consider for updates:

leverage cli: "v1.9.2"
  
terraform {
  required_version = "~> 1.3.5"

  required_providers {
    aws        = "~> 4.10"
    kubernetes = "~> 2.10"
    helm       = "~> 2.5"
    vault      = "~> 3.6"  
}

---

ChangeLog

• 14/02/23 : PR Enhacement | base-dns: Update versions constraint #475
• 03/03/23 : PR Migrate from JumpCloud to AWS IAM Identity Center (SSO) #481
• 22/03/23 : PR Enhacement | tools-cloud-nuke: Update versions constraint #489

---

Current Version Upgrade Status

├── apps-devstg
│ ├── global
│ │ └── base-identities ✅ (tf > 1.x / tf-aws > 4.x)
│ │ └── cli-test-layer ✅ (tf > 1.x / tf-aws > 4.x)
│ ├── us-east-1
│ │ ├── backups\ -- ✅ (tf > 1.x / tf-aws > 4.x)
│ │ ├── base-certificates ✅ (tf > 1.x / tf-aws > 4.x)
│ │ ├── base-network ✅ (tf > 1.x / tf-aws > 4.x)
│ │ ├── base-tf-backend ✅ (tf > 1.x / tf-aws > 4.x)
│ │ ├── cdn-s3-frontend\ -- ✅ (tf > 1.x / tf-aws = 3.x)
│ │ ├── databases-aurora ✅ (tf > 1.x / tf-aws > 4.x)
│ │ ├── databases-mysql\ -- ✅ (tf > 1.x / tf-aws > 4.x)
│ │ ├── databases-pgsql\ --
│ │ ├── ec2-fleet-ansible\ -- ✅ (tf > 1.x / tf-aws > 4.x)
│ │ ├── k8s-eks
│ │ │ ├── cluster ✅ (tf > 1.x / tf-aws > 4.x / tf-k8s > 2.x)
│ │ │ ├── identities ✅ (tf > 1.x / tf-aws > 4.x / tf-k8s > 2.x)
│ │ │ ├── k8s-resources. ✅ (tf > 1.x / tf-aws > 4.x / tf-k8s > 2.x)
│ │ │ ├── k8s-workloads. ✅ (tf > 1.x / tf-aws > 4.x / tf-k8s > 2.x)
│ │ │ └── network. ✅ (tf > 1.x / tf-aws > 4.x / tf-k8s > 2.x)
│ │ ├── k8s-eks-demoapps
│ │ │ ├── cluster ✅ (tf > 1.x / tf-aws > 4.x / tf-k8s > 2.x)
│ │ │ ├── identities ✅ (tf > 1.x / tf-aws > 4.x / tf-k8s > 2.x)
│ │ │ ├── k8s-resources. ✅ (tf > 1.x / tf-aws > 4.x / tf-k8s > 2.x)
│ │ │ ├── k8s-workloads. ✅ (tf > 1.x / tf-aws > 4.x / tf-k8s > 2.x)
│ │ │ └── network. ✅ (tf > 1.x / tf-aws > 4.x / tf-k8s > 2.x)
│ │ ├── k8s-eks-v1.17
│ │ │ ├── cluster ✅ (tf > 1.x / tf-aws > 4.x / tf-k8s > 2.x)
│ │ │ ├── identities ✅ (tf > 1.x / tf-aws > 4.x / tf-k8s > 2.x)
│ │ │ ├── k8s-resources. ✅ (tf > 1.x / tf-aws > 4.x / tf-k8s > 2.x)
│ │ │ ├── k8s-workloads. ✅ (tf > 1.x / tf-aws > 4.x / tf-k8s > 2.x)
│ │ │ └── network. ✅ (tf > 1.x / tf-aws > 4.x / tf-k8s > 2.x)
│ │ ├── k8s-kind
│ │ ├── k8s-kops\ --
│ │ ├── notifications ✅ (tf > 1.x / tf-aws > 4.x)
│ │ ├── security-audit ✅ (tf > 1.x / tf-aws > 4.x)
│ │ ├── security-base ✅ (tf > 1.x / tf-aws > 4.x)
│ │ ├── security-certs ✅ (tf > 1.x / tf-aws > 4.x)
│ │ ├── security-compliance\ --
│ │ ├── security-firewall\ -- ✅ (tf > 1.x / tf-aws > 4.x)
│ │ ├── security-keys ✅ (tf > 1.x / tf-aws > 4.x)
│ │ ├── storage
│   │ │ └── s3-bucket-demo-files. ✅ (tf > 1.x / tf-aws > 4.x)
│ │ └── tools-cloud-nuke ✅ (tf > 1.x / tf-aws > 4.x)
│ └── us-east-2
│ ├── k8s-eks-v.1.17
│ ├── security-compliance\ --
│ └── security-keys ✅ (tf > 1.x / tf-aws > 4.x)
|
├── apps-prd
│ ├── global
│ │ └── base-identities ✅ (tf > 1.x / tf-aws > 4.x)
│ └── us-east-1
│ ├── backups\ -- ✅ (tf > 1.x / tf-aws > 4.x)
│ ├── base-network ✅ (tf > 1.x / tf-aws > 4.x)
│ ├── base-tf-backend ✅ (tf > 1.x / tf-aws > 4.x)
│ ├── cdn-s3-frontend\ -- ✅ (tf > 1.x / tf-aws = 3.x)
│ ├── ec2-fleet\ -- ✅ (tf > 1.x / tf-aws > 4.x)
│ ├── notifications ✅ (tf > 1.x / tf-aws > 4.x)
│ ├── security-audit ✅ (tf > 1.x / tf-aws > 4.x)
│ ├── security-base ✅ (tf > 1.x / tf-aws > 4.x)
│ ├── security-certs ✅ (tf > 1.x/ tf-aws > 4.x)
│ ├── security-compliance\ --
│ └── security-keys ✅ (tf > 1.x / tf-aws > 4.x)
|
├── management
│ ├── global
│ │ ├── base-identities ✅ (tf > 1.x / tf-aws > 4.x)
│ │ ├── cost-mgmt
│ │ ├── organizations
│ │ └── sso ✅ (tf > 1.x / tf-aws > 4.x)
│ ├── us-east-1
│ │ ├── backups
│ │ ├── base-tf-backend ✅ (tf > 1.x / tf-aws > 4.x)
│ │ ├── firewall-manager
│ │ ├── notifications ✅ (tf > 1.x / tf-aws > 4.x)
│ │ ├── security-audit ✅ (tf > 1.x / tf-aws > 4.x)
│ │ ├── security-base ✅ (tf > 1.x / tf-aws > 4.x)
│ │ ├── security-compliance
│ │ ├── security-keys ✅ (tf > 1.x / tf-aws > 4.x)
│ │ └── security-monitoring ✅ (tf > 1.x/ tf-aws > 4.x)
│ └── us-east-2
│ └── security-monitoring\ -- ✅ (tf > 1.x/ tf-aws > 4.x)
|
├── network
│ ├── global
│ │ └── base-identities ✅ (tf > 1.x / tf-aws > 4.x)
│ ├── us-east-1
│ │ ├── base-network ✅ (tf > 1.x / tf-aws > 4.x)
│ │ ├── base-tf-backend ✅ (tf > 1.x / tf-aws > 4.x)
│ │ ├── network-firewall
│ │ ├── notifications ✅ (tf > 1.x / tf-aws > 4.x)
│ │ ├── security-audit ✅ (tf > 1.x / tf-aws > 4.x)
│ │ ├── security-base ✅ (tf > 1.x / tf-aws > 4.x)
│ │ ├── security-compliance\ --
│ │ ├── security-keys ✅ (tf > 1.x / tf-aws > 4.x)
│ │ └── transit-gateway
│ └── us-east-2
│ ├── base-network ✅ (tf > 1.x / tf-aws > 4.x)
│ ├── network-firewall
│ ├── security-compliance\ --
│ ├── security-keys ✅ (tf > 1.x / tf-aws > 4.x)
│ └── transit-gateway
|
├── security
│ ├── global
│ │ └── base-identities ✅ (tf > 1.x / tf-aws > 4.x)
│ ├── us-east-1
│ │ ├── base-tf-backend ✅ (tf > 1.x / tf-aws > 4.x)
│ │ ├── firewall-manager
│ │ ├── notifications ✅ (tf > 1.x / tf-aws > 4.x)
│ │ ├── security-audit ✅ (tf > 1.x / tf-aws > 4.x)
│ │ ├── security-base ✅ (tf > 1.x / tf-aws > 4.x)
│ │ ├── security-compliance\ --
│ │ ├── security-keys ✅ (tf > 1.x / tf-aws > 4.x)
│ │ └── security-monitoring ✅ (tf > 1.x/ tf-aws > 4.x)
│ └── us-east-2
│ ├── security-audit
│ ├── security-compliance\ --
│ └── security-monitoring\ -- ✅ (tf > 1.x/ tf-aws > 4.x)
|
└── shared
├── global
│ ├── base-dns ✅ (tf > 1.x / tf-aws > 4.x)
│ └── base-identities ✅ (tf > 1.x / tf-aws > 4.x)
├── us-east-1
│ ├── backups ✅ (tf > 1.x / tf-aws > 4.x)
│ ├── base-network ✅ (tf > 1.x / tf-aws > 4.x)
│ ├── base-tf-backend ✅ (tf > 1.x / tf-aws > 4.x)
│ ├── container-registry ✅ (tf > 1.3.x / tf-aws > 4.10)
│ ├── ec2-fleet\ -- ✅ (tf > 1.x / tf-aws > 4.x)
│ ├── ec2-fleet-bastions\ --
│ ├── k8s-eks
│ ├── k8s-eks-demoapps
│ ├── k8s-eks-prd
│ ├── notifications ✅ (tf > 1.x / tf-aws > 4.x)
│ ├── secrets-manager\ --
│ ├── security-audit ✅ (tf > 1.x / tf-aws > 4.x)
│ ├── security-base ✅ (tf > 1.x / tf-aws > 4.x)
│ ├── security-compliance\ --
│ ├── security-keys ✅ (tf > 1.x / tf-aws > 4.x)
│ ├── storage
│ │ ├── backup-gdrive ✅ (tf > 1.x / tf-aws > 4.x)
│ │ ├── object-file-shares-for-users-list ✅ (tf > 1.x / tf-aws > 4.x)
│ │ ├── object-file-shares-for-sftp ✅ (tf > 1.x / tf-aws > 4.x)
│ ├── tools-cloud-scheduler-stop-start
│ ├── tools-eskibana
│ ├── tools-github-selfhosted-runners
│ ├── tools-jenkins\ --
│ ├── tools-managedeskibana
│ ├── tools-prometheus
│ ├── tools-vault
│ ├── tools-vpn-server ✅ (tf > 1.x / tf-aws > 4.x)
│ └── tools-webhooks\ --
└── us-east-2
├── base-network ✅ (tf > 1.x / tf-aws > 4.x)
├── container-registry ✅ (tf > 1.3.x / tf-aws > 4.10)
├── security-compliance\ --
├── security-keys ✅ (tf > 1.x / tf-aws > 4.x)
├── tools-eskibana
└── tools-prometheus

---

Ref Links

• https://github.com/tfverch/tfvc
• https://github.com/minamijoyo/tfupdate
• Info about tfupdate: Enhancement | Check and Update Ref Architecture layers to use terraform, module and provider resources in latest versions #370 (comment)
• Steps to update le-ref-arch layers: Enhancement | Check and Update Ref Architecture layers to use terraform, module and provider resources in latest versions #370 (comment)

[rodriguez-matias]

Layer le-tf-infra-aws/management/global/sso updated by this PR #481

[rodriguez-matias]

Hey, I found this custom Github Action workflow that is a wrapper for the tfupdate command:

tfupdate-action
https://github.com/HENNGE/tfupdate-action

I'll be testing this workflow to try to automate the steps for update the layers.

[exequielrafaela]

@rodriguez-matias may be as part of this ongoing layer effort we can start integrating default tags in the terraform aws provider as best practice. Ideally having a dedicated issue for this could help segmenting the scope for this task and being able to prioritize accordingly. So If you have a few mins and can create the issue I really appreciate it. I think it should look somthing similar too:

One consideration I haven't tested yet is the possibility to parametrize this default tags and pass it arguments from our common configs tfvars files.

CC: @binbashar/leverage-ref-architecture-aws-admin @binbashar/leverage-ref-architecture-aws-dev

[exequielrafaela]

This new leverage cli feature request binbashar/leverage#259 should help with this.

[exequielrafaela]

@lgallard Let's discuss the upgrade of the following layers:

• https://www.binbash.co/services-catalog/landing-zone
• https://www.binbash.co/services-catalog/security-baseline
• https://www.binbash.co/services-catalog/cost-monitoring
• https://leverage.binbash.co/user-guide/ref-architecture-eks/overview/#control-plane

@diego-ojeda-binbash Let's review and adjust this versions accordingly:

leverage cli: "v1.9.2"
  
terraform {
  required_version = "~> 1.3.5"

  required_providers {
    aws        = "~> 4.10"
    kubernetes = "~> 2.10"
    helm       = "~> 2.5"
    vault      = "~> 3.6"  
}

[diego-ojeda-binbash]

@exequielrafaela @lgallard
Sure, here:

leverage cli: "v1.12.2"     => latest available if possible
  
terraform {
  required_version = "~> 1.6"      => latest available if possible, if it fails due to constraints we can move to 1.5

  required_providers {
    aws        = "~> 5.0"
    kubernetes = "~> 2.10"            => latest available if possible, must be tested by standing up the cluster
    helm       = "~> 2.5"             => latest available if possible, must be tested by standing up the cluster
    vault      = "~> 3.6"             => we don't use this any more
}

And keep in mind you can spin up the demoapps cluster using these instructions: https://binbash.atlassian.net/wiki/spaces/BDPS/pages/2270527489/DemoApps#Standing-up-the-DemoApps