diff --git a/.gitignore b/.gitignore index 88c4a51d9..c5986d3b7 100644 --- a/.gitignore +++ b/.gitignore @@ -22,6 +22,8 @@ # Except for these specific files. # These are the index files for the components, modules, and GitHub Actions. !/docs/components/library/aws/aws.mdx +!/docs/components/library/aws/spacelift/README.md +!/docs/components/library/aws/tgw/README.md !/docs/modules/library/artifactory/artifactory.mdx !/docs/modules/library/cloudflare/cloudflare.mdx !/docs/modules/library/example/example.mdx diff --git a/docs/components/library/aws/spacelift/README.md b/docs/components/library/aws/spacelift/README.md new file mode 100644 index 000000000..501fa1666 --- /dev/null +++ b/docs/components/library/aws/spacelift/README.md @@ -0,0 +1,413 @@ +--- +tags: + - layer/spacelift + - provider/aws + - provider/spacelift +--- + +# spacelift + +These components are responsible for setting up Spacelift and include three components: `spacelift/admin-stack`, +`spacelift/spaces`, and `spacelift/worker-pool`. + +Spacelift is a specialized, Terraform-compatible continuous integration and deployment (CI/CD) platform for +infrastructure-as-code. It's designed and implemented by long-time DevOps practitioners based on previous experience +with large-scale installations - dozens of teams, hundreds of engineers and tens of thousands of cloud resources. + +## Stack Configuration + +Spacelift exists outside of the AWS ecosystem, so we define these components as unique to our standard stack +organization. Spacelift Spaces are required before tenant-specific stacks are created in Spacelift, and the root +administrator stack, referred to as `root-gbl-spacelift-admin-stack`, also does not belong to a specific tenant. +Therefore, we define both outside of the standard `core` or `plat` stacks directories. That root administrator stack is +responsible for creating the tenant-specific administrator stacks, `core-gbl-spacelift-admin-stack` and +`plat-gbl-spacelift-admin-stack`. + +Our solution is to define a spacelift-specific configuration file per Spacelift Space. Typically our Spaces would be +`root`, `core`, and `plat`, so we add three files: + +```diff ++ stacks/orgs/NAMESPACE/spacelift.yaml ++ stacks/orgs/NAMESPACE/core/spacelift.yaml ++ stacks/orgs/NAMESPACE/plat/spacelift.yaml +``` + +### Global Configuration + +In order to apply common Spacelift configuration to all stacks, we need to set a few global Spacelift settings. The +`pr-comment-triggered` label will be required to trigger stacks with GitHub comments but is not required otherwise. More +on triggering Spacelift stacks to follow. + +Add the following to `stacks/orgs/NAMESPACE/_defaults.yaml`: + +```yaml +settings: + spacelift: + workspace_enabled: true # enable spacelift by default + before_apply: + - spacelift-configure-paths + before_init: + - spacelift-configure-paths + - spacelift-write-vars + - spacelift-tf-workspace + before_plan: + - spacelift-configure-paths + labels: + - pr-comment-triggered +``` + +Furthermore, specify additional tenant-specific Space configuration for both `core` and `plat` tenants. + +For example, for `core` add the following to `stacks/orgs/NAMESPACE/core/_defaults.yaml`: + +```yaml +terraform: + settings: + spacelift: + space_name: core +``` + +And for `plat` add the following to `stacks/orgs/NAMESPACE/plat/_defaults.yaml`: + +```yaml +terraform: + settings: + spacelift: + space_name: plat +``` + +### Spacelift `root` Space + +The `root` Space in Spacelift is responsible for deploying the root administrator stack, `admin-stack`, and the Spaces +component, `spaces`. This Spaces component also includes Spacelift policies. Since the root administrator stack is unique +to tenants, we modify the stack context to create a unique stack slug, `root-gbl-spacelift`. + +`stacks/orgs/NAMESPACE/spacelift.yaml`: + +```yaml +import: + - mixins/region/global-region + - orgs/NAMESPACE/_defaults + - catalog/terraform/spacelift/admin-stack + - catalog/terraform/spacelift/spaces + +# These intentionally overwrite the default values +vars: + tenant: root + environment: gbl + stage: spacelift + +components: + terraform: + # This admin stack creates other "admin" stacks + admin-stack: + metadata: + component: spacelift/admin-stack + inherits: + - admin-stack/default + settings: + spacelift: + root_administrative: true + labels: + - root-admin + - admin + vars: + enabled: true + root_admin_stack: true # This stack will be created in the root space and will create all the other admin stacks as children. + context_filters: # context_filters determine which child stacks to manage with this admin stack + administrative: true # This stack is managing all the other admin stacks + root_administrative: false # We don't want this stack to also find itself in the config and add itself a second time + labels: + - admin + # attachments only on the root stack + root_stack_policy_attachments: + - TRIGGER Global administrator + # this creates policies for the children (admin) stacks + child_policy_attachments: + - TRIGGER Global administrator +``` + +#### Deployment + +> [!TIP] +> +> The following steps assume that you've already authenticated with Spacelift locally. + +First deploy Spaces and policies with the `spaces` component: + +```bash +atmos terraform apply spaces -s root-gbl-spacelift +``` + +In the Spacelift UI, you should see each Space and each policy. + +Next, deploy the `root` `admin-stack` with the following: + +```bash +atmos terraform apply admin-stack -s root-gbl-spacelift +``` + +Now in the Spacelift UI, you should see the administrator stacks created. Typically these should look similar to the +following: + +```diff ++ root-gbl-spacelift-admin-stack ++ root-gbl-spacelift-spaces ++ core-gbl-spacelift-admin-stack ++ plat-gbl-spacelift-admin-stack ++ core-ue1-auto-spacelift-worker-pool +``` + +> [!TIP] +> +> The `spacelift/worker-pool` component is deployed to a specific tenant, stage, and region but is still deployed by the +> root administrator stack. Verify the administrator stack by checking the `managed-by:` label. + +Finally, deploy the Spacelift Worker Pool (change the stack-slug to match your configuration): + +```bash +atmos terraform apply spacelift/worker-pool -s core-ue1-auto +``` + +### Spacelift Tenant-Specific Spaces + +A tenant-specific Space in Spacelift, such as `core` or `plat`, includes the administrator stack for that specific Space +and _all_ components in the given tenant. This administrator stack uses `var.context_filters` to select all components +in the given tenant and create Spacelift stacks for each. Similar to the root administrator stack, we again create a +unique stack slug for each tenant. For example `core-gbl-spacelift` or `plat-gbl-spacelift`. + +For example, configure a `core` administrator stack with `stacks/orgs/NAMESPACE/core/spacelift.yaml`. + +```yaml +import: + - mixins/region/global-region + - orgs/NAMESPACE/core/_defaults + - catalog/terraform/spacelift/admin-stack + +vars: + tenant: core + environment: gbl + stage: spacelift + +components: + terraform: + admin-stack: + metadata: + component: spacelift/admin-stack + inherits: + - admin-stack/default + settings: + spacelift: + labels: # Additional labels for this stack + - admin-stack-name:core + vars: + enabled: true + context_filters: + tenants: ["core"] + labels: # Additional labels added to all children + - admin-stack-name:core # will be used to automatically create the `managed-by:stack-name` label + child_policy_attachments: + - TRIGGER Dependencies +``` + +Deploy the `core` `admin-stack` with the following: + +```bash +atmos terraform apply admin-stack -s core-gbl-spacelift +``` + +Create the same for the `plat` tenant in `stacks/orgs/NAMESPACE/plat/spacelift.yaml`, update the tenant and +configuration as necessary, and deploy with the following: + +```bash +atmos terraform apply admin-stack -s plat-gbl-spacelift +``` + +Now all stacks for all components should be created in the Spacelift UI. + +## Triggering Spacelift Runs + +Cloud Posse recommends two options to trigger Spacelift stacks. + +### Triggering with Policy Attachments + +Historically, all stacks were triggered with three `GIT_PUSH` policies: + +1. [GIT_PUSH Global Administrator](https://github.com/cloudposse/terraform-spacelift-cloud-infrastructure-automation/blob/main/catalog/policies/git_push.administrative.rego) + triggers admin stacks +2. [GIT_PUSH Proposed Run](https://github.com/cloudposse/terraform-spacelift-cloud-infrastructure-automation/blob/main/catalog/policies/git_push.proposed-run.rego) + triggers Proposed runs (typically Terraform Plan) for all non-admin stacks on Pull Requests +3. [GIT_PUSH Tracked Run](https://github.com/cloudposse/terraform-spacelift-cloud-infrastructure-automation/blob/main/catalog/policies/git_push.tracked-run.rego) + triggers Tracked runs (typically Terraform Apply) for all non-admin stacks on merges into `main` + +Attach these policies to stacks and Spacelift will trigger them on the respective git push. + +### Triggering with GitHub Comments (Preferred) + +Atmos support for `atmos describe affected` made it possible to greatly improve Spacelift's triggering workflow. Now we +can add a GitHub Action to collect all affected components for a given Pull Request and add a GitHub comment to the +given PR with a formatted list of the affected stacks. Then Spacelift can watch for a GitHub comment event and then +trigger stacks based on that comment. + +In order to set up GitHub Comment triggers, first add the following `GIT_PUSH Plan Affected` policy to the `spaces` +component. + +For example, `stacks/catalog/spacelift/spaces.yaml` + +```yaml +components: + terraform: + spaces: + metadata: + component: spacelift/spaces + settings: + spacelift: + administrative: true + space_name: root + vars: + spaces: + root: + policies: +--- +# This policy will automatically assign itself to stacks and is used to trigger stacks directly from the `cloudposse/github-action-atmos-affected-trigger-spacelift` GitHub action +# This is only used if said GitHub action is set to trigger on "comments" +"GIT_PUSH Plan Affected": + type: GIT_PUSH + labels: + - autoattach:pr-comment-triggered + body: | + package spacelift + + # This policy runs whenever a comment is added to a pull request. It looks for the comment body to contain either: + # /spacelift preview input.stack.id + # /spacelift deploy input.stack.id + # + # If the comment matches those patterns it will queue a tracked run (deploy) or a proposed run (preview). In the case of + # a proposed run, it will also cancel all of the other pending runs for the same branch. + # + # This is being used on conjunction with the GitHub actions `atmos-trigger-spacelift-feature-branch.yaml` and + # `atmos-trigger-spacelift-main-branch.yaml` in .github/workflows to automatically trigger a preview or deploy run based + # on the `atmos describe affected` output. + + track { + commented + contains(input.pull_request.comment, concat(" ", ["/spacelift", "deploy", input.stack.id])) + } + + propose { + commented + contains(input.pull_request.comment, concat(" ", ["/spacelift", "preview", input.stack.id])) + } + + # Ignore if the event is not a comment + ignore { + not commented + } + + # Ignore if the PR has a `spacelift-no-trigger` label + ignore { + input.pull_request.labels[_] = "spacelift-no-trigger" + } + + # Ignore if the PR is a draft and deesnt have a `spacelift-trigger` label + ignore { + input.pull_request.draft + not has_spacelift_trigger_label + } + + has_spacelift_trigger_label { + input.pull_request.labels[_] == "spacelift-trigger" + } + + commented { + input.pull_request.action == "commented" + } + + cancel[run.id] { + run := input.in_progress[_] + run.type == "PROPOSED" + run.state == "QUEUED" + run.branch == input.pull_request.head.branch + } + + # This is a random sample of 10% of the runs + sample { + millis := round(input.request.timestamp_ns / 1e6) + millis % 100 <= 10 + } +``` + +This policy will automatically attach itself to _all_ components that have the `pr-comment-triggered` label, already +defined in `stacks/orgs/NAMESPACE/_defaults.yaml` under `settings.spacelift.labels`. + +Next, create two new GitHub Action workflows: + +```diff ++ .github/workflows/atmos-trigger-spacelift-feature-branch.yaml ++ .github/workflows/atmos-trigger-spacelift-main-branch.yaml +``` + +The feature branch workflow will create a comment event in Spacelift to run a Proposed run for a given stack. Whereas +the main branch workflow will create a comment event in Spacelift to run a Deploy run for those same stacks. + +#### Feature Branch + +```yaml +name: "Plan Affected Spacelift Stacks" + +on: + pull_request: + types: + - opened + - synchronize + - reopened + branches: + - main + +jobs: + context: + runs-on: ["self-hosted"] + steps: + - name: Atmos Affected Stacks Trigger Spacelift + uses: cloudposse/github-action-atmos-affected-trigger-spacelift@v1 + with: + atmos-config-path: ./rootfs/usr/local/etc/atmos + github-token: ${{ secrets.GITHUB_TOKEN }} +``` + +This will add a GitHub comment such as: + +``` +/spacelift preview plat-ue1-sandbox-foobar +``` + +#### Main Branch + +```yaml +name: "Deploy Affected Spacelift Stacks" + +on: + pull_request: + types: [closed] + branches: + - main + +jobs: + run: + if: github.event.pull_request.merged == true + runs-on: ["self-hosted"] + steps: + - name: Atmos Affected Stacks Trigger Spacelift + uses: cloudposse/github-action-atmos-affected-trigger-spacelift@v1 + with: + atmos-config-path: ./rootfs/usr/local/etc/atmos + deploy: true + github-token: ${{ secrets.GITHUB_TOKEN }} + head-ref: ${{ github.sha }}~1 +``` + +This will add a GitHub comment such as: + +``` +/spacelift deploy plat-ue1-sandbox-foobar +``` diff --git a/docs/components/library/aws/tgw/README.md b/docs/components/library/aws/tgw/README.md new file mode 100644 index 000000000..e2ea39b03 --- /dev/null +++ b/docs/components/library/aws/tgw/README.md @@ -0,0 +1,491 @@ +--- +tags: + - component/tgw + - layer/network + - provider/aws +--- + +# tgw + +AWS Transit Gateway connects your Amazon Virtual Private Clouds (VPCs) and on-premises networks through a central hub. +This connection simplifies your network and puts an end to complex peering relationships. Transit Gateway acts as a +highly scalable cloud router—each new connection is made only once. + +For more on Transit Gateway, see [the AWS documentation](https://aws.amazon.com/transit-gateway/). + +## Requirements + +In order to connect accounts with Transit Gateway, we deploy Transit Gateway to a central account, typically +`core-network`, and then deploy Transit Gateway attachments for each connected account. Each connected accounts needs a +Transit Gateway attachment for the given account's VPC, either by VPC attachment or by Peering Connection attachment. +Furthermore, each private subnet in each connected VPC needs to explicitly list the CIDRs for all allowed connections. + +## Solution + +First we deploy the Transit Gateway Hub, `tgw/hub`, to a central network account. The component prepares the Transit +Gateway network with the following steps: + +1. Provision Transit Gateway in the network account +2. Collect VPC and EKS component output from every account connected to Transit Gateway +3. Share the Transit Gateway with the Organization using Resource Access Manager (RAM) + +By using the `tgw/hub` component to collect Terraform output from connected accounts, only this single component +requires access to the Terraform state of all connected accounts. + +Next we deploy `tgw/spoke` to the network account and then to every connected account. This spoke component connects the +given account to the central hub and any listed connection with the following steps: + +1. Create a Transit Gateway VPC attachment in the spoke account. This connects the account's VPC to the shared Transit + Gateway from the hub account. +2. Define all allowed routes for private subnets. Each private subnet in an account's VPC has it's own route table. This + route table needs to explicitly list any allowed connection to another account's VPC CIDR. +3. (Optional) Create an EKS Cluster Security Group rule to allow traffic to the cluster in the given account. + +## Implementation + +1. Deploy `tgw/hub` to the network account. List every allowed connection: + +```yaml +# stacks/catalog/tgw/hub +components: + terraform: + tgw/hub/defaults: + metadata: + type: abstract + component: tgw/hub + vars: + enabled: true + name: tgw-hub + tags: + Team: sre + Service: tgw-hub + + tgw/hub: + metadata: + inherits: + - tgw/hub/defaults + component: tgw/hub + vars: + # These are all connections available for spokes in this region + # Defaults environment to this region + connections: + - account: + tenant: core + stage: network + - account: + tenant: core + stage: auto + eks_component_names: + - eks/cluster + - account: + tenant: plat + stage: sandbox + eks_component_names: [] # No clusters deployed for sandbox + - account: + tenant: plat + stage: dev + eks_component_names: + - eks/cluster + - account: + tenant: plat + stage: staging + eks_component_names: + - eks/cluster + - account: + tenant: plat + stage: prod + eks_component_names: + - eks/cluster +``` + +2. Deploy `tgw/spoke` to network. List every account connected to network (all accounts): + +```yaml +# stacks/catalog/tgw/spoke +components: + terraform: + tgw/spoke-defaults: + metadata: + type: abstract + component: tgw/spoke + vars: + enabled: true + name: tgw-spoke + tgw_hub_tenant_name: core + tgw_hub_stage_name: network # default, added for visibility + tags: + Team: sre + Service: tgw-spoke +``` + +```yaml +# stacks/orgs/acme/core/network/us-east-1/network.yaml +tgw/spoke: + metadata: + inherits: + - tgw/spoke-defaults + vars: + # This is what THIS spoke is allowed to connect to + connections: + - account: + tenant: core + stage: network + - account: + tenant: core + stage: auto + - account: + tenant: plat + stage: sandbox + - account: + tenant: plat + stage: dev + - account: + tenant: plat + stage: staging + - account: + tenant: plat + stage: prod +``` + +3. Finally, deploy `tgw/spoke` for each connected account and list the allowed connections: + +```yaml +# stacks/orgs/acme/plat/dev/us-east-1/network.yaml +tgw/spoke: + metadata: + inherits: + - tgw/spoke-defaults + vars: + connections: + # Always list self + - account: + tenant: plat + stage: dev + - account: + tenant: core + stage: network + - account: + tenant: core + stage: auto +``` + +### Alternate Regions + +In order to connect any account to the network, the given account needs: + +1. Access to the shared Transit Gateway hub +2. An attachment for the given Transit Gateway hub +3. Routes to and from each private subnet + +However, sharing the Transit Gateway hub via RAM is only supported in the same region as the primary hub. Therefore, we +must instead deploy a new hub in the alternate region and create a +[Transit Gateway Peering Connection](https://docs.aws.amazon.com/vpc/latest/tgw/tgw-peering.html) between the two +Transit Gateway hubs. + +Furthermore, since this Transit Gateway hub for the alternate region is now peered, we must create a Peering Transit +Gateway attachment, opposed to a VPC Transit Gateway Attachment. + +#### Cross Region Deployment + +1. Deploy `tgw/hub` and `tgw/spoke` into the primary region as described in [Implementation](#implementation) + +2. Deploy `tgw/hub` and `tgw/cross-region-hub` into the new region in the network account. See the following + configuration: + +```yaml +# stacks/catalog/tgw/cross-region-hub +import: + - catalog/tgw/hub + +components: + terraform: + # Cross region TGW requires additional hub in the alternate region + tgw/hub: + vars: + # These are all connections available for spokes in this region + # Defaults environment to this region + connections: + # Hub for this region is always required + - account: + tenant: core + stage: network + # VPN source + - account: + tenant: core + stage: network + environment: use1 + # Github Runners + - account: + tenant: core + stage: auto + environment: use1 + eks_component_names: + - eks/cluster + # All stacks where a spoke will be deployed + - account: + tenant: plat + stage: dev + - account: + tenant: plat + stage: staging + - account: + tenant: plat + stage: prod + + # This alternate hub needs to be connected to the primary region's hub + tgw/cross-region-hub-connector: + vars: + enabled: true + primary_tgw_hub_region: us-east-1 +``` + +3. Deploy a `tgw/spoke` for network in the new region. For example: + +```yaml +# stacks/orgs/acme/core/network/us-west-2/network.yaml +tgw/spoke: + metadata: + inherits: + - tgw/spoke-defaults + vars: + peered_region: true # Required for alternate region spokes + connections: + # This stack, always included + - account: + tenant: core + stage: network + # VPN + - account: + tenant: core + environment: use1 + stage: network + # Automation runners + - account: + tenant: core + environment: use1 + stage: auto + eks_component_names: + - eks/cluster + # All other connections + - account: + tenant: plat + stage: dev + - account: + tenant: plat + stage: staging + - account: + tenant: plat + stage: prod +``` + +4. Deploy the `tgw/spoke` components for all connected accounts. For example: + +```yaml +# stacks/orgs/acme/plat/dev/us-west-2/network.yaml +tgw/spoke: + metadata: + inherits: + - tgw/spoke-defaults + vars: + peered_region: true # Required for alternate region spokes + connections: + # This stack, always included + - account: + tenant: plat + stage: dev + # TGW Hub, always included + - account: + tenant: core + stage: network + # VPN + - account: + tenant: core + environment: use1 + stage: network + # Automation runners + - account: + tenant: core + environment: use1 + stage: auto + eks_component_names: + - eks/cluster +``` + +5. Update any existing `tgw/spoke` connections to allow the new account and region. For example: + +```yaml +# stacks/orgs/acme/core/auto/us-east-1/network.yaml +tgw/spoke: + metadata: + inherits: + - tgw/spoke-defaults + vars: + connections: + - account: + tenant: core + stage: network + - account: + tenant: core + stage: corp + - account: + tenant: core + stage: auto + - account: + tenant: plat + stage: sandbox + - account: + tenant: plat + stage: dev + - account: + tenant: plat + stage: staging + - account: + tenant: plat + stage: prod + + # Alternate regions <-------- These are added for alternate region + - account: + tenant: core + stage: network + environment: usw2 + - account: + tenant: plat + stage: dev + environment: usw2 + - account: + tenant: plat + stage: staging + environment: usw2 + - account: + tenant: plat + stage: prod + environment: usw2 +``` + +## Destruction + +When destroying Transit Gateway components, order of operations matters. Always destroy any removed `tgw/spoke` +components before removing a connection from the `tgw/hub` component. + +The `tgw/hub` component creates map of VPC resources that each `tgw/spoke` component references. If the required +reference is removed before the `tgw/spoke` is destroyed, Terraform will fail to destroy the given `tgw/spoke` +component. + +:::info Pro Tip! + +[Atmos Workflows](https://atmos.tools/core-concepts/workflows/) make applying and destroying Transit Gateway much +easier! For example, to destroy components in the correct order, use a workflow similar to the following: + +```yaml +# stacks/workflows/network.yaml +workflows: + destroy/tgw: + description: Destroy the Transit Gateway "hub" and "spokes" for connecting VPCs. + steps: + - command: echo 'Destroying platform spokes for Transit Gateway' + type: shell + name: plat-spokes + - command: terraform destroy tgw/spoke -s plat-use1-sandbox --auto-approve + - command: terraform destroy tgw/spoke -s plat-use1-dev --auto-approve + - command: terraform destroy tgw/spoke -s plat-use1-staging --auto-approve + - command: terraform destroy tgw/spoke -s plat-use1-prod --auto-approve + - command: echo 'Destroying core spokes for Transit Gateway' + type: shell + name: core-spokes + - command: terraform destroy tgw/spoke -s core-use1-auto --auto-approve + - command: terraform destroy tgw/spoke -s core-use1-network --auto-approve + - command: echo 'Destroying Transit Gateway Hub' + type: shell + name: hub + - command: terraform destroy tgw/hub -s core-use1-network --auto-approve +``` + +::: + +# FAQ + +## `tgw/spoke` Fails to Recreate VPC Attachment with `DuplicateTransitGatewayAttachment` Error + +```bash +╷ +│ Error: creating EC2 Transit Gateway VPC Attachment: DuplicateTransitGatewayAttachment: tgw-0xxxxxxxxxxxxxxxx has non-deleted Transit Gateway Attachments with same VPC ID. +│ status code: 400, request id: aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee +│ +│ with module.tgw_spoke_vpc_attachment.module.standard_vpc_attachment.aws_ec2_transit_gateway_vpc_attachment.default["core-use2-network"], +│ on .terraform/modules/tgw_spoke_vpc_attachment.standard_vpc_attachment/main.tf line 43, in resource "aws_ec2_transit_gateway_vpc_attachment" "default": +│ 43: resource "aws_ec2_transit_gateway_vpc_attachment" "default" { +│ +╵ +Releasing state lock. This may take a few moments... +exit status 1 +``` + +This is caused by Terraform attempting to create the replacement VPC attachment before the original is completely +destroyed. Retry the apply. Now you should see only "create" actions. + +# Changelog + +## Upgrading to `v1.276.0` + +Components PR [#804](https://github.com/cloudposse/terraform-aws-components/pull/804) + +### Affected Components + +- `tgw/hub` +- `tgw/spoke` +- `tgw/cross-region-hub-connector` + +### Summary + +This change to the Transit Gateway components, +[PR #804](https://github.com/cloudposse/terraform-aws-components/pull/804), added support for cross-region connections. + +As part of that change, we've added `environment` to the component identifier used in the Terraform Output created by +`tgw/hub`. Because of that map key change, all resources in Terraform now have a new resource identifier and therefore +must be recreated with Terraform or removed from state and imported into the new resource ID. + +Recreating the resources is the easiest solution but means that Transit Gateway connectivity will be lost while the +changes apply, which typically takes an hour. Alternatively, removing the resources from state and importing back into +the new resource ID is much more complex operationally but means no lost Transit Gateway connectivity. + +Since we use Transit Gateway for VPN and GitHub Automation runner access, a temporarily lost connection is not a +significant concern, so we choose to accept lost connectivity and recreate all `tgw/spoke` resources. + +### Steps + +1. Notify your team of a temporary VPN and Automation outage for accessing private networks +2. Deploy all `tgw/hub` components. There should be a hub component in each region of your network account connected to + Transit Gateway +3. Deploy all `tgw/spoke` components. There should be a spoke component in every account and every region connected to + Transit Gateway + +#### Tips + +Use workflows to deploy `tgw` across many accounts with a single command: + +```bash +atmos workflow deploy/tgw -f network +``` + +```yaml +# stacks/workflows/network.yaml +workflows: + deploy/tgw: + description: Provision the Transit Gateway "hub" and "spokes" for connecting VPCs. + steps: + - command: terraform deploy tgw/hub -s core-use1-network + name: hub + - command: terraform deploy tgw/spoke -s core-use1-network + - command: echo 'Creating core spokes for Transit Gateway' + type: shell + name: core-spokes + - command: terraform deploy tgw/spoke -s core-use1-corp + - command: terraform deploy tgw/spoke -s core-use1-auto + - command: terraform deploy tgw/spoke -s plat-use1-sandbox + - command: echo 'Creating platform spokes for Transit Gateway' + type: shell + name: plat-spokes + - command: terraform deploy tgw/spoke -s plat-use1-dev + - command: terraform deploy tgw/spoke -s plat-use1-staging + - command: terraform deploy tgw/spoke -s plat-use1-prod +``` diff --git a/docs/layers/accounts/tutorials/deprecated-cold-start-components.mdx b/docs/layers/accounts/tutorials/deprecated-cold-start-components.mdx index c7878c413..96f7a8338 100644 --- a/docs/layers/accounts/tutorials/deprecated-cold-start-components.mdx +++ b/docs/layers/accounts/tutorials/deprecated-cold-start-components.mdx @@ -52,7 +52,7 @@ The following guidance provides clarity on the security of the metadata files. ::: -Learn more about the [sso](/components/library/aws/aws-sso/) component. +Learn more about the [sso](/components/library/aws/identity-center/) component. #### Provision `iam-primary-roles` component diff --git a/docs/layers/eks/foundational-platform.mdx b/docs/layers/eks/foundational-platform.mdx index 566c1b8f2..e3b576736 100644 --- a/docs/layers/eks/foundational-platform.mdx +++ b/docs/layers/eks/foundational-platform.mdx @@ -47,7 +47,7 @@ those implementations in follow up topics. For details, see the - [`eks/cluster`](/components/library/aws/eks/cluster/): This component is responsible for provisioning an end-to-end EKS Cluster, including IAM role to Kubernetes Auth Config mapping. -- [`eks/karpenter`](/components/library/aws/eks/karpenter/): Installs the Karpenter chart on the EKS cluster and +- [`eks/karpenter`](/components/library/aws/eks/karpenter-controller/): Installs the Karpenter chart on the EKS cluster and prepares the environment for provisioners. - [`eks/karpenter-provisioner`](/components/library/aws/eks/karpenter-node-pool/): Deploys Karpenter Node Pools using CRDs made available by `eks/karpenter` diff --git a/docs/layers/identity/tutorials/leapp/leapp.mdx b/docs/layers/identity/tutorials/leapp/leapp.mdx index 1a797682a..25e9db369 100644 --- a/docs/layers/identity/tutorials/leapp/leapp.mdx +++ b/docs/layers/identity/tutorials/leapp/leapp.mdx @@ -37,7 +37,7 @@ easier. We expect you’ve gone through our guide on [Geodesic](/resources/legacy/fundamentals/geodesic) prior to using Leapp since that contains some important understanding of what Geodesic is, how it works, and what it’s doing. -Your organization is already set up with either the [sso](/components/library/aws/aws-sso/) or the +Your organization is already set up with either the [sso](/components/library/aws/identity-center/) or the [iam-primary-roles](https://github.com/cloudposse/terraform-aws-components/tree/main/deprecated/iam-primary-roles) and [iam-delegated-roles](https://github.com/cloudposse/terraform-aws-components/tree/main/deprecated/iam-delegated-roles) components. diff --git a/docs/layers/monitoring/datadog/setup.mdx b/docs/layers/monitoring/datadog/setup.mdx index 426415a5c..b6d4b322c 100644 --- a/docs/layers/monitoring/datadog/setup.mdx +++ b/docs/layers/monitoring/datadog/setup.mdx @@ -63,7 +63,7 @@ plat is in place, usually EKS or ECS. There are two core components to the Datadog implementation -1. [**datadog-configuration**](https://docs.cloudposse.com/components/library/aws/datadog-configuration) +1. [**datadog-configuration**](https://docs.cloudposse.com/components/library/aws/datadog-credentials) 2. [**datadog-integration**](https://docs.cloudposse.com/components/library/aws/datadog-integration) @@ -107,7 +107,7 @@ This component also handles default configurations such as Datadog URL and provi components to utilize via its submodule `datadog_keys`. Use a configuration similar to the following but check the -[`README.md`](https://docs.cloudposse.com/components/library/aws/datadog-configuration/) for exact input references. +[`README.md`](https://docs.cloudposse.com/components/library/aws/datadog-credentials/) for exact input references. ```yaml components: diff --git a/docs/layers/monitoring/monitoring.mdx b/docs/layers/monitoring/monitoring.mdx index 0a4bc7fa8..55a31b268 100644 --- a/docs/layers/monitoring/monitoring.mdx +++ b/docs/layers/monitoring/monitoring.mdx @@ -31,7 +31,7 @@ We have broken down the monitoring solution into several components to make it e #### Foundation -- [`datadog-configuration`](/components/library/aws/datadog-configuration/): This is a **utility** component. This component expects Datadog API and APP keys to be stored in SSM or ASM, it then copies the keys to SSM/ASM of each account this component is deployed to. This is for several reasons: +- [`datadog-configuration`](/components/library/aws/datadog-credentials/): This is a **utility** component. This component expects Datadog API and APP keys to be stored in SSM or ASM, it then copies the keys to SSM/ASM of each account this component is deployed to. This is for several reasons: 1. Keys can be easily rotated from one place 2. Keys can be set for a group and then copied to all accounts in that group, meaning you could have a pair of api keys and app keys for production accounts and another set for non-production accounts. This component is **required** for all other components to work. As it also stores information about your Datadog account, which other components will use, such as your Datadog site url, along with providing an easy interface for other components to configure the Datadog provider. diff --git a/docs/layers/software-delivery/eks-argocd/eks-argocd.mdx b/docs/layers/software-delivery/eks-argocd/eks-argocd.mdx index 73ee7a393..b10b2e6fa 100644 --- a/docs/layers/software-delivery/eks-argocd/eks-argocd.mdx +++ b/docs/layers/software-delivery/eks-argocd/eks-argocd.mdx @@ -337,7 +337,7 @@ runs: ### Implementation - [`eks/argocd`](/components/library/aws/eks/argocd/): This component is responsible for provisioning [ArgoCD](https://argoproj.github.io/cd/). -- [`argocd-repo`](/components/library/aws/argocd-repo/): This component is responsible for creating and managing an ArgoCD desired state repository. +- [`argocd-repo`](/components/library/aws/argocd-github-repo/): This component is responsible for creating and managing an ArgoCD desired state repository. - [`sso-saml-provider`](/components/library/aws/sso-saml-provider/): This component reads sso credentials from SSM Parameter store and provides them as outputs ## References diff --git a/docs/resources/adrs/adopted/use-basic-provider-block-for-root-level-components.mdx b/docs/resources/adrs/adopted/use-basic-provider-block-for-root-level-components.mdx index 41780d7f1..e054ebd31 100644 --- a/docs/resources/adrs/adopted/use-basic-provider-block-for-root-level-components.mdx +++ b/docs/resources/adrs/adopted/use-basic-provider-block-for-root-level-components.mdx @@ -19,7 +19,7 @@ The content in this ADR may be out-of-date and needing an update. For questions, We [Use Terraform Provider Block with compatibility for Role ARNs and Profiles](/resources/adrs/adopted/use-terraform-provider-block-with-compatibility-for-role-arns-an) in all components other than the root-level components. By _root-level_ we are referring to components that are provisioned in the top-level AWS account that we generally refer to as the `root` account. -The problem arises when working with the `root` account during a cold-start when there’s no SSO, Federated IAM or IAM roles provisioned, so if we used the `role_arn` or `profile` it would not work. That’s why we assume the administrator will use their current AWS session to provision these components, which is why we do not define the `role_arn` or `profile` in `provider { ... }` block for the components like [sso](/components/library/aws/aws-sso/) or [account](/components/library/aws/account/) . +The problem arises when working with the `root` account during a cold-start when there’s no SSO, Federated IAM or IAM roles provisioned, so if we used the `role_arn` or `profile` it would not work. That’s why we assume the administrator will use their current AWS session to provision these components, which is why we do not define the `role_arn` or `profile` in `provider { ... }` block for the components like [sso](/components/library/aws/identity-center/) or [account](/components/library/aws/account/) . ## Decision diff --git a/docs/resources/adrs/proposed/proposed-use-aws-federated-iam-over-aws-sso.mdx b/docs/resources/adrs/proposed/proposed-use-aws-federated-iam-over-aws-sso.mdx index e44ae4e5a..49edb9fbd 100644 --- a/docs/resources/adrs/proposed/proposed-use-aws-federated-iam-over-aws-sso.mdx +++ b/docs/resources/adrs/proposed/proposed-use-aws-federated-iam-over-aws-sso.mdx @@ -94,11 +94,10 @@ AWS Federated IAM and AWS SSO can coexist and are not mutually exclusive. - GSuite does not support mapping group attributes to SAML attributes (But they don't really solve it for AWS SSO either, and if you can script the GSuite API you can achieve the same effect.) -- Our current implementation with [iam-primary-roles](https://github.com/cloudposse/terraform-aws-components/tree/main/deprecated/iam-primary-roles) and [iam-delegated-roles](https://github.com/cloudposse/terraform-aws-components/tree/main/deprecated/iam-delegated-roles) is outdated and should be updated to use the interface we developed for AWS [sso](/components/library/aws/aws-sso/). +- Our current implementation with [iam-primary-roles](https://github.com/cloudposse/terraform-aws-components/tree/main/deprecated/iam-primary-roles) and [iam-delegated-roles](https://github.com/cloudposse/terraform-aws-components/tree/main/deprecated/iam-delegated-roles) is outdated and should be updated to use the interface we developed for AWS [sso](/components/library/aws/identity-center/). ### When to use AWS SSO? - -AWS SSO is ideally suited for business users of AWS that interact with the AWS Web Console. It does work well with the `aws` CLI, but not together with EKS. +lnmgafj/....[;..................................................................................................................................................................................................../PLU5D5YT6FTYUNFVHGT6FHGNU VDYBHBGYBDFR G YGDAWS SSO is ideally suited for business users of AWS that interact with the AWS Web Console. It does work well with the `aws` CLI, but not together with EKS. ### When to use AWS Federated IAM with SAML? diff --git a/docs/tags.yml b/docs/tags.yml index dbb13a0e8..ea1cd459f 100644 --- a/docs/tags.yml +++ b/docs/tags.yml @@ -813,9 +813,9 @@ component/eks/idp-roles: description: The eks/idp-roles Component. label: component/eks/idp-roles -component/eks/karpenter: +component/eks/karpenter-controller: description: The eks/karpenter Component. - label: component/eks/karpenter + label: component/eks/karpenter-controller component/eks/keda: description: The eks/keda Component. diff --git a/package-lock.json b/package-lock.json index 8d20ec124..3f666001f 100644 --- a/package-lock.json +++ b/package-lock.json @@ -6428,8 +6428,8 @@ "license": "MIT" }, "node_modules/custom-loaders": { - "version": "0.0.0", - "resolved": "file:plugins/custom-loaders" + "resolved": "plugins/custom-loaders", + "link": true }, "node_modules/cytoscape": { "version": "3.30.1", @@ -18860,6 +18860,9 @@ "type": "github", "url": "https://github.com/sponsors/wooorm" } + }, + "plugins/custom-loaders": { + "version": "0.0.0" } } } diff --git a/plugins/staticRedirects/redirects/components-migration.json b/plugins/staticRedirects/redirects/components-migration.json new file mode 100644 index 000000000..ac755c01d --- /dev/null +++ b/plugins/staticRedirects/redirects/components-migration.json @@ -0,0 +1,42 @@ +[ + { + "from": "/components/library/aws/argocd-repo", + "to": "/components/library/aws/argocd-github-repo" + }, + { + "from": "/components/library/aws/argocd-repo/templates", + "to": "/components/library/aws/argocd-github-repo" + }, + { + "from": "/components/library/aws/aws-sso", + "to": "/components/library/aws/identity-center" + }, + { + "from": "/components/library/aws/datadog-configuration", + "to": "/components/library/aws/datadog-credentials" + }, + { + "from": "/components/library/aws/datadog-configuration/modules/datadog_keys", + "to": "/components/library/aws/datadog-credentials/modules/datadog_keys" + }, + { + "from": "/components/library/aws/eks/aws-node-termination-handler", + "to": "/components/library/aws/eks/node-termination-handler" + }, + { + "from": "/components/library/aws/eks/karpenter", + "to": "/components/library/aws/eks/karpenter-controller" + }, + { + "from": "/components/library/aws/eks/karpenter/docs", + "to": "/components/library/aws/eks/karpenter-controller/docs" + }, + { + "from": "/components/library/aws/spacelift/worker-pool", + "to": "/components/library/aws/spacelift/worker-pool-asg" + }, + { + "from": "/components/library/aws/tgw/cross-region-hub-connector", + "to": "/components/library/aws/tgw/hub-connector" + } +] diff --git a/scripts/Dockerfile b/scripts/Dockerfile index 6ff0626b2..5d01599a5 100644 --- a/scripts/Dockerfile +++ b/scripts/Dockerfile @@ -4,6 +4,10 @@ FROM python:3.9-slim # Set the working directory WORKDIR / +RUN apt-get update && apt-get install -y apt-utils curl +RUN curl -1sLf 'https://dl.cloudsmith.io/public/cloudposse/packages/cfg/setup/bash.deb.sh' | bash +RUN apt-get install -y terraform-docs + # Copy the Python requirements file into the container COPY ./docs-collator/requirements.txt scripts/docs-collator/requirements.txt diff --git a/scripts/Makefile b/scripts/Makefile index 12f6d4eb4..df54a46ec 100644 --- a/scripts/Makefile +++ b/scripts/Makefile @@ -3,4 +3,4 @@ build: docker build -t docs-collator --progress=plain . run: build - docker run -it --rm -e PUBLIC_REPO_ACCESS_TOKEN=$(PUBLIC_REPO_ACCESS_TOKEN). -v $(PWD)/docs:/docs docs-collator /scripts/render-docs-for-components.sh + docker run -it --rm -e PUBLIC_REPO_ACCESS_TOKEN=$(PUBLIC_REPO_ACCESS_TOKEN) -v $(PWD)/docs:/docs docs-collator /scripts/render-docs-for-modules.sh diff --git a/scripts/docs-collator/AbstractRenderer.py b/scripts/docs-collator/AbstractRenderer.py index 0a19b388f..d8d235dfb 100644 --- a/scripts/docs-collator/AbstractRenderer.py +++ b/scripts/docs-collator/AbstractRenderer.py @@ -6,7 +6,7 @@ README_YAML = "README.yaml" DOCS_DIR = "docs" - +CHANGELOG_MD = "CHANGELOG.md" class TerraformDocsRenderingError(Exception): def __init__(self, message): @@ -15,6 +15,10 @@ def __init__(self, message): class AbstractRenderer: + + def render(self): + raise NotImplementedError + def _pre_rendering_fixes(self, repo, module_download_dir, submodule_dir=""): readme_yaml_file = os.path.join(module_download_dir, submodule_dir, README_YAML) content = io.read_file_to_string(readme_yaml_file) @@ -27,6 +31,25 @@ def _pre_rendering_fixes(self, repo, module_download_dir, submodule_dir=""): ) io.save_string_to_file(readme_yaml_file, content) + if os.path.exists(os.path.join(module_download_dir, submodule_dir, CHANGELOG_MD)): + import yaml + + with open(readme_yaml_file, 'r') as file: + readme_yaml = yaml.safe_load(file) + if "include" not in readme_yaml: + readme_yaml["include"] = [] + readme_yaml["include"].append(CHANGELOG_MD) + with open(readme_yaml_file, 'w') as file: + yaml.dump(readme_yaml, file) + + change_md_file = os.path.join(module_download_dir, submodule_dir, CHANGELOG_MD) + content = io.read_file_to_string(change_md_file) + content = rendering.shift_headings(content) + lines = content.splitlines() + lines.insert(0, "## Changelog") + content = "\n".join(lines) + io.save_string_to_file(change_md_file, content) + def _post_rendering_fixes(self, repo, readme_md_file, submodule_dir=""): content = io.read_file_to_string(readme_md_file) content = rendering.fix_self_non_closing_br_tags(content) diff --git a/scripts/docs-collator/ComponentRenderer.py b/scripts/docs-collator/ComponentRenderer.py deleted file mode 100644 index 802b96831..000000000 --- a/scripts/docs-collator/ComponentRenderer.py +++ /dev/null @@ -1,142 +0,0 @@ -import os - -from utils import io, rendering, templating - -README_MD = "README.md" -CHANGELOG_MD = "CHANGELOG.md" -GITHUB_REPO = "cloudposse/terraform-aws-components" -INDEX_CATEGORY_JSON = "_category_.json" - -SCRIPT_DIR = os.path.dirname(os.path.realpath(__file__)) -TEMPLATES_DIR = os.path.join(SCRIPT_DIR, "templates/components") - -jenv = templating.init_templating(TEMPLATES_DIR) -DOC_TEMPLATE = jenv.get_template("readme.md") -INDEX_CATEGORY_TEMPLATE = jenv.get_template("index_category.json") - - -class ComponentRenderer: - def __init__(self, download_dir, docs_dir): - self.download_dir = download_dir - self.docs_dir = docs_dir - - def render(self, component): - module_download_dir = os.path.join(self.download_dir, "modules", component) - - files = io.get_filenames_in_dir(module_download_dir, README_MD, True) - files += io.get_filenames_in_dir(module_download_dir, "*.md", True) - - images = io.get_filenames_in_dir(module_download_dir, "*.png", True) - - for file in files: - if file.endswith(CHANGELOG_MD): - continue - self.__render_doc(component, file) - - for image in images: - io.copy_file( - image, - os.path.join( - self.docs_dir, - component, - os.path.relpath(image, module_download_dir), - ), - ) - - def __render_doc(self, component, file): - module_download_dir = os.path.join(self.download_dir, "modules") - - # Render Terraform docs using template for website doc format - # This will update the given README in place - module_path = os.path.join(module_download_dir, component) - rendering.render_terraform_docs( - module_path, os.path.join(TEMPLATES_DIR, "terraform-docs.yml") - ) - - content = io.read_file_to_string(file) - - # Previously we set tags to this short list. - # However now we instead use the tags from the given component's frontmatter - # tags = ["terraform", "aws", component] - content, frontmatter = rendering.strip_frontmatter(content) - tags = rendering.get_tags_from_frontmatter(frontmatter) - - # Static replacement and corrections for docusaurus - content = rendering.strip_title(content) - content = rendering.fix_self_non_closing_br_tags(content) - content = rendering.fix_custom_non_self_closing_tags_in_pre(content) - content = rendering.remove_logo_from_the_bottom(content) - content = rendering.reformat_admonitions(content) - content = rendering.remove_https_cloudposse_docs(content) - content = rendering.replace_broken_links(content) - content = rendering.fix_mdx_format(content) - - change_log_file = os.path.join(os.path.dirname(file), CHANGELOG_MD) - change_log_content = ( - io.read_file_to_string(change_log_file) - if os.path.exists(change_log_file) - else "" - ) - change_log_content = rendering.reformat_admonitions(change_log_content) - change_log_content = rendering.shift_headings(change_log_content) - - relative_path = os.path.relpath(file, module_download_dir) - result_file = os.path.join( - self.docs_dir, os.path.relpath(file, module_download_dir) - ) # /README.md - - name = ( - component - if os.path.basename(file) == "README.md" - else os.path.basename(file).replace(".md", "") - ) - label = name - title = name - github_edit_url = ( - f"https://github.com/{GITHUB_REPO}/blob/main/modules/{relative_path}" - ) - - if ( - len(relative_path.split("/")) > 2 and relative_path.split("/")[1] != "docs" - ): # this is submodule - submodule_name = os.path.basename(os.path.dirname(result_file)) - - label = submodule_name - title = submodule_name - - # renaming final file /.mdx - result_file = os.path.join( - os.path.dirname(result_file), f"{submodule_name}.mdx" - ) - else: - # renaming final file /README.mdx - result_file = os.path.join(os.path.dirname(result_file), f"{name}.mdx") - - io.create_dirs(os.path.dirname(result_file)) - - doc_content = DOC_TEMPLATE.render( - label=label, - title=title, - content=content, - change_log_content=change_log_content, - github_repository=GITHUB_REPO, - github_edit_url=github_edit_url, - tags=tags, - ) - - io.save_string_to_file(result_file, doc_content) - - # Breaking builds, not sure why and not adding value. Disabling. - # self.__create_indexes_for_subfolder(component) - - def __create_indexes_for_subfolder(self, component): - for root, dirs, files in os.walk(os.path.join(self.docs_dir, component)): - if not files and all(os.path.isdir(os.path.join(root, d)) for d in dirs): - self.__render_category_index(root) - - def __render_category_index(self, dir): - name = os.path.basename(dir) - - content = INDEX_CATEGORY_TEMPLATE.render(label=name, title=name) - - io.save_string_to_file(os.path.join(dir, INDEX_CATEGORY_JSON), content) diff --git a/scripts/docs-collator/GitHubProvider.py b/scripts/docs-collator/GitHubProvider.py index 9519c6122..0179ac470 100644 --- a/scripts/docs-collator/GitHubProvider.py +++ b/scripts/docs-collator/GitHubProvider.py @@ -5,12 +5,17 @@ import pickle from github import Github, GithubException from functools import lru_cache + +from github import Auth + from utils import io -GITHUB_ORG = "cloudposse" TERRAFORM_MODULE_NAME_PATTERN = re.compile( "^terraform-[a-zA-Z0-9]+-.*" ) # convention is terraform-- +COMPONENTS_NAME_PATTERN = re.compile( + "^[a-zA-Z0-9]+.*" +) # convention is terraform-- GITHUB_ACTION_NAME_PATTERN = re.compile( "^github-action-.*" ) # convention is github-action- @@ -21,41 +26,59 @@ class GitHubProvider: - def __init__(self, github_api_token): - self.github = Github(github_api_token) + def __init__(self, github_api_token = None): + if github_api_token: + auth = Auth.Token(github_api_token) + else: + logging.info("No GitHub API token provided, using unauthenticated access") + auth = None + self.github = Github(auth=auth) self.cache = self.load_cache(CACHE_FILE) self.repos_cache = self.load_cache(REPOS_CACHE_FILE) - def get_terraform_repos(self, includes_csv, excludes_csv): + def __del__(self): + self.github.close() + + def get_terraform_repos(self, github_org, includes_csv, excludes_csv): key = (includes_csv, excludes_csv) if key in self.repos_cache: return self.repos_cache[key] repos = self.__get_repos( - includes_csv, excludes_csv, TERRAFORM_MODULE_NAME_PATTERN + github_org, includes_csv, excludes_csv, TERRAFORM_MODULE_NAME_PATTERN ) self.repos_cache[key] = repos self.save_cache(REPOS_CACHE_FILE, self.repos_cache) return repos - def get_github_actions_repos(self, includes_csv, excludes_csv): - return self.__get_repos(includes_csv, excludes_csv, GITHUB_ACTION_NAME_PATTERN) + def get_components_repos(self, github_org, includes_csv, excludes_csv): + key = (includes_csv, excludes_csv) + if key in self.repos_cache: + return self.repos_cache[key] + repos = self.__get_repos( + github_org, includes_csv, excludes_csv, COMPONENTS_NAME_PATTERN + ) + self.repos_cache[key] = repos + self.save_cache(REPOS_CACHE_FILE, self.repos_cache) + return repos + + def get_github_actions_repos(self, github_org, includes_csv, excludes_csv): + return self.__get_repos(github_org, includes_csv, excludes_csv, GITHUB_ACTION_NAME_PATTERN) - def __get_repos(self, includes_csv, excludes_csv, pattern): + def __get_repos(self, github_org, includes_csv, excludes_csv, pattern): repos = [] excludes = self.__csv_to_set(excludes_csv) includes = self.__csv_to_set(includes_csv) - if len(includes) > 0: for include in includes: - repo = self.github.get_organization(GITHUB_ORG).get_repo(include) + repo = self.github.get_organization(github_org).get_repo(include) if not self.__is_valid(repo, pattern): continue repos.append(repo) else: - for repo in self.github.get_organization(GITHUB_ORG).get_repos(): + for repo in self.github.get_organization(github_org).get_repos(): if not self.__is_valid(repo, pattern): continue diff --git a/scripts/docs-collator/__init__.py b/scripts/docs-collator/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/scripts/docs-collator/component/__init__.py b/scripts/docs-collator/component/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/scripts/docs-collator/component/component.py b/scripts/docs-collator/component/component.py new file mode 100644 index 000000000..648d77940 --- /dev/null +++ b/scripts/docs-collator/component/component.py @@ -0,0 +1,85 @@ +import os + +from utils import io + +README_YAML = "README.yaml" +README_MD = "README.md" +SUBMODULES_DIR = "modules" + + +class Component: + _terraform_dir = None + _name = None + _provider = None + _module_download_dir = None + _repo = None + _subdirs = [] + + def __init__(self, repo, module_download_dir, terraform_dir, name, subdirs=[]): + self._repo = repo + self._module_download_dir = module_download_dir + self._terraform_dir = terraform_dir + self._name = name + self._subdirs = subdirs + + @property + def name(self): + return self._name + + @property + def dir(self): + return self._module_download_dir + + @property + def terraform_dir(self): + return self._terraform_dir + + def has_readme_yaml(self): + return os.path.exists(os.path.join(self._module_download_dir, README_YAML)) + + @property + def provider(self): + return self._repo.provider + + @property + def subdirs(self): + return self._subdirs + + @property + def repo(self): + return self._repo + + def modules(self): + extra_resources_dir = os.path.join(self._terraform_dir, SUBMODULES_DIR) + files = io.get_filenames_in_dir(extra_resources_dir, "*", True) + readme_files = {} + + for file in files: + base_name = os.path.basename(file) + dir_name = os.path.dirname(file) + name = os.path.basename(os.path.dirname(file)) + + rel_dir = os.path.relpath(dir_name, self._terraform_dir).split("/")[:-1] + + subdirs = self._subdirs.copy() + subdirs.append(self.name) + subdirs.extend(rel_dir) + + if base_name == README_YAML: + readme_files[dir_name] = Component( + repo=self._repo, + module_download_dir=dir_name, + terraform_dir=dir_name, + name=name, + subdirs=subdirs + ) + elif base_name == README_MD and dir_name not in readme_files: + readme_files[dir_name] = Component( + repo=self._repo, + module_download_dir=dir_name, + terraform_dir=dir_name, + name=name, + subdirs=subdirs + ) + + return readme_files.values() diff --git a/scripts/docs-collator/component/fetcher.py b/scripts/docs-collator/component/fetcher.py new file mode 100644 index 000000000..49c28d5d0 --- /dev/null +++ b/scripts/docs-collator/component/fetcher.py @@ -0,0 +1,84 @@ +import os + +from AbstractFetcher import AbstractFetcher, MissingReadmeYamlException +from .repository.single import ComponentRepositorySingle + +DOCS_DIR = "docs" +IMAGES_DIR = "images" +TERRAFORM_DIR = "src" +README_YAML = "README.yaml" +README_MD = "README.md" +SUBMODULES_DIR = "modules" +CHANGELOG_MD = "CHANGELOG.md" + + +class ComponentFetcher(AbstractFetcher): + def __init__(self, github_provider, download_dir): + super().__init__(github_provider, download_dir) + + def fetch(self, repo): + module_download_dir = os.path.join(self.download_dir, repo.name) + + remote_files = set(self.github_provider.list_repo_dir(repo, "", False)) + + if README_YAML not in remote_files: + raise MissingReadmeYamlException() + + self._fetch_readme_yaml(repo, module_download_dir) + self._fetch_readme_md(repo, module_download_dir) + + if CHANGELOG_MD in remote_files: + self._fetch_changelog_md(repo, module_download_dir) + + if DOCS_DIR in remote_files: + self._fetch_docs(repo, module_download_dir) + + if IMAGES_DIR in remote_files: + self.__fetch_images(repo, module_download_dir) + + if TERRAFORM_DIR in remote_files: + self.__fetch_terraform_files(repo, module_download_dir) + + if SUBMODULES_DIR in remote_files: + self.__fetch_submodules(repo, module_download_dir) + + return ComponentRepositorySingle(repo, module_download_dir) + + def __fetch_images(self, repo, module_download_dir): + remote_files = self.github_provider.list_repo_dir(repo, IMAGES_DIR) + + for remote_file in remote_files: + self.github_provider.fetch_file(repo, remote_file, module_download_dir) + + def __fetch_terraform_files(self, repo, module_download_dir): + remote_files = self.github_provider.list_repo_dir(repo, TERRAFORM_DIR) + for remote_file in remote_files: + self.github_provider.fetch_file(repo, remote_file, module_download_dir) + + def _fetch_readme_md(self, repo, module_download_dir): + self.github_provider.fetch_file(repo, README_MD, module_download_dir) + + def _fetch_changelog_md(self, repo, module_download_dir): + self.github_provider.fetch_file(repo, CHANGELOG_MD, module_download_dir) + + def __fetch_submodules(self, repo, module_download_dir): + remote_files = self.github_provider.list_repo_dir(repo, SUBMODULES_DIR) + readme_files = {} + + for remote_file in remote_files: + base_name = os.path.basename(remote_file) + dir_name = os.path.dirname(remote_file) + + if base_name == README_YAML: + readme_files[dir_name] = remote_file + elif base_name == README_MD and dir_name not in readme_files: + readme_files[dir_name] = remote_file + + for readme_file in readme_files.values(): + self.github_provider.fetch_file(repo, readme_file, module_download_dir) + if os.path.basename(readme_file) == README_YAML: + self._fetch_docs( + repo, + module_download_dir, + submodule_dir=os.path.dirname(readme_file), + ) diff --git a/scripts/docs-collator/component/renderer/__init__.py b/scripts/docs-collator/component/renderer/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/scripts/docs-collator/component/renderer/factory.py b/scripts/docs-collator/component/renderer/factory.py new file mode 100644 index 000000000..9fc29e66d --- /dev/null +++ b/scripts/docs-collator/component/renderer/factory.py @@ -0,0 +1,21 @@ +import os + +from AbstractRenderer import AbstractRenderer +from .yml import ComponentRendererYaml +from .md import ComponentRendererMD +from component.repository.abstract import ComponentRepositoryAbstract + + +class ComponentRendererFactory: + + @staticmethod + def produce(repo: ComponentRepositoryAbstract) -> list[AbstractRenderer]: + result = [] + for component in repo.components: + if component.has_readme_yaml(): + result.append(ComponentRendererYaml(component)) + else: + result.append(ComponentRendererMD(component)) + for module in component.modules(): + result.append(ComponentRendererMD(module)) + return result diff --git a/scripts/docs-collator/component/renderer/md.py b/scripts/docs-collator/component/renderer/md.py new file mode 100644 index 000000000..100a37f5e --- /dev/null +++ b/scripts/docs-collator/component/renderer/md.py @@ -0,0 +1,111 @@ +import os +import logging + +from utils import io, rendering, templating +from AbstractRenderer import AbstractRenderer +from component.component import Component + +SCRIPT_DIR = os.path.dirname(os.path.realpath(__file__)) +README_YAML = "README.yaml" +README_MD = "README.md" +IMAGES_DIR = "images" +MODULES_README_TEMPLATE = "readme.md" +CHANGELOG_MD = "CHANGELOG.md" + + +class ComponentRendererMD(AbstractRenderer): + def __init__(self, component: Component): + self.templates_dir = os.path.join(SCRIPT_DIR, "templates/components/md") + self.component = component + + def render(self, docs_dir): + logging.info(f"Rendering doc for: {self.component.repo.full_name}") + logging.info(f"Provider: {self.component.provider}, Module: {self.component.name}") + + path_components = [docs_dir, self.component.provider] + path_components.extend(self.component.subdirs) + path_components.append(self.component.name) + module_docs_dir = os.path.join(*path_components) + logging.info(f"Module docs dir: {module_docs_dir}") + + self.__render_doc() + content = io.read_file_to_string(os.path.join(self.component.dir, README_MD)) + + change_log_content = None + if os.path.exists(os.path.join(self.component.dir, CHANGELOG_MD)): + change_md_file = os.path.join(self.component.dir, CHANGELOG_MD) + content = io.read_file_to_string(change_md_file) + content = rendering.shift_headings(content) + lines = content.splitlines() + lines.insert(0, "## Changelog") + content = "\n".join(lines) + io.save_string_to_file(change_md_file, content) + + change_log_content = io.read_file_to_string(change_md_file) + change_log_content = rendering.reformat_admonitions(change_log_content) + change_log_content = rendering.shift_headings(change_log_content) + + rel_dir = os.path.relpath(self.component.terraform_dir, self.component.repo.dir) + + jenv = templating.init_templating(self.templates_dir) + self.doc_template = jenv.get_template("readme.md") + + content = self.doc_template.render( + label=self.component.name, + title=self.component.name, + description=self.component.name, + github_edit_url=f"https://github.com/{self.component.repo.full_name}/blob/{self.component.repo.default_branch}/{rel_dir}/README.md", + content=content, + change_log_content=change_log_content + ) + + dest_file = os.path.join(module_docs_dir, README_MD) + io.create_dirs(os.path.dirname(dest_file)) + io.save_string_to_file(dest_file, content) + + self._copy_extra_resources_for_docs(self.component.dir, module_docs_dir) + self.__copy_extra_resources_for_images(module_docs_dir) + + # self.__post_rendering_fixes_for_submodule(dest_file) + + def __render_doc(self): + file = os.path.join(self.component.dir, README_MD) + # Render Terraform docs using template for website doc format + # This will update the given README in place + rendering.render_terraform_docs( + self.component.terraform_dir, os.path.join(self.templates_dir, "terraform-docs.yml") + ) + + content = io.read_file_to_string(file) + + # Previously we set tags to this short list. + # However now we instead use the tags from the given component's frontmatter + # tags = ["terraform", "aws", component] + content, frontmatter = rendering.strip_frontmatter(content) + tags = rendering.get_tags_from_frontmatter(frontmatter) + + # Static replacement and corrections for docusaurus + content = rendering.strip_title(content) + content = rendering.fix_self_non_closing_br_tags(content) + content = rendering.fix_custom_non_self_closing_tags_in_pre(content) + content = rendering.remove_logo_from_the_bottom(content) + content = rendering.reformat_admonitions(content) + content = rendering.remove_https_cloudposse_docs(content) + content = rendering.replace_broken_links(content) + content = rendering.fix_mdx_format(content) + + io.save_string_to_file(file, content) + + def __copy_extra_resources_for_images(self, module_docs_dir): + extra_resources_dir = os.path.join(self.component.dir, IMAGES_DIR) + files = io.get_filenames_in_dir(extra_resources_dir, "*", True) + + for file in files: + if os.path.isdir(file): + continue + + dest_file = os.path.join( + module_docs_dir, IMAGES_DIR, os.path.relpath(file, extra_resources_dir) + ) + io.copy_file(file, dest_file) + logging.info(f"Copied extra file: {dest_file}") diff --git a/scripts/docs-collator/templates/components/index_category.json b/scripts/docs-collator/component/renderer/templates/components/md/index_category.json similarity index 100% rename from scripts/docs-collator/templates/components/index_category.json rename to scripts/docs-collator/component/renderer/templates/components/md/index_category.json diff --git a/scripts/docs-collator/templates/components/readme.md b/scripts/docs-collator/component/renderer/templates/components/md/readme.md similarity index 95% rename from scripts/docs-collator/templates/components/readme.md rename to scripts/docs-collator/component/renderer/templates/components/md/readme.md index 32c62d415..82e6b657b 100644 --- a/scripts/docs-collator/templates/components/readme.md +++ b/scripts/docs-collator/component/renderer/templates/components/md/readme.md @@ -16,7 +16,5 @@ tags: [] {{ content }} {%- if change_log_content|trim != "" %} -## Changelog - {{ change_log_content }} {%- endif %} diff --git a/scripts/docs-collator/templates/components/terraform-docs.yml b/scripts/docs-collator/component/renderer/templates/components/md/terraform-docs.yml similarity index 100% rename from scripts/docs-collator/templates/components/terraform-docs.yml rename to scripts/docs-collator/component/renderer/templates/components/md/terraform-docs.yml diff --git a/scripts/docs-collator/component/renderer/templates/components/yml/index_category.json b/scripts/docs-collator/component/renderer/templates/components/yml/index_category.json new file mode 100644 index 000000000..e673d5349 --- /dev/null +++ b/scripts/docs-collator/component/renderer/templates/components/yml/index_category.json @@ -0,0 +1,9 @@ +{ + "label": "{{ label }}", + "collapsible": true, + "collapsed": true, + "link": { + "type": "generated-index", + "title": "{{ title }}" + } +} diff --git a/scripts/docs-collator/component/renderer/templates/components/yml/readme.md b/scripts/docs-collator/component/renderer/templates/components/yml/readme.md new file mode 100644 index 000000000..fe08e30f1 --- /dev/null +++ b/scripts/docs-collator/component/renderer/templates/components/yml/readme.md @@ -0,0 +1,79 @@ +{{- defineDatasource "config" .Env.README_YAML | regexp.Replace ".*" "" -}} +{{- defineDatasource "includes" .Env.README_INCLUDES -}} +{{- $deprecated := has (ds "config") "deprecated" -}} +{{- $fullModuleName := (ds "config").name -}} +{{- $shortModuleName := ( conv.Default $fullModuleName (conv.Join (coll.GoSlice ($fullModuleName | strings.SplitN "-" 3) 1) "-")) -}} +--- +title: {{ $fullModuleName }} +sidebar_label: {{ $fullModuleName }} +sidebar_class_name: command +{{- if has (ds "config") "description" }} +description: |- +{{ index ( (ds "config").description | strings.Split "." ) 0 | strings.Indent 2 }} +{{- end }} +{{- if has (ds "config") "tags" }} +tags: +{{ (ds "config").tags | data.ToYAML | strings.Indent 2 -}} +{{- end }} +custom_edit_url: https://github.com/cloudposse-terraform-components/{{ $fullModuleName }}/edit/main/README.md +--- + +{{- if $deprecated }} +## Deprecated + +{{ if has (ds "config").deprecated "notice" }} +{{- (ds "config").deprecated.notice }} +{{- else }} +This module is no longer actively maintained +{{- end }} + +{{ if has (ds "config") "description" }} +### Historical Description + +{{(ds "config").description }} +{{- end }} +{{- else }} +{{- if has (ds "config") "description" }} +{{ (ds "config").description }} + +{{- end }} +{{- end }} + +{{ if has (ds "config") "screenshots" }} +## Screenshots + +{{ range $screenshot := (ds "config").screenshots }} +{{ printf "![%s](%s)\n*%s*" $screenshot.name $screenshot.url $screenshot.description }}{{ end }} +{{- end }} + +{{ if has (ds "config") "introduction" }} +## Introduction + +{{ (ds "config").introduction }} +{{ end }} + +{{ if has (ds "config") "usage" }} +## Usage + +{{ (ds "config").usage -}} +{{ end }} + +{{ if not $deprecated -}} +{{ if has (ds "config") "quickstart" -}} +## Quick Start + +{{ (ds "config").quickstart -}} +{{ end }} + +{{ if has (ds "config") "examples" }} +## Examples + +{{(ds "config").examples }} +{{ end }} + +{{ if has (ds "config") "include" }} +{{ range $file := (datasource "config").include -}} +{{ (include "includes" $file) }} +{{- end }} +{{- end }} +{{- end }} diff --git a/scripts/docs-collator/component/renderer/templates/components/yml/terraform-docs.yml b/scripts/docs-collator/component/renderer/templates/components/yml/terraform-docs.yml new file mode 100644 index 000000000..50b752c14 --- /dev/null +++ b/scripts/docs-collator/component/renderer/templates/components/yml/terraform-docs.yml @@ -0,0 +1,225 @@ +# https://pkg.go.dev/github.com/terraform-docs/terraform-docs/terraform#Module +content: |- + {{- $context_variables := list "context" "enabled" "namespace" "tenant" "environment" "stage" "name" "delimiter" "attributes" "tags" "additional_tag_map" "label_order" "regex_replace_chars" "id_length_limit" "label_key_case" "label_value_case" "descriptor_formats" "labels_as_tags" -}} + + {{ .Header }} + + ## Variables + + ### Required Variables + +
+ {{- range .Module.Inputs }} + {{- if and (not (has .Name $context_variables)) .Required }} +
`{{ .Name }}`{{ if lt (len (split "\n" (tostring .Type))) 2 }} (`{{ tostring .Type }}`){{end}} required{{ if contains "OBSOLETE: " (tostring .Description) }} OBSOLETE{{ end }}
+
+ {{- $lines := regexSplit "\n" (tostring .Description | replace "OBSOLETE: " "") -1 -}} + {{- range $lines }} + {{ . }}
+ {{- end }} + + {{ if gt (len (split "\n" (tostring .Type))) 2 }} + **Type:** + ```hcl + {{ .Type.Raw }} + ``` +
+ {{ end }} + +
+ {{- end }} + {{- end }} +
+ + {{ $optional := false -}} + {{ range .Module.Inputs }}{{ if not .Required }}{{ $optional = true -}}{{ end -}}{{ end -}} + + {{ if $optional -}} + + ### Optional Variables + +
+ {{- range .Module.Inputs }} + {{- if and (not (has .Name $context_variables)) (not .Required) }} +
`{{ .Name }}`{{ if lt (len (split "\n" (tostring .Type))) 2 }} (`{{ tostring .Type }}`){{end}} optional{{ if contains "OBSOLETE: " (tostring .Description) }} OBSOLETE{{ end }}
+
+ {{- $lines := regexSplit "\n" (tostring .Description | replace "OBSOLETE: " "") -1 -}} + {{- range $lines }} + {{ . }}
+ {{- end }} +
+ {{ if gt (len (split "\n" (tostring .Type))) 2 }} + **Type:** + ```hcl + {{ tostring .Type }} + ``` +
+ {{ end }} + **Default value:** {{ if lt (len (split "\n" .GetValue)) 2 }}`{{ .GetValue | replace "{}" "{ }" | replace "[]" "[ ]" }}`{{ else }} + ```hcl + {{- $lines := regexSplit "\n" .GetValue -1 -}} + {{- range $lines }} + {{ . }} + {{- end }} + ``` + {{ end }} +
+ {{- end -}} + {{ end -}} +
+ {{ end }} + + ### Context Variables + +
+ + The following variables are defined in the `context.tf` file of this module and part of the [terraform-null-label](https://registry.terraform.io/modules/cloudposse/label/null) pattern. + + +
+ {{- range .Module.Inputs }} + {{- if and (has .Name $context_variables) }} +
`{{ .Name }}`{{ if lt (len (split "\n" (tostring .Type))) 2 }} (`{{ tostring .Type }}`){{end}} {{ ternary .Required "required" "optional" }}{{ if contains "OBSOLETE: " (tostring .Description) }} OBSOLETE{{ end }}
+
+ {{- $lines := regexSplit "\n" (tostring .Description | replace "OBSOLETE: " "") -1 -}} + {{- range $lines }} + {{ . }}
+ {{- end }} + **Required:** {{ ternary .Required "Yes" "No" }}
+ {{ if gt (len (split "\n" (tostring .Type))) 2 }} + **Type:** + ```hcl + {{ .Type.Raw }} + ``` +
+ {{ end }} + **Default value:** {{ if lt (len (split "\n" .GetValue)) 2 }}`{{ .GetValue | replace "{}" "{ }" | replace "[]" "[ ]" }}`{{ else }} + ```hcl + {{- $lines := regexSplit "\n" .GetValue -1 -}} + {{- range $lines }} + {{ . }} + {{- end }} + ``` + {{ end }} +
+ {{- end }} + {{- end }} +
+
+ + {{ if ne (len .Module.Outputs) 0 -}} + ## Outputs + +
+ {{- range .Module.Outputs }} +
`{{ .Name }}`{{ if contains "OBSOLETE: " (tostring .Description) }} OBSOLETE{{ end }}
+
+ {{- $lines := regexSplit "\n" (tostring .Description | replace "OBSOLETE: " "" | default "n/a" ) -1 -}} + {{- range $lines }} + {{ . }}
+ {{- end }} +
+ {{- end }} +
+ {{- end }} + + ## Dependencies + + {{ if ne (len .Module.Requirements) 0 -}} + ### Requirements + {{ range .Module.Requirements }} + - `{{ .Name }}`{{ if .Version }}, version: `{{ .Version }}`{{ end }} + {{- end }} + {{- end }} + + {{ if ne (len .Module.Providers) 0 -}} + ### Providers + {{ range .Module.Providers }} + - `{{ .Name }}`{{ if .Version }}, version: `{{ .Version }}`{{ end }} + {{- end }} + {{- end }} + + {{ if ne (len .Module.ModuleCalls) 0 -}} + ### Modules + + Name | Version | Source | Description + --- | --- | --- | --- + {{ range .Module.ModuleCalls }} + {{- $baseUrl := "https://registry.terraform.io/modules" -}} + {{ if contains "//" .Source -}} + {{- $moduleParts := regexSplit "//" .Source 2 -}} + {{- $moduleName := first $moduleParts -}} + {{- $submodulePath := last $moduleParts -}} + `{{ .Name }}` | {{ .Version | default "latest" }} | {{ printf "[`%s`](%s/%s/%s/%s)" .Source $baseUrl $moduleName $submodulePath .Version }} | {{ tostring .Description | default "n/a" }} + {{ else -}} + `{{ .Name }}` | {{ .Version | default "latest" }} | {{ printf "[`%s`](%s/%s/%s)" .Source $baseUrl .Source .Version }} | {{ tostring .Description | default "n/a" }} + {{ end }} + {{- end -}} + {{- end -}} + + {{- if ne (len .Module.Resources) 0 }} + + ## Resources + + The following resources are used by this module: + {{ range .Module.Resources }} + {{- $isResource := and $.Config.Sections.Resources ( eq "resource" (printf "%s" .GetMode)) }} + {{- if $isResource }} + {{- $fullspec := ternary .URL (printf "[`%s`](%s)" .Spec .URL) .Spec }} + - {{ $fullspec }} {{ printf "(%s)" .GetMode -}} + {{- end }} + {{- end }} + + ## Data Sources + + The following data sources are used by this module: + {{ range .Module.Resources }} + {{- $isDataResource := and $.Config.Sections.DataSources ( eq "data source" (printf "%s" .GetMode)) }} + {{- if $isDataResource }} + {{- $fullspec := ternary .URL (printf "[`%s`](%s)" .Spec .URL) .Spec }} + - {{ $fullspec }} {{ printf "(%s)" .GetMode -}} + {{- end }} + {{- end }} + {{- end }} + +formatter: "markdown document" # this is required +version: "" + +recursive: + enabled: false + path: modules + +sections: + hide: [] + show: [] + +sort: + enabled: true + by: required + +output-values: + enabled: false + from: '' + +output: + file: ../README.md + mode: inject + template: |- + + {{ .Content }} + + +settings: + anchor: true + color: true + default: true + description: true + escape: true + hide-empty: false + html: true + indent: 2 + lockfile: false + read-comments: true + required: true + sensitive: true + type: true diff --git a/scripts/docs-collator/component/renderer/templates/make/Makefile b/scripts/docs-collator/component/renderer/templates/make/Makefile new file mode 100644 index 000000000..b25677208 --- /dev/null +++ b/scripts/docs-collator/component/renderer/templates/make/Makefile @@ -0,0 +1,3 @@ +SHELL := /bin/bash + +-include $(shell curl -sSL -o .build-harness "https://cloudposse.tools/build-harness"; echo .build-harness) diff --git a/scripts/docs-collator/component/renderer/yml.py b/scripts/docs-collator/component/renderer/yml.py new file mode 100644 index 000000000..f5d4d8938 --- /dev/null +++ b/scripts/docs-collator/component/renderer/yml.py @@ -0,0 +1,112 @@ +import os +import subprocess +import logging + +from utils import io, rendering, templating +from AbstractRenderer import AbstractRenderer, TerraformDocsRenderingError +from component.component import Component + +SCRIPT_DIR = os.path.dirname(os.path.realpath(__file__)) +README_YAML = "README.yaml" +README_MD = "README.md" + +IMAGES_DIR = "images" +MODULES_README_TEMPLATE = "readme.md" + + +class ComponentRendererYaml(AbstractRenderer): + def __init__(self, component: Component): + self.templates_dir = os.path.join(SCRIPT_DIR, "templates/components/yml") + self.component = component + + def render(self, docs_dir): + logging.info(f"Rendering doc for: {self.component.repo.full_name}") + logging.info(f"Provider: {self.component.provider}, Module: {self.component.name}") + + path_components = [docs_dir, self.component.provider] + path_components.extend(self.component.subdirs) + path_components.append(self.component.name) + module_docs_dir = os.path.join(*path_components) + logging.info(f"Module docs dir: {module_docs_dir}") + + self._pre_rendering_fixes(self.component, self.component.dir) + readme_md_file = self.__render_readme(module_docs_dir) + self._post_rendering_fixes(readme_md_file) + + self._copy_extra_resources_for_docs(self.component.dir, module_docs_dir) + self.__copy_extra_resources_for_images(module_docs_dir) + + def __render_readme(self, module_docs_dir): + readme_yaml_file = os.path.join(self.component.dir, README_YAML) + readme_md_file = os.path.join(self.component.dir, README_MD) + + readme_tmpl_file = os.path.join(self.templates_dir, MODULES_README_TEMPLATE) + + io.create_dirs(module_docs_dir) + # Run the make readme command in the module directory to compile README.md + logging.debug(f"Rendering README.md for: {self.component.dir}") + logging.debug(f"make readme") + logging.debug(f"README_TEMPLATE_FILE: {readme_tmpl_file}") + logging.debug(f"README_FILE: {readme_md_file}") + logging.debug(f"README_YAML: {readme_yaml_file}") + logging.debug(f"README_TEMPLATE_YAML: {readme_yaml_file}") + logging.debug(f"README_INCLUDES: {self.component.dir}") + response = subprocess.run( + [ + "make", + "readme", + f"README_TEMPLATE_FILE={os.path.abspath(readme_tmpl_file)}", + f"README_FILE={os.path.abspath(readme_md_file)}", + f"README_YAML={os.path.abspath(readme_yaml_file)}", + f"README_TEMPLATE_YAML={os.path.abspath(readme_yaml_file)}", + f"README_INCLUDES={os.path.abspath(self.component.dir)}/", + ], + capture_output=True, + cwd=os.path.join(SCRIPT_DIR, "templates", "make"), + ) + + if response.returncode != 0: + error_message = response.stderr.decode("utf-8") + raise TerraformDocsRenderingError(error_message) + + logging.info(f"Rendered: {readme_md_file}") + + # Re-render terraform docs with this repo's terraform-docs template for modules. + # This replaces docs/terraform.md for the given module in place + logging.debug(f"Rendering terraform docs for: {self.component.dir}") + rendering.render_terraform_docs( + self.component.terraform_dir, os.path.join(self.templates_dir, "terraform-docs.yml") + ) + + readme_md_file = os.path.join(self.component.dir, README_MD) + io.copy_file(readme_md_file, os.path.join(module_docs_dir, README_MD)) + return os.path.join(module_docs_dir, README_MD) + + def _post_rendering_fixes(self, readme_md_file, submodule_dir=""): + content = io.read_file_to_string(readme_md_file) + content = rendering.fix_self_non_closing_br_tags(content) + content = rendering.fix_custom_non_self_closing_tags_in_pre(content) + content = rendering.fix_github_edit_url(content, self.component.repo, submodule_dir) + content = rendering.fix_sidebar_label( + content, self.component.repo, self.component.name + ) + content = rendering.replace_relative_links_with_github_links( + self.component.repo, content, self.component.name + ) + content = rendering.fix_mdx_format(content) + content = rendering.reformat_admonitions(content) + io.save_string_to_file(readme_md_file, content) + + def __copy_extra_resources_for_images(self, module_docs_dir): + extra_resources_dir = os.path.join(self.component.dir, IMAGES_DIR) + files = io.get_filenames_in_dir(extra_resources_dir, "*", True) + + for file in files: + if os.path.isdir(file): + continue + + dest_file = os.path.join( + module_docs_dir, IMAGES_DIR, os.path.relpath(file, extra_resources_dir) + ) + io.copy_file(file, dest_file) + logging.info(f"Copied extra file: {dest_file}") diff --git a/scripts/docs-collator/component/repository/__init__.py b/scripts/docs-collator/component/repository/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/scripts/docs-collator/component/repository/abstract.py b/scripts/docs-collator/component/repository/abstract.py new file mode 100644 index 000000000..0d822f735 --- /dev/null +++ b/scripts/docs-collator/component/repository/abstract.py @@ -0,0 +1,77 @@ + +class ComponentRepositoryAbstract: + _full_name = None + _name = None + _default_branch = None + _module_download_dir = None + + def __init__(self, repo, module_download_dir): + self._full_name = repo.full_name + self._name = repo.name + self._module_download_dir = module_download_dir + self._default_branch = repo.default_branch + + @property + def components(self): + raise NotImplementedError() + @property + def full_name(self): + return self._full_name + + @property + def name(self): + return self._name + + @property + def dir(self): + return self._module_download_dir + + @property + def default_branch(self): + return self._default_branch + + @property + def provider(self): + return self._parse_terraform_repo_name()[0] + + @property + def subdirs(self): + return self._parse_terraform_repo_name()[1] + + @property + def module_name(self): + return self._parse_terraform_repo_name()[2] + + def _parse_terraform_repo_name(self): + name_items = self._name.split("-") + provider = name_items[0] + module_name = "-".join(name_items[1:]) + subdirs = [] + + groups = [ + "eks", "auth0", "dms", "glue", "managed-grafana-data-source", + "managed-grafana", "managed-prometheus", "spacelift", "tgw" + ] + + for group in groups: + if module_name.startswith(group): + subdirs = [group] + module_name = module_name[len(f"{group}-"):] + break + + preserve_provider_prefix = [ + "team-roles", "teams", "saml", "shield", "inspector", + "inspector2", "backup", "ssosync", "config" + ] + + for item in preserve_provider_prefix: + if module_name == item: + module_name = f"{provider}-{item}" + break + + if module_name == "": + provider = "null" + module_name = self._name + elif module_name == "argocd": + subdirs = ["eks"] + return provider, subdirs, module_name diff --git a/scripts/docs-collator/component/repository/multiple.py b/scripts/docs-collator/component/repository/multiple.py new file mode 100644 index 000000000..98117e83f --- /dev/null +++ b/scripts/docs-collator/component/repository/multiple.py @@ -0,0 +1,47 @@ +import os + +from .abstract import ComponentRepositoryAbstract +from ..component import Component +from utils import io + +TERRAFORM_DIR = "src" + +# Multiple components in a single repository is not used at the moment +# Leaving this here for future use +class ComponentRepositoryMultiple(ComponentRepositoryAbstract): + @property + def components(self): + result = [] + if self.subdirs and len(self.subdirs) == 1 and self.subdirs[0] == self.module_name: + result.append( + Component( + repo=self, + module_download_dir=self._module_download_dir, + terraform_dir=self._module_download_dir, + name=self.module_name, + subdirs=[] + ) + ) + + files = io.get_filenames_in_dir(os.path.join(self._module_download_dir, TERRAFORM_DIR), "*.tf", True) + components_dirs = {} + + for file in files: + dir_name = os.path.dirname(file) + name = os.path.basename(os.path.dirname(file)) + + rel_dir = os.path.relpath(dir_name, os.path.join(self._module_download_dir, TERRAFORM_DIR)) + if rel_dir not in components_dirs: + components_dirs[rel_dir] = name + for component_dir in components_dirs.keys(): + path = os.path.join(self._module_download_dir, TERRAFORM_DIR, component_dir) + result.append( + Component( + repo=self, + module_download_dir=path, + terraform_dir=path, + name=component_dir, + subdirs=self.subdirs + ) + ) + return result diff --git a/scripts/docs-collator/component/repository/single.py b/scripts/docs-collator/component/repository/single.py new file mode 100644 index 000000000..21b17aab0 --- /dev/null +++ b/scripts/docs-collator/component/repository/single.py @@ -0,0 +1,19 @@ +import os + +from .abstract import ComponentRepositoryAbstract +from ..component import Component + +TERRAFORM_DIR = "src" + + +class ComponentRepositorySingle(ComponentRepositoryAbstract): + + @property + def components(self): + return [Component( + repo=self, + module_download_dir=self._module_download_dir, + terraform_dir=os.path.join(self._module_download_dir, TERRAFORM_DIR), + name=self.module_name, + subdirs=self.subdirs + )] diff --git a/scripts/docs-collator/render_docs_for_components.py b/scripts/docs-collator/render_docs_for_components.py index 9dbd653df..79e17f92c 100644 --- a/scripts/docs-collator/render_docs_for_components.py +++ b/scripts/docs-collator/render_docs_for_components.py @@ -1,58 +1,142 @@ import logging -import os +import sys import click -from ComponentRenderer import ComponentRenderer -from utils import io +from GitHubProvider import GitHubProvider +from component.fetcher import ComponentFetcher +from ModuleFetcher import MissingReadmeYamlException +from AbstractRenderer import TerraformDocsRenderingError +from component.renderer.factory import ComponentRendererFactory OUTPUT_DOC_DIR = "content/components/library/aws" -CLONED_REPO_DIR = "tmp/components/terraform-aws-components" +DOWNLOAD_TMP_DIR = "tmp/components" +GITHUB_ORG = "cloudposse-terraform-components" +# def main(input_dir, output_dir): +# modules_dir = os.path.join(input_dir, "modules") +# +# logging.info(f"Looking for modules in: {modules_dir}") +# +# components = io.get_subfolders(modules_dir) +# +# logging.info(f"Found {len(components)} components") +# +# renderer = ComponentRenderer(input_dir, output_dir) +# +# for component in components: +# logging.info(f"Processing component: {component}") +# renderer.render(component) -def main(input_dir, output_dir): - modules_dir = os.path.join(input_dir, "modules") +def main( + github_api_token, + output_dir, + download_dir, + includes, + excludes, + fail_on_rendering_error, +): + github_provider = GitHubProvider(github_api_token) + fetcher = ComponentFetcher(github_provider, download_dir) + logging.info(f"Fetching repositories ...") - logging.info(f"Looking for modules in: {modules_dir}") + repos = github_provider.get_components_repos(GITHUB_ORG, includes, excludes) - components = io.get_subfolders(modules_dir) + logging.info(f"Found {len(repos)} repositories to process") - logging.info(f"Found {len(components)} components") + for repo in repos: + # logging.info(f"Debugging module: {repo.full_name}") + # if repo.full_name != "cloudposse/terraform-aws-ec2-ami-backup": + # logging.info(f"skipping...") + # continue - renderer = ComponentRenderer(input_dir, output_dir) + try: + logging.info(f"Fetching files for: {repo.full_name}") + component_repo = fetcher.fetch(repo) + except MissingReadmeYamlException as e: + logging.warning(e.message) + continue + + try: + for renderer in ComponentRendererFactory.produce(component_repo): + renderer.render(output_dir) + except TerraformDocsRenderingError as e: + if fail_on_rendering_error: + logging.error(e.message) + sys.exit(1) + else: + logging.warning(e.message) + continue - for component in components: - logging.info(f"Processing component: {component}") - renderer.render(component) @click.command() @click.option( - "--input-dir", - default=CLONED_REPO_DIR, + "--github-api-token", + envvar="PUBLIC_REPO_ACCESS_TOKEN", required=True, - help="Path to cloned repository", + help="GitHub API token", ) @click.option( "--output-dir", default=OUTPUT_DOC_DIR, - required=False, + required=True, help="Rendered documentation output directory", ) @click.option( - "--log-level", + "--download-dir", + default=DOWNLOAD_TMP_DIR, + required=True, + help="Temporary download directory", +) +@click.option( + "--includes", + required=False, + help="Comma separated list of repos to include. Conflicts with --excludes. Example: 'terraform-aws-s3-log-storage,terraform-aws-network-firewall'", +) +@click.option( + "--excludes", + required=False, + help="Comma separated list of repos to exclude. Conflicts with --includes. Example: 'terraform-aws-s3-log-storage,terraform-aws-network-firewall'", +) +@click.option( + "--fail-on-rendering-error", + is_flag=True, + show_default=True, default=False, required=False, - help="Recursive lookup in sub folders of README.md files", + help="Fail on rendering error", +) +@click.option( + "--log-level", default="INFO", required=False, help="Log level. Available options" ) -def cli_main(input_dir, output_dir, log_level): +def cli_main( + github_api_token, + output_dir, + download_dir, + includes, + excludes, + fail_on_rendering_error, + log_level, +): logging.basicConfig( format="[%(asctime)s] %(levelname)s %(message)s", datefmt="%d-%m-%Y %H:%M:%S", level=logging.getLevelName(log_level), ) - main(input_dir, output_dir) + logging.info( + f"Download directory: {download_dir}, documentation output directory: {output_dir}" + ) + + main( + github_api_token, + output_dir, + download_dir, + includes, + excludes, + fail_on_rendering_error, + ) if __name__ == "__main__": diff --git a/scripts/docs-collator/render_docs_for_github_actions.py b/scripts/docs-collator/render_docs_for_github_actions.py index 6f9b36ed2..b235530cd 100644 --- a/scripts/docs-collator/render_docs_for_github_actions.py +++ b/scripts/docs-collator/render_docs_for_github_actions.py @@ -12,6 +12,7 @@ OUTPUT_DOC_DIR = "content/github-actions/library" REPOS_SKIP_LIST = {"terraform-aws-components"} +GITHUB_ORG = "cloudposse" def main( github_api_token, @@ -27,7 +28,7 @@ def main( logging.info(f"Fetching repositories ...") - repos = github_provider.get_github_actions_repos(includes_csv, excludes_csv) + repos = github_provider.get_github_actions_repos(GITHUB_ORG, includes_csv, excludes_csv) logging.info(f"Found {len(repos)} repositories to process") diff --git a/scripts/docs-collator/render_docs_for_modules.py b/scripts/docs-collator/render_docs_for_modules.py index 7cbbc6d5c..0f09a7763 100644 --- a/scripts/docs-collator/render_docs_for_modules.py +++ b/scripts/docs-collator/render_docs_for_modules.py @@ -11,6 +11,7 @@ DOWNLOAD_TMP_DIR = "tmp/modules" OUTPUT_DOC_DIR = "content/modules/library" +GITHUB_ORG = "cloudposse" def main( github_api_token, @@ -26,7 +27,7 @@ def main( logging.info(f"Fetching repositories ...") - repos = github_provider.get_terraform_repos(includes_csv, excludes_csv) + repos = github_provider.get_terraform_repos(GITHUB_ORG, includes_csv, excludes_csv) logging.info(f"Found {len(repos)} repositories to process") diff --git a/scripts/docs-collator/requirements.txt b/scripts/docs-collator/requirements.txt index 73159f85f..775d693db 100644 --- a/scripts/docs-collator/requirements.txt +++ b/scripts/docs-collator/requirements.txt @@ -7,7 +7,7 @@ idna==3.4 Jinja2==3.1.2 MarkupSafe==2.1.2 pycparser==2.21 -PyGithub==1.57 +PyGithub==2.5.0 PyJWT==2.6.0 PyNaCl==1.5.0 requests==2.28.2 diff --git a/scripts/docs-collator/testComponents.py b/scripts/docs-collator/testComponents.py new file mode 100644 index 000000000..ce27769f5 --- /dev/null +++ b/scripts/docs-collator/testComponents.py @@ -0,0 +1,45 @@ +import os +import unittest + +from GitHubProvider import GitHubProvider +from component.fetcher import ComponentFetcher +from component.repository.single import ComponentRepositorySingle +from component.repository.multiple import ComponentRepositoryMultiple +from component.renderer.factory import ComponentRendererFactory + + +class TestComponentFetcher(unittest.TestCase): + github_provider = None + def setUp(self): + token = None + if os.environ['PUBLIC_REPO_ACCESS_TOKEN']: + token = os.environ['PUBLIC_REPO_ACCESS_TOKEN'] + self.github_provider = GitHubProvider(github_api_token=token) + self.download_dir = "tmp/test/components" + self.output_dir = "tmp/test/docs" + + def test_single_with_modules(self): + fetcher = ComponentFetcher(self.github_provider, self.download_dir) + + repo = self.github_provider.github.get_repo("cloudposse-terraform-components/aws-account-map") + self.assertEqual(repo.name, "aws-account-map") + component_repo = fetcher.fetch(repo) + self.assertIsInstance(component_repo, ComponentRepositorySingle) + renderers = ComponentRendererFactory.produce(component_repo) + self.assertEqual(len(renderers), 4) + for renderer in renderers: + renderer.render(self.output_dir) + + def test_single_with_changlelog(self): + fetcher = ComponentFetcher(self.github_provider, self.download_dir) + + repo = self.github_provider.github.get_repo("cloudposse-terraform-components/aws-argocd-github-repo") + self.assertEqual(repo.name, "aws-argocd-github-repo") + component_repo = fetcher.fetch(repo) + self.assertIsInstance(component_repo, ComponentRepositorySingle) + renderers = ComponentRendererFactory.produce(component_repo) + self.assertEqual(len(renderers), 1) + for renderer in renderers: + renderer.render(self.output_dir) + + diff --git a/scripts/docs-collator/utils/rendering.py b/scripts/docs-collator/utils/rendering.py index bcaa741c8..25a581558 100644 --- a/scripts/docs-collator/utils/rendering.py +++ b/scripts/docs-collator/utils/rendering.py @@ -1,3 +1,4 @@ +import os import re import subprocess import yaml @@ -6,7 +7,7 @@ BR_REGEX = re.compile(re.escape("
"), re.IGNORECASE) SIDEBAR_LABEL_REGEX = re.compile("sidebar_label: .*", re.IGNORECASE) CUSTOM_EDIT_URL_REGEX = re.compile("custom_edit_url: .*", re.IGNORECASE) -NAME_REGEX = re.compile("name: .*", re.IGNORECASE) +NAME_REGEX = re.compile("^name: .*", re.IGNORECASE) RELATIVE_LINK_PATTERN = r"\]\((?!http[s]?://)([^)\s]+)\)" @@ -258,6 +259,7 @@ def strip_frontmatter(content): return "\n".join(content_lines), "\n".join(frontmatter) + def reformat_admonitions(content): """ Reformat admonitions to be compatible with Docusaurus. @@ -308,6 +310,7 @@ def reformat_admonitions(content): return '\n'.join(result) + def remove_https_cloudposse_docs(content): """ In component readmes, we have links to the cloudposse docs, @@ -317,6 +320,7 @@ def remove_https_cloudposse_docs(content): """ return content.replace("https://docs.cloudposse.com/", "/") + def strip_title(content): """ The title is created with frontmatter. @@ -324,6 +328,7 @@ def strip_title(content): """ return re.sub(r"# Component:(.*)", "", content).strip() + def replace_broken_links(content): """ Replace broken links in content. diff --git a/scripts/render-docs-for-components.sh b/scripts/render-docs-for-components.sh index a0eb2d7b5..9d268e89c 100755 --- a/scripts/render-docs-for-components.sh +++ b/scripts/render-docs-for-components.sh @@ -5,12 +5,10 @@ set -e GITHUB_ORG=${GITHUB_ORG:-"cloudposse"} GITHUB_REPO=${GITHUB_REPO:-"terraform-aws-components"} GIT_BRANCH=${GIT_BRANCH:-"main"} -TMP_CLONE_DIR="${TMP_CLONE_DIR:-tmp/components/${GITHUB_REPO}}" -RENDERED_DOCS_DIR="${RENDERED_DOCS_DIR:-docs/components/library/aws/}" - -echo "Cloning repo '${GITHUB_ORG}/${GITHUB_REPO}'" -git clone --depth 1 --branch "${GIT_BRANCH}" https://github.com/${GITHUB_ORG}/${GITHUB_REPO}.git "${TMP_CLONE_DIR}" || true +RENDERED_DOCS_DIR="${RENDERED_DOCS_DIR:-docs/components/library}" +DOWNLOAD_TMP_DIR="${DOWNLOAD_TMP_DIR:-tmp/components}" python scripts/docs-collator/render_docs_for_components.py \ - --input-dir "${TMP_CLONE_DIR}" \ - --output-dir "${RENDERED_DOCS_DIR}" + --download-dir "${DOWNLOAD_TMP_DIR}" \ + --output-dir "${RENDERED_DOCS_DIR}" \ + --excludes "vpc"