diff --git a/docs/layers/github-actions/github-oidc-with-aws.mdx b/docs/layers/github-actions/github-oidc-with-aws.mdx index 4b6787357..4e20355c8 100644 --- a/docs/layers/github-actions/github-oidc-with-aws.mdx +++ b/docs/layers/github-actions/github-oidc-with-aws.mdx @@ -88,10 +88,18 @@ sequenceDiagram ### Deploy GitHub OIDC Provider Component After deploying the [GitHub OIDC Provider component](/components/library/aws/github-oidc-provider/) into an account, you should see the Identity Provider in IAM in the AWS Web Console. + + Deploy this component in each account where GitHub Actions need to assume a role. + + + - Import `catalog/github-oidc-provider` in the `gbl` stack for the given account + - Deploy the `github-oidc-provider` component: `atmos terraform apply github-oidc-provider -s plat-gbl-dev` + + - ### Configure GitHub OIDC Mixin Role and Policy + ### Option 1: Configure GitHub OIDC Mixin Role and Policy Use the mixin to grant GitHub the ability to assume a role for a specific component. @@ -100,7 +108,7 @@ sequenceDiagram - ### Deploy GitHub OIDC Role Component + ### Option 2: Deploy GitHub OIDC Role Component Deploy the [GitHub OIDC Role component](/components/library/aws/github-oidc-role/) to create a generalized role for GitHub to access several resources in AWS. diff --git a/docs/layers/software-delivery/ecs-ecspresso/ecs-partial-task-definitions.mdx b/docs/layers/software-delivery/ecs-ecspresso/ecs-partial-task-definitions.mdx new file mode 100644 index 000000000..f360e0eef --- /dev/null +++ b/docs/layers/software-delivery/ecs-ecspresso/ecs-partial-task-definitions.mdx @@ -0,0 +1,221 @@ +--- +title: "ECS Partial Task Definitions" +sidebar_label: "Partial Task Definitions" +sidebar_position: 20 +--- + +import Intro from '@site/src/components/Intro'; +import Steps from '@site/src/components/Steps'; + + + This document describes what partial task definitions are and how we can use them to set up ECS services using Terraform and GitHub Actions. + + +## The Problem + +Managing ECS Services is challenging. Ideally, we want our services to be managed by Terraform so everything is living +in code. However, we also want to update the task definition via GitOps as through the GitHub release lifecycle. This is +challenging because Terraform can create the task definition, but if updated by the application repository, the task +definition will be out of sync with the Terraform state. + +Managing it entirely through Terraform means we cannot easily update the newly built image by the application repository +unless we directly commit to the infrastructure repository, which is not ideal. + +Managing it entirely through the application repository means we cannot codify the infrastructure and have to hardcode +ARNs, secrets, and other infrastructure-specific configurations. + +## Introduction + +ECS Partial task definitions is the idea of breaking the task definition into smaller parts. This allows for easier +management of the task definition and makes it easier to update the task definition. + +We do this by setting up Terraform to manage a portion of the task definition, and the application repository to manage +another portion. + +The Terraform (infrastructure) portion is created first. It will create an ECS Service in ECS, and then upload the task +definition JSON to S3 as `task-template.json`.The application repository will have a `task-definition.json` git +controlled, during the development lifecycle, the application repository will download the task definition from S3, +merge the task definitions, then update the ECS Service with the new task definition. Finally, GitHub actions will +update the S3 bucket with the deployed task definition under `task-definition.json`. If Terraform is planned again, it +will use the new task definition as the base for the next deployment, thus not resetting the image or application +configuration. + +
+ +### Pros + +The **benefit** to using this approach is that we can manage the task definition portion in Terraform with the +infrastructure, meaning secrets, volumes, and other ARNs can be managed in Terraform. If a filesystem ID updates we can +re-apply Terraform to update the task definition with the new filesystem ID. The application repository can manage the +container definitions, environment variables, and other application-specific configurations. This allows developers who +are closer to the application to quickly update the environment variables or other configuration. + +### Cons + +The drawback to this approach is that it is more complex than managing the task definition entirely in Terraform or the +application repository. It requires more setup and more moving parts. It can be confusing for a developer who is not +familiar with the setup to understand how the task definition is being managed and deployed. + +This also means that when something goes wrong, it becomes harder to troubleshoot as there are more moving parts. + +### Getting Setup + +#### Pre-requisites + +- Application Repository - [Cloud Posse Example ECS Application](https://github.com/cloudposse-examples/app-on-ecs) +- Infrastructure Repository +- ECS Cluster - [Cloud Posse Docs](https://docs.cloudposse.com/components/library/aws/ecs/) - + [Component](https://github.com/cloudposse/Terraform-aws-components/tree/main/modules/ecs). +- `ecs-service` - [Cloud Posse Docs](https://docs.cloudposse.com/components/library/aws/ecs-service/) - + [Component](https://github.com/cloudposse/Terraform-aws-components/tree/main/modules/ecs-service). + - **Must** use the Cloud Posse Component. + - [`v1.416.0`](https://github.com/cloudposse/Terraform-aws-components/releases/tag/1.416.0) or later. +- S3 Bucket - [Cloud Posse Docs](https://docs.cloudposse.com/components/library/aws/s3-bucket/) - + [Component](https://github.com/cloudposse/Terraform-aws-components/tree/main/modules/s3-bucket). + +#### Steps + + + +1. Set up the S3 Bucket that will store the task definition. + +
This bucket should be in the same account as the ECS Cluster. + +
+
+ S3 Bucket Default Definition + + ```yaml + components: + terraform: + s3-bucket/defaults: + metadata: + type: abstract + vars: + enabled: true + account_map_tenant_name: core + # Suggested configuration for all buckets + user_enabled: false + acl: "private" + grants: null + force_destroy: false + versioning_enabled: false + allow_encrypted_uploads_only: true + block_public_acls: true + block_public_policy: true + ignore_public_acls: true + restrict_public_buckets: true + allow_ssl_requests_only: true + lifecycle_configuration_rules: + - id: default + enabled: true + abort_incomplete_multipart_upload_days: 90 + filter_and: + prefix: "" + tags: {} + # Move to Glacier after 2 years + transition: + - storage_class: GLACIER + days: 730 + # Never expire + expiration: {} + # Versioning isnt enabled, but these default values are still required + noncurrent_version_transition: + - storage_class: GLACIER + days: 90 + noncurrent_version_expiration: {} + ``` + +
+ + ```yaml + import: + - catalog/s3-bucket/defaults + + components: + terraform: + s3-bucket/ecs-tasks-mirror: #NOTE this is the component instance name. + metadata: + component: s3-bucket + inherits: + - s3-bucket/defaults + vars: + enabled: true + name: ecs-tasks-mirror + ``` + +2. Create an ECS Service in Terraform + +
Set up the ECS Service in Terraform using the + [`ecs-service` component](https://github.com/cloudposse/Terraform-aws-components/tree/main/modules/ecs-service). This + will create the ECS Service and upload the task definition to the S3 bucket. + +
To enable Partial Task Definitions, set the variable `s3_mirror_name` to be the component instance name of the + bucket to mirror to. For example `s3-bucket/ecs-tasks-mirror` + + ```yaml + components: + terraform: + ecs-services/defaults: + metadata: + component: ecs-service + type: abstract + vars: + enabled: true + ecs_cluster_name: "ecs/cluster" + s3_mirror_name: s3-bucket/ecs-tasks-mirror + ``` + +3. Set up an Application repository with GitHub workflows. + + An example application repository can be found [here](https://github.com/cloudposse-examples/app-on-ecs). + +
Two things need to be pulled from this repository: + + - The `task-definition.json` file under `deploy/task-definition.json` + - The GitHub Workflows. + + An important note about the GitHub Workflows, in the example repository they all live under `.github/workflows`. This + is done so development of workflows can be fast, however we recommend moving the shared workflows to a separate + repository and calling them from the application repository. The application repository should only contain the + workflows `main-branch.yaml`, `release.yaml` and `feature-branch.yml`. + +
To enable Partial Task Definitions in the workflows, the call to + [`cloudposse/github-action-run-ecspresso` (link)](https://github.com/cloudposse-examples/app-on-ecs/blob/main/.github/workflows/workflow-cd-ecspresso.yml#L133-L147) + should have the input `mirror_to_s3_bucket` set to the S3 bucket name. the variable `use_partial_taskdefinition` + should be set to `'true'` + +
+ Example GitHub Action Step + + ```yaml + - name: Deploy + uses: cloudposse/github-action-deploy-ecspresso@0.6.0 + continue-on-error: true + if: ${{ steps.db_migrate.outcome != 'failure' }} + id: deploy + with: + image: ${{ steps.image.outputs.out }} + image-tag: ${{ inputs.tag }} + region: ${{ steps.environment.outputs.region }} + operation: deploy + debug: false + cluster: ${{ steps.environment.outputs.cluster }} + application: ${{ steps.environment.outputs.name }} + taskdef-path: ${{ inputs.path }} + mirror_to_s3_bucket: ${{ steps.environment.outputs.s3-bucket }} + use_partial_taskdefinition: "true" + timeout: 10m + ``` + +
+ + + +## Operation + +Changes through Terraform will not immediately be reflected in the ECS Service. This is because the task template has +been updated, but whatever was in the `task-definition.json` file in the S3 bucket will be used for deployment. + +To update the ECS Service after updating the Terraform for it, you must deploy through GitHub Actions. This will then +download the new template and create a new updated `task-defintion.json` to store in s3. diff --git a/docs/layers/software-delivery/ecs-ecspresso/setup.mdx b/docs/layers/software-delivery/ecs-ecspresso/setup.mdx index 9ee445561..147c4c414 100644 --- a/docs/layers/software-delivery/ecs-ecspresso/setup.mdx +++ b/docs/layers/software-delivery/ecs-ecspresso/setup.mdx @@ -23,7 +23,8 @@ import AtmosWorkflow from '@site/src/components/AtmosWorkflow'; | 3. Validate the environment configuration | Click Ops | | 4. Create a GitHub PAT | Click Ops | | 5. Set all Example App repository secrets | Click Ops | -| 6. Deploy the example ECS services | `atmos workflow deploy/app-on-ecs -f app-on-ecs` | +| 6. Deploy the shared ECS Task Definition S3 Bucket | `atmos apply s3-bucket/ecs-tasks-mirror -s < YOUR STACK >` | +| 7. Deploy the example ECS services | `atmos workflow deploy/app-on-ecs -f app-on-ecs` | @@ -306,6 +307,12 @@ We do not recommend keeping all shared workflows in the same repository as in th + + ### Configure the S3 Mirror Bucket, if not already configured + + If you haven't already configured the S3 mirror bucket, deploy and configure the shared S3 bucket for ECS tasks definitions now. Follow the [ECS Partial Task Definitions guide](/layers/software-delivery/ecs-ecspresso/ecs-partial-task-definitions/#steps) + + ### Deploy the Example App ECS Service @@ -368,7 +375,7 @@ We do not recommend keeping all shared workflows in the same repository as in th - Apply this component with the following: + Finally, apply the `ecs-services/example-app-on-ecs` component to deploy the Example App ECS service. diff --git a/package-lock.json b/package-lock.json index 3f666001f..8d20ec124 100644 --- a/package-lock.json +++ b/package-lock.json @@ -6428,8 +6428,8 @@ "license": "MIT" }, "node_modules/custom-loaders": { - "resolved": "plugins/custom-loaders", - "link": true + "version": "0.0.0", + "resolved": "file:plugins/custom-loaders" }, "node_modules/cytoscape": { "version": "3.30.1", @@ -18860,9 +18860,6 @@ "type": "github", "url": "https://github.com/sponsors/wooorm" } - }, - "plugins/custom-loaders": { - "version": "0.0.0" } } } diff --git a/static/assets/ecs-partial-task-definitions.png b/static/assets/ecs-partial-task-definitions.png new file mode 100644 index 000000000..cbdfeef95 Binary files /dev/null and b/static/assets/ecs-partial-task-definitions.png differ