From 2e949f586c42f2ce1bf3223cc99738abb55fe3b6 Mon Sep 17 00:00:00 2001 From: Murdock9803 Date: Wed, 9 Oct 2024 06:53:00 +0530 Subject: [PATCH 1/5] added the documentation related to CICD best practices Signed-off-by: Murdock9803 --- .../registering_workflows.md | 28 +++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/docs/user_guide/flyte_fundamentals/registering_workflows.md b/docs/user_guide/flyte_fundamentals/registering_workflows.md index 1c53a78020..d0344fb50f 100644 --- a/docs/user_guide/flyte_fundamentals/registering_workflows.md +++ b/docs/user_guide/flyte_fundamentals/registering_workflows.md @@ -358,6 +358,34 @@ two GitHub actions that facilitates this: of Flyte packages, for example, the `.tgz` archives that are created by `pyflyte package`. +### Some CI/CD best practices + +Several best practices are essential for ensuring smooth versioning, secure automation and image management. Tools like Docker, Helm and Github Actions help teams streamline their build, tagging, and release processes while maintaining security and scalability. + +#### Approach for Building, Tagging and Versioning + +- Build pipelines currently register Flyte workflows on each commit. Determine the version of the build. For commits on feature branches, use `-`. For commits on main branch, use `main-`. For released (tagged) versions we usually use the version number. + +- Build, tag and push docker image based on the version from step (1). + +- Serialize and register the Flyte workflows based on the version from step (1) and the docker image from step (2). This step is dependent on the success of step (2). Depending on whether it’s a feature branch, etc. we’re adjusting the domain to register to accordingly. + +#### Best practices to follow + +- **GitHub Actions for Workflow Automation**: Leverage GitHub Actions to automate the release process, including critical parts such as generating git tags, Docker images, and releasing Helm and manifest files. Automating these workflows reduces manual errors and speeds up the deployment cycle. + +- **Docker for Image Management**: +Use Docker to build, tag, and push images based on the version of the build. These Docker images are integral to the deployment pipeline, allowing Flyte workflows and other tasks to reference the correct image without manual work. + +- **Parallel Execution with Matrix Strategy**: +To optimize performance and efficiency through parallel execution of tasks, employ a matrix strategy in GitHub Actions. This is used in the `publish-flyte-component-image` job. + +- **Secure Authentication with GitHub Secrets**: +GitHub Secrets can be used to store sensitive data such as authentication tokens and credentials securely. This ensures that essential data is protected and not exposed in the codebase, safeguarding your automation processes. + +- **Helm and GoReleaser for Packaging and Releasing**: +Use Helm for managing Kubernetes package releases and GoReleaser for automating the release of Go applications. These tools streamline the packaging and distribution processes, making the CI/CD pipeline more robust and manageable. These are used in the `helm-release` and `manifest-release` jobs. + ## What's next? In this guide, you learned about the Flyte demo cluster, Flyte configuration, and From 5c0d2d31c64cfa3faf3a4bffec311964d78c9db7 Mon Sep 17 00:00:00 2001 From: AYUSH SAHU Date: Thu, 24 Oct 2024 17:38:03 +0530 Subject: [PATCH 2/5] Improved the documentation based on suggestions Signed-off-by: AYUSH SAHU --- .../registering_workflows.md | 25 +++++-------------- 1 file changed, 6 insertions(+), 19 deletions(-) diff --git a/docs/user_guide/flyte_fundamentals/registering_workflows.md b/docs/user_guide/flyte_fundamentals/registering_workflows.md index 916af8282f..d1ef9d71ea 100644 --- a/docs/user_guide/flyte_fundamentals/registering_workflows.md +++ b/docs/user_guide/flyte_fundamentals/registering_workflows.md @@ -367,31 +367,18 @@ two GitHub actions that facilitates this: ### Some CI/CD best practices -Several best practices are essential for ensuring smooth versioning, secure automation and image management. Tools like Docker, Helm and Github Actions help teams streamline their build, tagging, and release processes while maintaining security and scalability. +In case Flyte workflows are registered on each commit in your build pipelines, you can consider the following recommendations and approach: -#### Approach for Building, Tagging and Versioning +- **Versioning Strategy** : Determining the version of the build for different types of commits makes them consistent and identifiable. For commits on feature branches, use `-` and for the ones on main branches, use `main-`. Use version numbers for the released (tagged) versions. -- Build pipelines currently register Flyte workflows on each commit. Determine the version of the build. For commits on feature branches, use `-`. For commits on main branch, use `main-`. For released (tagged) versions we usually use the version number. +- **Container Image Management** : With Docker or any other CRI-compliant engine, build and push images to maintain traceability and consistency across environments. -- Build, tag and push docker image based on the version from step (1). +- **Workflow Serialization and Registration** : Workflows should be serialized and registered based on the versioning of the build and the container image. Depending on whether the build is for a feature branch or main, the registration domain should be adjusted accordingly. -- Serialize and register the Flyte workflows based on the version from step (1) and the docker image from step (2). This step is dependent on the success of step (2). Depending on whether it’s a feature branch, etc. we’re adjusting the domain to register to accordingly. +- **Container Image Specification** : When managing multiple images across tasks within a Flyte workflow, use the `--image` flag during registration to specify which image to use. This avoids hardcoding the image within the task definition, promoting reusability and flexibility in workflows. -#### Best practices to follow +- **Helm and GoReleaser for Packaging and Releasing**: Use Helm for managing Kubernetes package releases and GoReleaser for automating the release of Go applications. These tools streamline the packaging and distribution processes, making the CI/CD pipeline more robust and manageable. These are used in the `helm-release` and `manifest-release` jobs. -- **GitHub Actions for Workflow Automation**: Leverage GitHub Actions to automate the release process, including critical parts such as generating git tags, Docker images, and releasing Helm and manifest files. Automating these workflows reduces manual errors and speeds up the deployment cycle. - -- **Docker for Image Management**: -Use Docker to build, tag, and push images based on the version of the build. These Docker images are integral to the deployment pipeline, allowing Flyte workflows and other tasks to reference the correct image without manual work. - -- **Parallel Execution with Matrix Strategy**: -To optimize performance and efficiency through parallel execution of tasks, employ a matrix strategy in GitHub Actions. This is used in the `publish-flyte-component-image` job. - -- **Secure Authentication with GitHub Secrets**: -GitHub Secrets can be used to store sensitive data such as authentication tokens and credentials securely. This ensures that essential data is protected and not exposed in the codebase, safeguarding your automation processes. - -- **Helm and GoReleaser for Packaging and Releasing**: -Use Helm for managing Kubernetes package releases and GoReleaser for automating the release of Go applications. These tools streamline the packaging and distribution processes, making the CI/CD pipeline more robust and manageable. These are used in the `helm-release` and `manifest-release` jobs. ## What's next? From 4251158eea3f715b03f4c95061b044a33e3bded8 Mon Sep 17 00:00:00 2001 From: AYUSH SAHU Date: Thu, 24 Oct 2024 22:36:28 +0530 Subject: [PATCH 3/5] Update CICD doc for registering and serialyzing Signed-off-by: AYUSH SAHU --- docs/user_guide/flyte_fundamentals/registering_workflows.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user_guide/flyte_fundamentals/registering_workflows.md b/docs/user_guide/flyte_fundamentals/registering_workflows.md index d1ef9d71ea..bbd0bf8e92 100644 --- a/docs/user_guide/flyte_fundamentals/registering_workflows.md +++ b/docs/user_guide/flyte_fundamentals/registering_workflows.md @@ -373,7 +373,7 @@ In case Flyte workflows are registered on each commit in your build pipelines, y - **Container Image Management** : With Docker or any other CRI-compliant engine, build and push images to maintain traceability and consistency across environments. -- **Workflow Serialization and Registration** : Workflows should be serialized and registered based on the versioning of the build and the container image. Depending on whether the build is for a feature branch or main, the registration domain should be adjusted accordingly. +- **Workflow Serialization and Registration** : Workflows should be serialized and registered based on the versioning of the build and the container image. Depending on whether the build is for a feature branch or main, the registration domain should be adjusted accordingly. For more context, please visit the [Registering workflows](https://docs.flyte.org/en/latest/user_guide/flyte_fundamentals/registering_workflows.html) page. - **Container Image Specification** : When managing multiple images across tasks within a Flyte workflow, use the `--image` flag during registration to specify which image to use. This avoids hardcoding the image within the task definition, promoting reusability and flexibility in workflows. From 0b6c00fe23ce986dd900ba2b42547f41d4df0950 Mon Sep 17 00:00:00 2001 From: AYUSH SAHU Date: Sat, 26 Oct 2024 17:09:16 +0530 Subject: [PATCH 4/5] removed the line about Go releaser Signed-off-by: AYUSH SAHU --- docs/user_guide/flyte_fundamentals/registering_workflows.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/user_guide/flyte_fundamentals/registering_workflows.md b/docs/user_guide/flyte_fundamentals/registering_workflows.md index bbd0bf8e92..372457cd87 100644 --- a/docs/user_guide/flyte_fundamentals/registering_workflows.md +++ b/docs/user_guide/flyte_fundamentals/registering_workflows.md @@ -377,8 +377,6 @@ In case Flyte workflows are registered on each commit in your build pipelines, y - **Container Image Specification** : When managing multiple images across tasks within a Flyte workflow, use the `--image` flag during registration to specify which image to use. This avoids hardcoding the image within the task definition, promoting reusability and flexibility in workflows. -- **Helm and GoReleaser for Packaging and Releasing**: Use Helm for managing Kubernetes package releases and GoReleaser for automating the release of Go applications. These tools streamline the packaging and distribution processes, making the CI/CD pipeline more robust and manageable. These are used in the `helm-release` and `manifest-release` jobs. - ## What's next? From d2df63ab7bb5ec1b4bed45956e2437dc27cd5f37 Mon Sep 17 00:00:00 2001 From: AYUSH SAHU Date: Sat, 26 Oct 2024 17:10:48 +0530 Subject: [PATCH 5/5] Removed some general point about containerization Signed-off-by: AYUSH SAHU --- docs/user_guide/flyte_fundamentals/registering_workflows.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/user_guide/flyte_fundamentals/registering_workflows.md b/docs/user_guide/flyte_fundamentals/registering_workflows.md index 372457cd87..15f6fb6874 100644 --- a/docs/user_guide/flyte_fundamentals/registering_workflows.md +++ b/docs/user_guide/flyte_fundamentals/registering_workflows.md @@ -371,8 +371,6 @@ In case Flyte workflows are registered on each commit in your build pipelines, y - **Versioning Strategy** : Determining the version of the build for different types of commits makes them consistent and identifiable. For commits on feature branches, use `-` and for the ones on main branches, use `main-`. Use version numbers for the released (tagged) versions. -- **Container Image Management** : With Docker or any other CRI-compliant engine, build and push images to maintain traceability and consistency across environments. - - **Workflow Serialization and Registration** : Workflows should be serialized and registered based on the versioning of the build and the container image. Depending on whether the build is for a feature branch or main, the registration domain should be adjusted accordingly. For more context, please visit the [Registering workflows](https://docs.flyte.org/en/latest/user_guide/flyte_fundamentals/registering_workflows.html) page. - **Container Image Specification** : When managing multiple images across tasks within a Flyte workflow, use the `--image` flag during registration to specify which image to use. This avoids hardcoding the image within the task definition, promoting reusability and flexibility in workflows.