From 5bc0a25620035dd5a9aa5545f412ebee913651b8 Mon Sep 17 00:00:00 2001 From: George MacRorie Date: Mon, 25 Nov 2024 16:52:08 +0000 Subject: [PATCH 1/5] feat(docs/guides): add gitops implementation guide --- docs/guides/_sidebar.md | 3 + docs/guides/gitops-implementation.md | 453 +++++++++++++++++++++++++++ docs/guides/gitops-overview.md | 136 ++++++++ docs/guides/index.md | 8 +- 4 files changed, 599 insertions(+), 1 deletion(-) create mode 100644 docs/guides/_sidebar.md create mode 100644 docs/guides/gitops-implementation.md create mode 100644 docs/guides/gitops-overview.md diff --git a/docs/guides/_sidebar.md b/docs/guides/_sidebar.md new file mode 100644 index 0000000..24a3a58 --- /dev/null +++ b/docs/guides/_sidebar.md @@ -0,0 +1,3 @@ +* Glu and GitOps + * [Overview](/guides/gitops-overview.md) + * [Implementation](/guides/gitops-implementation.md) diff --git a/docs/guides/gitops-implementation.md b/docs/guides/gitops-implementation.md new file mode 100644 index 0000000..aec833c --- /dev/null +++ b/docs/guides/gitops-implementation.md @@ -0,0 +1,453 @@ +GitOps and Glu: Implementation +============================== + +This guide follows on from the previous [GitOps and Glu: Overview](/guides/gitops-overview.md) guide. + +In this guide we will walkthrough the actual implementation (in Go) of the pipeline found in the [GitOps Example Repository](https://github.com/get-glu/gitops-example). + +The pipeline code in particular is rooted in this directory: https://github.com/get-glu/gitops-example/tree/main/cmd/pipeline. + +## The Goal + +The goal of this codebase is to model and execute a promotion pipeline. +Whenever a new version of our application is pushed and tagged as `latest` in some source OCI repository we want to update our configuration Git repository, so that FluxCD can deploy it to our _staging_ environment. +Additionally, when a user decides the application in production is ready, we want them to be able to promote it to the _production_ environment. + +Our example repository, acts as the source of truth for FluxCD to apply to the target environments. +It contains the manifests which our pipeline is going to update for us. +The different `staging` and `production` directories in the `env` folder is where you will find these managed manifests: + +```yaml +env +├── production +│   ├── deployment.yaml # our pipeline will update this based on staging +│   └── service.yaml +└── staging + ├── deployment.yaml # our pipeline will update this based on a source OCI repository + └── service.yaml +``` + +Glu doesn't perform deployments directly (we're using FluxCD for that in this situation). +However, it instead keeps changes flowing in an orderely fashion from OCI through Git. + +In Glu, we will implement a _system_, with a single release _pipeline_, consisting of three phases _oci_ (to mode the source of new versions), _staging_ and _production_. + +At the end, we will add a trigger for the staging phase to attempt promotions based on a schedule. + +## The System + +Every Glu codebase starts with a `glu.System`. A system is a container for your pipelines, and entrypoint for command line interactions and starting the built-in server, as well a scheduler for running promotions based on triggers. + +In the GitOps example you will find a `main.go`. In this you will find `main()` function, which calls a function `run(ctx) error`. + +This `run()` function is where we first get introduced to our new glu system instance. + +```go +func run(ctx context.Context) error { + return glu.NewSystem(ctx, glu.Name("gitops-example"), glu.WithUI(ui.FS())). + // ... +``` + +A glu system needs some metadata (a container for a name and optional labels/annotations), as well as some optional extras. +`glu.Name` is a handy utility for creating an instance of `glu.Metadata` with the require field `name` set to the first argument passed. +We happen to want the UI, which is distributed as its own separate Go module: + +```go +// go get github.com/get-glu/glu/ui@latest +import "github.com/get-glu/glu/ui" +``` + +> Keeping it as a separate module means that you don't have to include all the assets in your resulting pipeline to use Glu. +> The UI module bundles in a pre-built React/Typescript app as an instance of Go's `fs.FS`. + +The `System` type exposes a few functions for adding new pipelines (`AddPipeline`), declaring triggers (`AddTrigger`) and running the entire system (`Run`). + +## The Pipeline + +In this example, we have a single pipeline we have chosen to call `gitops-example-app`. + +> We tend to model our pipelines around the particular applications they deliver. + +To add a new pipeline, we call `AddPipeline()`, which takes a function that returns a Pipeline for some provided configuration: + +```go +system.AddPipeline(func(ctx context.Context, config *glu.Config) (glu.Pipeline, error) { + // ... +}) +``` + +It is up to the implementer to build a pipeline using the provided configuration. +The returned pipeline from this function will be registered on the system. +In this function, we're expected to define our pipeline and all its child phases and their promotion dependencies on one another. + +The provided configuration has lots of useful conventions baked into it for the built-in phase types. +Our pipelines phases will interaction with both OCI and Git sources of truth to make everything work. + +### Definition + +```go +pipeline := glu.NewPipeline(glu.Name("gitops-example-app"), func() *AppResource { + return &AppResource{ + Image: "ghcr.io/get-glu/gitops-example/app", + } +}) +``` + +As with our system, pipelines require some metadata. +Notice this type called `*AppResource`. We won't get into this right now (we will learn about this in [The Resource](#the-resource) section), however, this type is intrinsic for defining _what_ flows through our pipeline and _how_ it is represented in our target **sources**. + +Before we can define our phases, each phase will likely need to source their state from somewhere (e.g. OCI or Git). +We can use the config argument passed to the builder function to make this process easier. + +### Config: OCI + +```go +// fetch the configured OCI repositority source named "app" +ociRepo, err := config.OCIRepository("app") +if err != nil { + return nil, err +} + +ociSource := oci.New[*AppResource](ociRepo) +``` + +The `OCIRepository` method fetches a pre-configured OCI repository client. +This client implementation is intended for the `github.com/get-glu/glu/pkg/src/oci.Source` implementation. +Notice we provide the name `"app"` when creating the repository. + +> Notice again the type `*AppResource` supplied as a type argument this time. +> Ignore this for now, we will come to that later on in this guide. + +Remember, Glu brings some conventions around configuration. +You can now provided OCI specific configuration for this source repository by using the same name `"app"` in a `glu.yaml` (or this can alternatively be supplied via environment variables). + +```yaml +sources: + oci: + app: # this is where our config.OCIRepository("app") argument comes in + reference: "ghcr.io/get-glu/gitops-example/app:latest" + credential: "github" + +credentials: + github: + type: "basic" + basic: + username: "glu" + password: "password" +``` + +### Config: Git + +```go +// fetch the configured Git repository source named "gitopsexample" +gitRepo, gitProposer, err := config.GitRepository(ctx, "gitopsexample") +if err != nil { + return nil, err +} + +gitSource := git.NewSource[*AppResource](gitRepo, gitProposer) +``` + +As with OCI, Git has a similar convenience function for getting a pre-configured Git repository. +Here, we ask configuration for a git repository with the name `"gitopsexample"`. + +In its current form, this returns both a repository and proposer. +We can ignore the proposer as an implementation detail for now. +However, for future reference, the proposer is so that we can support opening pull or merge requests in target SCMs (e.g. GitHub). + +Again, as with the OCI source, we can now configure this named repository in our `glu.yaml` or via environment variables. + +```yaml +sources: + git: + gitopsexample: # this is where our config.GitRepository("gitopsexample") argument comes in + remote: + name: "origin" + url: "https://github.com/get-glu/gitops-example.git" + credential: "github" + +credentials: + github: + type: "basic" + basic: + username: "glu" + password: "password" +``` + +## The Phases + +Now that we have our pipeline named and our sources configured, we can begin defining our phases and their promotion dependencies. + +Remember, we said that we plan to define three phases: + +- OCI (the source phase) +- Staging (represented as configuration in our Git repository) +- Production (represented as configuration in our Git repository) + +### OCI + +```go +// build a phase which sources from the OCI repository +ociPhase, err := phases.New(glu.Name("oci"), pipeline, ociSource) +if err != nil { + return nil, err +} +``` + +The initial phase is nice and simple. We're going to use the `phases` package to make a new phase using our pipeline and our source. +As with our pipeline and system, we need to give it some metadata (name and optional labels/annotations). + +### Staging (Git) + +```go +// build a phase for the staging environment which source from the git repository +// configure it to promote from the OCI phase +stagingPhase, err := phases.New(glu.Name("staging", glu.Label("url", "http://0.0.0.0:30081")), + pipeline, gitSource, core.PromotesFrom(ociPhase)) +if err != nil { + return nil, err +} +``` + +Again, here we give the phase some metadata (this time with a helpful label) and pass additional dependencies (pipeline and git source). +However, we also now pass a new option `core.PromotesFrom(ociPhase)`. +This particular option creates a _promotion_ dependency to the _staging_ phase from the OCI _phase_. +In other words, we make it so that you can promote from _oci_ to _staging_. + +This is how we create the promotions paths from one phase to the next in Glu. + +### Production (Git) + +```go +// build a phase for the production environment which source from the git repository +// configure it to promote from the staging git phase +_, err = phases.New(glu.Name("production", glu.Label("url", "http://0.0.0.0:30082")), + pipeline, gitSource, core.PromotesFrom(stagingPhase)) +if err != nil { + return nil, err +} +``` + +Finally, we describe our _production_ phase. As with staging, we pass metadata, pipeline, the git source and this time we add a promotion relationship to the `stagingPhase`. This means we can promote from _staging_ to _production_. + +Now we have described out entire end to end _phase_. +However, it is crucial to now understand more about the `*AppResource`. + +## The Resource + +Remember that `*AppResource` type that has been floating around? + +This particular piece of the puzzle has two goals in the system: + +1. What are we promoting? + +2. How is it represented in _each_ source? + +### The What + +```go +// AppResource is a custom envelope for carrying our specific repository configuration +// from one source to the next in our pipeline. +type AppResource struct { + Image string + ImageDigest string +} + +// Digest is a core required function for implementing glu.Resource. +// It should return a unique digest for the state of the resource. +// In this instance we happen to be reading a unique digest from the source +// and so we can lean into that. +// This will be used for comparisons in the phase to decided whether or not +// a change has occurred when deciding if to update the target source. +func (c *AppResource) Digest() (string, error) { + return c.ImageDigest, nil +} +``` + +In our particular GitOps repository, the _what_ is OCI image digests. +We're interested in updating an applications (bundled into an OCI repositories) version across different target environments. + +By defining a type which implements the `core.Resource` interface, we can use it in our pipeline. + +```go +// Resource is an instance of a resource in a phase. +// Primarilly, it exposes a Digest method used to produce +// a hash digest of the resource instances current state. +type Resource interface { + Digest() (string, error) +} +``` + +A resource (currently) only needs a single method `Digest()`. +It is up to the implementer to return a string, which is a content digest of the resource itself. +In our example, we return the actual image digest field, as this is the unique digest we're using to make promotion decision with. + +> This is used for comparision when making promotion decisions. +> If two instances of your resources (i.e. the version in oci, compared with the version in staging) differ, then a promotion will take place. + +### The How + +This part is important, and the functions you need to implement are depend on the sources you're using. +Whenever you attempt to integrate a source into a phase for a given resource type, the source will add further compile constraints. + +#### oci.Source[Resource] + +```go +type Resource interface { + core.Resource + ReadFromOCIDescriptor(v1.Descriptor) error +} +``` + +The OCI source (currently a read-only phase source) requires a single method `ReadFromOCIDescriptor(...)`. +By implementing this method, you can combine your resource type and this source type into a phase for your pipeline. +The method should extract any necessary details onto your type structure. +These details should be the ones that change and are copied between phases. + +```go +// ReadFromOCIDescriptor is an OCI specific resource requirement. +// Its purpose is to read the resources state from a target OCI metadata descriptor. +// Here we're reading out the images digest from the metadata. +func (c *AppResource) ReadFromOCIDescriptor(d v1.Descriptor) error { + c.ImageDigest = d.Digest.String() + return nil +} +``` + +In our example, we're interested in moving the OCI image digest from one phase to the next. +So, we extract that from the image descriptor provided to this function and set it on the field we defined on our type. + +#### git.Source[Resource] + +```go +type Resource interface { + core.Resource + ReadFrom(context.Context, core.Metadata, fs.Filesystem) error + WriteTo(context.Context, core.Metadata, fs.Filesystem) error +} +``` + +The Git source requires a resource to be readable from and writeable to a target filesystem. +This source is particularly special, as it takes care of details such as checking checking out branches, staging changes, creating commits, opening pull requests and so on. +Instead, all it requires the implementer to do, is explain how to read and write the definition to a target repositories root tree. +The source then takes care of the rest of the contribution lifecycle. + +> There are further ways to configure the resulting commit message, PR title and body via other methods on your type. + +**GitOps Example: Reading from filesystem** + +```go +func (r *AppResource) ReadFrom(_ context.Context, meta core.Metadata, fs fs.Filesystem) error { + deployment, err := readDeployment(fs, fmt.Sprintf("env/%s/deployment.yaml", meta.Name)) + if err != nil { + return err + } + + if containers := deployment.Spec.Template.Spec.Containers; len(containers) > 0 { + ref, err := registry.ParseReference(containers[0].Image) + if err != nil { + return err + } + + digest, err := ref.Digest() + if err != nil { + return err + } + + r.ImageDigest = digest.String() + } + + return nil +} +``` + +Here we see that we read a file at a particular path: `fmt.Sprintf("env/%s/deployment.yaml", meta.Name)` + +The metadata supplied by Glu here happens to be the phases metadata. +This is how we can read different paths, dependent on the phase being read or written to. + +This particular implementations reads the file as a Kubernetes deployment encoded as YAML. +It then extracts the containers image reference directly from the pod spec. + +The resulting image digest is again set on the receiving resource type `c.ImageDigest = digest.String()`. + +**GitOps Example: Writing to filesystem** + +```go +func (r *AppResource) WriteTo(ctx context.Context, meta glu.Metadata, fs fs.Filesystem) error { + path := fmt.Sprintf("env/%s/deployment.yaml", meta.Name) + deployment, err := readDeployment(fs, path) + if err != nil { + return err + } + + // locate the target container + if containers := deployment.Spec.Template.Spec.Containers; len(containers) > 0 { + // update containers image @ + containers[0].Image = fmt.Sprintf("%s@%s", c.Image, c.ImageDigest) + + // update an informative environment variable with the digest too + for i := range containers[0].Env { + if containers[0].Env[i].Name == "APP_IMAGE_DIGEST" { + containers[0].Env[i].Value = r.ImageDigest + } + } + } + + // re-open the file for writing + fi, err := fs.OpenFile( + path, + os.O_WRONLY|os.O_TRUNC, + 0644, + ) + if err != nil { + return err + } + + defer fi.Close() + + // re-encode the deployment and copy it into the file + data, err := yaml.Marshal(deployment) + if err != nil { + return err + } + + _, err = io.Copy(fi, bytes.NewReader(data)) + return err +} +``` + +When it comes to writing, again we look to the file in the path dictated by metadata from the phase. + +Here, we're taking the image digest from our receiving `r *AppResource` and setting it on the target container in our deployment pod spec. +Finally, we're rewriting our target file with the new updated contents of our deployment. + +## The Triggers + +## Now Run + +Finally, we can now run our pipeline. + +```go +system.Run() +``` + +This is usually the last function call in a Glu system binary. +It takes care of setting up signal traps and propagating terminations through context cancellation to the various components. + +Additionally, if you have configured any triggers and your invoking your glu pipeline as a server binary, then these will be enabled for the duration of the process. + +## Recap + +OK, that was a lot. I appreciate you taking the time to walkthrough this journey! + +We have: + +- Created a glu system to orchestrate our promotion pipelines with +- Configured two sources (OCI and Git) for reading and writing to +- Declared three phases to promote change between +- Defined a resource type for carrying and our promotion material through the pipeline + +The byproduct of doing all this, is that we get an instant API for introspecting and manually promote changes through pipelines with. +All changes flow in and out of our disperate sources, via whichever encoding formats / configuration langauges and filesystem layours we choose. +Additionally, we can add a dashboard interface for humans to read and interact with, in order to trigger manual promotions. diff --git a/docs/guides/gitops-overview.md b/docs/guides/gitops-overview.md new file mode 100644 index 0000000..cd11bb5 --- /dev/null +++ b/docs/guides/gitops-overview.md @@ -0,0 +1,136 @@ +Glu and GitOps: Overview +======================== + +In this guide, we will create a GitOps pipeline for deploying some simple applications to multiple "environments". +For the purposes of keeping the illustration simple to create (and destroy), our environments are simply two separate pods in the same cluster. However, Glu can extend to multiple namespaces, clusters, or even non-Kubernetes deployment environments. + +In this guide, we will: + +- Deploy a bunch of applications and CD components to a Kubernetes-in-Docker cluster on our local machine. +- Visit our applications in their different "environments" to see what is deployed. +- Look at the Glu pipeline UI to see the latest version of our application, as well as the versions deployed to each environment. +- Trigger a promotion in the UI to update one target environment, to version it is configured to promote from. +- Observe that our promoted environment has updated to the new version. + +All the content in this guide works around our [GitOps Example Repository](https://github.com/get-glu/gitops-example). +You can follow the README there, however, we will go over the steps again here in this guide. + +The repository contains a script, which will: + +- Spin up a Kind cluster to run all of our components. +- Deploy FluxCD into the cluster to perform CD for our environments. +- Add two Flux Git sources pointed at two separate "environment" folders (`staging` and `production`). +- Flux will then deploy our demo application into both of these "environments". +- Deploy a Glu-implemented pipeline for visualizing and progressing promotions across our environments. + +### Requirements + +- [Docker](https://www.docker.com/) +- [Kind](https://kind.sigs.k8s.io/) +- [Go](https://go.dev/) + +> The start script below will also install [Timoni](https://timoni.sh/) (using `go install` to do so). +> This is used to configure our Kind cluster. +> Big love to Timoni :heart:. + +### Running + +Before you get started you're going to want to do the following: + +1. Fork [this repo](https://github.com/get-glu/gitops-example)! +2. Clone your fork locally. +3. Make a note of your fork's GitHub URL (likely `https://github.com/{your_username}/gitops-example`). +4. Generate a [GitHub Personal Access Token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens) (if you want to experiment with promotions). + +> You will need at least read and write contents scope (`contents:write`). + +Once you have done the above, you can run the start script found in the root of your local fork. +The script will prompt you for your fork's repository URL and access token (given you want to perform promotions). + +```console +./start.sh + +Enter your target gitops repo URL [default: https://github.com/get-glu/gitops-example]: +Enter your GitHub personal access token [default: (read-only pipeline)]: +Creating cluster... +``` + +This process can take a few minutes in total, sit tight! + +Once finished, you will be presented with a link to take you to the Glu UI. + +```console +########################################## +# # +# Pipeline Ready: http://localhost:30080 # +# # +########################################## +``` + +Following the link, you should see a view similar to the one in the screenshot below. +From here, we need to select a pipeline to get started. + +Dashboard Welcome Screen + +Head to the pipeline drop down at the top-left of the dashboard. +Then select the `gitops-example-app` pipeline. + +Pipeline selection dropdown + +You will notice there are three phase "oci", "staging" and "production". + +Pipeline dashboard view + +If we click on a phase (for example, "staging") we can see an expanded view of the phases details in a pane below. + +Pipeline staging phase expanded view + +Notice that the "staging" and "production" phases have a `url` label on them. +These contain links to localhost ports that are exposed on our kind cluster. +You can follow these links to see the "staging" and "production" deployed applications. + +Click on the URL for the "staging" phase (it should be [https://localhost:30081](https://localhost:30081)). +The page should open in a new tab and show a small welcome screen with some details regarding the deployed application. +Notice we can see details regarding the built version of the application (Git and OCI digests). + +Staging application landing page + +Navigating back to the Glu pipeline view, we will now visit our "production" application. + +Pipeline view with production url underlined + +Click on the URL for the "production" phase (it should be [https://localhost:30082](https://localhost:30082)). + +Production application landing page + +This page is similar to our staging application phase. +It contains all the same information, but it lacks all the styles. +This is because "staging" is currently ahead of "production". + +We're going to now trigger a promotion. +Head back to the glu dashboard and click on the up arrow in the "production" phase node. + +Promotion dialogue + +You will be presented with a modal, with a cancel and promote button. +Click on `Promote` to trigger a promotion from "staging" to "production". + +> Congratulations, a promotion is in progress! + +This will have triggered a promotion to take place. +In this particular example, this will have updated a manifest in our forked Git repository. +Try navigating in your browser to your forked repository in GitHub. +You should see a new commit labelled `Update production`. +Clicking on this, you should see a git patch similar to the following. + +Git patch to production deployment manifest + +Eventually, once FluxCD has caught up with a new revision you should see it take effect in the [production application](https://localhost:30082). + +Git patch to production deployment manifest + +### Next Steps + +This guide has walked through an interactive example of a pipeline implemented in Glu. + +In the [next guide](/guides/gitops-implementation?id=gitops-and-glu-implementation), we will look into how this particular pipeline is implemented in Go, using the Glu framework. diff --git a/docs/guides/index.md b/docs/guides/index.md index b77f7a1..b3463b6 100644 --- a/docs/guides/index.md +++ b/docs/guides/index.md @@ -1,7 +1,9 @@ -# Guides +Guides +====== This section contains guides for exploring pre-built Glu pipelines in action. +<<<<<<< Updated upstream ## Glu Orchestrated GitOps Pipeline In this guide, we will create a GitOps pipeline for deploying some simple applications to multiple "environments". @@ -138,3 +140,7 @@ This guide has walked through an interactive example of a pipeline implemented i In the next guide, we will look into how this particular pipeline is implemented in Go, using the Glu framework. TODO(georgemac): Write the guide and link it here. +======= +1. [Glu and GitOps: Overview](/guides/gitops-overview.md): Explore our example pipeline +2. [Glu and GitOps: Implementation](/guides/gitops-implementation.md): A dive into the code behind the pipeline +>>>>>>> Stashed changes From ca9a895a79c8538f6dff3501b3a0220777679e6c Mon Sep 17 00:00:00 2001 From: George MacRorie Date: Mon, 25 Nov 2024 16:56:32 +0000 Subject: [PATCH 2/5] chore(docs/guides): talk about adding triggers in gitops implementation --- docs/guides/gitops-implementation.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/docs/guides/gitops-implementation.md b/docs/guides/gitops-implementation.md index aec833c..1eeea7f 100644 --- a/docs/guides/gitops-implementation.md +++ b/docs/guides/gitops-implementation.md @@ -424,6 +424,23 @@ Finally, we're rewriting our target file with the new updated contents of our de ## The Triggers +Triggers are used to automate promotions based on some event or schedule. +Currently, scheduled triggers are the only prebuilt triggers available (we intend to add more, e.g. on oci or git push). + +```go +system.AddTrigger( + schedule.New( + schedule.WithInterval(10*time.Second), + schedule.MatchesLabel("env", "staging"), + // alternatively, the phase instance can be target directly with: + // glu.ScheduleMatchesPhase(stagingPhase), + ), +) +``` + +Here, we configure the system to attempt a promotion on any phase with particular label pair (`"env" == "staging"`) every `10s`. +Remember, a phase will only perform a real promotion if the resource derived from the two source differs (based on comparing the result of `Digest()`). + ## Now Run Finally, we can now run our pipeline. From d14ec4ac9c00a19ad98b6f49d5be4bfa3eb99f1d Mon Sep 17 00:00:00 2001 From: George MacRorie Date: Mon, 25 Nov 2024 17:02:55 +0000 Subject: [PATCH 3/5] fix(docs/guides): bad stash and patch --- docs/guides/gitops-overview.md | 15 ++-- docs/guides/index.md | 139 --------------------------------- 2 files changed, 8 insertions(+), 146 deletions(-) diff --git a/docs/guides/gitops-overview.md b/docs/guides/gitops-overview.md index cd11bb5..bfd212b 100644 --- a/docs/guides/gitops-overview.md +++ b/docs/guides/gitops-overview.md @@ -1,4 +1,4 @@ -Glu and GitOps: Overview +GitOps and Glu: Overview ======================== In this guide, we will create a GitOps pipeline for deploying some simple applications to multiple "environments". @@ -9,7 +9,7 @@ In this guide, we will: - Deploy a bunch of applications and CD components to a Kubernetes-in-Docker cluster on our local machine. - Visit our applications in their different "environments" to see what is deployed. - Look at the Glu pipeline UI to see the latest version of our application, as well as the versions deployed to each environment. -- Trigger a promotion in the UI to update one target environment, to version it is configured to promote from. +- Trigger a promotion in the UI to update one target environment, to the version it is configured to promote from. - Observe that our promoted environment has updated to the new version. All the content in this guide works around our [GitOps Example Repository](https://github.com/get-glu/gitops-example). @@ -21,7 +21,7 @@ The repository contains a script, which will: - Deploy FluxCD into the cluster to perform CD for our environments. - Add two Flux Git sources pointed at two separate "environment" folders (`staging` and `production`). - Flux will then deploy our demo application into both of these "environments". -- Deploy a Glu-implemented pipeline for visualizing and progressing promotions across our environments. +- Deploy a Glu-implemented pipeline to visualize and progress promotions across our environments. ### Requirements @@ -72,12 +72,12 @@ From here, we need to select a pipeline to get started. Dashboard Welcome Screen -Head to the pipeline drop down at the top-left of the dashboard. +Head to the pipeline dropdown at the top-left of the dashboard. Then select the `gitops-example-app` pipeline. Pipeline selection dropdown -You will notice there are three phase "oci", "staging" and "production". +You will notice there are three phases "oci", "staging" and "production". Pipeline dashboard view @@ -120,7 +120,7 @@ Click on `Promote` to trigger a promotion from "staging" to "production". This will have triggered a promotion to take place. In this particular example, this will have updated a manifest in our forked Git repository. Try navigating in your browser to your forked repository in GitHub. -You should see a new commit labelled `Update production`. +You should see a new commit labeled `Update production`. Clicking on this, you should see a git patch similar to the following. Git patch to production deployment manifest @@ -132,5 +132,6 @@ Eventually, once FluxCD has caught up with a new revision you should see it take ### Next Steps This guide has walked through an interactive example of a pipeline implemented in Glu. +In the next guide, we will look into how this particular pipeline is implemented in Go, using the Glu framework. -In the [next guide](/guides/gitops-implementation?id=gitops-and-glu-implementation), we will look into how this particular pipeline is implemented in Go, using the Glu framework. +TODO(georgemac): Write the guide and link it here. diff --git a/docs/guides/index.md b/docs/guides/index.md index b3463b6..c3fe852 100644 --- a/docs/guides/index.md +++ b/docs/guides/index.md @@ -3,144 +3,5 @@ Guides This section contains guides for exploring pre-built Glu pipelines in action. -<<<<<<< Updated upstream -## Glu Orchestrated GitOps Pipeline - -In this guide, we will create a GitOps pipeline for deploying some simple applications to multiple "environments". -For the purposes of keeping the illustration simple to create (and destroy), our environments are simply two separate pods in the same cluster. However, Glu can extend to multiple namespaces, clusters, or even non-Kubernetes deployment environments. - -In this guide, we will: - -- Deploy a bunch of applications and CD components to a Kubernetes-in-Docker cluster on our local machine. -- Visit our applications in their different "environments" to see what is deployed. -- Look at the Glu pipeline UI to see the latest version of our application, as well as the versions deployed to each environment. -- Trigger a promotion in the UI to update one target environment, to the version it is configured to promote from. -- Observe that our promoted environment has updated to the new version. - -All the content in this guide works around our [GitOps Example Repository](https://github.com/get-glu/gitops-example). -You can follow the README there, however, we will go over the steps again here in this guide. - -The repository contains a script, which will: - -- Spin up a Kind cluster to run all of our components. -- Deploy FluxCD into the cluster to perform CD for our environments. -- Add two Flux Git sources pointed at two separate "environment" folders (`staging` and `production`). -- Flux will then deploy our demo application into both of these "environments". -- Deploy a Glu-implemented pipeline to visualize and progress promotions across our environments. - -### Requirements - -- [Docker](https://www.docker.com/) -- [Kind](https://kind.sigs.k8s.io/) -- [Go](https://go.dev/) - -> The start script below will also install [Timoni](https://timoni.sh/) (using `go install` to do so). -> This is used to configure our Kind cluster. -> Big love to Timoni :heart:. - -### Running - -Before you get started you're going to want to do the following: - -1. Fork [this repo](https://github.com/get-glu/gitops-example)! -2. Clone your fork locally. -3. Make a note of your fork's GitHub URL (likely `https://github.com/{your_username}/gitops-example`). -4. Generate a [GitHub Personal Access Token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens) (if you want to experiment with promotions). - -> You will need at least read and write contents scope (`contents:write`). - -Once you have done the above, you can run the start script found in the root of your local fork. -The script will prompt you for your fork's repository URL and access token (given you want to perform promotions). - -```console -./start.sh - -Enter your target gitops repo URL [default: https://github.com/get-glu/gitops-example]: -Enter your GitHub personal access token [default: (read-only pipeline)]: -Creating cluster... -``` - -This process can take a few minutes in total, sit tight! - -Once finished, you will be presented with a link to take you to the Glu UI. - -```console -########################################## -# # -# Pipeline Ready: http://localhost:30080 # -# # -########################################## -``` - -Following the link, you should see a view similar to the one in the screenshot below. -From here, we need to select a pipeline to get started. - -Dashboard Welcome Screen - -Head to the pipeline dropdown at the top-left of the dashboard. -Then select the `gitops-example-app` pipeline. - -Pipeline selection dropdown - -You will notice there are three phases "oci", "staging" and "production". - -Pipeline dashboard view - -If we click on a phase (for example, "staging") we can see an expanded view of the phases details in a pane below. - -Pipeline staging phase expanded view - -Notice that the "staging" and "production" phases have a `url` label on them. -These contain links to localhost ports that are exposed on our kind cluster. -You can follow these links to see the "staging" and "production" deployed applications. - -Click on the URL for the "staging" phase (it should be [https://localhost:30081](https://localhost:30081)). -The page should open in a new tab and show a small welcome screen with some details regarding the deployed application. -Notice we can see details regarding the built version of the application (Git and OCI digests). - -Staging application landing page - -Navigating back to the Glu pipeline view, we will now visit our "production" application. - -Pipeline view with production url underlined - -Click on the URL for the "production" phase (it should be [https://localhost:30082](https://localhost:30082)). - -Production application landing page - -This page is similar to our staging application phase. -It contains all the same information, but it lacks all the styles. -This is because "staging" is currently ahead of "production". - -We're going to now trigger a promotion. -Head back to the glu dashboard and click on the up arrow in the "production" phase node. - -Promotion dialogue - -You will be presented with a modal, with a cancel and promote button. -Click on `Promote` to trigger a promotion from "staging" to "production". - -> Congratulations, a promotion is in progress! - -This will have triggered a promotion to take place. -In this particular example, this will have updated a manifest in our forked Git repository. -Try navigating in your browser to your forked repository in GitHub. -You should see a new commit labeled `Update production`. -Clicking on this, you should see a git patch similar to the following. - -Git patch to production deployment manifest - -Eventually, once FluxCD has caught up with a new revision you should see it take effect in the [production application](https://localhost:30082). - -Git patch to production deployment manifest - -### Next Steps - -This guide has walked through an interactive example of a pipeline implemented in Glu. -In the next guide, we will look into how this particular pipeline is implemented in Go, using the Glu framework. - -TODO(georgemac): Write the guide and link it here. -======= 1. [Glu and GitOps: Overview](/guides/gitops-overview.md): Explore our example pipeline 2. [Glu and GitOps: Implementation](/guides/gitops-implementation.md): A dive into the code behind the pipeline ->>>>>>> Stashed changes From 59844472070400e8da65bcc777d76b7704d6e67c Mon Sep 17 00:00:00 2001 From: Mark Phelps <209477+markphelps@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:13:06 -0500 Subject: [PATCH 4/5] chore: Update gitops-implementation.md --- docs/guides/gitops-implementation.md | 82 ++++++++++++++-------------- 1 file changed, 41 insertions(+), 41 deletions(-) diff --git a/docs/guides/gitops-implementation.md b/docs/guides/gitops-implementation.md index 1eeea7f..8856e92 100644 --- a/docs/guides/gitops-implementation.md +++ b/docs/guides/gitops-implementation.md @@ -3,7 +3,7 @@ GitOps and Glu: Implementation This guide follows on from the previous [GitOps and Glu: Overview](/guides/gitops-overview.md) guide. -In this guide we will walkthrough the actual implementation (in Go) of the pipeline found in the [GitOps Example Repository](https://github.com/get-glu/gitops-example). +In this guide, we will walkthrough the actual implementation (in Go) of the pipeline found in the [GitOps Example Repository](https://github.com/get-glu/gitops-example). The pipeline code in particular is rooted in this directory: https://github.com/get-glu/gitops-example/tree/main/cmd/pipeline. @@ -13,9 +13,9 @@ The goal of this codebase is to model and execute a promotion pipeline. Whenever a new version of our application is pushed and tagged as `latest` in some source OCI repository we want to update our configuration Git repository, so that FluxCD can deploy it to our _staging_ environment. Additionally, when a user decides the application in production is ready, we want them to be able to promote it to the _production_ environment. -Our example repository, acts as the source of truth for FluxCD to apply to the target environments. -It contains the manifests which our pipeline is going to update for us. -The different `staging` and `production` directories in the `env` folder is where you will find these managed manifests: +Our example repository acts as the source of truth for FluxCD to apply to the target environments. +It contains the manifests that our pipeline is going to update for us. +The different `staging` and `production` directories in the `env` folder are where you will find these managed manifests: ```yaml env @@ -28,7 +28,7 @@ env ``` Glu doesn't perform deployments directly (we're using FluxCD for that in this situation). -However, it instead keeps changes flowing in an orderely fashion from OCI through Git. +However, it instead keeps changes flowing in an orderly fashion from OCI through Git. In Glu, we will implement a _system_, with a single release _pipeline_, consisting of three phases _oci_ (to mode the source of new versions), _staging_ and _production_. @@ -36,9 +36,9 @@ At the end, we will add a trigger for the staging phase to attempt promotions ba ## The System -Every Glu codebase starts with a `glu.System`. A system is a container for your pipelines, and entrypoint for command line interactions and starting the built-in server, as well a scheduler for running promotions based on triggers. +Every Glu codebase starts with a `glu.System`. A system is a container for your pipelines, and entrypoint for command line interactions and starting the built-in server, as well as a scheduler for running promotions based on triggers. -In the GitOps example you will find a `main.go`. In this you will find `main()` function, which calls a function `run(ctx) error`. +In the GitOps example, you will find a `main.go`. In this, you will find `main()` function, which calls a function `run(ctx) error`. This `run()` function is where we first get introduced to our new glu system instance. @@ -49,7 +49,7 @@ func run(ctx context.Context) error { ``` A glu system needs some metadata (a container for a name and optional labels/annotations), as well as some optional extras. -`glu.Name` is a handy utility for creating an instance of `glu.Metadata` with the require field `name` set to the first argument passed. +`glu.Name` is a handy utility for creating an instance of `glu.Metadata` with the required field `name` set to the first argument passed. We happen to want the UI, which is distributed as its own separate Go module: ```go @@ -60,7 +60,7 @@ import "github.com/get-glu/glu/ui" > Keeping it as a separate module means that you don't have to include all the assets in your resulting pipeline to use Glu. > The UI module bundles in a pre-built React/Typescript app as an instance of Go's `fs.FS`. -The `System` type exposes a few functions for adding new pipelines (`AddPipeline`), declaring triggers (`AddTrigger`) and running the entire system (`Run`). +The `System` type exposes a few functions for adding new pipelines (`AddPipeline`), declaring triggers (`AddTrigger`), and running the entire system (`Run`). ## The Pipeline @@ -81,7 +81,7 @@ The returned pipeline from this function will be registered on the system. In this function, we're expected to define our pipeline and all its child phases and their promotion dependencies on one another. The provided configuration has lots of useful conventions baked into it for the built-in phase types. -Our pipelines phases will interaction with both OCI and Git sources of truth to make everything work. +Our pipeline phases will interact with both OCI and Git sources of truth to make everything work. ### Definition @@ -96,13 +96,13 @@ pipeline := glu.NewPipeline(glu.Name("gitops-example-app"), func() *AppResource As with our system, pipelines require some metadata. Notice this type called `*AppResource`. We won't get into this right now (we will learn about this in [The Resource](#the-resource) section), however, this type is intrinsic for defining _what_ flows through our pipeline and _how_ it is represented in our target **sources**. -Before we can define our phases, each phase will likely need to source their state from somewhere (e.g. OCI or Git). +Before we can define our phases, each phase will likely need to source its state from somewhere (e.g. OCI or Git). We can use the config argument passed to the builder function to make this process easier. ### Config: OCI ```go -// fetch the configured OCI repositority source named "app" +// fetch the configured OCI repository source named "app" ociRepo, err := config.OCIRepository("app") if err != nil { return nil, err @@ -119,7 +119,7 @@ Notice we provide the name `"app"` when creating the repository. > Ignore this for now, we will come to that later on in this guide. Remember, Glu brings some conventions around configuration. -You can now provided OCI specific configuration for this source repository by using the same name `"app"` in a `glu.yaml` (or this can alternatively be supplied via environment variables). +You can now provide OCI-specific configuration for this source repository by using the same name `"app"` in a `glu.yaml` (or this can alternatively be supplied via environment variables). ```yaml sources: @@ -149,9 +149,9 @@ gitSource := git.NewSource[*AppResource](gitRepo, gitProposer) ``` As with OCI, Git has a similar convenience function for getting a pre-configured Git repository. -Here, we ask configuration for a git repository with the name `"gitopsexample"`. +Here, we ask for configuration for a git repository with the name `"gitopsexample"`. -In its current form, this returns both a repository and proposer. +In its current form, this returns both a repository and a proposer. We can ignore the proposer as an implementation detail for now. However, for future reference, the proposer is so that we can support opening pull or merge requests in target SCMs (e.g. GitHub). @@ -200,7 +200,7 @@ As with our pipeline and system, we need to give it some metadata (name and opti ### Staging (Git) ```go -// build a phase for the staging environment which source from the git repository +// build a phase for the staging environment which sources from the git repository // configure it to promote from the OCI phase stagingPhase, err := phases.New(glu.Name("staging", glu.Label("url", "http://0.0.0.0:30081")), pipeline, gitSource, core.PromotesFrom(ociPhase)) @@ -214,12 +214,12 @@ However, we also now pass a new option `core.PromotesFrom(ociPhase)`. This particular option creates a _promotion_ dependency to the _staging_ phase from the OCI _phase_. In other words, we make it so that you can promote from _oci_ to _staging_. -This is how we create the promotions paths from one phase to the next in Glu. +This is how we create the promotion paths from one phase to the next in Glu. ### Production (Git) ```go -// build a phase for the production environment which source from the git repository +// build a phase for the production environment which sources from the git repository // configure it to promote from the staging git phase _, err = phases.New(glu.Name("production", glu.Label("url", "http://0.0.0.0:30082")), pipeline, gitSource, core.PromotesFrom(stagingPhase)) @@ -230,7 +230,7 @@ if err != nil { Finally, we describe our _production_ phase. As with staging, we pass metadata, pipeline, the git source and this time we add a promotion relationship to the `stagingPhase`. This means we can promote from _staging_ to _production_. -Now we have described out entire end to end _phase_. +Now we have described our entire end-to-end _phase_. However, it is crucial to now understand more about the `*AppResource`. ## The Resource @@ -257,7 +257,7 @@ type AppResource struct { // It should return a unique digest for the state of the resource. // In this instance we happen to be reading a unique digest from the source // and so we can lean into that. -// This will be used for comparisons in the phase to decided whether or not +// This will be used for comparisons in the phase to decide whether or not // a change has occurred when deciding if to update the target source. func (c *AppResource) Digest() (string, error) { return c.ImageDigest, nil @@ -265,14 +265,14 @@ func (c *AppResource) Digest() (string, error) { ``` In our particular GitOps repository, the _what_ is OCI image digests. -We're interested in updating an applications (bundled into an OCI repositories) version across different target environments. +We're interested in updating an application (bundled into an OCI repository) version across different target environments. -By defining a type which implements the `core.Resource` interface, we can use it in our pipeline. +By defining a type that implements the `core.Resource` interface, we can use it in our pipeline. ```go // Resource is an instance of a resource in a phase. -// Primarilly, it exposes a Digest method used to produce -// a hash digest of the resource instances current state. +// Primarily, it exposes a Digest method used to produce +// a hash digest of the resource instances' current state. type Resource interface { Digest() (string, error) } @@ -282,12 +282,12 @@ A resource (currently) only needs a single method `Digest()`. It is up to the implementer to return a string, which is a content digest of the resource itself. In our example, we return the actual image digest field, as this is the unique digest we're using to make promotion decision with. -> This is used for comparision when making promotion decisions. -> If two instances of your resources (i.e. the version in oci, compared with the version in staging) differ, then a promotion will take place. +> This is used for comparison when making promotion decisions. +> If two instances of your resources (i.e. the version in OCI, compared with the version in staging) differ, then a promotion will take place. ### The How -This part is important, and the functions you need to implement are depend on the sources you're using. +This part is important, and the functions you need to implement depend on the sources you're using. Whenever you attempt to integrate a source into a phase for a given resource type, the source will add further compile constraints. #### oci.Source[Resource] @@ -305,7 +305,7 @@ The method should extract any necessary details onto your type structure. These details should be the ones that change and are copied between phases. ```go -// ReadFromOCIDescriptor is an OCI specific resource requirement. +// ReadFromOCIDescriptor is an OCI-specific resource requirement. // Its purpose is to read the resources state from a target OCI metadata descriptor. // Here we're reading out the images digest from the metadata. func (c *AppResource) ReadFromOCIDescriptor(d v1.Descriptor) error { @@ -328,11 +328,11 @@ type Resource interface { ``` The Git source requires a resource to be readable from and writeable to a target filesystem. -This source is particularly special, as it takes care of details such as checking checking out branches, staging changes, creating commits, opening pull requests and so on. -Instead, all it requires the implementer to do, is explain how to read and write the definition to a target repositories root tree. +This source is particularly special, as it takes care of details such as checking out branches, staging changes, creating commits, opening pull requests, and so on. +Instead, all it requires the implementer to do is explain how to read and write the definition to a target repository root tree. The source then takes care of the rest of the contribution lifecycle. -> There are further ways to configure the resulting commit message, PR title and body via other methods on your type. +> There are further ways to configure the resulting commit message, PR title, and body via other methods on your type. **GitOps Example: Reading from filesystem** @@ -366,8 +366,8 @@ Here we see that we read a file at a particular path: `fmt.Sprintf("env/%s/deplo The metadata supplied by Glu here happens to be the phases metadata. This is how we can read different paths, dependent on the phase being read or written to. -This particular implementations reads the file as a Kubernetes deployment encoded as YAML. -It then extracts the containers image reference directly from the pod spec. +This particular implementation reads the file as a Kubernetes deployment encoded as YAML. +It then extracts the container's image reference directly from the pod spec. The resulting image digest is again set on the receiving resource type `c.ImageDigest = digest.String()`. @@ -420,7 +420,7 @@ func (r *AppResource) WriteTo(ctx context.Context, meta glu.Metadata, fs fs.File When it comes to writing, again we look to the file in the path dictated by metadata from the phase. Here, we're taking the image digest from our receiving `r *AppResource` and setting it on the target container in our deployment pod spec. -Finally, we're rewriting our target file with the new updated contents of our deployment. +Finally, we're rewriting our target file with the newly updated contents of our deployment. ## The Triggers @@ -438,8 +438,8 @@ system.AddTrigger( ) ``` -Here, we configure the system to attempt a promotion on any phase with particular label pair (`"env" == "staging"`) every `10s`. -Remember, a phase will only perform a real promotion if the resource derived from the two source differs (based on comparing the result of `Digest()`). +Here, we configure the system to attempt a promotion on any phase with a particular label pair (`"env" == "staging"`) every `10s`. +Remember, a phase will only perform a real promotion if the resource derived from the two sources differs (based on comparing the result of `Digest()`). ## Now Run @@ -452,19 +452,19 @@ system.Run() This is usually the last function call in a Glu system binary. It takes care of setting up signal traps and propagating terminations through context cancellation to the various components. -Additionally, if you have configured any triggers and your invoking your glu pipeline as a server binary, then these will be enabled for the duration of the process. +Additionally, if you have configured any triggers and are invoking your glu pipeline as a server binary, then these will be enabled for the duration of the process. ## Recap -OK, that was a lot. I appreciate you taking the time to walkthrough this journey! +OK, that was a lot. I appreciate you taking the time to walk through this journey! We have: - Created a glu system to orchestrate our promotion pipelines with - Configured two sources (OCI and Git) for reading and writing to - Declared three phases to promote change between -- Defined a resource type for carrying and our promotion material through the pipeline +- Defined a resource type for carrying our promotion material through the pipeline -The byproduct of doing all this, is that we get an instant API for introspecting and manually promote changes through pipelines with. -All changes flow in and out of our disperate sources, via whichever encoding formats / configuration langauges and filesystem layours we choose. +The byproduct of doing all this is that we get an instant API for introspecting and manually promoting changes through pipelines. +All changes flow in and out of our disparate sources, via whichever encoding formats/configuration languages and filesystem layouts we choose. Additionally, we can add a dashboard interface for humans to read and interact with, in order to trigger manual promotions. From 962cec52675efb73b668040bbe6f5874765e7e5c Mon Sep 17 00:00:00 2001 From: George Date: Mon, 25 Nov 2024 17:19:35 +0000 Subject: [PATCH 5/5] chore: Update docs/guides/gitops-overview.md --- docs/guides/gitops-overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/gitops-overview.md b/docs/guides/gitops-overview.md index bfd212b..73c7dda 100644 --- a/docs/guides/gitops-overview.md +++ b/docs/guides/gitops-overview.md @@ -134,4 +134,4 @@ Eventually, once FluxCD has caught up with a new revision you should see it take This guide has walked through an interactive example of a pipeline implemented in Glu. In the next guide, we will look into how this particular pipeline is implemented in Go, using the Glu framework. -TODO(georgemac): Write the guide and link it here. +Next: [GitOps and Glu: Implementation](/guides/gitops-implementation)