Document end-to-end GitOps solutions

[balopat]

Document solutions that walk the user through best practices from dev, testing and production deployment using GitOps using Skaffold.

[leefernandes]

In particular I'm curious how:

• one uses skaffold to commit the built k8s manifests to a target repo or branch when using skaffold deploy. Is there a way to get the manifest build output from skaffold deploy similarly to kustomize build?

• how to add a tag for a promoted image going from one environment to another which incorporates the original tag. For example if your development image is tagged with the gitCommit, when promoted to staging how to add a tag such as staging-${gitCommit}

[nkubala]

Is there a way to get the manifest build output from skaffold deploy similarly to kustomize build?

currently the only way to see them is by running with -v DEBUG, but we could also accomplish this with a --dry-run flag. see #1739

For example if your development image is tagged with the gitCommit, when promoted to staging how to add a tag such as staging-${gitCommit}

right now in skaffold we don't support multiple tagging strategies, this has been discussed in #1269. maybe it's worth revisiting different solutions for this now.

not very pretty, but you can accomplish something similar with this right now using profiles (and a little unix foo):

build:
  tagPolicy:
    sha256: {}
...
profiles:
- name: staging
   build:
     tagPolicy:
       envTemplate:
         template: staging-{{.GIT_COMMIT}}

$ skaffold dev -p staging $GIT_COMMIT=$(git rev-parse --short HEAD)

[tbarrella]

I'm also pretty curious about this... maybe I should just be patient, but related to this, do you think Skaffold is meant to be a good building block for supporting using artifacts from one workflow in another? (Sorry if this isn't the best place to ask this, although I wasn't able to access the skaffold-users mailing list.)

• Workflow 1 builds image A; workflow 2 builds image B and deploys latest A and B: For deploying, this seems easy enough if workflow 2 uses skaffold deploy with --images or --build-artifacts that combines output from two different builds. Although for skaffold dev, without something like an --images flag to fix image A, this seems to be extra work (e.g. using auto-generated Kustomizations that update image A's tag)

• Workflow 1 builds image A; workflow 2 builds image B which is meant to use the latest A as a base image: I guess this is actually a general problem not limited to Skaffold that I could research more (just found this for example). I have a fear that auto-updating the tag for A in B's Dockerfile could lead to issues (e.g. merge conflicts), but maybe it's actually fine or even standard...

[alnr]

You can get a step closer by passing the --dry-run and --output=yaml flags to your deployer (specify in skaffold.yaml):

deploy:
  kustomize:
    path: Infrastructure/dev
    flags:
      apply:
        - --dry-run
        - --output=yaml

Unfortunately, the output is interleaved with skaffold logging. The global -v flags to skaffold don't work as expected.

[lucasmpr]

@alnr probably you could use the -o flag to output to a file using kustomize?

[alnr]

@alnr probably you could use the -o flag to output to a file using kustomize?

Not possible unfortunately. skaffold first calls kustomize to render/build the manifest, modifies that output to adjust image tags, and passes that to kubectl apply. kubectl does not support writing to files directly (-o/--output in kubectl controls output format, not output destination).

[alnr]

Ref #2161

[tejal29]

@itsleeowen, Recently @marlon-gamez added a functionality to output build output.
In #922, we actually separated skaffold build and skaffold deploy.

Now users can potentially do

1. skaffold build --file-output=build.out
2. skaffold deploy --build-artifacts=build.out

[tstromberg]

I'm closing this issue as it hasn't seen activity in awhile, and it doesn't seem to be making any headway for prioritization. Let's re-open the issue if and when we feel ready to address it.

[StevenACoffman]

I'm leaving breadcrumbs for others. Skaffold supports gitops by rendering the output files after a build:

skaffold build -p production
skaffold render -p production  --output rendered-deployment.yaml

You can then use skaffold apply:

skaffold apply -p production rendered-deployment.yaml

With the above steps, the altered files need to be checked into git (hopefully before applying) and you are on on your own for that. If the fully rendered kubernetes manifests need to be in a separate git repository from the application source code files... then things get really awkward. There's a profusion of different possibilities and very little guidance.

Suppose, we have 2 git repositories: A (application source files + skaffold.yaml) and B (production rendered kubernetes manifests). How would we automate making a PR in B (manifest) with the rendered output from A (app skaffold)?

Kustomize (and skaffold) allows us to refer to repository A in some kustomize files from repository B.
For this, Kustomize's git getter accepts both URL-style SSH addresses like git::ssh://[email protected]/foo/bar, and "scp-style" addresses like git::[email protected]/foo/bar. In the latter case, omitting the git:: force prefix is allowed if the username prefix is exactly git@.

From repository B (manifests only), you can use remoteBuild to use the kustomizations in A (source code, skaffold, kustomizations), and it can update the deployment. However, repository B would not have knowledge of the tag skaffold had pushed to unless skaffold had rendered it out as a new "base". (or if it was implied to be the full git sha1 of repository A main/master branch).

Alternatively, Suppose in production manifest repository B, we have cicd/kustomization.yaml:

namespace:
  cicd-prod
resources:
  - github.com/<Org>/k8s-infra/cicd/k8s-deploy/overlays/prod?ref=workflow-deploy

We can take the fully rendered production manifest as the base, and as long as there is some trigger (github action dispatch event from repository A or daisy chained cloudbuilds) it will work.

If Skaffold's render is not working for you, Skaffold also allows the final image to be retrieved via:

$ skaffold build -p production -q -o "{{json .}}"
{"builds":[{"imageName":"stevenacoffman/small","tag":"stevenacoffman/small:e89e17663a78e21b167fc5268b1e86bdee51113a29c7ea3dda8fa5943758adaf"}]}

skaffold build -q -o "{{(index .Builds 0).ImageName }}" or skaffold build -q -o "{{(index .Builds 0).Tag }}" select these.

kustomize allows you to edit the image with:

kustomize edit set image $(skaffold build -q -o "{{(index .Builds 0).ImageName }}")=$(git rev-parse HEAD)

This will change the overlay files and require it to be checked into git.