Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC-0003] Flux OCI support for Kubernetes manifests #2601

Merged
merged 13 commits into from
Jul 8, 2022

Conversation

stefanprodan
Copy link
Member

@stefanprodan stefanprodan commented Mar 31, 2022

Proposal for adding OCI support for Kubernetes manifests to Flux.

Flux should be able to distribute and reconcile Kubernetes configuration packaged as OCI artifacts.

On the client-side, the Flux CLI should offer a command for packaging Kubernetes configs into an OCI artifact and pushing the artifact to a container registry using the Docker config file and the Docker credential helpers for authentication.

flux push artifact oci://ghcr.io/org/my-app-config:v1.0.0 --path ./deploy

On the server-side, the Flux source-controller should offer a dedicated API Kind for defining how OCI artifacts are pulled from container registries and how the artifact's authenticity can be verified. Flux source-controller should be able to work with any type of artifact even if it's not created with the Flux CLI.

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
  name: app-config
spec:
  interval: 10m
  url: oci://ghcr.io/org/my-app-config
  ref:
    semver: "1.x"
  secretRef:
    name: my-app-regcred
  verify:
    provider: cosign
    secretRef:
      name: my-app-cosgin-key

To apply manifests stored in OCI artifacts, Flux kustomize-controller should accept OCIRepository as a source:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: app
spec:
  interval: 10m
  serviceAccountName: app-applier
  sourceRef:
    kind: OCIRepository
    name: app-config
  path: ./
  wait: true
  prune: true

This RFC is sponsored by @hiddeco

@stefanprodan stefanprodan added the area/rfc Feature request proposals in the RFC format label Mar 31, 2022
@stefanprodan stefanprodan changed the title Add proposal for adding OCI support for Kubernetes manifests to Flux [RFC] Flux OCI support for Kubernetes manifests Mar 31, 2022
rfcs/kubernetes-oci/README.md Outdated Show resolved Hide resolved
@pjbgf pjbgf added this to the GA milestone May 12, 2022
@rashedkvm
Copy link
Member

@stefanprodan @hiddeco @pjbgf quick update. I am completing a different work assignment. I expect to have a draft PR for this RFC in the next 2 to 3 weeks. Let me know if this timeline conflicts with the road to GA. Thank you!

@hiddeco
Copy link
Member

hiddeco commented May 20, 2022

This is absolutely fine I think, there is still quite some work to do for us in other controller areas around GA.

@tis-rpage
Copy link

Any chance of using cosign verification of the OCI artifacts?
https://github.com/sigstore/cosign

@stefanprodan
Copy link
Member Author

@tis-rpage Please read the RFC, cosign verification is there.

--path="./deploy"
```

The Flux CLI will produce artifacts of type `application/vnd.docker.distribution.manifest.v2+json`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should produce application/vnd.oci.image.manifest.v1+json, manifest types. This is supported by all major container registries, docker included, and is the only one currently registered: https://www.iana.org/assignments/media-types/application/vnd.oci.image.manifest.v1+json

Copy link
Member Author

@stefanprodan stefanprodan Jul 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe when crane would allow changing the type we can switch to it. But since source-controller doesn't care about types and should work with any type, I don't think vnd.docker should block the RFC.

Copy link
Member

@souleb souleb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Copy link
Contributor

@darkowlzz darkowlzz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Copy link
Member

@pjbgf pjbgf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM


Flux users should be able to package a local directory containing Kubernetes configs into a tarball
and push the archive to a container registry as an OCI artifact.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It means every OCI artifact will only have 1 layer and will contains 100% of the kube object to create. We will have to be really careful else flux will garbage collect kube object.

I did not saw an "flux artifact diff" command. For me it will be a mandatory one.

Perhaps the "flux push" could also be oriented to be more careful on how to build such artifact ( add a "parent" artifact and if not set ask user to validate the fact it is an initial state, else display a diff and ask for validation. ) A metadata can be set inside the OCI artifact to "link" to the parent, potentially, fluxcd (the image source component) can be setup to accept new version only if new version is linked to the previous version via a chain ( I want to deploy version 5, I got version 3, I check the parent of version 5 which is 4, I check parent of version 4 which is 3 OK I can deploy, and if I want to deploy version 2 because I rollback, the version 2 will give parent 1 which give no parent, so flux detect it is not an update, try the rollback check if current version ( 3 ) has a parent yes, it is 2 which is the one you try to deploy => OK let's deploy this rollback).

"flux push" could also have other option like " flux artifact push --parent oci://docker.io/org/app-config:v1.2.3 --update minor --diff -path ./deploy " which state take the version 1.2.3 and create an updated version with minor change (only work with semver, and here it will create a version v1.3.0 ) and take the content of the path ./deploy as a diff from previous version ( take the previous version and add/replace file from the ./deploy path )

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO chaining artifacts would bring a lot of complexity with no real benefit, we push a single layer that should contain a snapshot of a Git repository content for the commit that’s checked out. You have the option to tag the artifact with the Git SHA or Git Tag. As for flux diff artifact, this can be easily done with flux pull artifact then using gnu diff.

rfcs/kubernetes-oci/README.md Outdated Show resolved Hide resolved
Signed-off-by: Stefan Prodan <[email protected]>
@stefanprodan stefanprodan marked this pull request as ready for review July 6, 2022 14:02
@stefanprodan stefanprodan changed the title [RFC] Flux OCI support for Kubernetes manifests [RFC-0003] Flux OCI support for Kubernetes manifests Jul 6, 2022
@stefanprodan stefanprodan merged commit abeea06 into main Jul 8, 2022
@stefanprodan stefanprodan deleted the rfc-kubernetes-oci branch July 8, 2022 09:11
@davinkevin
Copy link

Sorry if I ask this question too late, but is there a possibility to interact with this OCI entry after download by the OCIRepository component?

As an end user, I would like to use it as the HelmRelease component, where I can provide some values.yaml. Here, it seems to be really static. I have a set of manifests (called base in our case) and every env (dev, prod…) would like to consume one (using OCI entry) but with different config, secret, patches and other stuff provided by kustomize.

The main idea would be to replace our base by an OCI entry but without extension possible, we are not able to use them.

@kingdonb
Copy link
Member

kingdonb commented Aug 1, 2022

I see that GitRepositorySpec is where the includes feature is defined, so it appears to be correct that OCIRepository cannot currently be "included" in a GitRepository or vice versa.

Should we try to better generalize this part of the spec so those includes would be possible, or is there something blocking?

@davinkevin
Copy link

For information, I've elaborated my question in this comment, with some feedbacks on the current OCI specification for manifests:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/rfc Feature request proposals in the RFC format
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.