[RFC-0003] Flux OCI support for Kubernetes manifests

[stefanprodan]

Proposal for adding OCI support for Kubernetes manifests to Flux.

Flux should be able to distribute and reconcile Kubernetes configuration packaged as OCI artifacts.

On the client-side, the Flux CLI should offer a command for packaging Kubernetes configs into an OCI artifact and pushing the artifact to a container registry using the Docker config file and the Docker credential helpers for authentication.

flux push artifact oci://ghcr.io/org/my-app-config:v1.0.0 --path ./deploy

On the server-side, the Flux source-controller should offer a dedicated API Kind for defining how OCI artifacts are pulled from container registries and how the artifact's authenticity can be verified. Flux source-controller should be able to work with any type of artifact even if it's not created with the Flux CLI.

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
  name: app-config
spec:
  interval: 10m
  url: oci://ghcr.io/org/my-app-config
  ref:
    semver: "1.x"
  secretRef:
    name: my-app-regcred
  verify:
    provider: cosign
    secretRef:
      name: my-app-cosgin-key

To apply manifests stored in OCI artifacts, Flux kustomize-controller should accept OCIRepository as a source:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: app
spec:
  interval: 10m
  serviceAccountName: app-applier
  sourceRef:
    kind: OCIRepository
    name: app-config
  path: ./
  wait: true
  prune: true

This RFC is sponsored by @hiddeco

[rashedkvm]

@stefanprodan @hiddeco @pjbgf quick update. I am completing a different work assignment. I expect to have a draft PR for this RFC in the next 2 to 3 weeks. Let me know if this timeline conflicts with the road to GA. Thank you!

[hiddeco]

This is absolutely fine I think, there is still quite some work to do for us in other controller areas around GA.

[tis-rpage]

Any chance of using cosign verification of the OCI artifacts?
https://github.com/sigstore/cosign

[stefanprodan]

@tis-rpage Please read the RFC, cosign verification is there.

[souleb]

I think we should produce application/vnd.oci.image.manifest.v1+json, manifest types. This is supported by all major container registries, docker included, and is the only one currently registered: https://www.iana.org/assignments/media-types/application/vnd.oci.image.manifest.v1+json

[stefanprodan]

Maybe when crane would allow changing the type we can switch to it. But since source-controller doesn't care about types and should work with any type, I don't think vnd.docker should block the RFC.

[souleb]

LGTM.

[darkowlzz]

LGTM!

[pjbgf]

LGTM

[GregoireW]

It means every OCI artifact will only have 1 layer and will contains 100% of the kube object to create. We will have to be really careful else flux will garbage collect kube object.

I did not saw an "flux artifact diff" command. For me it will be a mandatory one.

Perhaps the "flux push" could also be oriented to be more careful on how to build such artifact ( add a "parent" artifact and if not set ask user to validate the fact it is an initial state, else display a diff and ask for validation. ) A metadata can be set inside the OCI artifact to "link" to the parent, potentially, fluxcd (the image source component) can be setup to accept new version only if new version is linked to the previous version via a chain ( I want to deploy version 5, I got version 3, I check the parent of version 5 which is 4, I check parent of version 4 which is 3 OK I can deploy, and if I want to deploy version 2 because I rollback, the version 2 will give parent 1 which give no parent, so flux detect it is not an update, try the rollback check if current version ( 3 ) has a parent yes, it is 2 which is the one you try to deploy => OK let's deploy this rollback).

"flux push" could also have other option like " flux artifact push --parent oci://docker.io/org/app-config:v1.2.3 --update minor --diff -path ./deploy " which state take the version 1.2.3 and create an updated version with minor change (only work with semver, and here it will create a version v1.3.0 ) and take the content of the path ./deploy as a diff from previous version ( take the previous version and add/replace file from the ./deploy path )

[stefanprodan]

IMO chaining artifacts would bring a lot of complexity with no real benefit, we push a single layer that should contain a snapshot of a Git repository content for the commit that’s checked out. You have the option to tag the artifact with the Git SHA or Git Tag. As for flux diff artifact, this can be easily done with flux pull artifact then using gnu diff.

[davinkevin]

Sorry if I ask this question too late, but is there a possibility to interact with this OCI entry after download by the OCIRepository component?

As an end user, I would like to use it as the HelmRelease component, where I can provide some values.yaml. Here, it seems to be really static. I have a set of manifests (called base in our case) and every env (dev, prod…) would like to consume one (using OCI entry) but with different config, secret, patches and other stuff provided by kustomize.

The main idea would be to replace our base by an OCI entry but without extension possible, we are not able to use them.

[kingdonb]

I see that GitRepositorySpec is where the includes feature is defined, so it appears to be correct that OCIRepository cannot currently be "included" in a GitRepository or vice versa.

Should we try to better generalize this part of the spec so those includes would be possible, or is there something blocking?

[davinkevin]

For information, I've elaborated my question in this comment, with some feedbacks on the current OCI specification for manifests: