Support for OCI registries in Flux plugin

[gfichtenholt]

This is a long-running issue I started a little over a month ago.
Flux CD added support for OCI registries recently
fluxcd/flux2#2597

My work is to expose this functionality to kubeapps end user via flux plugin.

Areas that I worked on so far:

• Basic CRUD support for OCI registries, including caching of repos and charts in redis
  (like an index for regular helm repositories)
  U(pdate) implementation still pending.
• Introduce an API for repository lister (catalog). Implementation so far is
  docker api v2 compatible registry
  https://github.com/opencontainers/distribution-spec/blob/main/spec.md#api using
  ORAS v2 libraries. Works for GitHub (ghcr.io), others TBD
  Full discussion here: Flux oci support 2 #4932. I stopped
  when I got blocked by Support last parameter for listing catalog and tags oras-project/oras-go#196 below. Once I verify it is fixed and will pick up
  where I left off

Areas which I have not started yet:

• Support other kinds of repositories, such as Harbor, Azure CR, AWS CR, etc.
• Support for k8s docker registry credentials (OCI registries may use them)
• Support for InsecureTLS flag
• unit tests. I wrote one integration test so far which is testing (back)end-2-end,
  because I wanted to see what flux engine really does on the server-side
  I know I need to do the client-side tests.

Currently outstanding issues that will require verification and/or code changes when fixed:

• Add support for insecure registries in HelmRepositories of Type oci Add support for insecure registries in HelmRepositories of Type oci fluxcd/source-controller#807
• Flux fails to login to HTTP OCI registry where helm CLI logs in Flux fails to login to HTTP OCI registry where helm CLI logs in fluxcd/source-controller#805
• Support last parameter for listing catalog and tags Support last parameter for listing catalog and tags oras-project/oras-go#196

I plan to be updating this issue as I make further progress with OCI support in flux plugin

[ppbaena]

Hi Greg, could you update the current status of this issue and next steps planned?

[gfichtenholt]

Current status: in progress.
Items left to do before I can mark this Closed:

• Update operation implementation for OCI repos + integration tests.
  Time estimate: 2 days
• Production code cleanup (removing instrumentation via log.Info()).
  Other tidying up: 2 days
• Integration test that creates a helm release out of OCI helm repository. Time estimate: 2 days
• Integration tests: use cases involving TLS: 2 days
• Regular unit tests (i.e. when k8s isn't running).
  I have so far only written and relied on integration tests running locally (not in CI)
  which was fine until a point. Now I need to write at least some rudimentary CRUD tests that will be run on CI
  Time estimate: 2 days min. The more time I am able to spend the more tests I can write, including some edge scenarios

These items are optional and can be addressed at any point in the future:

• Integration tests: remove runtime dependency on github ghcr.io as a test registry host,
  in favor of a locally running registry. Blocked by Flux fails to login to HTTP OCI registry where helm CLI logs in fluxcd/source-controller#805 and How can I push chart to an insecure OCI registry with helm v3 helm/helm#6324
• Add support for insecure registries in HelmRepositories. Blocked by Add support for insecure registries in HelmRepositories of Type oci fluxcd/source-controller#807
• Support for non-docker api v2 compatible OCI registries, such as Harbor, Azure CR, AWS CR, etc.

[gfichtenholt]

1. OCI Distribution Spec - https://github.com/opencontainers/distribution-spec/blob/main/spec.md#api
2. Docker Registry HTTP API V2 Spec - https://docs.docker.com/registry/spec/api/#docker-registry-http-api-v2

Registry	(1)	(2)
ghcr.io	yes	yes
docker hub official 'registry' image (https://hub.docker.com/_/registry)	yes	yes
harbor	yes	no

[gfichtenholt]

Current status:

• Update operation implementation for OCI repos + integration tests is blocked by
  Requested feature: flux OCI helm repositories notice when tags on remote registry change fluxcd/source-controller#839
  Will begin working on remaining items until it is resolved

[gfichtenholt]

This PR is still in progress with lots of changes coming in, so adding this as comment here rather than creating a new issue: The current redis cache for helm OCI repos is broken. It loads the initial state into the cache just fine, but when the state on the remote changes, such as:

• a new chart is pushed to repo
• an existing chart is removed from repo
• a new chart version (aka tag) is pushed
• an existing tag is removed from a chart

will result in out-of-sync cache. Basically any change on remote server, the cache will get out of sync with the remote. The reason is that fluxcd chose not to implement a feature where the caller gets notified when source on remote changes (ref fluxcd/source-controller#839). I understand the seriousness of the issue and I am currently considering different fixes for this, such as:

• disable cache for helm OCI repositories
• implement logic to poll remote for changes and react (i.e. what flux is already doing)
• manual resync, including delete/create of helm OCI repo

will update this PR as I narrow the solution down

[gfichtenholt]

fyi, flux v. 33 adds HelmRepository.spec.provider. I will need to find out more about it and whether it affects our code or not

update Sept 7. The decision was to go ahead and add this as a feature. Implementation wise, I was thinking this:

1. setup an integration scenario that requires this feature (probably on GCP, since I've already done a bunch of work there, but I am open to others)
2. add code to flux plugin to handle this setting. Should be simple as plugin-specific (custom) setting
3. verify (2) is working with an integration test
4. when we have cycles, we expose (2) in the UI

[gfichtenholt]

Just FYI, this feature mentioned above appears to be something flux authors call "auto-login" or
"contextual login" that basically means using a "secured" HelmRepository without
a secret ref. It is important to understand that in order for this scenario to work,
the workload (i.e. flux controllers and kubeapps) needs to actually run on GKE cluster.
Another way to put it is "Workload Identity can't be used by Pods running on the host network"
ref https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#limitations

If I had to summarize this approach in one sentence it would something like "instead of
configuring secrets based off of as service access tokens or service JSON keys,
per helm repository, one configures flux source-controller pod with a certain
'workload identity' which is provided by flux to GKE when needed for authentication"

[castelblanque]

Thanks for the info, Greg, that makes it more understandable. The term "contextual login" or "contextual auth" looks like a good fit for referring to it.

Wondering how we can manage the feature from the UI point of view. For example: user creating a HelmRepository from Kubeapps UI could select the provider from e.g. a dropdown. Initially it could be a fixed list of the options in the provider spec.
But for the experience to be optimal, the allowed options in the dropdown could be only two: 1) generic and 2) the provider in which Flux machinery runs only (e.g. gcp)?
Is the cloud provider something that we could find out from within Kubeapps?
In the multicluster scenario, the UI in which the repo would be created might not have access to the backend of the actual Flux running.

[gfichtenholt]

Haven't thought this far ahead yet. As a first cut I was thinking of having users pick from a fixed list. I haven't done enough research to see if it's possible to find out automatically. Flux guys chose to put the burden on the user, so there is that

[gfichtenholt]

Status update: took 3 full days but I have now verified that the feature works as advertised in flux. That is, having the field spec.provider set to "gcp" is sufficient to make the HelmRepository work without a secret ref on GCP. Pretty neat.
The caveat is you have to set up your GKE cluster for workload identity, which is non-trivial and what took me the vast majority of the time. Thinking about how to set up flux plugin integration test for this. Will probably be guarded by a flag. Unset by default. If set, rely on some existing GKE cluster that is pre-created and pre-configured by someone prior to running the test

[gfichtenholt]

I believe this issue can be now closed. All planned features have been implemented. If any new features rise we shall open separate PR targeting those