Using the OCI Helm Repository is impossible

[hobbsh]

Describe the solution you'd like
There was recently a change to use an OCI Helm Repository, however it does not work. The documented way of using it also results in a failure. It seems like at the very least, documentation needs to be updated.

Helm install locally according to the documentation (looks like the version is just incorrect):

$ helm install --set router.configuration.telemetry.metrics.prometheus.enabled=true --set managedFederation.apiKey="REDACTED" --set managedFederation.graphRef="REDACTED" --create-namespace --namespace router-deploy router-test oci://ghcr.io/apollographql/helm-charts/router --version 0.14.0 --values router/values.yaml

Error: INSTALLATION FAILED: failed to download "oci://ghcr.io/apollographql/helm-charts/router" at version "0.14.0"

Trying to add the repository manually locally:

$ helm repo add apollo-router oci://ghcr.io/apollographql/helm-charts/router

Error: looks like "oci://ghcr.io/apollographql/helm-charts/router" is not a valid chart repository or cannot be reached: object required

Using an oci type HelmRepository with flux:

chart pull error: chart pull error: failed to get chart version for remote reference: GET "https://ghcr.io/v2/apollographql/helm-charts/router/router/tags/list": GET "https://ghcr.io/token?scope=repository%!!(MISSING)A(MISSING)apollographql%!!(MISSING)F(MISSING)helm-charts%!!(MISSING)F(MISSING)router%!!(MISSING)F(MISSING)router%!!(MISSING)A(MISSING)pull&service=ghcr.io": unexpected status code 403: denied: requested access to the resource is denied

 ---
 apiVersion: source.toolkit.fluxcd.io/v1beta2
 kind: HelmRepository
 metadata:
   name: apollo-router
 spec:
   type: "oci"
   interval: 30m0s
   url: oci://ghcr.io/apollographql/helm-charts/router

Describe alternatives you've considered
It looks like tags are not being used on the repository itself anymore so it's even more painful to pin releases that way, despite the fact that this was not a great solution to begin with.

Additional context
Additionally, the README in the chart itself is out of date and that is where the instructions should be updated, not here, in my opinion.

Local helm version:

helm version
version.BuildInfo{Version:"v3.9.4", GitCommit:"dbc6d8e20fe1d58d50e6ed30f09a04a77e4c68db", GitTreeState:"clean", GoVersion:"go1.19"}

[hobbsh]

Currently investigating if our flux version is out of date which could be causing the 403s. However it's still concerning that I can't helm repo add.

[garypen]

It is possible. We use it internally.

There may well be issues with the documentation, so I will work look through the docs to see what errors I can fix. Or I'm also always happy to accept PRs.

If you read the helm documentation for OCI hosted charts, you'll note that helm repo add is not supported. (I agree that this is tricky to find, so it may be that the helm documentation on this subject could be improved.)

I can't comment on whether or not flux works with OCI hosted charts, since I'm not currently using flux.

[garypen]

(I'm on vacation today, so won't get the chance to look at this properly until tomorrow. However, here's a snippet of how we use the OCI chart in our helm command that may be helpful)

HELM_CHART="oci://ghcr.io/apollographql/helm-charts/router"
HELM_CHART_VERSION="0.1.21"

helm upgrade router "${HELM_CHART}" \
    --version "${HELM_CHART_VERSION}" \
    <etc...>

[hobbsh]

Thanks for the quick reply @garypen - no need to respond on your vacation though! Unfortunately, manually helm installs are a no-go for us, our clusters are fully automated with flux. Just wanted to drop an update that it looks like updating flux (to v0.33.0) does not help the situation and could come down to a lack of support or some configuration issue in the OCI registry itself:

Error message in flux, seems like it could be a registry permission issue:

chart pull error: chart pull error: failed to download chart for remote reference: failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden

[garypen]

Error message in flux, seems like it could be a registry permission issue:

That's an odd error. There's no requirement to authorise to pull that helm chart. You can confirm this by running:

helm pull oci://ghcr.io/apollographql/helm-charts/router --version 0.1.21

manually. That should just succeed and deliver router-0.1.21.tgz to you.

[garypen]

I've raised a PR (#1649) to fix the documentation generation for the repo section of the helm chart docs. The next time we do a release the helm docs (README.md) will be correct.

With that fix in place, I think the issues remaining here are flux issues, so I'll close this issue. Feel free to re-open it if that's not the case.

[ecoupal-believe]

@hobbsh I have the same exact error did you find a workaround ?

[ecoupal-believe]

@hobbsh just in case you still have this issue: fluxcd/flux2#3438

This will fix it:

apiVersion: source.toolkit.fluxcd.io/v1beta2
 kind: HelmRepository
 metadata:
   name: apollo-router
 spec:
   type: "oci"
   interval: 30m0s
   url: oci://ghcr.io/apollographql/helm-charts

[rorychatterton]

I ran into the same issue with helm on the desktop.

It appears that you can't download GHCR OCI repositories without being logged into Github first in docker. To fix, you can do something like this:

# Fails
➜  ~ helm pull oci://ghcr.io/apollographql/helm-charts/router --version 0.1.21
Error: failed to authorize: failed to fetch oauth token: unexpected status: 403 Forbidden

# Login to Github
➜  ~ echo $GITHUB_TOKEN | docker login ghcr.io -u <Your Username> --password-stdin
Login Succeeded

# Now Works
➜  ~ helm pull oci://ghcr.io/apollographql/helm-charts/router --version 0.1.21
Pulled: ghcr.io/apollographql/helm-charts/router:0.1.21
Digest: sha256:cd4055ad5f6530136f10cccbc6d4f3596fa632e55ca3efcdddfc64367db69c1a

@garypen I'm not sure if this is a known issue, but it would be good to get this in the standard documentation 👍

[garypen]

I'm fairly sure you don't need to have docker logged in to ghcr.io. I use the chart in many environments where I'm not logged in to docker. To be sure I tested as follows:

garypen@Garys-MBP docs % docker logout ghcr.io.
Removing login credentials for ghcr.io
garypen@Garys-MBP docs % helm pull oci://ghcr.io/apollographql/helm-charts/router --version 0.1.21
Pulled: ghcr.io/apollographql/helm-charts/router:0.1.21
Digest: sha256:cd4055ad5f6530136f10cccbc6d4f3596fa632e55ca3efcdddfc64367db69c1a
garypen@Garys-MBP docs % ls -lart router-0.1.21.tgz 
-rw-r--r--  1 garypen  staff  7451 31 Jan 08:29 router-0.1.21.tgz

I don't know why it failed for you, could there be something unusual about your environment? Perhaps you can try my test from a different environment and see what happens for you?

[rorychatterton]

Very strange.

Previously it was not working for me on either device (PopOS using Podman + Helm, and ARM Mac using Docker + Helm) until I had logged into the registry.

Now it's working perfectly fine after logging out of both, on both. 🤷

The gremlins are at it again... Sorry for the misguidance.

[zackerydev]

I'm seeing this again unfortunately...
https://ghcr.io/v2/apollographql/helm-charts/router/router/manifests/1.40.1

Is returning 403 for me.

[crazyelectron-io]

I have the same issue with Ansible as the OP has with Flux, using Ansible's kubernetes.core.helm. It requires authentication to ghcr.io even though it is a public repo and works without login if running helm install.