Istio: failed to download artifact from source-controller: 503 service unavailable

[Edblakejitsuin]

Running:

flux create kustomization flux-system
--source=flux-system
--path="./clusters/my-cluster"
--prune=true
--interval=10m

Get the error:

faild to download artifact from http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/7759744efeeaf1823e07e5f911f192beb0a08c97.tar.gz, status: 503 Service Unavailable

flux check reports:

► checking prerequisites
✔ kubectl 1.18.12 >=1.18.0
✔ Kubernetes 1.17.13 >=1.16.0
► checking controllers
✔ source-controller is healthy
► regsitry/fluxcd/source-controller:v0.5.4
✔ kustomize-controller is healthy
► regsitry/fluxcd/kustomize-controller:v0.5.0
✔ helm-controller is healthy
► regsitry/fluxcd/helm-controller:v0.4.3
✔ notification-controller is healthy
► regsitry/fluxcd/notification-controller:v0.5.0
✔ all checks passed

[hiddeco]

We assume a default cluster domain since v0.5.x (cluster.local), but this likely does not match your cluster configuration. You can configure your cluster domain using flux bootstrap --cluster-domain your.domain [..].

[Edblakejitsuin]

I am using the default cluster.local domain...
noticed there is a perod after cluster.local in the URL, could this be the problem?

http://source-controller.flux-system.svc.cluster.local./gitrepository

[Edblakejitsuin]

removing the period from the source controller deployment setting --storage-adv-addr and in gotk-components.yaml allowed me to create the kustomization - presumably a bug?

[hiddeco]

The period after the cluster domain is to prevent an additional DNS lookup, and should work fine.

Are you able to curl to the given URL from another pod? If not, are you able to curl to e.g. http://source-controller.flux-system/gitrepository/flux-system/flux-system/7759744efeeaf1823e07e5f911f192beb0a08c97.tar.gz?

[Edblakejitsuin]

I think this was related to my flux-system namespace having istio sidecar injection enabled. I can apply the kustomization with the default URL and injection disabled.

[MichaelHudgins]

For anyone who runs into this and doesn't want to disable injection the following virtual service should fix the resolving issue until istio can fix FQDN with . at the end. Modify cluster.local if you need:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: source-controller-fqdn-fix
  namespace: flux-system
spec:
  hosts:
  - source-controller.flux-system.svc.cluster.local.
  http:
  - route:
    - destination:
        host: source-controller.flux-system.svc.cluster.local