Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

registry facade won't start in air gap network #8462

Closed
SirLemyDanger opened this issue Feb 25, 2022 · 4 comments · Fixed by #8694
Closed

registry facade won't start in air gap network #8462

SirLemyDanger opened this issue Feb 25, 2022 · 4 comments · Fixed by #8694
Assignees
@corneliusludmann
Labels
feature: documentation team: delivery Issue belongs to the self-hosted team

Comments

@SirLemyDanger
Copy link

Bug description

After a fresh installation of GitPod (Jan'22) the registry facade pod is crashing permanently.

Environment: K3s on Ubuntu
I set up K3s to pull images from a proxy (artifactory) by having a /etc/rancher/k3s/registries.yaml like this:

mirrors:
  docker.io:
    endpoint:
      - "<my container proxy>"
  gcr.io:
    endpoint:
      - "<my container proxy>"
  eu.gcr.io:
    endpoint:
      - "<my container proxy>"
  quay.io:
    endpoint:
      - "<my container proxy>"

This works fine; all containers are being pulled and I can manually pull images with crictl.

However the registry-facade pod tries to access https://eu.gcr.io/v2/gitpod-core-dev/build/supervisor/manifests/commit-dbe106c19401e780829827dc50a0154c1b93b96c which times out, as the machine has no direct access to the internet. See log at the end.

In a postpressing step of the gitlab.config.yaml I changed the runtime setting to (not sure though if this is a good idea):

workspace:
  resources:
    requests:
      cpu: "1"
      memory: 2Gi
  runtime:
    containerdRuntimeDir: /var/lib/rancher/k3s/agent/containerd/io.containerd.runtime.v2.task/k8s.io
    containerdSocket: /run/k3s/containerd/containerd.sock
    fsShiftMethod: shiftfs

error log:

Head \"https://eu.gcr.io/v2/gitpod-core-dev/build/supervisor/manifests/commit-dbe106c19401e780829827dc50a0154c1b93b96c\": dial tcp 108.177.15.82:443: i/o timeout
es/docker.(*request).do
	github.com/containerd/[email protected]/remotes/docker/resolver.go:569
github.com/containerd/containerd/remotes/docker.(*request).doWithRetries
olver.go:576
github.com/containerd/containerd/remotes/docker.(*dockerResolver).Resolve
	github.com/containerd/[email protected]/remotes/docker/resolver.go:280
github.comceFromImage
	github.com/gitpod-io/gitpod/registry-facade/pkg/registry/layersource.go:227
github.com/gitpod-io/gitpod/registry-facade/pkg/registry.buildStaticLayer
	githy.go:48
github.com/gitpod-io/gitpod/registry-facade/pkg/registry.NewRegistry
	github.com/gitpod-io/gitpod/registry-facade/pkg/registry/registry.go:104
github.com/gitpod-od-io/gitpod/registry-facade/cmd/run.go:113
github.com/spf13/cobra.(*Command).execute
	github.com/spf13/[email protected]/command.go:856
github.com/spf13/cobra.(*Command).Exeb.com/spf13/cobra.(*Command).Execute
	github.com/spf13/[email protected]/command.go:897
github.com/gitpod-io/gitpod/registry-facade/cmd.Execute
	github.com/gitpod-io/gitpod/od-io/gitpod/registry-facade/main.go:10
runtime.main
	runtime/proc.go:255
runtime.goexit
	runtime/asm_amd64.s:1581","host":"eu.gcr.io","level":"info","message":"trying
{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","error":"cannot source layer from eu.gcr.io/gitpod-core-dev/build/supervisor:coom/gitpod-io/gitpod/registry-facade/pkg/registry.buildStaticLayer
        github.com/gitpod-io/gitpod/registry-facade/pkg/registry/registry.go:50
  - Head \"https://eu.gcr106c19401e780829827dc50a0154c1b93b96c\": dial tcp 108.177.15.82:443: i/o timeout
    failed to do request
    github.com/containerd/containerd/remotes/docker.(*request).doer/resolver.go:569
    github.com/containerd/containerd/remotes/docker.(*request).doWithRetries
    	github.com/containerd/[email protected]/remotes/docker/resolver.go:57kerResolver).Resolve
    	github.com/containerd/[email protected]/remotes/docker/resolver.go:280
    github.com/gitpod-io/gitpod/registry-facade/pkg/registry.NewStaticSouade/pkg/registry/layersource.go:227
    github.com/gitpod-io/gitpod/registry-facade/pkg/registry.buildStaticLayer
    	github.com/gitpod-io/gitpod/registry-facade/pkg/regry-facade/pkg/registry.NewRegistry
    	github.com/gitpod-io/gitpod/registry-facade/pkg/registry/registry.go:104
    github.com/gitpod-io/gitpod/registry-facade/cmd.glob.d/run.go:113
    github.com/spf13/cobra.(*Command).execute
    	github.com/spf13/[email protected]/command.go:856
    github.com/spf13/cobra.(*Command).ExecuteC
    	github/cobra.(*Command).Execute
    	github.com/spf13/[email protected]/command.go:897
    github.com/gitpod-io/gitpod/registry-facade/cmd.Execute
    	github.com/gitpod-io/gitpod.com/gitpod-io/gitpod/registry-facade/main.go:10
    runtime.main
    	runtime/proc.go:255
    runtime.goexit
    	runtime/asm_amd64.s:1581","level":"fatal","message":ry-facade","version":"commit-c6fdc8f36defd46b07ede1dd7a294846f1093c51"},"severity":"CRITICAL","time":"2022-02-25T15:21:29Z"}

Steps to reproduce

K3s without traeffik, single node; configured with /etc/rancher/k3s/registries.yaml

install GitPod with installer of the Jan'22 release

Workspace affected

No response

Expected behavior

no request to the internet

Example repository

No response

Anything else?

No response
@sagor999 sagor999 added the team: delivery Issue belongs to the self-hosted team label Feb 25, 2022
@SirLemyDanger
Copy link
Author

re-checked with feb '2022 release. Still the case

@SirLemyDanger
Copy link
Author

looks releated to the limitations metioned here: #6983

@corneliusludmann corneliusludmann mentioned this issue Mar 9, 2022
Merged
@corneliusludmann
Copy link
Contributor

@SirLemyDanger Sorry that it took so long but I'm currently adding instructions on how to install Gitpod in an air-gap network with this pull request. Would love to hear if this works for you.

@corneliusludmann corneliusludmann moved this from 🧊Backlog to 🕶In Review / Measuring in 🚚 Security, Infrastructure, and Delivery Team (SID) Mar 9, 2022
@corneliusludmann corneliusludmann self-assigned this Mar 9, 2022
Repository owner moved this from 🕶In Review / Measuring to ✨Done in 🚚 Security, Infrastructure, and Delivery Team (SID) Mar 9, 2022
@SirLemyDanger
Copy link
Author

that worked as far as I can tell. registry-facade is still crashing but it seems that this is now because of #8559


{"error":"Head \"https://myregistry/v2/pleniter/control/gitpod/supervisor/manifests/commit-a8dd8e421ba0ddec468ae78dcf336f6c1aad74b5\": x509: certificate signed by unknown authority
failed to do request
github.com/containerd/containerd/remotes/docker.(*request).do
	github.com/containerd/[email protected]/remotes/docker/resolver.go:569
github.com/containerd/containerd/remotes/docker.(*request).doWithRetries
	github.com/containerd/[email protected]/remotes/docker/resolver.go:576
github.com/containerd/containerd/remotes/docker.(*dockerResolver).Resolve
	github.com/containerd/[email protected]/remotes/docker/resolver.go:280
github.com/gitpod-io/gitpod/registry-facade/pkg/registry.NewStaticSourceFromImage
	github.com/gitpod-io/gitpod/registry-facade/pkg/registry/layersource.go:227
github.com/gitpod-io/gitpod/registry-facade/pkg/registry.buildStaticLayer
	github.com/gitpod-io/gitpod/registry-facade/pkg/registry/registry.go:48
github.com/gitpod-io/gitpod/registry-facade/pkg/registry.NewRegistry
	github.com/gitpod-io/gitpod/registry-facade/pkg/registry/registry.go:104
github.com/gitpod-io/gitpod/registry-facade/cmd.glob..func2
	github.com/gitpod-io/gitpod/registry-facade/cmd/run.go:113
github.com/spf13/cobra.(*Command).execute
	github.com/spf13/[email protected]/command.go:856
github.com/spf13/cobra.(*Command).ExecuteC
	github.com/spf13/[email protected]/command.go:960
github.com/spf13/cobra.(*Command).Execute
	github.com/spf13/[email protected]/command.go:897
github.com/gitpod-io/gitpod/registry-facade/cmd.Execute
	github.com/gitpod-io/gitpod/registry-facade/cmd/root.go:41
main.main
	github.com/gitpod-io/gitpod/registry-facade/main.go:10
runtime.main
	runtime/proc.go:255
runtime.goexit
	runtime/asm_amd64.s:1581","host":"myregistry","level":"info","message":"trying next host","severity":"INFO","time":"2022-03-10T10:56:22Z"}
{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","error":"cannot source layer from myregistry/pleniter/control/gitpod/supervisor:commit-a8dd8e421ba0ddec468ae78dcf336f6c1aad74b5:
    github.com/gitpod-io/gitpod/registry-facade/pkg/registry.buildStaticLayer
        github.com/gitpod-io/gitpod/registry-facade/pkg/registry/registry.go:50
  - Head \"https://myregistry/v2/pleniter/control/gitpod/supervisor/manifests/commit-a8dd8e421ba0ddec468ae78dcf336f6c1aad74b5\": x509: certificate signed by unknown authority
    failed to do request
    github.com/containerd/containerd/remotes/docker.(*request).do
    	github.com/containerd/[email protected]/remotes/docker/resolver.go:569
    github.com/containerd/containerd/remotes/docker.(*request).doWithRetries
    	github.com/containerd/[email protected]/remotes/docker/resolver.go:576
    github.com/containerd/containerd/remotes/docker.(*dockerResolver).Resolve
    	github.com/containerd/[email protected]/remotes/docker/resolver.go:280
    github.com/gitpod-io/gitpod/registry-facade/pkg/registry.NewStaticSourceFromImage
    	github.com/gitpod-io/gitpod/registry-facade/pkg/registry/layersource.go:227
    github.com/gitpod-io/gitpod/registry-facade/pkg/registry.buildStaticLayer
    	github.com/gitpod-io/gitpod/registry-facade/pkg/registry/registry.go:48
    github.com/gitpod-io/gitpod/registry-facade/pkg/registry.NewRegistry
    	github.com/gitpod-io/gitpod/registry-facade/pkg/registry/registry.go:104
    github.com/gitpod-io/gitpod/registry-facade/cmd.glob..func2
    	github.com/gitpod-io/gitpod/registry-facade/cmd/run.go:113
    github.com/spf13/cobra.(*Command).execute
    	github.com/spf13/[email protected]/command.go:856
    github.com/spf13/cobra.(*Command).ExecuteC
    	github.com/spf13/[email protected]/command.go:960
    github.com/spf13/cobra.(*Command).Execute
    	github.com/spf13/[email protected]/command.go:897
    github.com/gitpod-io/gitpod/registry-facade/cmd.Execute
    	github.com/gitpod-io/gitpod/registry-facade/cmd/root.go:41
    main.main
    	github.com/gitpod-io/gitpod/registry-facade/main.go:10
    runtime.main
    	runtime/proc.go:255
    runtime.goexit
    	runtime/asm_amd64.s:1581","level":"fatal","message":"cannot create registry","serviceContext":{"service":"registry-facade","version":"commit-fbb3e1d7cf1ae8c31a8db44029e9a6d6660cbbb9"},"severity":"CRITICAL","time":"2022-03-10T10:56:22Z"}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@corneliusludmann corneliusludmann

Labels
feature: documentation team: delivery Issue belongs to the self-hosted team
Projects
No open projects
🚚 Security, Infrastructure, and Delivery Team (SID)
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[installer] Add air-gap instructions gitpod-io/gitpod
3 participants
@SirLemyDanger @sagor999 @corneliusludmann