Building custom image is stuck when "RUN brew install tfenv"

[vavdoshka]

Bug description

Gitpod being stuck on attempt to build the image with tfenv. More then 20 minutes for sure without any progreess.

initializing base image context
running base image build
Step 1/4 : FROM gitpod/workspace-full
latest: Pulling from gitpod/workspace-full
Digest: sha256:3ba16dad787233abc63227ff0f0effa9db584c6aaee15a7dae023f8fe002cb9d
Status: Image is up to date for gitpod/workspace-full:latest
 ---> 0741af90db04
Step 2/4 : RUN brew install tfenv
 ---> Running in 10f31d5eba82
==> Downloading https://ghcr.io/v2/linuxbrew/core/bzip2/manifests/1.0.8-1
==> Downloading https://ghcr.io/v2/linuxbrew/core/bzip2/blobs/sha256:2a4cbb681a3aeda2361625bd10d329e47bf986feaccd50bf3a1dce7753ad42d6
==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:2a4cbb681a3aeda2361625bd10d329e47bf986feaccd50bf3a1dce7753ad42d6?se=2021-09-26T12%3A05%3A00Z&sig=nStf%2B0L%2BSsiHGpLbrfohIag0seXhlAuu8BrXx2XI7zA%3D&sp=r&spr=https&sr=b&sv=2019-12-12

Steps to reproduce

run gitpod with

.gitpod.yaml

image:
  file: .gitpod.Dockerfile

.gitpod.Dockerfile

FROM gitpod/workspace-full

RUN brew install tfenv
RUN tfenv install latest && tfenv use latest

Expected behavior

the gitpod is started with terraform command available

Example repository

No response

Anything else?

Just started to play around with Gitpod, I wonder if I do something wrong or it is some sort of a bug.
Meanwhile as a workaround I will continue with a public pod image https://github.com/Vlaaaaaaad/gitpod-terraform

[tob1k]

FWIW I'm seeing the exact same behaviour

[shaal]

Please help, I have the same issue.
Running brew update && brew install drud/ddev/ddev in .gitpod.Dockerfile is slow with errors.

Typing the same command in a new Gitpod workspace runs fast with no errors.

[Decipher]

Also getting the same(ish) error for pretty much the same command as above: https://gitpod.io/#github.com/realityloop/presentation-druxt_101/tree/feature/dceu

[shaal]

Minimum replicable repo -
https://github.com/shaal/testing-gitpod-imagebuild

[iQQBot]

It look like image build server had some network issue

[pawlean]

Thank you for all the detail everyone! Especially to @shaal for the minimum replicable repo, super useful. I've come across the same issue. Let me raise it internally :)

[purkhusid]

Any update? It's pretty much impossible to try to build any custom images at the moment.

[geropl]

Hm, this sounds like a DinD issue we saw a few days ago in another environment where we use Docker in Docker as well: Here image-builder uses it to build the base image for the actual workspace.

Google basically enabled some form of protection against a CVE a few days back, which somehow makes DinD containers loose packages (we saw roughly 40% in some cases, which in turn breaks a lot of downloads).

We'll have a closer look andsee if that's the issue.

[csweichel]

This issue increases the priority for the image-builder-mk3 rollout. Once #5693 is merged, we'll deploy that one which should fix this issue, too.

[csweichel]

Root cause of this issue: https://status.cloud.google.com/incidents/QSirAFiyN5yMeeE6GNxq?_ga=2.66285422.-1381644037.1603351900

[csweichel]

/schedule

[roboquat]

@csweichel: Issue scheduled in the workspace team (WIP: 0)

In response to this:

/schedule

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[aledbf]

Closing. This GKE issue is fixed.