Fix broken integration tests TestBaseImageBuild and TestParallelBaseImageBuild

[kylos101]

1. Assess which tests are passing and failing.
2. Fix broken tests for workspace components.
3. If other tests are failing, but not related to workspace components, create a separate issue to track that breakage.

[kylos101]

@jenting when you feel ready to start this, may I ask you to move this issue's status from Scheduled to In-Progress? 🙏

[jenting]

Failed E2E test cases

• TestBaseImageBuild ✅
• TestParallelBaseImageBuild ✅
• TestRegularWorkspaceTasks

[jenting]

These two E2E tests TestBaseImageBuild and TestParallelBaseImageBuild failed because the secret gcp-sa-registry-auth does not render correctly which is deployed by the werft.

It should be fixed in the werft deployment script.

[kylos101]

These two E2E tests TestBaseImageBuild and TestParallelBaseImageBuild failed because the secret gcp-sa-registry-auth does not render correctly which is deployed by the werft.

It should be fixed in the werft deployment script.

Hi @jenting I triggered an image build in this PR's preview envirionment with this URL, and it works. Here is the related workspace that I started. 👍 Cool!

These two tests pass when I run with go. Do they pass with werft too, or, is that where you bump into the issue with gcp-sa-registry-auth?

1. TestBaseImageBuild
2. TestParallelBaseImageBuild

Given:

go test -v ./... \
   -kubeconfig=/home/gitpod/.kube/config \
   -namespace=staging-jenting-fi2f9fc7a4ea \
   -username=kylos101 \
   -enterprise=false \
   -gitlab=false \
   -run=TestRegularWorkspaceTasks

I encountered failures, but, assume you are debugging:

?       github.com/gitpod-io/gitpod/test/pkg/agent/daemon       [no test files]
?       github.com/gitpod-io/gitpod/test/pkg/agent/daemon/api   [no test files]
?       github.com/gitpod-io/gitpod/test/pkg/agent/workspace    [no test files]
?       github.com/gitpod-io/gitpod/test/pkg/agent/workspace/api        [no test files]
?       github.com/gitpod-io/gitpod/test/pkg/integration        [no test files]
?       github.com/gitpod-io/gitpod/test/pkg/integration/common [no test files]
testing: warning: no tests to run
PASS
ok      github.com/gitpod-io/gitpod/test/tests/components/content-service       2.604s [no tests to run]
testing: warning: no tests to run
PASS
ok      github.com/gitpod-io/gitpod/test/tests/components/database      2.605s [no tests to run]
testing: warning: no tests to run
PASS
ok      github.com/gitpod-io/gitpod/test/tests/components/image-builder 2.637s [no tests to run]
testing: warning: no tests to run
PASS
ok      github.com/gitpod-io/gitpod/test/tests/components/server        2.611s [no tests to run]
testing: warning: no tests to run
PASS
ok      github.com/gitpod-io/gitpod/test/tests/components/ws-daemon     2.600s [no tests to run]
=== RUN   TestRegularWorkspaceTasks
=== RUN   TestRegularWorkspaceTasks/ws-manager
=== RUN   TestRegularWorkspaceTasks/ws-manager/it_can_run_workspace_tasks
=== RUN   TestRegularWorkspaceTasks/ws-manager/it_can_run_workspace_tasks/init
    tasks_test.go:96: rpc error: code = Unavailable desc = connection closed
=== RUN   TestRegularWorkspaceTasks/ws-manager/it_can_run_workspace_tasks/before
    tasks_test.go:96: rpc error: code = Unavailable desc = connection closed
=== RUN   TestRegularWorkspaceTasks/ws-manager/it_can_run_workspace_tasks/command
    tasks_test.go:96: rpc error: code = Unavailable desc = connection closed
--- FAIL: TestRegularWorkspaceTasks (34.85s)
    --- FAIL: TestRegularWorkspaceTasks/ws-manager (34.85s)
        --- FAIL: TestRegularWorkspaceTasks/ws-manager/it_can_run_workspace_tasks (34.85s)
            --- FAIL: TestRegularWorkspaceTasks/ws-manager/it_can_run_workspace_tasks/init (15.72s)
            --- FAIL: TestRegularWorkspaceTasks/ws-manager/it_can_run_workspace_tasks/before (10.05s)
            --- FAIL: TestRegularWorkspaceTasks/ws-manager/it_can_run_workspace_tasks/command (9.07s)
FAIL
FAIL    github.com/gitpod-io/gitpod/test/tests/components/ws-manager    37.452s
testing: warning: no tests to run
PASS
ok      github.com/gitpod-io/gitpod/test/tests/ide/jetbrains    2.597s [no tests to run]
testing: warning: no tests to run
PASS
ok      github.com/gitpod-io/gitpod/test/tests/ide/vscode       2.605s [no tests to run]
testing: warning: no tests to run
PASS
ok      github.com/gitpod-io/gitpod/test/tests/workspace        2.721s [no tests to run]
?       github.com/gitpod-io/gitpod/test/tests/workspace/common [no test files]
FAIL

Let me know if you need anything?

[jenting]

These two tests pass when I run with go. Do they pass with werft too, or, is that where you bump into the issue with gcp-sa-registry-auth?

Yes, I believe it would pass with werft because werft uses the same secret in the testing namespace.

1. TestBaseImageBuild
2. TestParallelBaseImageBuild

Given:

go test -v ./... \
   -kubeconfig=/home/gitpod/.kube/config \
   -namespace=staging-jenting-fi2f9fc7a4ea \
   -username=kylos101 \
   -enterprise=false \
   -gitlab=false \
   -run=TestRegularWorkspaceTasks

I'll start to debug the test case TestRegularWorkspaceTasks failed. The failure message is different from the above two test cases.

[jenting]

The culprit of TestRegularWorkspaceTasks test failure seems like the port forwarding not working correctly.

Let's say this E2E test creates a temporary workspace Pod ws-xxx, the Pod runs the supervisor container image.
The supervisor endpoint :22999 is available within the Pod. After that, the E2E test creates the port-forward to this temporary workspace Pod, for example, kubectl port-forward --address=0.0.0.0 pod/<ws-xxx> <localPort>:22999, and then perform grpc request to the localhost:<localPort> in order to get the task status.
But somehow, the grpc request can't reach the Pods ws-xxx port 22999 via the port forwarding way.

[kylos101]

@jenting can you close this issue, create a new one for TestRegularWorkspaceTasks, and reach out to the IDE team for help in the new issue? Also, for the new issue, add it to the list of tasks in #8799, at the end. 🙏

This way #8800 is done, and you can move onto other issues in #8799, like sharing the status of tests in Slack. 😄 👍

[jenting]

Create a new issue #9575 to address to failed E2E TestRegularWorkspaceTasks because it involved cross IDE team as well as workspace team.

Close this issue.