Epic: Workspace component integration tests

[kylos101]

Summary

Workspace component integration tests are fixed to reliably pass, run daily against main, and report status in team channel.

Context

The integration tests compile to separate binaries and can be executed individually via this script. Additionally, Leeway builds them as an asset, and we can [run them in werft].

Tasks:

• Fix broken integration test
  • Fix broken integration tests TestBaseImageBuild and TestParallelBaseImageBuild #8800
  • Fix the workspace integration tests that need a username in DB first #9627
  • Fix broken integration test TestRegularWorkspaceTasks #9575
  • Workspace integration test TestBaseImageBuild and TestParallelBaseImageBuild failed again #9782
  • Fix broken workspace integration test TestMissingBackup(PVC) #9990 (Moves to Epic: Ensure durability for user workspace files #7901)
• Enhance integration test
  • Enhance the workspace integration tests TestUploadDownloadBlob and TestUploadDownloadBlobViaServer #9953
• Add new integration test
  • Add integration test for Git LFS #8122
• Nightly integration test trigger by werft
  • Run workspace integration tests nightly against main on core-dev #9597 (on core-dev environment)
  • Change workspace integration tests nightly from core-dev to preview environment #9749 (on preview environment)

Value

We will know to the day (not necessarily which PR) if workspace components are passing or failing. This will help us have more confidence in the tests, ship with confidence (assuming tests are passing), and identify breakage before shipping to production.

Acceptance criteria

Tests are run daily against default branch and results are shared in the team workspace Slack channel #workspace-jobs.

Complexities

1. Werft jobs for the Gitpod mono repo were recently refactored to accommodate the new with-vm annotation. Now, the default environment is preview environment (with-vm annotation enabled by default).
2. Tests are compiled into separate binaries.
3. The entry point for tests will need to be modified, so that from a werft job, we can specify parameters that make test run conditional by component name. For example, this way, we can have an exclusive workspace team werft job that only runs workspace component tests.

[kylos101]

Hi @jenting 👋 , this is the epic tracking the issues (work breakdown) for our team workspace integration tests. May I ask you to review, and let me know if you have any questions?

It'd be ideal if you familiarized yourself with running the tests via go and werft, prior to getting started with #8800.

[jenting]

The integration test result https://werft.gitpod-dev.com/job/gitpod-custom-jenting-inte-test.0
The only one left failed test case is tracking under #7901.

@kylos101 @atduarte should we close this issue? Feel free to close it once you think it's done.

[kylos101]

Amazing work, @jenting ! Thank you for looping in @princerachit and @utam0k to help support you on this effort, too.

@jenting let's wait till the next scheduled run is complete and successful, so we can see success in Slack. That should be tomorrow, right?

@atduarte this is a tech debt epic, and almost done. We wanted to consult with you before closing. Is there anything else we need to do?

[atduarte]

@kylos101 agree with you that we should wait for tomorrow to validate that the scheduling is working and the run is successful. Will it continue to fail because of TestMissingBackup/CreateWorkspace/it_can_run_workspace_tasks/pvc_backup_init? Would be preferable to avoid broken windows.

[kylos101]

No, the plan is to ignore that test (@jenting will do, we spoke about earlier), this way we always see success in Slack, unless there's a real issue. 💪

Tests can fail due to GCR rate limiting, which is being mitigated by this bot.