Configure ide integration test to run automatically #7380

jeanp413 · 2021-12-29T10:44:31Z

Description

Configure ide integration test to run automatically inside a werft job.
Requires USER_TOKEN secret to inject a user inside DB

Related Issue(s)

Related #3065

How to test

Before merged:

Click Gitpod button or open this https://gitpod.io/#github.com/gitpod-io/gitpod/pull/7380
Run in terminal werft run github -j .werft/integration-tests-startup.yaml -f. It will start 3 werft jobs Configure ide integration test to run automatically #7380 (comment)
For now, after output [version] jp-run-integration-test.*, you can go to step3 check the latest job's log
Check in https://werft.gitpod-dev.com/ for a gitpod-run-integration-tests-ide-main.* job and check the ide test run, right now one test is failing but that's independant of this PR. Check logs of werft, you'll see

After merged:

Open any workspace
Do like Before merged step 2\3

Release Notes

NONE

Documentation

/werft no-preview

jeanp413 · 2021-12-30T17:05:20Z

/werft run with-helm

👍 started the job as gitpod-build-jp-run-integration-test.7

jeanp413 · 2021-12-30T17:42:12Z

/werft run with-helm

👍 started the job as gitpod-build-jp-run-integration-test.9

jeanp413 · 2021-12-31T03:12:17Z

/werft run with-helm

👍 started the job as gitpod-build-jp-run-integration-test.12

jeanp413 · 2021-12-31T03:28:17Z

/werft run with-helm

👍 started the job as gitpod-build-jp-run-integration-test.13

jeanp413 · 2021-12-31T04:31:07Z

/werft run with-helm

👍 started the job as gitpod-build-jp-run-integration-test.15

jeanp413 · 2022-01-03T20:10:20Z

hey @csweichel, I'm trying to run the ide integration test automatically and for that I need a default user I could use to run the test, I tried to use the default user builtin-workspace-prober and added some missing entries in d_b_identity and d_b_token_entry that were required but in the end, it looks I need a valid github token as internally this github query is made and it fails with an invalid token. @akosyakov mentioned that maybe there are some other default users injected in staging that I could use instead of builtin-workspace-prober where can I find those users? or in any case, do you know a proper way to get this token or a workaround it?

csweichel · 2022-01-04T09:15:36Z

I reckon we'll want to pass a GitHub token to the test and create a new user using that token. Relying on pre-existing users is very fragile and binds us to particular environments.

jeanp413 · 2022-01-05T04:33:26Z

@csweichel where/how can I get/generate this github token? This new user that you mention, should I create it using a db migration or should I follow another approach?

csweichel · 2022-01-05T13:12:38Z

@csweichel where/how can I get/generate this github token?

The GitHub token would be injected as parameter to the test. When executing the test in a CI setting, we'd take the GitHub token from a secret or the CI system itself.

This new user that you mention, should I create it using a db migration or should I follow another approach?

We should create this new user in the integration test on-demand, i.e. add the DB entries so that the user exists. Akin to how the buildin user DB migration works, except not a migration but in Go code and connected to an identity which carries the aforementioned GitHub token.

codecov · 2022-01-12T20:06:06Z

Codecov Report

Merging #7380 (07b58c4) into main (a21f568) will decrease coverage by 3.88%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #7380      +/-   ##
==========================================
- Coverage   15.06%   11.17%   -3.89%     
==========================================
  Files          51       18      -33     
  Lines        4899      993    -3906     
==========================================
- Hits          738      111     -627     
+ Misses       4089      880    -3209     
+ Partials       72        2      -70

Flag	Coverage Δ
components-gitpod-cli-app	`11.17% <ø> (ø)`
components-local-app-app-darwin-amd64	`?`
components-local-app-app-darwin-arm64	`?`
components-local-app-app-linux-amd64	`?`
components-local-app-app-linux-arm64	`?`
components-local-app-app-windows-386	`?`
components-local-app-app-windows-amd64	`?`
components-local-app-app-windows-arm64	`?`
components-ws-daemon-app	`?`
components-ws-daemon-lib	`?`
install-installer-raw-app	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...staller/pkg/components/ws-manager/networkpolicy.go
components/ws-daemon/pkg/content/archive.go
components/ws-daemon/pkg/internal/session/store.go
components/ws-daemon/pkg/content/hooks.go
install/installer/pkg/common/display.go
components/ws-daemon/pkg/content/service.go
components/ws-daemon/pkg/daemon/markunmount.go
install/installer/pkg/common/storage.go
components/ws-daemon/pkg/quota/xfs.go
...mponents/ws-daemon/pkg/daemon/cgroup_customizer.go
... and 23 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a21f568...07b58c4. Read the comment docs.

jeanp413 · 2022-01-17T16:20:18Z

/werft run with-helm

👍 started the job as gitpod-build-jp-run-integration-test.19

jeanp413 · 2022-01-18T15:30:14Z

/werft run with-helm

👍 started the job as gitpod-build-jp-run-integration-test.21

jeanp413 · 2022-01-18T17:50:20Z

/werft run with-helm

👍 started the job as gitpod-build-jp-run-integration-test.22

jeanp413 · 2022-01-19T03:40:56Z

/werft run with-helm

👍 started the job as gitpod-build-jp-run-integration-test.24

iQQBot · 2022-02-24T08:26:42Z

@mustard-mh @filiptronicek could you try for a test?

mustard-mh · 2022-02-24T08:40:41Z

I updated How to test section

mustard-mh · 2022-02-24T08:54:07Z

How to check the ide test run?

mustard-mh · 2022-02-24T09:00:45Z

Test env build took a long time...

After build, we should output our task job gitpod-run-integration-tests-ide-main.18 but not [version] jp-run-integration-test.80

With job gitpod-run-integration-tests-ide-main.18, we can see job log

mustard-mh · 2022-02-24T09:01:11Z

I'll update how to test section later

iQQBot · 2022-02-24T09:01:48Z

The startup job will trigger 3 job

build a preview environment on main branch, the name should be like gitpod-test-main.*
start integration test in new job, the name should be like gitpod-run-integration-tests-ide-main.*
start wipe preview environment, the name should be like gitpod-wipe-devstaging-main.*

akosyakov · 2022-02-28T08:59:36Z

@jeanp413 @iQQBot I tried, but now I'm getting an error like

[int-tests]  --- FAIL: TestPythonExtWorkspace (25.29s)
[int-tests]      --- FAIL: TestPythonExtWorkspace/PythonExtensionWorkspace (25.29s)
[int-tests]          --- FAIL: TestPythonExtWorkspace/PythonExtensionWorkspace/it_can_run_python_extension_in_a_workspace (25.29s)
[int-tests]              python_ws_test.go:65: cannot start workspace: "workspace instance e62a2fff-83ba-4024-b66c-b72878f95333 failed: container workspace completed; containers of a workspace pod are not supposed to do that"

It seems to be related to our latests security measurements, maybe it should be disabled for test envs?

iQQBot · 2022-02-28T09:02:18Z

Yes, it's relate #8476

csweichel · 2022-02-28T09:58:48Z

.werft/integration-tests-startup.yaml

+      # this is because we don't want to increate gitpod-build-main number, we can choose a name
+      sudo cp .werft/build.yaml .werft/test.yaml
+      # version is actual namespace that werft is build, --job-file should be same with above
+      VERSION=$(werft run github -j .werft/test.yaml -s .werft/test.yaml github.com/gitpod-io/gitpod:main)


if you use --follow-with-prefix, the run will wait until the other job is done and integrate the log output correctly.

The main purpose here is to get the build ID in order to get the version number of the corresponding integration-test in the next step

kylos101 · 2022-02-28T14:37:30Z

Hey there, as a heads up, existing integration tests are broken (this PR did not break them), I am just sharing for awareness).

These are the most recent.

These are older but used to pass.

iQQBot · 2022-03-01T13:44:35Z

So lets we go and merge this, I will delete some debug message

iQQBot · 2022-03-01T14:45:27Z

Hey there, as a heads up, existing integration tests are broken (this PR did not break them), I am just sharing for awareness).

These are the most recent.

These are older but used to pass.

because this branch not have preview environment

iQQBot · 2022-03-01T15:12:48Z

All test pass, except it_can_start_a_ghost_workspace This is normal

iQQBot · 2022-03-01T16:02:22Z

/unhold

roboquat added do-not-merge/work-in-progress release-note-none size/XS size/L and removed size/XS labels Dec 29, 2021

jeanp413 force-pushed the jp/run-integration-test branch 2 times, most recently from db1ae6a to d19eb52 Compare December 29, 2021 15:41

roboquat added the team: webapp Issue belongs to the WebApp team label Dec 30, 2021

jeanp413 force-pushed the jp/run-integration-test branch from f5b519d to 6fb28a5 Compare December 30, 2021 17:26

jeanp413 force-pushed the jp/run-integration-test branch from f0f7232 to b042836 Compare December 31, 2021 02:39

jeanp413 force-pushed the jp/run-integration-test branch from b042836 to 8f2bac5 Compare December 31, 2021 04:30

jeanp413 force-pushed the jp/run-integration-test branch from 8f2bac5 to bc48b5d Compare January 12, 2022 19:56

jeanp413 force-pushed the jp/run-integration-test branch 2 times, most recently from 6f318a7 to 49a2e1d Compare January 17, 2022 15:58

jeanp413 force-pushed the jp/run-integration-test branch from 49a2e1d to 81b18c6 Compare January 18, 2022 15:30

jeanp413 force-pushed the jp/run-integration-test branch from 81b18c6 to 3f93ea3 Compare January 19, 2022 03:40

roboquat added the do-not-merge/hold label Feb 24, 2022

iQQBot previously approved these changes Feb 24, 2022

View reviewed changes

iQQBot dismissed their stale review via fc76810 February 24, 2022 11:25

iQQBot requested a review from a team February 27, 2022 16:31

iQQBot self-assigned this Feb 27, 2022

iQQBot mentioned this pull request Feb 28, 2022

Jetbrains integration smoke test #8180

Merged

csweichel reviewed Feb 28, 2022

View reviewed changes

loujaybee added the team: IDE label Mar 1, 2022

iQQBot force-pushed the jp/run-integration-test branch from fc76810 to e0f313f Compare March 1, 2022 15:35

jeanp413 and others added 2 commits March 1, 2022 15:49

Run ide integration test in werft job

86b3fec

update script

07b58c4

iQQBot force-pushed the jp/run-integration-test branch from e0f313f to 07b58c4 Compare March 1, 2022 16:02

roboquat removed the do-not-merge/hold label Mar 1, 2022

iQQBot approved these changes Mar 1, 2022

View reviewed changes

roboquat merged commit 9cb8a91 into main Mar 1, 2022

roboquat deleted the jp/run-integration-test branch March 1, 2022 16:06

roboquat added the deployed: IDE IDE change is running in production label Mar 2, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Configure ide integration test to run automatically #7380

Configure ide integration test to run automatically #7380

jeanp413 commented Dec 29, 2021 •

edited by mustard-mh

Loading

jeanp413 commented Dec 30, 2021 •

edited by werft-gitpod-dev-com bot

Loading

jeanp413 commented Dec 30, 2021 •

edited by werft-gitpod-dev-com bot

Loading

jeanp413 commented Dec 31, 2021 •

edited by werft-gitpod-dev-com bot

Loading

jeanp413 commented Dec 31, 2021 •

edited by werft-gitpod-dev-com bot

Loading

jeanp413 commented Dec 31, 2021 •

edited by werft-gitpod-dev-com bot

Loading

jeanp413 commented Jan 3, 2022 •

edited

Loading

csweichel commented Jan 4, 2022

jeanp413 commented Jan 5, 2022

csweichel commented Jan 5, 2022

codecov bot commented Jan 12, 2022 •

edited

Loading

jeanp413 commented Jan 17, 2022 •

edited by werft-gitpod-dev-com bot

Loading

jeanp413 commented Jan 18, 2022 •

edited by werft-gitpod-dev-com bot

Loading

jeanp413 commented Jan 18, 2022 •

edited by werft-gitpod-dev-com bot

Loading

jeanp413 commented Jan 19, 2022 •

edited by werft-gitpod-dev-com bot

Loading

iQQBot commented Feb 24, 2022

mustard-mh commented Feb 24, 2022

mustard-mh commented Feb 24, 2022

mustard-mh commented Feb 24, 2022 •

edited

Loading

mustard-mh commented Feb 24, 2022

iQQBot commented Feb 24, 2022

akosyakov commented Feb 28, 2022

iQQBot commented Feb 28, 2022

csweichel Feb 28, 2022

iQQBot Mar 1, 2022

kylos101 commented Feb 28, 2022

iQQBot commented Mar 1, 2022

iQQBot commented Mar 1, 2022

iQQBot commented Mar 1, 2022

iQQBot commented Mar 1, 2022

Configure ide integration test to run automatically #7380

Configure ide integration test to run automatically #7380

Conversation

jeanp413 commented Dec 29, 2021 • edited by mustard-mh Loading

Description

Related Issue(s)

How to test

Release Notes

Documentation

jeanp413 commented Dec 30, 2021 • edited by werft-gitpod-dev-com bot Loading

jeanp413 commented Dec 30, 2021 • edited by werft-gitpod-dev-com bot Loading

jeanp413 commented Dec 31, 2021 • edited by werft-gitpod-dev-com bot Loading

jeanp413 commented Dec 31, 2021 • edited by werft-gitpod-dev-com bot Loading

jeanp413 commented Dec 31, 2021 • edited by werft-gitpod-dev-com bot Loading

jeanp413 commented Jan 3, 2022 • edited Loading

csweichel commented Jan 4, 2022

jeanp413 commented Jan 5, 2022

csweichel commented Jan 5, 2022

codecov bot commented Jan 12, 2022 • edited Loading

Codecov Report

jeanp413 commented Jan 17, 2022 • edited by werft-gitpod-dev-com bot Loading

jeanp413 commented Jan 18, 2022 • edited by werft-gitpod-dev-com bot Loading

jeanp413 commented Jan 18, 2022 • edited by werft-gitpod-dev-com bot Loading

jeanp413 commented Jan 19, 2022 • edited by werft-gitpod-dev-com bot Loading

iQQBot commented Feb 24, 2022

mustard-mh commented Feb 24, 2022

mustard-mh commented Feb 24, 2022

mustard-mh commented Feb 24, 2022 • edited Loading

mustard-mh commented Feb 24, 2022

iQQBot commented Feb 24, 2022

akosyakov commented Feb 28, 2022

iQQBot commented Feb 28, 2022

csweichel Feb 28, 2022

Choose a reason for hiding this comment

iQQBot Mar 1, 2022

Choose a reason for hiding this comment

kylos101 commented Feb 28, 2022

iQQBot commented Mar 1, 2022

iQQBot commented Mar 1, 2022

iQQBot commented Mar 1, 2022

iQQBot commented Mar 1, 2022

jeanp413 commented Dec 29, 2021 •

edited by mustard-mh

Loading

jeanp413 commented Dec 30, 2021 •

edited by werft-gitpod-dev-com bot

Loading

jeanp413 commented Dec 30, 2021 •

edited by werft-gitpod-dev-com bot

Loading

jeanp413 commented Dec 31, 2021 •

edited by werft-gitpod-dev-com bot

Loading

jeanp413 commented Dec 31, 2021 •

edited by werft-gitpod-dev-com bot

Loading

jeanp413 commented Dec 31, 2021 •

edited by werft-gitpod-dev-com bot

Loading

jeanp413 commented Jan 3, 2022 •

edited

Loading

codecov bot commented Jan 12, 2022 •

edited

Loading

jeanp413 commented Jan 17, 2022 •

edited by werft-gitpod-dev-com bot

Loading

jeanp413 commented Jan 18, 2022 •

edited by werft-gitpod-dev-com bot

Loading

jeanp413 commented Jan 18, 2022 •

edited by werft-gitpod-dev-com bot

Loading

jeanp413 commented Jan 19, 2022 •

edited by werft-gitpod-dev-com bot

Loading

mustard-mh commented Feb 24, 2022 •

edited

Loading