Run Skupper integration tests with the skupper-router image

[jiridanek]

Build the image for pull requests. Only push the image if we are on main branch. Run skupper integration tests with the image before push.

[jiridanek]

2022-03-11T21:46:44.2939778Z --- FAIL: TestGateway/local-gateway-podman (6.45s)

That happened because the image is only present among docker images and not among podman images.

[jiridanek]

--- SKIP: TestGateway/local-gateway-service (0.03s)

That is because the systemd gateway test expects to find qdrouterd binary on the PATH (on the machine running the test). Clear evidence that #141 will require renames in skupper repository as well.

I'm inclined to leave this test skipped, because installing dispatch is inconvenient...

[fgiorgetti]

why not leaving master as the default?

[jiridanek]

My thinking is that we want to catch issues in skupper-router here. If both skupper and skupper-router are changing from one run of the job to the next, we will never be able to guess which is the culprit of a failure.

So, I want to have the latest known working version of skupper here. Using latest release is the only way I can think of to get a working version. I could somehow figure out how to find the latest commit on master that passed all skupper tests, for example... Or possibly even something more complicated.

[fgiorgetti]

Ok, it might be better using 0.8 instead, in this case.

[jiridanek]

Certainly not. There has to be a skupper version that works with the skupper-router after the @kgiusti 's renaming. So, at first, I expect there will have to be a commit hash of whatever version where you fix skupper for it. Then, going forward, we can pick some suitable release, yes.

[fgiorgetti]

you can remove -p=1 here, which would speed things up

[jiridanek]

That is a good call, I probably should do that. I worried the output would become less clear, but the way it is now is simply too slow to be useful. Removing -p1 it is.

[jiridanek]

-p=3 now, and it passed a few times in a row

[jiridanek]

At this point, there is one fundamental question that needs to be asked. @kgiusti @ganeshmurthy (@cliffjansen) can you live with a CI test that runs 40 minutes and if it fails, it is quite unclear why it failed? (even more unclear than the system-tests)?

To summarize, this brings in integration-tests from skupper, from some known-working version of skupper, but puts in the latest skupper-router image. Then the tests run. These are the same tests that Renato is (was?, still is?) so unhappy about.

[codecov]

Codecov Report

Patch and project coverage have no change.

Comparison is base (d83e009) 30.85% compared to head (f12570c) 30.85%.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #169   +/-   ##
=======================================
  Coverage   30.85%   30.85%           
=======================================
  Files         174      174           
  Lines       38052    38052           
  Branches     5331     5331           
=======================================
  Hits        11741    11741           
  Misses      25155    25155           
  Partials     1156     1156           

Flag	Coverage Δ
pyunittests	54.50% <ø> (ø)
unittests	30.85% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[jiridanek]

https://github.com/skupperproject/skupper-router/actions/runs/4584747691/jobs/8096438425?pr=169#step:4:744

go: downloading github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578
client/podman/container.go:12:2: no required module provides package github.com/skupperproject/skupper/client/generated/libpod/client/containers; to add it:
	go get github.com/skupperproject/skupper/client/generated/libpod/client/containers
client/podman/container.go:13:2: no required module provides package github.com/skupperproject/skupper/client/generated/libpod/client/exec; to add it:
	go get github.com/skupperproject/skupper/client/generated/libpod/client/exec
client/podman/image.go:9:2: no required module provides package github.com/skupperproject/skupper/client/generated/libpod/client/images; to add it:
	go get github.com/skupperproject/skupper/client/generated/libpod/client/images
client/podman/network.go:10:2: no required module provides package github.com/skupperproject/skupper/client/generated/libpod/client/networks; to add it:
	go get github.com/skupperproject/skupper/client/generated/libpod/client/networks
client/podman/version.go:5:2: no required module provides package github.com/skupperproject/skupper/client/generated/libpod/client/system; to add it:
	go get github.com/skupperproject/skupper/client/generated/libpod/client/system
client/podman/volume.go:8:2: no required module provides package github.com/skupperproject/skupper/client/generated/libpod/client/volumes; to add it:
	go get github.com/skupperproject/skupper/client/generated/libpod/client/volumes
client/podman/container.go:14:2: no required module provides package github.com/skupperproject/skupper/client/generated/libpod/models; to add it:
	go get github.com/skupperproject/skupper/client/generated/libpod/models
make: *** [Makefile:23: build-cmd] Error 1
Error: Process completed with exit code 2.

@fgiorgetti does this by any chance mean anything to you? I'm thinking maybe I'm missing something obvious. If not, I will consider investigating further (or maybe burying this PR and never coming back ;)