Cirrus: Update to F37 CI VM Images

[cevich]

No description provided.

[cevich]

@mtrmac PTAL, this seems like a really odd error to get on a F36 -> F37 image update. Any idea what I might have screwed up in the image?

[mtrmac]

Just to link things together, #1739 is failing similarly.

I don’t really have any idea; notably the later registry tests successfully use podman to run a registry container, so the configuration on the system can’t be that broken.

Purely as a lazy guess, something from

skopeo/systemtest/helpers.bash

Lines 304 to 312 in e024c43

	local runtime=
	cgroup_type=$(stat -f -c %T /sys/fs/cgroup)
	if [[ $cgroup_type == "tmpfs" ]]; then
	runtime="--runtime runc"
	fi
	# cgroup option necessary under podman-in-podman (CI tests),
	# and doesn't seem to do any harm otherwise.
	PODMAN="podman $runtime --cgroup-manager=cgroupfs"

might make a difference, but I don’t understand at all how that could be related.

[cevich]

Just to link things together, #1739 is failing similarly.

Good, so at least we know it's not related at all to the package removal changes in @lsm5 PR. The issue must be due to an updated package. That diff will be smaller in @lsm5 PR than this one, but still could be large. I'm thinking the best approach is probably to go hands-on with get_ci_vm.sh, reproduce it. Then start pressing random buttons until it either explodes or starts working 😁

[cevich]

Purely as a lazy guess, something from

Hmmm, so I wonder...maybe the difference is in the container image, not the VM image.

[cevich]

Force-push: newer images. Not expecting a different result.

[cevich]

Force-push: newer images. Not expecting a different result.

[mtrmac]

(FWIW, I realize that this is important work that will inevitably need to happen. At this point I’m prioritizing the sigstore feature work, given upcoming deadlines; let me know if that’s not the right judgement call.)

[cevich]

At this point I’m prioritizing the sigstore feature work

Please do, and ignore this for now. I think I know what the next step[*] is to move this and @lsm5 PR forward. I'll ping you (likely next year) if I need help.

[*] Use get_ci_vm.sh and reproduce the failure, and try to expose what changed in the environment to cause it. Specifically, paying attention to runc/crun stuffs.

[cevich]

Now this is weird...when I run the system tests manually via get_ci_vm.sh Skopeo Test they all pass. The only significant difference I'm aware of is that I did not first run the integration tests. I'll play with this setup more, to see if maybe the integration tests are somehow breaking the system tests...

[cevich]

...I cannot reproduce the problem manually on a get_ci_vm.sh. The tests pass no matter how or what order I run (int. vs. sys.) them. This is bringing back some memories now. IIRC Lokesh mentioned this fact to me last year about his PR. Maybe I'll have some luck using the 're-run w/ terminal' button...

[cevich]

...random idea: updated to images w/ older kernels (minus the multiple-port bindings bug).

[cevich]

Hmmm, so that change simply kicked the can down the road. It def. seems like the setup is wonky, but I cannot explain why running this stuff manually works perfectly fine. That's what has me really puzzled.

[cevich]

@lsm5 @mtrmac uh huh, well, I don't know how or why 0e94474 did it, but the tests are fixed now 😁

P.S. The commit message I wrote came directly out of the hole which I normally sit upon. It's a complete guess - I was never able to reproduce the test failures manually 😕

[mtrmac]

It’s weird that podman system reset reports A storage.conf file exists at /usr/share/containers/storage.conf but then podman run seems to complain that the config file doesn’t contain the standard options.

It’s also weird that the podman run … /bin/true does not contain the error messages that were plaguing us before, on the first run of a podman run command. Does either of --runtime runc (if used) and --cgroup-manager=cgroupfs make a difference in this behavior?

---

Either way… if there isn’t a way to investigate further, and this works, let’s merge.

Just, please, explicitly write down (in the runner.sh comments and the commit message) that we don’t know why this works / is necessary, and in one of the places document what failures this is working around. Documenting the hypothesis is useful as well, but to me the important thing is that in 3 years we will know not to treat that hypothesis as a certainty.

[cevich]

It’s weird that podman system reset reports

To be clear, it's not the system reset that's new, that's always been there. The thing that "fixed" this issue is the podman run ... true. Suggesting something in the database or other runtime state is mucking with the tests. Likely by changing the output (which system tests are particularly sensitive to).

Either way… if there isn’t a way to investigate further, and this works, let’s merge.

I tried but couldn't reproduce the problem using hack/get_ci_vm.sh, the tests all pass w/o error. I also tried running the tests in a different order, which similarly did not seem to help - so test-order isn't to blame 😕

Just, please, explicitly write down

Good suggestion, thanks.

[mtrmac]

it's not the `system reset that's new, that's always been there.

True; I meant that it seems to be saying something about the configuration, as interpreted at that point.

[mtrmac]

Thanks! LGTM.

[cevich]

I took a quick stroll through the system-test code-path. I believe do to legacy reasons, the system-tests modify the host's container storage configuration. IIRC, the system-tests in particular are very sensitive to unexpected output from podman. If I had to make a guess, the problem is fixed (by running a container) because that prevents certain warning messages from being printed at test-time. It's a wild-guess though.

[cevich]

Force-push: Rebased.