Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lots of failures on Cirrus CI / nightly freebsd-15 #3967

Closed
tgross35 opened this issue Oct 15, 2024 · 6 comments · Fixed by #3988
Closed

Lots of failures on Cirrus CI / nightly freebsd-15 #3967

tgross35 opened this issue Oct 15, 2024 · 6 comments · Fixed by #3988
Labels
A-CI Area: CI-related items O-bsd

Comments

@tgross35
Copy link
Contributor

Almost everything has been failing at least once on freebsd-15, e.g.:

I don't know why this would be, considering freebsd-13 and freebsd-14 seem to work fine.

Cc @tesuji (since you did the CI matrix recently) @asomers

@tgross35 tgross35 added A-CI Area: CI-related items O-bsd labels Oct 15, 2024
@asomers
Copy link
Contributor

asomers commented Oct 15, 2024

Something must be broken in the gcp image. I'll take a look tomorrow, if I have the time. In the mean time, we should pin CI to the last known good image.

@tgross35
Copy link
Contributor Author

tgross35 commented Oct 15, 2024

Any chance you know which one would be good, or even what the options are? I'm not sure why we seem to be using an image with snap, but I can't find the image list anywhere.

Thanks for the quick reply.

@tesuji
Copy link
Contributor

tesuji commented Oct 15, 2024

I don't know the root cause of this, but I think sometimes CirrusCI fails to download the GCE image.
Maybe we are on free CirrusCI tier so the build image agents are delayed or we hit the rate limits 🤷 .
The freeebsd15-snap image is listed here: https://cirrus-ci.org/guide/FreeBSD/#list-of-available-image-families.
Also if you have a gcloud account, you could check what freebsd images there are with

gcloud compute images list --no-standard-images --project=freebsd-org-cloud-dev

I don't have any gcloud accounts to check so I'm sorry I can't help.

asomers added a commit to asomers/libc that referenced this issue Oct 15, 2024
The latest FreeBSD 15 GCE image has a problem that prevents it from
working with Cirrus CI.  As a result, libc's CI is broken.  Try the
latest 3 images to see if any work.

Issue rust-lang#3967
@asomers
Copy link
Contributor

asomers commented Oct 15, 2024

tesuji's command is the right one. The linked PR tries several different images as an experiment.

asomers added a commit to asomers/libc that referenced this issue Oct 15, 2024
The latest FreeBSD 15 GCE image has a problem that prevents it from
working with Cirrus CI.  As a result, libc's CI is broken.  Try the
latest 6 images to see if any work.

Issue rust-lang#3967
@asomers
Copy link
Contributor

asomers commented Oct 15, 2024

So due to the libmd .so version bump, even old 15.0-CURRENT images aren't working. And the newest images don't work with a different error message, possibly for a different reason. At this point I think the best action for libc to take would be to temporarily disable CI on FreeBSD 15.
Upstream ticket: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=282112

asomers added a commit to asomers/libc that referenced this issue Oct 15, 2024
FreeBSD 15 is the unstable development release.  Currently its GCE
images available to Cirrus CI don't work because the solib version of
libmd was just bumped, and the package builders haven't yet caught up.

Issue rust-lang#3967
asomers added a commit to asomers/libc that referenced this issue Oct 15, 2024
FreeBSD 15 is the unstable development release.  Currently its GCE
images available to Cirrus CI don't work because the solib version of
libmd was just bumped, and the package builders haven't yet caught up.

Issue rust-lang#3967
@tgross35
Copy link
Contributor Author

tgross35 commented Oct 16, 2024

I asked on Zulip, @syphar was able to run the command to get the full list:

NAME                                        PROJECT                FAMILY             DEPRECATED  STATUS
freebsd-13-2-stable-amd64-v20231228         freebsd-org-cloud-dev  freebsd-13-2-snap              READY
freebsd-13-2-stable-amd64-v20240104         freebsd-org-cloud-dev  freebsd-13-2-snap              READY
freebsd-13-2-stable-amd64-v20240111         freebsd-org-cloud-dev  freebsd-13-2-snap              READY
freebsd-13-2-stable-amd64-v20240118         freebsd-org-cloud-dev  freebsd-13-2-snap              READY
freebsd-13-3-prerelease-amd64-v20240125     freebsd-org-cloud-dev  freebsd-13-3-snap              READY
freebsd-13-3-prerelease-amd64-v20240201     freebsd-org-cloud-dev  freebsd-13-3-snap              READY
freebsd-13-3-release-amd64                  freebsd-org-cloud-dev  freebsd-13-3                   READY
freebsd-13-3-stable-amd64-v20240208         freebsd-org-cloud-dev  freebsd-13-3-snap              READY
freebsd-13-3-stable-amd64-v20240215         freebsd-org-cloud-dev  freebsd-13-3-snap              READY
freebsd-13-3-stable-amd64-v20240222         freebsd-org-cloud-dev  freebsd-13-3-snap              READY
freebsd-13-3-stable-amd64-v20240229         freebsd-org-cloud-dev  freebsd-13-3-snap              READY
freebsd-13-3-stable-amd64-v20240307         freebsd-org-cloud-dev  freebsd-13-3-snap              READY
freebsd-13-3-stable-amd64-v20240321         freebsd-org-cloud-dev  freebsd-13-3-snap              READY
freebsd-13-3-stable-amd64-v20240328         freebsd-org-cloud-dev  freebsd-13-3-snap              READY
freebsd-13-3-stable-amd64-v20240404         freebsd-org-cloud-dev  freebsd-13-3-snap              READY
freebsd-13-3-stable-amd64-v20240411         freebsd-org-cloud-dev  freebsd-13-3-snap              READY
freebsd-13-3-stable-amd64-v20240425         freebsd-org-cloud-dev  freebsd-13-3-snap              READY
freebsd-13-3-stable-amd64-v20240502         freebsd-org-cloud-dev  freebsd-13-3-snap              READY
freebsd-13-3-stable-amd64-v20240509         freebsd-org-cloud-dev  freebsd-13-3-snap              READY
freebsd-13-3-stable-amd64-v20240517         freebsd-org-cloud-dev  freebsd-13-3-snap              READY
freebsd-13-3-stable-amd64-v20240523         freebsd-org-cloud-dev  freebsd-13-3-snap              READY
freebsd-13-3-stable-amd64-v20240530         freebsd-org-cloud-dev  freebsd-13-3-snap              READY
freebsd-13-3-stable-amd64-v20240606         freebsd-org-cloud-dev  freebsd-13-3-snap              READY
freebsd-13-3-stable-amd64-v20240613         freebsd-org-cloud-dev  freebsd-13-3-snap              READY
freebsd-13-3-stable-amd64-v20240620         freebsd-org-cloud-dev  freebsd-13-3-snap              READY
freebsd-13-3-stable-amd64-v20240628         freebsd-org-cloud-dev  freebsd-13-3-snap              READY
freebsd-13-3-stable-amd64-v20240705         freebsd-org-cloud-dev  freebsd-13-3-snap              READY
freebsd-13-3-stable-amd64-v20240711         freebsd-org-cloud-dev  freebsd-13-3-snap              READY
freebsd-13-3-stable-amd64-v20240718         freebsd-org-cloud-dev  freebsd-13-3-snap              READY
freebsd-13-4-beta1-amd64                    freebsd-org-cloud-dev  freebsd-13-4                   READY
freebsd-13-4-beta2-amd64                    freebsd-org-cloud-dev  freebsd-13-4                   READY
freebsd-13-4-beta3-amd64                    freebsd-org-cloud-dev  freebsd-13-4                   READY
freebsd-13-4-prerelease-amd64-v20240725     freebsd-org-cloud-dev  freebsd-13-4-snap              READY
freebsd-13-4-prerelease-amd64-v20240801     freebsd-org-cloud-dev  freebsd-13-4-snap              READY
freebsd-13-4-rc1-amd64                      freebsd-org-cloud-dev  freebsd-13-4                   READY
freebsd-13-4-rc2-amd64                      freebsd-org-cloud-dev  freebsd-13-4                   READY
freebsd-13-4-rc3-amd64                      freebsd-org-cloud-dev  freebsd-13-4                   READY
freebsd-13-4-release-amd64                  freebsd-org-cloud-dev  freebsd-13-4                   READY
freebsd-13-4-stable-amd64-v20240808         freebsd-org-cloud-dev  freebsd-13-4-snap              READY
freebsd-13-4-stable-amd64-v20240815         freebsd-org-cloud-dev  freebsd-13-4-snap              READY
freebsd-13-4-stable-amd64-v20240822         freebsd-org-cloud-dev  freebsd-13-4-snap              READY
freebsd-13-4-stable-amd64-v20240829         freebsd-org-cloud-dev  freebsd-13-4-snap              READY
freebsd-13-4-stable-amd64-v20240905         freebsd-org-cloud-dev  freebsd-13-4-snap              READY
freebsd-13-4-stable-amd64-v20240912         freebsd-org-cloud-dev  freebsd-13-4-snap              READY
freebsd-13-4-stable-amd64-v20240919         freebsd-org-cloud-dev  freebsd-13-4-snap              READY
freebsd-13-4-stable-amd64-v20240926         freebsd-org-cloud-dev  freebsd-13-4-snap              READY
freebsd-13-4-stable-amd64-v20241003         freebsd-org-cloud-dev  freebsd-13-4-snap              READY
freebsd-13-4-stable-amd64-v20241010         freebsd-org-cloud-dev  freebsd-13-4-snap              READY
freebsd-14-0-release-amd64-ufs              freebsd-org-cloud-dev  freebsd-14-0                   READY
freebsd-14-0-stable-amd64-ufs-20231228      freebsd-org-cloud-dev  freebsd-14-0-snap              READY
freebsd-14-0-stable-amd64-ufs-20240104      freebsd-org-cloud-dev  freebsd-14-0-snap              READY
freebsd-14-0-stable-amd64-ufs-20240111      freebsd-org-cloud-dev  freebsd-14-0-snap              READY
freebsd-14-0-stable-amd64-ufs-20240118      freebsd-org-cloud-dev  freebsd-14-0-snap              READY
freebsd-14-0-stable-amd64-ufs-20240125      freebsd-org-cloud-dev  freebsd-14-0-snap              READY
freebsd-14-0-stable-amd64-ufs-20240201      freebsd-org-cloud-dev  freebsd-14-0-snap              READY
freebsd-14-0-stable-amd64-ufs-20240208      freebsd-org-cloud-dev  freebsd-14-0-snap              READY
freebsd-14-0-stable-amd64-ufs-20240215      freebsd-org-cloud-dev  freebsd-14-0-snap              READY
freebsd-14-0-stable-amd64-ufs-20240222      freebsd-org-cloud-dev  freebsd-14-0-snap              READY
freebsd-14-0-stable-amd64-ufs-20240229      freebsd-org-cloud-dev  freebsd-14-0-snap              READY
freebsd-14-0-stable-amd64-ufs-20240307      freebsd-org-cloud-dev  freebsd-14-0-snap              READY
freebsd-14-0-stable-amd64-ufs-20240321      freebsd-org-cloud-dev  freebsd-14-0-snap              READY
freebsd-14-0-stable-amd64-ufs-20240329      freebsd-org-cloud-dev  freebsd-14-0-snap              READY
freebsd-14-0-stable-amd64-ufs-20240404      freebsd-org-cloud-dev  freebsd-14-0-snap              READY
freebsd-14-0-stable-amd64-ufs-20240411      freebsd-org-cloud-dev  freebsd-14-0-snap              READY
freebsd-14-0-stable-amd64-ufs-20240418      freebsd-org-cloud-dev  freebsd-14-0-snap              READY
freebsd-14-1-prerelease-amd64-ufs-20240425  freebsd-org-cloud-dev  freebsd-14-1-snap              READY
freebsd-14-1-prerelease-amd64-ufs-20240503  freebsd-org-cloud-dev  freebsd-14-1-snap              READY
freebsd-14-1-release-amd64-ufs              freebsd-org-cloud-dev  freebsd-14-1                   READY
freebsd-14-1-stable-amd64-ufs-20240509      freebsd-org-cloud-dev  freebsd-14-1-snap              READY
freebsd-14-1-stable-amd64-ufs-20240517      freebsd-org-cloud-dev  freebsd-14-1-snap              READY
freebsd-14-1-stable-amd64-ufs-20240530      freebsd-org-cloud-dev  freebsd-14-1-snap              READY
freebsd-14-1-stable-amd64-ufs-20240606      freebsd-org-cloud-dev  freebsd-14-1-snap              READY
freebsd-14-1-stable-amd64-ufs-20240620      freebsd-org-cloud-dev  freebsd-14-1-snap              READY
freebsd-14-1-stable-amd64-ufs-20240628      freebsd-org-cloud-dev  freebsd-14-1-snap              READY
freebsd-14-1-stable-amd64-ufs-20240705      freebsd-org-cloud-dev  freebsd-14-1-snap              READY
freebsd-14-1-stable-amd64-ufs-20240711      freebsd-org-cloud-dev  freebsd-14-1-snap              READY
freebsd-14-1-stable-amd64-ufs-20240718      freebsd-org-cloud-dev  freebsd-14-1-snap              READY
freebsd-14-1-stable-amd64-ufs-20240725      freebsd-org-cloud-dev  freebsd-14-1-snap              READY
freebsd-14-1-stable-amd64-ufs-20240801      freebsd-org-cloud-dev  freebsd-14-1-snap              READY
freebsd-14-1-stable-amd64-ufs-20240808      freebsd-org-cloud-dev  freebsd-14-1-snap              READY
freebsd-14-1-stable-amd64-ufs-20240815      freebsd-org-cloud-dev  freebsd-14-1-snap              READY
freebsd-14-1-stable-amd64-ufs-20240822      freebsd-org-cloud-dev  freebsd-14-1-snap              READY
freebsd-14-1-stable-amd64-ufs-20240829      freebsd-org-cloud-dev  freebsd-14-1-snap              READY
freebsd-14-1-stable-amd64-ufs-20240905      freebsd-org-cloud-dev  freebsd-14-1-snap              READY
freebsd-14-1-stable-amd64-ufs-20240912      freebsd-org-cloud-dev  freebsd-14-1-snap              READY
freebsd-14-1-stable-amd64-ufs-20240919      freebsd-org-cloud-dev  freebsd-14-1-snap              READY
freebsd-14-1-stable-amd64-ufs-20240926      freebsd-org-cloud-dev  freebsd-14-1-snap              READY
freebsd-14-1-stable-amd64-ufs-20241003      freebsd-org-cloud-dev  freebsd-14-1-snap              READY
freebsd-14-1-stable-amd64-ufs-20241010      freebsd-org-cloud-dev  freebsd-14-1-snap              READY
freebsd-15-0-current-amd64-ufs-20231228     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240104     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240111     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240118     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240125     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240126     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240201     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240208     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240215     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240222     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240229     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240307     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240329     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240404     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240415     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240418     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240425     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240502     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240509     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240517     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240523     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240530     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240606     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240613     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240620     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240628     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240705     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240711     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240718     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240725     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240801     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240808     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240815     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240822     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240829     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240905     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240912     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240919     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20240926     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20241003     freebsd-org-cloud-dev  freebsd-15-0-snap              READY
freebsd-15-0-current-amd64-ufs-20241010     freebsd-org-cloud-dev  freebsd-15-0-snap              READY

Based on asomers' testing, I guess that we probably can't resolve this just by changing the image though.

tgross35 pushed a commit to tgross35/rust-libc that referenced this issue Oct 16, 2024
FreeBSD 15 is the unstable development release.  Currently its GCE
images available to Cirrus CI don't work because the solib version of
libmd was just bumped, and the package builders haven't yet caught up.

Issue rust-lang#3967

(backport <rust-lang#3974>)
(cherry picked from commit 9b0ccb1)
asomers added a commit to asomers/libc that referenced this issue Oct 17, 2024
asomers added a commit to asomers/libc that referenced this issue Oct 17, 2024
asomers added a commit to asomers/libc that referenced this issue Oct 17, 2024
The freebsd-15-0-current-amd64-ufs-20241017 GCE image fixes the libmd.so
problem, and once again works with Cirrus CI.

Fixes rust-lang#3967
tgross35 pushed a commit to tgross35/rust-libc that referenced this issue Nov 6, 2024
The freebsd-15-0-current-amd64-ufs-20241017 GCE image fixes the libmd.so
problem, and once again works with Cirrus CI.

Fixes rust-lang#3967

(backport <rust-lang#3988>)
(cherry picked from commit 86d3d1e)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-CI Area: CI-related items O-bsd
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants