Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GH Actions (docker): Run a job for "make build-local" first, cache image for job "make build" #33062

Closed
mkoeppe opened this issue Dec 21, 2021 · 112 comments
Closed

GH Actions (docker): Run a job for "make build-local" first, cache image for job "make build" #33062

mkoeppe opened this issue Dec 21, 2021 · 112 comments
Labels
c: porting p: critical / 2 t: enhancement
Milestone
sage-9.8

Comments

@mkoeppe
Copy link
Contributor

mkoeppe commented Dec 21, 2021

Follow up from #32703, where this was implemented for macOS.

Here we implement it for the docker-based workflows
via incremental builds (#34228) on top of previous stages pushed to ghcr.io

Depends on #34115
Depends on #34228

CC: @tobiasdiez @dimpase

Component: porting

Author: Matthias Koeppe

Branch/Commit: 5713937

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/33062
@mkoeppe mkoeppe added this to the sage-9.6 milestone Dec 21, 2021
@mkoeppe
Copy link
Contributor Author

mkoeppe commented Jan 13, 2022

Dependencies: #30933, #33103, #33140

@mkoeppe

This comment has been minimized.

Sign in to view
@mkoeppe
Copy link
Contributor Author

mkoeppe commented Jan 30, 2022

Changed dependencies from #30933, #33103, #33140 to #33103, #33140, #33222

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Feb 5, 2022

Changed dependencies from #33103, #33140, #33222 to #33103, #33140

@mkoeppe

This comment has been minimized.

Sign in to view
@mkoeppe
Copy link
Contributor Author

mkoeppe commented Feb 7, 2022

Branch: u/mkoeppe/gh_actions__docker___run_a_job_for__make_build_local__first__cache_image_for_job__make_build_

@mkoeppe

This comment has been minimized.

Sign in to view
@mkoeppe
Copy link
Contributor Author

mkoeppe commented Feb 7, 2022

Last 10 new commits:

531cd14Merge #33103
51191fe.github/workflows/tox.yml (local-macos): Fix filtering of experimental packages
12f88cf.github/workflows/tox.yml (local-macos): Group optional/experimental package builds into fewer jobs
49236ddtox.ini: Add local-macos-python3_pythonorg-python3.9 etc., refactor CONFIG_CONFIGURE_ARGS_1=...--with-python
1fcdc91tox.ini: Updates for macos and manylinux from #31396
41ec3aatox.ini: Reindent
d7066aaMerge #33140
98a5baeMerge tag '9.6.beta0' into t/33062/gh_actions__docker___run_a_job_for__make_build_local__first__cache_image_for_job__make_build_
c446d33.github/workflows, src/doc/en/developer/portability_testing.rst: Switch from docker.pkg.github.com to ghcr.io
c9c7d9ftox.ini (docker): Support Docker build layer caching

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Feb 7, 2022

Commit: c9c7d9f

@mkoeppe

This comment has been minimized.

Sign in to view
@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Feb 7, 2022

Branch pushed to git repo; I updated commit sha1. New commits:

e56bf3e.github/workflows/tox.yml: Integrate optional and experimental runs here
c830cdb.github/workflows/ci-{linux,macos}.yml: Split from tox.yml

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Feb 7, 2022

Changed commit from c9c7d9f to c830cdb

@mkoeppe

This comment has been minimized.

Sign in to view
@mkoeppe
Copy link
Contributor Author

mkoeppe commented Feb 7, 2022

Author: Matthias Koeppe

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Feb 7, 2022

Reviewer: https://github.com/mkoeppe/sage/actions/runs/1804137820

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Feb 7, 2022

comment:11

Obviously there is again a lot of cut+paste going on here, but I'm not sure yet whether this is best refactored using (1) scripts to be put in .ci/; (2) new top-level Makefile targets; (3) reusable workflows in the same repo; (4) reusable workflows from a different repo. So until then, copy+paste it is

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Feb 7, 2022

Changed commit from c830cdb to 47b636d

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Feb 7, 2022

Branch pushed to git repo; I updated commit sha1. New commits:

47b636d.github/workflows/ci-linux.yml: Fix typo

@tobiasdiez
Copy link
Contributor

comment:14

A lot of jobs are failing because there doesn't exist a corresponding maximal image.

What is the distinction between the docker and docker-optional jobs?

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Feb 7, 2022

comment:15

Replying to @tobiasdiez:

What is the distinction between the docker and docker-optional jobs?

docker-optional also runs the stages that build optional and experimental packages.

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Feb 7, 2022

comment:17

Replying to @tobiasdiez:

A lot of jobs are failing because there doesn't exist a corresponding maximal image.

Not sure if that's it but there are a number of new failures introduced by switching to DOCKER_BUILDKIT=1, for example https://github.com/mkoeppe/sage/runs/5087229478?check_suite_focus=true

 > [with-system-packages 130/204] RUN zypper --ignore-unknown install --no-confirm --auto-agree-with-licenses --no-recommends --details "pkgconfig(gf2x)" || echo "(ignoring error)":
------
failed to prepare d4bec9ys962tp15dnd39jkkya: max depth exceeded

This will require some changes in build/bin/write-dockerfile.sh

@tobiasdiez
Copy link
Contributor

comment:18

Replying to @mkoeppe:

Replying to @tobiasdiez:

What is the distinction between the docker and docker-optional jobs?

docker-optional also runs the stages that build optional and experimental packages.

Okay, thanks! Would it then make sense to base the docker-optional on the image created in the docker job to speed up the build (i.e. as a "stage")?

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Feb 7, 2022

comment:19

It's currently using a different package configuration (-maximal instead of -standard)

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Aug 1, 2022

Changed commit from 5b425ac to 31c4722

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Aug 1, 2022

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

b354a38tox.ini, .github/workflows/docker.yml: Reimplement -incremental via SKIP_SYSTEM_PACKAGES
5decaddbuild/bin/write-dockerfile.sh: Do not use persistent env var PACKAGES (except on nix)
6c32034Merge #34228
979779atox.ini (ubuntu-jammy, debian-bookworm, fedora-35, fedora-36): Do not IGNORE_MISSING_SYSTEM_PACKAGES
326f61dReplace tox-optional, tox-experimental.yml by use of reusable workflow in ci-linux.yml
31c4722.github/workflows/ci-linux.yml: Run following stages even when a job in the previous stage fails

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Aug 1, 2022

Branch pushed to git repo; I updated commit sha1. New commits:

a6ee7a5.github/workflows/ci-linux.yml: Use success() | failure()
f2b52ef.github/workflows/ci-linux.yml: First run all 'standard' jobs, then 'minimal', ...

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Aug 1, 2022

Changed commit from 31c4722 to f2b52ef

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Aug 1, 2022

Branch pushed to git repo; I updated commit sha1. New commits:

ca6d163.github/workflows/ci-linux.yml: Rename stages

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Aug 1, 2022

Changed commit from f2b52ef to ca6d163

@mkoeppe

This comment has been minimized.

Sign in to view
@mkoeppe
Copy link
Contributor Author

mkoeppe commented Aug 3, 2022

Work Issues: TARGETS_OPTIONAL not passed correctly

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Aug 3, 2022

Branch pushed to git repo; I updated commit sha1. New commits:

5713937sage -package list --has-file, --no-file: Handle disjunctions

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Aug 3, 2022

Changed commit from ca6d163 to 5713937

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Aug 3, 2022

Changed work issues from TARGETS_OPTIONAL not passed correctly to none

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Sep 19, 2022

comment:85

Marking it as critical because it is a dependency of #34266

@mkoeppe mkoeppe modified the milestones: sage-9.7, sage-9.8 Sep 19, 2022
@dimpase
Copy link
Member

dimpase commented Sep 21, 2022

comment:87

lgtm

@dimpase
Copy link
Member

dimpase commented Sep 21, 2022

Changed reviewer from https://github.com/mkoeppe/sage/actions/runs/2751836720 to Dima Pasechnik

@mkoeppe
Copy link
Contributor Author

mkoeppe commented Sep 21, 2022

comment:88

Thanks!

@vbraun
Copy link
Member

vbraun commented Sep 25, 2022

Changed branch from u/mkoeppe/gh_actions__docker___run_a_job_for__make_build_local__first__cache_image_for_job__make_build_ to 5713937

@vbraun vbraun closed this as completed in 02bfe77 Sep 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
c: porting p: critical / 2 t: enhancement
Projects
None yet
Milestone
sage-9.8
Development

No branches or pull requests
4 participants
@vbraun @dimpase @tobiasdiez @mkoeppe