[WIP] - Proof of concept - optimising CI build

[trallard]

Related to #1407

Notes:

• I am moving away from Make to using doit some of the advantages of this are:
  • Python-based so we get all the syntactic sugar
  • it allows for much more flexibility when it comes to set tasks, dependencies, and manipulating artefacts/build outputs
  • others projects in the Jupyter ecosystem have started adopting it for automation and to complement their CI workflows and it works pretty well

Done:

• started to adopt the workflow shown in Improve performance of CI system #1407
• right-hand side of the workflow is now implemented - see image attached (the side in green is implemented except that the push is not working as we need a PAT - see below)
• added cache for the Docker build and for Python dependencies
• created doit tasks
• parallel GH jobs for: testing and manifest/tag creation
• Docker images and manifests are saved as GH artifacts
• Add types to the dodo.py file

Questions

• What would you suggest doing after?
• Completing the main workflow in Improve performance of CI system #1407 -> pushing all the tags to DockerHub and the manifests to the Wiki? (The advantage is that if that is done now at least we get an optimised workflow from what we currently have for x86 images)
• Working on the left-hand side of the workflow?
• How long do we want to keep the artefacts for? right now I set a 1-day policy while I was doing some tests but not sure how long would be a reasonable time here

TODO:

• Get a PAT (personal access token) so that we can push to GHCR

---

---

Updated by Ayaz:
Moved Add types to the dodo.py file to Done section.

[mathbunnyru]

@trallard I merged the PR where tests are run for all dependent images.
So running tests for a specific image now only takes one line of code.
Please, merge master to this branch and fix the code so it uses new way to run the tests.

[schettino72]

def task_docker_load_images() -> dict[str, Any]:
    """
    Load docker images from previously downloaded artifacts 📥
    Since we are running in a CI environment and within a separate job than
    the one where the images are built, we need to load the images from the
    `CI_IMAGE_TAR`
    """

    def check_image():
        assert Utils.IS_CI
        assert (
            Utils.CI_IMAGE_TAR.exists()
        ), f"not found images archive in: {Utils.CI_IMAGE_TAR}"

    return dict(
        name="load_images",
        doc="Load and inspect Docker images",
        actions=[
            check_image,
            Utils.do("docker", "load", "--input", Utils.CI_IMAGE_TAR),
            # TODO: @trallard to add a more robust inspect
            (Utils.inspect_image, [Utils.get_images()[-1]]),
            Utils.do("docker", "images"),
        ],
    )

[mathbunnyru]

Fail of Build Sphinx Documentation and check links is ok - somewhy github returned 403 during linkcheck.

[mathbunnyru]

@trallard I merged some simple parts of this PR to master, to make this PR a bit more focused.
Would you like to continue the work here or you don't have plans to finish this?

No pressure here, I would like to know to better understand the current situation ❤️

[trallard]

Hey @mathbunnyru I have had some deadlines and stuff at work so had not had a chance to finish this off. I do plan to finish this soon

Something I have been thinking of is actually deferring deleting stuff in the Makefile since we have some bits that are reused on other workflows and I do not want to break things unintentionally

[trallard]

Follow up

I merged some simple parts of this PR to master
What has already been merged? - do I need to remove anything from this PR then?

[mathbunnyru]

No, this PR only has the necessary parts.
I removed some inconsistencies between this PR and master (like having beautiful icons, a bit different wording and other small changes).
So, this PR is easier to read / smaller and doesn't contain too many things.

[mathbunnyru]

I'm closing this PR, because approach with self-hosted runners and many jobs seems to work fine.