Build & Push Docker Images on releases

[arjxn-py]

Description

Push Docker images on releases, starting from either 23.9 or 24.1 before the 24.1rc0 release

Type of change

Please add a line in the relevant section of CHANGELOG.md to document the change (include PR #) - note reverse order of PR #s. If necessary, also add to the list of breaking changes.

• New feature (non-breaking change which adds functionality)
• Optimization (back-end change that speeds up the code)
• Bug fix (non-breaking change which fixes an issue)

Key checklist:

• No style issues: $ pre-commit run (or $ nox -s pre-commit) (see CONTRIBUTING.md for how to set this up to run automatically when committing locally, in just two lines of code)
• All tests pass: $ python run-tests.py --all (or $ nox -s tests)
• The documentation builds: $ python run-tests.py --doctest (or $ nox -s doctests)

You can run integration tests, unit tests, and doctests together at once, using $ python run-tests.py --quick (or $ nox -s quick).

Further checks:

• Code is commented, particularly in hard-to-understand areas
• Tests added that prove fix is effective or that feature works

[arjxn-py]

Related to #3312
cc: @agriyakhetarpal

[Saransh-cpp]

I'm very rusty at dockerhub stuff at the moment, but IIRC, shouldn't we tag the "release" images with the git tag of the pybamm's release?

I mean - won't this overwrite the "latest" (develop) image on dockerhub with the "release" image, and then as soon as a new commit is pushed to develop, the "release" image will be overwritten by a new "latest" image?

[agriyakhetarpal]

Yes, I'm aware. We're still discussing atm about how the tags will be set since we want to label the image with either the PyBaMM version or the name of the release (since it contains the version) – the latter is easier to extract from the GitHub events API. Marking the PR as draft for now since it's not ready yet

[arjxn-py]

shouldn't we tag the "release" images with the git tag of the pybamm's release?

Yes, you're right. We'll Push Docker Images Based on release tags only, that way we'll have a different image for each release but as @agriyakhetarpal mentioned we're discussing which Image should we push for release to keep it single and simple for each release (i.e. We're mostly in favor of image with IDAKLU or ALL solver), We'd be happy to have your opinion on this @Saransh-cpp

[agriyakhetarpal]

Yes, we're also discussing reducing the number of images we build at this time – the reason is that the latest image without any solvers is just a simple git clone command and doesn't make sense to include, the jax image can be superseded by pip install pybamm[jax] where wheels are currently available for all platforms, which makes the installation a matter of seconds. The images where there are any compilation steps present are odes, idaklu and subsequently all – all of them use the same SUNDIALS that has been compiled in .local/. My opinion is towards just pushing the all image to accommodate all solvers and no other images (regardless of pushes to develop or releases). We run the image builds on commits to the main branch and the arm64 image workflow runs for ~40 mins due to QEMU emulation, so we are not constrained about time or size either. @arjxn-py, could you open a new issue about this?

[agriyakhetarpal]

Also, we've been mixing conda and virtualenv environments in the image – while that does work without issues now, it's not really a good idea to do so, even in late 2023. I would suggest switching to using a manylinux2014-compliant base image (i.e., with glibc) in the not-so-long term; it will essentially be the process of containerising PyBaMM from the ground up once again, but the added experience in this area for all of us in the past months would mean it can be done pretty quickly.

[arjxn-py]

could you open a new issue about this?

Sure, I'm up for making the all image as standard one if we do not consider size. Although, this is also to be noted that latest, jax & odes image with all also manages the environment & installation from source which I feel is different than pip install pybamm[extras]. Happy to open a new Issue about the same and have a discussion there 🙂
See #3666

[agriyakhetarpal]

Yeah ultimately it doesn't make sense for the jax image to have SUNDIALS pre-installed. The [odes] extra is the only one where we don't have wheels on PyPI so we have to compile from source – perhaps they are not publishing because of copyleft licensing issues...

[arjxn-py]

won't this overwrite the "latest" (develop) image on dockerhub with the "release" image, and then as soon as a new commit is pushed to develop, the "release" image will be overwritten by a new "latest" image?

Guess I missed this while answering previously 😬 , Sorry.
No, this wont overwrite the previous release image as two different release images would have two different tags and as soon as tags are different the images are distinct. Maybe you can notice here how there are different tags and different images.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (8dc0421) 99.59% compared to head (bc0fa0b) 99.59%.
Report is 2 commits behind head on develop.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #3661   +/-   ##
========================================
  Coverage    99.59%   99.59%           
========================================
  Files          258      258           
  Lines        20797    20797           
========================================
  Hits         20712    20712           
  Misses          85       85           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[agriyakhetarpal]

Question: what will happen if the git tag is manually adjusted through a force push, a few commits ahead or behind in the tree? Will that trigger a push?

We won't need to do that usually, but maybe a way out of that situation would be to be able to set the image tags from inputs to the workflow (we already support the workflow_dispatch: event) and overwrite the release image – this would tie in with reducing the amount of images we push. But then we currently clone the repository inside the Dockerfile, which would always fetch the latest changes from develop... perhaps @Saransh-cpp can confirm the most recent instance of having to adjust the GitHub tag, was it for 23.9rc0?

Edit: if we don't push Docker images on pre-releases and ensure that the non-pre-release tag is never modified then this shouldn't be a problem 🙂 (i.e., use types: [released] instead of [published])

[agriyakhetarpal]

The entire file shows up as changed, possibly due to a difference in line endings schemes?

[agriyakhetarpal]

Suggested change

	release:
	types: [published]
	release:
	types: [released]

in light of #3661 (comment). The changes from this PR won't be available until 24.5rc0 anyway, so we have time to resolve #3692 and #3666 till then – they should be resolved prior to resuming work on this PR.