Current CI issues

[bilderbuchi]

Let's keep a list of CI issues that are affecting our runs:

• The paths caught by our sphinx-doctest problem matcher are not picked up correctly by Actions annotations: Problem matcher relative file path should be rooted in working-directory, not workspace actions/runner#765
• There is no support yet for multiline error messages (that e.g. sphinx-doctest produces) in the problem matcher: Multiline support for Toolkit Annotations actions/toolkit#193. As a temporary workaround, we just match the first line for now, which will just show in the CI overview that a problem exists
• We don't yet use linting (Start using linting #346) and don't have annotations/actions for flake8
• We don't have annotations for pytest errors, but maybe that works ootb? If not, search for a problem matcher example. not that useful without multiline annotations
• We should generate coverage information to visualise how much of the codebase our tests cover -- it will be little :-(.
• We could use mamba instead of conda for faster python installation
• Consider if we should use pyupgrade in our linting stage (to generate remark bubbles in the Changes view like flake8) to maintain a modern code style. In that case we should research first if there is a Github Action plugin for the decorations, or if the tool can even report back specific changes instead of changing in-place (the default behaviour).

For later

• Automatically deploy tagged commits to PyPI if CI successful.
• If possible, also update conda-forge feedstock?

[bilderbuchi]

• Using python setup.py test is deprecated, and we should move away from that again. see Deprecate test command pypa/setuptools#1684 setup.py test is going to be deprecated pytest-dev/pytest#5534
  • We need to undo the related changes from @cjermain's 4357bc7
  • Then identify and implement the currently recommended best practice.

[bilderbuchi]

• See if we can store (at least for a short time) the generated documentation (~1 MB) as an artifact so we can check online if docs generation works as expected. So users do not need to run sphinx locally, and devs can check PRs without having to fetch them locally, first.

[bilderbuchi]

• Upgrade GH actions plugin version to 3: https://github.com/actions/cache/releases/tag/v3.0.0

[bilderbuchi]

• We could start adding pydocstyle checking to CI. Enable "D212 | Multi-line docstring summary should start at the first line"