PreparingReleases

This document outlines how to manage the MDAnalysis sources and how to prepare distribution packages. See also DevelopmentWorkflow for new ideas how to organize codebase and workflow.

Release policy and release numbering

We use a MAJOR.MINOR.PATCH scheme to label releases. We adhere to the idea of semantic versioning (semantic versioning was introduced with release 0.9, see Issue 200): Given a version number MAJOR.MINOR.PATCH, we increment the:

• MAJOR version when we make incompatible API changes,
• MINOR version when we add functionality in a backwards-compatible manner, and
• PATCH version when we make backwards-compatible bug fixes.

However, as long as the MAJOR number is 0 (i.e. the API has not stabilized), even MINOR increases may introduce incompatible API changes. As soon as we have a 1.0.0 release, the public API can only be changed in a backward-incompatible manner with an increase in MAJOR version.

Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.

The CHANGELOG lists important changes for each release.

MAJOR, MINOR, PATCH number are integers that increase monotonically.

The release number is set in setup.py and in MDAnalysis.__version__ (MDAnalysis/version.py), e.g.

RELEASE = '0.7.5'

While the code is in development (i.e. whenever we are not preparing a release!) the release number gets the suffix -dev0, e.g.

RELEASE = '0.7.6-dev0'

so that people using the development branch from the sources see immediately that this is not a final release. For example, "0.7.6-dev0" is the state before the 0.7.6 release.

Summary

• declare feature freeze on develop via developer list
• finalize the CHANGELOG file with date
• increment version (4 places)
• merge changes into develop
• create a tag from develop named release-<version_number>
• create a package-<version_number> branch from develop
• add rebuilt c files (See Issue #667) to package-<version_number>
• create a tag from package-<version_number>
• this will trigger github actions to create a source distribution and wheels for both package and testsuite based on the contents of package-<version_number>, upload them to testpypi and then check them
• check that the github action checks have all completed successfully
• if everything goes well, create a new release from the newly created tag
• this will re-trigger the github action, create the necessary distribution files and upload them to pypi
• increment develop branch ready for next version and make PR
• update the docs (manually change sitemapindex.xml)

Preparing distribution tar balls

Preparing

• create a pre-release branch (i.e. pre-release-<versionnumber>) based on the current develop.
• Update local repository (git pull) and switch to the pre-release branch (git checkout pre-release<versionnumber>)
• Update CHANGELOG with the release number and summarize important changes. Add all authors that contributed to this release. See the file itself for guidelines and formatting.
• Update the release version under: maintainer/conda/MDAnalysis/meta.yaml, package/MDAnalysis/version.py, package/setup.py, testsuite/MDAnalysisTests/__init__.py, and testsuite/setup.py.
• Create a PR against develop with these committed changes.

Note:

• Make sure to add and commit the changes!
• Test if the distribution (MDAnalysis and MDAnalysisTests) builds successfully and run UnitTests

Packaging the release

• From the release complete develop branch, create a new tag and push it to github. This tag contains a snapshot of the Python source files as they were when the release was created.

git tag -m 'release 0.7.5 of MDAnalysis and MDAnalysisTests' release-0.7.5
git push --tags origin

• Create a package-<version_number> branch from develop

Note on Cython code (see Issue 85 for details):

• If no Cython, setup.py will use C-code that was previously generated; of you updated any *.pyx files then you must have Cython installed (but you probably new this already).
  
• If Cython if installed, it is used to compile extension and .pyx source files are used instead of .c files.
  
• From there, .pyx files are converted to .c files if they are newer than the already present .c files or if the --force flag is set (e.g setup.py build --force).

Therefore, as a developer you should (1) have Cython installed and (2) only use --force if you are positive that you want to regenerate all .c files.

• All branches named package-* have branch protection, so will need to temporarily remove the branch protection rule or create a new branch from this package-<version_number> branch which you will use to create a PR.
• Create new C/C++ files by creating a build and commit them to the new package-<version_number> branch via PR. We recommend generating the C/C++ files by building a source distribution tarball so you can check at the same time that there are no issues with this process.

cd package
python setup.py sdist

• Once committed to package-<version_number>, create a new tag named package-<version_number>. This tag contains a record of all the files as they were deployed to users for that version.

git tag -m 'package 0.7.5 of MDAnalysis and MDAnalysisTests' package-0.7.5
git push --tags origin

Testing the release

• Upon creation of the package-<version_number> tag, a github action will automatically build the source distribution and wheels for the package and testuite.
• These will also be automatically pushed to the main testpypi and testsuite testpypi registries so that they can be tested.
• The action will then pull these entries down and check that they can be properly installed.
• If the tests all come back green, then you are good to go!

What if the action doesn't come back green?

In that case the first thing to check is what was the cause.

1. There was a temporary issue with the github actions services

In that case, just re-run the action, it will re-upload any necessary files, ignore those that were already uploaded, and re-run the checks.

2. There is an issue with the source code

In that case, you will need to:

• Delete the current package-<version_number> branch, and the newly created tags.
• Add the new changes to develop
• Restart the release process
• Before creating the package-* tag, create a separate test tag with an extra commit that bumps up the version number slightly so that it can be uploaded to testpypi and checked for correctness.
• If the action for the test tag works, delete the test tag, and create a new package-* tag, with the correct version number.
• The github action triggered from the re-created tag will fail, but this is not a problem as we've checked in the previous step that everything worked fine.

Completing the release

• At this point you only need to create a new release on GitHub based on the package-<version_number> tag you created above.
• Make sure you include relevant release notes, including any known issues and highlights on the release.
• Once published, the release will trigger the deploy github action again, uploading the source distributions and wheels to the main pypi and testsuite pypi registries.

What if the action doesn't come back green?

This is slightly more problematic. If nothing was uploaded to pypi then you can restart the action in the hope that it now works. If some portion of the package was uploaded to pypi, then you won't be able to re-start the action as it will fail telling you that packages are already present.

Whilst we could enable the action to skip over existing files and upload the missing one, in order to avoid potential issues with broken uploads, we have disabled this. In this case the advice is to yank the new releases on pypi, and then create a new minor release and deploy it.

Update release on anaconda

conda packages are built on conda-forge.

Upon push to pypi, the conda-forge bot should automatically pick up the presence of a new version and create a pull request on the MDAnalysis feedstock.

• Check the contents of the PR, update the meta.yaml file as necessary, especially bumping up the python and dependency minimum versions as necessary.
• If all the check return green, approve and merge the pull request.

Create a blog post outlining the release

To ensure that users are aware of the new release, create a PR adding a blog post entry for the release release. An example of this can be seen here.

Don't forget to then publicize the post on the mailing lists / discord / twitter / etc...!

Clean up old doc builds

Currently -dev0 doc builds are not cleaned up automatically on new releases. Once a new release is made, remove the old -dev0 directory from the gh-pages branch and create amend the versions.json file accordingly.

Update/create a release page on the wiki (outdated / no longer done)

Create a ReleaseXYZ wiki page, modelled after e.g. Release062 (using the CHANGELOG as a reference). Also add it to the Release Notes.

Update docs (outdated?)

The docs for a release are displayed at https://docs.mdanalysis.org (also at https://docs.mdanalysis.org/stable). They are build automatically and versioned. Because the latest release (as sorted by semantic versioning) is copied to https://docs.mdanalysis.org/stable , any redirects that use https://docs.mdanalysis.org/stable will automatically update (e.g. https://www.mdanalysis.org/docs, see DNS: Pagerules)

1. Update the version information in https://github.com/MDAnalysis/MDAnalysis.github.io/blob/master/sitemapindex.xml to point to the new docs (keeps the search on the website current).

Extra note: Test data

UnitTests and test data are maintained in the same git repository as the MDAnalysis package. Tests are checked in together with code (for details see Issue 87, UnitTests and MDAnalysisTests).

The library, MDAnalysis and the tests, MDAnalysisTests share the same release number: whenever there's a new MDAnalysis there will be a corresponding MDAnalysisTests. Therefore, the two packages are maintained and built together (see above).