Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ARM Documentation updates #16

Merged
merged 42 commits into from
Mar 4, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
c0f5978
add aarch64 to support
aliciaaevans Nov 14, 2023
d3a18ab
add new datechanged directive
daler Nov 18, 2023
ee47a82
add section on platform nomenclature
daler Nov 18, 2023
7200564
update supported python version info
daler Nov 18, 2023
628dc0c
minor tweaks to guidelines
daler Nov 18, 2023
fab7027
fix typos
daler Nov 18, 2023
684368e
minor cleanup on bulk.rst
daler Nov 18, 2023
636d1e7
clarification on repodata patching
daler Nov 18, 2023
6dbbb30
add Alicia to core team
daler Nov 18, 2023
627984c
update notes on building docs
daler Nov 18, 2023
0c1b1ff
css on datechanged directives
daler Nov 18, 2023
fecf007
overhaul and update CI inventory page
daler Nov 18, 2023
ef5a87d
consolidate all FAQs together into top-level
daler Nov 18, 2023
4c2c62a
consolidate cb3.rst into top-level faqs
daler Nov 18, 2023
c777929
overheaul build-system.rst
daler Nov 19, 2023
4a7e86d
minor updates to bulk
daler Nov 19, 2023
d733189
add new aarch64 page
daler Nov 19, 2023
84fe3ce
update number of contributors
daler Nov 19, 2023
bec930c
make CI platforms more general
daler Nov 19, 2023
c5e8956
minor formatting
daler Nov 19, 2023
caa2685
clarifications in faqs
daler Nov 19, 2023
e98d181
update link to galaxy's mulled-build docs
daler Nov 19, 2023
15c6b33
clarifications and updates
daler Nov 19, 2023
3849bcc
fix build/host/run details
daler Nov 19, 2023
5c7ded5
Arm/ARM clarification
daler Nov 19, 2023
e425b40
correct arch for supported mac
daler Nov 19, 2023
a8fc0ee
rm "summary" table
daler Nov 19, 2023
a324f6e
make miniforge statement more accurate and add link
daler Nov 19, 2023
81f6bce
Update source/developer/aarch64.rst
daler Nov 20, 2023
d302e37
Update source/faqs.rst
daler Nov 20, 2023
f104c60
note about linux-aarch64 containers
daler Nov 20, 2023
7ff4680
add inventory of repos
daler Nov 24, 2023
c69024f
add inventory of dockerfiles/images
daler Nov 24, 2023
9726db0
Update source/faqs.rst
daler Dec 1, 2023
22a39f5
typo
daler Nov 25, 2023
2fc9c8c
css update
daler Nov 25, 2023
0e07cad
minor changes
daler Mar 4, 2024
38b0e0e
update images page
daler Mar 4, 2024
b09c54c
Merge branch 'main' into arm-updates
daler Mar 4, 2024
9c795b9
Update source/contributor/build-system.rst
daler Mar 4, 2024
6846aa8
update datechanged
daler Mar 4, 2024
26f99c5
address various review comments
daler Mar 4, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions source/_ext/datechanged_ext.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
"""
Slight modification of the .. versionchanged:: directive that prints "changed
on" rather than "changed in version".
"""

from sphinx.domains import changeset
from sphinx.locale import _

changeset.versionlabels['datechanged'] = _("Changed on %s")
changeset.versionlabel_classes['datechanged'] = 'changed'

def setup(app):
app.add_directive('datechanged', changeset.VersionChange)
return {
'version': 'builtin',
'env_version': 1,
'parallel_read_safe': True,
'parallel_write_safe': True,
}
1 change: 1 addition & 0 deletions source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ def setup(app):
extensions = [
"bioconda_sphinx_ext",
"details_ext",
"datechanged_ext",
"sphinx.ext.intersphinx",
"sphinx.ext.todo",
"sphinx.ext.mathjax",
Expand Down
177 changes: 57 additions & 120 deletions source/contributor/build-system.rst
Original file line number Diff line number Diff line change
@@ -1,153 +1,90 @@
Build system
============

.. datechanged:: 2024-03-04
Generalized descriptions to reflect multiple CI systems in use

The build system for Bioconda takes recipes and converts them into conda
packages that are uploaded to anaconda.org as well as Docker containers that
are uploaded to quay.io as part of the Biocontainers project. All of this happens in a transparent way, with all
build logs available for inspection. The code for the build system can be found
in `bioconda-utils <https://github.com/bioconda/bioconda-utils>`_, but parts
are also with the ``bioconda-recipes`` repo. This document serves as
a high-level overview; the code remains the authoritative source on exactly
what happens during a build.
are uploaded to quay.io as part of the Biocontainers project. All of this
happens in a transparent way, with all build logs available for inspection. The
code for the build system can be found in `bioconda-utils
<https://github.com/bioconda/bioconda-utils>`_, but parts are also with the
``bioconda-recipes`` repo. This document serves as a high-level overview; the
code remains the authoritative source on exactly what happens during a build.

Why so complicated? We have to work within the constraints of conda-build,
Docker, and Azure, while simultaneously supporting the same build system on
a local machine so contributors can test. We also have isolated bioconda-utils
from bioconda-recipes to better facilitate testing of the infrastructure, and
to (one day!) make it general enough that others can use the framework for
their own specific channels. So there are a lot of moving parts that have to be
coordinated, resulting in a complex system. That said, we do have some room to
simplify, and do so where we can.
Docker, CircleCI, GitHub Actions, and Azure DevOps . . . while simultaneously
supporting the same build system on a local machine so contributors can test.
We also have isolated bioconda-utils from bioconda-recipes to better facilitate
testing of the infrastructure, and to (one day!) make it general enough that
others can use the framework for their own specific channels. So there are
a lot of moving parts that have to be coordinated, resulting in a complex
system. That said, we do have some room to simplify, and do so where we can.

This page gives a high-level overview of the most relevant parts to building
recipes. See :ref:`ci-inventory` for *all* of the moving parts throughout the
bioconda ecosystem.

Stages of a bioconda build
--------------------------

A GitHub pull request, or any pushed changes to a GitHub pull request, triggers
a new build on Azure DevOps. One build can contain mulitple recipes, limited only
by the time limit imposed by Azure.
Each build on Azure starts with a fresh VM, so we
need to create the entire bioconda-build system environment from scratch for
each build.
a new build on CI/CD platforms. One build can contain mulitple recipes, limited
only by the time limit imposed by each CI platform. Typically, each build
starts with a fresh VM, so we need to create the entire bioconda-build system
environment from scratch for each build. When possible, we take advantage of
caching offered by the CI platform.

When testing locally, we use the
``quay.io/bioconda/bioconda-utils-build-env-cos7`` Docker container to avoid changing the
local system. This container is defined by `this Dockerfile
<https://github.com/bioconda/bioconda-utils/blob/master/Dockerfile>`_.


Otherwise, when running on Azure, a new Linux or MacOS VM is created for
each build.

The steps are orchestrated by the `Azure config file <https://github.com/bioconda/bioconda-recipes/blob/master/azure-pipeline.yml>`_.


Configure the environment
~~~~~~~~~~~~~~~~~~~~~~~~~
Build steps
~~~~~~~~~~~

N.B., due to transitioning to Azure, the remainder of this section is no longer relevant and requires rewritting.
Once the environment is configured in the first step, the rest of the steps are
orchestrated by bioconda-utils.

- Configure the CI environment:
- ``bioconda-recipes: .circleci/config.yml`` is the primary configuration
daler marked this conversation as resolved.
Show resolved Hide resolved
file for the steps that are run. See https://circleci.com/docs/2.0/ for
the configuration documentation.
- ``bioconda-common: common.sh`` defines the versions of Miniconda and
bioconda-utils to use.
- ``bioconda-recipes: .circleci/setup.sh`` installs Miniconda and
bioconda-utils, sets the correct channel order

- Run linting on changed recipes
- This is triggered by the ``bioconda-recipes: .circleci/config.yml`` "lint"
job, which runs ``bioconda-utils: bioconda_utils/cli.py`` and
``bioconda_utils/linting.py``
- **Configure the environment.** The `bioconda-common
<https://github.com/bioconda/bioconda-common>`_ repo has scripts for
configuring a working conda environment with `bioconda-utils
<https://github.com/bioconda/bioconda-utils>`_ installed. This is used across
the various CI systems to minimize maintenance burden.

- Build recipes
- Triggered by the ``bioconda-recipes: .circleci/config.yaml`` "test-linux"
job, which runs ``bioconda-utils build``. This performs the next steps.
- **Lint.** This step checks for common errors, formatting, and consistency.

- Filter recipes to only focus on recipes that satisfy the following criteria:
- changed recently (we use a ``git diff`` command to identify these
recipes; see ``bioconda-utils: bioconda_utils/build.py``
- not on any blacklists listed in ``config.yaml``
- **Build recipes.** Recipes to be built satisfy the following criteria:
- changed in this pull request
- not on the `build-fail-blacklist`
- does not have a build-failure yaml file in the recipe with a hash
matching this version
- package with that version number and build number does not exist in
bioconda channel (we check the channel for each of the changed recipes)

- Download the configured Docker container (currently based on CentOS 7)
- default configured in ``bioconda-utils: docker_utils.py``

- Build a new, temporary Docker container
- Dockerfile configured in ``bioconda-utils: docker_utils.py``; we hope to
move to simply pulling from DockerHub now that our build dependencies are
not changing as often)

- Topologically sort changed recipes, and build them one-by-one in the Docker
container. This runs ``conda-build`` on the recipe while also specifying the
correct environment variables.

- The conda-build directory is exported to the docker container to a temp
file and added as a channel. This way, packages built by one container
will be visible to containers building subsequent packages in the same
Travis-CI build.
- ``bioconda-utils: docker_utils.py`` specifies the build script that is
run in the container.
- At the end of the build, the build script copies the package to the
exported conda-bld directory
- A whitelist of env vars is exported. The whitelist is configured in
``bioconda-utils: utils.py``.

- Upon successfully building and testing via ``conda-build``, the built package
is added to a minimal BusyBox container using ``mulled-build`` (maintained in
`galaxy-lib <https://github.com/galaxyproject/galaxy-lib>`_). This acts as
a more stringent test than ``conda-build`` alone. The BusyBox container
bioconda channel

- **Isolated test.** Upon successfully building and testing via
``conda-build``, the built package is added to a minimal BusyBox container
using ``mulled-build`` (maintained in `galaxy-tool-util
<https://docs.galaxyproject.org/en/latest/admin/special_topics/mulled_containers.html#automatic-build-of-linux-containers>`_). This acts as a more
stringent test than ``conda-build`` alone, because the BusyBox container
purposefully is missing many system libraries (like libgcc) that may be
present in the CentOS 7 container. Note that it is common for a package to
build in the CentOS 7 container but fail in the BusyBox container. When this
happens, it is often because a dependency needs to be added to the recipe.

- Upon successfully testing the package in the BusyBox container, we have a branch point:

- if we are on a pull request:
- report the successful test back to the GitHub PR, at which time it
can be merged into the master branch
- if we are on the master branch:
- upload the built conda package to anaconda.org, with an optional label
- upload the BusyBox container to quay.io

As soon as the package is uploaded to anaconda.org, it is available for
installing via ``conda``. As soon as the BusyBox container is uploaded to
quay.io, it is available for use via ``docker pull``.

The ``bulk`` branch
-------------------

Periodically, large-scale maintenance needs to be done on the channel. For
example, when a new version of Bioconductor comes out, we need to update all
``bioconductor-*`` packages and rebuild them. Or if we change the version of
a pinned package in ``scripts/env.yaml``, then all packages depending
on that package need to be rebuilt. While our build infrastructure will build
recipes in the correct toplogically sorted order, if there are too many recipes
then Travis-CI will timeout and the build will fail.

Our solution to avoiding builds failing due to timeouts is the special ``bulk``
branch. This branch is used by the bioconda core team for maintenance and
behaves much like the ``master`` branch in that packages, once successfully
built and tested, are immediately uploaded to anaconda.org. The major
difference is that ``bulk`` does not go through the pull-request-and-review
process in order for packages to be built and uploaded to the channel. As such,
only bioconda core members are able to push to the ``bulk`` branch.

The workflow is to first merge the latest master into ``bulk`` branch and
resolve any conflicts. Then push (often a large number of) changes to the
branch, without opening a PR. Unlike the ``master`` branch, which uses
the shortcut of only checking for recipes in the channel if they have been changed
according to ``git``, the ``bulk`` branch is configured to do the exhaustive
check against the channel (which can take some time). Any existing recipe that
does not exist in the channel will therefore be re-built. As packages build,
they are uploaded; as they fail, the testing moves on to the next package. The
``bulk`` branch runs up until the Travis-CI timeout, at which time the entire
build fails. But since individual packages were uploaded as they are
successfully built, our work is saved and we can start the next build where we
left off. Failing tests are fixed in another round of commits, and these
changes are then pushed to ``bulk`` and the process repeats. Once ``bulk`` is
fully successful, a PR is opened to merge the changes into master.
- **Report.** If we are on a pull request, report the successful test back to
the GitHub PR, at which time it can be merged into the master branch

- **Upload.** If we are on the master branch, then upload the built conda
package to anaconda.org, and upload the BusyBox container to quay.io

- **Use!** As soon as the package is uploaded to anaconda.org, it is available
for installing via ``conda``. As soon as the BusyBox container is uploaded to
quay.io, it is available for use via ``docker pull``.

Labels
------
Expand Down
Loading