Skip to content

Commit

Permalink
despite everyone striving to improve, we may overestimate how far we'…
Browse files Browse the repository at this point in the history
…ve come
  • Loading branch information
h-vetinari committed May 7, 2023
1 parent 94e1f5c commit 88aefa5
Showing 1 changed file with 24 additions and 8 deletions.
32 changes: 24 additions & 8 deletions src/maintainer/infrastructure.rst
Original file line number Diff line number Diff line change
Expand Up @@ -283,11 +283,25 @@ Conda-forge builds and maintains its own set of compilers for various languages
and/or systems (e.g., ``C``, ``FORTRAN``, ``C++``, ``CUDA``, etc.). These are used
in all of our CI builds to build essentially all artefacts published by conda-forge.

In the past, compiler upgrades often required full rebuilds of basically all of
conda-forge due to potential ABI breaks. These breaks have become much rarer in
frequency and scope, so it is not a current concern. However, we keep our policies
for such ABI breaks in place for the next time it should occur.

This compiler infrastructure has a critical role beyond building everything, which
is to ensure that packages stay compatible with each other. This is due to how compiled
packages have a so-called `Application Binary Interface <https://pypackaging-native.github.io/background/binary_interface>`_
(ABI), and how changes in the compiler infrastructure may break this ABI, leading
to crashes, miscalculations etc. Generally speaking, using a consistent compiler
version greatly reduces the risk of ABI breaks.

In the past, changes in the compiler upgrades in conda-forge sometimes required a
full rebuild of basically all compiled packages, to avoid breakage between packages
that have been compiled to different ABIs. Compilers also have different policies in
this regard: GCC promises to never break ABI, while MSVC changed (=broke) ABI with
every compiler release (although this is not the case anymore for the vc14 series
currently covering VS2015-VS2022).

While it is likely that large-scale ABI breaks will happen again which require
a full rebuild of conda-forge (e.g. MSVC is planning a vNext after vc14), in recent
years we have managed to use more targetted rebuilds with less disruptive roll-outs.

We keep our policies for full rebuilds in place for the next time it will occur.
While we do not have any promises of support for a generation of ABI-compatible
compilers, we have historically maintained them according to the following (non-binding)
principles.
Expand All @@ -312,9 +326,9 @@ internal use (e.g. ``comp7``). We note again that the existence of these names
does not imply any level of support or stability for the compilers
that form the given stack.

The more recent status quo is that compilers can be upgraded in an ABI-compatible manner,
meaning that we can just increase the version in our global pinning, and it will slowly
roll out in a compatible manner to the ecosystem as feedstocks get rerendered.
For the cases that do not require a complete rebuild of conda-forge (i.e. if the ABI
of a new compiler remains compatible), we can just increase the version in our global
pinning, and it will slowly roll out to the ecosystem as feedstocks get rerendered.
For such ABI-compatible upgrades, similar but looser principles apply:

* The pins are similarly in the global pinning, see :ref:`globally_pinned_packages`.
Expand Down Expand Up @@ -388,6 +402,8 @@ Linux or OSX, ensure that the respective above-mentioned feedstocks have been re
for the new major version, that all interrelated versions are lifted at the same time,
and obviously that the compilers work (e.g. by testing them on some feedstocks by
specifying the new version through the feedstock-local `conda_build_config.yaml`).
You should also check the compiler release notes for warnings about ABI incompatibilities,
and mention any such notices in the discussion about the upgrade.

For Windows, we stay on older compilers for longer, because using a newer toolchain
would force everyone wanting to locally develop with conda-forge artefacts to use
Expand Down

0 comments on commit 88aefa5

Please sign in to comment.