Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document microarchitecture-optimized builds #2091

Merged
merged 2 commits into from
Feb 28, 2024
Merged

Document microarchitecture-optimized builds #2091

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
72cd37c
document microarchitecture-optimized builds
jaimergp Feb 28, 2024
e496664
Merge branch 'main' into microarchitectures
beckermr Feb 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/glossary.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,3 +78,7 @@ An environment is a tool that helps to keep dependencies required by different p
## Recipe

A recipe is a collection of files required to build a conda package. This includes, at minimum, a [`meta.yaml`](maintainer/adding_pkgs.md#the-recipe-meta-yaml) file, but can also include license files, patches, build scripts, test scripts etc. [Learn More](https://docs.conda.io/projects/conda-build/en/stable/resources/define-metadata.html).

## Virtual package

Virtual packages are not real packages that can be downloaded. They are injected by the conda clients at runtime so the solver can consider that metadata as part of the constraints of the problem. By convention, they always start with a double underscore (`__`). Some examples include the type of operating system (Linux, Windows, macOS), or the CUDA version supported by the system (if any). [Learn More](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-virtual.html).
34 changes: 34 additions & 0 deletions docs/maintainer/knowledge_base.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1271,6 +1271,40 @@ which has `openblas` as a dependency and has a symlink from `libblas.so.3` to `l
BLAS `3.8.0` API. This means that, at install time, the user can select what BLAS implementation
they like without any knowledge of the version of the BLAS implementation needed.

### Microarchitecture-optimized builds {#microarch}

conda [virtual packages](../glossary.md#virtual-package) include `__archspec`, which expose the processor architecture to the solver. However, `__archspec` should not be used directly in recipes; instead, users should rely on the [`microarch-level`](https://github.com/conda-forge/microarch-level-feedstock) helper packages (contributed in [staged-recipes#24306](https://github.com/conda-forge/staged-recipes/pull/24306)).

Before learning how to use it, please read these considerations:

- Adding microarchitecture variants can result in too many entries in the build matrix. Do not overuse it.
- These optimized builds should only be used when the performance improvements are significant.
- Preferrably, the project should rely on runtime dispatch for arch-specific optimizations.
- If the package is already too large, consider using smaller outputs for the arch-optimized variants.

To implement microarchitecture-optimized builds in your feedstock, you'll end up with something like:

```yaml title="recipe/conda_build_config.yaml"
microarch_level: # [unix and x86_64]
- 1 # [unix and x86_64]
- 3 # [unix and x86_64]
- 4 # [unix and x86_64]
```
Comment on lines +1287 to +1292
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, the very recent archspec 0.2.3 now has windows support, in large part due to @isuruf's work on this. 🥳

Not sure what else is necessary to wire this up though, just tried on a fully up-to-date environment:

>conda info
    [...]
       virtual packages : __archspec=1=x86_64
                          __conda=24.1.2=0
                          __win=0=0


```yaml title="recipe/meta.yaml"
# ...
requirements:
build:
- x86_64-microarch-level {{ microarch_level }} # [unix and x86_64]
- {{ compiler('c') }}
# ...
# ...
```

That's it. Note that the activation scripts behind the `microarch-level` packages are already injecting the necessary compiler flags for you. Since they also have `run_exports` entries, your
package will have the necessary runtime requirements to ensure the most adequate variant gets installed. Refer to [this comment](https://github.com/conda-forge/staged-recipes/pull/24306#issuecomment-1800095471) for more information.


<a id="knowledge-mpl"></a>

<a id="matplotlib"></a>
Expand Down