Skip to content

Commit

Permalink
Update conda-store explanations (#726)
Browse files Browse the repository at this point in the history 
Signed-off-by: Pavithra Eswaramoorthy <[email protected]>
Co-authored-by: Peyton Murray <[email protected]>
  • Loading branch information
@pavithraes @peytondmurray
pavithraes and peytondmurray authored Jul 18, 2024
1 parent 8535d3c commit 94f7a26
Show file tree
Hide file tree
Showing 4 changed files with 327 additions and 223 deletions.
211 changes: 91 additions & 120 deletions docusaurus-docs/conda-store/explanations/artifacts.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,187 +4,158 @@ description: Understand environment artifacts generated by conda-store

# Artifacts

:::warning
This page is in active development, some content may be inaccurate.
:::

conda environments can be created in a few different ways.
conda-store creates "artifacts" (corresponding to different environment creation options) that can be shared with colleagues and can be used to reproduce environments.
In the conda-store UI, these are available in the "Logs and Artifacts" section at the end of the environment page.
conda-store creates "artifacts" (corresponding to different environment creation options) for every environment, that can be shared with colleagues and used to reproduce environments.
In the conda-store UI, these are available in the **"Logs and Artifacts"** section
at the end of the environment page.

The following sections describe the various artifacts generated and how to create environments with them.

### YAML file (pinned)
Environments in shared namespaces on conda-store can be accessed by everyone with access to that namespace, in which case you may not need to share the artifacts manually.
Artifacts are used to share your environment with external collaborators who don't have access to conda-store.

YAML files that follow the conda specification is a common way to create environments.
conda-store creates a "pinned" YAML, where all the exact versions of requested packages (including `pip` packages) as well as all their dependencies are specified, to ensure new environments created match the original environment as closely as possible.
:::note
The libraries (conda, conda-lock, conda-pack, etc.) mentioned in the following sections are separate projects in the conda ecosystem. The environments created using them are not managed by conda-store.
:::

A pinned YAML file is generated for each environment ta is built.
This includes pinning of the `pip`` packages as well.
## YAML file (pinned)

YAML files that follow the conda specification are a common way to create environments.
conda-store creates a "pinned" YAML, where all the exact versions of requested packages (including `pip` packages) as well as all their dependencies are specified, to ensure new environments created match the original environment as closely as possible.

:::info
In rare cases, the completely pinned packages may not solve because packages are
routinely marked as broken and removed.
In rare cases, building environments from "pinned" YAML files may not solve because packages are routinely marked as broken and removed at the repository level.

**conda-forge** (default channel in conda-store)
has a [policy that packages are never removed but are marked as
broken][conda-forge-immutability-policy].
Most other channels do not have such a policy.
:::

Assuming you have `conda` installed, to create a conda environment (on any machine) using this file:
Click on **"Show yml file"** link in the conda-store UI to open the file in a new browser tab. You can copy-and-past this file in [conda-store UI's YAML editor][cs-ui-yaml] to create a new environment managed by conda-store in a different namespace.

1. Click on **"Show yml file"** link in the conda-store UI to open the file in a new browser tab.
2. Save the file with: Right-click on the page -> Select "Save As" -> Give the file a meaningful name (like `environment.yml`)
3. Run the following command and use the corresponding filename:
```bash
conda env create --file <environment.yml>
```
You can download the file and share with someone or use it to create an environment on a different machine. Assuming `conda` is installed, run the [CLI commands mentioned in the conda-documentation][conda-docs-create-env] with the corresponding filename to create a conda environment (on any machine).

### Lockfile
## Lockfile

A conda lockfile is a representation of only the `conda` dependencies in
A conda lockfile is a representation of all (`conda` and `pip`) dependencies in
a given environment.
conda-store created lockfiles using the [conda-lock][conda-lock-github] project.
conda-store creates lockfiles using the [conda-lock][conda-lock-github] project.

Click on **"Show lockfile"** to open the lockfile in a new browser tab.
You can download the file and share with someone or use it to create an environment in a different space.

To create an environment att the new location, follow the [commands in the conda-lock documentation][conda-lock-install-env].

## Tarballs or archives

:::warning
This file will not reproduce the `pip` dependencies in a given environment.
It is usually a good practice to not mix pip and conda dependencies.
Building environments from archives is only supported on Linux machines
because the tarballs are built on Linux machines.
:::

Click the `lockfile` icon to download the
lockfile. First install `conda-lock` if it is not already installed.
A tarball or archive is a _packaged_ environment that can be moved, unpacked, and used in a different location or on a different machine.

```shell
conda install -c conda-forge lockfile
```
conda-store uses [Conda-Pack][conda-pack], a library for
creating tarballs of conda environments.

Install the locked environment file from conda-store.
Click **"Download archive"** button to download the archive of your conda environment, and share/move it to the desired location.

```shell
conda-lock install <lockfile-filename>
```
To install the tarball, follow the [instructions for the target machine in the conda-pack documentation][conda-pack-usage].

### conda-pack archive
## Docker images

[Conda-Pack](https://conda.github.io/conda-pack/) is a package for
creating tarballs of given Conda environments. Creating a Conda archive
is not as simple as packing and unpacking a given directory. This is
due to the base path for the environment that may
change. [Conda-Pack](https://conda.github.io/conda-pack/) handles all
of these issues. Click the `archive` button and download the given
environment. The size of the archive will be less than the size seen
on the environment UI element due to compression.
:::warning
Docker image creation is currently only supported on Linux.

```shell
conda install -c conda-forge conda-pack
```
The docker image generation and registry features are experimental,
and the following instructions are not thoroughly tested.
If you face any difficulties, open an issue on the GitHub repository.
:::

Install the Conda-Pack tarball. The directions are [slightly
complex](https://conda.github.io/conda-pack/#commandline-usage). Note
that `my_env` can be any name in any given prefix.
conda-store acts as a docker registry.
It leverages [Conda Docker][conda-docker], which builds docker images without Docker, allowing for advanced caching, reduced image sizes, and does not require elevated privileges.

```shell
mkdir -p my_env
tar -xzf <conda-pack-tarfile>.tar.gz -C my_env
### Authentication

source my_env/bin/activate
The `conda-store` docker registry requires authentication.
You can use **any username** and your **user token as the password**.

conda-unpack
```bash
docker login -u <any-username> -p <conda-store-token>
```

### Docker images
To get your user token:

:::note
Docker image creation is currently only supported on Linux.
:::
1. Visit your user page at `<your-conda-store-domain>/admin/user`
2. Click on "Create token", which displays your token
3. Click on "copy" to copy the token to your clipboard

conda-store acts as a docker registry which allows for interesting
ways to handle Conda environment. In addition this registry leverages
[conda-docker](https://github.com/conda-incubator/conda-docker) which
builds docker images without docker allowing for advanced caching,
reduced image sizes, and does not require elevated privileges. Click
on the `docker` link this will copy a url to your clipboard. Note the
beginning of the url for example `localhost:8080/`. This is required to tell
docker where the docker registry is located. Otherwise by default it
will try and user docker hub. Your url will likely be different.

The `conda-store` docker registry requires authentication via any
username with password set to a token that is generated by visiting
the user page to generate a token. Alternatively in the
`conda_store_config.py` you can set
`c.AuthenticationBackend.predefined_tokens` which have environment
read permissions on the given docker images needed for pulling.
Alternatively, you can set `c.AuthenticationBackend.predefined_tokens` in `conda_store_config.py`, which have environment read permissions on the given docker images required for pulling images.

```
docker login -u token -p <conda-store-token>
docker pull <docker-url>
docker run -it <docker-url> python
```
### General usage

#### General usage
To use a specific environment build, click on the **"Show Docker image"** to get the URL to the docker image. For example: `localhost:8080/analyst/python-numpy-env:583dd55140491c6b4cfa46e36c203e10280fe7e180190aa28c13f6fc35702f8f-20210825-180211-244815-3-python-numpy-env`.

```shell
docker run -it localhost:8080/<namespace>/<environment-name>
```
The URL consists of: `<conda-store-domain>/<namespace>/<environment-name>:<build_key>`

* The conda-store domain (for example `localhost:8080/`) at the beginning tells Docker where the docker registry is located. Otherwise, Docker will try to use Docker Hub by default.
* The `<namespace>/<environment-name>` refers to the specific conda environment
* The "build key" is a combination of `<specification-sha256>-<build
date>-<build id>-<environment name>` which points to specific build of the environment. For example, a past version of the environment.

If you want to use a specific build (say one that was built in the
past and is not the current environment) you can visit the specific
build that you want in the UI and copy its docker registry tag
name. The tag name is a combination of `<specification-sha256>-<build
date>-<build id>-<environment name>` that we will refer to as build
key. An example would be
`localhost:5000/filesystem/python-numpy-env:583dd55140491c6b4cfa46e36c203e10280fe7e180190aa28c13f6fc35702f8f-20210825-180211-244815-3-python-numpy-env`.
To use a conda-store environment docker image:

```shell
docker run -it localhost:8080/<namespace>/<environment-name>:<build_key>
```bash
docker run -it <docker-url>
```

#### On Demand Docker Image
### On-demand (dynamic) docker image

In conda-store, you can also specify the required packages within the docker image name itself, without needing an actual environment to be created by conda-store UI.

The URL format is: `<registry-url>:<registry-port>/conda-store-dynamic/<package-constraint-1>/.../<package-constraint-n>`.

conda-store has an additional feature which allow for specifying the
packages within the docker image name itself without requiring an
actual environment to be created on the conda-store UI side.
After `conda-store-dynamic`, you can specify packages with constraints separated by
slashes in the following format:
* `<=1.10` as `.lt.1.10`
* `>=1.10` as `.gt.1.10`

The following convention is used
`<registry-url>:<registry-port>/conda-store-dynamic/`. After
`conda-store-dynamic` you specify packages needed separated by
slashes. Additionally you may specify package constraints
for example `<=1.10` as `.lt.1.10`.
For example, if you need Python less than `3.10` and NumPy
greater than `1.0`, this would be the docker image
name: `<registry-url>:<registry-port>/conda-store-dynamic/python.lt.3.10/numpy.gt.1.0`.

As full example support we want python less than `3.8` and NumPy
greater than `1.0`. This would be the following docker image
name. `<registry-url>:<registry-port>/conda-store-dynamic/python.lt.3.8/numpy.gt.1.0`. conda-store
will then create the following environment and the docker image will
download upon the docker image being built.
conda-store creates the environment ands builds the docker image, which you can then download.

### Installers
## Installers

conda-store uses [constructor] to generate an installer for the current platform
(where the server is running):
Installers are another way to share and use a set of (bundled) packages.
conda-store uses [constructor][constructor-docs] to generate an installer for the current platform (where the server is running):

- on Linux and macOS, it generates a `.sh` installer
- on Windows, it generates a `.exe` installer using NSIS.
- on Linux and MacOS, it generates a `.sh` installer
- on Windows, it generates a `.exe` installer using NSIS

conda-store automatically adds `conda` and `pip` to the target environment
because these are required for the installer to work.

Also note that `constructor` uses a separate dependency solver instead of
:::note
`constructor` uses a separate dependency solver instead of
utilizing the generated lockfile, so the package versions used by the installer
might be different compared to the environment available in conda-store. There
are plans to address this issue in the future.
:::

#### Existing Deployments

conda-store saves environment settings and doesn't automatically update them on
startup (see `CondaStore.ensure_settings`). Existing deployments need to
manually enable installer builds via the admin interface. This can be done by
going to `<CondaStoreServer.url_prefix>/admin/setting/<namespace>/<env>/` (or
clicking on the `Settings` button on the environment page) and adding
`"CONSTRUCTOR_INSTALLER"` to `build_artifacts`.
<!-- Internal links -->
[cs-ui-yaml]: ../../conda-store-ui/tutorials/create-envs#yaml-editor

<!-- External links -->
[conda-docs]: https://docs.conda.io/projects/conda/en/latest/user-guide/concepts/environments.html
[conda-forge-immutability-policy]: https://conda-forge.org/docs/maintainer/updating_pkgs.html#packages-on-conda-forge-are-immutable
[conda-docs-create-env]: https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-from-an-environment-yml-file
[conda-lock-github]: https://github.com/conda-incubator/conda-lock
[conda-lock-install-env]: https://conda.github.io/conda-lock/output/#environment-lockfile
[constructor]: https://github.com/conda/constructor
[conda-pack]: https://conda.github.io/conda-pack/
[conda-pack-usage]: https://conda.github.io/conda-pack/index.html#commandline-usage
[conda-docker]: https://github.com/conda-incubator/conda-docker
[constructor-docs]: https://conda.github.io/constructor/
Loading

0 comments on commit 94f7a26

Please sign in to comment.