sagemathgh-38728: Improve conda setup

    
<!-- ^ Please provide a concise and informative title. -->
<!-- ^ Don't put issue numbers in the title, do this in the PR
description below. -->
<!-- ^ For example, instead of "Fixes sagemath#12345" use "Introduce new method
to calculate 1 + 2". -->
<!-- v Describe your changes below in detail. -->
<!-- v Why is this change required? What problem does it solve? -->
<!-- v If this PR resolves an open issue, please link to it here. For
example, "Fixes sagemath#12345". -->

A few small quality of life improvements to the conda setup.
Notably the environment files under `src` are now moved to the root, as
the old env files there were non-functional and only confused users.
Relately, removed the outdated,not working and untested instructions to
use conda solely to provide the system packages for sage-the-distro.

A few other improvements along the way:
- Make conda in devcontainer working again by forcing mamba v1 (v2 was
released a few days ago and breaks a few things related to the lock
files)
- Force usage of conda-forge everywhere (mixing of channels is no longer
supported)


### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->

- [ ] The title is concise and informative.
- [ ] The description explains in detail what this PR is about.
- [ ] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation and checked the documentation
preview.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on. For example,
-->
<!-- - sagemath#12345: short description why this is a dependency -->
<!-- - sagemath#34567: ... -->
    
URL: sagemath#38728
Reported by: Tobias Diez
Reviewer(s): Dima Pasechnik, Kwankyu Lee, Tobias Diez, Vincent Macri

.devcontainer/onCreate-conda.sh

@@ -2,8 +2,11 @@
 set -e
 
 # Create conda environment
-conda install mamba -n base -c conda-forge -y
-mamba env create --file src/environment-dev-3.11-linux.yml || mamba env update --file src/environment-dev-3.11-linux.yml
+conda config --env --add channels conda-forge
+conda config --env --set channel_priority strict
+conda update -y --all --override-channels -c conda-forge
+conda install mamba=1 -n base -y
+mamba env create -y --file environment-dev-3.11-linux.yml || mamba env update -y --file environment-dev-3.11-linux.yml
 conda init bash
 
 # Build sage

.github/workflows/ci-conda.yml

@@ -59,7 +59,7 @@ jobs:
         with:
           path: ~/conda_pkgs_dir
           key:
-            ${{ runner.os }}-conda-${{ hashFiles('src/environment-3.11.yml') }}
+            ${{ runner.os }}-conda-${{ hashFiles('environment-3.11.yml') }}
 
       - name: Setup Conda environment
         uses: conda-incubator/setup-miniconda@v3
@@ -70,7 +70,7 @@ jobs:
           channels: conda-forge
           channel-priority: true
           activate-environment: sage
-          environment-file: src/${{ matrix.conda-env }}-${{ matrix.python }}-${{ startsWith(matrix.os, 'macos') && (startsWith(runner.arch, 'ARM') && 'macos' || 'macos-x86_64') || 'linux' }}.yml
+          environment-file: ${{ matrix.conda-env }}-${{ matrix.python }}-${{ startsWith(matrix.os, 'macos') && (startsWith(runner.arch, 'ARM') && 'macos' || 'macos-x86_64') || 'linux' }}.yml
 
       - name: Print Conda environment
         shell: bash -l {0}

.github/workflows/ci-meson.yml

@@ -39,7 +39,7 @@ jobs:
         with:
           path: ~/conda_pkgs_dir
           key:
-            ${{ runner.os }}-conda-${{ hashFiles('src/environment-3.11-linux.yml') }}
+            ${{ runner.os }}-conda-${{ hashFiles('environment-3.11-linux.yml') }}
 
       - name: Compiler cache
         uses: hendrikmuhs/[email protected]
@@ -55,7 +55,7 @@ jobs:
           channels: conda-forge
           channel-priority: true
           activate-environment: sage
-          environment-file: src/environment-${{ matrix.python }}-${{ startsWith(matrix.os, 'macos') && (startsWith(runner.arch, 'ARM') && 'macos' || 'macos-x86_64') || 'linux' }}.yml
+          environment-file: environment-${{ matrix.python }}-${{ startsWith(matrix.os, 'macos') && (startsWith(runner.arch, 'ARM') && 'macos' || 'macos-x86_64') || 'linux' }}.yml
 
       - name: Print Conda environment
         shell: bash -l {0}

.github/workflows/conda-lock-update.py

@@ -4,30 +4,53 @@
 import subprocess
 
 script_dir = Path(__file__).resolve().parent
-root_dir = script_dir / '..' / '..'
+root_dir = script_dir / ".." / ".."
 
 subprocess.run([str(root_dir / "bootstrap-conda")])
 
 platforms = {
     "linux-64": "linux",
     "linux-aarch64": "linux-aarch64",
     "osx-64": "macos-x86_64",
-    "osx-arm64": "macos"
-    #"win-64": "win",
+    "osx-arm64": "macos",
+    # "win-64": "win",
 }
 pythons = ["3.9", "3.10", "3.11"]
 tags = ["", "-dev"]
-sources = ["", "src"]
 
 for platform_key, platform_value in platforms.items():
     for python in pythons:
         for tag in tags:
-            for src in sources:
-                env_file = root_dir / src / f"environment{tag}-{python}.yml"
-                lock_file = root_dir / src / f"environment{tag}-{python}-{platform_value}"
+            env_file = root_dir / f"environment{tag}-{python}.yml"
+            lock_file = root_dir / f"environment{tag}-{python}-{platform_value}"
+            lock_file_gen = root_dir / f"environment{tag}-{python}-{platform_value}.yml"
 
-                if not env_file.exists():
-                    continue
+            if not env_file.exists():
+                continue
 
-                print(f"Updating lock file for {env_file} at {lock_file}", flush=True)
-                subprocess.run(["conda-lock", "--channel", "conda-forge", "--kind", "env", "--platform", platform_key, "--file", str(env_file), "--lockfile", str(lock_file), "--filename-template", str(lock_file)])
+            print(f"Updating lock file for {env_file} at {lock_file_gen}", flush=True)
+            subprocess.run(
+                [
+                    "conda-lock",
+                    "--mamba",
+                    "--channel",
+                    "conda-forge",
+                    "--kind",
+                    "env",
+                    "--platform",
+                    platform_key,
+                    "--file",
+                    str(env_file),
+                    "--lockfile",
+                    str(lock_file),
+                    "--filename-template",
+                    str(lock_file),
+                ],
+                check=True,
+            )
+
+            # Add conda env name to lock file at beginning
+            with open(lock_file_gen, "r+") as f:
+                content = f.read()
+                f.seek(0, 0)
+                f.write(f"name: sage{tag}\n{content}")

.gitignore

@@ -28,12 +28,15 @@
 /src/lib/pkgconfig
 
 # Environment files generated by bootstrap-conda.
-# The files without Python version are no longer generated
+# The files without Python version and in src are no longer generated
 # but may still be in users' directories.
 /environment.yml
 /environment-3.9.yml
 /environment-3.10.yml
 /environment-3.11.yml
+/environment-dev-3.9.yml
+/environment-dev-3.10.yml
+/environment-dev-3.11.yml
 /environment-optional.yml
 /environment-optional-3.9.yml
 /environment-optional-3.10.yml

.gitpod.yml

@@ -7,7 +7,7 @@ tasks:
   - name: Setup
     # Create conda environment, then configure and build sage
     init: >-
-      && mamba env create --file src/environment-dev-3.11-linux.yml --prefix venv
+      && mamba env create --file environment-dev-3.11-linux.yml --prefix venv
       && conda config --append envs_dirs $(pwd)
       && conda activate $(pwd)/venv
       && ./bootstrap

.vscode/settings.json

@@ -1,7 +1,5 @@
 {
-    // This settings file is not ignored by git. It should be kept in sync with
-    // the trac repo.
-    "python.defaultInterpreterPath": "./venv/bin/python3",
+    // This settings file is not ignored by git.
     "files.exclude": {
         "**/__pycache__": true,
         "src/**/*.cpp": true,

bootstrap

@@ -227,9 +227,6 @@ save () {
 	src/doc/en/installation/*.txt \
 	$(find src/doc/en/reference/spkg -name index.rst -prune -o -maxdepth 1 -name "*.rst" -print) \
 	environment-3.[89].yml environment-3.1[0-9].yml \
-	src/environment-3.[89].yml src/environment-3.1[0-9].yml \
-	environment-optional-3.[89].yml environment-optional-3.1[0-9].yml \
-	src/environment-optional-3.[89].yml src/environment-optional-3.1[0-9].yml \
 	src/Pipfile \
 	src/pyproject.toml \
 	src/requirements.txt \

bootstrap-conda

@@ -84,7 +84,7 @@ unset ENABLE_SYSTEM_SITE_PACKAGES
 echo >&2 $0:$LINENO: generate conda environment files
 
     (
-        echo "name: sage-build"
+        echo "name: sage"
         echo "channels:"
         echo "  - conda-forge"
         echo "  - nodefaults"
@@ -106,58 +106,18 @@ echo >&2 $0:$LINENO: generate conda environment files
         for pkg in $SAGELIB_SYSTEM_PACKAGES; do
             echo "  - $pkg"
         done
-    ) > src/environment-template.yml
-
-    (
-        cat environment-template.yml
-        echo "  # optional packages"
-        for pkg in $OPTIONAL_SYSTEM_PACKAGES; do
-            echo "  - $pkg"
-        done
-    ) > environment-optional-template.yml
+    ) > environment-template.yml
 
     (
-        sed 's/name: sage/name: sage-dev/' src/environment-template.yml
+        sed 's/name: sage/name: sage-dev/' environment-template.yml
         echo "  # Additional dev tools"
         echo "  - conda-lock"
         for pkg in $DEVELOP_SYSTEM_PACKAGES; do
             echo "  - $pkg"
         done
-    ) > src/environment-dev-template.yml
-
-    (
-        cat src/environment-template.yml
-        echo "  # optional packages"
-        for pkg in $OPTIONAL_SYSTEM_PACKAGES $SAGELIB_OPTIONAL_SYSTEM_PACKAGES; do
-            echo "  - $pkg"
-        done
-    ) > src/environment-optional-template.yml
-
-    (
-        echo >&4 "  - pip:"
-        echo >&5 "  - pip:"
-        for PKG_BASE in $(sage-package list :standard: :optional: --has-file requirements.txt --no-file distros/conda.txt --no-file src; sage-package list :standard: :optional: --has-file version_requirements.txt --no-file requirements.txt --no-file distros/conda.txt --no-file src); do
-            eval PKG_SCRIPTS=\$path_$PKG_BASE PKG_TYPE=\$type_$PKG_BASE
-            SYSTEM_PACKAGES_FILE=$PKG_SCRIPTS/requirements.txt
-            if [ ! -f $SYSTEM_PACKAGES_FILE ]; then
-                SYSTEM_PACKAGES_FILE=$PKG_SCRIPTS/version_requirements.txt
-            fi
-            if grep -q SAGERUNTIME $PKG_SCRIPTS/dependencies $PKG_SCRIPTS/dependencies_order_only 2>/dev/null; then
-                : # cannot install packages that depend on the Sage library
-            else
-                case "$PKG_BASE:$PKG_TYPE" in
-                    $DEVELOP_SPKG_PATTERN:*) FD=4;;
-                    *:standard)              FD="4 5";;
-                    *)                       FD=5;;
-                esac
-                ${STRIP_COMMENTS} $SYSTEM_PACKAGES_FILE | while read -r line; do
-                    [ -n "$line" ] && for fd in $FD; do echo >&$fd "    - $line"; done
-                done
-            fi
-        done
-    ) 4>> /dev/null 5>> src/environment-optional-template.yml
+    ) > environment-dev-template.yml
 
-for f in environment environment-optional src/environment src/environment-optional src/environment-dev; do
+for f in environment environment-dev; do
     for python_version in 3.9 3.10 3.11; do
         sed -E 's/^( *- *)python *$/\1python='$python_version'/' $f-template.yml > $f-$python_version.yml
     done

condarc.yml

@@ -3,4 +3,3 @@
 channel_priority: strict
 channels:
   - conda-forge
-  - defaults