sagemathgh-35986: Use conda-lock for reproducible conda env

    
<!-- ^^^^^
Please provide a concise, informative and self-explanatory title.
Don't put issue numbers in there, do this in the PR body below.
For example, instead of "Fixes sagemath#1234" use "Introduce new method to
calculate 1+1"
-->
<!-- Describe your changes here in detail -->

<!-- Why is this change required? What problem does it solve? -->
<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes sagemath#12345". -->
<!-- If your change requires a documentation PR, please link it
appropriately. -->

The conda workflow is broken since a few months due to various issues
that arise from some packages being updated in conda-forge and sage not
being quite compatible with the new version. For example, currently it
fails as Cython 3.0.3 is used.

To prevent such issues happening in the feature, we use [conda-
lock](https://conda.github.io/conda-lock/) to create a lock file for
each os + python combination, from which one can then create a
reproducible conda env. These lock files are created using commands of
the form:
```bash
conda-lock --channel conda-forge --kind env --platform linux-64
--platform osx-64 --file ./src/environment-dev-3.10.yml --filename-
template "src/environment-dev-3.10-{platform}"
```


Follow-up: Add a github workflow that updates these lock files
automatically, similar to:
- https://github.com/ibis-
project/ibis/blob/master/.github/workflows/conda-lock.yml
- https://github.com/svopper/munaiah-
analyser/blob/main/out/SciTools/iris/.github_workflows_refresh-
lockfiles.yml
- https://github.com/chipsalliance/fpga-interchange-
tests/blob/main/.github/workflows/update_conda_lock.yml
- https://github.com/ESMValGroup/ESMValTool/blob/main/.github/workflows/
create-condalock-file.yml

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->
<!-- If your change requires a documentation PR, please link it
appropriately -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
<!-- Feel free to remove irrelevant items. -->

- [x] The title is concise, informative, and self-explanatory.
- [x] The description explains in detail what this PR is about.
- [ ] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

- sagemath#36405: to get env files for each
python version
- sagemath#36767

<!-- List all open PRs that this PR logically depends on
- sagemath#12345: short description why this is a dependency
- sagemath#34567: ...
-->

<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
    
URL: sagemath#35986
Reported by: Tobias Diez
Reviewer(s): Dima Pasechnik, Isuru Fernando, Matthias Köppe, Tobias Diez

.devcontainer/devcontainer.json

@@ -8,7 +8,7 @@
 	"remoteUser": "vscode",
 
 	// Setup conda environment
-	"onCreateCommand": ".devcontainer/onCreate-conda.sh",
+	"onCreateCommand": ".devcontainer/onCreate-conda.sh || true",
 
 	// Install additional features.
 	"features": {

.devcontainer/onCreate-conda.sh

@@ -2,9 +2,8 @@
 set -e
 
 # Create conda environment
-./bootstrap-conda
 conda install mamba -n base -c conda-forge -y
-mamba env create --file src/environment-dev-3.11.yml || mamba env update --file src/environment-dev-3.11.yml
+mamba env create --file src/environment-dev-3.11-linux.yml || mamba env update --file src/environment-dev-3.11-linux.yml
 conda init bash
 
 # Build sage

.github/workflows/ci-conda.yml

@@ -18,11 +18,11 @@ concurrency:
 jobs:
   test:
     name: Conda
-    runs-on: ${{ matrix.os }}
+    runs-on: ${{ matrix.os }}-latest
 
     strategy:
       matrix:
-        os: [ubuntu-latest, macos-latest]
+        os: [ubuntu, macos]
         python: ['3.9', '3.10', '3.11']
         # Optional environment is disabled for now as its not yet working
         # environment: [environment, environment-optional]
@@ -38,9 +38,6 @@ jobs:
           GH_TOKEN: ${{ github.token }}
           SAGE_CI_FIXES_FROM_REPOSITORIES: ${{ vars.SAGE_CI_FIXES_FROM_REPOSITORIES }}
 
-      - name: Create conda environment files
-        run: ./bootstrap-conda
-
       - name: Cache conda packages
         uses: actions/cache@v3
         with:
@@ -57,7 +54,7 @@ jobs:
           channels: conda-forge
           channel-priority: true
           activate-environment: sage
-          environment-file: src/${{ matrix.conda-env }}-${{ matrix.python }}.yml
+          environment-file: src/${{ matrix.conda-env }}-${{ matrix.python }}-${{ startsWith(matrix.os, 'macos') && 'macos' || 'linux' }}.yml
 
       - name: Print Conda environment
         shell: bash -l {0}

.github/workflows/conda-lock-update.py

@@ -0,0 +1,33 @@
+#!/usr/bin/env python3
+
+from pathlib import Path
+import subprocess
+
+script_dir = Path(__file__).resolve().parent
+root_dir = script_dir / '..' / '..'
+
+subprocess.run([str(root_dir / "bootstrap-conda")])
+
+platforms = {
+    "linux-64": "linux",
+    "linux-aarch64": "linux-aarch64",
+    "osx-64": "macos",
+    "osx-arm64": "macos-arm64"
+    #"win-64": "win",
+}
+pythons = ["3.9", "3.10", "3.11"]
+tags = ["", "-dev"]
+sources = ["", "src"]
+
+for platform_key, platform_value in platforms.items():
+    for python in pythons:
+        for tag in tags:
+            for src in sources:
+                env_file = root_dir / src / f"environment{tag}-{python}.yml"
+                lock_file = root_dir / src / f"environment{tag}-{python}-{platform_value}"
+
+                if not env_file.exists():
+                    continue
+
+                print(f"Updating lock file for {env_file} at {lock_file}", flush=True)
+                subprocess.run(["conda-lock", "--channel", "conda-forge", "--kind", "env", "--platform", platform_key, "--file", str(env_file), "--lockfile", str(lock_file), "--filename-template", str(lock_file)])

.gitpod.yml

@@ -7,8 +7,7 @@ tasks:
   - name: Setup
     # Create conda environment, then configure and build sage
     init: >-
-      ./bootstrap-conda
-      && mamba env create --file src/environment-dev-3.11.yml --prefix venv
+      && mamba env create --file src/environment-dev-3.11-linux.yml --prefix venv
       && conda config --append envs_dirs $(pwd)
       && conda activate $(pwd)/venv
       && ./bootstrap

.vscode/settings.json

@@ -35,5 +35,6 @@
         "sagemath",
         "Cython"
     ],
-    "editor.formatOnType": true
+    "editor.formatOnType": true,
+    "esbonio.sphinx.confDir": ""
 }

bootstrap-conda

@@ -115,6 +115,7 @@ echo >&2 $0:$LINENO: generate conda environment files
     (
         sed 's/name: sage/name: sage-dev/' src/environment-template.yml
         echo "  # Additional dev tools"
+        echo "  - conda-lock"
         for pkg in $DEVELOP_SYSTEM_PACKAGES; do
             echo "  - $pkg"
         done

build/pkgs/sphinx/distros/conda.txt

@@ -1 +1 @@
-sphinx<8,>=5.2
+sphinx>=5.2

environment-3.10-linux-aarch.yml

@@ -0,0 +1,193 @@
+# Generated by conda-lock.
+# platform: linux-aarch64
+# input_hash: d745aab1c14fd9333fe506502bdbc2bb5ab40c689cf58e8b1eb71d8e283a8e6d
+
+channels:
+  - conda-forge
+dependencies:
+  - _sysroot_linux-aarch64_curr_repodata_hack=4=h57d6b7b_13
+  - ca-certificates=2023.7.22=hcefe29a_0
+  - font-ttf-dejavu-sans-mono=2.37=hab24e00_0
+  - font-ttf-inconsolata=3.000=h77eed37_0
+  - font-ttf-source-code-pro=2.038=h77eed37_0
+  - font-ttf-ubuntu=0.83=hab24e00_0
+  - kernel-headers_linux-aarch64=4.18.0=h5b4a56d_13
+  - ld_impl_linux-aarch64=2.40=h2d8c526_0
+  - libboost-headers=1.82.0=h8af1aa0_2
+  - libgcc-devel_linux-aarch64=12.3.0=h8b5ab12_2
+  - libgomp=13.2.0=hf8544c7_2
+  - libstdcxx-devel_linux-aarch64=12.3.0=h8b5ab12_2
+  - libstdcxx-ng=13.2.0=h9a76618_2
+  - mathjax=3.2.2=h8af1aa0_0
+  - pari-elldata=0.0.20161017=0
+  - pari-galdata=0.0.20180411=0
+  - pari-galpol=0.0.20180625=0
+  - pari-seadata-small=0.0.20090618=0
+  - python_abi=3.10=4_cp310
+  - tzdata=2023c=h71feb2d_0
+  - fonts-conda-forge=1=0
+  - pari-seadata=0.0.20090618=0
+  - sysroot_linux-aarch64=2.17=h5b4a56d_13
+  - binutils_impl_linux-aarch64=2.40=h870a726_0
+  - fonts-conda-ecosystem=1=0
+  - binutils=2.40=h64c2a2e_0
+  - binutils_linux-aarch64=2.40=h94bbfa1_2
+  - _openmp_mutex=4.5=2_kmp_llvm
+  - libgcc-ng=13.2.0=hf8544c7_2
+  - bc=1.07.1=hf897c2e_0
+  - bdw-gc=8.0.6=hd62202e_0
+  - bzip2=1.0.8=hf897c2e_4
+  - c-ares=1.20.1=h31becfc_0
+  - cliquer=1.22=hb9de7d4_0
+  - gengetopt=2.23=h01db608_0
+  - gf2x=1.3.0=h1b3b3a3_2
+  - giflib=5.2.1=hb4cce97_3
+  - gmp=6.2.1=h7fd3ca4_0
+  - icu=72.1=hcf00150_0
+  - jpeg=9e=h2a766a3_3
+  - keyutils=1.6.1=h4e544f5_0
+  - lerc=4.0.0=h4de3ea5_0
+  - libatomic_ops=7.6.14=h4e544f5_0
+  - libbraiding=1.1=h7fd3ca4_0
+  - libdeflate=1.17=hb4cce97_0
+  - libev=4.33=h516909a_1
+  - libexpat=2.5.0=hd600fc2_1
+  - libffi=3.4.2=h3557bc0_5
+  - libgfortran5=13.2.0=h582850c_2
+  - libiconv=1.17=h9cdd2b7_0
+  - libnsl=2.0.0=h31becfc_1
+  - libsanitizer=12.3.0=h8ebda82_2
+  - libsodium=1.0.18=hb9de7d4_1
+  - libtool=2.4.7=h4de3ea5_0
+  - libuuid=2.38.1=hb4cce97_0
+  - libuv=1.46.0=h31becfc_0
+  - libwebp-base=1.2.4=h4e544f5_0
+  - libzlib=1.2.13=h31becfc_5
+  - lrcalc=2.1=h4de3ea5_5
+  - m4=1.4.18=h516909a_1001
+  - make=4.3=h309ac5b_1
+  - metis=5.1.0=h2f0025b_1007
+  - nauty=2.7.2=hf897c2e_0
+  - ncurses=6.4=h2e1726e_0
+  - ninja=1.11.1=hdd96247_0
+  - openssl=3.1.3=h31becfc_0
+  - palp=2.20=hb9de7d4_0
+  - patch=2.7.6=hf897c2e_1002
+  - pkg-config=0.29.2=hb9de7d4_1008
+  - planarity=3.0.0.5=hb9de7d4_1002
+  - primesieve=11.1=h2f0025b_0
+  - qhull=2020.2=hd62202e_2
+  - rhash=1.4.4=h31becfc_0
+  - symmetrica=3.0.1=hd600fc2_0
+  - xz=5.2.6=h9cdd2b7_0
+  - cddlib=1!0.94m=h719063d_0
+  - ecm=7.0.4=h719063d_1002
+  - expat=2.5.0=hd600fc2_1
+  - gcc_impl_linux-aarch64=12.3.0=hcde2664_2
+  - givaro=4.1.1=h192cbe9_1
+  - glpk=5.0=h66325d0_0
+  - libedit=3.1.20191231=he28a2e2_2
+  - libgfortran-ng=13.2.0=he9431aa_2
+  - libhomfly=1.02r6=hb9de7d4_0
+  - libnghttp2=1.52.0=h250e5c5_0
+  - libpng=1.6.39=hf9034f9_0
+  - libsqlite=3.43.0=h194ca79_0
+  - libssh2=1.11.0=h492db2e_0
+  - libxml2=2.10.4=h430b14f_0
+  - mpfr=4.2.0=h96f194b_0
+  - ntl=11.4.3=h0d7519b_1
+  - perl=5.32.1=4_h31becfc_perl5
+  - primecount=7.9=hd600fc2_0
+  - readline=8.2=h8fc344f_1
+  - tar=1.34=h048efde_0
+  - tk=8.6.13=h194ca79_0
+  - zeromq=4.3.4=h01db608_1
+  - zlib=1.2.13=h31becfc_5
+  - zstd=1.5.5=h4c53e97_0
+  - autoconf=2.71=pl5321h2148fe1_1
+  - ecl=21.2.1=haa44c19_2
+  - freetype=2.12.1=hf0a5ef3_2
+  - gcc=12.3.0=hc1b51f9_2
+  - gcc_linux-aarch64=12.3.0=h464a8f7_2
+  - gfan=0.6.2=h5f589ec_1003
+  - gfortran_impl_linux-aarch64=12.3.0=hb7244be_2
+  - gxx_impl_linux-aarch64=12.3.0=hcde2664_2
+  - krb5=1.21.2=hc419048_0
+  - libboost=1.82.0=hbfc56d7_2
+  - libflint=2.9.0=hd3470fa_ntl_100
+  - libhwloc=2.9.1=h21e8147_0
+  - libopenblas=0.3.23=pthreads_hd703e6f_0
+  - libtiff=4.5.0=h4c1066a_2
+  - llvm-openmp=17.0.2=h8b0cb96_0
+  - m4ri=20140914=h75e8696_1005
+  - mpc=1.3.1=hf4c8f4c_0
+  - mpfi=1.5.4=h846f343_1001
+  - pari=2.15.4=h169c2a7_2_pthread
+  - ppl=1.2=h984aac9_1006
+  - python=3.10.12=hbbe8eec_0_cpython
+  - qd=2.3.22=h05efe27_1004
+  - sqlite=3.43.0=h3b3482f_0
+  - tachyon=0.99b6=h63ab1d9_1001
+  - texinfo=7.0=pl5321h17f021e_0
+  - appdirs=1.4.4=pyh9f0ad1d_0
+  - arb=2.23.0=h37d5dab_0
+  - automake=1.16.5=pl5321h8af1aa0_0
+  - c-compiler=1.6.0=h31becfc_0
+  - cachetools=5.3.1=pyhd8ed1ab_0
+  - chardet=5.2.0=py310hbbe02a8_1
+  - colorama=0.4.6=pyhd8ed1ab_0
+  - distlib=0.3.7=pyhd8ed1ab_0
+  - eclib=20230424=h0bc7b0f_0
+  - filelock=3.12.4=pyhd8ed1ab_0
+  - fontconfig=2.14.2=ha9a116f_0
+  - fplll=5.4.4=h5cd656c_0
+  - gfortran=12.3.0=h8d4031d_2
+  - gfortran_linux-aarch64=12.3.0=h1993883_2
+  - gxx=12.3.0=hc1b51f9_2
+  - gxx_linux-aarch64=12.3.0=h21accf6_2
+  - lcalc=2.0.5=h3264cc0_1
+  - libblas=3.9.0=17_linuxaarch64_openblas
+  - libboost-devel=1.82.0=h37bb5a9_2
+  - libbrial=1.2.12=h17533bf_1
+  - libcurl=8.3.0=h4e8248e_0
+  - libwebp=1.2.4=h7bdf6e5_1
+  - m4rie=20150908=h75e8696_1001
+  - maxima=5.45.0=haa44c19_3
+  - openblas=0.3.23=pthreads_hef96516_0
+  - packaging=23.2=pyhd8ed1ab_0
+  - pluggy=1.3.0=pyhd8ed1ab_0
+  - singular=4.2.1.p3=h3d4c4c6_2
+  - sympow=2.023.6=h157afb5_3
+  - tbb=2021.9.0=h4c384f3_0
+  - tomli=2.0.1=pyhd8ed1ab_0
+  - typing_extensions=4.8.0=pyha770c72_0
+  - zipp=3.17.0=pyhd8ed1ab_0
+  - boost-cpp=1.82.0=h62f3a30_2
+  - brial=1.2.12=pyh694c41f_1
+  - cmake=3.27.6=hef020d8_0
+  - curl=8.3.0=h4e8248e_0
+  - cxx-compiler=1.6.0=h2a328a1_0
+  - fortran-compiler=1.6.0=h7048d53_0
+  - importlib-metadata=6.8.0=pyha770c72_0
+  - libcblas=3.9.0=17_linuxaarch64_openblas
+  - libgd=2.3.3=h5fc1a20_5
+  - liblapack=3.9.0=17_linuxaarch64_openblas
+  - pyproject-api=1.6.1=pyhd8ed1ab_0
+  - typing-extensions=4.8.0=hd8ed1ab_0
+  - arpack=3.7.0=hf862f49_2
+  - compilers=1.6.0=h8af1aa0_0
+  - fflas-ffpack=2.4.3=hf104d39_2
+  - gsl=2.7=h294027d_0
+  - iml=1.0.5=h9076c59_1003
+  - importlib_metadata=6.8.0=hd8ed1ab_0
+  - liblapacke=3.9.0=17_linuxaarch64_openblas
+  - platformdirs=3.11.0=pyhd8ed1ab_0
+  - suitesparse=5.10.1=h1404dd6_1
+  - blas-devel=3.9.0=17_linuxaarch64_openblas
+  - giac=1.9.0.21=h04922a4_1
+  - igraph=0.9.10=hefb87a8_1
+  - linbox=1.6.3=h31716a8_7
+  - virtualenv=20.24.4=pyhd8ed1ab_0
+  - blas=2.117=openblas
+  - rw=0.9=hf897c2e_0
+  - tox=4.11.3=pyhd8ed1ab_0