Add debian/fedora/arch/conda package information to build/pkgs, generate Dockerfiles and installation help; add tox.ini

[mkoeppe]

https://doc.sagemath.org/html/en/installation/source.html#linux-recommended-installation
gives apt-get install and yum install command lines to install system packages that will be recognized by build/pkgs/SPKG/spkg-configure.m4.

Several packages are missing, see comments in
https://github.com/sagemath/sage-prod/files/10659286/Dockerfile-ubuntu-minimal.gz

This ticket puts this information on a per-package basis into build/pkgs/SPKG/distros/debian.txt, .../fedora.txt, .../arch.txt for the following purposes:

• sage -info SPKG prints out this information.

• Provide a tox.ini that generates and runs Dockerfiles that test the installation on various systems. Example (running 3 builds in parallel):

     $ tox -p auto -e archlinux-latest,ubuntu-focal-minimal,debian-buster-maximal
  ⠸
  

Incidental change on this ticket:

• Mark gfortran as a type=standard package. It is a prereq of standard package numpy. Its status as a type=optional package predates the clarifications brought by the development of the spkg-configure.m4 mechanism (Meta-ticket: spkg-configure: Try to use as many system packages as possible #27330).

---

Possible follow-up and related tickets:

• Add GitHub Actions workflow for testing the build on various Linux distributions via docker; macOS with homebrew; and Windows (cygwin) #29087 - Add a GitHub workflow that runs this automatically, with many parallel jobs.
• Also the apt-cyg command lines at https://trac.sagemath.org/wiki/Cygwin64Port could be generated in the same way.
• ./bootstrap: Format build/pkgs/SPKG/distros/ information to produce the apt-get/yum command lines shown in the installation manual #26964: Aggregate and format it (during ./bootstrap) to produce the command lines shown in the manual. This ticket already prepares it by moving these command lines into separate .txt files.
• sage_bootstrap: Add command "sage -package list --output={install-requires,requirements,pipfile,debian,....}" #29041: at ./bootstrap time, generate src/requirements.txt, src/constraints.txt, src/setup.cfg [install_requires] from build/pkgs
• Add SAGE_ROOT/environment.yml for "conda env create -f". #28745: Add SAGE_ROOT/environment.yml for conda env create -f.
• Add to tox.ini a conda run (either with a fresh environment or fresh conda install) without docker - to test with macOS conda
• Add to tox.ini a homebrew run (with a fresh install not in /usr/local)

Resources:

• https://repology.org/

CC: @dimpase @embray @antonio-rojas @isuruf @timokau

Component: build

Author: Matthias Koeppe

Branch: 23c6334

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/29053

[mkoeppe]

Branch: u/mkoeppe/installation_manual__generate__apt_get_install__and__yum_install__lines_from_build_pkgs_at___bootstrap_time

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

3a5cfb7

build/pkgs/*/debian.txt: Add optional packages

[sagetrac-git]

Commit: 3a5cfb7

[sagetrac-git]

Changed commit from 3a5cfb7 to 79d912e

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

79d912e

build/pkgs/*/fedora.txt: New, from installation manual 9.0

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

8a739b1

sage -info: Print info on equivalent system packages

[sagetrac-git]

Changed commit from 79d912e to 8a739b1

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

3ef64c4	Document the debian.txt, fedora.txt files.
ebee631	Add build/pkgs/gcc/fedora.txt
221c344	Strip comments from debian.txt, fedora.txt files

[sagetrac-git]

Changed commit from 8a739b1 to 221c344

[dimpase]

comment:7

how do you generate these per package?

by the way, Arch has namewise
a naming scheme close to Gentoo and Homebrew.

[mkoeppe]

comment:8

Replying to @dimpase:

how do you generate these per package?

I created these files by manual copy-paste from the installation manual.

[sagetrac-git]

Changed commit from 221c344 to ff4a551

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

46957b4	tox.ini: New - test spkg-configure on ubuntu docker images
ce99fad	Merge tag '9.1.beta1' into t/29053/installation_manual__generate__apt_get_install__and__yum_install__lines_from_build_pkgs_at___bootstrap_time
0dd2c03	build/bin/sage-spkg: Reformat the install command to 1 line
b5abc8b	build/bin/sage-spkg: fixup
97b09de	Make gfortran a standard package - it is a prereq of standard package numpy
ff4a551	Fix typo

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

a472a9a	Add fedora and debian Docker tests
1a31b40	src/doc/en/installation/{debian|fedora}[-optional].txt
0c2148e	more envs

[sagetrac-git]

Changed commit from ff4a551 to 0c2148e

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

e38dd8e

src/doc/en/installation/source.rst: Use literalinclude

[sagetrac-git]

Changed commit from 0c2148e to e38dd8e

[mkoeppe]

Author: Matthias Koeppe

[dimpase]

Reviewer: Dima Pasechnik

[mkoeppe]

comment:65

Thank you!

[dimpase]

comment:66

What's suprising to me is that these GH Actions don't time out, as these builds take quite a while...

[mkoeppe]

comment:67

It's quite generous - it provides 6 hours per job and up to 20 Jobs in parallel.

[vbraun]

Changed branch from u/mkoeppe/installation_manual__generate__apt_get_install__and__yum_install__lines_from_build_pkgs_at___bootstrap_time to 23c6334

[mkoeppe]

Changed commit from 23c6334 to none

[mkoeppe]

comment:69

added follow-up ticket to the description

[embray]

comment:71

Somehow missed this in the fervor over other issues, but this kicks ass, thank you! I'll open a ticket to adding cygwin to this as well. repology.org also just recently added Cygwin.

[embray]

comment:72

One thing I might suggest is that rather than having the distro-specific commands hard-coded in sage-spkg, they could go in a text file or something, with the option to override and/or add support for other distros without having to patch sage-spkg.

That could be done in a follow-up though.

[dimpase]

comment:73

Replying to @embray:

Somehow missed this in the fervor over other issues, but this kicks ass, thank you! I'll open a ticket to adding cygwin to this as well. repology.org also just recently added Cygwin.

see, mkoeppe practically buried me under his tickets, and then came you with #29118 :P

[mkoeppe]

comment:74

Replying to @embray:

I'll open a ticket to adding cygwin to this as well. repology.org also just recently added Cygwin.

The cygwin ticket is #29106 (Add cygwin package information). It would be great if also corresponding CI scripts could be added, to help elevate the cygwin platform to a first class target for sage.

[mkoeppe]

comment:75

Replying to @embray:

One thing I might suggest is that rather than having the distro-specific commands hard-coded in sage-spkg, they could go in a text file or something, with the option to override and/or add support for other distros without having to patch sage-spkg.

That could be done in a follow-up though.

Thanks, I have added this suggestion to #29146, which already has a number of ideas for improvements.

[mkoeppe]

Changed branch from 23c6334 to u/mkoeppe/23c6334cf2927a4d917a27b3963471e8242d86a3

[mkoeppe]

comment:77

(sorry, pushed to the wrong ticket by accident)

[mkoeppe]

Changed branch from u/mkoeppe/23c6334cf2927a4d917a27b3963471e8242d86a3 to 23c6334

[kiwifb]

comment:78

Somehow in recent merges I experienced failure to build the documentation in sage-on-gentoo. It seems the literalinclude are causing me trouble.

[installat] The HTML pages are in ../../build_doc/html/en/installation.
Error building the documentation.
Traceback (most recent call last):
  File "sage_setup/docbuild/__main__.py", line 2, in <module>
    main()
  File "/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_7/sage_setup/docbuild/__init__.py", line 1720, in main
    builder()
  File "/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_7/sage_setup/docbuild/__init__.py", line 336, in _wrapper
    build_many(build_other_doc, L)
  File "/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_7/sage_setup/docbuild/__init__.py", line 280, in build_many
    _build_many(target, args, processes=NUM_THREADS)
  File "/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_7/sage_setup/docbuild/utils.py", line 283, in build_many
    raise worker_exc.original_exception
OSError: /dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_7/doc/en/installation/source.rst:228: WARNING: Include file '/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_7/doc/en/installation/debian.txt' not found or reading it failed

I am thinking of removing the installation instruction from sage-on-gentoo but I am curious at what could have caused this. Especially when you consider that it started happening sometime after this ticket was merged. My last successful build of volker's branch is dated 20th of February and would corresponding to the first merging attempt of #21785.

[mkoeppe]

comment:79

Replying to @kiwifb:

Somehow in recent merges I experienced failure to build the documentation in sage-on-gentoo. It seems the literalinclude are causing me trouble.

[installat] The HTML pages are in ../../build_doc/html/en/installation.
Error building the documentation.
...
OSError: /dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_7/doc/en/installation/source.rst:228: WARNING: Include file '/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_7/doc/en/installation/debian.txt' not found or reading it failed

Yes, as of #26964, these files are being generated by bootstrap and are not sources in the git tree any more.

Distributors should build either from a bootstrapped source tree or from a sagelib sdist (see ticket #21516 - Fix sagelib sdist (src/setup.py sdist); which needs help).

I am thinking of removing the installation instruction from sage-on-gentoo

Yes, there's no point in shipping the installation manual with a distribution package.

[kiwifb]

comment:80

I see, a sagelib sdist is a good development as it makes it more like a regular python package I guess. But right now I am missing the ability to build sagelib directly from git head. Thanks for explanation.

[dimpase]

comment:81

This has missed build/pkgs/r/ from #28884.
I've opened #29273 to deal with it.

[isuruf]

comment:82

Replying to @mkoeppe:

Replying to @isuruf:

Replying to @mkoeppe:

@isuruf: Compiling python3 on conda-forge gives me an error that looks like it's related to ffi - https://github.com/mkoeppe/sage/runs/410329852

Looks like you are not activating the conda environment.

Thank you! That helped.
But now (https://github.com/mkoeppe/sage/runs/412042067), still in the python3 build, I am hitting again #29012:

Testing importing of various modules...
ctypes module imported OK
math module imported OK
hashlib module imported OK
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/sage/local/var/tmp/sage/build/python3-3.7.3.p1/src/Lib/crypt.py", line 3, in <module>
    import _crypt
ModuleNotFoundError: No module named '_crypt'
crypt module failed to import

This patch might help https://github.com/conda-forge/python-feedstock/blob/master/recipe/patches/0014-Fix-cross-compilation-on-Debian-based-distros.patch