Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New package: sagemath-9.5 #34030

Merged
merged 1 commit into from
Feb 3, 2022
Merged

Conversation

tornaria
Copy link
Contributor

@tornaria tornaria commented Nov 11, 2021

This is WIP, so I can get feedback.

Supported architectures: x86_64, x86_64-musl, i686. We also try to make sure it works with musl-1.2.

The testsuite can be run normal (-Q) or long (-K). For CI it runs normal (~ 1/2 time) and it uses a fixed random seed 0 to avoid random failures. When running locally, the random seed is random so one may get occasional test failures that depend on a particular value of the random seed.

The package sagemath-test is a dummy package whose purpose is to run the testsuite with sage installed as a package and without the build directory. For this to make sense, the masterdir has to be clean (without traces of the sagemath build, this should be ok after running "pkg" or "pkg -Q").

@dkwo
Copy link
Contributor

dkwo commented Nov 11, 2021

I'd go with sagemath.
Is it possible to not build as root, or otherwise pass the option?

checking for root user... yes
configure: error: You cannot build Sage as root, switch to an unprivileged user.  (If building in a container, use --enable-build-as-root.)

We need more expert advice with relocation.

@fosslinux
Copy link
Contributor

Should use the --enable-build-as-root option

@dkwo
Copy link
Contributor

dkwo commented Nov 12, 2021

With this, at least it gets to the end

=> Registering new packages to /host/binpkgs/sagemath
index: added `sage-9.5.beta5_1' (x86_64).
index: 1 packages registered.
=> sage-9.5.beta5_1: removing autodeps, please wait...
=> sage-9.5.beta5_1: cleaning build directory...
=> sage: removing files from destdir...

on the x86_64

make[2]: Leaving directory '/builddir/sage-9.5.beta5/build/make'

real    22m23.377s
user    164m31.944s
sys     6m1.798s
Sage build/upgrade complete!

Obviously, it complains that

=> sage-9.5.beta5_1: running pre-pkg hook: 99-pkglint ...
=> WARNING: sage-9.5.beta5_1: liblinbox.so.0 not found in common/shlibs!
=> WARNING: sage-9.5.beta5_1: liblrcalc.so.1 not found in common/shlibs!
=> WARNING: sage-9.5.beta5_1: libbrial_groebner.so.3 not found in common/shlibs!
=> WARNING: sage-9.5.beta5_1: libcliquer.so.1 not found in common/shlibs!
=> WARNING: sage-9.5.beta5_1: libpython3.9.so.1.0 not found in common/shlibs!
=> WARNING: sage-9.5.beta5_1: libbrial.so.3 not found in common/shlibs!
=> WARNING: sage-9.5.beta5_1: libgap.so.0 not found in common/shlibs!
=> WARNING: sage-9.5.beta5_1: libbraiding.so.0 not found in common/shlibs!
=> WARNING: sage-9.5.beta5_1: libhomfly.so.0 not found in common/shlibs!

and lots of => WARNING: sage-9.5.beta5_1: removed empty dir: ...

@dkwo
Copy link
Contributor

dkwo commented Nov 12, 2021

Other than that, it installs fine.
I will run doctests and perhaps test on musl, but it looks not so bad.

[nicolo@mabragor ~]$ sage
/usr/lib/sage-9.5.beta5/bin/sage-env: line 122: cd: /builddir/sage-9.5.beta5: No such file or directory
Warning: overwriting SAGE_ROOT environment variable:
Old SAGE_ROOT=/builddir/sage-9.5.beta5
New SAGE_ROOT=
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 9.5.beta5, Release Date: 2021-10-28               │
│ Using Python 3.9.7. Type "help()" for help.                        │
└────────────────────────────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Warning: this is a prerelease version, and it may be unstable.     ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
sage: 2+2
4

@dkwo
Copy link
Contributor

dkwo commented Nov 12, 2021

Either ./configure --prefix=SAGE_LOCAL or edit #SAGE_ROOT=/path/to/sage-version at the beginning of the scritp should fix that.

@dkwo
Copy link
Contributor

dkwo commented Nov 12, 2021

About the doctests, I'm still unable to perform them
From within sage

sage: run_doctests(sage.combinat.affine_permutation)
too few successful tests, not using stored timings
Running doctests with ID 2021-11-12-21-34-16-32069d04.
Using --optional=sage
Doctesting 1 file.
sage -t --random-seed=0 /builddir/sage-9.5.beta5/src/sage/combinat/affine_permutation.py
    FileNotFoundError in doctesting framework
**********************************************************************
Traceback (most recent call last):
  File "/usr/lib/sage-9.5.beta5/lib/python3.9/site-packages/sage/doctest/forker.py", line 2465, in __call__
    doctests, extras = self._run(runner, options, results)
  File "/usr/lib/sage-9.5.beta5/lib/python3.9/site-packages/sage/doctest/forker.py", line 2513, in _run
    doctests, extras = self.source.create_doctests(sage_namespace)
  File "/usr/lib/sage-9.5.beta5/lib/python3.9/site-packages/sage/doctest/sources.py", line 725, in create_doctests
    raise IOError(errno.ENOENT, "File does not exist", self.path)
FileNotFoundError: [Errno 2] File does not exist: '/builddir/sage-9.5.beta5/src/sage/combinat/affine_permutation.py'

----------------------------------------------------------------------
sage -t --random-seed=0 /builddir/sage-9.5.beta5/src/sage/combinat/affine_permutation.py  # FileNotFoundError in doctesting framework
----------------------------------------------------------------------
Total time for all tests: 0.0 seconds
    cpu time: 0.0 seconds
    cumulative wall time: 0.0 seconds

From terminal

[nicolo@mabragor ~]$ sage --testall
/usr/lib/sage-9.5.beta5/bin/sage-env: line 122: cd: /builddir/sage-9.5.beta5: No such file or directory
Warning: overwriting SAGE_ROOT environment variable:
Old SAGE_ROOT=/builddir/sage-9.5.beta5
New SAGE_ROOT=
Running doctests with ID 2021-11-12-21-36-43-b1335b9b.
Using --optional=dochtml,pip,sage,sage.geometry.polyhedron,sage.rings.real_double
Doctesting entire Sage library.
Traceback (most recent call last):
  File "/usr/lib/sage-9.5.beta5/bin/sage-runtests", line 151, in <module>
    err = DC.run()
  File "/usr/lib/sage-9.5.beta5/lib/python3.9/site-packages/sage/doctest/control.py", line 1251, in run
    self.expand_files_into_sources()
  File "/usr/lib/sage-9.5.beta5/lib/python3.9/site-packages/sage/doctest/control.py", line 835, in expand_files_into_sources
    self.sources = [FileDocTestSource(path, self.options) for path in expand()]
  File "/usr/lib/sage-9.5.beta5/lib/python3.9/site-packages/sage/doctest/control.py", line 835, in <listcomp>
    self.sources = [FileDocTestSource(path, self.options) for path in expand()]
  File "/usr/lib/sage-9.5.beta5/lib/python3.9/site-packages/sage/doctest/sources.py", line 527, in __init__
    raise ValueError("unknown file extension %r"%ext)
ValueError: unknown file extension ''

What am I missing?

@fosslinux
Copy link
Contributor

You can ignore the common/shlibs and empty dir messages.

The doctest failures probably have something to do with using python 3.9; why is this?

srcpkgs/sage/template Outdated Show resolved Hide resolved
@tornaria
Copy link
Contributor Author

We need more expert advice with relocation.

I think one way would be to compile sage in-place, then remove all unnecessary stuff (or vinstall all necessary stuff).

I'll give it a try later.

The alternative is how sage builds binary tarballs: compile sage in a directory with a particular pathname (long enough), then search-and-replace this pathname with the location it will be installed.

In their binary tarballs they do this search-and-replace step the first time sage runs after being extracted, in this way sage can be extracted anywhere and will fix its paths the first time it runs. We would do this at xbps-src install time.

@tornaria
Copy link
Contributor Author

Obviously, it complains that

=> sage-9.5.beta5_1: running pre-pkg hook: 99-pkglint ...
=> WARNING: sage-9.5.beta5_1: liblinbox.so.0 not found in common/shlibs!
=> WARNING: sage-9.5.beta5_1: liblrcalc.so.1 not found in common/shlibs!
=> WARNING: sage-9.5.beta5_1: libbrial_groebner.so.3 not found in common/shlibs!
=> WARNING: sage-9.5.beta5_1: libcliquer.so.1 not found in common/shlibs!
=> WARNING: sage-9.5.beta5_1: libpython3.9.so.1.0 not found in common/shlibs!
=> WARNING: sage-9.5.beta5_1: libbrial.so.3 not found in common/shlibs!
=> WARNING: sage-9.5.beta5_1: libgap.so.0 not found in common/shlibs!
=> WARNING: sage-9.5.beta5_1: libbraiding.so.0 not found in common/shlibs!
=> WARNING: sage-9.5.beta5_1: libhomfly.so.0 not found in common/shlibs!

These are false warnings, we don't want this pkg to provide any shlibs and those shlibs will not be in any standard path anyway. As a matter of fact, all of those should eventually be gone and used from system instead.

@tornaria
Copy link
Contributor Author

You can ignore the common/shlibs and empty dir messages.

The doctest failures probably have something to do with using python 3.9; why is this?

No, this is because sage needs the source files to be available, and I didn't include in the pkg. This is not just for testing, but also for inline help. You can see it complains about missing /builddir/sage-9.5.beta5/... files, which is where the source files were.

I'll do a second attempt soon.

@tornaria
Copy link
Contributor Author

Second draft:

  • rename pkg from sage to sagemath (please remove sage before installing sagemath as I didn't set replaces=)
  • updated sage to sage-9.5.beta6
  • now it builds in place at /usr/lib/sage-$version
  • the resulting pkg includes a lot of useless stuff that comes with the distribution; we'll have to clean up. E.g. SAGE_ROOT/upstream is definitely unnecessary (1.2G); possibly SAGE_ROOT/pkgs or at least SAGE_ROOT/pkgs/sagemath-standard/build (2.6G).

A useful reference is https://github.com/sagemath/binary-pkg.

Now it's possible to run tests on the installed sage, for instance:

$ sage -tp 8 --all

to test all with 8 parallel jobs.

@dkwo
Copy link
Contributor

dkwo commented Nov 13, 2021

It's getting better.
Could you include your pyzmq patch, at least until we're using sage's python modules, so that it also builds on musl?

@dkwo
Copy link
Contributor

dkwo commented Nov 13, 2021

* now it builds in place at `/usr/lib/sage-$version`

@leahneukirchen What do you think?

@dkwo
Copy link
Contributor

dkwo commented Nov 13, 2021

* the resulting pkg includes a lot of useless stuff that comes with the distribution; we'll have to clean up. E.g. SAGE_ROOT/upstream is definitely unnecessary (1.2G); possibly SAGE_ROOT/pkgs or at least SAGE_ROOT/pkgs/sagemath-standard/build (2.6G).

Definitely :)
Size required on disk: 7300MB

@dkwo
Copy link
Contributor

dkwo commented Nov 13, 2021

Dumb question: does this in htop mean it is using system python?
python3 /usr/lib/sage-9.5.beta6/src/bin/sage-runtests -p 12 --all

@dkwo
Copy link
Contributor

dkwo commented Nov 13, 2021

Could you add gengetopt to the makedepends?

@dkwo
Copy link
Contributor

dkwo commented Nov 13, 2021

It seems giac is not picked up, could you add it also to depends?

 RuntimeError: unable to start giac because the command 'giac --sage' failed: The command was not found or was not executable: giac.

    In order to use the Giac interface you need to have Giac installed
    and have a program called "giac" in your PATH. You need a giac version
    supporting "giac --sage" ( roughly after 0.9.1 of march 2011). Some giac
    instructions  and the help's langage depend of you LANG variable. To obtain
    inline help for  giac commands, you also need to have the program "cas_help"
    in your PATH.

@tornaria tornaria force-pushed the sagemath branch 2 times, most recently from b0a333c to ad557eb Compare November 13, 2021 10:59
@tornaria
Copy link
Contributor Author

New on sagemath-9.5.beta6_2:

  • include patch fixing build of pyzmq on musl
  • add gengetopts and texinfo to makedepends
  • first attempt at cleanup: remove upstream sources and build artifacts

This one should save ~ 3.8G installed size.

Again, it should be possible to run tests on the installed sage, for instance:

$ sage -tp 8 --all

to test all with 8 parallel jobs. There are a few tests failing will look into it when I get a chance. It may also be useful to compare tests failing on ./xbps-src check sagemath vs test failing on installed sage.

----------------------------------------------------------------------
sage -t --random-seed=45812884732240074843516850899920199933 /usr/lib/sage-9.5.beta6/src/sage/env.py  # 1 doctest failed
sage -t --random-seed=45812884732240074843516850899920199933 /usr/lib/sage-9.5.beta6/src/sage/functions/min_max.py  # 2 doctests failed
sage -t --random-seed=45812884732240074843516850899920199933 /usr/lib/sage-9.5.beta6/src/sage/lfunctions/sympow.py  # 10 doctests failed
sage -t --random-seed=45812884732240074843516850899920199933 /usr/lib/sage-9.5.beta6/src/sage/lfunctions/pari.py  # 1 doctest failed
sage -t --random-seed=45812884732240074843516850899920199933 /usr/lib/sage-9.5.beta6/src/sage/modular/abvar/abvar.py  # 1 doctest failed
sage -t --random-seed=45812884732240074843516850899920199933 /usr/lib/sage-9.5.beta6/src/sage/modular/hecke/submodule.py  # 1 doctest failed
sage -t --random-seed=45812884732240074843516850899920199933 /usr/lib/sage-9.5.beta6/src/sage/repl/ipython_kernel/install.py  # 1 doctest failed
sage -t --random-seed=45812884732240074843516850899920199933 /usr/lib/sage-9.5.beta6/src/sage/schemes/elliptic_curves/ell_rational_field.py  # 9 doctests failed
sage -t --random-seed=45812884732240074843516850899920199933 /usr/lib/sage-9.5.beta6/src/sage_docbuild/__init__.py  # 1 doctest failed
sage -t --random-seed=45812884732240074843516850899920199933 /usr/lib/sage-9.5.beta6/src/sage_setup/find.py  # 3 doctests failed
sage -t --random-seed=45812884732240074843516850899920199933 /usr/lib/sage-9.5.beta6/src/sage/tests/cmdline.py  # 1 doctest failed
----------------------------------------------------------------------
Total time for all tests: 1186.4 seconds
    cpu time: 7017.0 seconds
    cumulative wall time: 8303.5 seconds

@tornaria
Copy link
Contributor Author

It seems giac is not picked up, could you add it also to depends?

Good point, we have to explicitly add anything that is needed at runtime that is not implicitly required by shlibs. Maybe we should try to run all tests in a barebones chroot with only sagemath installed.

@dkwo
Copy link
Contributor

dkwo commented Nov 13, 2021

Was the building taking 90+ min on the ci also before?

I'm not yet using a chroot, but my system is quite clean: it seems we also miss singular, pari from depends.
As a consequence, I was still getting a bit more doctest failures with previous version:

sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/tests/cmdline.py  # 5 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/schemes/elliptic_curves/ell_rational_field.py  # 14 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/rings/number_field/number_field.py  # 6 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/misc/sageinspect.py  # 35 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/misc/sagedoc.py  # 6 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/modular/abvar/abvar.py  # 1 doctest failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/rings/number_field/number_field_element.pyx  # 2 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/modular/hecke/submodule.py  # 1 doctest failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/rings/integer.pyx  # 1 doctest failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/repl/ipython_extension.py  # 6 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/graphs/connectivity.pyx  # 2 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/parallel/decorate.py  # 2 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/rings/number_field/number_field_rel.py  # 1 doctest failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/rings/number_field/galois_group.py  # 11 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/misc/persist.pyx  # 2 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/structure/element.pyx  # 37 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/misc/inline_fortran.py  # 3 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/functions/min_max.py  # 2 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage_docbuild/__init__.py  # 5 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/features/__init__.py  # 2 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/lfunctions/pari.py  # 1 doctest failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage_setup/find.py  # 3 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/misc/superseded.py  # 6 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/groups/galois_group.py  # 2 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/interfaces/gp.py  # 1 doctest failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/groups/perm_gps/permgroup_morphism.py  # 3 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/env.py  # 4 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/rings/polynomial/ore_polynomial_element.pyx  # 6 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/repl/ipython_kernel/install.py  # 1 doctest failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/tests/books/judson-abstract-algebra/galois-sage.py  # 5 doctests
 failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/rings/tate_algebra_ideal.pyx  # 6 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/lfunctions/sympow.py  # 10 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/interfaces/qsieve.py  # 7 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/misc/cython.py  # 21 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/structure/factory.pyx  # 8 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/graphs/graph_decompositions/fast_digraph.pyx  # 5 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/misc/session.pyx  # 2 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/misc/lazy_attribute.pyx  # 3 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/ext/memory_allocator.pxd  # 3 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/ext/memory_allocator.pyx  # 7 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/groups/pari_group.py  # 2 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/misc/nested_class.pyx  # 6 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/arith/long.pxd  # 14 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/repl/load.py  # 2 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/docs/instancedoc.pyx  # 4 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/cpython/cython_metaclass.pyx  # 4 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/rings/integer_fake.pxd  # 1 doctest failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/cpython/wrapperdescr.pxd  # 6 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/symbolic/pynac.pxi  # 1 doctest failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/misc/inherit_comparison.pyx  # 5 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/cpython/string.pyx  # 1 doctest failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/docs/conf.py  # 1 doctest failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/cpython/getattr.pyx  # 4 doctests failed
sage -t --random-seed=210801943898287344358194260905231821388 /usr/lib/sage-9.5.beta6/src/sage/misc/cachefunc.pyx  # 54 doctests failed

@dkwo
Copy link
Contributor

dkwo commented Nov 13, 2021

perhaps also add to depends: flintqs pari-datapkgs and zn_poly

@dkwo
Copy link
Contributor

dkwo commented Nov 13, 2021

Sorry, forget about singular and pari.

@leahneukirchen
Copy link
Member

Running the test suite also requires git.

@leahneukirchen
Copy link
Member

leahneukirchen commented Nov 13, 2021

@tornaria
Copy link
Contributor Author

Note: I'm not making the sphinx update in #34789 a priority now, since that's needs more review and testing. Instead I'll try to see if we can build sagemath without it (I think so).

The other four tickets mentioned above will be needed for sure; they should be an easy merge as they are all well tested new packages (so don't affect anyone else). Merging them will save us 30-40 minutes on the CI so it'd be really useful and they are probably the last packages needed before merging sagemath itself without any vendored in dependency.

@tornaria
Copy link
Contributor Author

Rebased to master and squashed.

  • sphinx update is NOT needed now
  • all dependenciees are already available on void repo
  • this version has no vendored packages, only the sagelib + binary scripts is shipped in the xbps, which is 230M.
  • all doctests pass for me locally (including musl 1.2)

I kept the full commit history at https://github.com/tornaria/void-packages/tree/sagemath-v1.

I've only tested the CLI. I'm sure the notebook/jupyter interface will need fixing.

At this point it's probably convenient to switch the installation method so that the sagelib to the main site-packages. Also the src directory is probably not needed other than for doctesting, and in principle everything is already copied into site-packages so (a) introspection (display source for a function from inside sage) should work (b) it should be possible to configure something so that it can be doctested from site-packages.

@dkwo
Copy link
Contributor

dkwo commented Jan 19, 2022

Excellent, I will build the new version soon.

At this point it's probably convenient to switch the installation method

Do you mean we could now switch to python3 setup.py build?

@tornaria tornaria changed the title [WIP] New package: sagemath-9.5.rc2 [WIP] New package: sagemath-9.5.rc3 Jan 19, 2022
@dkwo
Copy link
Contributor

dkwo commented Jan 20, 2022

Everything seems to work fine.
Indeed, it only installs as SPKG

sage_conf-9.5.rc3:                           standard, will be installed as an SPKG
sage_docbuild-9.5.rc3:                       standard, will be installed as an SPKG
sage_setup-9.5.rc3:                          standard, will be installed as an SPKG
sagelib-9.5.rc3:                             standard, will be installed as an SPKG
sagemath_doc_html-none:                      standard, will be installed as an SPKG

@tornaria
Copy link
Contributor Author

Note that sage -notebook is broken atm.

If you need it, it should be possible to run it as

$ PYTHONPATH=/usr/lib/sage-9.5.rc3/venv/lib/python3.10/site-packages/ sage -notebook

which instructs python to search for packages in the venv.

If you use the notebook and notice anything is missing, please report (I mostly only use the CLI). Some standard packages are just force-disabled (e.g. jmol, mathjax, threejs) and thist may or may not break the notebook or reduce its functionality.

If setting PYTHONPATH doesn't work properly and you want to try the shebang hack, apply the following commit and rebuild:

commit 4430467ee2e19479a09961e902e89881efad358a
Author: Gonzalo Tornaría <[email protected]>
Date:   Thu Nov 11 09:39:20 2021 -0300

    sagemath: don't rewrite python shebangs

diff --git a/common/hooks/pre-pkg/03-rewrite-python-shebang.sh b/common/hooks/pre-pkg/03-rewrite-python-shebang.sh
index 07162ad2c6..c0f2198e44 100644
--- a/common/hooks/pre-pkg/03-rewrite-python-shebang.sh
+++ b/common/hooks/pre-pkg/03-rewrite-python-shebang.sh
@@ -4,6 +4,10 @@
 hook() {
        local pyver= shebang= off=
 
+       if [ -n "$_no_python_shebang" ]; then
+               return 0
+       fi
+
        if [ -d ${PKGDESTDIR}/usr/lib/python* ]; then
                pyver="$(find ${PKGDESTDIR}/usr/lib/python* -prune -type d | grep -o '[[:digit:]]\.[[:digit:]]\+$')"
        fi
diff --git a/srcpkgs/sagemath/template b/srcpkgs/sagemath/template
index de9fe51777..f01fff3eea 100644
--- a/srcpkgs/sagemath/template
+++ b/srcpkgs/sagemath/template
@@ -72,6 +72,9 @@ nostrip=yes
 # don't shlib_provide anything
 noshlibprovides=yes
 
+# don't rewrite python shebang as sage uses its own python (for now)
+_no_python_shebang=yes
+
 # compile python code in sage venv
 pycompile_dirs="$_SAGE_ROOT/venv/"
 

I expect this will fix itself once we start installing sagemath in /usr as opposed to a private venv (the latter doesn't make sense anymore now that all python modules other than sagelib are from system).

@dkwo
Copy link
Contributor

dkwo commented Jan 20, 2022

Thank you. What seems to work for me:

ln -s /usr/lib/sage-9.5.rc3/local/var/lib/sage/venv-python3.10/share/jupyter /usr/local/share/
alias juplab='jupyter lab --app-dir /usr/lib/python3.10/site-packages/jupyterlab'
PYTHONPATH=/usr/lib/sage-9.5.rc3/venv/lib/python3.10/site-packages/ juplab

with no further changes to sage.
(The 2nd is some issue with jupyter lab, the 3rd with sage-venv as you say.)

Note that the ./configure --disable-doc option can be used to suppress the installation of packages needed for docbuilding (see trac ticket 32759).

@tornaria tornaria changed the title [WIP] New package: sagemath-9.5.rc3 [WIP] New package: sagemath-9.5.rc4 Jan 25, 2022
@leahneukirchen
Copy link
Member

Are you sure you wanna remove documentation? Will it break ?command?

@tornaria
Copy link
Contributor Author

Are you sure you wanna remove documentation? Will it break ?command?

That's introspection and it still works. It should be using either the function docstring or the actual source as-installed in site-packages, so in fact we don't even need src/sage. There are two reasons I'm still shipping src: (a) the scripts in src/bin are needed because our installation is imperfect (b) for --t --all it needs the source in src/sage, however there must be a way to make test to work on the installed files.

@tornaria tornaria force-pushed the sagemath branch 2 times, most recently from d45a20c to d63f5f0 Compare January 30, 2022 22:13
@tornaria tornaria changed the title [WIP] New package: sagemath-9.5.rc4 [WIP] New package: sagemath-9.5 Jan 30, 2022
@tornaria
Copy link
Contributor Author

sage 9.5 is released, so I've updated this branch.

However, now that all dependencies are already available in void, I've been working on a new template which will install sagelib as a python module, in the system site-packages.

I'm not sure about all the binary scripts, I lean towards installing all of them in /usr/lib/sagemath/bin or /usr/lib/sagemath/9.5/bin and have just a symlink in /usr/bin. The point is that it should be easy to install a local or user version of sagemath from source to coexist with the system sagemath.

@dkwo
Copy link
Contributor

dkwo commented Jan 31, 2022

Great. What are the obstacles to cross compile it, besides ntl?

@dkwo
Copy link
Contributor

dkwo commented Jan 31, 2022

Just a note to self: use

ln -s /usr/lib/sage-9.5/local/share/jupyter /usr/local/share/
PYTHONPATH=/usr/lib/sage-9.5/local/lib/python3.10/site-packages/

@tornaria
Copy link
Contributor Author

tornaria commented Feb 2, 2022

I updated this branch to the new "sage-as-a-python-module" version from #35339 which seems to work very well.

The previous approach with its full history is at https://github.com/tornaria/void-packages/tree/sagemath-v1, rebased to current master. I won't be updating or rebasing it further.

Assuming CI passes (should take about 2h to finish), this PR is done and ready to merge. Any problems we find later we can follow up in individual issues, please @ me so I'm notified.

We have:

  • 11 patches which can be upstreamed, I'll work on that.
  • 2 patches are taken from positive_review tickets in sage trac, should be merged soon.
  • only 2 are void specific, no patches that are musl specific.

@tornaria tornaria changed the title [WIP] New package: sagemath-9.5 New package: sagemath-9.5 Feb 2, 2022
@tornaria
Copy link
Contributor Author

tornaria commented Feb 3, 2022

Note that #35367 should be merged before this so that the binary package of sagemath runs ok on older cpus (e.g. nehalem).

@tornaria
Copy link
Contributor Author

tornaria commented Feb 3, 2022

BTW, doctesting with --long on my nehalem box finished: after #35367 all tests passed using the packages for nauty and sagemath built on the cascade lake. No more illegal instructions found.

@leahneukirchen leahneukirchen merged commit 9289a1a into void-linux:master Feb 3, 2022
@tornaria tornaria deleted the sagemath branch February 3, 2022 13:45
@Vindaar
Copy link

Vindaar commented Feb 3, 2022

Huge thank you @tornaria! This is amazing and really appreciated! All working fine on my end.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
new-package This PR adds a new package
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants