spkg-configure.m4 for pari

[dimpase]

libpari-gmp is a dependency of many packages, so it's important to be able to use the one from the system.

We check that system's PARI/GP environment is close enough to what we would install in Sage (i.e. versions of GP and libpari, and that they share the datadir). Also, PARI packages need to be checked.

CC: @embray @kiwifb @isuruf @timokau @antonio-rojas

Component: build: configure

Author: Dima Pasechnik

Branch: dd2b3fa

Reviewer: Isuru Fernando

Issue created by migration from https://trac.sagemath.org/ticket/28242

[dimpase]

comment:1

we should use sample GP commands to test for packages, as in https://pari.math.u-bordeaux.fr/packages.html:

  ellinit("11a1")     /* package: elldata */
  galoisgetpol(8)     /* package: galpol */
  galoisgetname(12,1) /* package: galpol 4.0 and up */
  ellmodulareqn(11)   /* package: seadata */
  polgalois(x^8-2)    /* package: galdata */

if only the small version of seadata is installed, ellmodulareqn(211) produces an error (it does work with the "normal" seadata version).

need to figure out how to test for nftables.

[dimpase]

comment:2

The following may be used to test whether libpari knows about galdata
(Basically, obtained by editing the output of gp2c run on polygalois(x^8-2)[1]).

#include <pari/pari.h>  /* cc t.c -lpari && ./a.out */
GEN t_t(long prec)
{
  GEN x = pol_x(fetch_user_var("x"));
  return gel(polgalois(gsubgs(gpowgs(x, 8), 2), prec), 1);
}

int main() 
{
GEN r;
pari_init(5000000, 0);
r=t_t(100);
pari_printf("group order=%P4d\n", r); /* should be 16 */
pari_close();
return 0;
}

[dimpase]

Branch: u/dimpase/packages/pariconfig

[dimpase]

New commits:

83d26e5

spkg-configure for pari

[dimpase]

Commit: 83d26e5

[dimpase]

Author: Dima Pasechnik

[dimpase]

comment:4

This is still WIP, more to be done here.

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

4088bfc	spkg-configure for pari
712fb95	spkg-configure for pari_elldata

[sagetrac-git]

Changed commit from 83d26e5 to 712fb95

[sagetrac-git]

Changed commit from 712fb95 to 2e11aed

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

2f45c9e	spkg-configure for pari
b622942	spkg-configure for pari_elldata
2e11aed	spkg-configure's for PARI packages

[dimpase]

comment:7

still needed to sort out dependencies properly, and provide spkg-configure.m4 for gp2c.

For installing gp2c, one must provide pari.cfg location, something that it's not really discoverable. (save for find / -name pari.cfg :-))

[dimpase]

comment:8

It seems that all the PARI package "deps" should be checked within spkg-configure of PARI itself (as PARI does not provide means to install extra packages, we need to disable system PARI lacking these packages). -- the branch does not yet implement this.

And not only standard packages pari_galdata and
pari_seadata_small, but also the optional packages pari_elldata,
pari_galpol, and pari_seadata.

The optional package pari_nftables is clearly an afterthought (not used anywhere in sagelib, too), and I don't want to rule out an otherwise OK installation of PARI/GP, is it is missing (e.g. Debian doesn't even ship it, although Fedora does ship it; not sure about Conda and Gentoo).

pari_jupyter should be installable regardless.

gp2c only needs the correct location of pari.cfg, and while it's a pain to find it, it is doable in some ad hoc way. I wonder what Conda does does with gp2c, and where it places pari.cfg.

[kiwifb]

comment:9

Distros have a bone to pick with the data packages from pari. That may be why some distro decided not to bother. There are ways to install the stuff manually if you want too.

Upstream is committing the cardinal sin of not versioning their data packages. The tarball is always called exactly the same - even when the content is changed. You understand that's a bit of a problem for automated build scripts that rely on a stored checksum to verify that the package is authentic.

For info, I had the stuff packaged in Gentoo and I recently updated it.

[dimpase]

comment:10

I'm trying to convince the upstream to make configuration info for libpari discoverable.
E.g. there is no way to get the location of pari.cfg from gp. I'm thinking of getting its location from the full path to libpari.so, I see no other good way...

[kiwifb]

comment:11

pari.cfg is installed in something called sysdatadir at configure time. I don't know if it is possible to get that variable at runtime (without knowing pari.cfg already that is).

[sagetrac-git]

Changed commit from 2e11aed to 183f47c

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

183f47c

check for PARI packages in PARI's spkg-configure

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

72f3c20

spkg-configure for gp2c

[sagetrac-git]

Changed commit from 183f47c to 72f3c20

[isuruf]

comment:14

Replying to @dimpase:

And not only standard packages pari_galdata and
pari_seadata_small, but also the optional packages pari_elldata,
pari_galpol, and pari_seadata.

Is there a way to avoid checking for the optional packages? Maybe build PARI from SPKG if the optional package is not found, but requested.

The optional package pari_nftables is clearly an afterthought (not used anywhere in sagelib, too), and I don't want to rule out an otherwise OK installation of PARI/GP, is it is missing (e.g. Debian doesn't even ship it, although Fedora does ship it; not sure about Conda and Gentoo).

I just packaged all the optional PARI database packages on conda including nftables.

pari_jupyter should be installable regardless.

gp2c only needs the correct location of pari.cfg, and while it's a pain to find it, it is doable in some ad hoc way. I wonder what Conda does does with gp2c, and where it places pari.cfg.

Conda puts it into <prefix>/lib/pari/pari.cfg. We don't package gp2c yet.

[kiwifb]

comment:15

Someone put pari.cfg in a unhelpful location in Gentoo. In fact it broke my ebuild of gp2c. I don't know for how long but no one ever complained so I don't think there are many users for that program. Fixed in sage-on-gentoo's pari (2.11.1 and 2.11.2). Current sage-on-gentoo location is ${EPREFIX}/usr/share/pari/pari.cfg.

[dimpase]

comment:16

by the way, it seems that cypari needs either a patch or an amendend
spkg-install script. With the branch on this ticket, its installation fails with

Installing cypari-2.1.1
Installing package cypari2 using pip2
Ignoring indexes: https://pypi.org/simple
Created temporary directory: /tmp/pip-ephem-wheel-cache-Nyv_mU
Created temporary directory: /tmp/pip-req-tracker-_Zruyw
Created requirements tracker '/tmp/pip-req-tracker-_Zruyw'
Created temporary directory: /tmp/pip-install-rsVw0C
Processing /home/dimpase/sage/local/var/tmp/sage/build/cypari-2.1.1/src
  Created temporary directory: /tmp/pip-req-build-tpOOw3
  Added file:///home/dimpase/sage/local/var/tmp/sage/build/cypari-2.1.1/src to build tracker '/tmp/pip-req-tracker-_Zruyw'
  Running setup.py (path:/tmp/pip-req-build-tpOOw3/setup.py) egg_info for package from file:///home/dimpase/sage/local/var/tmp/sage/build/cypari-2
.1.1/src
    Running command python setup.py egg_info
    running egg_info
    creating pip-egg-info/cypari2.egg-info
    writing requirements to pip-egg-info/cypari2.egg-info/requires.txt
    writing pip-egg-info/cypari2.egg-info/PKG-INFO
    writing top-level names to pip-egg-info/cypari2.egg-info/top_level.txt
    writing dependency_links to pip-egg-info/cypari2.egg-info/dependency_links.txt
    writing manifest file 'pip-egg-info/cypari2.egg-info/SOURCES.txt'
    Generating PARI functions: (!_) (#_) (%) (%#) (+_) (-_) Catalan/home/dimpase/sage/local/share/pari/doc not found at /usr/bin/gphelp line 147.
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-tpOOw3/setup.py", line 83, in <module>
        cmdclass=dict(build_ext=build_ext, bdist_egg=no_egg)
      File "/home/dimpase/sage/local/lib/python2.7/site-packages/setuptools/__init__.py", line 143, in setup
        return distutils.core.setup(**attrs)
      File "/home/dimpase/sage/local/lib/python2.7/distutils/core.py", line 151, in setup
        dist.run_commands()
      File "/home/dimpase/sage/local/lib/python2.7/distutils/dist.py", line 953, in run_commands
        self.run_command(cmd)
      File "/home/dimpase/sage/local/lib/python2.7/distutils/dist.py", line 972, in run_command
        cmd_obj.run()
      File "/home/dimpase/sage/local/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 296, in run
        self.find_sources()
      File "/home/dimpase/sage/local/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 303, in find_sources
        mm.run()
      File "/home/dimpase/sage/local/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 534, in run
        self.add_defaults()
      File "/home/dimpase/sage/local/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 570, in add_defaults
        sdist.add_defaults(self)
      File "/home/dimpase/sage/local/lib/python2.7/site-packages/setuptools/command/py36compat.py", line 36, in add_defaults
        self._add_defaults_ext()
      File "/home/dimpase/sage/local/lib/python2.7/site-packages/setuptools/command/py36compat.py", line 119, in _add_defaults_ext
        build_ext = self.get_finalized_command('build_ext')
      File "/home/dimpase/sage/local/lib/python2.7/distutils/cmd.py", line 312, in get_finalized_command
        cmd_obj.ensure_finalized()
      File "/home/dimpase/sage/local/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
        self.finalize_options()
      File "/tmp/pip-req-build-tpOOw3/setup.py", line 33, in finalize_options
        rebuild()
      File "autogen/__init__.py", line 26, in rebuild
        G()
      File "autogen/generator.py", line 341, in __call__
        self.handle_pari_function(**v)
      File "autogen/generator.py", line 234, in handle_pari_function
        doc = get_rest_doc(function)
      File "autogen/doc.py", line 345, in get_rest_doc
        raw = get_raw_doc(function)
      File "autogen/doc.py", line 277, in get_raw_doc
        doc = subprocess.check_output(["gphelp", "-raw", function])
      File "/home/dimpase/sage/local/lib/python2.7/subprocess.py", line 223, in check_output
        raise CalledProcessError(retcode, cmd, output=output)
    subprocess.CalledProcessError: Command '[u'gphelp', u'-raw', u'Catalan']' returned non-zero exit status 2
Cleaning up...
  Removing source in /tmp/pip-req-build-tpOOw3
Removed file:///home/dimpase/sage/local/var/tmp/sage/build/cypari-2.1.1/src from build tracker '/tmp/pip-req-tracker-_Zruyw'
Removed build tracker '/tmp/pip-req-tracker-_Zruyw'
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-req-build-tpOOw3/

which appears to indicate that it does not try to figure out where PARI/GP docs are, but just assumes that they are in SAGE_LOCAL/share/pari/doc, instead of using
`echo "default(datadir)" | gp -qf`/doc

[dimpase]

comment:17

Oops, this is simply due to

# For PARI/GP
GP_DATA_DIR="$SAGE_LOCAL/share/pari" && export GP_DATA_DIR
GPHELP="$SAGE_LOCAL/bin/gphelp" && export GPHELP
GPDOCDIR="$SAGE_LOCAL/share/pari/doc" && export GPDOCDIR

in src/bin/sage-env.

OK, I can move this stuff into src/bin/sage-env-config(.in) and fill them in correctly.

[isuruf]

comment:39

Thanks. Works for me

[kiwifb]

comment:40

Sorry for the late intervention.

diff --git a/src/sage_setup/docbuild/__init__.py b/src/sage_setup/docbuild/__init__.py
index e406bca..0e07c71 100644
--- a/src/sage_setup/docbuild/__init__.py
+++ b/src/sage_setup/docbuild/__init__.py
@@ -294,7 +294,7 @@ def _build_many(target, args):
             raise
     return ret
 
-if not (CYGWIN_VERSION and CYGWIN_VERSION[0] < 3):
+if (os.environ['SAGE_PARI_CFG'] !='') and (not (CYGWIN_VERSION and CYGWIN_VERSION[0] < 3)):
     build_many = _build_many
 else:
     # Cygwin 64-bit < 3.0.0 has a bug with exception handling when exceptions

That bit causes me trouble when building the doc

 * python3_7: running distutils-r1_run_phase python_compile_all
Setting permissions of DOT_SAGE directory so only you can read and write it.
Traceback (most recent call last):
  File "sage_setup/docbuild/__main__.py", line 1, in <module>
    from sage_setup.docbuild import main
  File "/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_7/sage_setup/docbuild/__init__.py", line 297, in <module>
    if (os.environ['SAGE_PARI_CFG'] !='') and (not (CYGWIN_VERSION and CYGWIN_VERSION[0] < 3)):
  File "/usr/lib/python3.7/os.py", line 678, in __getitem__
    raise KeyError(key) from None
KeyError: 'SAGE_PARI_CFG'

[kiwifb]

comment:41

I would be happy for this ticket to proceed as is and the issue to be addressed in a follow up ticket.

But the general assumption should be sage-on-distros don't run configure. In fact we have pushed a few things so that sage-distro can run without sage-env. I did so a few release ago and Antonio Rojas from arch has pushed so he could follow my lead there in sage-8.9. We can define the variable at build time, I could even flip the value depending on whether pari has been built with tls in Gentoo. But it would be cleaner if it just worked, with the assumption that pari is provided externally, if the variable is undefined.

[kiwifb]

comment:42

Follow up will take place at #28341.

[vbraun]

Changed branch from u/dimpase/packages/pariconfig to dd2b3fa

[dimpase]

Changed commit from dd2b3fa to none

[dimpase]

comment:44

Can Archlinux get more PARI packages (as much as Debian 10), so that its PARI may be used?

[antonio-rojas]

comment:45

Replying to @dimpase:

Can Archlinux get more PARI packages (as much as Debian 10), so that its PARI may be used?

Standard packages are available from the official repositories, optional packages can easily be installed from AUR - https://aur.archlinux.org/packages/?K=pari-

[mkoeppe]

comment:46

Follow up at #29319

[mkoeppe]

comment:47

Replying to @dimpase:

by the way, it seems that cypari needs either a patch or an amendend
spkg-install script. With the branch on this ticket, its installation fails with

Installing cypari-2.1.1
...
      File "autogen/doc.py", line 277, in get_raw_doc
        doc = subprocess.check_output(["gphelp", "-raw", function])
      File "/home/dimpase/sage/local/lib/python2.7/subprocess.py", line 223, in check_output
        raise CalledProcessError(retcode, cmd, output=output)
    subprocess.CalledProcessError: Command '[u'gphelp', u'-raw', u'Catalan']' returned non-zero exit status 2 ...

which appears to indicate that it does not try to figure out where PARI/GP docs are, but just assumes that they are in SAGE_LOCAL/share/pari/doc, instead of using
`echo "default(datadir)" | gp -qf`/doc

Follow up for this problem: #29342

[thierry-FreeBSD]

comment:48

I was working on FreeBSD with 9.1.beta9 and pari as a system package without any problem related to gp / pari.

Yesterday I switched to 9.1.rc1, with the same external package, and now the compilation of cysignals aborts with this message:

[cysignals-1.10.2]     cc -pthread -shared -L/usr/local/lib -fstack-protector-strong -L/usr/ports/math/sage/work/stage/usr/local/lib -Wl,-rpath,/usr/ports/math/sage/work/stage/usr/local/lib -L/usr/local/llvm90/lib -L/usr/local/lib -Wl,-rpath=/usr/local/lib/gcc9 -L/usr/local/lib/gcc9 -B/usr/local/bin -L/usr/local/lib -fstack-protector-strong -O2 -pipe -DLIBICONV_PLUG -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing -DLIBICONV_PLUG -I/usr/local/include -isystem /usr/local/include -Wp,-U_FORTIFY_SOURCE build/temp.freebsd-12.1-STABLE-amd64-3.7/build/src/cysignals/signals.o -L/usr/local/lib -lpython3.7m -o build/lib.freebsd-12.1-STABLE-amd64-3.7/cysignals/signals.so -lpari -lomp -pthread -L/usr/local/lib
[cysignals-1.10.2]     /usr/local/bin/ld: PARI_SIGINT_block: TLS definition in /usr/local/lib/libpari.so section .tbss mismatches non-TLS reference in build/temp.freebsd-12.1-STABLE-amd64-3.7/build/src/cysignals/signals.o
[cysignals-1.10.2]     /usr/local/bin/ld: /usr/local/lib/libpari.so: error adding symbols: bad value

Anyone with the same problem on other OS, or is it specific to FreeBSD?

[dimpase]

comment:49

Hmm, not sure what's happening here. I don't think I saw this before. Something related to --enable-tls (tls=thread-local storage) option while buildig Pari/GP - which works on Linux smoothly.

[thierry-FreeBSD]

comment:50

But it seems that this is disabled when Sage builds Pari?

In build/pkgs/pari/spkg-install.in there is "unset enable_tls".

[dimpase]

comment:51

I mean to say that we can use Pari built with --enable-tls with Sage on several Linux distributions (not the Pari built by Sage). Does FreeBSD Pari port use some special clang flags for TLS?

Is Pari built with pthread on FreeBSD?

[dimpase]

comment:52

On the other hand, cysignals is not known to be thread-safe, so it could be that FreeBSD tools are better than Linux tools in detecting this:
sagemath/cysignals#21

In view of this, linking cysignals with -pthreads might be what causes this error.

[thierry-FreeBSD]

comment:53

Actually the problem was different! Pari's spkg-configure.m4 is broken: I thought that I was using pari from the system packages, but I was not.

spkg-configure.m4 includes unescaped [ ] and it produces an error.

A fix could be:

--- build/pkgs/pari/spkg-configure.m4.orig      2020-04-27 19:20:17.006373000 +0200
+++ build/pkgs/pari/spkg-configure.m4   2020-04-27 19:34:16.220830000 +0200
@@ -74,7 +74,7 @@
             sage_spkg_install_pari=yes
         fi
         AC_MSG_CHECKING([whether hyperellcharpoly bug is fixed])
-        bug_check=$(echo "hyperellcharpoly([x^10 + x^9 + x^8 + x,0]*Mod(1,3))" | $GP -qf 2>> config.log)
+        bug_check=$(echo "hyperellcharpoly([[x^10 + x^9 + x^8 + x,0]]*Mod(1,3))" | $GP -qf 2>> config.log)
         expected="%1 = x^8 + 731*x^7 + 6*x^6 - 720*x^5 + 18*x^4 - 2160*x^3 + 54*x^2 + 19737*x + 81"
         if test x"$bug_check" = x"$expected"; then
            AC_MSG_RESULT([yes])

(But ATM I removed this part)

[mkoeppe]

comment:54

That's fixed already by #29554

[dimpase]

comment:55

Replying to @mkoeppe:

That's fixed already by #29554

however, this still begs a question why cysignals is built with -pythread (as set in its setup.py), while it's known not to be thread-safe, cf.
sagemath/cysignals#21