Update Singular to 3-1-3-3

[malb]

Singular spkg upgrade

Singular 3-1-2 fixes a critical bug, cf. #10902. There are more bugfixes in the newest stable release. We should update as soon as possible.

Sage libsingular upgrade

There are various issues where currRing is not correctly set. Some of these are triggered by refcounting the rings and freeing them when they are no longer needed, see the dependency ticket #11339.

Installation instructions

• Install: http://boxen.math.washington.edu/home/jdemeyer/spkg/singular-3-1-3-3.p1.spkg

• Apply the two dependency patches from Refcounting for Singular rings #11339

• Apply: attachment: 10903_flat.patch

Depends on #11339

CC: @zimmermann6 @simon-king-jena

Component: packages: standard

Keywords: singular SageDays34 sd34

Author: Burcin Erocal, Martin Albrecht, Volker Braun, Simon King

Reviewer: Martin Albrecht, Volker Braun, Simon King

Merged: sage-4.7.2.alpha4

Issue created by migration from https://trac.sagemath.org/ticket/10903

[kiwifb]

comment:2

Last time I tried libsingular from 3.1.2 didn't play well with sage (sage couldn't build). It is documented somewhere on sage-devel.

[kiwifb]

comment:3

http://groups.google.com/group/sage-devel/browse_thread/thread/9544eb87400b5e43/e2424823682d7d53?lnk=gst&q=singular+francois#e2424823682d7d53http://groups.google.com/group/sage-devel/browse_thread/thread/9544eb87400b5e43/e2424823682d7d53?lnk=gst&q=singular+francois#e2424823682d7d53

[kiwifb]

comment:4

sorry for the double pasting the mouse is vintage.

[SnarkBoojum]

comment:5

Updating singular to at least 3-1-3 (if it's out...) will also help the ARM port, as it's one of the few packages which still give issues with 4.6.2.

[kedlaya]

comment:6

Indeed 3-1-3 is out now. Already 3-1-2 adds a lot of functionality for polynomials over the integers which would be very helpful to have in Sage!

[kiwifb]

comment:7

Trying to compile libsingular with this version and trying to link it to sage is on my TODO list. The singular 3-1-3 executable as called by the pexpect interface breaks a few doctest because normalization has been changed.

[kcrisman]

comment:8

Just FYI, I'm cutting a p10 shortly with a very small fix for Cygwin in it (see #11550), so if someone if making this package, it would be great to base off that.

[nexttime]

comment:9

Replying to @SnarkBoojum:

Updating singular to at least 3-1-3 (if it's out...) will also help the ARM port, as it's one of the few packages which still give issues with 4.6.2.

Cf. #10810.

---

If someone touches the Singular spkg, please also add a patch for #11645 in case it should then still be necessary. (I've submitted one upstream, see http://www.singular.uni-kl.de:8002/trac/ticket/352#comment:15).

[nexttime]

Changed keywords from singular to singular SageDays34

[nexttime]

comment:10

Replying to @kcrisman:

Just FYI, I'm cutting a p10 shortly with a very small fix for Cygwin in it (see #11550), so if someone if making this package, it would be great to base off that.

FWIW, the Singular spkg from #11550 will probably become a p12, since #11663 meanwhile provides a p11 (which Karl-Dieter's will have to get rebased on again).

I'll then perhaps provide a p13 based on that fixing #11645. :)

Anyway, I guess upgrading Sage's Singular will be attempted during the Sage/Singular Days in Kaiserslautern at the end of September, so we have some time until then...

[nexttime]

Changed keywords from singular SageDays34 to singular SageDays34 sd34

[malb]

comment:12

I haven't run any tests yet and the changes in the SPKG are not committed (because our patches will probably be accepted upstream before 3-1-4 comes out).

[malb]

Author: Burcin Erocal, Martin Albrecht

[nexttime]

comment:14

Replying to @malb:

I haven't run any tests yet and the changes in the SPKG are not committed (because our patches will probably be accepted upstream before 3-1-4 comes out).

The spkg name should contain the svn version you are using.

Regarding the patch to module_list.py:

Please don't randomly add all libraries any extension module using Singular might also use to singular_libs.

[malb]

comment:15

doctests hang here:

Trying:
    R = PolynomialRing(ZZ, Integer(5), order='lex', names=('x', 'y', 'a', 'b', 'u',)); (x, y, a, b, u,) = R._first_ngens(5)###line 4461:_sage_    >>> R.<x,y,a,b,u>=PolynomialRing(ZZ, 5, order='lex')

they also crash in other places. Thus, more fun ahead.

[nexttime]

comment:16

Replying to @malb:

doctests hang here:

Trying:
    R = PolynomialRing(ZZ, Integer(5), order='lex', names=('x', 'y', 'a', 'b', 'u',)); (x, y, a, b, u,) = R._first_ngens(5)###line 4461:_sage_    >>> R.<x,y,a,b,u>=PolynomialRing(ZZ, 5, order='lex')

they also crash in other places. Thus, more fun ahead.

How about starting with 3-1-3 and some important upstream patches?

[malb]

comment:17

I'm in Kaiserslautern right now, so I have direct access to the Singular developers. Hence, actually fixing the issues seems like the order of the day.

[malb]

Dependencies: #11339

[malb]

comment:19

With

$ hg qap
trac_11339_refcount_singular_rings.patch
trac_11339_refcount_singular_polynomials.patch
10903_singular-3-1-4.patch

I currently get these doctest failures.

sage -t  "devel/sage-main/sage/rings/polynomial/polynomial_element.pyx"
sage -t  "devel/sage-main/sage/rings/polynomial/multi_polynomial_ideal.py"
sage -t  "devel/sage-main/sage/rings/polynomial/toy_d_basis.py"
sage -t  "devel/sage-main/sage/rings/polynomial/multi_polynomial_sequence.py"
sage -t  "devel/sage-main/sage/rings/polynomial/multi_polynomial_element.py"
sage -t  "devel/sage-main/sage/rings/polynomial/multi_polynomial_libsingular.pyx"

[malb]

comment:20

With the updated SPKG & patch I get:

• sage -t "devel/sage-main/sage/rings/polynomial/polynomial_element.pyx"

Numerical noise, nothing serious

• sage -t "devel/sage-main/sage/rings/polynomial/multi_polynomial_ideal.py"

crash

• sage -t "devel/sage-main/sage/rings/polynomial/multi_polynomial_sequence.py"

crash

• sage -t "devel/sage-main/sage/rings/polynomial/multi_polynomial_element.py"

      sage: f=(a*x-1)*((a+1)*y-1); f
  Expected:
      -x*y + (-a)*x + (-a - 1)*y + 1
  Got:
      (a^2 + a)*x*y + (-a)*x + (-a - 1)*y + 1
  

• sage -t "devel/sage-main/sage/rings/polynomial/multi_polynomial_libsingular.pyx"

Factorisation over number fields goes horribly wrong

[malb]

comment:22

Hans created a new branch of Singular, where the algebraic extensions are reverted to the old implementation. Using this branch, available in SPKG form at:

http://sage.math.washington.edu/home/malb/spkgs/singular-3-1-3.svn-algnumbers.spkg

the crashes seem to go away (at least for everything in sage/rings).

[malb]

comment:23

Output of make ptestlong with this patch and #11339 applied:

sage -t  -long -force_lib devel/sage/sage/misc/sageinspect.py # 1 doctests failed
sage -t  -long -force_lib devel/sage/sage/tests/cmdline.py # 1 doctests failed
sage -t  -long -force_lib devel/sage/sage/interfaces/singular.py # 1 doctests failed
sage -t  -long -force_lib devel/sage/sage/crypto/mq/sr.py # 0 doctests failed
sage -t  -long -force_lib devel/sage/sage/schemes/elliptic_curves/ell_generic.py # 0 doctests failed
sage -t  -long -force_lib devel/sage/sage/schemes/elliptic_curves/ell_curve_isogeny.py # 0 doctests failed
sage -t  -long -force_lib devel/sage/sage/schemes/plane_conics/con_field.py # 1 doctests failed
sage -t  -long -force_lib devel/sage/sage/schemes/plane_conics/con_finite_field.py # 0 doctests failed
sage -t  -long -force_lib devel/sage/sage/schemes/generic/algebraic_scheme.py # 0 doctests failed

i.e., crashes.

[malb]

comment:24

I just updated the spkg such that SAGE_DEBUG="yes" will build Singular with debugging enabled (internal consistency checks, asserts etc.)

[alexanderdreyer]

comment:112

Replying to @nexttime:

Well, smells more as if --without-dynamic-kernel does no longer work... (which we pass on any OS but Linux)

But --with-dl is active. Even thought we do not load parts of the kernel dynamically, we can load other dynamic modules (for instance, there is an experimental python interface for Singular).

[jdemeyer]

comment:113

Replying to @nexttime:

Replying to @jdemeyer:

New spkg needs review: http://boxen.math.washington.edu/home/jdemeyer/spkg/singular-3-1-3-3.p1.spkg. See attachment: singular-3-1-3-3.p0-p1.diff for changes.

 * Exit when `patch` fails.  We do not print an error message, patch is 
   verbose enough by itself.

Not true. patch's error messages do not start with Error nor Failed, such that it's hard to find out which spkg actually failed to build / install in parallel builds.

What do you mean with this? How is the message

Error: Patch failed to apply.

more clear than something like

1 out of 7 hunks FAILED -- saving rejects to file sage/rings/number_field/number_field_element.pyx.rej

In any case, I don't see what this has to do with parallel builds.

Can anybody tell me how we should proceed?

[vbraun]

comment:114

Leif wants to be able to grep the install.log for "Error", I think. Thats not particularly high on my agenda, though. Especially since patch will either already fail at the developer's machine or always work (assuming that the hardware is working).

[alexanderdreyer]

comment:115

Replying to @vbraun:

Leif wants to be able to grep the install.log for "Error", I think. Thats not particularly high on my agenda, though. Especially since patch will either already fail at the developer's machine or always work (assuming that the hardware is working).

BTW: So it's not mandatory anymore to "unroll" patches in the spkg, i.e. to deliver the patched file and use cp?
Despite of this, you're right from my point of view.

[alexanderdreyer]

comment:116

Replying to @jdemeyer:

New spkg needs review: http://boxen.math.washington.edu/home/jdemeyer/spkg/singular-3-1-3-3.p1.spkg. See attachment: singular-3-1-3-3.p0-p1.diff for changes.

The new spkg builds and installs on SuSE 11 amd64 and OS X 10.5 ppc. The tests are running (fine so far). Somebody should test it on Solaris though.

[vbraun]

comment:117

Replying to @alexanderdreyer:

BTW: So it's not mandatory anymore to "unroll" patches in the spkg, i.e. to deliver the patched file and use cp?

Not any more, we have included patch as a standard spkg!

[alexanderdreyer]

comment:118

Replying to @vbraun:

Not any more, we have included patch as a standard spkg!

Nice!

[nexttime]

comment:119

Replying to @jdemeyer:

Replying to @nexttime:

Replying to @jdemeyer:

New spkg needs review: http://boxen.math.washington.edu/home/jdemeyer/spkg/singular-3-1-3-3.p1.spkg. See attachment: singular-3-1-3-3.p0-p1.diff for changes.

 * Exit when `patch` fails.  We do not print an error message, patch is 
   verbose enough by itself.

Not true. patch's error messages do not start with Error nor Failed, such that it's hard to find out which spkg actually failed to build / install in parallel builds.

What do you mean with this? How is the message

Error: Patch failed to apply.

more clear than something like

1 out of 7 hunks FAILED -- saving rejects to file sage/rings/number_field/number_field_element.pyx.rej

It isn't more clear (the "Error: ..." would be printed in addition anyway, although an "ordinary" user would perhaps not know what a hunk is or the message is all about), but you can easily grep for it, so every error message should start with "Failed" or (preferably) "Error".

(Volker, btw., the new ATLAS spkg is an IMHO poor counterexample to that, as it raises unhandled exceptions such that one gets tracebacks instead of meaningful error messages.)

The Sage library, which is built in parallel by default, is also a bad exception to this; it's often hard to see what really failed, not to mention the odd warnings about -Wstrict-prototypes not being valid for C++.

In any case, I don't see what this has to do with parallel builds.

Well, the screen output (and hence install.log) is pretty messed up in parallel builds, and make usually doesn't stop immediately if one spkg fails to build, but continues and waits for other jobs to finish. [Another issue is that stdout and stderr frequently get "out of sync".]

To see what went wrong or which spkgs failed to build / didn't pass the (self-)test suite, one can usually do

$ egrep '^(Error|Failed) ' spkg/logs/*

with variations on -i, -h, -n and -A/B/C... etc.

It wouldn't be bad to add something similar to the top-level Makefile; since we always append to the logs, we could create a fresh temporary log file upon every build and let sage-spkg write to it whenever a build or running spkg-check failed, such that an informative summary can be printed at the end, at least on errors. (We could also log the so far successfully built spkgs somewhere, such that one can tail -f that file, perhaps even with timings or date/time, "i/N spkgs built".)

Can anybody tell me how we should proceed?

Well, it's a minor issue of course. But we should IMHO avoid such in the future.

I still cannot really tell whether this spkg (along with the patches) causes more doctests to fail on Linux PPC64 (the Skynet machine Silius; SLES 11.1, POWER7), since I've experimented with various things on that machine, and I haven't yet had the time to reliably compare builds which really only differ w.r.t. this ticket (and #11339).

In case this ticket should cause more problems on that platform, we could (hopefully) fix these on a follow-up, since there are other spkgs which apparently are still broken on it (also depending on the compiler version and options), so building Sage there is currently pretty experimental anyway.

[jdemeyer]

comment:120

Replying to @nexttime:

To see what went wrong or which spkgs failed to build / didn't pass the (self-)test suite, one can usually do

$ egrep '^(Error|Failed) ' spkg/logs/*

with variations on -i, -h, -n and -A/B/C... etc.

But every failed spkg also prints the message

sage: An error occurred while installing mpir-2.1.3.p5
Please email sage-devel http://groups.google.com/group/sage-devel
explaining the problem and send the relevant part of
of /home/buildbot/build/sage/iras-1/iras_full/build/sage-4.7.2.alpha4/install.log.  Describe your computer, operating system, etc.
If you want to try to fix the problem yourself, *don't* just cd to
/home/buildbot/build/sage/iras-1/iras_full/build/sage-4.7.2.alpha4/spkg/build/mpir-2.1.3.p5 and type 'make check' or whatever is appropriate.
Instead, the following commands setup all environment variables
correctly and load a subshell for you to debug the error:
(cd '/home/buildbot/build/sage/iras-1/iras_full/build/sage-4.7.2.alpha4/spkg/build/mpir-2.1.3.p5' && '/home/buildbot/build/sage/iras-1/iras_full/build/sage-4.7.2.alpha4/sage' -sh)
When you are done debugging, you can type "exit" to leave the
subshell.

One can easily grep for this to find out which package caused the problem and then look at spkg/logs/$PACKAGENAME.log.

[jdemeyer]

Attachment: singular-3-1-3-3.p0-p1.diff.gz

diff for the new Singular spkg, for review only

[jdemeyer]

comment:121

New spkg with error messages for failed patches: http://boxen.math.washington.edu/home/jdemeyer/spkg/singular-3-1-3-3.p1.spkg. Needs review.

[vbraun]

comment:122

Looks good. I take it from your comment that it works on OSX/PPC. Positive review.

[alexanderdreyer]

comment:124

Replying to @vbraun:

Looks good. I take it from your comment that it works on OSX/PPC. Positive review.

For the record: I also managed to build, install and succesfully run all tests on SuSE 11.1 SP1 and OSX 10.6 ppc.

[nexttime]

comment:125

Replying to @vbraun:

Looks good.

Looks better.

Btw., I sometimes also grep for "An error". Grepping for "^(Error|Failed) " (perhaps with context, which doesn't make much sense in the former case) immediately shows you what went wrong. Of course the logs of spkgs that are themselves built in parallel are sometimes harder to read.

[saraedum]

comment:126

trac_10903_singular-fixes.patch seems to remove the functionality of the parameter omit_underscore_names in sageinspect.py's sage_getvariablename(). Was that done on purpose?