python-2.7 meta issue #51
Comments
|
It looks like the use of pickling is very extensive now. It is very much necessary to include the pickling patch to find out what's noise and what's a problem. |
|
We have AttributeError: 'InstanceTester' object has no attribute '_type_equality_funcs' all over the place |
|
2.7.1-r1 is stable on amd64 so we'll really have to solve issues. |
|
Shall we create a branch for this, with 2.7 enabled in the ebuilds and the 2.7 python version activated in the update-python script? That way we could share current status even in the presence of failures. How would we determine the python slot to use? Hardwired into the ebuilds, as 2.6 is now, or determined at emerge time, or shall we even try to support multiple ABIs simultaneously, as many python packages do? I have no strong feelings either way. |
|
I have given some thought to this. I am leaning to force the python slot. This is because of sage-clib which links explicitly to libpython2.6 (currently). If we want concurent python version to exist we have to build it twice and move it /usr/lib(64)/python2.x instead of the current /usr/lib(64). That's a big hassle so I say just one python at a time. |
|
On another note for people not following sage-devel, most the issues with python-2.7 are resolved. There is a big bunch of doctest failures but they are caused by: a change in the number of significant figures displayed [aka numerical noise], and a change in the content of error messages [aka formatting noise]. |
|
OK sage-4.7.alpha4 is in the overlay and is python-2.7. There are a few wrinkles to to iron in sage-notebook but it is otherwise behaving. Because I enabled PYTHONWARNINGS we get to see plenty of ugly stuff as well as the stuff we want to see. We want to see sage's deprecation messages because they are tested. But now we also see python complaining about deprecated stuff in sage. In the end I am sure that's a good thing (tm). |
|
I have updated the sympy-0.6.6 ebuild. This is to allow it to build both with python-2.6 and python-2.7. It escaped my radar on macos because the only python available there is 2.7 now so it built on that. On a system where both 2.6 and 2.7 are available only 2.6 will be built. I am looking at the other python ebuilds in overlay for similar problems. |
|
sage-latex now builds for python-2.6 and 2.7 concurrently. pynac-0.2.1.ebuild is for python-2.6, pynac-0.2.1-r1.ebuild is for python-2.7. Strangely enough libpynac.so doesn't appear to use any libpython but python headers are used. Polybori-0.7 is now set as a python-2.7 only ebuild. scons blows - nothing new here. |
|
Adopt Steve Trogdon patch and remove PYTHONWARNINGS. Less chatter. We will still need to deal with the deprecation warning about "sets" in sagenb. I am sure Christopher dealt with that before but probably in sage only? |
|
There are some killed doctests (3 but not consistently). Here is how to get one along with a backtrace: sage: f = Zmod(6).cover()
sage: f.kernel()
Principal ideal (6) of Integer Ring
sage: R.<x,y> = PolynomialRing(QQ, 2)
sage: S.<a,b> = R.quo(x^2 + y^2)
sage: phi = S.cover()
sage: phi == loads(dumps(phi))
True
sage: phi == R.quo(x^2 + y^3).cover()
Program received signal SIGSEGV, Segmentation fault.
id_Delete (h=0x50687d8, r=0x0) at ideals.cc:127
127 ideals.cc: No such file or directory.
in ideals.cc
(gdb) bt
#0 id_Delete (h=0x50687d8, r=0x0) at ideals.cc:127
#1 0x00007fffb9670b49 in __pyx_pf_4sage_4libs_8singular_17groebner_strategy_16GroebnerStrategy_1__dealloc__ (
o=) at sage/libs/singular/groebner_strategy.cpp:2570
#2 __pyx_tp_dealloc_4sage_4libs_8singular_17groebner_strategy_GroebnerStrategy (o=)
at sage/libs/singular/groebner_strategy.cpp:3190
#3 0x00007ffff7aa98c7 in PyDict_Clear (op=) at Objects/dictobject.c:891
#4 0x00007ffff7aa990f in dict_tp_clear (op=) at Objects/dictobject.c:2088
#5 0x00007ffff7b32f27 in delete_garbage (generation=0) at Modules/gcmodule.c:769
#6 collect (generation=0) at Modules/gcmodule.c:930
#7 0x00007ffff7b33616 in collect_generations (basicsize=) at Modules/gcmodule.c:996
#8 _PyObject_GC_Malloc (basicsize=) at Modules/gcmodule.c:1457
#9 0x00007ffff7ac4a96 in PyType_GenericAlloc (type=0x7ffff7d9ad40, nitems=0) at Objects/typeobject.c:744
#10 0x00007ffff7a8d256 in BaseException_new (type=, args=, kwds=)
at Objects/exceptions.c:34
#11 0x00007ffff7ac57b5 in type_call (type=0x7ffff7d9ad40, args=0x4ae46d0, kwds=0x0) at Objects/typeobject.c:712
#12 0x00007ffff7a77539 in PyObject_Call (func=0x7ffff7d9ad40, arg=0x4ae46d0, kw=0x0) at Objects/abstract.c:2529
#13 0x00007ffff7b00b45 in PyEval_CallObjectWithKeywords (func=0x7ffff7d9ad40, arg=0x4ae46d0, kw=0x0) at Python/ceval.c:3881
#14 0x00007ffff7b10443 in PyErr_NormalizeException (exc=0x7fffffffa778, val=0x7fffffffa770, tb=0x7fffffffa768)
at Python/errors.c:190
#15 0x00007fffe433021c in __Pyx_GetException (type=0x7fffffffa7c0, value=0x7fffffffa7b8, tb=0x7fffffffa7c8)
at sage/structure/parent.c:20909
#16 0x00007fffe433e4d1 in __pyx_pf_4sage_9structure_6parent_6Parent_2element_class (__pyx_v_self=0x4aea500,
unused=) at sage/structure/parent.c:4251
#17 0x00007ffff7aac1e2 in PyCFunction_Call (func=0x504ebd8, arg=0x7ffff7f81050, kw=)
at Objects/methodobject.c:90
#18 0x00007ffff7a77539 in PyObject_Call (func=0x504ebd8, arg=0x7ffff7f81050, kw=0x0) at Objects/abstract.c:2529
#19 0x00007ffff7b00b45 in PyEval_CallObjectWithKeywords (func=0x504ebd8, arg=0x7ffff7f81050, kw=0x0) at Python/ceval.c:3881
#20 0x00007ffff7a8b51f in methoddescr_call (descr=, args=0x7ffff7f81050, kwds=0x0)
at Objects/descrobject.c:246
#21 0x00007ffff7a77539 in PyObject_Call (func=0x138e950, arg=0x7ffff7e9c050, kw=0x0) at Objects/abstract.c:2529
#22 0x00007ffff7b05efe in do_call (f=, throwflag=) at Python/ceval.c:4230
#23 call_function (f=, throwflag=) at Python/ceval.c:4035
#24 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2665
#25 0x00007ffff7b07b65 in PyEval_EvalCodeEx (co=0x1159e30, globals=, locals=,
args=, argcount=3, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3252
#26 0x00007ffff7a99a97 in function_call (func=0x115d5f0, arg=0xfadeb0, kw=0x0) at Objects/funcobject.c:526
#27 0x00007ffff7a77539 in PyObject_Call (func=0x115d5f0, arg=0xfadeb0, kw=0x0) at Objects/abstract.c:2529
#28 0x00007ffff7a77cca in PyObject_CallFunctionObjArgs (callable=0x115d5f0) at Objects/abstract.c:2760
#29 0x00007ffff7ac8a8e in slot_tp_descr_get (self=0x13179d0, obj=0x4aea500, type=0x19e91f0) at Objects/typeobject.c:5621
#30 0x00007ffff7aae433 in _PyObject_GenericGetAttrWithDict (obj=0x4aea500, name=0x13c5a78, dict=0x5068fb0) at Objects/object.c:1432
#31 0x00007ffff7aae4d1 in PyObject_GenericGetAttr (obj=, name=) at Objects/object.c:1454
#32 0x00007fffe434ed62 in __pyx_tp_getattro_4sage_9structure_6parent_Parent (o=0x4aea500, n=0x13c5a78)
at sage/structure/parent.c:18702
#33 0x00007ffff7aad8f4 in PyObject_GetAttr (v=0x4aea500, name=) at Objects/object.c:1189
#34 0x00007ffff7afdbab in builtin_hasattr (self=, args=) at Python/bltinmodule.c:885
#35 0x00007ffff7aac1a0 in PyCFunction_Call (func=0x7ffff7fae3b0, arg=0x503d680, kw=)
at Objects/methodobject.c:81
#36 0x00007ffff7b05ba3 in call_function (f=, throwflag=) at Python/ceval.c:4012
#37 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2665
#38 0x00007ffff7b05cfa in fast_function (f=, throwflag=) at Python/ceval.c:4098
#39 call_function (f=, throwflag=) at Python/ceval.c:4033
#40 PyEval_EvalFrameEx (f=, throwflag=) at Python/ceval.c:2665
#41 0x00007ffff7b07b65 in PyEval_EvalCodeEx (co=0x1159e30, globals=, locals=,
args=, argcount=3, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3252
#42 0x00007ffff7a99a97 in function_call (func=0x115d5f0, arg=0xfadbe0, kw=0x0) at Objects/funcobject.c:526
|
|
With a better backtrace we have this at the beginning: Program received signal SIGSEGV, Segmentation fault.
id_Delete (h=0x509b078, r=0x0) at ideals.cc:127
127 p_Delete(&((*h)->m[--j]), r);
(gdb) bt
#0 id_Delete (h=0x509b078, r=0x0) at ideals.cc:127
#1 0x00007fffb9a3db49 in __pyx_pf_4sage_4libs_8singular_17groebner_strategy_16GroebnerStrategy_1__dealloc__ (
o=) at sage/libs/singular/groebner_strategy.cpp:2570
#2 __pyx_tp_dealloc_4sage_4libs_8singular_17groebner_strategy_GroebnerStrategy (o=)
Martin, I have a suspicion that in fact p_Delete is not found, can you confirm that suspicion? How would I check libsingular.so to see if the function is in there? readelf -s doesn't show "the right" p_Delete: readelf -s /usr/lib64/libsingular.so.3.1.1 | grep p_Delete 630: 0000000000324f87 112 FUNC GLOBAL DEFAULT 11 _Z41p_Delete__FieldQ_Leng 3361: 0000000000323330 83 FUNC GLOBAL DEFAULT 11 _Z42p_Delete__FieldZp_Len 4474: 0000000000322e26 117 FUNC GLOBAL DEFAULT 11 _Z46p_Delete__RingGeneral 5613: 0000000000323383 117 FUNC GLOBAL DEFAULT 11 _Z47p_Delete__FieldGenera |
|
I'm still in the process of upgrading to 4.7, so I can't reproduce (yet), but some ideas up front:
0000000000272540 g DF .text 000000000000006f Base p_Delete__FieldZp_LengthGeneral_OrdGeneral 00000000002725c0 g DF .text 0000000000000093 Base p_Delete__FieldGeneral_LengthGeneral_OrdGeneral 0000000000275180 g DF .text 000000000000008d Base p_Delete__FieldQ_LengthGeneral_OrdGeneral So perhaps it would be best to use the gdb command |
|
Upgrade complete, still can't reproduce. This is singular-3.1.2-r2 if that makes a difference. |
|
doesn't make any difference the important ebuild is libsingular not singular. Do any of your tests get killed/crashed? I also inspected the code so I know where the p_Delete__Field* come from they are under control. |
|
According to Alexander Dreyer - posted from libsingular-devel: the real problem is we somehow have a ring that is pointing to NULL and is undefined. Will look at disassemble. |
|
I seem to have incriminated python garbage collection module... sage: import gc sage: gc.disable() and then the sequence pass just fine. |
|
See no killed or crashed tests here after running |
|
After using the gc.disable() trick Steve experience SIGSEGV when he quit sage after going through our sequence of commands. Why do I smell a memory leak of some kind? In other merges I added a new patch by friendly upstream dev Nicolas M. Thierry to fix a doctest. I also enforced the use of cython-2.7. While it is without big consequences (I think) I found that on my system on which python-2.6 is the default, cython-2.6 was actually called even though we made python 2.7 the active one. Anyone think it is a bug? |
|
Adjusting the EPYTHON variable in sage-env should have taken care of that. Perhaps you could scatter some |
|
I am not talking about which cython sage is calling. I am talking about the building of sage at the ebuild level: sage/coding/binary_code.pyx --> build/sage/coding/binary_code.pyx
python `which cython` --disable-function-redefinition --embed-positions --directive cdivision=True,autotestdict=False,fast_getattr=True -I/var/tmp/portage/sci-mathematics/sage-4.7_alpha5/work/sage-4.7.alpha5 -o sage/combinat/expnums.c sage/combinat/expnums.pyx
sandbox:stop caught signal 2 in pid 1394
sandbox:stop signal already caught and busy still cleaning up!
Traceback (most recent call last):
File "/usr/bin/cython-2.6", line 7, in
from Cython.Compiler.Main import main
File "/usr/lib/python2.6/site-packages/Cython/Compiler/Main.py", line 29, in
from Symtab import BuiltinScope, ModuleScope
File "/usr/lib/python2.6/site-packages/Cython/Compiler/Symtab.py", line 12, in
import TypeSlots
File "/usr/lib/python2.6/site-packages/Cython/Compiler/TypeSlots.py", line 606, in
MethodSlot(ibinaryfunc, "nb_inplace_and", "__iand__"),
File "/usr/lib/python2.6/site-packages/Cython/Compiler/TypeSlots.py", line 240, in __init__
for alt in (self.py2, self.py3):
KeyboardInterrupt
Error running command, failed with status 256.
/home/francois/Work/Overlays/Gentoo-sage/sci-mathematics/sage/sage-4.7_alpha5.ebuild: src_compile aborted; exiting.
I interrupted this build because I wanted to change something. Imagine my surprise when I found that cython-2.6 was actually called. |
|
cython is a wrapper script which calls As far as I can tell, we don't have a sage-notebook ebuild for python 2.7 in tree. I'd like to have one, and have dependencies in such a way that sage and sage-notebook match one another in terms of the python version they use. |
|
OK, I was wrong about there not being a sage-notebook version for 2.7; my automatic layman updates had failed without me noticing. WRT supporting multiple ABIs, gagern/sage-on-gentoo@0988670fe1ab523157bc is an attempt to support that for sage-clib. It should be possible to switch the other sage packages as well. Haven't done so yet, though. |
|
Thanks for that Martin, it is quite instructive. The next question we have to ask ourselves is: Is it worth it? |
|
hum considering we have python 2.4 and 2.5 in the tree unless they are retired as well, we'll probably have a 2.6 hanging around. I'll have to find a definite answer to that. |
|
I committed some doctest patch for python-2.7 but I goofed, they only apply with maximalib. They were applying nicely on my private ebuild of 4.7.1.alpha1 in which maximalib will land. I have been working on 4.7.1.alpha1 to get my patch upstream. Some editing will be needed for 4.7 final. |
|
Martin, is your posted test-log file a live file? I couldn't access it when you posted it but now that I have looked at it, it looks like a lot of stuff is python-2.6 in there. |
|
You're right, that was at a time when my layman didn't sync and therefore used an old alpha3 ebuild which fixed python to 2.6. I've just rerun with the current 4.7_rc0 against Python 2.7 and now I do indeed get segfaults: $ grep -B1 'signal 11' ~/.sage/tmp/test.log | grep ^sage sage -t -force_lib "devel/sage/sage/schemes/generic/scheme.py" sage -t -force_lib "devel/sage/sage/rings/morphism.pyx" sage -t -force_lib "devel/sage/sage/rings/homset.py" I've updated the complete log on my web server. |
|
OK, I've debugged the issue in What happens is that the garbage collector invokes the tp_clear function for the object. Its implementation is provided by Cython, and one of its effects is that every python reference will be set to None. A bit later on, tp_dealloc is called and invokes the Others have had similar problems before. There are crude workarounds floating around. I guess a proper solution would be twaking cython to allow custom code in the |
|
Any ideas why it appeared with python 2.7? Considering some comments in the sage code there may have been problems before, from sage/libs/singular/rings.pyx cdef void singular_ring_delete(ring *doomed):
"""
Carefully deallocate the ring, without changing "currRing" (since
this method can be called at unpredictable times due to garbage
collection).
TESTS:
This example caused a segmentation fault with a previous version
of this method::
sage: import gc
sage: from sage.rings.polynomial.multi_polynomial_libsingular import MPolynomialRing_libsingular
sage: R1 = MPolynomialRing_libsingular(GF(5), 2, ('x', 'y'), TermOrder('degrevlex', 2))
sage: R2 = MPolynomialRing_libsingular(GF(11), 2, ('x', 'y'), TermOrder('degrevlex', 2))
sage: R3 = MPolynomialRing_libsingular(GF(13), 2, ('x', 'y'), TermOrder('degrevlex', 2))
sage: _ = gc.collect()
sage: foo = R1.gen(0)
sage: del foo
sage: del R1
sage: _ = gc.collect()
sage: del R2
sage: _ = gc.collect()
sage: del R3
sage: _ = gc.collect()
"""
cdef ring *oldRing = NULL
if currRing != doomed:
oldRing = currRing
rChangeCurrRing(doomed)
rDelete(doomed)
rChangeCurrRing(oldRing)
else:
(&currRing)[0] = NULL
rDelete(doomed)
I think it should be taken to cython upstream, Robert Bradshaw is a good guy and I am hoping he will be responsive to that. |
|
Reported the above as Sage Ticket 11339. |
|
While I am not hot on supporting multiple python I am pushing a patch upstream that would actually make it easier to build sage with multiple python: |
|
sage 5.0 will bring python 2.7 upstream and we are good with 4.8 in the overlay. Closing. |
OK so on macos and possibly prefix in general we will be pushed out of python-2.6 rather soon. So it is important to find the issues with 2.7.
I am building sage on macos prefix with the currently available 2.7.1-r00.1 ebuild, without any patch for pickles (that will be a few identified test failures that can be fixed later).
So far the building of 4.6.2 is successful after modifying sage-notebook and sage to accept 2.7.
sage-baselayout will need a change as well.
The text was updated successfully, but these errors were encountered: