Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SystemError: <class 'astropy.io.ascii.cparser.FastWriter'> returned NULL without setting an error #7744

Closed
Gabriel-p opened this issue Aug 16, 2018 · 16 comments
Closed

SystemError: <class 'astropy.io.ascii.cparser.FastWriter'> returned NULL without setting an error #7744

Gabriel-p opened this issue Aug 16, 2018 · 16 comments
Labels
Close? closed-by-bot io.ascii

Comments

@Gabriel-p
Copy link
Contributor

Gabriel-p commented Aug 16, 2018 •
edited
Loading

I'm getting this error when trying to write a file out using:

 ascii.write(data, 'name.dat', format='csv', overwrite=True)

It appears to be related to this cython closed issue. Removing format='csv' makes the issue go away.

Here's what's in my conda environment:

$ conda list
# packages in environment at /home/gabriel/anaconda3/envs/asteca3:
#
# Name                    Version                   Build  Channel
astropy                   3.0.4            py37h14c3975_0  
atomicwrites              1.1.5                    py37_0  
attrs                     18.1.0                   py37_0  
autograd                  1.2                       <pip>
blas                      1.0                         mkl  
ca-certificates           2018.03.07                    0  
certifi                   2018.8.13                py37_0  
cycler                    0.10.0                   py37_0  
dbus                      1.13.2               h714fa37_1  
emcee                     2.2.1              pyh24bf2e0_4    astropy
expat                     2.2.5                he0dffb1_0  
fontconfig                2.13.0               h9420a91_0  
freetype                  2.9.1                h8a8886c_0  
future                    0.16.0                    <pip>
glib                      2.56.1               h000015b_0  
gst-plugins-base          1.14.0               hbbd80ab_1  
gstreamer                 1.14.0               hb453b48_1  
icu                       58.2                 h9c2bf20_1  
intel-openmp              2018.0.3                      0  
jpeg                      9b                   h024ee3a_2  
kiwisolver                1.0.1            py37hf484d3e_0  
libedit                   3.1.20170329         h6b74fdf_2  
libffi                    3.2.1                hd88cf55_4  
libgcc-ng                 7.2.0                hdf63c60_3  
libgfortran-ng            7.2.0                hdf63c60_3  
libpng                    1.6.34               hb9fc6fc_0  
libstdcxx-ng              7.2.0                hdf63c60_3  
libuuid                   1.0.3                h1bed415_2  
libxcb                    1.13                 h1bed415_1  
libxml2                   2.9.8                h26e45fe_1  
matplotlib                2.2.2            py37hb69df0a_2  
mkl                       2018.0.3                      1  
mkl_fft                   1.0.4            py37h4414c95_1  
mkl_random                1.0.1            py37h4414c95_1  
more-itertools            4.2.0                    py37_0  
ncurses                   6.1                  hf484d3e_0  
numpy                     1.15.0           py37h1b885b7_0  
numpy-base                1.15.0           py37h3dfced4_0  
openssl                   1.0.2p               h14c3975_0  
pandas                    0.23.3           py37h04863e7_0  
patsy                     0.5.0                    py37_0  
pcre                      8.42                 h439df22_0  
pip                       10.0.1                   py37_0  
pip                       18.0                      <pip>
pluggy                    0.6.0                    py37_0  
psutil                    5.4.6            py37h14c3975_0  
py                        1.5.4                    py37_0  
pyparsing                 2.2.0                    py37_1  
pyqt                      5.9.2            py37h22d08a2_0  
pytest                    3.6.3                    py37_0  
pytest-arraydiff          0.2              py37h39e3cac_0  
pytest-astropy            0.4.0                    py37_0  
pytest-doctestplus        0.1.3                    py37_0  
pytest-openfiles          0.3.0                    py37_0  
pytest-remotedata         0.3.0                    py37_0  
python                    3.7.0                hc3d631a_0  
python-dateutil           2.7.3                    py37_0  
pytz                      2018.5                   py37_0  
qt                        5.9.6                h52aff34_0  
readline                  7.0                  ha6073c6_4  
sampyl-mcmc               0.3                       <pip>
scipy                     1.1.0            py37hc49cb51_0  
seaborn                   0.9.0                    py37_0  
setuptools                39.2.0                   py37_0  
sip                       4.19.8           py37hf484d3e_0  
six                       1.11.0                   py37_1  
sqlite                    3.24.0               h84994c4_0  
statsmodels               0.9.0            py37h035aef0_0  
tk                        8.6.7                hc745277_3  
tornado                   5.0.2            py37h14c3975_0  
wheel                     0.31.1                   py37_0  
xz                        5.2.4                h14c3975_4  
zlib                      1.2.11               ha838bed_2

Trace:

Traceback (most recent call last):
...
  File "/home/gabriel/anaconda3/envs/asteca3/lib/python3.7/site-packages/astropy/io/ascii/ui.py", line 881, in write
    writer.write(table, output)
  File "/home/gabriel/anaconda3/envs/asteca3/lib/python3.7/site-packages/astropy/io/ascii/fastbasic.py", line 194, in write
    self._write(table, output, {'fill_values': [(core.masked, '')]})
  File "/home/gabriel/anaconda3/envs/asteca3/lib/python3.7/site-packages/astropy/io/ascii/fastbasic.py", line 170, in _write
    writer = cparser.FastWriter(table, **write_kwargs)
SystemError: <class 'astropy.io.ascii.cparser.FastWriter'> returned NULL without setting an error
@pllim pllim added the io.ascii label Aug 17, 2018
@pllim
Copy link
Member

pllim commented Aug 17, 2018

It didn't say what Cython version you have.

@Gabriel-p
Copy link
Contributor Author

Cython is not installed in that environment.

@pllim
Copy link
Member

pllim commented Aug 17, 2018

Does the problem go away if you install Cython? Just curious because you mentioned a Cython issue in your report above but you don't have Cython.

@Gabriel-p
Copy link
Contributor Author

It does not.

$ conda list
# packages in environment at /home/gabriel/anaconda3/envs/asteca3:
#
# Name                    Version                   Build  Channel
astropy                   3.0.4            py37h14c3975_0  
atomicwrites              1.1.5                    py37_0  
attrs                     18.1.0                   py37_0  
autograd                  1.2                       <pip>
blas                      1.0                         mkl  
ca-certificates           2018.03.07                    0  
certifi                   2018.8.13                py37_0  
cycler                    0.10.0                   py37_0  
cython                    0.28.5           py37hf484d3e_0  
dbus                      1.13.2               h714fa37_1  
emcee                     2.2.1              pyh24bf2e0_4    astropy
expat                     2.2.5                he0dffb1_0  
fontconfig                2.13.0               h9420a91_0  
freetype                  2.9.1                h8a8886c_0  
future                    0.16.0                    <pip>
glib                      2.56.1               h000015b_0  
gst-plugins-base          1.14.0               hbbd80ab_1  
gstreamer                 1.14.0               hb453b48_1  
icu                       58.2                 h9c2bf20_1  
intel-openmp              2018.0.3                      0  
jpeg                      9b                   h024ee3a_2  
kiwisolver                1.0.1            py37hf484d3e_0  
libedit                   3.1.20170329         h6b74fdf_2  
libffi                    3.2.1                hd88cf55_4  
libgcc-ng                 7.2.0                hdf63c60_3  
libgfortran-ng            7.2.0                hdf63c60_3  
libpng                    1.6.34               hb9fc6fc_0  
libstdcxx-ng              7.2.0                hdf63c60_3  
libuuid                   1.0.3                h1bed415_2  
libxcb                    1.13                 h1bed415_1  
libxml2                   2.9.8                h26e45fe_1  
matplotlib                2.2.2            py37hb69df0a_2  
mkl                       2018.0.3                      1  
mkl_fft                   1.0.4            py37h4414c95_1  
mkl_random                1.0.1            py37h4414c95_1  
more-itertools            4.2.0                    py37_0  
ncurses                   6.1                  hf484d3e_0  
numpy                     1.15.0           py37h1b885b7_0  
numpy-base                1.15.0           py37h3dfced4_0  
openssl                   1.0.2p               h14c3975_0  
pandas                    0.23.3           py37h04863e7_0  
patsy                     0.5.0                    py37_0  
pcre                      8.42                 h439df22_0  
pip                       10.0.1                   py37_0  
pip                       18.0                      <pip>
pluggy                    0.6.0                    py37_0  
psutil                    5.4.6            py37h14c3975_0  
py                        1.5.4                    py37_0  
pyparsing                 2.2.0                    py37_1  
pyqt                      5.9.2            py37h22d08a2_0  
pytest                    3.6.3                    py37_0  
pytest-arraydiff          0.2              py37h39e3cac_0  
pytest-astropy            0.4.0                    py37_0  
pytest-doctestplus        0.1.3                    py37_0  
pytest-openfiles          0.3.0                    py37_0  
pytest-remotedata         0.3.0                    py37_0  
python                    3.7.0                hc3d631a_0  
python-dateutil           2.7.3                    py37_0  
pytz                      2018.5                   py37_0  
qt                        5.9.6                h52aff34_0  
readline                  7.0                  ha6073c6_4  
sampyl-mcmc               0.3                       <pip>
scipy                     1.1.0            py37hc49cb51_0  
seaborn                   0.9.0                    py37_0  
setuptools                39.2.0                   py37_0  
sip                       4.19.8           py37hf484d3e_0  
six                       1.11.0                   py37_1  
sqlite                    3.24.0               h84994c4_0  
statsmodels               0.9.0            py37h035aef0_0  
tk                        8.6.7                hc745277_3  
tornado                   5.0.2            py37h14c3975_0  
wheel                     0.31.1                   py37_0  
xz                        5.2.4                h14c3975_4  
zlib                      1.2.11               ha838bed_2

@Gabriel-p
Copy link
Contributor Author

I also tried removing and re-installing astropy after installing cython. No dice.

@pllim
Copy link
Member

pllim commented Aug 17, 2018

Thanks for checking!

@bsipocz
Copy link
Member

bsipocz commented Aug 17, 2018

I'm getting this error when trying to write a file out using:

ascii.write(data, 'name.dat', format='csv', overwrite=True)

have you tried to turn off the fast writer, with the fast_writer=False option?

@cdeil
Copy link
Member

cdeil commented Aug 24, 2018

I can reproduce this bug with Python 3.7 and Astropy 3.0.4 .

Minimal test case to reproduce:

$ conda create --name trouble python=3.7 astropy
$ conda activate trouble
$ python
Python 3.7.0 | packaged by conda-forge | (default, Aug  7 2018, 14:28:25) 
[Clang 6.1.0 (clang-602.0.53)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import astropy
>>> astropy.__version__
'3.0.4'
>>> from astropy.table import Table
>>> t = Table({'a': [1, 2, 3]})
>>> t.write('test.csv')
>>> t.write('test.csv')
WARNING: AstropyDeprecationWarning: test.csv already exists. Automatically overwriting ASCII files is deprecated. Use the argument 'overwrite=True' in the future. [astropy.io.ascii.ui]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/deil/software/anaconda3/envs/gammapy-dev-37/lib/python3.7/site-packages/astropy/table/table.py", line 2562, in write
    io_registry.write(self, *args, **kwargs)
  File "/Users/deil/software/anaconda3/envs/gammapy-dev-37/lib/python3.7/site-packages/astropy/io/registry.py", line 560, in write
    writer(data, *args, **kwargs)
  File "/Users/deil/software/anaconda3/envs/gammapy-dev-37/lib/python3.7/site-packages/astropy/io/ascii/connect.py", line 86, in write_csv
    return write(table, filename, **kwargs)
  File "/Users/deil/software/anaconda3/envs/gammapy-dev-37/lib/python3.7/site-packages/astropy/io/ascii/ui.py", line 881, in write
    writer.write(table, output)
  File "/Users/deil/software/anaconda3/envs/gammapy-dev-37/lib/python3.7/site-packages/astropy/io/ascii/fastbasic.py", line 194, in write
    self._write(table, output, {'fill_values': [(core.masked, '')]})
  File "/Users/deil/software/anaconda3/envs/gammapy-dev-37/lib/python3.7/site-packages/astropy/io/ascii/fastbasic.py", line 170, in _write
    writer = cparser.FastWriter(table, **write_kwargs)
SystemError: <class 'astropy.io.ascii.cparser.FastWriter'> returned NULL without setting an error

Note how the error is only triggered on the second write call.

I also tried with Astropy master (e9dee2e) and the issue seems to be resolved; so possibly just making a new Astropy release would resolve it.

But maybe it's worth adding a regression test for this?

Note that if you try to run Astropy tests with Python 3.7 you might run into this pytest bug at the moment: pytest-dev/pytest#3854 (comment)

@taldcroft or anyone - thoughts?

@cdeil cdeil mentioned this issue Aug 24, 2018
Merged
@Gabriel-p
Copy link
Contributor Author

I'm sorry @bsipocz for some reason I never got a notification of your comment. Disabling fast_writer=False makes the issue go away.

@taldcroft
Copy link
Member

@cdeil - I assume this is only with Python 3.7, not 3.6. At least I don't reproduce the error you saw on 3.6 with astropy 3.0.4.

I'm not 100% sure, but I think #5578 is the main thing in master that isn't in 3.0.4 as far as the C fast reader.

@saimn
Copy link
Contributor

saimn commented Aug 30, 2018

Something to check maybe: it could be that you are installing a wheel or conda package that was built with a problematic cython version, not including the fix on their side?

@cdeil
Copy link
Member

cdeil commented Sep 10, 2018

Not sure if it helps, but here's the error in a Gammapy CI build:
https://travis-ci.org/gammapy/gammapy/jobs/426433957#L4305
It only occurs with Python 3.7, not with 3.6.

For now I'll skip that one test on Python 3.7 in Gammapy

@pllim
Copy link
Member

pllim commented Sep 10, 2018 •
edited
Loading

Since this was addressed at cython/cython#2043 , which was fixed back in Dec 2017, why is this still an issue now? Given the Cython timeline, the fix should be in Cython >=0.28?

Hmm, @cdeil 's log shows cython: 0.28.5...

@pllim
Copy link
Member

pllim commented Feb 20, 2024

I don't think this is a problem anymore in 2024? 👀

@pllim pllim added the Close? label Feb 20, 2024
@github-actions GitHub Actions
Copy link
Contributor

Hi humans 👋 - this issue was labeled as Close? approximately 12 hours ago. If you think this issue should not be closed, a maintainer should remove the Close? label - otherwise, I will close this issue in 7 days.

If you believe I commented on this issue incorrectly, please report this here

@github-actions GitHub Actions
Copy link
Contributor

I'm going to close this issue as per my previous message, but if you feel that this issue should stay open, then feel free to re-open and remove the Close? label.

If this is the first time I am commenting on this issue, or if you believe I closed this issue incorrectly, please report this here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Close? closed-by-bot io.ascii
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@saimn @taldcroft @cdeil @pllim @Gabriel-p @bsipocz