Running AlignTraj on GRO/XTC causes Travis segfault

[IAlibay]

Expected behavior

AlignTraj and Transform should not segfault when running Travis.

Actual behavior

Running AlignTraj / Transform on the MDA GRO/XTC testfiles causes Travis to fail on python 3.6-3.8 w/ Linux. Interestingly we cannot reproduce that behaviour locally. See: MDAnalysis/UserGuide#83 (transform), and https://travis-ci.com/github/bigginlab/MDRestraintsGenerator/jobs/357265043 (aligntraj)

Note: I'm not 100% convinced this isn't user error somehow (i.e. Travis setup related), so I'm going to open a test PR here to see if I can reproduce it.

Code to reproduce the behavior

Will work fine for the PSF/DCD pair, but fails on GRO/XTC.

Transform:

import MDAnalysis as mda
from MDAnalysis.tests.datafiles import TPR, XTC
from MDAnalysis import transformations as trans

def test_transform():
    u = mda.Universe(TPR, XTC)
    protein = u.select_atoms('protein')
    align_transform = trans.fit_rot_trans(protein, protein, weights=protein.masses)
    u.trajectory.add_transformations(align_transform)
    assert 1 == 1

Aligntraj:

import MDAnalysis as mda
from MDAnalysis.tests.datafiles import GRO, XTC

@pytest.mark.parametrize('top,traj', [(PSF, DCD), (GRO, XTC)])
def test_aligntraj(tmpdir, top, traj):
    """AlignTraj is failing, so let's test it here"""
    copy_u = mda.Universe(top, traj)
    prealigner = align.AlignTraj(copy_u, copy_u, select="protein and name CA",
                                                 in_memory=True)
    prealigner.run()

    assert 1 == 1

Current version of MDAnalysis

• Which version are you using? (run python -c "import MDAnalysis as mda; print(mda.__version__)") 1.0.0 and 0.20.1 tested
• Which version of Python (python -V)? 3.6 - 3.8
• Which operating system? Ubuntu 18.04 & 16.04 LTS (Travis-CI)

[lilyminium]

#2819 keeps passing despite the same code segmentation-faulting on the user guide (history of MDAnalysis/UserGuide#83 and notes in MDAnalysis/UserGuide#82), and @IAlibay's IAlibay/MDRestraintsGenerator#23. Between us we've iterated through the MDAnalysis version, gcc, OS, Python, dependency versions, and Travis image to fit this repo's. We also can't reproduce locally so we can't debug that way. Any @MDAnalysis/coredevs have any ideas why this is happening? @tylerjereddy maybe?

Also of interest is that for the user guide it also segfaults on ReadTheDocs.

[IAlibay]

Update here, I changed the Travis yml files to use astropy's CI helpers like we do here and magically everything started working... It is rather concerning that Travis will fail with a standard "get the latest miniconda, build an environment from this yml file, go".

We probably should get to the bottom of what kind of magic astropy is doing...

[kain88-de]

ABI incompatibilities, Fun! Astropi helpers installs everything from conda forge. Not sure what the default does. I guess mix mini conda packages with pip packages. Pip packages usually are build with a manylinux image. There could be some subtle issue there.

…

On Fri 3. Jul 2020 at 15:24, Irfan Alibay ***@***.***> wrote: Update here, I changed the Travis yml files to use astropy's CI helpers like we do here and magically everything started working... It is rather concerning that Travis will fail with a standard "get the latest miniconda, build an environment from this yml file, go". We probably should get to the bottom of what kind of magic astropy is doing... — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub <#2818 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABA2OVVFOVUZMAVEJSFCLDTRZXLZLANCNFSM4OPTBZPQ> .

[orbeckst]

Good point. Possibly make sure that you do not use the default conda channel and only conda-forge when not using the helpers?

Any of our cython code that use numpy arrays is probably very sensitive to what numpy was build against.

[lilyminium]

Thank you @kain88-de! This version where everything looks like it's conda-forge is still failing, though (and previously the user guide was just calling conda install mdanalysis anyway) -- am I missing something here?

$ conda list

# packages in environment at /home/travis/miniconda/envs/mda-user-guide:

#

# Name                    Version                   Build  Channel

_libgcc_mutex             0.1                 conda_forge    conda-forge

_openmp_mutex             4.5                      1_llvm    conda-forge

alabaster                 0.7.12                     py_0    conda-forge

attrs                     19.3.0                     py_0    conda-forge

babel                     2.8.0                      py_0    conda-forge

backcall                  0.2.0              pyh9f0ad1d_0    conda-forge

biopython                 1.77             py38h1e0a361_0    conda-forge

bleach                    3.1.5              pyh9f0ad1d_0    conda-forge

brotlipy                  0.7.0           py38h1e0a361_1000    conda-forge

bzip2                     1.0.8                h516909a_2    conda-forge

ca-certificates           2020.6.20            hecda079_0    conda-forge

certifi                   2020.6.20        py38h32f6830_0    conda-forge

cffi                      1.14.0           py38hd463f26_0    conda-forge

cftime                    1.1.3            py38h8790de6_0    conda-forge

chardet                   3.0.4           py38h32f6830_1006    conda-forge

cryptography              2.9.2            py38h766eaa4_0    conda-forge

curl                      7.71.1               he644dc0_0    conda-forge

cycler                    0.10.0                     py_2    conda-forge

cython                    0.29.20          py38h950e882_0    conda-forge

decorator                 4.4.2                      py_0    conda-forge

defusedxml                0.6.0                      py_0    conda-forge

docutils                  0.16             py38h32f6830_1    conda-forge

entrypoints               0.3             py38h32f6830_1001    conda-forge

freetype                  2.10.2               he06d7ca_0    conda-forge

griddataformats           0.5.0                      py_0    conda-forge

gsd                       2.1.2            py38h8790de6_0    conda-forge

hdf4                      4.2.13            hf30be14_1003    conda-forge

hdf5                      1.10.6          nompi_h3c11f04_100    conda-forge

hypothesis                5.19.0                     py_0    conda-forge

icu                       64.2                 he1b5a44_1    conda-forge

idna                      2.10               pyh9f0ad1d_0    conda-forge

imagesize                 1.2.0                      py_0    conda-forge

importlib-metadata        1.7.0            py38h32f6830_0    conda-forge

importlib_metadata        1.7.0                         0    conda-forge

ipykernel                 5.3.0            py38h23f93f0_0    conda-forge

ipython                   7.16.1           py38h23f93f0_0    conda-forge

ipython_genutils          0.2.0                      py_1    conda-forge

ipywidgets                7.5.1                      py_0    conda-forge

jedi                      0.17.1           py38h32f6830_0    conda-forge

jinja2                    2.11.2             pyh9f0ad1d_0    conda-forge

joblib                    0.16.0                     py_0    conda-forge

jpeg                      9d                   h516909a_0    conda-forge

jsonschema                3.2.0            py38h32f6830_1    conda-forge

jupyter_client            6.1.5                      py_0    conda-forge

jupyter_contrib_core      0.3.3                      py_2    conda-forge

jupyter_contrib_nbextensions 0.5.1                    py38_0    conda-forge

jupyter_core              4.6.3            py38h32f6830_1    conda-forge

jupyter_highlight_selected_word 0.2.0                 py38_1000    conda-forge

jupyter_latex_envs        1.4.6                 py38_1000    conda-forge

jupyter_nbextensions_configurator 0.4.1                    py38_0    conda-forge

kiwisolver                1.2.0            py38hbf85e49_0    conda-forge

krb5                      1.17.1               hfafb76e_1    conda-forge

latexcodec                2.0.1              pyh9f0ad1d_0    conda-forge

ld_impl_linux-64          2.34                 h53a641e_5    conda-forge

libblas                   3.8.0               17_openblas    conda-forge

libcblas                  3.8.0               17_openblas    conda-forge

libcurl                   7.71.1               hcdd3856_0    conda-forge

libedit                   3.1.20191231         h46ee950_0    conda-forge

libffi                    3.2.1             he1b5a44_1007    conda-forge

libgcc-ng                 9.2.0                h24d8f2e_2    conda-forge

libgfortran-ng            7.5.0                hdf63c60_6    conda-forge

libiconv                  1.15              h516909a_1006    conda-forge

liblapack                 3.8.0               17_openblas    conda-forge

libnetcdf                 4.7.4           nompi_h84807e1_104    conda-forge

libopenblas               0.3.10               h5ec1e0e_0    conda-forge

libpng                    1.6.37               hed695b0_1    conda-forge

libsodium                 1.0.17               h516909a_0    conda-forge

libssh2                   1.9.0                hab1572f_2    conda-forge

libstdcxx-ng              9.2.0                hdf63c60_2    conda-forge

libxml2                   2.9.10               hee79883_0    conda-forge

libxslt                   1.1.33               h31b3aaa_0    conda-forge

llvm-openmp               10.0.0               hc9558a2_0    conda-forge

lxml                      4.5.1            py38hbb43d70_0    conda-forge

markupsafe                1.1.1            py38h1e0a361_1    conda-forge

matplotlib-base           3.2.2            py38h2af1d28_0    conda-forge

mdanalysis                1.0.0            py38h950e882_0    conda-forge

mdanalysistests           1.0.0            py38h32f6830_1    conda-forge

mistune                   0.8.4           py38h1e0a361_1001    conda-forge

mmtf-python               1.1.2                      py_0    conda-forge

mock                      4.0.2            py38h32f6830_0    conda-forge

more-itertools            8.4.0                      py_0    conda-forge

msgpack-python            1.0.0            py38hbf85e49_1    conda-forge

msmb-theme                1.2.0                    pypi_0    pypi

nbconvert                 5.6.1            py38h32f6830_1    conda-forge

nbformat                  5.0.6                      py_0    conda-forge

nbsphinx                  0.7.1              pyh9f0ad1d_0    conda-forge

ncurses                   6.1               hf484d3e_1002    conda-forge

netcdf4                   1.5.3           nompi_py38hfd55d45_105    conda-forge

networkx                  2.4                        py_1    conda-forge

nglview                   2.7.7              pyh5ca1d4c_1    conda-forge

notebook                  6.0.3            py38h32f6830_0    conda-forge

numpy                     1.18.5           py38h8854b6b_0    conda-forge

openssl                   1.1.1g               h516909a_0    conda-forge

oset                      0.1.3                      py_1    conda-forge

packaging                 20.4               pyh9f0ad1d_0    conda-forge

pandas                    1.0.5            py38hcb8c335_0    conda-forge

pandoc                    2.10                          0    conda-forge

pandocfilters             1.4.2                      py_1    conda-forge

parso                     0.7.0              pyh9f0ad1d_0    conda-forge

patsy                     0.5.1                      py_0    conda-forge

pexpect                   4.8.0            py38h32f6830_1    conda-forge

pickleshare               0.7.5           py38h32f6830_1001    conda-forge

pip                       20.1.1                     py_1    conda-forge

plotly                    4.8.2              pyh9f0ad1d_0    conda-forge

pluggy                    0.13.1           py38h32f6830_2    conda-forge

prometheus_client         0.8.0              pyh9f0ad1d_0    conda-forge

prompt-toolkit            3.0.5                      py_1    conda-forge

psutil                    5.7.0            py38h1e0a361_1    conda-forge

ptyprocess                0.6.0                   py_1001    conda-forge

py                        1.9.0              pyh9f0ad1d_0    conda-forge

pybtex                    0.22.2           py38h32f6830_1    conda-forge

pybtex-docutils           0.2.2            py38h32f6830_1    conda-forge

pycparser                 2.20               pyh9f0ad1d_2    conda-forge

pygments                  2.6.1                      py_0    conda-forge

pyopenssl                 19.1.0                     py_1    conda-forge

pyparsing                 2.4.7              pyh9f0ad1d_0    conda-forge

pyrsistent                0.16.0           py38h1e0a361_0    conda-forge

pysocks                   1.7.1            py38h32f6830_1    conda-forge

pytest                    5.4.3            py38h32f6830_0    conda-forge

python                    3.8.3           cpython_he5300dc_0    conda-forge

python-dateutil           2.8.1                      py_0    conda-forge

python_abi                3.8                      1_cp38    conda-forge

pytz                      2020.1             pyh9f0ad1d_0    conda-forge

pyyaml                    5.3.1            py38h1e0a361_0    conda-forge

pyzmq                     19.0.1           py38ha71036d_0    conda-forge

readline                  8.0                  hf8c457e_0    conda-forge

requests                  2.24.0             pyh9f0ad1d_0    conda-forge

retrying                  1.3.3                      py_2    conda-forge

scikit-learn              0.23.1           py38h3a94b23_0    conda-forge

scipy                     1.5.0            py38h18bccfc_0    conda-forge

seaborn                   0.10.1                        1    conda-forge

seaborn-base              0.10.1                     py_1    conda-forge

send2trash                1.5.0                      py_0    conda-forge

setuptools                48.0.0           py38h32f6830_0    conda-forge

six                       1.15.0             pyh9f0ad1d_0    conda-forge

snowballstemmer           2.0.0                      py_0    conda-forge

sortedcontainers          2.2.2              pyh9f0ad1d_0    conda-forge

sphinx                    2.2.0                      py_0    conda-forge

sphinx-sitemap            1.0.2                    pypi_0    pypi

sphinx_rtd_theme          0.5.0              pyh9f0ad1d_0    conda-forge

sphinxcontrib-applehelp   1.0.2                      py_0    conda-forge

sphinxcontrib-bibtex      1.0.0                      py_0    conda-forge

sphinxcontrib-devhelp     1.0.2                      py_0    conda-forge

sphinxcontrib-htmlhelp    1.0.3                      py_0    conda-forge

sphinxcontrib-jsmath      1.0.1                      py_0    conda-forge

sphinxcontrib-qthelp      1.0.3                      py_0    conda-forge

sphinxcontrib-serializinghtml 1.1.4                      py_0    conda-forge

sqlite                    3.32.3               hcee41ef_0    conda-forge

statsmodels               0.11.1           py38h1e0a361_2    conda-forge

tabulate                  0.8.7              pyh9f0ad1d_0    conda-forge

terminado                 0.8.3            py38h32f6830_1    conda-forge

testpath                  0.4.4                      py_0    conda-forge

threadpoolctl             2.1.0              pyh5ca1d4c_0    conda-forge

tk                        8.6.10               hed695b0_0    conda-forge

tornado                   6.0.4            py38h1e0a361_1    conda-forge

tqdm                      4.47.0             pyh9f0ad1d_0    conda-forge

traitlets                 4.3.3            py38h32f6830_1    conda-forge

urllib3                   1.25.9                     py_0    conda-forge

wcwidth                   0.2.5              pyh9f0ad1d_0    conda-forge

webencodings              0.5.1                      py_1    conda-forge

wheel                     0.34.2                     py_1    conda-forge

widgetsnbextension        3.5.1                    py38_0    conda-forge

xz                        5.2.5                h516909a_0    conda-forge

yaml                      0.2.5                h516909a_0    conda-forge

zeromq                    4.3.2                he1b5a44_2    conda-forge

zipp                      3.1.0                      py_0    conda-forge

zlib                      1.2.11            h516909a_1006    conda-forge

[kain88-de]

OK. I had a little closer look. This could be a problem with our conda package. Did you check locally if the error occurs using the environment file and python 3.8? In #86 @IAlibay has switched to python 3.6. Worth a try if he sees the failure with astropy builds as well on 3.8.

[IAlibay]

I've tried bumping up the python version to 3.8 on the astropy build and Travis still runs fine. Similarly locally running under 3.8 also is still fine.

I'll let @lilyminium test out the environment build with 3.8 to see if the Travis failure with that keeps going.

[IAlibay]

Just as a follow up here, we have some indication that numpy might be at least partly involved in this issue: MDAnalysis/UserGuide#95

Particularly numpy 1.19. Since we enforce the stable version of numpy in the main MDA CI (1.18.5), I'm just testing out what happens if we pass 1.19.0 instead in #2861

[orbeckst]

Random comment from @richardjgowers MDAnalysis/pytng#29 (comment) (my highlighting)

A lot of methods you find won't work because of the memory being allocated by malloc/realloc/free, whereas memory in numpy is (by default) handled using PyMem_malloc which is a different system and mixing the two causes segfaults. I.e. you can't malloc then PyMem_Free. This is all to avoid doing copies of data, but I'd really like to keep the bindings tight else we're letting everything downstream down.

Is it possible that our Cython-based XDRReader does something somewhere where we are not careful with how we pass arrays around (and maybe something changed in that respect in numpy recently?)

Sorry, grasping at straws here...

P.S.: I cross-posted in MDAnalysis/UserGuide#95 (comment) because I couldn't find this issue at first.

[orbeckst]

Irrelevant today... closing.