Failed: DID NOT WARN && EOFError

[chy-2003]

I've installed MDAnalysis[analysis] and MDAnalysisTests with pip3. When I was running pytest --disable-pytest-warnings --pyargs MDAnalysisTests, 3 failures occurred. Here is the short test summary info:

============================================================================ short test summary info ============================================================================
FAILED coordinates/test_xdr.py::TestXTCReader_offsets::test_persistent_offsets_readonly - Failed: DID NOT WARN. No warnings of type (<class 'UserWarning'>,) were emitted.
FAILED coordinates/test_xdr.py::TestTRRReader_offsets::test_persistent_offsets_readonly - Failed: DID NOT WARN. No warnings of type (<class 'UserWarning'>,) were emitted.
FAILED core/test_universe.py::TestUniverseCreation::test_Universe_invalidpermissionfile_IE_msg - EOFError: Compressed file ended before the end-of-stream marker was reached
========================================= 3 failed, 18077 passed, 695 skipped, 6 xfailed, 2 xpassed, 8949 warnings in 727.06s (0:12:07) =========================================

I did the operations under MDAnalysis 2.4.2, Ubuntu 20.04, python 3.10.10, pip 23.0.1.
And the same problem just still happens under python3.8.1.

Following are detailed informations:

=================================================================================== FAILURES ====================================================================================
____________________________________________________________ TestXTCReader_offsets.test_persistent_offsets_readonly _____________________________________________________________

self = <MDAnalysisTests.coordinates.test_xdr.TestXTCReader_offsets object at 0x7f909bb54ac0>, tmpdir = local('/tmp/pytest-of-root/pytest-2/test_persistent_offsets_readon0')

    def test_persistent_offsets_readonly(self, tmpdir):
        shutil.copy(self.filename, str(tmpdir))

        if os.name == 'nt':
            # Windows platform has a unique way to deny write access
            subprocess.call("icacls {fname} /deny Users:W".format(fname=tmpdir),
                            shell=True)
        else:
            os.chmod(str(tmpdir), 0o555)

        filename = str(tmpdir.join(os.path.basename(self.filename)))
        # try to write a offsets file
>       with (pytest.warns(UserWarning, match="Couldn't save offsets") and
              pytest.warns(UserWarning, match="Cannot write")):
E             Failed: DID NOT WARN. No warnings of type (<class 'UserWarning'>,) were emitted.
E             The list of emitted warnings is: [].

/usr/local/lib/python3.10/dist-packages/MDAnalysisTests/coordinates/test_xdr.py:885: Failed
____________________________________________________________ TestTRRReader_offsets.test_persistent_offsets_readonly _____________________________________________________________

self = <MDAnalysisTests.coordinates.test_xdr.TestTRRReader_offsets object at 0x7f909bb55960>, tmpdir = local('/tmp/pytest-of-root/pytest-2/test_persistent_offsets_readon1')

    def test_persistent_offsets_readonly(self, tmpdir):
        shutil.copy(self.filename, str(tmpdir))

        if os.name == 'nt':
            # Windows platform has a unique way to deny write access
            subprocess.call("icacls {fname} /deny Users:W".format(fname=tmpdir),
                            shell=True)
        else:
            os.chmod(str(tmpdir), 0o555)

        filename = str(tmpdir.join(os.path.basename(self.filename)))
        # try to write a offsets file
>       with (pytest.warns(UserWarning, match="Couldn't save offsets") and
              pytest.warns(UserWarning, match="Cannot write")):
E             Failed: DID NOT WARN. No warnings of type (<class 'UserWarning'>,) were emitted.
E             The list of emitted warnings is: [].

/usr/local/lib/python3.10/dist-packages/MDAnalysisTests/coordinates/test_xdr.py:885: Failed
________________________________________________________ TestUniverseCreation.test_Universe_invalidpermissionfile_IE_msg ________________________________________________________

self = <MDAnalysisTests.core.test_universe.TestUniverseCreation object at 0x7f909ae30d00>, tmpdir = local('/tmp/pytest-of-root/pytest-2/test_Universe_invalidpermissio0')

    def test_Universe_invalidpermissionfile_IE_msg(self, tmpdir):
        # check for file with invalid permissions (eg. no read access)
        with tmpdir.as_cwd():
            temp_file = 'permission.denied.tpr'
            with open(temp_file, 'w'):
                pass

            if os.name == 'nt':
                subprocess.call("icacls {filename} /deny Users:RX".format(filename=temp_file),
                                shell=True)
            else:
                os.chmod(temp_file, 0o200)

            # Issue #3221 match by PermissionError and error number instead
            with pytest.raises(PermissionError, match=f"Errno {errno.EACCES}"):
>               mda.Universe('permission.denied.tpr')

/usr/local/lib/python3.10/dist-packages/MDAnalysisTests/core/test_universe.py:171:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/lib/python3.10/dist-packages/MDAnalysis/core/universe.py:346: in __init__
    topology = _topology_from_file_like(self.filename,
/usr/local/lib/python3.10/dist-packages/MDAnalysis/core/universe.py:110: in _topology_from_file_like
    topology = p.parse(**kwargs)
/usr/local/lib/python3.10/dist-packages/MDAnalysis/topology/TPRParser.py:202: in parse
    with openany(self.filename, mode='rb') as infile:
/usr/lib/python3.10/contextlib.py:135: in __enter__
    return next(self.gen)
/usr/local/lib/python3.10/dist-packages/MDAnalysis/lib/util.py:318: in openany
    stream = anyopen(datasource, mode=mode, reset=reset)
/usr/local/lib/python3.10/dist-packages/MDAnalysis/lib/util.py:397: in anyopen
    stream = _get_stream(datasource, openfunc, mode=mode)
/usr/local/lib/python3.10/dist-packages/MDAnalysis/lib/util.py:444: in _get_stream
    stream.readline()
/usr/lib/python3.10/bz2.py:198: in readline
    return self._buffer.readline(size)
/usr/lib/python3.10/_compression.py:68: in readinto
    data = self.read(len(byte_view))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <_compression.DecompressReader object at 0x7f9083c2bcd0>, size = 8192

    def read(self, size=-1):
        if size < 0:
            return self.readall()

        if not size or self._eof:
            return b""
        data = None  # Default if EOF is encountered
        # Depending on the input data, our call to the decompressor may not
        # return any data. In this case, try again after reading another block.
        while True:
            if self._decompressor.eof:
                rawblock = (self._decompressor.unused_data or
                            self._fp.read(BUFFER_SIZE))
                if not rawblock:
                    break
                # Continue to next stream.
                self._decompressor = self._decomp_factory(
                    **self._decomp_args)
                try:
                    data = self._decompressor.decompress(rawblock, size)
                except self._trailing_error:
                    # Trailing data isn't a valid compressed stream; ignore it.
                    break
            else:
                if self._decompressor.needs_input:
                    rawblock = self._fp.read(BUFFER_SIZE)
                    if not rawblock:
>                       raise EOFError("Compressed file ended before the "
                                       "end-of-stream marker was reached")
E                       EOFError: Compressed file ended before the end-of-stream marker was reached

/usr/lib/python3.10/_compression.py:99: EOFError

[IAlibay]

I can't reproduce this locally (nor did it show up on CI).

This looks like it might be related to a permission error, could you tell me more about the platform you are running tests on?

• How did you install your Python environment?
• Is it a pure Ubuntu installation?
• Are you on a shared machine?
• Do you have write permissions to /tmp?
• What type of CPU architecture are you using?

[chy-2003]

• I'm on an Elastic Cloud Server. System(Ubuntu20.04-server) is provided by host.
• Python3.8 is pre-installed. I installed Python3.10 by apt-get install python3.10, and set it as default by update-alternatives --install and update-alternatives --config.
• The only user on the computer is root. And I do have write permissions to /tmp in terminal.
• CPU: amd64

Before I tried to install mdanalysis on server, I'd successfully installed mda on Ubuntu22.04-desktop on my local vitural computer(Hyper-V, amd64), with python3.10. I guess there's something wrong with the pre-installed python3.8 and pip.

[IAlibay]

I would suspect this is an issue with the elastic cloud server? Possibly that the error type isn't being thrown isn't the way it would on a normal machine.

The only way to know for this would be to manually reproduce the conditions of the test and see what gets thrown in a python shell.

[chy-2003]

That's really a lot of work to do. I'll try my best to figure it out.
I've just done something terrible. Now problems like No module named 'apt_pkg' appear which cannot be fixed by reinstall and ln command. I'm crazy.

[chy-2003]

Finally, I've installed mda successfully. But the problem remains unknown. I did following operations:

• uninstall all the things about python and pip (because I mistakenly broke the pip)
• reinstall python3.8.5 and pip3(now there's only python3.8 in the machine)
• add 127.0.0.1 <my_CT-ECS_servername> to /etc/hosts(to remove annoying warnings when using sudo)
• create a normal user
• install mda through the normal user, not root(commands without prefix 'sudo')

I've tried using python3.10 and python3.8, and I've tried reinstall them before. So it turns out that root cannot install mda in my case. I got this idea because not considering the problems caused by ECS, the only difference between two machines is root and normal user.

Following are my assumptions:

• Access problems caused by ECS and root together.
• pip install should not be called by root. This may mess python packages and system packages up.

[IAlibay]

Re-opening issue, I've been able to reproduce the issue on an aarch64 VM build using CirrusCI. See: https://cirrus-ci.com/task/6205116001288192?logs=ci#L639

[IAlibay]

So I can confirm that the issue here is related to running the tests as root - essentially you can't have a readonly file or a file that yields a PermissionError if you have root permissions (well at least, not using normal, and safe, chmod-like means).

For the sake of sanity we probably can just skip those specific tests and assume that everything else passes then we're good. However in practice, you probably don't want to run MDAnalysis as root unless you really have to (which I know docker probably forces you to).