test failures on s390x #105

rathann · 2020-05-31T21:53:10Z

I'm getting 11 failed tests on Fedora 32 on s390x architecture. s390x is big-endian, could this be the reason?

$ pytest  --last-failed
================================================= test session starts ==================================================
platform linux -- Python 3.8.3, pytest-4.6.9, py-1.8.0, pluggy-0.13.0
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/builddir/build/BUILD/zstandard-0.13.0/.hypothesis/examples')
rootdir: /builddir/build/BUILD/zstandard-0.13.0
plugins: xdist-1.31.0, forked-1.1.1, hypothesis-4.23.8
collected 341 items / 330 deselected / 11 selected                                                                     
run-last-failure: rerun previous 11 failures (skipped 137 files)

tests/test_compressor.py FFFF                                                                                    [ 36%]
tests/test_decompressor.py FFFFFFF                                                                               [100%]

======================================================= FAILURES =======================================================
__________________________________ TestCompressor_stream_reader.test_constant_methods __________________________________

self = <tests.test_compressor.TestCompressor_stream_reader testMethod=test_constant_methods>

    def test_constant_methods(self):
        cctx = zstd.ZstdCompressor()
    
        with cctx.stream_reader(b"boo") as reader:
            self.assertTrue(reader.readable())
            self.assertFalse(reader.writable())
            self.assertFalse(reader.seekable())
            self.assertFalse(reader.isatty())
            self.assertFalse(reader.closed)
            self.assertIsNone(reader.flush())
            self.assertFalse(reader.closed)
    
>       self.assertTrue(reader.closed)
E       AssertionError: False is not true

tests/test_compressor.py:638: AssertionError
_________________________________ TestCompressor_stream_reader.test_no_context_manager _________________________________

self = <tests.test_compressor.TestCompressor_stream_reader testMethod=test_no_context_manager>

    def test_no_context_manager(self):
        cctx = zstd.ZstdCompressor()
    
        reader = cctx.stream_reader(b"foo")
        reader.read(4)
        self.assertFalse(reader.closed)
    
        reader.close()
>       self.assertTrue(reader.closed)
E       AssertionError: False is not true

tests/test_compressor.py:600: AssertionError
____________________________________ TestCompressor_stream_reader.test_read_closed _____________________________________

self = <tests.test_compressor.TestCompressor_stream_reader testMethod=test_read_closed>

    def test_read_closed(self):
        cctx = zstd.ZstdCompressor()
    
        with cctx.stream_reader(b"foo" * 60) as reader:
            reader.close()
>           self.assertTrue(reader.closed)
E           AssertionError: False is not true

tests/test_compressor.py:645: AssertionError
_______________________________________ TestCompressor_stream_writer.test_close ________________________________________

self = <tests.test_compressor.TestCompressor_stream_writer testMethod=test_close>

    def test_close(self):
        buffer = NonClosingBytesIO()
        cctx = zstd.ZstdCompressor(level=1)
        writer = cctx.stream_writer(buffer)
    
        writer.write(b"foo" * 1024)
        self.assertFalse(writer.closed)
        self.assertFalse(buffer.closed)
        writer.close()
>       self.assertTrue(writer.closed)
E       AssertionError: False is not true

tests/test_compressor.py:922: AssertionError
_________________________________ TestDecompressor_stream_reader.test_constant_methods _________________________________

self = <tests.test_decompressor.TestDecompressor_stream_reader testMethod=test_constant_methods>

    def test_constant_methods(self):
        dctx = zstd.ZstdDecompressor()
    
        with dctx.stream_reader(b"foo") as reader:
            self.assertFalse(reader.closed)
            self.assertTrue(reader.readable())
            self.assertFalse(reader.writable())
            self.assertTrue(reader.seekable())
            self.assertFalse(reader.isatty())
            self.assertFalse(reader.closed)
            self.assertIsNone(reader.flush())
            self.assertFalse(reader.closed)
    
>       self.assertTrue(reader.closed)
E       AssertionError: False is not true

tests/test_decompressor.py:351: AssertionError
________________________________ TestDecompressor_stream_reader.test_no_context_manager ________________________________

self = <tests.test_decompressor.TestDecompressor_stream_reader testMethod=test_no_context_manager>

    def test_no_context_manager(self):
        source = b"foobar" * 60
        cctx = zstd.ZstdCompressor()
        frame = cctx.compress(source)
    
        dctx = zstd.ZstdDecompressor()
        reader = dctx.stream_reader(frame)
    
        self.assertEqual(reader.read(6), b"foobar")
        self.assertEqual(reader.read(18), b"foobar" * 3)
        self.assertFalse(reader.closed)
    
        # Calling close prevents subsequent use.
        reader.close()
>       self.assertTrue(reader.closed)
E       AssertionError: False is not true

tests/test_decompressor.py:533: AssertionError
_________________________________ TestDecompressor_stream_reader.test_read_after_exit __________________________________

self = <tests.test_decompressor.TestDecompressor_stream_reader testMethod=test_read_after_exit>

    def test_read_after_exit(self):
        cctx = zstd.ZstdCompressor()
        frame = cctx.compress(b"foo" * 60)
    
        dctx = zstd.ZstdDecompressor()
    
        with dctx.stream_reader(frame) as reader:
            while reader.read(16):
                pass
    
>       self.assertTrue(reader.closed)
E       AssertionError: False is not true

tests/test_decompressor.py:465: AssertionError
___________________________________ TestDecompressor_stream_reader.test_read_buffer ____________________________________

self = <tests.test_decompressor.TestDecompressor_stream_reader testMethod=test_read_buffer>

    def test_read_buffer(self):
        cctx = zstd.ZstdCompressor()
    
        source = b"".join([b"foo" * 60, b"bar" * 60, b"baz" * 60])
        frame = cctx.compress(source)
    
        dctx = zstd.ZstdDecompressor()
    
        with dctx.stream_reader(frame) as reader:
            self.assertEqual(reader.tell(), 0)
    
            # We should get entire frame in one read.
            result = reader.read(8192)
            self.assertEqual(result, source)
            self.assertEqual(reader.tell(), len(source))
    
            # Read after EOF should return empty bytes.
            self.assertEqual(reader.read(1), b"")
            self.assertEqual(reader.tell(), len(result))
    
>       self.assertTrue(reader.closed)
E       AssertionError: False is not true

tests/test_decompressor.py:397: AssertionError
___________________________________ TestDecompressor_stream_reader.test_read_closed ____________________________________

self = <tests.test_decompressor.TestDecompressor_stream_reader testMethod=test_read_closed>

    def test_read_closed(self):
        dctx = zstd.ZstdDecompressor()
    
        with dctx.stream_reader(b"foo") as reader:
            reader.close()
>           self.assertTrue(reader.closed)
E           AssertionError: False is not true

tests/test_decompressor.py:358: AssertionError
___________________________________ TestDecompressor_stream_reader.test_read_stream ____________________________________

self = <tests.test_decompressor.TestDecompressor_stream_reader testMethod=test_read_stream>

    def test_read_stream(self):
        cctx = zstd.ZstdCompressor()
        source = b"".join([b"foo" * 60, b"bar" * 60, b"baz" * 60])
        frame = cctx.compress(source)
    
        dctx = zstd.ZstdDecompressor()
        with dctx.stream_reader(io.BytesIO(frame)) as reader:
            self.assertEqual(reader.tell(), 0)
    
            chunk = reader.read(8192)
            self.assertEqual(chunk, source)
            self.assertEqual(reader.tell(), len(source))
            self.assertEqual(reader.read(1), b"")
            self.assertEqual(reader.tell(), len(source))
            self.assertFalse(reader.closed)
    
>       self.assertTrue(reader.closed)
E       AssertionError: False is not true

tests/test_decompressor.py:434: AssertionError
______________________________________ TestDecompressor_stream_writer.test_close _______________________________________

self = <tests.test_decompressor.TestDecompressor_stream_writer testMethod=test_close>

    def test_close(self):
        foo = zstd.ZstdCompressor().compress(b"foo")
    
        buffer = NonClosingBytesIO()
        dctx = zstd.ZstdDecompressor()
        writer = dctx.stream_writer(buffer)
    
        writer.write(foo)
        self.assertFalse(writer.closed)
        self.assertFalse(buffer.closed)
        writer.close()
>       self.assertTrue(writer.closed)
E       AssertionError: False is not true

tests/test_decompressor.py:944: AssertionError
====================================== 11 failed, 330 deselected in 0.32 seconds =======================================
$

The text was updated successfully, but these errors were encountered:

indygreg · 2020-06-13T17:53:50Z

If this were an endian issue, it would be a very interesting manifestation!

Looking at the code, this is potentially due to uninitialized memory. But it would be very strange that this isn't occurring in other environments!

Are you using the system Python build or a custom one? Any chance you could build Python from source (perhaps via pyenv) and try to reproduce with that?

This memory should be initialized to NULL, which I thought would be a 0 everywhere. However, #105 reports tests failing and the only explanation I could come up with is that these int fields are never explicitly set to 0. This commit explicitly sets these fields to 0 to possibly fix this issue.

indygreg · 2020-06-13T18:05:35Z

I just pushed a commit that may fix this. Could you please test and see if it does?

rathann · 2020-06-14T22:53:10Z

I've just tested it and there's no change in the test results. The same 11 tests are still failing with the same assertion errors. I am using system python. I don't know when I'll have some time to build Python from source, but I'll let you know if running under vanilla Python changes anything here.

rathann · 2021-02-03T18:04:33Z

With 0.15.1, there are 24 failures now:

======================================================= FAILURES =======================================================
_______________________________________ TestCompressor_stream_reader.test_close ________________________________________
[gw3] linux -- Python 3.9.1 /usr/bin/python3

self = <tests.test_compressor.TestCompressor_stream_reader testMethod=test_close>

    def test_close(self):
        buffer = NonClosingBytesIO(b"foo" * 1024)
        cctx = zstd.ZstdCompressor()
        reader = cctx.stream_reader(buffer)
    
        reader.read(3)
        self.assertFalse(reader.closed)
        self.assertFalse(buffer.closed)
        reader.close()
>       self.assertTrue(reader.closed)
E       AssertionError: False is not true

tests/test_compressor.py:871: AssertionError
_________________________________ TestCompressor_stream_reader.test_no_context_manager _________________________________
[gw3] linux -- Python 3.9.1 /usr/bin/python3

self = <tests.test_compressor.TestCompressor_stream_reader testMethod=test_no_context_manager>

    def test_no_context_manager(self):
        cctx = zstd.ZstdCompressor()
    
        reader = cctx.stream_reader(b"foo")
        reader.read(4)
        self.assertFalse(reader.closed)
    
        reader.close()
>       self.assertTrue(reader.closed)
E       AssertionError: False is not true

tests/test_compressor.py:625: AssertionError
_______________________________________ TestCompressor_stream_writer.test_close ________________________________________
[gw3] linux -- Python 3.9.1 /usr/bin/python3

self = <tests.test_compressor.TestCompressor_stream_writer testMethod=test_close>

    def test_close(self):
        buffer = NonClosingBytesIO()
        cctx = zstd.ZstdCompressor(level=1)
        writer = cctx.stream_writer(buffer)
    
        writer.write(b"foo" * 1024)
        self.assertFalse(writer.closed)
        self.assertFalse(buffer.closed)
        writer.close()
>       self.assertTrue(writer.closed)
E       AssertionError: False is not true

tests/test_compressor.py:1057: AssertionError
__________________________________ TestCompressor_stream_reader.test_constant_methods __________________________________
[gw0] linux -- Python 3.9.1 /usr/bin/python3

self = <tests.test_compressor.TestCompressor_stream_reader testMethod=test_constant_methods>

    def test_constant_methods(self):
        cctx = zstd.ZstdCompressor()
    
        with cctx.stream_reader(b"boo") as reader:
            self.assertTrue(reader.readable())
            self.assertFalse(reader.writable())
            self.assertFalse(reader.seekable())
            self.assertFalse(reader.isatty())
            self.assertFalse(reader.closed)
            self.assertIsNone(reader.flush())
            self.assertFalse(reader.closed)
    
>       self.assertTrue(reader.closed)
E       AssertionError: False is not true

tests/test_compressor.py:663: AssertionError
____________________________________ TestCompressor_stream_reader.test_read_closed _____________________________________
[gw0] linux -- Python 3.9.1 /usr/bin/python3

self = <tests.test_compressor.TestCompressor_stream_reader testMethod=test_read_closed>

    def test_read_closed(self):
        cctx = zstd.ZstdCompressor()
    
        with cctx.stream_reader(b"foo" * 60) as reader:
            reader.close()
>           self.assertTrue(reader.closed)
E           AssertionError: False is not true

tests/test_compressor.py:670: AssertionError
________________________________ TestCompressor_stream_reader.test_close_closefd_false _________________________________
[gw1] linux -- Python 3.9.1 /usr/bin/python3

self = <tests.test_compressor.TestCompressor_stream_reader testMethod=test_close_closefd_false>

    def test_close_closefd_false(self):
        buffer = NonClosingBytesIO(b"foo" * 1024)
        cctx = zstd.ZstdCompressor()
        reader = cctx.stream_reader(buffer, closefd=False)
    
        reader.read(3)
        self.assertFalse(reader.closed)
        self.assertFalse(buffer.closed)
        reader.close()
>       self.assertTrue(reader.closed)
E       AssertionError: False is not true

tests/test_compressor.py:919: AssertionError
________________________________ TestCompressor_stream_writer.test_close_closefd_false _________________________________
[gw1] linux -- Python 3.9.1 /usr/bin/python3

self = <tests.test_compressor.TestCompressor_stream_writer testMethod=test_close_closefd_false>

    def test_close_closefd_false(self):
        buffer = io.BytesIO()
        cctx = zstd.ZstdCompressor(level=1)
        writer = cctx.stream_writer(buffer, closefd=False)
    
        writer.write(b"foo" * 1024)
        self.assertFalse(writer.closed)
        self.assertFalse(buffer.closed)
        writer.close()
>       self.assertTrue(writer.closed)
E       AssertionError: False is not true

tests/test_compressor.py:1109: AssertionError
______________________________________ TestDecompressor_stream_reader.test_close _______________________________________
[gw0] linux -- Python 3.9.1 /usr/bin/python3

self = <tests.test_decompressor.TestDecompressor_stream_reader testMethod=test_close>

    def test_close(self):
        foo = zstd.ZstdCompressor().compress(b"foo" * 1024)
    
        buffer = io.BytesIO(foo)
        dctx = zstd.ZstdDecompressor()
        reader = dctx.stream_reader(buffer)
    
        reader.read(3)
        self.assertFalse(reader.closed)
        self.assertFalse(buffer.closed)
        reader.close()
>       self.assertTrue(reader.closed)
E       AssertionError: False is not true

tests/test_decompressor.py:489: AssertionError
_______________________________ TestDecompressor_stream_reader.test_close_closefd_false ________________________________
[gw0] linux -- Python 3.9.1 /usr/bin/python3

self = <tests.test_decompressor.TestDecompressor_stream_reader testMethod=test_close_closefd_false>

    def test_close_closefd_false(self):
        foo = zstd.ZstdCompressor().compress(b"foo" * 1024)
    
        buffer = io.BytesIO(foo)
        dctx = zstd.ZstdDecompressor()
        reader = dctx.stream_reader(buffer, closefd=False)
    
        reader.read(3)
        self.assertFalse(reader.closed)
        self.assertFalse(buffer.closed)
        reader.close()
>       self.assertTrue(reader.closed)
E       AssertionError: False is not true

tests/test_decompressor.py:539: AssertionError
_________________________________ TestDecompressor_stream_reader.test_constant_methods _________________________________
[gw0] linux -- Python 3.9.1 /usr/bin/python3

self = <tests.test_decompressor.TestDecompressor_stream_reader testMethod=test_constant_methods>

    def test_constant_methods(self):
        dctx = zstd.ZstdDecompressor()
    
        with dctx.stream_reader(b"foo") as reader:
            self.assertFalse(reader.closed)
            self.assertTrue(reader.readable())
            self.assertFalse(reader.writable())
            self.assertFalse(reader.seekable())
            self.assertFalse(reader.isatty())
            self.assertFalse(reader.closed)
            self.assertIsNone(reader.flush())
            self.assertFalse(reader.closed)
    
>       self.assertTrue(reader.closed)
E       AssertionError: False is not true

tests/test_decompressor.py:374: AssertionError
________________________________ TestDecompressor_stream_reader.test_no_context_manager ________________________________
[gw0] linux -- Python 3.9.1 /usr/bin/python3

self = <tests.test_decompressor.TestDecompressor_stream_reader testMethod=test_no_context_manager>

    def test_no_context_manager(self):
        source = b"foobar" * 60
        cctx = zstd.ZstdCompressor()
        frame = cctx.compress(source)
    
        dctx = zstd.ZstdDecompressor()
        reader = dctx.stream_reader(frame)
    
        self.assertEqual(reader.read(6), b"foobar")
        self.assertEqual(reader.read(18), b"foobar" * 3)
        self.assertFalse(reader.closed)
    
        # Calling close prevents subsequent use.
        reader.close()
>       self.assertTrue(reader.closed)
E       AssertionError: False is not true

tests/test_decompressor.py:659: AssertionError
_________________________________ TestDecompressor_stream_reader.test_read_after_exit __________________________________
[gw0] linux -- Python 3.9.1 /usr/bin/python3

self = <tests.test_decompressor.TestDecompressor_stream_reader testMethod=test_read_after_exit>

    def test_read_after_exit(self):
        cctx = zstd.ZstdCompressor()
        frame = cctx.compress(b"foo" * 60)
    
        dctx = zstd.ZstdDecompressor()
    
        with dctx.stream_reader(frame) as reader:
            while reader.read(16):
                pass
    
>       self.assertTrue(reader.closed)
E       AssertionError: False is not true

tests/test_decompressor.py:588: AssertionError
___________________________________ TestDecompressor_stream_reader.test_read_buffer ____________________________________
[gw0] linux -- Python 3.9.1 /usr/bin/python3

self = <tests.test_decompressor.TestDecompressor_stream_reader testMethod=test_read_buffer>

    def test_read_buffer(self):
        cctx = zstd.ZstdCompressor()
    
        source = b"".join([b"foo" * 60, b"bar" * 60, b"baz" * 60])
        frame = cctx.compress(source)
    
        dctx = zstd.ZstdDecompressor()
    
        with dctx.stream_reader(frame) as reader:
            self.assertEqual(reader.tell(), 0)
    
            # We should get entire frame in one read.
            result = reader.read(8192)
            self.assertEqual(result, source)
            self.assertEqual(reader.tell(), len(source))
    
            # Read after EOF should return empty bytes.
            self.assertEqual(reader.read(1), b"")
            self.assertEqual(reader.tell(), len(result))
    
>       self.assertTrue(reader.closed)
E       AssertionError: False is not true

tests/test_decompressor.py:420: AssertionError
___________________________________ TestDecompressor_stream_reader.test_read_closed ____________________________________
[gw0] linux -- Python 3.9.1 /usr/bin/python3

self = <tests.test_decompressor.TestDecompressor_stream_reader testMethod=test_read_closed>

    def test_read_closed(self):
        dctx = zstd.ZstdDecompressor()
    
        with dctx.stream_reader(b"foo") as reader:
            reader.close()
>           self.assertTrue(reader.closed)
E           AssertionError: False is not true

tests/test_decompressor.py:381: AssertionError
___________________________________ TestDecompressor_stream_reader.test_read_stream ____________________________________
[gw0] linux -- Python 3.9.1 /usr/bin/python3

self = <tests.test_decompressor.TestDecompressor_stream_reader testMethod=test_read_stream>

    def test_read_stream(self):
        cctx = zstd.ZstdCompressor()
        source = b"".join([b"foo" * 60, b"bar" * 60, b"baz" * 60])
        frame = cctx.compress(source)
    
        dctx = zstd.ZstdDecompressor()
        with dctx.stream_reader(io.BytesIO(frame)) as reader:
            self.assertEqual(reader.tell(), 0)
    
            chunk = reader.read(8192)
            self.assertEqual(chunk, source)
            self.assertEqual(reader.tell(), len(source))
            self.assertEqual(reader.read(1), b"")
            self.assertEqual(reader.tell(), len(source))
            self.assertFalse(reader.closed)
    
>       self.assertTrue(reader.closed)
E       AssertionError: False is not true

tests/test_decompressor.py:457: AssertionError
______________________________________ TestDecompressor_stream_writer.test_close _______________________________________
[gw0] linux -- Python 3.9.1 /usr/bin/python3

self = <tests.test_decompressor.TestDecompressor_stream_writer testMethod=test_close>

    def test_close(self):
        foo = zstd.ZstdCompressor().compress(b"foo")
    
        buffer = NonClosingBytesIO()
        dctx = zstd.ZstdDecompressor()
        writer = dctx.stream_writer(buffer)
    
        writer.write(foo)
        self.assertFalse(writer.closed)
        self.assertFalse(buffer.closed)
        writer.close()
>       self.assertTrue(writer.closed)
E       AssertionError: False is not true

tests/test_decompressor.py:1072: AssertionError
_______________________________ TestDecompressor_stream_writer.test_close_closefd_false ________________________________
[gw0] linux -- Python 3.9.1 /usr/bin/python3

self = <tests.test_decompressor.TestDecompressor_stream_writer testMethod=test_close_closefd_false>

    def test_close_closefd_false(self):
        foo = zstd.ZstdCompressor().compress(b"foo")
    
        buffer = NonClosingBytesIO()
        dctx = zstd.ZstdDecompressor()
        writer = dctx.stream_writer(buffer, closefd=False)
    
        writer.write(foo)
        self.assertFalse(writer.closed)
        self.assertFalse(buffer.closed)
        writer.close()
>       self.assertTrue(writer.closed)
E       AssertionError: False is not true

tests/test_decompressor.py:1122: AssertionError
__________________________________________ TestOpen.test_read_binary_filename __________________________________________
[gw1] linux -- Python 3.9.1 /usr/bin/python3

self = <tests.test_open.TestOpen testMethod=test_read_binary_filename>

    def test_read_binary_filename(self):
        with tempfile.TemporaryDirectory() as td:
            p = os.path.join(td, "testfile")
            with open(p, "wb") as fh:
                cctx = zstd.ZstdCompressor()
                fh.write(cctx.compress(b"foo" * 1024))
    
            fh = zstd.open(p, "rb")
    
            self.assertEqual(fh.read(6), b"foofoo")
            self.assertEqual(len(fh.read()), 1024 * 3 - 6)
            self.assertFalse(fh.closed)
    
            fh.close()
>           self.assertTrue(fh.closed)
E           AssertionError: False is not true

tests/test_open.py:91: AssertionError
__________________________________________ TestOpen.test_read_binary_fileobj ___________________________________________
[gw1] linux -- Python 3.9.1 /usr/bin/python3

self = <tests.test_open.TestOpen testMethod=test_read_binary_fileobj>

    def test_read_binary_fileobj(self):
        cctx = zstd.ZstdCompressor()
        buffer = io.BytesIO(cctx.compress(b"foo" * 1024))
    
        fh = zstd.open(buffer, "rb")
    
        self.assertEqual(fh.read(6), b"foofoo")
        self.assertFalse(fh.closed)
        self.assertFalse(buffer.closed)
    
        fh.close()
>       self.assertTrue(fh.closed)
E       AssertionError: False is not true

tests/test_open.py:66: AssertionError
___________________________________________ TestOpen.test_read_text_filename ___________________________________________
[gw1] linux -- Python 3.9.1 /usr/bin/python3

self = <tests.test_open.TestOpen testMethod=test_read_text_filename>

    def test_read_text_filename(self):
        with tempfile.TemporaryDirectory() as td:
            p = os.path.join(td, "testfile")
            cctx = zstd.ZstdCompressor()
            with open(p, "wb") as fh:
                fh.write(cctx.compress(b"foo\n" * 1024))
    
            fh = zstd.open(p, "r")
    
            self.assertEqual(fh.read(4), "foo\n")
            self.assertEqual(fh.readline(), "foo\n")
            self.assertFalse(fh.closed)
    
            fh.close()
>           self.assertTrue(fh.closed)
E           AssertionError: False is not true

tests/test_open.py:120: AssertionError
___________________________________________ TestOpen.test_read_text_fileobj ____________________________________________
[gw1] linux -- Python 3.9.1 /usr/bin/python3

self = <tests.test_open.TestOpen testMethod=test_read_text_fileobj>

    def test_read_text_fileobj(self):
        cctx = zstd.ZstdCompressor()
        buffer = io.BytesIO(cctx.compress(b"foo\n" * 1024))
    
        fh = zstd.open(buffer, "r")
        self.assertIsInstance(fh, io.TextIOWrapper)
    
        self.assertEqual(fh.readline(), "foo\n")
    
        fh.close()
>       self.assertTrue(fh.closed)
E       AssertionError: False is not true

tests/test_open.py:103: AssertionError
_________________________________________ TestOpen.test_write_binary_filename __________________________________________
[gw1] linux -- Python 3.9.1 /usr/bin/python3

self = <tests.test_open.TestOpen testMethod=test_write_binary_filename>

    def test_write_binary_filename(self):
        with tempfile.TemporaryDirectory() as td:
            p = os.path.join(td, "testfile")
    
            fh = zstd.open(p, "wb")
            fh.write(b"foo" * 1024)
            self.assertFalse(fh.closed)
    
            fh.close()
>           self.assertTrue(fh.closed)
E           AssertionError: False is not true

tests/test_open.py:31: AssertionError
__________________________________________ TestOpen.test_write_binary_fileobj __________________________________________
[gw1] linux -- Python 3.9.1 /usr/bin/python3

self = <tests.test_open.TestOpen testMethod=test_write_binary_fileobj>

    def test_write_binary_fileobj(self):
        buffer = io.BytesIO()
    
        fh = zstd.open(buffer, "wb")
        fh.write(b"foo" * 1024)
        self.assertFalse(fh.closed)
        self.assertFalse(buffer.closed)
    
        fh.close()
>       self.assertTrue(fh.closed)
E       AssertionError: False is not true

tests/test_open.py:19: AssertionError
__________________________________________ TestOpen.test_write_text_filename ___________________________________________
[gw1] linux -- Python 3.9.1 /usr/bin/python3

self = <tests.test_open.TestOpen testMethod=test_write_text_filename>

    def test_write_text_filename(self):
        with tempfile.TemporaryDirectory() as td:
            p = os.path.join(td, "testfile")
    
            fh = zstd.open(p, "w")
            self.assertIsInstance(fh, io.TextIOWrapper)
    
            fh.write("foo\n")
            fh.write("bar\n")
            fh.close()
>           self.assertTrue(fh.closed)
E           AssertionError: False is not true

tests/test_open.py:50: AssertionError

ghost · 2021-03-01T02:43:36Z

In ZstdCompressionWriter struct, closed is int.

typedef struct {
    PyObject_HEAD
    ...
    int closed;
    ...
} ZstdCompressionWriter;

But in PyMemberDef, it's T_BOOL, which means the type is char.
See https://docs.python.org/3/c-api/structures.html#c.PyMemberDef

static PyMemberDef ZstdCompressionWriter_members[] = {
    {"closed", T_BOOL, offsetof(ZstdCompressionWriter, closed), READONLY, NULL},
    {NULL}};

This should fix this problem.

 static PyMemberDef ZstdCompressionWriter_members[] = {
-    {"closed", T_BOOL, offsetof(ZstdCompressionWriter, closed), READONLY, NULL},
+    {"closed", T_INT, offsetof(ZstdCompressionWriter, closed), READONLY, NULL},
     {NULL}};

Or change the closed to char.

rathann · 2021-03-01T13:05:33Z

Great catch, @animalize . The following patch makes all tests pass on s390x:

diff -up zstandard-0.15.1/c-ext/compressionreader.c.orig zstandard-0.15.1/c-ext/compressionreader.c
--- zstandard-0.15.1/c-ext/compressionreader.c.orig     2020-12-31 23:18:06.000000000 +0000
+++ zstandard-0.15.1/c-ext/compressionreader.c  2021-03-01 11:41:09.056060857 +0000
@@ -793,7 +793,7 @@ static PyMethodDef compressionreader_met
     {NULL, NULL}};
 
 static PyMemberDef compressionreader_members[] = {
-    {"closed", T_BOOL, offsetof(ZstdCompressionReader, closed), READONLY,
+    {"closed", T_INT, offsetof(ZstdCompressionReader, closed), READONLY,
      "whether stream is closed"},
     {NULL}};
 
diff -up zstandard-0.15.1/c-ext/decompressionreader.c.orig zstandard-0.15.1/c-ext/decompressionreader.c
--- zstandard-0.15.1/c-ext/decompressionreader.c.orig   2020-12-31 23:18:06.000000000 +0000
+++ zstandard-0.15.1/c-ext/decompressionreader.c        2021-03-01 11:40:35.686060857 +0000
@@ -761,7 +761,7 @@ static PyMethodDef decompressionreader_m
     {NULL, NULL}};
 
 static PyMemberDef decompressionreader_members[] = {
-    {"closed", T_BOOL, offsetof(ZstdDecompressionReader, closed), READONLY,
+    {"closed", T_INT, offsetof(ZstdDecompressionReader, closed), READONLY,
      "whether stream is closed"},
     {NULL}};
 
diff -up zstandard-0.15.1/c-ext/compressionwriter.c.orig zstandard-0.15.1/c-ext/compressionwriter.c
--- zstandard-0.15.1/c-ext/compressionwriter.c.orig     2020-12-31 23:18:06.000000000 +0000
+++ zstandard-0.15.1/c-ext/compressionwriter.c  2021-03-01 11:50:29.546150542 +0000
@@ -323,7 +323,7 @@ static PyMethodDef ZstdCompressionWriter
     {NULL, NULL}};
 
 static PyMemberDef ZstdCompressionWriter_members[] = {
-    {"closed", T_BOOL, offsetof(ZstdCompressionWriter, closed), READONLY, NULL},
+    {"closed", T_INT, offsetof(ZstdCompressionWriter, closed), READONLY, NULL},
     {NULL}};
 
 PyTypeObject ZstdCompressionWriterType = {
diff -up zstandard-0.15.1/c-ext/decompressionwriter.c.orig zstandard-0.15.1/c-ext/decompressionwriter.c
--- zstandard-0.15.1/c-ext/decompressionwriter.c.orig   2020-12-31 23:18:06.000000000 +0000
+++ zstandard-0.15.1/c-ext/decompressionwriter.c        2021-03-01 11:49:31.386150542 +0000
@@ -243,7 +243,7 @@ static PyMethodDef ZstdDecompressionWrit
     {NULL, NULL}};
 
 static PyMemberDef ZstdDecompressionWriter_members[] = {
-    {"closed", T_BOOL, offsetof(ZstdDecompressionWriter, closed), READONLY,
+    {"closed", T_INT, offsetof(ZstdDecompressionWriter, closed), READONLY,
      NULL},
     {NULL}};

ghost · 2021-03-01T13:56:27Z

Thanks for verification.

After thinking about it, I think change closed to char is better, the attribute is a boolean value.

    case T_BOOL:
        v = PyBool_FromLong(*(char*)addr);
        break;
    ...
    case T_INT:
        v = PyLong_FromLong(*(int*)addr);
        break;

rathann · 2021-03-01T15:18:33Z

It gets assigned values 0 and 1, so I guess it doesn't matter much, as long as the type is used consistently everywhere. I think char is signed on s390x and unsigned on others.

ghost · 2021-03-02T04:37:19Z

Yes, it doesn't matter much in this case.

I searched, char is up to the implementation.
In msvc, char is signed on x86.
In clang, char is signed on x86, but it is unsigned on an arm target.
https://stackoverflow.com/questions/2054939/is-char-signed-or-unsigned-by-default

ghost · 2021-03-02T04:48:05Z

It's up to compiler, this article tested gcc on these machines:
https://trofi.github.io/posts/203-signed-char-or-unsigned-char.html

signed: alpha, hppa, x86, ia64, m68k, mips, sh, sparc
unsigned: arm, powerpc, s390

skriesch · 2021-07-14T17:06:44Z

We have got same issues on openSUSE Tumbleweed.

In Zstd(De)?Compression(Reader|Writer) structs, `closed` is declared as `int`: ```C typedef struct { PyObject_HEAD ... int closed; ... } ZstdCompressionWriter; ``` but in [`PyMemberDef`](https://docs.python.org/3/c-api/structures.html#c.PyMemberDef), it's `T_BOOL`: ```C static PyMemberDef ZstdCompressionWriter_members[] = { {"closed", T_BOOL, offsetof(ZstdCompressionWriter, closed), READONLY, NULL}, {NULL}}; ``` which is `char`. This fixes regression tests on s390x (#105). Closes #164.

indygreg · 2022-01-19T01:15:29Z

#164 merged yesterday and fixes the int/char mismatch. Does someone with an impacted machine want to verify that this issue is resolved?

ghost · 2022-01-19T02:23:53Z

You may run tests on s390x with uraimo/run-on-arch-action GitHub action.

rathann · 2022-01-19T08:11:37Z

FWIW, the Fedora package has been carrying the merged patch since October last year and s390x builds had their test suites pass.

skriesch · 2022-01-30T18:25:01Z

I have tested the version 0.17.0 in OBS with openSUSE Tumbleweed. I do not receive any error any more for s390x.

indygreg added the bug label Jun 13, 2020

rathann changed the title ~~11 test failures on s390x~~ test failures on s390x Feb 3, 2021

rathann mentioned this issue Oct 18, 2021

fix inconsistent type for closed attribute #164

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test failures on s390x #105

test failures on s390x #105

rathann commented May 31, 2020

indygreg commented Jun 13, 2020

indygreg commented Jun 13, 2020

rathann commented Jun 14, 2020

rathann commented Feb 3, 2021

ghost commented Mar 1, 2021 •

edited by ghost

Loading

rathann commented Mar 1, 2021

ghost commented Mar 1, 2021

rathann commented Mar 1, 2021

ghost commented Mar 2, 2021 •

edited by ghost

Loading

ghost commented Mar 2, 2021

skriesch commented Jul 14, 2021

indygreg commented Jan 19, 2022

ghost commented Jan 19, 2022

rathann commented Jan 19, 2022

skriesch commented Jan 30, 2022

test failures on s390x #105

test failures on s390x #105

Comments

rathann commented May 31, 2020

indygreg commented Jun 13, 2020

indygreg commented Jun 13, 2020

rathann commented Jun 14, 2020

rathann commented Feb 3, 2021

ghost commented Mar 1, 2021 • edited by ghost Loading

rathann commented Mar 1, 2021

ghost commented Mar 1, 2021

rathann commented Mar 1, 2021

ghost commented Mar 2, 2021 • edited by ghost Loading

ghost commented Mar 2, 2021

skriesch commented Jul 14, 2021

indygreg commented Jan 19, 2022

ghost commented Jan 19, 2022

rathann commented Jan 19, 2022

skriesch commented Jan 30, 2022

ghost commented Mar 1, 2021 •

edited by ghost

Loading

ghost commented Mar 2, 2021 •

edited by ghost

Loading