cffi ZstdDecompressor.stream_reader broken

[rtkbkish]

Here is simple example of the broken behaviour

from zstandard import ZstdDecompressor
from zstandard import ZstdCompressor
from cStringIO import StringIO
import os

def zstd_readlines(handle):
  """
  Read data from a Zstandard-compressed stream and split by newlines. Returns
  an iterator over the stream.
  :param file handle: the file to read from
  :return: a generator that yields text lines
  :rtype : Iterator[unicode]
  """
  decompressor = ZstdDecompressor()
  with decompressor.stream_reader(handle) as reader:
    while True:
      read = reader.read(8192)
      if not read:
        break

def compress(data):
  handle = StringIO()
  compressor = ZstdCompressor()
  with compressor.stream_writer(handle) as writer:
    writer.write(data)

  handle.seek(0)
  return handle


data = bytearray(os.urandom(1000000))
handle = compress(data)
zstd_readlines(handle)

print "Success"

(venv) root@a74460b9653b:~# pypy read.py
Traceback (most recent call last):
  File "read4.py", line 33, in <module>
    zstd_readlines(handle)
  File "read4.py", line 17, in zstd_readlines
    read = reader.read(8192)
  File "/opt/awn/venv/site-packages/zstd_cffi.py", line 1521, in read
    result = decompress()
  File "/opt/awn/venv/site-packages/zstd_cffi.py", line 1491, in decompress
    _zstd_error(zresult))
ZstdError: (u'zstd decompress error: %s', u'Corrupted block detected')
(venv) root@a74460b9653b:~# export PYTHON_ZSTANDARD_IMPORT_POLICY='cext'
(venv) root@a74460b9653b:~# pypy read.py
Success
(venv) root@a74460b9653b:~#

We worked around with by switching to read_to_iter, but block reading should work as well.

[indygreg]

I am able to reproduce the issue and it looks legitimate. I am quite surprised the fuzz testing didn't catch this!