Memory leak copy_stream

[mikeheddes]

This is similar to #35 and #40. With zstandard 0.10.2 and Python 3.6.0.

def write_pkl(data, file):
    cctx = zstd.ZstdCompressor()
    stream = io.BytesIO()
    with open(file, 'wb') as f:
        pickle.dump(data, stream)
        stream.seek(0)
        cctx.copy_stream(stream, f)

The data input is a list with ~270000 tuples with Numpy arrays and single numbers. No bytes are written with such a large array but with smaller arrays it works fine. It was working for me with an array of ~200000 long.

My memory consumption goes from ~1.5 GB while creating the array to +35 GB when write_pkl is executed.

[mikeheddes]

I looked a bit further and saw that stream.seek(0) is never called so I think there is something wrong with pickle.dump. I will close this issue because I think it's not related.

[indygreg]

Using io.BytesIO() here will buffer the pickled output then copy that buffered output to a zstd compressor.

Try using a zstd.ZstdCompressor().stream_writer() and feed an instance of that directly to pickle.dump(). That may not work though, as stream_writer() doesn't implement the full io.rawIOBase interface. That's on my TODO list.

[mikeheddes]

Thank you for your suggestion it works.

def write_pkl(data, file):
    with open(file, 'wb') as f:
        cctx = zstd.ZstdCompressor()
        with cctx.stream_writer(f) as compressor:
            pickle.dump(data, compressor)