Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

six.moves cannot be pickled with cloudpickle #194

Open
li-dan opened this issue May 2, 2017 · 0 comments
Open

six.moves cannot be pickled with cloudpickle #194

li-dan opened this issue May 2, 2017 · 0 comments

Comments

@li-dan
Copy link

li-dan commented May 2, 2017

six.moves cannot be pickled with cloudpickle. Reproducer:

#!/usr/bin/env python
from __future__ import absolute_import, division, print_function

import cloudpickle
import six.moves

if __name__ == "__main__":
    print("cloudpickle version:", cloudpickle.__version__)
    print("six version:", six.__version__)
    cloudpickle.loads(cloudpickle.dumps(six.moves))

With Python version 2.7.11, cloudpickle version 0.2.2 and six version 1.10.0, this throws a RuntimeError with the message "maximum recursion depth exceeded while calling a Python object":

cloudpickle version: 0.2.2
six version: 1.10.0
Traceback (most recent call last):
  File "test.py", line 10, in <module>
    cloudpickle.loads(cloudpickle.dumps(six.moves))
  File "/opt/python/python-2.7/lib64/python2.7/pickle.py", line 1388, in loads
    return Unpickler(file).load()
  File "/opt/python/python-2.7/lib64/python2.7/pickle.py", line 864, in load
    dispatch[key](self)
  File "/opt/python/python-2.7/lib64/python2.7/pickle.py", line 1221, in load_build
    setstate = getattr(inst, "__setstate__", None)
  File "/usr/local/python/python-2.7/std/lib/python2.7/site-packages/six.py", line 118, in __getattr__
    _module = self._resolve()
  File "/usr/local/python/python-2.7/std/lib/python2.7/site-packages/six.py", line 115, in _resolve
    return _import_module(self.mod)
...

The reason for pickling six.moves is that a recent commit to cloudpickle changed it to pickle any modules that are used by functions, so pickling a function will also pickle any modules that it uses. For example, the bug is triggered by:

#!/usr/bin/env python
from __future__ import absolute_import, division, print_function

import cloudpickle
import six.moves

def f():
    C = six.moves.http_cookies.SimpleCookie()
    return C

if __name__ == "__main__":
    cloudpickle.loads(cloudpickle.dumps(f))

I think the issue can be fixed by defining _LazyModule.__reduce__:

def __reduce__(self):
    return _import_module, (self.__name__,)
li-dan pushed a commit to li-dan/six that referenced this issue May 2, 2017
Since importing six automatically sets up six.moves, _LazyModule
instances can be unpickled by just importing them.

Fixes benjaminp#194.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant