-
Notifications
You must be signed in to change notification settings - Fork 169
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cannot pickle attrs classes #320
Comments
I am not sure if it's a bug, a regression or a missing feature but someone should definitely investigate :) Thanks for the report. |
At least the first part with The second part of the ticket (with |
I thought #321 was only about Python 3.6 while this issue is reported for 3.8.1. Will need to investigate but cannot do it in the short term. |
With the merge of @pierreglaser MR ( 8f851f7 ) now both of my above test-cases produce the same output: |
* add and document API * implement run batch in parallel (+ tests) * simplify model context manager * fix pickle process classes See cloudpipe/cloudpickle#320 python-attrs/attrs#458 * fix zarr in-memory store and dask processes * disable dask parallel schedulers on CI Is it really supported? * get a dask lock for create / resize zarr datasets * clean-up * add test for DummyLock * run model processes in parallel + more docstrings * update release notes * docstrings tweaks * doc: add run parallel section
Hello, I was looking at this and found that for some reason I cannot recreate the thread.local error in pytest. I was wondering if anyone had any thoughts on why that is? or possibly the solution! for reference the stdout/err is:
patch below
|
Generated `__repr__` methods for Pythons with f-strings included a `threading.local` object in that method's `globals()` dictionary. Because `cloudpickle` attempts to serialize all globals of this method, it ends up trying to pickle the `threading.local`, which cannot be pickled. Instead, we now pull use of the thread-local out into its own function, which `cloudpickle` will happily serialize. As an added benefit, this eliminates some duplicated code between f-string and non–f-string `__repr__`s. Should fix: - python-attrs#458 - cloudpipe/cloudpickle#320
Generated `__repr__` methods for Pythons with f-strings included a `threading.local` object in that method's `globals()` dictionary. Because `cloudpickle` attempts to serialize all globals of this method, it ends up trying to pickle the `threading.local`, which cannot be pickled. Instead, we now pull use of the thread-local out into its own function, which `cloudpickle` will happily serialize. As an added benefit, this eliminates some duplicated code between f-string and non–f-string `__repr__`s. Should fix: - python-attrs#458 - cloudpipe/cloudpickle#320
Generated `__repr__` methods for Pythons with f-strings included a `threading.local` object in that method's `globals()` dictionary. Because `cloudpickle` attempts to serialize all globals of this method, it ends up trying to pickle the `threading.local`, which cannot be pickled. Instead, we now pull use of the thread-local out into its own function, which `cloudpickle` will happily serialize. As an added benefit, this eliminates some duplicated code between f-string and non–f-string `__repr__`s. Should fix: - python-attrs#458 - cloudpipe/cloudpickle#320
Because cloudpickle tries to pickle a function's globals, when it pickled an attrs instance, it would try to pickle the `__repr__` method and its globals, which included a `threading.local`. This broke cloudpickle for all attrs classes unless they explicitly specified `repr=False`. Modules, however, are pickled by reference, not by value, so moving the repr into a different module means we can put `_compat` into the function's globals and not worry about direct references. Includes a test to ensure that attrs and cloudpickle remain compatible. Also adds an explanation of the reason we even *have* that global thread-local variable. It wasn't completely obvious to a reader why the thread-local was needed to track reference cycles in `__repr__` calls, and the test did not previously contain a cycle that touched a non-attrs value. This change adds a comment explaining the need and also adds a non-attrs value in the reference cycle of `test_infinite_recursion`. Fixes: - python-attrs#458 - cloudpipe/cloudpickle#320
Because cloudpickle tries to pickle a function's globals, when it pickled an attrs instance, it would try to pickle the `__repr__` method and its globals, which included a `threading.local`. This broke cloudpickle for all attrs classes unless they explicitly specified `repr=False`. Modules, however, are pickled by reference, not by value, so moving the repr into a different module means we can put `_compat` into the function's globals and not worry about direct references. Includes a test to ensure that attrs and cloudpickle remain compatible. Also adds an explanation of the reason we even *have* that global thread-local variable. It wasn't completely obvious to a reader why the thread-local was needed to track reference cycles in `__repr__` calls, and the test did not previously contain a cycle that touched a non-attrs value. This change adds a comment explaining the need and tests a cycle that contains non-attrs values. Fixes: - python-attrs#458 - cloudpipe/cloudpickle#320
Because cloudpickle tries to pickle a function's globals, when it pickled an attrs instance, it would try to pickle the `__repr__` method and its globals, which included a `threading.local`. This broke cloudpickle for all attrs classes unless they explicitly specified `repr=False`. Modules, however, are pickled by reference, not by value, so moving the repr into a different module means we can put `_compat` into the function's globals and not worry about direct references. Includes a test to ensure that attrs and cloudpickle remain compatible. Also adds an explanation of the reason we even *have* that global thread-local variable. It wasn't completely obvious to a reader why the thread-local was needed to track reference cycles in `__repr__` calls, and the test did not previously contain a cycle that touched a non-attrs value. This change adds a comment explaining the need and tests a cycle that contains non-attrs values. Fixes: - #458 - cloudpipe/cloudpickle#320
Hi, any update on the issue? from tenacity import retry
import cloudpickle
@retry
def foo(bar):
print(bar)
cloudpickle.dumps(foo) |
With attrs release 21.4.0, this incompatibility has been fixed and attrs instances can be cloudpickled. |
* add and document API * implement run batch in parallel (+ tests) * simplify model context manager * fix pickle process classes See cloudpipe/cloudpickle#320 python-attrs/attrs#458 * fix zarr in-memory store and dask processes * disable dask parallel schedulers on CI Is it really supported? * get a dask lock for create / resize zarr datasets * clean-up * add test for DummyLock * run model processes in parallel + more docstrings * update release notes * docstrings tweaks * doc: add run parallel section
Fails with
TypeError: cannot pickle 'property' object
for me.python 3.8.1, attrs = "19.3.0", cloudpickle = "1.2.2"
As mentioned here: python-attrs/attrs#458 there is also some issue with
TypeError: cannot pickle '_thread._local' object
that can be worked around by using@attrs(repr=False)
. The repr fix does not help with the above.See below for a snipet to reproduce the
TypeError: cannot pickle '_thread._local' object
:The text was updated successfully, but these errors were encountered: