Serializing attrs classes does not work?

[bitbucketboss]

Working minimal example using Dask's default scheduler:

import time
import attr
import dask

class MyClass:
    def __init__(self, i):
        self.i = i

@attr.s
class MyClassAttrs:
    i = attr.ib()

my_class = MyClass(1)
my_class_attrs = MyClassAttrs(1)

@dask.delayed
def Increment(c):
    time.sleep(1)
    c.i = c.i + 1
    return c

delayed = Increment(my_class)
result = delayed.compute()
result.i

delayed_attrs = Increment(my_class_attrs)
result_attrs = delayed_attrs.compute()
result_attrs.i

This code correctly returns 2 for result.i and result_attrs.i using Dask's default scheduler.

Failing minimal example using Dask's distributed scheduler:

import time
import attr
import dask

# Start new code
from dask.distributed import Client
client = Client()
# End new code

class MyClass:
    def __init__(self, i):
        self.i = i

@attr.s
class MyClassAttrs:
    i = attr.ib()

my_class = MyClass(1)
my_class_attrs = MyClassAttrs(1)

@dask.delayed
def Increment(c):
    time.sleep(1)
    c.i = c.i + 1
    return c

delayed = Increment(my_class)
result = delayed.compute()
result.i

delayed_attrs = Increment(my_class_attrs)
result_attrs = delayed_attrs.compute()
result_attrs.i

client.shutdown()

This code correctly returns 2 for result.i but raises

TypeError: cannot pickle '_thread._local' object

, at line:

result_attrs = delayed_attrs.compute()

when using the Dask distributed scheduler.

attrs is a widely used package so I'm not sure why its use is causing a failure.

Attrs classes do pickle properly by themselves. For example, this does work:

from pickle import dumps, loads
loads(dumps(my_class_attrs))

I have also successfully used the standard multiprocessing library to do multiprocessing with attrs classes.

Environment

• Dask version: 2.30.0
• Python version: 3.8.5
• Operating System: Windows 10
• Install method (conda, pip, source): conda

Any advice on how to deal with this issue would be welcome.

[bitbucketboss]

Upon further investigation I note that cloudpickle does not work on my_class_attrs:

import cloudpickle
cloudpickle.loads(cloudpickle.dumps(my_class_attrs))

, raises the same error encountered previously:

TypeError: cannot pickle '_thread._local' object

[bitbucketboss]

This appears to be a cloudpickle bug:

Cloudpickle issue 320
Attrs issue 458

As such I will close this issue.

[kylebarron]

Note that as mentioned in the linked issues, this will work if you set repr=False on your attrs classes.

[thetorpedodog]

With attrs release 21.4.0, that workaround is no longer needed; attrs classes can now be pickled with cloudpickle (e.g. via Dask).