[core] AttributeError: module 'abc' has no attribute '_get_dump'

[PidgeyBE]

ray 1.1.0

What is the problem?

In ray 1.1.0 the following critical lines where removed from the codebase in 871cde9:

            if hasattr(abc, '_get_dump'):
                (registry, _, _, _) = abc._get_dump(obj)
                clsdict["_abc_impl"] = [subclass_weakref()
                                        for subclass_weakref in registry]
            else:
                # FIXME(suquark): The upstream cloudpickle cannot work in Ray
                # because sometimes both '_abc_registry' and '_get_dump' does
                # not exist. Some strange typing objects may cause this issue.
                # Here the workaround just set "_abc_impl" to None.
                clsdict["_abc_impl"] = None

As a result, our software CICD fails after upgrading to ray 1.1.0 with:
AttributeError: module 'abc' has no attribute '_get_dump'
If I just open ipython and import abc, I get the same error on abc._get_dump. We are using python 3.6.

Reproduction (REQUIRED)

The issue is hard to reproduce in a standalone script. But anyhow

import abc
abc._get_dump

result in the same error (in Python 3.6), so I assume the workaround should have stayed?

• I have verified my script runs in a clean environment and reproduces the issue.
• I have verified the issue also occurs with the latest wheels.

[simon-mo]

@suquark would you mind taking a look a this?
also @PidgeyBE can you list the typing package version if there is one? pip freeze | grep typing.

[PidgeyBE]

@simon-mo

# pip freeze | grep typing
typing-extensions==3.7.4.1

[PidgeyBE]

Additionally we see here that some _abc_ cache keys are popped before pickling serialization:

ray/python/ray/cloudpickle/cloudpickle_fast.py

Line 187 in 871cde9

clsdict.pop('_abc_negative_cache', None)

but not all of them. Is there a reason e.g. _abc_generic_negative_cache and _abc_generic_negative_cache_version are not popped?
Because these keys/values are causing pickle issues in our use cases...

[suquark]

_abc_generic_negative_cache and _abc_generic_negative_cache_version are likely generated by third-party libraries, and they do not exist in the python stdlib: https://github.com/python/cpython/blob/master/Lib/_py_abc.py.

we can hardcode these fields in cloudpickle as the workaround. Such issues would not be fully solved before python3.7 (where some patches are appiled), because python typing object has poor serialization support and failed a wide range of softwares: python/typing#511

[simon-mo]

The PR was merged. @PidgeyBE please try out latest master!

[PidgeyBE]

@simon-mo I've tested it on master and it works!

[hellofinch]

@simon-mo hello, the same bug came out in ray 2.0.0 with python 3.6. I tried to fix it as your PR shows. And it works. Maybe you can merge a PR again?