AttributeError: type object 'Callable' has no attribute '_abc_registry'

[ghost]

I'm trying to deploy scrapy project to scrapinghub using the following command:

shub deploy 470662

Versions:

• Scrapy version 2.2.1
• shub version 2.11.0
• python3.8

it generates the following error:

/usr/local/lib/python3.8/site-packages/setuptools/distutils_patch.py:25: UserWarning: Distutils was imported before Setuptools. This usage is discouraged and may exhibit undesirable behaviors or errors. Please use Setuptools' objects directly or at least import Setuptools first.
  warnings.warn(
Packing version 1.0
Deploying to Scrapy Cloud project "470662"
Deploy log last 30 lines:
  File "/app/python/lib/python3.8/site-packages/scrapy/http/__init__.py", line 15, in <module>
    from scrapy.http.response import Response
  File "/app/python/lib/python3.8/site-packages/scrapy/http/response/__init__.py", line 7, in <module>
    from typing import Generator
  File "/app/python/lib/python3.8/site-packages/typing.py", line 1359, in <module>
    class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
  File "/app/python/lib/python3.8/site-packages/typing.py", line 1007, in __new__
    self._abc_registry = extra._abc_registry
AttributeError: type object 'Callable' has no attribute '_abc_registry'
Traceback (most recent call last):
  File "/usr/local/bin/shub-image-info", line 8, in <module>
    sys.exit(shub_image_info())
  File "/usr/local/lib/python3.8/site-packages/sh_scrapy/crawl.py", line 200, in shub_image_info
    from scrapy.exceptions import ScrapyDeprecationWarning
  File "/app/python/lib/python3.8/site-packages/scrapy/__init__.py", line 12, in <module>
    from scrapy.spiders import Spider
  File "/app/python/lib/python3.8/site-packages/scrapy/spiders/__init__.py", line 10, in <module>
    from scrapy.http import Request
  File "/app/python/lib/python3.8/site-packages/scrapy/http/__init__.py", line 15, in <module>
    from scrapy.http.response import Response
  File "/app/python/lib/python3.8/site-packages/scrapy/http/response/__init__.py", line 7, in <module>
    from typing import Generator
  File "/app/python/lib/python3.8/site-packages/typing.py", line 1359, in <module>
    class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
  File "/app/python/lib/python3.8/site-packages/typing.py", line 1007, in __new__
    self._abc_registry = extra._abc_registry
AttributeError: type object 'Callable' has no attribute '_abc_registry'
{"message": "shub-image-info exit code: 1", "details": null, "error": "image_info_error"}

{"status": "error", "message": "Internal error"}
Deploy log location: /var/folders/rp/xxjnjsvn70g2ndh68l0g10bh0000gn/T/shub_deploy_6h9fra1k.log
Error: Deploy failed: b'{"status": "error", "message": "Internal error"}'

And here are requirements.txt and scrapinghub.yml

requirements.txt:

itemadapter==0.1.0
requests==2.23.0
Scrapy==2.2.1
scrapy-user-agents==0.1.1
scrapy-rotating-proxies==0.6.2

scrapinghub.yml:

projects:
  default: 470662
requirements:
  file: requirements.txt

[vshlapakov]

Seems to be an issue in scrapy-rotating-proxies: it requires typing library explicitly which is a backport of stdlib' typing and not needed for Python >=3.5. Ideally it just shouldn't have any effect on Python 3.8, but it seems there's a known issue starting with Python>=3.7 (python/typing#721).

I'd suggest to create an issue in scrapy-rotating-proxies tracker, they should require the typing package only for specific python versions (note their setup.py has nothing about Python 3.7/3.8 support, so it can take some time).