Ray Core: AttributeError: 'NoneType' object has no attribute 'enum_types_by_name'

[Nithanaroy]

What happened + What you expected to happen

I started the head node using ray start. Then to create a cluster I started the worker node, passing in the head server's address, ray start --address='100.96.243.93:54140' --redis-password='5241590000000000' -v --block. The worker node fails to start and dies with the follow error,

2022-09-24 00:45:22,881	INFO agent.py:83 -- Parent pid is 319
2022-09-24 00:45:22,882	INFO agent.py:109 -- Dashboard agent grpc address: 0.0.0.0:58037
2022-09-24 00:45:22,883	INFO utils.py:99 -- Get all modules by type: DashboardAgentModule
2022-09-24 00:45:23,241	ERROR agent.py:385 -- Agent is working abnormally. It will exit immediately.
Traceback (most recent call last):
  File "/home/jobuser/.local/lib/python3.7/site-packages/ray/dashboard/agent.py", line 383, in <module>
    loop.run_until_complete(agent.run())
  File "/export/apps/python/3.7/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()
  File "/home/jobuser/.local/lib/python3.7/site-packages/ray/dashboard/agent.py", line 167, in run
    modules = self._load_modules()
  File "/home/jobuser/.local/lib/python3.7/site-packages/ray/dashboard/agent.py", line 126, in _load_modules
    dashboard_utils.DashboardAgentModule
  File "/home/jobuser/.local/lib/python3.7/site-packages/ray/dashboard/utils.py", line 108, in get_all_modules
    importlib.import_module(name)
  File "/export/apps/python/3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/jobuser/.local/lib/python3.7/site-packages/ray/dashboard/modules/reporter/reporter_agent.py", line 24, in <module>
    from ray._private.metrics_agent import MetricsAgent, Gauge, Record
  File "/home/jobuser/.local/lib/python3.7/site-packages/ray/_private/metrics_agent.py", line 28, in <module>
    import ray._private.prometheus_exporter as prometheus_exporter
  File "/home/jobuser/.local/lib/python3.7/site-packages/ray/_private/prometheus_exporter.py", line 18, in <module>
    from opencensus.common.transports import sync
  File "/home/jobuser/.local/lib/python3.7/site-packages/opencensus/common/transports/sync.py", line 16, in <module>
    from opencensus.trace import execution_context
  File "/home/jobuser/.local/lib/python3.7/site-packages/opencensus/trace/__init__.py", line 15, in <module>
    from opencensus.trace.span import Span
  File "/home/jobuser/.local/lib/python3.7/site-packages/opencensus/trace/span.py", line 32, in <module>
    from opencensus.trace import status as status_module
  File "/home/jobuser/.local/lib/python3.7/site-packages/opencensus/trace/status.py", line 15, in <module>
    from google.rpc import code_pb2
  File "/home/jobuser/.local/lib/python3.7/site-packages/google/rpc/code_pb2.py", line 36, in <module>
    _CODE = DESCRIPTOR.enum_types_by_name["Code"]
AttributeError: 'NoneType' object has no attribute 'enum_types_by_name'

Versions / Dependencies

• Ray 1.12.1 on both server and worker nodes
• Python 3.7.10
• Linux OS (RHEL 7.6)

Reproduction script

• ray start on machine 1 with the above config
• ray start --address='<server address>' --redis-password='5241590000000000' -v --block on machine 2 with the above config

Issue Severity

High: It blocks me from completing my task.

[cadedaniel]

cc @rickyyx @scv119

[scv119]

@Nithanaroy i think this is due the compatibility issue between protobuf and grpcio in the old ray versions. Can you try to use latest ray (2.0.0) which have fixed this issue?

I believe #25648 fixed the issue.

[Nithanaroy]

@scv119, looks like the condition, protobuf >= 3.15.3, < 4.0.0 mentioned in the above PR is satisfied in my case

Head

• protobuf==3.19.4
• grpcio==1.43.0

Worker

• protobuf==3.18.1
• grpcio==1.41.0

Unfortunately, it is bit tricky to migrate to 2.0 yet in my setup.

[scv119]

@Nithanaroy hmm could you try to pin grpcio to an earlier version, such 1.30.0 ?

[richardliaw]

Going to close this out for now but @Nithanaroy feel free to reopen if the grpcio doesn't resolve it!

[Nithanaroy]

Sorry for the delay. That didn't solve the issue. Attached all the dependencies and versions I used in head and worker nodes.

head.txt
worker.txt

[Nithanaroy]

I don't see a reopen option, @richardliaw @scv119