Error at from ray import tune due to new protobuf release

[saitcakmak]

What happened + What you expected to happen

As of recent protobuf 4.21.1 release, from ray import tune throws the following error:

(py37) saitcakmak@saitcakmak-mbp ~ % python
Python 3.7.0 (default, Jun 28 2018, 07:39:16) 
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from ray import tune
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/saitcakmak/opt/anaconda3/envs/py37/lib/python3.7/site-packages/ray/__init__.py", line 108, in <module>
    import ray._raylet  # noqa: E402
  File "python/ray/_raylet.pyx", line 115, in init ray._raylet
  File "/Users/saitcakmak/opt/anaconda3/envs/py37/lib/python3.7/site-packages/ray/exceptions.py", line 7, in <module>
    from ray.core.generated.common_pb2 import RayException, Language, PYTHON
  File "/Users/saitcakmak/opt/anaconda3/envs/py37/lib/python3.7/site-packages/ray/core/generated/common_pb2.py", line 15, in <module>
    from . import runtime_env_common_pb2 as src_dot_ray_dot_protobuf_dot_runtime__env__common__pb2
  File "/Users/saitcakmak/opt/anaconda3/envs/py37/lib/python3.7/site-packages/ray/core/generated/runtime_env_common_pb2.py", line 42, in <module>
    serialized_options=None, json_name='packages', file=DESCRIPTOR,  create_key=_descriptor._internal_create_key),
  File "/Users/saitcakmak/opt/anaconda3/envs/py37/lib/python3.7/site-packages/google/protobuf/descriptor.py", line 560, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates

This was originally encountered on failure of one of our nightly workflows: https://github.com/facebook/Ax/runs/6634795167?check_suite_focus=true

Versions / Dependencies

Python 3.7.0
Ray 1.12.1
Protobuf 4.21.1

Reproduction script

conda create -n test python=3.7
conda activate test
pip install "ray[tune]"
python
from ray import tune

Issue Severity

Medium: It is a significant difficulty but I can work around it.

[ddelange]

fixed by #25211

[mburaksayici]

Downgraded protobuf to 3.19 and its solved for our case.

[junbridge]

pip3 install --upgrade protobuf==3.20.0

[trahloff]

I also get this error when trying to spin up a minimal AWS cluster from my local machine. Is there any workaround I can use to make it work?

[ddelange]

This issue can be closed as ray 1.13.0 has been released