Upgrade protobuf dependency to version 4.x

[markfickett]

What's the use case?

I have a Python project that includes both Dagster and Pulumi. Pulumi has upgraded to protobuf major version 4, but Dagster still requires protobuf>=3.13.0,<4. I would like Dagster (and other libraries including OpenTelemetry) to upgrade to v4 so that I can move our overall project dependencies up.

Ideas of implementation

No response

Additional information

Repro:

$ python3 -m venv venv
$ source venv/bin/activate
(venv) $ pip install pipdeptree dagster pulumi
...
ERROR: pulumi 3.47.0 has requirement grpcio==1.47, but you'll have grpcio 1.48.0 which is incompatible.
ERROR: pulumi 3.47.0 has requirement protobuf~=4.21, but you'll have protobuf 3.20.3 which is incompatible.
$ pipdeptree
Warning!!! Possibly conflicting dependencies found:
* pulumi==3.47.0
 - grpcio [required: ==1.47, installed: 1.48.0]
 - protobuf [required: ~=4.21, installed: 3.20.3]
------------------------------------------------------------------------
dagster==1.1.1
  ...
  - grpcio [required: >=1.32.0,<1.48.1, installed: 1.48.0]
    - six [required: >=1.5.2, installed: 1.16.0]
  - grpcio-health-checking [required: >=1.32.0,<1.44.0, installed: 1.43.0]
    - grpcio [required: >=1.43.0, installed: 1.48.0]
      - six [required: >=1.5.2, installed: 1.16.0]
    - protobuf [required: >=3.6.0, installed: 3.20.3]
  - protobuf [required: >=3.13.0,<4, installed: 3.20.3]
pulumi==3.47.0
  ...
  - grpcio [required: ==1.47, installed: 1.48.0]
    - six [required: >=1.5.2, installed: 1.16.0]
  - protobuf [required: ~=4.21, installed: 3.20.3]

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

[markfickett]

Opentelemetry was able to allow proto3 or proto4, with this change.

Dagster v0.14.17 "Added a pin to protobuf version 3 due to a backwards incompatible change in the probobuf version 4 release." I think a change similar to OTel's would work around this backwards incompatibility.

[mkleinbort-ic]

I would highly welcome this feature. I have another tool that is much faster using proto4+ which makes this incompatibility a big annoyance

[scottincrypto]

I'm keen to see this one resolved too, I have a dependency clash on another package

[dbfreem]

I also, have a dependency clash due to this issue. I would like to see it upgraded to 4.

[npezolano]

Came here to +1 this issue, how big is the performance impact from using PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python

[gibsondan]

Just an update on this: There's a PR that makes dagster work with both protobuf 3 and protobuf 4 here: #12466

The main blocker to getting that shipped is relaxing our grpcio pin, since the grpcio version we are pinned to still has a protobuf 3 pin - we are working with the grpc team to make that possible here: grpc/grpc#31885

[gibsondan]

Actually it looks like we don't need to fully relax our grpcio pin, we just need to bump the min grpcio version that we support up a bit. So this will likely go live in the 1.2 release in a couple of weeks.

[0x26res]

If I'm not mistaken this has been resolved.

[gibsondan]

Definitely resolved, yes! Sorry for not following up.