Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make client work with gevent monkey patching #328

Open
ns-dzhang opened this issue Apr 11, 2022 · 1 comment
Open

Make client work with gevent monkey patching #328

ns-dzhang opened this issue Apr 11, 2022 · 1 comment
Labels
api: pubsublite Issues related to the googleapis/python-pubsublite API. priority: p3 Desirable enhancement or fix. May not be included in next release. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.

Comments

@ns-dzhang
Copy link

Hello,

  • OS type and version: Mac OS Big Sur ver 11.6.1
  • Python version: Python 3.8.9
  • pip version: pip 21.1.2
  • google-cloud-pubsublite version: 1.4.1

Steps to reproduce

  1. using tutorial codes : https://cloud.google.com/pubsub/lite/docs/publishing
  2. add two more lines at the beginning:
from gevent import monkey
monkey.patch_all()

Code example

from gevent import monkey
monkey.patch_all()

from google.cloud.pubsublite.cloudpubsub import PublisherClient
from google.cloud.pubsublite.types import (
    CloudRegion,
    CloudZone,
    MessageMetadata,
    TopicPath,
)

# TODO(developer):
# project_number = 1122334455
# cloud_region = "us-central1"
# zone_id = "a"
# topic_id = "your-topic-id"
# num_messages = 100
# regional = True

if regional:
    location = CloudRegion(cloud_region)
else:
    location = CloudZone(CloudRegion(cloud_region), zone_id)

topic_path = TopicPath(project_number, location, topic_id)

# PublisherClient() must be used in a `with` block or have __enter__() called before use.
with PublisherClient() as publisher_client:
    for message in range(num_messages):
        data = f"{message}"
        # Messages of the same ordering key will always get published to the same partition.
        # When ordering_key is unset, messsages can get published ot different partitions if
        # more than one partition exists for the topic.
        api_future = publisher_client.publish(
            topic_path, data.encode("utf-8"), ordering_key="testing"
        )
        # result() blocks. To resolve api futures asynchronously, use add_done_callback().
        message_id = api_future.result()
        message_metadata = MessageMetadata.decode(message_id)
        print(
            f"Published {data} to partition {message_metadata.partition.value} and offset {message_metadata.cursor.offset}."
        )

print(
    f"Finished publishing {num_messages} messages with an ordering key to {str(topic_path)}."
)

When running the code with the monkey.patch_all(), publish() hangs.

I debugged a little bit, looks "thread" and "select" patch affect pubsublite. If I didn't patch these two as below, it worked as expected. Not sure if pubsublite is gevent friendly? My case is I need patch all.

from gevent import monkey
monkey.patch_all(thread=False, select=False)

Thanks!

@product-auto-label product-auto-label bot added the api: pubsublite Issues related to the googleapis/python-pubsublite API. label Apr 11, 2022
@ns-dzhang
Copy link
Author

cc: @ns-xliu @ns-zquan

@yoshi-automation yoshi-automation added triage me I really want to be triaged. 🚨 This issue needs some love. labels Apr 12, 2022
@meredithslota meredithslota added type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. priority: p2 Moderately-important priority. Fix may not be included in next release. and removed 🚨 This issue needs some love. triage me I really want to be triaged. labels Apr 20, 2022
@dpcollins-google dpcollins-google added priority: p3 Desirable enhancement or fix. May not be included in next release. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design. and removed priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. labels Apr 22, 2022
@dpcollins-google dpcollins-google changed the title publish() hang when using gevent monkey patch_all() Make client work with gevent monkey patching Apr 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: pubsublite Issues related to the googleapis/python-pubsublite API. priority: p3 Desirable enhancement or fix. May not be included in next release. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.
Projects
None yet
Development

No branches or pull requests

4 participants