Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: use thread-safe iterator to generate context ids #716

Merged
merged 2 commits into from
Sep 7, 2021

Conversation

chrisrossi
Copy link
Contributor

Fixes #715

@chrisrossi chrisrossi requested review from andrewsg and a team as code owners September 2, 2021 20:54
@product-auto-label product-auto-label bot added the api: datastore Issues related to the googleapis/python-ndb API. label Sep 2, 2021
@google-cla google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Sep 2, 2021
self.lock = threading.Lock()

def __next__(self):
with self.lock:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there's a good chance itertools.count is threadsafe, but it doesn't seem to be officially documented as such, so I figured it was better not to assume that.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By inspection, the C implementation doesn't make any calls which would release the GIL, AFAICT. @jimfulton do you have any other intuition about it?

@chrisrossi
Copy link
Contributor Author

Notably, I did write a regression test, using orchestrate, that exposed the original bug, but the test didn't really survive the transition from generator function to iterator class.

google/cloud/ndb/context.py Show resolved Hide resolved
self.lock = threading.Lock()

def __next__(self):
with self.lock:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By inspection, the C implementation doesn't make any calls which would release the GIL, AFAICT. @jimfulton do you have any other intuition about it?



_context_ids = _generate_context_ids()
_context_ids = _ContextIds()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there no explicit tests for _generate_context_ids which need updating?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There aren't, no. It was a private implementation detail of the constructor and got called in the majority of unit tests, so I didn't need it for coverage. I did go ahead and add some asserts for it in the test for the Context constructor. I also went ahead and added a non-deterministic regression test for this bug. I figure it's better than nothing.

@chrisrossi chrisrossi merged commit 92ec8ac into googleapis:main Sep 7, 2021
@chrisrossi chrisrossi deleted the fix-715 branch September 7, 2021 13:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: datastore Issues related to the googleapis/python-ndb API. cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

'Generator already executing'
2 participants