-
Notifications
You must be signed in to change notification settings - Fork 207
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
chore: transition the library to the new microgenerator #158
Conversation
a18ea36
to
1ecc48f
Compare
The argument is not supported anymore in the new generated client.
This configuration is not anymore supported by the generated subscriber client.
1ecc48f
to
2b01cc4
Compare
I believe this PR is now ready to be reviewed. The things that still fail are related to incompatible changes in the client itself, for example:
We need to figure out how to handle these and I will add the Since this PR is quite complex, every pair of eyes would be beneficial. 🙂
Let's not rush and take the time to review this thoroughly, thanks! |
I don't think I understand the benchmark result very well; are the two different throughput numbers for Also, just had a thought that may be relevant: is the manual layer invoking the asynchronous client or the synchronous? I see a lot of task/future names and semantics, which made me ask. The asynchronous surface has had basically no optimization focus. |
I did some publisher profiling with from google.pubsub_v1 import types as gapic_types
...
# Create the Pub/Sub message object.
message = gapic_types.PubsubMessage(
data=data, ordering_key=ordering_key, attributes=attrs
) FWIW, profiling the current released version (i.e. the Is instantiating a |
Yes, constructing messages is more expensive with the microgenerator because of its use of proto-plus. Depending on what Can you send me the raw data in some way? I'd like to take a closer look at which part of construction is expensive. E.g. vanilla_pb = gapic_types.PubsubMessage.pb(data=data, ordering_key=ordering_key, attributes=attrs)
proto_plus_pb = gapic_types.PubsubMessage.wrap(vanilla_pb) |
This actually improved things considerably - the benchmark showed a publisher throughput of ~57 MB/s, which is only ~11% worse than the current stable version. In addition, the
The following is roughly how the messages look like in the benchmark and my local script for profiling: data = b"A" * 250
ordering_key = ""
attrs = {'clientId': '0', 'sendTime': '1598046901869', 'sequenceNumber': '0'}
...
message = gapic_types.PubsubMessage(
data=data, ordering_key=ordering_key, attributes=attrs
) ( If using the hack, instantiation is considerably faster, but there is still quite a lot of time spent in I'll also see if the same can be used to improve subscriber performance.
Correct. The benchmark framework spins up separate compute instances for publisher and subscriber and then measures how well they perform individually.
It's the synchronous client, i.e. |
I have some good news - circumventing the wrapper classes around the raw protobuf messages to speed up instantiation and attribute access seems to help a lot. Running the benchmarks with the two most recent experimental commits produces the following:
The performance hit is now only around 10% (publisher ~12%, subscriber ~8%), which might actually be acceptable. |
Profiling shows that the speed of creating a new pubsub message and the speed of accessing the message's attributes significantly affects the throughput of publisher and subscriber. This commit makes everything faster by circumventing the wrapper class around the raw protobuf pubsub messages where possible.
30dce66
to
c29d7f8
Compare
Optimizing |
My bad, when I said "the raw data" above, I meant the profiling data. I don't particularly want to set up the benchmark environment, but I do want to try looking through the data to see what the hot spots are and if they could be optimized. |
I have a special, optimized-but-not-reviewed-or-production-ready tarball of proto-plus. Is it possible to run the benchmark suite using it? My own benchmarking has these changes saving about 10-15 milliseconds out of about 500 milliseconds. YMMV, but if it improves the throughput enough it could be worth putting the proto-plus changes up for review. |
If Just to clarify, would you like to run the benchmark using the tip of this PR branch, i.e. the version that already includes the optimizations that try to bypass the protobuf buffer classes? Or on the version that excludes this last commit and uses the wrapper classes more heavily?
No worries, sent you an email with the data just now. |
The tip of this PR branch. Changes are also available in this branch of my fork: https://github.com/software-dov/proto-plus-python/tree/hackaday |
First run, using proto-plus 1.7.1 on the tip of this PR branch. The results seem more or less similar to the previous benchmark:
I also did another run, the measured throughput was around 55 MB/s - a bit worse, but within the normal results variance (the typical difference appears to be somewhere in the 1-2 MB/s range). |
Okay. In light of those numbers I'm inclined not to merge the proto-plus changes. Is there an approval process for accepting the throughput regression? |
Please just mind that that the proto-plus optimizations might not have a noticeable effect here, as the optimizations added here actively try to circumvent proto-plus, but other libraries might still see benefits. I'm not aware of any formal processes, but we do have weekly PubSub meetings on Thursdays where we discuss these things. I added the question to the agenda whether a -10% performance hit is acceptable in the new major release. Update: Was confirmed, -10% is good enough for now, although we should strive to improve this further in the mid-term. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks good to me. Sorry this took so long. Had to go commit by commit.
@cguardia That's fine, appreciated. I will re-generate and benchmark the code again now, just in case, and see if we are ready for the major release. |
After re-generating the code yet again, performance still appears to be in line with the previous benchmarks:
Merging. 🎉 Edit: Ah, @kamalaboulhosn expressed a wish to review the UPGRADING guide, will wait with merging a bit more. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The upgrade guide looks good.
Closes #131.
Closes #168.
This PR replaces the old code generator, the generated parts of the code now use microgenerator. The latter also implies dropping support for Python 2.7 and 3.5.
There are a lot of changes and the transition itself was far from smooth, thus it's probably best to review this commit by commit (I tried to make the commits self-contained with each containing a single change/fix).
Things to focus on in reviews
Regenerating the code overrides some of the URLs in the samples README. Seems like a synthtool issue.
The clients do not support the
client_config
argument anymore. At least the ordering keys feature used that to change the timeout to "infinity". We need to see if that's crucial, and if the same can be set in a different way (maybe through the retry policy...)SERVICE_ADDRESS
and_DEFAULT_SCOPES
constants in clients might be obsolete. Let's see if there are more modern alternatives (they are currently still injected into the generated clients).Regenerating the code out of the box fails, because the
Bazel
tool incorrectly tries to use Python 2, resulting in syntax errors (could be just a problem on my machine, but it is a known Bazel issue).Workaround:
google/pubsub/v1/BUILD.bazel
file in the local googleapis clone and point thesynthtool
to it:synthtool
installation. Add the following two Bazel arguments to the_generate_code()
method insynthtool/gcp/gapic_bazel.py
(lines 177-178):The workaround should convince Bazel to use Python 3, as this is the Python version in the configs.
Things left to do
Double check that the new client performance is adequate. There have been reports of possibly degraded performance with the new microgenerator.The 10% performance hit was declared acceptable, not a release blocker anymore.get_iam_policy()
, or do not support all config options anymore, e.gcreate_subscription()
. Adjust or delete?Determine a replacement for now-unsupportedclient_config
argument to client constructor. Or does the code generator need an update?client_config
has been replaced by custom Retry settings passed to the GAPIC clientpublish()
. If we want to support custom retries, we must update the user-facing client'spublish()
method.PR checklist