-
Notifications
You must be signed in to change notification settings - Fork 2.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[EventHubs] Performance improvement #7533
[EventHubs] Performance improvement #7533
Conversation
yunhaoling
commented
Oct 1, 2019
•
edited
Loading
edited
- Disable accept
- Receive settle mode -> ReceiveAndDelete
- auto_complete -> False
- Lazy parse message properties
Can one of the admins verify this patch? |
@yunhaoling What does the improvement look like with these changes? |
Hello @bryevdv , After making these changes, async receiving events (without accessing the properties of each event as accessing the properties would require parsing the underlying c_message object), the performance could be 3 times faster then previous eh. Previously eh parses each underlying c message properties once received.
unit: msg(size of 1024bytes) per second |
@yunhaoling nice, thanks. The body of the message is still always accessible, correct? It's just asking for the properties that incurs additional cost? I would suggest adding some tests that maintain that properties are not filled in unless they are explicitly asked for. |
@bryevdv yes, the body is always accessible. only the properties (message_annotation, delivery_annotation, header, application_properties) needs extra parsing. no problem. |
if self._need_further_parse: | ||
self._parse_message_properties() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
uamqp already lazily loads properties. I guess another lazy loading won't improve much performance
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm afraid I have to do the lazy assignment here.
because the parsing actually takes place when properties of message are assigned to events (the property method of message in uamqp will call parse function).
so we need make the assignment lazy too.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we can potentially only put the lazy loading into uamqp... we can discuss offline.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lazy parsing removed
Hello @annatisch as we actually don't need accept messages in EventHubs, can you check whether I can set the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
if self._need_further_parse: | ||
self._parse_message_properties() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we can potentially only put the lazy loading into uamqp... we can discuss offline.
* [EventHubs] Eventhubs runtime metric (#7431) * runtime metric init commit * evenhubts-runtime-metric implementation * Update code, and test and docstring * Add example code * Update property name * Update name to last enqueued event properties * Eventhubs remove iot and linkredirect (#7450) * Remove iothub/link-redirect related code * Remove self._running from consumer and producer * Remove IoT related params "operation" and "device" * Remove exception from close() * add iterator long running test * small bug fix * small bug fix * Fix connection properties bug and format * Changed product to azure-eventhub in user agent * Fix a type hint * Improve stress script * Print to console configurable * small changes * Disable tracking last enqueued event properties for uamqp 1.2.2 * use different consumer group * fix an issue about consumer group * fix an issue about consumer group * Fix a get_properties bug * [EventHubs] Performance improvement (#7533) * performance improvement * batch offset * Review feedback * Remove lazy parse in EventData * Add annotation assertions * Small fix for missing annotation initialization (#7575) * Python EventHubs test code revamp (#7594) * fix pylint error * fix mypy error * fix for 2.7 * fix for 2.7 * Sleep 300 seconds to test reconnect * Remove Pypy from eventhub live test * Remove Pypy from eventhub live test * Add "Matrix:" * Fix bug passing timeout in seconds to uamqp * update checkpointstore HISTORY * [EventHubs] Update documentations (#7603) * Update docs * Update iothub related description * Docs update * Small fix * Small fix * revert setup.py
* [EventHubs] Eventhubs runtime metric (Azure#7431) * runtime metric init commit * evenhubts-runtime-metric implementation * Update code, and test and docstring * Add example code * Update property name * Update name to last enqueued event properties * Eventhubs remove iot and linkredirect (Azure#7450) * Remove iothub/link-redirect related code * Remove self._running from consumer and producer * Remove IoT related params "operation" and "device" * Remove exception from close() * add iterator long running test * small bug fix * small bug fix * Fix connection properties bug and format * Changed product to azure-eventhub in user agent * Fix a type hint * Improve stress script * Print to console configurable * small changes * Disable tracking last enqueued event properties for uamqp 1.2.2 * use different consumer group * fix an issue about consumer group * fix an issue about consumer group * Fix a get_properties bug * [EventHubs] Performance improvement (Azure#7533) * performance improvement * batch offset * Review feedback * Remove lazy parse in EventData * Add annotation assertions * Small fix for missing annotation initialization (Azure#7575) * Python EventHubs test code revamp (Azure#7594) * fix pylint error * fix mypy error * fix for 2.7 * fix for 2.7 * Sleep 300 seconds to test reconnect * Remove Pypy from eventhub live test * Remove Pypy from eventhub live test * Add "Matrix:" * Fix bug passing timeout in seconds to uamqp * update checkpointstore HISTORY * [EventHubs] Update documentations (Azure#7603) * Update docs * Update iothub related description * Docs update * Small fix * Small fix * revert setup.py
* switched vault_url to vault_endpoint across key vault * moved enabled and tags to kwargs * changed the name parameter for get_policy and update_policy to certificate_name * updated changelog * [Core] Updated release notes (#7627) * Core doc fix (#7638) * Fix azure-core examples * Fix azure-core Readme * Simplify doc * Kill common.py file * Remove tracing.ext * update app config doc (#7618) * update app config doc * move match_condition to internal * [AutoPR] netapp/resource-manager (#7092) * Generated from 34d744550227bee7b380604d38fc1de6c9af6397 (#7089) Update netapp.json * changelog for netapp * ANF-374 update for R5 RP version 2019-07-01 API * ANF-374 update for R5 RP version 2019-07-01 API (#7460) * ANF-374 update for R5 RP version 2019-07-01 API * ANF-374 update for R5 RP version 2019-07-01 API * ANF-374 update for R5 RP version 2019-07-01 API * extending pypy timeout. (#7640) * Eventhubs preview4 merge to master (#7605) * [EventHubs] Eventhubs runtime metric (#7431) * runtime metric init commit * evenhubts-runtime-metric implementation * Update code, and test and docstring * Add example code * Update property name * Update name to last enqueued event properties * Eventhubs remove iot and linkredirect (#7450) * Remove iothub/link-redirect related code * Remove self._running from consumer and producer * Remove IoT related params "operation" and "device" * Remove exception from close() * add iterator long running test * small bug fix * small bug fix * Fix connection properties bug and format * Changed product to azure-eventhub in user agent * Fix a type hint * Improve stress script * Print to console configurable * small changes * Disable tracking last enqueued event properties for uamqp 1.2.2 * use different consumer group * fix an issue about consumer group * fix an issue about consumer group * Fix a get_properties bug * [EventHubs] Performance improvement (#7533) * performance improvement * batch offset * Review feedback * Remove lazy parse in EventData * Add annotation assertions * Small fix for missing annotation initialization (#7575) * Python EventHubs test code revamp (#7594) * fix pylint error * fix mypy error * fix for 2.7 * fix for 2.7 * Sleep 300 seconds to test reconnect * Remove Pypy from eventhub live test * Remove Pypy from eventhub live test * Add "Matrix:" * Fix bug passing timeout in seconds to uamqp * update checkpointstore HISTORY * [EventHubs] Update documentations (#7603) * Update docs * Update iothub related description * Docs update * Small fix * Small fix * revert setup.py * Resolve Global Install Errors (#7642) * swapping all dev_requirements to leverage the non-develop mode. opencensus-tracing will fail if azure-core is installed in dev mode. all the rest of the tests are ok with using non-dev, so this seems to be a good compromise * Setup core for unified pipelines. (#7577) * Setup core for unified pipelines. * Added open census to the core pipeline (can be released independently). * Publish the packages artifact. (#7643) * Enable template for unified templates. (#7644) * Setup template for UP. * Fixed service directory. * ensure error_map is used consistently and correctly (#7637) * switched vault_url to vault_endpoint across key vault * ran black on keyvault * reverted <<<HEAD additions to code * got rid of black changes that broke python 3.5
* [EventHubs] Eventhubs runtime metric (Azure#7431) * runtime metric init commit * evenhubts-runtime-metric implementation * Update code, and test and docstring * Add example code * Update property name * Update name to last enqueued event properties * Eventhubs remove iot and linkredirect (Azure#7450) * Remove iothub/link-redirect related code * Remove self._running from consumer and producer * Remove IoT related params "operation" and "device" * Remove exception from close() * add iterator long running test * small bug fix * small bug fix * Fix connection properties bug and format * Changed product to azure-eventhub in user agent * Fix a type hint * Improve stress script * Print to console configurable * small changes * Disable tracking last enqueued event properties for uamqp 1.2.2 * use different consumer group * fix an issue about consumer group * fix an issue about consumer group * Fix a get_properties bug * [EventHubs] Performance improvement (Azure#7533) * performance improvement * batch offset * Review feedback * Remove lazy parse in EventData * Add annotation assertions * Small fix for missing annotation initialization (Azure#7575) * Python EventHubs test code revamp (Azure#7594) * fix pylint error * fix mypy error * fix for 2.7 * fix for 2.7 * Sleep 300 seconds to test reconnect * Remove Pypy from eventhub live test * Remove Pypy from eventhub live test * Add "Matrix:" * Fix bug passing timeout in seconds to uamqp * update checkpointstore HISTORY * [EventHubs] Update documentations (Azure#7603) * Update docs * Update iothub related description * Docs update * Small fix * Small fix * revert setup.py
* switched vault_url to vault_endpoint across key vault * moved enabled and tags to kwargs * changed the name parameter for get_policy and update_policy to certificate_name * updated changelog * [Core] Updated release notes (Azure#7627) * Core doc fix (Azure#7638) * Fix azure-core examples * Fix azure-core Readme * Simplify doc * Kill common.py file * Remove tracing.ext * update app config doc (Azure#7618) * update app config doc * move match_condition to internal * [AutoPR] netapp/resource-manager (Azure#7092) * Generated from 34d744550227bee7b380604d38fc1de6c9af6397 (Azure#7089) Update netapp.json * changelog for netapp * ANF-374 update for R5 RP version 2019-07-01 API * ANF-374 update for R5 RP version 2019-07-01 API (Azure#7460) * ANF-374 update for R5 RP version 2019-07-01 API * ANF-374 update for R5 RP version 2019-07-01 API * ANF-374 update for R5 RP version 2019-07-01 API * extending pypy timeout. (Azure#7640) * Eventhubs preview4 merge to master (Azure#7605) * [EventHubs] Eventhubs runtime metric (Azure#7431) * runtime metric init commit * evenhubts-runtime-metric implementation * Update code, and test and docstring * Add example code * Update property name * Update name to last enqueued event properties * Eventhubs remove iot and linkredirect (Azure#7450) * Remove iothub/link-redirect related code * Remove self._running from consumer and producer * Remove IoT related params "operation" and "device" * Remove exception from close() * add iterator long running test * small bug fix * small bug fix * Fix connection properties bug and format * Changed product to azure-eventhub in user agent * Fix a type hint * Improve stress script * Print to console configurable * small changes * Disable tracking last enqueued event properties for uamqp 1.2.2 * use different consumer group * fix an issue about consumer group * fix an issue about consumer group * Fix a get_properties bug * [EventHubs] Performance improvement (Azure#7533) * performance improvement * batch offset * Review feedback * Remove lazy parse in EventData * Add annotation assertions * Small fix for missing annotation initialization (Azure#7575) * Python EventHubs test code revamp (Azure#7594) * fix pylint error * fix mypy error * fix for 2.7 * fix for 2.7 * Sleep 300 seconds to test reconnect * Remove Pypy from eventhub live test * Remove Pypy from eventhub live test * Add "Matrix:" * Fix bug passing timeout in seconds to uamqp * update checkpointstore HISTORY * [EventHubs] Update documentations (Azure#7603) * Update docs * Update iothub related description * Docs update * Small fix * Small fix * revert setup.py * Resolve Global Install Errors (Azure#7642) * swapping all dev_requirements to leverage the non-develop mode. opencensus-tracing will fail if azure-core is installed in dev mode. all the rest of the tests are ok with using non-dev, so this seems to be a good compromise * Setup core for unified pipelines. (Azure#7577) * Setup core for unified pipelines. * Added open census to the core pipeline (can be released independently). * Publish the packages artifact. (Azure#7643) * Enable template for unified templates. (Azure#7644) * Setup template for UP. * Fixed service directory. * ensure error_map is used consistently and correctly (Azure#7637) * switched vault_url to vault_endpoint across key vault * ran black on keyvault * reverted <<<HEAD additions to code * got rid of black changes that broke python 3.5
* [EventHubs] Eventhubs runtime metric (Azure#7431) * runtime metric init commit * evenhubts-runtime-metric implementation * Update code, and test and docstring * Add example code * Update property name * Update name to last enqueued event properties * Eventhubs remove iot and linkredirect (Azure#7450) * Remove iothub/link-redirect related code * Remove self._running from consumer and producer * Remove IoT related params "operation" and "device" * Remove exception from close() * add iterator long running test * small bug fix * small bug fix * Fix connection properties bug and format * Changed product to azure-eventhub in user agent * Fix a type hint * Improve stress script * Print to console configurable * small changes * Disable tracking last enqueued event properties for uamqp 1.2.2 * use different consumer group * fix an issue about consumer group * fix an issue about consumer group * Fix a get_properties bug * [EventHubs] Performance improvement (Azure#7533) * performance improvement * batch offset * Review feedback * Remove lazy parse in EventData * Add annotation assertions * Small fix for missing annotation initialization (Azure#7575) * Python EventHubs test code revamp (Azure#7594) * fix pylint error * fix mypy error * fix for 2.7 * fix for 2.7 * Sleep 300 seconds to test reconnect * Remove Pypy from eventhub live test * Remove Pypy from eventhub live test * Add "Matrix:" * Fix bug passing timeout in seconds to uamqp * update checkpointstore HISTORY * [EventHubs] Update documentations (Azure#7603) * Update docs * Update iothub related description * Docs update * Small fix * Small fix * revert setup.py
* switched vault_url to vault_endpoint across key vault * moved enabled and tags to kwargs * changed the name parameter for get_policy and update_policy to certificate_name * updated changelog * [Core] Updated release notes (Azure#7627) * Core doc fix (Azure#7638) * Fix azure-core examples * Fix azure-core Readme * Simplify doc * Kill common.py file * Remove tracing.ext * update app config doc (Azure#7618) * update app config doc * move match_condition to internal * [AutoPR] netapp/resource-manager (Azure#7092) * Generated from 34d744550227bee7b380604d38fc1de6c9af6397 (Azure#7089) Update netapp.json * changelog for netapp * ANF-374 update for R5 RP version 2019-07-01 API * ANF-374 update for R5 RP version 2019-07-01 API (Azure#7460) * ANF-374 update for R5 RP version 2019-07-01 API * ANF-374 update for R5 RP version 2019-07-01 API * ANF-374 update for R5 RP version 2019-07-01 API * extending pypy timeout. (Azure#7640) * Eventhubs preview4 merge to master (Azure#7605) * [EventHubs] Eventhubs runtime metric (Azure#7431) * runtime metric init commit * evenhubts-runtime-metric implementation * Update code, and test and docstring * Add example code * Update property name * Update name to last enqueued event properties * Eventhubs remove iot and linkredirect (Azure#7450) * Remove iothub/link-redirect related code * Remove self._running from consumer and producer * Remove IoT related params "operation" and "device" * Remove exception from close() * add iterator long running test * small bug fix * small bug fix * Fix connection properties bug and format * Changed product to azure-eventhub in user agent * Fix a type hint * Improve stress script * Print to console configurable * small changes * Disable tracking last enqueued event properties for uamqp 1.2.2 * use different consumer group * fix an issue about consumer group * fix an issue about consumer group * Fix a get_properties bug * [EventHubs] Performance improvement (Azure#7533) * performance improvement * batch offset * Review feedback * Remove lazy parse in EventData * Add annotation assertions * Small fix for missing annotation initialization (Azure#7575) * Python EventHubs test code revamp (Azure#7594) * fix pylint error * fix mypy error * fix for 2.7 * fix for 2.7 * Sleep 300 seconds to test reconnect * Remove Pypy from eventhub live test * Remove Pypy from eventhub live test * Add "Matrix:" * Fix bug passing timeout in seconds to uamqp * update checkpointstore HISTORY * [EventHubs] Update documentations (Azure#7603) * Update docs * Update iothub related description * Docs update * Small fix * Small fix * revert setup.py * Resolve Global Install Errors (Azure#7642) * swapping all dev_requirements to leverage the non-develop mode. opencensus-tracing will fail if azure-core is installed in dev mode. all the rest of the tests are ok with using non-dev, so this seems to be a good compromise * Setup core for unified pipelines. (Azure#7577) * Setup core for unified pipelines. * Added open census to the core pipeline (can be released independently). * Publish the packages artifact. (Azure#7643) * Enable template for unified templates. (Azure#7644) * Setup template for UP. * Fixed service directory. * ensure error_map is used consistently and correctly (Azure#7637) * switched vault_url to vault_endpoint across key vault * ran black on keyvault * reverted <<<HEAD additions to code * got rid of black changes that broke python 3.5
* [EventHubs] Eventhubs runtime metric (#7431) * runtime metric init commit * evenhubts-runtime-metric implementation * Update code, and test and docstring * Add example code * Update property name * Update name to last enqueued event properties * Eventhubs remove iot and linkredirect (#7450) * Remove iothub/link-redirect related code * Remove self._running from consumer and producer * Remove IoT related params "operation" and "device" * Remove exception from close() * add iterator long running test * small bug fix * small bug fix * Fix connection properties bug and format * Changed product to azure-eventhub in user agent * Fix a type hint * Improve stress script * Print to console configurable * small changes * Disable tracking last enqueued event properties for uamqp 1.2.2 * use different consumer group * fix an issue about consumer group * fix an issue about consumer group * Fix a get_properties bug * [EventHubs] Performance improvement (#7533) * performance improvement * batch offset * Review feedback * Remove lazy parse in EventData * Add annotation assertions * Small fix for missing annotation initialization (#7575) * Python EventHubs test code revamp (#7594) * fix pylint error * fix mypy error * fix for 2.7 * fix for 2.7 * Sleep 300 seconds to test reconnect * Remove Pypy from eventhub live test * Remove Pypy from eventhub live test * Add "Matrix:" * Fix bug passing timeout in seconds to uamqp * update checkpointstore HISTORY * [EventHubs] Update documentations (#7603) * Update docs * Update iothub related description * Docs update * Small fix * Small fix * revert setup.py
* switched vault_url to vault_endpoint across key vault * moved enabled and tags to kwargs * changed the name parameter for get_policy and update_policy to certificate_name * updated changelog * [Core] Updated release notes (#7627) * Core doc fix (#7638) * Fix azure-core examples * Fix azure-core Readme * Simplify doc * Kill common.py file * Remove tracing.ext * update app config doc (#7618) * update app config doc * move match_condition to internal * [AutoPR] netapp/resource-manager (#7092) * Generated from 34d744550227bee7b380604d38fc1de6c9af6397 (#7089) Update netapp.json * changelog for netapp * ANF-374 update for R5 RP version 2019-07-01 API * ANF-374 update for R5 RP version 2019-07-01 API (#7460) * ANF-374 update for R5 RP version 2019-07-01 API * ANF-374 update for R5 RP version 2019-07-01 API * ANF-374 update for R5 RP version 2019-07-01 API * extending pypy timeout. (#7640) * Eventhubs preview4 merge to master (#7605) * [EventHubs] Eventhubs runtime metric (#7431) * runtime metric init commit * evenhubts-runtime-metric implementation * Update code, and test and docstring * Add example code * Update property name * Update name to last enqueued event properties * Eventhubs remove iot and linkredirect (#7450) * Remove iothub/link-redirect related code * Remove self._running from consumer and producer * Remove IoT related params "operation" and "device" * Remove exception from close() * add iterator long running test * small bug fix * small bug fix * Fix connection properties bug and format * Changed product to azure-eventhub in user agent * Fix a type hint * Improve stress script * Print to console configurable * small changes * Disable tracking last enqueued event properties for uamqp 1.2.2 * use different consumer group * fix an issue about consumer group * fix an issue about consumer group * Fix a get_properties bug * [EventHubs] Performance improvement (#7533) * performance improvement * batch offset * Review feedback * Remove lazy parse in EventData * Add annotation assertions * Small fix for missing annotation initialization (#7575) * Python EventHubs test code revamp (#7594) * fix pylint error * fix mypy error * fix for 2.7 * fix for 2.7 * Sleep 300 seconds to test reconnect * Remove Pypy from eventhub live test * Remove Pypy from eventhub live test * Add "Matrix:" * Fix bug passing timeout in seconds to uamqp * update checkpointstore HISTORY * [EventHubs] Update documentations (#7603) * Update docs * Update iothub related description * Docs update * Small fix * Small fix * revert setup.py * Resolve Global Install Errors (#7642) * swapping all dev_requirements to leverage the non-develop mode. opencensus-tracing will fail if azure-core is installed in dev mode. all the rest of the tests are ok with using non-dev, so this seems to be a good compromise * Setup core for unified pipelines. (#7577) * Setup core for unified pipelines. * Added open census to the core pipeline (can be released independently). * Publish the packages artifact. (#7643) * Enable template for unified templates. (#7644) * Setup template for UP. * Fixed service directory. * ensure error_map is used consistently and correctly (#7637) * switched vault_url to vault_endpoint across key vault * ran black on keyvault * reverted <<<HEAD additions to code * got rid of black changes that broke python 3.5