bug fixes after June 6 release #56

weidongh · 2012-08-21T00:39:44Z

This contains all the bug fixes after first official publish on June 7. We also added more tests for the coverage and doctests.

The issues fixed are listed here: https://github.com/WindowsAzure/azure-sdk-for-python/issues?page=1&state=closed

MergeDevToMaster

pip install pyazure -> azure

Fix for issue Azure#56

* Initial commit * Initial commit * Initial commit * event hub client * Update README.md * Update README.md Fix typos * Memory leak * Support timestamp filter * Support timestamp filter * Update README.md * Add sender and refactor * Added abstract classes Todo - Migrate Base Class Wireframes - Migrate Azure Classes * First draft of class wires directly ported from .net (might be some minor gaps) * send example * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Skip batch tests pending uAMQP release

* Migrate Event Hub SDK to central Repo (#4764) * Initial commit * Initial commit * Initial commit * event hub client * Update README.md * Update README.md Fix typos * Memory leak * Support timestamp filter * Support timestamp filter * Update README.md * Add sender and refactor * Added abstract classes Todo - Migrate Base Class Wireframes - Migrate Azure Classes * First draft of class wires directly ported from .net (might be some minor gaps) * send example * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Skip batch tests pending uAMQP release * Fixed iothub_recv sample Changes from PR: https://github.com/Azure/azure-event-hubs-python/pull/109/files * Review feedback

* Move to under sdk * Remove policies * Remove debugging files * Rename Offset to EventPosition * make tests a namespace package * Revised test receive for new code * Revised test send for track two * Update async code from sync * Revise async receive and send live test for track2 * Use uamqp 1.2 * Resolve code review feedback * add queue_message to async sender * send_batch receives both list and iterator * Update after adp review * send accepts EventData, list, iteratable * Event Hub Track 2 (#5) * Initial commit * Initial commit * Initial commit * event hub client * Update README.md * Update README.md Fix typos * Memory leak * Support timestamp filter * Support timestamp filter * Update README.md * Add sender and refactor * Added abstract classes Todo - Migrate Base Class Wireframes - Migrate Azure Classes * First draft of class wires directly ported from .net (might be some minor gaps) * send example * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Add get_partition_info in Event Hub * add get_partition_info * Add telemetry information to the connection properties * Disable smart split in batch message * change epoch to exclusive_receiver_priority * fix small problem * remove uamqp dependency * Eventhub track2 (#6) * Initial commit * Initial commit * Initial commit * event hub client * Update README.md * Update README.md Fix typos * Memory leak * Support timestamp filter * Support timestamp filter * Update README.md * Add sender and refactor * Added abstract classes Todo - Migrate Base Class Wireframes - Migrate Azure Classes * First draft of class wires directly ported from .net (might be some minor gaps) * send example * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Add get_partition_info in Event Hub * add get_partition_info * Add telemetry information to the connection properties * Disable smart split in batch message * 1. Add amqp over websocket test 2. Add proxy sample 3. Update some comment and code * Changes from cross-lang * Change debug to network_tracing * Sync Client Constructor * auto_reconnect True and keep_alive None * consumer_group $default * hide open() * partition -> partition_id * credentials -> credential in init * set running=true after opened * Eventhub track2 - Update livetest (#7) * Update README.md * Update README.md Fix typos * Memory leak * Support timestamp filter * Support timestamp filter * Update README.md * Add sender and refactor * Added abstract classes Todo - Migrate Base Class Wireframes - Migrate Azure Classes * First draft of class wires directly ported from .net (might be some minor gaps) * send example * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Add get_partition_info in Event Hub * add get_partition_info * Add telemetry information to the connection properties * Disable smart split in batch message * 1. Add amqp over websocket test 2. Add proxy sample 3. Update some comment and code * update some test code * Add __str__ to EventData * Update test code * Add eh error classes * EventHubError extends AzureError * Fix EventPosition default value issue * change $default to $Default * Handle TokenAuthError * wait for ready in _reconnect * fix get_partition_ids issue * Fix reconnect issue * small fix * fix async live test * Eventhub track2 Live test update (#8) * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Add get_partition_info in Event Hub * add get_partition_info * Add telemetry information to the connection properties * Disable smart split in batch message * 1. Add amqp over websocket test 2. Add proxy sample 3. Update some comment and code * update some test code * Add __str__ to EventData * Update test code * Update event position * Update live test * Update reconnect live test * Update too large data size * debug->network_tracing * Negative test fix * Remove partition_key, send with batching_label * Fix review problems * Fix a log issue * fix get_partition_properties bug * add client properties live test * Revised setup.py for track 2

…its type is byte. (#56)

* Initial commit * Initial commit * Initial commit * wip: initial work - incomplete * Copy code from https://github.com/Azure/azure-cli/tree/master/src/azure-cli-testsdk * Add vcrpy version from azure-cli * Add VSCode ignores * Add Azure imports * Start converting CLI commands to SDK calls * Minor proofreading * Add required common and mock modules * Add deps, update pkgs and url * Package structure * Add more editor ignores * Remove azure deps and nonexistent HISTORY ref * Remove Azure mgmt dependencies * Remove azure.common dependency * Add dependency on six * Remove CLI-specific stuff * Remove CLI exception * Remove most CLI-centric patches * Remove older CLI-centric base class * Add sleep patch back in * First steps toward a unified config * Config and no-record headers Add config object to ScenarioTest Add fake header for deactivating recording Align cmdline options with vcr.py options * Move recording-deactivation header detection to right place * Add way to set default config file from scenario test * Remove 'CLI' from env vars * Reinstate patch_long_run_operation_delay * New record disabling mechanism * Modify config options for better backwards compatibility * Ignore more VS stuff * Remove jmespath dependency, update vcrpy * Remove checkers * Remove unuseds, make patches/processors kwargs * Remove checkers, used only for CLI * Update import and instance var name * Remove unused imports * line organization * Update version number and vcrpy version * Add a little more to README, add setup.cfg * Add unit tests for TestConfig * Add travis.yml and update dependencies * Update author email * Dummy push to try to trigger a Travis build * Another dummy commit for Travis * Switch to README.rst * Update Travis tag pointer * Use io to get encoding arg when reading README * Remove unnecessary import * Expose most everything in top-level namespace * ScenarioTest -> ReplayableTest; update version * Update version * Remove unused os.path import * Fix super call * Fix super call in setUp * Update ReplayableTest import * Use earlier vcrpy, update version num * Don't fix vcrpy version * Drop "cli" from default random name prefix * Add deployment to Travis * Specify universal wheel * Bump version to 0.2.2 for testing CI release process * Let preparer subclasses override create_random_name * De-"privatize" mock_in_unit_test and expose it * Bump version to 0.3.0 * Set disable_recording from kwarg * Don't treat mock_in_unit_test as a unittest * Remove skip for mock_in_unit_test, specify test dir * Update version numbers * Remove extraneous requirements * Remove outdated README.md stuff * Update install for new requirements.txt * Add 3.6-dev to pythons * Update code style and run pylint in Travis * Integrate with codecov.io * Add test cover IntegrationTestBase Also check in IntelliJ IDEA workspace configuration just in case someone else is using PyCharm * Add test case for LiveTest constructor * Add tests cover create_random_name * Add test covering get_sha1_hash * Add test cover RecordingProcessor base class * Add test covering SubscriptionRecordingProcessor * Configure code coverage using configuration file * Use unittest instead of nosetests to drive the automation * Fix a few code style issues * Rename coveragerc file * Add AccessTokenProcessor * Split intro and overview into clauses * Back to paragraphs * Update intro, add vcr link * Update intro paragraph to be more general * Remove reference to command modules * Update test and module refs; clarify VCR.py role * Remove outdated reference to class with builtin preparer * Replace intro text Use intro stolen from defunct recording_vcr_tests.md * s/ScenarioTest/ReplayableTest/ * Add note about semantic linefeeds * Make semantic linefeeds note an HTML comment * Remove CLI examples and add links to consumers * Remove doc for legacy test case class * Finish sentence * Add subclass kwarg information * Fix subscription ID removal and test new cases * Fix no-self-use linter complaint * Downgrade the version of vcrpy dependency from 1.11.1 to 1.11.0 due to a sympton similar to this kevin1024/vcrpy#318 * Bump version to 0.5.0 to prepare for release * Fix import on Py3 not-TravisCI * Fix recordmode * Tight config test * url parsing fix * fixed pylint import order * pylint import fix * added record processor for slashes * changed processor name * Bump version 0.5.0 => 0.5.1 * Fix SubscriptionIdReplacer * Update version 0.5.1 => 0.5.2 * Update __init__.py * Detect leaked LRO poller * PyLint happiness * Improve error message * wip * fix error * use base 64 * simplify * add comments * fix tests * fix lint error * use newer pylint to avoid invalid line errors * add a new test * address review feedback * address review feedback * support large response payload * remove useless single quots from the error message * setup: update version * do not use preparer model for allowing large payload * address review feedback * Support Autorest.Python 3.x LRO leak (#42) * Support Autorest.Python 3.x LRO leak * PyLint happyness * Bump version to 0.5.5 * Update the resource removal sequence Also: 1. Adopt py.test over nosetest 2. Cleanup some test code * remove x-ms-authorization-auxiliary header * fix lint error * update version * Add some CI tools to devtools * Readme update * Some tests configuration * Use Pytest as test launcher * PyLint fixes * CI tools tests are Py3.6 only * PyLint CI tools only if 3.6 * CI tools tests recording * Dont't Pylint the tests * Fix incorrect usage of os.environ * Make tests X-platform valid * Fix test dependent of traceback * Robust preparer testing * PyGithub 1.40 * Release 1.1.0 * Allow clone_to_path to have both PR and SHA1 (#49) * Allow clone_to_path to have both PR and SHA1 * Fix test * Update github_tools.py (#55) * SingleValueReplacer.process_request properly decodes request.body if its type is byte. (#56) * Remove pointless files * Move to tools * Update dev_requirement file * Don't record requests to AAD OAuth2 v2.0 endpoint * Use devtools from repo, not from PyPI * Fix mock dep

* remove async_ops * EventHubs track2 starter (#5330) * Move to under sdk * Remove policies * Remove debugging files * Rename Offset to EventPosition * make tests a namespace package * Revised test receive for new code * Revised test send for track two * Update async code from sync * Revise async receive and send live test for track2 * Use uamqp 1.2 * Resolve code review feedback * add queue_message to async sender * send_batch receives both list and iterator * Eventhubs track2 python main issues (#5575) * Move to under sdk * Remove policies * Remove debugging files * Rename Offset to EventPosition * make tests a namespace package * Revised test receive for new code * Revised test send for track two * Update async code from sync * Revise async receive and send live test for track2 * Use uamqp 1.2 * Resolve code review feedback * add queue_message to async sender * send_batch receives both list and iterator * Update after adp review * send accepts EventData, list, iteratable * Event Hub Track 2 (#5) * Initial commit * Initial commit * Initial commit * event hub client * Update README.md * Update README.md Fix typos * Memory leak * Support timestamp filter * Support timestamp filter * Update README.md * Add sender and refactor * Added abstract classes Todo - Migrate Base Class Wireframes - Migrate Azure Classes * First draft of class wires directly ported from .net (might be some minor gaps) * send example * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Add get_partition_info in Event Hub * add get_partition_info * Add telemetry information to the connection properties * Disable smart split in batch message * change epoch to exclusive_receiver_priority * fix small problem * remove uamqp dependency * Eventhub track2 (#6) * Initial commit * Initial commit * Initial commit * event hub client * Update README.md * Update README.md Fix typos * Memory leak * Support timestamp filter * Support timestamp filter * Update README.md * Add sender and refactor * Added abstract classes Todo - Migrate Base Class Wireframes - Migrate Azure Classes * First draft of class wires directly ported from .net (might be some minor gaps) * send example * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Add get_partition_info in Event Hub * add get_partition_info * Add telemetry information to the connection properties * Disable smart split in batch message * 1. Add amqp over websocket test 2. Add proxy sample 3. Update some comment and code * Changes from cross-lang * Change debug to network_tracing * Sync Client Constructor * auto_reconnect True and keep_alive None * consumer_group $default * hide open() * partition -> partition_id * credentials -> credential in init * set running=true after opened * Eventhub track2 - Update livetest (#7) * Update README.md * Update README.md Fix typos * Memory leak * Support timestamp filter * Support timestamp filter * Update README.md * Add sender and refactor * Added abstract classes Todo - Migrate Base Class Wireframes - Migrate Azure Classes * First draft of class wires directly ported from .net (might be some minor gaps) * send example * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Add get_partition_info in Event Hub * add get_partition_info * Add telemetry information to the connection properties * Disable smart split in batch message * 1. Add amqp over websocket test 2. Add proxy sample 3. Update some comment and code * update some test code * Add __str__ to EventData * Update test code * Add eh error classes * EventHubError extends AzureError * Fix EventPosition default value issue * change $default to $Default * Handle TokenAuthError * wait for ready in _reconnect * fix get_partition_ids issue * Fix reconnect issue * small fix * fix async live test * Eventhub track2 Live test update (#8) * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Add get_partition_info in Event Hub * add get_partition_info * Add telemetry information to the connection properties * Disable smart split in batch message * 1. Add amqp over websocket test 2. Add proxy sample 3. Update some comment and code * update some test code * Add __str__ to EventData * Update test code * Update event position * Update live test * Update reconnect live test * Update too large data size * debug->network_tracing * Negative test fix * Remove partition_key, send with batching_label * Fix review problems * Fix a log issue * fix get_partition_properties bug * add client properties live test * Revised setup.py for track 2 * Error hierarchy, sample code and docstring (#5743) * Recover from fork repo * Packaging update of azure-eventhubs * Fix error message * update iterator example * Revert "Packaging update of azure-eventhubs" This reverts commit 56fc4f01126daa85956222ea6f2a992146349bee. * disable autorest auto update * Sender/Receiver -> EventSender/Receiver * Change _batching_label back to partition_key * Remove transfer examples * move async to async folder * Update docstring string, sample codes and test codes (#5793) * catch and process LinkRedirect * Add receiver iterator pytest * small fix of iterator example * add retrieval_time to partition prop * fix open and re-send bugs * small fixes * fix reconnect test case * close iterator when closing receiver * Misc changes for code review fix * client.py type hints * catch KeyboardInterrupt * add next() for 2.7 iterator * raise KeyboardInterrupt instead of exit() * Fix missing consumer group directory in EPH * Fix livetest code problem * history and readme * Update history and setup (#5902) * EventPosition.first_available_event -> earliest new_events_only -> latest * Change EventSender's event_position to be mandatory * Update uamqp shared_req to 1.2.0 * Disable network_tracing * update uamqp dependency ~=1.2.0 * Remove EventPosition helper functions * Names changed to EventHubConsumer/Producer * Avoid nested with statement * Skip forced reconnect test * Update naming in eventhub (consumer and producer). (#5984) * Remove azure-core requirement tentatively * Warn when eventhubs or storage teardown fails * add type hints comments * add azure identity in dev_requirements * put TransportType in __init__ directly * change EventData's offset to be str, not EventPosition * remove from_iot_connection_string * small fix * docstring timeout from int to float * fix TransportType import * add pytest option sleep for reconnect test * Fix eventposition issue in receive test * fix constants import issue * remove azure identity dev_req * fix some example issues * iot string fix * Change filename for consumer and producer * Vendor azure-storage-blob in eventprocesshost (#6018) * Update storage dependency * Vendor storage blob v1.3.1 in eph * Add vendored blob-storage required pkg * Vendor storage 2.0.1 within EPH (#6031) * catch exception for mgmt_request * Update comment and code structure (#6042) * code review changes * Add python-dateutil in shared_requirements * Add aad credential env var to tests.yml * Change example code assertion for parallel running * Enable iothub receive test case * Revert "Enable iothub receive test case" This reverts commit 5abdcd780cfb2085472d73c8d57fff120bbbb322. * fix auth test error * change offset.value to offset * Fix an eventposition problem * Remove path append * trying removing the module init py within tests * Separate MockEventProcessor to a different file * remove tests to path * trying a run based on a nested conftest to establish the async fixtures without shattering on python 2.7

* Initial commit * Initial commit * Initial commit * wip: initial work - incomplete * Copy code from https://github.com/Azure/azure-cli/tree/master/src/azure-cli-testsdk * Add vcrpy version from azure-cli * Add VSCode ignores * Add Azure imports * Start converting CLI commands to SDK calls * Minor proofreading * Add required common and mock modules * Add deps, update pkgs and url * Package structure * Add more editor ignores * Remove azure deps and nonexistent HISTORY ref * Remove Azure mgmt dependencies * Remove azure.common dependency * Add dependency on six * Remove CLI-specific stuff * Remove CLI exception * Remove most CLI-centric patches * Remove older CLI-centric base class * Add sleep patch back in * First steps toward a unified config * Config and no-record headers Add config object to ScenarioTest Add fake header for deactivating recording Align cmdline options with vcr.py options * Move recording-deactivation header detection to right place * Add way to set default config file from scenario test * Remove 'CLI' from env vars * Reinstate patch_long_run_operation_delay * New record disabling mechanism * Modify config options for better backwards compatibility * Ignore more VS stuff * Remove jmespath dependency, update vcrpy * Remove checkers * Remove unuseds, make patches/processors kwargs * Remove checkers, used only for CLI * Update import and instance var name * Remove unused imports * line organization * Update version number and vcrpy version * Add a little more to README, add setup.cfg * Add unit tests for TestConfig * Add travis.yml and update dependencies * Update author email * Dummy push to try to trigger a Travis build * Another dummy commit for Travis * Switch to README.rst * Update Travis tag pointer * Use io to get encoding arg when reading README * Remove unnecessary import * Expose most everything in top-level namespace * ScenarioTest -> ReplayableTest; update version * Update version * Remove unused os.path import * Fix super call * Fix super call in setUp * Update ReplayableTest import * Use earlier vcrpy, update version num * Don't fix vcrpy version * Drop "cli" from default random name prefix * Add deployment to Travis * Specify universal wheel * Bump version to 0.2.2 for testing CI release process * Let preparer subclasses override create_random_name * De-"privatize" mock_in_unit_test and expose it * Bump version to 0.3.0 * Set disable_recording from kwarg * Don't treat mock_in_unit_test as a unittest * Remove skip for mock_in_unit_test, specify test dir * Update version numbers * Remove extraneous requirements * Remove outdated README.md stuff * Update install for new requirements.txt * Add 3.6-dev to pythons * Update code style and run pylint in Travis * Integrate with codecov.io * Add test cover IntegrationTestBase Also check in IntelliJ IDEA workspace configuration just in case someone else is using PyCharm * Add test case for LiveTest constructor * Add tests cover create_random_name * Add test covering get_sha1_hash * Add test cover RecordingProcessor base class * Add test covering SubscriptionRecordingProcessor * Configure code coverage using configuration file * Use unittest instead of nosetests to drive the automation * Fix a few code style issues * Rename coveragerc file * Add AccessTokenProcessor * Split intro and overview into clauses * Back to paragraphs * Update intro, add vcr link * Update intro paragraph to be more general * Remove reference to command modules * Update test and module refs; clarify VCR.py role * Remove outdated reference to class with builtin preparer * Replace intro text Use intro stolen from defunct recording_vcr_tests.md * s/ScenarioTest/ReplayableTest/ * Add note about semantic linefeeds * Make semantic linefeeds note an HTML comment * Remove CLI examples and add links to consumers * Remove doc for legacy test case class * Finish sentence * Add subclass kwarg information * Fix subscription ID removal and test new cases * Fix no-self-use linter complaint * Downgrade the version of vcrpy dependency from 1.11.1 to 1.11.0 due to a sympton similar to this kevin1024/vcrpy#318 * Bump version to 0.5.0 to prepare for release * Fix import on Py3 not-TravisCI * Fix recordmode * Tight config test * url parsing fix * fixed pylint import order * pylint import fix * added record processor for slashes * changed processor name * Bump version 0.5.0 => 0.5.1 * Fix SubscriptionIdReplacer * Update version 0.5.1 => 0.5.2 * Update __init__.py * Detect leaked LRO poller * PyLint happiness * Improve error message * wip * fix error * use base 64 * simplify * add comments * fix tests * fix lint error * use newer pylint to avoid invalid line errors * add a new test * address review feedback * address review feedback * support large response payload * remove useless single quots from the error message * setup: update version * do not use preparer model for allowing large payload * address review feedback * Support Autorest.Python 3.x LRO leak (Azure#42) * Support Autorest.Python 3.x LRO leak * PyLint happyness * Bump version to 0.5.5 * Update the resource removal sequence Also: 1. Adopt py.test over nosetest 2. Cleanup some test code * remove x-ms-authorization-auxiliary header * fix lint error * update version * Add some CI tools to devtools * Readme update * Some tests configuration * Use Pytest as test launcher * PyLint fixes * CI tools tests are Py3.6 only * PyLint CI tools only if 3.6 * CI tools tests recording * Dont't Pylint the tests * Fix incorrect usage of os.environ * Make tests X-platform valid * Fix test dependent of traceback * Robust preparer testing * PyGithub 1.40 * Release 1.1.0 * Allow clone_to_path to have both PR and SHA1 (Azure#49) * Allow clone_to_path to have both PR and SHA1 * Fix test * Update github_tools.py (Azure#55) * SingleValueReplacer.process_request properly decodes request.body if its type is byte. (Azure#56) * Remove pointless files * Move to tools * Update dev_requirement file * Don't record requests to AAD OAuth2 v2.0 endpoint * Use devtools from repo, not from PyPI * Fix mock dep

* remove async_ops * EventHubs track2 starter (#5330) * Move to under sdk * Remove policies * Remove debugging files * Rename Offset to EventPosition * make tests a namespace package * Revised test receive for new code * Revised test send for track two * Update async code from sync * Revise async receive and send live test for track2 * Use uamqp 1.2 * Resolve code review feedback * add queue_message to async sender * send_batch receives both list and iterator * Eventhubs track2 python main issues (#5575) * Move to under sdk * Remove policies * Remove debugging files * Rename Offset to EventPosition * make tests a namespace package * Revised test receive for new code * Revised test send for track two * Update async code from sync * Revise async receive and send live test for track2 * Use uamqp 1.2 * Resolve code review feedback * add queue_message to async sender * send_batch receives both list and iterator * Update after adp review * send accepts EventData, list, iteratable * Event Hub Track 2 (#5) * Initial commit * Initial commit * Initial commit * event hub client * Update README.md * Update README.md Fix typos * Memory leak * Support timestamp filter * Support timestamp filter * Update README.md * Add sender and refactor * Added abstract classes Todo - Migrate Base Class Wireframes - Migrate Azure Classes * First draft of class wires directly ported from .net (might be some minor gaps) * send example * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Add get_partition_info in Event Hub * add get_partition_info * Add telemetry information to the connection properties * Disable smart split in batch message * change epoch to exclusive_receiver_priority * fix small problem * remove uamqp dependency * Eventhub track2 (#6) * Initial commit * Initial commit * Initial commit * event hub client * Update README.md * Update README.md Fix typos * Memory leak * Support timestamp filter * Support timestamp filter * Update README.md * Add sender and refactor * Added abstract classes Todo - Migrate Base Class Wireframes - Migrate Azure Classes * First draft of class wires directly ported from .net (might be some minor gaps) * send example * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Add get_partition_info in Event Hub * add get_partition_info * Add telemetry information to the connection properties * Disable smart split in batch message * 1. Add amqp over websocket test 2. Add proxy sample 3. Update some comment and code * Changes from cross-lang * Change debug to network_tracing * Sync Client Constructor * auto_reconnect True and keep_alive None * consumer_group $default * hide open() * partition -> partition_id * credentials -> credential in init * set running=true after opened * Eventhub track2 - Update livetest (#7) * Update README.md * Update README.md Fix typos * Memory leak * Support timestamp filter * Support timestamp filter * Update README.md * Add sender and refactor * Added abstract classes Todo - Migrate Base Class Wireframes - Migrate Azure Classes * First draft of class wires directly ported from .net (might be some minor gaps) * send example * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Add get_partition_info in Event Hub * add get_partition_info * Add telemetry information to the connection properties * Disable smart split in batch message * 1. Add amqp over websocket test 2. Add proxy sample 3. Update some comment and code * update some test code * Add __str__ to EventData * Update test code * Add eh error classes * EventHubError extends AzureError * Fix EventPosition default value issue * change $default to $Default * Handle TokenAuthError * wait for ready in _reconnect * fix get_partition_ids issue * Fix reconnect issue * small fix * fix async live test * Eventhub track2 Live test update (#8) * Set allowed sasl mechs * Remove client.py * Receiver update * Add dummy send api * logging updates * Error handling, reconnect and logging * Add app properties to event data * unbind transport on connection close * timestamp filter on py2 * module version * Reconnect once when link/session/connection close * Add SessionPolicy * Add client info * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * Updates - Cleaned wireframes to be PEP compliant - Implemented single partition pump and single event_hub partition pump scenario Todo - Add Unit Tests for partition pump and event hub partition pump - Implement Partition Manager - Implement Checkpointing and Lease Managment * run client in non-blocking mode * Added unit testing * Implemented the following functionality - Azure_storage_checkpoint_manager - AzureBlobLease isExpired Todo Implement partition manager Implement partition context Test full implementation * Implemented Processing of First Epoh Todo - Fix lease bug that is breaking subsequent epochs * Changes - Completed End to End EPH Flow - Removed storage dependancy on downloading full blob to check lease state Todo - Add thread and queue for checking lease state and other storage operations - Ensure eventhub client shuts down properly - Find way to update partition pumps without restarting them - Other optimizations * Move examples out * Changes - Added thread pool executor to enable conncurent execution of partitions - Removed partition pump dependency on max_batch Todo - Ensure eventhub client shuts down properly (This is causing errors) - Add thread pool for making checkpoint code conccurent - Add thread and queue for checking lease state and other storage operations to enable async - Find way to reassign active partition pumps without restarting them - Other optimizations * Add async receive * Changes - Added logs - Fixed error causing client to prematurely shutdown * Manual link flow control for async receive * Workaround for stuck async receiver * Local variable names * Changes - Optimized logging and comments Todo - Add concurecny mechanim for azure storage - Depricate partition pump event queue and update to latest version of the client * Create Dockerfile * Stuck async receiver * credit keeps increasing in async receiver * Changes - Added asnyc event hub client support - Optimized logging and comments Todo - Add concurecny mechanim for azure storage * Updated docker file as requested * Added EPH example * Fix hardcoded HTTP header * Made suggested changes * Bug fix - Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one. * Updated loop naming convention to be consistent * Added option to pass asyncio event_loop to eph * Updated docker file * Fixed critical bug with partition manager and aquirec mechanisiims Todo : Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts * Bug fixes - Fixed bug where closing a pump closed a host - Fixed bug where error partitioned were not removed - Fixed bug where leases were renewed at an incorrect interval * Updated file headers Removed author reference * - Fixed bug in eph example that caused host to terminate prematurely - Made the lease renewal and checkpoint creation "multithreaded" * Increase the size of the connection pool The default connection pool size was too small for scenarios where multiple partitions were handled by one EventProcessorHost. If the amount of partitions handled is large, we might end up doing very many connections at the same time due to the multi-threaded blob-handling. For this reason, you might hit the OS limits that restrict the number of open files per process that in MacOS is not very big. This can be worked around with something like: `ulimit -n 2560` * Decrease info logging verbosity * added ability to toggle pump shutdown when all messages on a pump are processed. * Install also eventhubsprocessor * Default to keeping the pumps It is more optimal to keep the pumps alive even if there are no messages so that it is faster to pickup when messages start to arrive. * Pipe and event injector for Windows * Event injector updates * EHClient refactoring. EHClient leaks. Sender part 1. * Send support * ren eventhubsprocessor eventprocessorhost * Changes - Added event hub config to simplify installation story * Changes - Added optional eventprocessor_params for passing context to the event processor - Made the storage manager mandatatory * Fix memory leaks * logging * Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging. * tests * test: recv from multiple partitions * test utility * logging update * Support callback based send for high throughput * Workaroud memory issue in proton.reactor.ApplicationEvent * renamed eventprocessor to eventprocessorhost for consistency * updated docker file * fixed typo in url * Added amqp port to address * Updated sample documentation since url is auto encoded by config * Updated docs * Implement timeout for send * Async sender and example * Close injector pipe * Use send timer to also check queued messages * Add partition pump loop to partition_context This gives the EventProcessor access to the partition_pump loop object. This way if One desires to run synchronous code inside process_events_async one can utilize the loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla) * Include details in send error * Release deliveries when sender is closed * added validation to unquoted sas key * added support for custom eventhub client prefetch size * Update README.md * Update README.md * Added Docker instructions and fixed Dockerfile (#18) * Removed Dockerfile from the main folder and fixed Dockerfile example * Added build and run Dockerfile documentation * Update Readme * Removed rm qpid-proton folder * Removed /usr/share copy * Disallow a sender/receiver to be registered more than once * Make everything async in EPH I have removed all usage of threads thoroughout the code. Using threads to run pumps etc. Causes async code written into the event-processor to become caotic (you need to follow which loop is currently being used in the call to prevent loops not being found or using the wrong loop (There is the main loop and then loops that are created inside threads) Things become caotic when the event processor is being called by objects that run under different loops. So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls. Also changed the bla_async methods to not block. this way, when calling open_async for the the event-processor-host, the command will exit once the EPH is started. Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running (Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts on the eventhubs connections. * Started removing proton code * Removed most of proton _impl * Removed more code * Working sender * Updates to sender * Added some tests/samples * Some progress on clients * Fixed samples * Added azure namespace * #25 Partition key cannot be set for events * Updated version * Updated README * Renamed package to eventhub * Started EPH modifications * Updated imports * Fixed target urls * Updated logging * Updated async message receive * updated test imports * Added mgmt call to get eh info * Updated samples * Updated receive test * Added send and receive test clients * Updated uamqp dependency * Merged updates from dev * Fixed typos * Updated EPH sample * Started docstrings * Converted tests to pytest * Updates to batch receive * Started adding docstrings * More docstrings * bumped version * Started porting test suite * More tests and improvements * Moved eph tests * Some sample cleanup * Some test updates * Some test restructure * Docstring cleanup * Fixed some merge artifacts * Fixed formatting error * Removed delivery count * Nested package directory * Support custom URL suffix * Support custom URL suffix * Support for EventData device ID * Reverted nested directory * Updated release notes * Workaround for partitionkey * Finished partition key workaround * beta2 fixes * pylint fixes * Trigger CI * Test fixes * Added package manifest * Added warning for Python 2.7 support Support for issues #36 and #38 * Started adding scenario tests * More test scenarios * Better docstring formatting * Started iothub support * Fixed long running test * Fixed typo and memory leak * Restructure * IoThub support * Updates for RC1 release * Fix long running test * Docstring and sample cleanups * Working on error retry * Improved error processing * Fixed partition manager * Progress on IotHub error * Some test updates * Updated uamqp dependency * Restructure for independent connections * Added HTTP proxy support Fix for issue #41 * Fixed some tests + samples * pylint fixes * bumped version * Added keepalive config and some eph fixes * Made reconnect configurable * Added more EPH options * Bumped version * Pylint fix * Pylint fix * Added send and auth timeouts * Changed log formatting. Retry on reconnect * Pylint fixes * Renamed internal async module * Updated send example to match recv Fix for issue #56 * Added build badge to readme * Fix for repeat startup * Added more storage connect options to EPH * Bumped version * Handler blocked until client started * Added event data methods * Fix pylint * Fix 3.7 CI * Fix 3.7 CI * Updated pylint version * Pylint fixes * Updated README * Fixed readme badge refresh * Fixed bug in Azure namespace package * Updated manifest * Parse enqueued time as UTC Fixes #72. * Updates for release 1.2.0 (#81) * Made setup 2.7 compatible * Separated async tests * Support 2.7 types * Bumped version * Added non-ascii tests * Fix CI * Fix Py27 pylint * Added iot sample * Updated sender/receiver client opening * bumped version * Updated tests * Fixed test name * Fixed test env settings * Skip eph test * Updates for v1.3.0 (#91) * Added support for storing the state of the Event Processor along the Checkpoint. Both Checkpoint and the EP state are stored as pickled objects. * Fixing pylint complaints. * Switched from pickle back to JSON for lease persistence. * Fixes bug when accessing leases that don't contain EP context. Also, minor renaming. * Better SAS token support * Fixed pylint * Improved auth error handling * Test stabilization * Improved stored EPH context * Updated EPH context storing * Skip test on OSX * Skip tests on OSX Fail due to large message body bug. * Some cleanup * Fixed error handling * Improved SAS token parsing * Fixed datetime offset (#99) * Fixed datetime offset * Updated pylint * Removed 3.4 pylint pass * Fixed bug in error handling (#100) * Migrate event hub sdk to central repo 1. add verifiable code snippets into docstring 2. update readme according to the template 3. add livetest mark and config 4. optimize code layout/structure * 1. document formatting 2. separate async/sync example tests * Fix build error: 1. uamqp dependency mismatch 2. rename test_examples in eventhub to avoid mismatch * This should fix build error * remove tests import and add sys path to solve build error * add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed * Add get_partition_info in Event Hub * add get_partition_info * Add telemetry information to the connection properties * Disable smart split in batch message * 1. Add amqp over websocket test 2. Add proxy sample 3. Update some comment and code * update some test code * Add __str__ to EventData * Update test code * Update event position * Update live test * Update reconnect live test * Update too large data size * debug->network_tracing * Negative test fix * Remove partition_key, send with batching_label * Fix review problems * Fix a log issue * fix get_partition_properties bug * add client properties live test * Revised setup.py for track 2 * Error hierarchy, sample code and docstring (#5743) * Recover from fork repo * Packaging update of azure-eventhubs * Fix error message * update iterator example * Revert "Packaging update of azure-eventhubs" This reverts commit 56fc4f01126daa85956222ea6f2a992146349bee. * disable autorest auto update * Sender/Receiver -> EventSender/Receiver * Change _batching_label back to partition_key * Remove transfer examples * move async to async folder * Update docstring string, sample codes and test codes (#5793) * catch and process LinkRedirect * Add receiver iterator pytest * small fix of iterator example * add retrieval_time to partition prop * fix open and re-send bugs * small fixes * fix reconnect test case * close iterator when closing receiver * Misc changes for code review fix * client.py type hints * catch KeyboardInterrupt * add next() for 2.7 iterator * raise KeyboardInterrupt instead of exit() * Fix missing consumer group directory in EPH * Fix livetest code problem * history and readme * Update history and setup (#5902) * EventPosition.first_available_event -> earliest new_events_only -> latest * Change EventSender's event_position to be mandatory * Update uamqp shared_req to 1.2.0 * Disable network_tracing * update uamqp dependency ~=1.2.0 * Remove EventPosition helper functions * Names changed to EventHubConsumer/Producer * Avoid nested with statement * Skip forced reconnect test * Update naming in eventhub (consumer and producer). (#5984) * Remove azure-core requirement tentatively * Warn when eventhubs or storage teardown fails * add type hints comments * add azure identity in dev_requirements * put TransportType in __init__ directly * change EventData's offset to be str, not EventPosition * remove from_iot_connection_string * small fix * docstring timeout from int to float * fix TransportType import * add pytest option sleep for reconnect test * Fix eventposition issue in receive test * fix constants import issue * remove azure identity dev_req * fix some example issues * iot string fix * Change filename for consumer and producer * Vendor azure-storage-blob in eventprocesshost (#6018) * Update storage dependency * Vendor storage blob v1.3.1 in eph * Add vendored blob-storage required pkg * Vendor storage 2.0.1 within EPH (#6031) * catch exception for mgmt_request * Update comment and code structure (#6042) * code review changes * Add python-dateutil in shared_requirements * Add aad credential env var to tests.yml * Change example code assertion for parallel running * Enable iothub receive test case * Revert "Enable iothub receive test case" This reverts commit 5abdcd780cfb2085472d73c8d57fff120bbbb322. * fix auth test error * change offset.value to offset * Fix an eventposition problem * Remove path append * trying removing the module init py within tests * Separate MockEventProcessor to a different file * remove tests to path * trying a run based on a nested conftest to establish the async fixtures without shattering on python 2.7

joostdenijs and others added 3 commits June 6, 2012 16:12

Merge pull request Azure#8 from DinoV/master

d2f9f48

MergeDevToMaster

change pip install from pyazure to azure

8d49ea9

Merge pull request Azure#11 from ptvs/master

7174fc2

pip install pyazure -> azure

weidongh closed this Aug 22, 2012

yunhaoling pushed a commit to yunhaoling/azure-sdk-for-python that referenced this pull request Mar 25, 2019

Updated send example to match recv

f428531

Fix for issue Azure#56

annatisch mentioned this pull request May 8, 2019

Migrate Event Hub SDK to central Repo (#4764) #5145

Merged

lmazuel pushed a commit that referenced this pull request Jun 7, 2019

SingleValueReplacer.process_request properly decodes request.body if …

a9cca7d

…its type is byte. (#56)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bug fixes after June 6 release #56

bug fixes after June 6 release #56

weidongh commented Aug 21, 2012

bug fixes after June 6 release #56

bug fixes after June 6 release #56

Conversation

weidongh commented Aug 21, 2012