Migrate Event Hub SDK to central Repo (#4764)

[annatisch]

• Initial commit

• Initial commit

• Initial commit

• event hub client

• Update README.md

• Update README.md

Fix typos

• Memory leak

• Support timestamp filter

• Support timestamp filter

• Update README.md

• Add sender and refactor

• Added abstract classes

Todo

• Migrate Base Class Wireframes
• Migrate Azure Classes

• First draft of class wires directly ported from .net (might be some minor gaps)

• send example

• Set allowed sasl mechs

• Remove client.py

• Receiver update

• Add dummy send api

• logging updates

• Error handling, reconnect and logging

• Add app properties to event data

• unbind transport on connection close

• timestamp filter on py2

• module version

• Reconnect once when link/session/connection close

• Add SessionPolicy

• Add client info

• Updates

• Cleaned wireframes to be PEP compliant
• Implemented single partition pump and single event_hub partition pump scenario

Todo

• Add Unit Tests for partition pump and event hub partition pump
• Implement Partition Manager
• Implement Checkpointing and Lease Managment

• Updates

• Cleaned wireframes to be PEP compliant
• Implemented single partition pump and single event_hub partition pump scenario

Todo

• Add Unit Tests for partition pump and event hub partition pump
• Implement Partition Manager
• Implement Checkpointing and Lease Managment

• run client in non-blocking mode

• Added unit testing

• Implemented the following functionality

• Azure_storage_checkpoint_manager
• AzureBlobLease isExpired

Todo

Implement partition manager
Implement partition context
Test full implementation

• Implemented Processing of First Epoh

Todo

• Fix lease bug that is breaking subsequent epochs

• Changes

• Completed End to End EPH Flow
• Removed storage dependancy on downloading full blob to check lease state

Todo

• Add thread and queue for checking lease state and other storage operations

• Ensure eventhub client shuts down properly

• Find way to update partition pumps without restarting them

• Other optimizations

• Move examples out

• Changes

• Added thread pool executor to enable conncurent execution of partitions

• Removed partition pump dependency on max_batch

Todo

• Ensure eventhub client shuts down properly (This is causing errors)

• Add thread pool for making checkpoint code conccurent

• Add thread and queue for checking lease state and other storage operations to enable async

• Find way to reassign active partition pumps without restarting them

• Other optimizations

• Add async receive

• Changes

• Added logs
• Fixed error causing client to prematurely shutdown

• Manual link flow control for async receive

• Workaround for stuck async receiver

• Local variable names

• Changes

• Optimized logging and comments

Todo

• Add concurecny mechanim for azure storage

• Depricate partition pump event queue and update to latest version of the client

• Create Dockerfile

• Stuck async receiver

• credit keeps increasing in async receiver

• Changes

• Added asnyc event hub client support
• Optimized logging and comments

Todo

• Add concurecny mechanim for azure storage

• Updated docker file as requested

• Added EPH example

• Fix hardcoded HTTP header

• Made suggested changes

• Bug fix

• Fixed event loop bugs. In windows eventloop is thread dependent but in ubuntu the eventloop is threadsafe so you need to differentiate the thread specific eventloop from the host one.

• Updated loop naming convention to be consistent

• Added option to pass asyncio event_loop to eph

• Updated docker file

• Fixed critical bug with partition manager and aquirec mechanisiims

Todo :
Identitfy and fix remaining bug that is causing all pumps to shut down when a second host starts

• Bug fixes

• Fixed bug where closing a pump closed a host
• Fixed bug where error partitioned were not removed
• Fixed bug where leases were renewed at an incorrect interval

• Updated file headers
  Removed author reference

• • Fixed bug in eph example that caused host to terminate prematurely

• Made the lease renewal and checkpoint creation "multithreaded"

• Increase the size of the connection pool

The default connection pool size was too small for scenarios where
multiple partitions were handled by one EventProcessorHost.

If the amount of partitions handled is large, we might end up doing
very many connections at the same time due to the multi-threaded
blob-handling. For this reason, you might hit the OS limits that
restrict the number of open files per process that in MacOS is not
very big. This can be worked around with something like:

ulimit -n 2560

• Decrease info logging verbosity

• added ability to toggle pump shutdown when all messages on a pump are processed.

• Install also eventhubsprocessor

• Default to keeping the pumps

It is more optimal to keep the pumps alive even if there are no
messages so that it is faster to pickup when messages start to arrive.

• Pipe and event injector for Windows

• Event injector updates

• EHClient refactoring. EHClient leaks. Sender part 1.

• Send support

• ren eventhubsprocessor eventprocessorhost

• Changes

• Added event hub config to simplify installation story

• Changes

• Added optional eventprocessor_params for passing context to the event processor

• Made the storage manager mandatatory

• Fix memory leaks

• logging

• Fix: 1. process crash due to race in client stop and connection remote close. 2. handle client close in async receiver. 3. fail pending sends when sender is closed. 4. some debug logging.

• tests

• test: recv from multiple partitions

• test utility

• logging update

• Support callback based send for high throughput

• Workaroud memory issue in proton.reactor.ApplicationEvent

• renamed eventprocessor to eventprocessorhost for consistency

• updated docker file

• fixed typo in url

• Added amqp port to address

• Updated sample documentation since url is auto encoded by config

• Updated docs

• Implement timeout for send

• Async sender and example

• Close injector pipe

• Use send timer to also check queued messages

• Add partition pump loop to partition_context

This gives the EventProcessor access to the partition_pump loop object. This way if
One desires to run synchronous code inside process_events_async one can utilize the
loop object to run the synchronous code using await context.pump_loop.run_in_executor(None, bla)

• Include details in send error

• Release deliveries when sender is closed

• added validation to unquoted sas key

• added support for custom eventhub client prefetch size

• Update README.md

• Update README.md

• Added Docker instructions and fixed Dockerfile (TableEnumResult is unused #18)

• Removed Dockerfile from the main folder and fixed Dockerfile example

• Added build and run Dockerfile documentation

• Update Readme

• Removed rm qpid-proton folder

• Removed /usr/share copy

• Disallow a sender/receiver to be registered more than once

• Make everything async in EPH

I have removed all usage of threads thoroughout the code.
Using threads to run pumps etc. Causes async code written into the event-processor
to become caotic (you need to follow which loop is currently being used in the call to
prevent loops not being found or using the wrong loop (There is the main loop and then
loops that are created inside threads) Things become caotic when the event processor is
being called by objects that run under different loops.

So, no Threading except usage of asyncio run_in_executor. This is done mostly for azure blob api calls.

Also changed the bla_async methods to not block. this way, when calling open_async for the the
event-processor-host, the command will exit once the EPH is started.

Due to the above, see the edited example/eph.py where I added a monitor that makes sure the EPH is still running
(Could be replaced by loop.run_forever()) in the example file I have also incorporated a test class for gracefully
killing the EPH after 30 seconds. this works, nevertheless takes a while to close as we are waiting for timeouts
on the eventhubs connections.

• Started removing proton code

• Removed most of proton _impl

• Removed more code

• Working sender

• Updates to sender

• Added some tests/samples

• Some progress on clients

• Fixed samples

• Added azure namespace

• can't pass booleans to service bus create_queue,create_topic,create_subscription  #25 Partition key cannot be set for events

• Updated version

• Updated README

• Renamed package to eventhub

• Started EPH modifications

• Updated imports

• Fixed target urls

• Updated logging

• Updated async message receive

• updated test imports

• Added mgmt call to get eh info

• Updated samples

• Updated receive test

• Added send and receive test clients

• Updated uamqp dependency

• Merged updates from dev

• Fixed typos

• Updated EPH sample

• Started docstrings

• Converted tests to pytest

• Updates to batch receive

• Started adding docstrings

• More docstrings

• bumped version

• Started porting test suite

• More tests and improvements

• Moved eph tests

• Some sample cleanup

• Some test updates

• Some test restructure

• Docstring cleanup

• Fixed some merge artifacts

• Fixed formatting error

• Removed delivery count

• Nested package directory

• Support custom URL suffix

• Support custom URL suffix

• Support for EventData device ID

• Reverted nested directory

• Updated release notes

• Workaround for partitionkey

• Finished partition key workaround

• beta2 fixes

• pylint fixes

• Trigger CI

• Test fixes

• Added package manifest

• Added warning for Python 2.7 support

Support for issues #36 and #38

• Started adding scenario tests

• More test scenarios

• Better docstring formatting

• Started iothub support

• Fixed long running test

• Fixed typo and memory leak

• Restructure

• IoThub support

• Updates for RC1 release

• Fix long running test

• Docstring and sample cleanups

• Working on error retry

• Improved error processing

• Fixed partition manager

• Progress on IotHub error

• Some test updates

• Updated uamqp dependency

• Restructure for independent connections

• Added HTTP proxy support

Fix for issue #41

• Fixed some tests + samples

• pylint fixes

• bumped version

• Added keepalive config and some eph fixes

• Made reconnect configurable

• Added more EPH options

• Bumped version

• Pylint fix

• Pylint fix

• Added send and auth timeouts

• Changed log formatting. Retry on reconnect

• Pylint fixes

• Renamed internal async module

• Updated send example to match recv

Fix for issue #56

• Added build badge to readme

• Fix for repeat startup

• Added more storage connect options to EPH

• Bumped version

• Handler blocked until client started

• Added event data methods

• Fix pylint

• Fix 3.7 CI

• Fix 3.7 CI

• Updated pylint version

• Pylint fixes

• Updated README

• Fixed readme badge refresh

• Fixed bug in Azure namespace package

• Updated manifest

• Parse enqueued time as UTC

Fixes #72.

• Updates for release 1.2.0 (azure.WindowsAzureError: Unknown error (40106: Access Control Serivce token supplied for this operation has expired. Retry the operation.) #81)

• Made setup 2.7 compatible

• Separated async tests

• Support 2.7 types

• Bumped version

• Added non-ascii tests

• Fix CI

• Fix Py27 pylint

• Added iot sample

• Updated sender/receiver client opening

• bumped version

• Updated tests

• Fixed test name

• Fixed test env settings

• Skip eph test

• Updates for v1.3.0 (get_blob_metadata returns metadata names in lower case #91)

• Added support for storing the state of the Event Processor along the
  Checkpoint. Both Checkpoint and the EP state are stored as pickled
  objects.

• Fixing pylint complaints.

• Switched from pickle back to JSON for lease persistence.

• Fixes bug when accessing leases that don't contain EP context. Also,
  minor renaming.

• Better SAS token support

• Fixed pylint

• Improved auth error handling

• Test stabilization

• Improved stored EPH context

• Updated EPH context storing

• Skip test on OSX

• Skip tests on OSX

Fail due to large message body bug.

• Some cleanup

• Fixed error handling

• Improved SAS token parsing

• Fixed datetime offset (Version difference github and pypi #99)

• Fixed datetime offset

• Updated pylint

• Removed 3.4 pylint pass

• Fixed bug in error handling (Configuring a credential for a Windows Server virtual machine #100)

• Migrate event hub sdk to central repo

1. add verifiable code snippets into docstring
2. update readme according to the template
3. add livetest mark and config
4. optimize code layout/structure

• 1. document formatting

2. separate async/sync example tests

• Fix build error:

1. uamqp dependency mismatch
2. rename test_examples in eventhub to avoid mismatch

• This should fix build error

• remove tests import and add sys path to solve build error

• add live test for sending BatchEvent with application_properties, new live test passed with new uamqp wheel locally installed

• Skip batch tests pending uAMQP release

[adxsdk6]

Can one of the admins verify this patch?

[adxsdk6]

Can one of the admins verify this patch?

[codecov-io]

Codecov Report

Merging #5145 into master will decrease coverage by 4.28%.
The diff coverage is 22.67%.

@@            Coverage Diff             @@
##           master    #5145      +/-   ##
==========================================
- Coverage   52.99%    48.7%   -4.29%     
==========================================
  Files       10849     6533    -4316     
  Lines      227609   184668   -42941     
==========================================
- Hits       120613    89938   -30675     
+ Misses     106996    94730   -12266

Impacted Files	Coverage Δ
...eventhubs/azure/eventhub/async_ops/sender_async.py	10.22% <10.22%> (ø)
...enthubs/azure/eventhub/async_ops/receiver_async.py	10.55% <10.55%> (ø)
azure-eventhubs/azure/__init__.py	100% <100%> (ø)
azure-eventhubs/azure/eventhub/sender.py	11.34% <11.34%> (ø)
azure-eventhubs/azure/eventhub/receiver.py	12.65% <12.65%> (ø)
...hubs/azure/eventprocessorhost/partition_manager.py	15.56% <15.56%> (ø)
...tprocessorhost/azure_storage_checkpoint_manager.py	18.07% <18.07%> (ø)
...hubs/azure/eventprocessorhost/partition_context.py	19.04% <19.04%> (ø)
...hubs/azure/eventprocessorhost/eh_partition_pump.py	20% <20%> (ø)
azure-eventhubs/azure/eventhub/client.py	20.47% <20.47%> (ø)
... and 4480 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7181cb5...7efa689. Read the comment docs.

[lmazuel]

Should be relative installation

[lmazuel]

Unless the dev_setup is now able to get that it has to install azure-servicebus first, to be checked with Scott K I'm not sure

[lmazuel]

You don't need azure-nspkg only because azure-common is doing it. You might want to be more explicit about it