✨ Rabbitmq: prepare for using topic queues

[sanderegg]

What do these changes do?

• some typing fixes
• allows to use RabbitMQ topic exchanges by passing an optional topic parameter in publish and subscribe methods

NOTE: since we currently use durable exchanges they are kept within rabbitmq memory. Therefore one needs to change the name of the exchange between deployments once they get used.
Since this PR does not yet make use of the new feature this is not yet necessary, but will be once the topic queues are put to use for logs.

Using a topic queue brings the following possibilities:

• say we have RabbitMQ exchange of type TOPIC
• then one can publish logs in that exchange everytime setting the routing_key with origin.criticality pattern (e.g. publish(message="this is a log message", routing_key="dynamic_sidecar.INFO")
• the webserver might connect to that exchange but only subscribing to topic of type *.WARNING
• it will only receive the messages from any origin but with WARNING set up.

Here is a summary of how RabbitMQ messaging works

1. First a quick image shows more than anything. This shows the different types of Exchanges (FANOUT - the current one we use, DIRECT and TOPICS)
   
2. in RabbitMQ an Exchange is just a kind of placeholder where messages are published to. They are not stored there unless 1 or more Queues are bound to the Exchange
3. How these Queues are bound to the Exchange defines what messages go there (this is defined by the routing key. in case of DIRECT, the routing key must be exactly the same, in case of FANOUT the routing key is discarded, in case of TOPICS the routing key can be made of * (exactly 1 word) and # (0 or more words) and can contain . between words (e.g. .. and a message might use "director-v2.event.black".
4. Additionally there can be 1 or more Consumers per Queue.
5. Everytime a consumer successfully consumes a message from a Queue, that message disappears. Therefore you cannot keep messages 10 minutes there. They are kept until a consumer consumes them, or their TTL ran out or the Queue max number of messages is attained.

Related issue/s

• relates to M1-12 Maintenance and DevOps osparc-issues#675
• relates to Dask backend: run 15600 sleepers #3991

How to test

DevOps Checklist

[codecov]

Codecov Report

Merging #4165 (45dddb0) into master (b2223f1) will increase coverage by 0.4%.
The diff coverage is 100.0%.

@@           Coverage Diff            @@
##           master   #4165     +/-   ##
========================================
+ Coverage    85.4%   85.9%   +0.4%     
========================================
  Files         955     799    -156     
  Lines       41378   35660   -5718     
  Branches      945     540    -405     
========================================
- Hits        35358   30636   -4722     
+ Misses       5803    4896    -907     
+ Partials      217     128     -89     

Flag	Coverage Δ
integrationtests	67.2% <ø> (-0.1%)	⬇️
unittests	82.2% <100.0%> (-0.1%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ackages/service-library/src/servicelib/rabbitmq.py	95.6% <100.0%> (+2.4%)	⬆️
...s/service-library/src/servicelib/rabbitmq_utils.py	89.1% <100.0%> (ø)

... and 161 files with indirect coverage changes

[pcrespov]

brilliant! thx!

[GitHK]

@sanderegg One thing is not clear to me. Let's say the fronted subscribes to *.WARNING. This means that the following are possible:

1. It will receive all messages with WARNING log level, from that moment onward?
2. It is still capable of fetching messages from a 10 minutes ago from the queue?

It is not clear to me how the message delivery works. Currently we are using the pub/sub pattern (if you are not subscribed you do not get messages). What do the new queues bring?

[GitHK]

👍

[sanderegg]

@sanderegg One thing is not clear to me. Let's say the fronted subscribes to *.WARNING. This means that the following are possible:

1. It will receive all messages with WARNING log level, from that moment onward?
   Yes
2. It is still capable of fetching messages from a 10 minutes ago from the queue?
   No.
   I updated the description above to try to better explain the concept behind RabbitMQ messageing

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[codeclimate]

Code Climate has analyzed commit 45dddb0 and detected 0 issues on this pull request.

View more on Code Climate.