✨Adding rabbitmq based RPC for IPC between services

[GitHK]

What do these changes do?

Adds RPC to the backend stack to be used between different services inside the backend.

Use cases:

• director-v2 to agent: ask to remove pending volumes and waits for the operation to be complete
• autoscaling -> agent: ask if agent still needs to cleanup before putting the node to drain
• director-v2 -> dy-sidecar (possibly): replaces http long running tasks polling with RPC polling

Why this implementation and not others?

• Very good decoupling via RabbitMQ.
• The aio_pika RPC helper has a very solid implementation. It was just thinly wrapped to deal with one edge case. For more details see tests.

Bonus fixes:

• ♻️ replacing some dict with {} notation
• 🐛 upon channel shutdown non longer raise an error because the channel is closed

Thus is required to fix issues

Related issue/s

• improving dynamic-sidecar design osparc-issues#638
• required by ♻️ volumes are removed via agent (⚠️ devops) #3941

How to test

Checklist

• Documentation reflects the changes
• Unit tests for the changes exist
• Runs in the swarm

[codecov]

Codecov Report

Merging #3909 (5a79662) into master (3de921f) will increase coverage by 0.5%.
The diff coverage is 96.2%.

@@           Coverage Diff            @@
##           master   #3909     +/-   ##
========================================
+ Coverage    85.0%   85.5%   +0.5%     
========================================
  Files         934     781    -153     
  Lines       40180   34537   -5643     
  Branches      839     447    -392     
========================================
- Hits        34175   29552   -4623     
+ Misses       5784    4853    -931     
+ Partials      221     132     -89     

Flag	Coverage Δ
integrationtests	66.5% <ø> (+<0.1%)	⬆️
unittests	82.3% <96.2%> (+<0.1%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ackages/service-library/src/servicelib/rabbitmq.py	93.1% <92.8%> (-1.6%)	⬇️
.../service-library/src/servicelib/rabbitmq_errors.py	100.0% <100.0%> (ø)
...s/service-library/src/servicelib/rabbitmq_utils.py	89.1% <100.0%> (+89.1%)	⬆️
.../server/src/simcore_service_webserver/users_api.py	95.1% <0.0%> (-1.4%)	⬇️
...e/src/simcore_postgres_database/models/services.py
...rvice-integration/src/service_integration/_meta.py
...tion/src/service_integration/osparc_image_specs.py
...gration/src/service_integration/commands/config.py
...e/src/simcore_postgres_database/utils_migration.py
... and 153 more

[sanderegg]

Ok here are a few comments:

• why do we need yet another rabbitmq module ? can't we have an extension on the current one? (i.e. rabbitmq.py and now rabbitmq_robust_rpc.py?)
• in rabbitmq.py I made use of connection pooling, which is gone from your implementation. Also you had to re-implement all the setup/close mechanism that already exists there.
• we will likely want to register a whole interface. is there any such mechanism?

[pcrespov]

I added some comments.
I have some questions about this new approach but perhaps we can speak offline.
Thx.

[sanderegg]

Very interesting! thanks for this. Please check my comments.
Did you try using that syntax?

await rpc.proxy.multiply(x=2, y=3) == 6

[codeclimate]

Code Climate has analyzed commit 5a79662 and detected 0 issues on this pull request.

View more on Code Climate.

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.1% Duplication