Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛 #2758 exclusive queue cannot be reacquired #2780

Conversation

sanderegg
Copy link
Member

@sanderegg sanderegg commented Feb 1, 2022

What do these changes do?

This PR changes how RabbitMQ queues are declared.
Before PR: queue set as "exclusive" (e.g. (used by only one connection and the queue will be deleted when that connection closes)
After PR: queue is not exclusive

This has the following implication:

  • when a webserver is restarted, for about 5-10 seconds it is a 2nd instance and it has issues accessing the queue (as explained here there can be a race condition and client side communication might ensure issues... that is what we are seeing)
  • since the queue is not deleted now when the webserver goes down, it could generate issues but I don't see any at the moment since the messages have a TTL of 60000 seconds, so they will be deleted anyhow.

Related issue/s

fixes #2758

How to test

Checklist

@mrnicegyu11
Copy link
Member

@sanderegg we could maybe monitor the number of queues using prometheus, there should only be two, right? If they accumulate, we would have to do a cleanup at some point I guess.

@codecov
Copy link

codecov bot commented Feb 1, 2022

Codecov Report

Merging #2780 (02d8fd5) into master (fec7a1f) will increase coverage by 0.0%.
The diff coverage is 100.0%.

Impacted file tree graph

@@          Coverage Diff           @@
##           master   #2780   +/-   ##
======================================
  Coverage    78.8%   78.8%           
======================================
  Files         680     680           
  Lines       27101   27101           
  Branches     2633    2633           
======================================
+ Hits        21359   21366    +7     
+ Misses       4998    4992    -6     
+ Partials      744     743    -1     
Flag Coverage Δ
integrationtests 65.7% <100.0%> (+<0.1%) ⬆️
unittests 74.4% <0.0%> (+<0.1%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...simcore_service_webserver/computation_subscribe.py 88.0% <100.0%> (ø)
...simcore_service_webserver/projects/project_lock.py 94.7% <0.0%> (-5.3%) ⬇️
.../director/src/simcore_service_director/producer.py 62.1% <0.0%> (ø)
...ce_webserver/resource_manager/garbage_collector.py 73.5% <0.0%> (+0.8%) ⬆️
...ce_webserver/resource_manager/websocket_manager.py 97.3% <0.0%> (+4.0%) ⬆️
...rvice_datcore_adapter/utils/requests_decorators.py 78.9% <0.0%> (+7.8%) ⬆️

@sanderegg sanderegg merged commit a51557b into ITISFoundation:master Feb 1, 2022
@sanderegg sanderegg deleted the bugfix/2758/exclusive_queue_cannot_be_reacquired branch February 1, 2022 17:16
@odeimaiz odeimaiz added this to the Rudolph+1 milestone Feb 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

webserver shows write is none when restarted while connecting to RabbitMQ
4 participants