Fix realtime sims in distributed architecture #56
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Neuron simulator does not reset itself automatically after a call to simulation.run() This causes in-correct simulation data because it (the data) also contains information from previous simulations that ran on the same worker. The best way to avoid this (as suggested by neuron forums) is to run the simulation inside the child process. Since we cannot use multiprocessing module with celery, billiard is used. See: - https://www.neuron.yale.edu/phpBB/viewtopic.php?t=4039 - https://stackoverflow.com/questions/54858326/python-multiprocessing-billiard-vs-multiprocessing - celery/celery#5362 (comment) Signed-off-by: Dinika Saxena <[email protected]>
pgetta
approved these changes
Nov 13, 2024
|
Great work, Dinika, the change looks good to me. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implements the following changes to fix the new architecture:
Replace celery state updates with redis pub sub. Motivations:
sleep()(or similar) then we risk losing state updates that were for data.Start a sub process to run the simulation inside the celery worker - If the simulation is not started in a child, the neuron simulator is not reset. It still holds state from the previous simulations that that particular celery worker ran. This is why we were seeing repeated data for current.
Reduce the frequency of health checks of celery workers - The command
celery -A bluenaas.infrastructure.celery statustakes around 7 seconds to reply. The worker gets busy replying to these health checks and the simulation performance goes down.