Optimize work horse forking #4314
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
arikfr
reviewed
Oct 29, 2019
| # Configure any SQLAlchemy mappers loaded until now so that the mapping configuration | ||
| # will already be available to the forked work horses and they won't need | ||
| # to spend valuable time re-doing that on every fork. | ||
| configure_mappers() |
There was a problem hiding this comment.
I've noticed that in redash.models.base we use the flask_sqlalchemy.SQLAlchemy's instance configure_mappers. I wonder what's the difference?
There was a problem hiding this comment.
So you can access everything through flask_sqlalchemy.SQLAlchemy, I guess.
There was a problem hiding this comment.
Yeah, the idea being that you define a db object like db = SQLAlchemy() when using flask-sqlalchemy and then can call anything inside the sqlalchemy.orm module. In other words, it's to simplify the Flask-like API of flask-sqlalchemy.
arikfr
approved these changes
Oct 30, 2019
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What type of PR is this? (check all applicable)
Description
We've noticed that worker horses are consuming a lot of CPU. After some profiling, I realized that
configure_mappingis called every time a work horse is forked (i.e. for every job). This PR callsconfigure_mappingonce in the parent worker process, which is then available to the forked horses. We've seen significant performance improvement with this.Related Tickets & Documents
Mobile & Desktop Screenshots/Recordings (if there are UI changes)