Download extra channels in the background #762
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dbnicholson
force-pushed
the
background-downloads
branch
from
6a22ea9 to
422dd5f
Compare
pwithnall
reviewed
Aug 24, 2023
The only modules left out are the empty `__init__.py` package initialization modules and the generated Django migration modules.
This separates the helpers for managing jobs from the task definitions and the views using them. This should allow for cleaner reuse. There should be no functional changes. One important change here is that the task types are initially stored as callable paths. This will allow the task data to be serialized later without immediately resolving the task functions.
Using the Django app `ready` method wasn't the right approach. Those run whenever Django loads the app and before the database models are prepared. Instead, use a Kolibri `StorageHook`, which allows immediately restarting the jobs. Previously jobs would only be restarted when the Kolibri was restarted. While here, include `CANCELED` jobs in the restart logic. We're not cancelling these jobs, but the worker will cancel running jobs when it stops. Since the storage hook runs even when the worker is shutting down, this will requeue an incomplete job for when the worker next starts.
dbnicholson
force-pushed
the
background-downloads
branch
from
422dd5f to
2804d97
Compare
Use a database model to keep track of background tasks. This will allow persistently keeping track of tasks that need to be run in the background.
Since Kolibri will run as many jobs as there are available workers, queueing up background tasks from the collection downloader will likely start many in parallel. That does not make for an ideal user experience as the app will make the system busy and prevent any user initiated downloads from being started. Instead, create the desired background tasks in the `BackgroundTask` model and use the existing storage hook to enqueue the next background task when one completes. This will ensure that only one background task is running at a time.
When initial downloads complete, the user is shown the Discovery view with the channels from the content pack. Since they're likely to explore these channels first, prioritize downloading these thumbnails over the extra channels from other manifests.
Now that there's a way to serialize background tasks, defer downloading the extra channels so that the initial download completes faster. Since the thumbnail tasks depend on this, they need to be generated on the fly when the channel download completes. Fixes: #592
dbnicholson
force-pushed
the
background-downloads
branch
from
2804d97 to
e9de59d
Compare
|
🚢 I’ll hit merge once CI has finished |
pwithnall
approved these changes
Aug 24, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This introduces a database model to keep track of the background download tasks to run after the initial download completes. That allows serializing the tasks to run one at a time with a
StorageHookas the needed tasks are tracked in a persistent location. With that in place, we can defer downloading the extra channels until after the initial download completes. Together these changes have 2 major benefits: the initial download finishes faster and the background downloads cause less load.Fixes: #592