Download backend: Request all thumbnails for channel

[dbnicholson]

In order to provide a rich display of unavailable content, thumbnails for the missing content is desired. When downloading the starter pack content, request all thumbnails for each channel. This is a new option for the remotecontentimport Kolibri task added after 0.16.0-alpha14. The option is ignored on older releases.

#549

[dbnicholson]

Since learningequality/kolibri#10780 was merged, I tested this out on my local checkout. Good news - it works. Bad news - it significantly slows down the initial starter pack download. I tried this with the Scientist starter pack since it appeared to be the smallest and it was spending a lot of time downloading thumbnails. Presumably this is because it includes wikiHow, which has 88 MB of thumbnails.

[wjt]

I tried this with the Scientist starter pack since it appeared to be the smallest and it was spending a lot of time downloading thumbnails. Presumably this is because it includes wikiHow, which has 88 MB of thumbnails.

Might it work to download the thumbnails later as a separate background step that happens after the onboarding flow?

[manuq]

This looks good. Moving it to draft to avoid merging it until Kolibri 0.16 alpha15 is released.

[dbnicholson]

I tried this with the Scientist starter pack since it appeared to be the smallest and it was spending a lot of time downloading thumbnails. Presumably this is because it includes wikiHow, which has 88 MB of thumbnails.

Might it work to download the thumbnails later as a separate background step that happens after the onboarding flow?

Yeah, that's a good idea. I believe we can just start tasks to do that from the backend and it will happen in the background. I've never done it exactly that way, but I don't think it should be too hard.

[dbnicholson]

I changed it so that once the initial download completes, the remaining thumbnails are queued up for downloading in the background. It worked, but I can't say this is particularly robust since the background tasks aren't tracked and I'm not sure if Kolibri will restart them if they fail.

Here's a screenshot with TechBridge Girls showing 2 unavailable nodes after I ran kolibri with KOLIBRI_NAVIGATE_UNAVAILABLE=true:

[manuq]

@dbnicholson it is a bit hacky but I think it will work. As long as the for loop (the one after marking "COMPLETED") ends without Kolibri stopping. Once the tasks are scheduled Kolibri persists them and they are ran across Kolibri sessions.

The less-hacky way is: direct the collection download from the App root Vue component (kolibri_explore_plugin/assets/src/app.js). Today it is directed from the Download page (kolibri_explore_plugin/assets/src/views/DownloadPage.vue).

[manuq]

@dbnicholson the other problem I see is that the UI won't be refreshed with the thumbnails after the background download completes. That's another other reason to have a frontend component directing/monitoring the background job.

[dbnicholson]

@dbnicholson it is a bit hacky but I think it will work. As long as the for loop (the one after marking "COMPLETED") ends without Kolibri stopping. Once the tasks are scheduled Kolibri persists them and they are ran across Kolibri sessions.

The less-hacky way is: direct the collection download from the App root Vue component (kolibri_explore_plugin/assets/src/app.js). Today it is directed from the Download page (kolibri_explore_plugin/assets/src/views/DownloadPage.vue).

Agreed it is hacky. I probably need a bit more help here. I don't understand how directing it from the app component rather than the download view affects what's going on here. Could you explain?

Regardless, I think the idea would be that once the collection download completes then the frontend would call a separate backend API to initiate the background thumbnail download. Is that right? Would we want to track those tasks in the session state like the collection download?

[manuq]

@dbnicholson it is a bit hacky but I think it will work. As long as the for loop (the one after marking "COMPLETED") ends without Kolibri stopping. Once the tasks are scheduled Kolibri persists them and they are ran across Kolibri sessions.
The less-hacky way is: direct the collection download from the App root Vue component (kolibri_explore_plugin/assets/src/app.js). Today it is directed from the Download page (kolibri_explore_plugin/assets/src/views/DownloadPage.vue).

Agreed it is hacky. I probably need a bit more help here. I don't understand how directing it from the app component rather than the download view affects what's going on here. Could you explain?

I wish I have a clear understanding too! My gut feeling is that we want to monitor the background task in order to refresh the nodes thumbnail property in the frontend state after the download is completed. And we want that to happen in any view, not just the download view. The component that is always present is the root app, the parent of any every component.

Regardless, I think the idea would be that once the collection download completes then the frontend would call a separate backend API to initiate the background thumbnail download. Is that right? Would we want to track those tasks in the session state like the collection download?

Yes, maybe split the foreground collection stages from the background stages (extra metadata and thumbnails). My point is that this will need more thinking to happen in the background. Maybe start easy by adding this in fhe foreground first?

About persisting, we decided to go with session state for the download. Maybe it deserves a Django model to persist it in the database? I wouldn't change it for now.

[dbnicholson]

@dbnicholson it is a bit hacky but I think it will work. As long as the for loop (the one after marking "COMPLETED") ends without Kolibri stopping. Once the tasks are scheduled Kolibri persists them and they are ran across Kolibri sessions.
The less-hacky way is: direct the collection download from the App root Vue component (kolibri_explore_plugin/assets/src/app.js). Today it is directed from the Download page (kolibri_explore_plugin/assets/src/views/DownloadPage.vue).

Agreed it is hacky. I probably need a bit more help here. I don't understand how directing it from the app component rather than the download view affects what's going on here. Could you explain?

I wish I have a clear understanding too! My gut feeling is that we want to monitor the background task in order to refresh the nodes thumbnail property in the frontend state after the download is completed. And we want that to happen in any view, not just the download view. The component that is always present is the root app, the parent of any every component.

OK, I think I understand. We more or less want something attached to the root app that tracks any outstanding download tasks (whether foreground or background) so that any view can see the status of those tasks if they need to.

Regardless, I think the idea would be that once the collection download completes then the frontend would call a separate backend API to initiate the background thumbnail download. Is that right? Would we want to track those tasks in the session state like the collection download?

Yes, maybe split the foreground collection stages from the background stages (extra metadata and thumbnails). My point is that this will need more thinking to happen in the background. Maybe start easy by adding this in fhe foreground first?

Note that @dylanmccall is doing something very similar in #579. I'm going to sync up with Dylan to see if we can converge on a way to handle these background downloads. If that doesn't seem like it's going to materialize quickly then I'll go back to the original commit that simply adds all_thumbnails=True to the content download. It really slows things down, though.

About persisting, we decided to go with session state for the download. Maybe it deserves a Django model to persist it in the database? I wouldn't change it for now.

I think the session state is fine. Kolibri already has all the information we need as long as we keep the job IDs. A couple lists of job IDs stored in the session state seems fine to me.

[manuq]

Note that @dylanmccall is doing something very similar in #579

Oh that's great! Seems to be the way to go for these thumbnails and the extras metadata.

[dbnicholson]

This is ready for review again. I attempted to use the process from #579 to queue the background tasks and it didn't work. That method depends on the frontend to keep calling the ek-collections/update-download API to query and enqueue the tasks. Obviously it can't do that if it wants to move on.

Instead, I simply queue them in the background as before but a little nicer. I added a separate queue for background tasks, made the plugin restart them when the app starts, and made the enqueing happen in an explicit stage so that we don't reach COMPLETED until they're enqueued.

The idea is that the frontend can query the background and foreground queues for completion and update the UI as needed. I started to play with that but my Vue is too weak. I'm hoping someone else (looking at you @manuq) can implement that part since I believe it's generally needed to monitor the content tasks like the Device plugin does.

[manuq]

@dbnicholson sorry for missing this one today. @dylanmccall could you review? Otherwise I'll be back on Tuesday.

[dylanmccall]

I read through this and it looks good to me. Adding a separate queue for background tasks definitely seems like the way to go, and we can work through the UI stuff separately 👍