Content download speed regression in Kolibri 0.16

[erikos]

Tracking upstream issue:

• Content download speed regression in 0.16 learningequality/kolibri#10848

[erikos]

Joana reports that she has not been able to download a content pack once in the last three days. Bumping up priority to unbreak now.

[wjt]

@dbnicholson @dylanmccall between you, can you figure out how we can work around this download speed regression for Ladybug? Particularly now that we download all channels' metadata during onboarding, and my reading of #549 is that the intention is to also eagerly download 500MB of thumbnails, this is a real issue. (I'm just a guy but, is it straightforward to revert the change that switched to chunked downloads? Or bump up the chunk size?)

[dbnicholson]

In #549 I'm just about done with background downloading and it worked fine in my testing. However, your point is completely valid.

The most straightforward thing to do would be to copy the file transfer code from 0.16.0a12 (last release before the range requests were landed), and then wire up our own download manager and task in the explore plugin. It should be pretty mechanical.

I updated my download test script to compare 0.15.12, 0.16.0a12 and 0.16.0a15 and 0.16.0a12 matched the 0.15.12 performance:

Min 0.15 time 7.535127711016685 seconds
Min 0.16.0a12 time 7.166952276020311 seconds
Min 0.16.0a15 time 55.26884927297942 seconds

[erikos]

#638

[erikos]

Perfect, has been merged and I kicked off a new release of the plugin.

[erikos]

Ladybird 6.16-347

[vanessa-chang]

Verified Failed & It's a release blocker:

It fails to proceed the download after you select a content pack.
Only a white screen is shown with no progress bar.
The storage size remains the same, so there is no downloading in the background.

Test packages: Ladybird 6.16-347 & Windows 6.16.0.0

[manuq]

Yes, as Vanessa says, the application now shows a white screen right after Kolibri starts. In the logs I see a browser console error "get_download_status is not a function":

06-20 11:57:28.646  3699  3699 E EKWebConsole: TypeError: external_kolibriCoreAppGlobal_urls_default(...).kolibri:kolibri_explore_plugin:get_download_status is not a function -- Source: http://127.0.0.1:45249/static/kolibri.core.default_frontend/kolibri.core.default_frontend-0.16.0a15.js
06-20 11:57:28.647  3699  3699 I EndlessKey: mMainWebView loading finished http://127.0.0.1:45249/en/explore/#/
06-20 11:57:28.647  3699  3699 I PythonActivity: displayMainWebView

[manuq]

Oh but most important is this traceback error that happens before ModuleNotFoundError: No module named 'kolibri_explore_plugin.vendor:

06-20 11:57:22.755  3699  3909 W org.endlessos.Key: root: Tried to import module api_urls from kolibri_explore_plugin but an error was raised
06-20 11:57:22.759  3699  3909 E org.endlessos.Key: root: No module named 'kolibri_explore_plugin.vendor'
06-20 11:57:22.759  3699  3909 E org.endlessos.Key: Traceback (most recent call last):
06-20 11:57:22.759  3699  3909 E org.endlessos.Key:   File "/home/jenkins/workspace/kolibri-installer-android/src/kolibri/plugins/__init__.py", line 256, in _return_module
06-20 11:57:22.759  3699  3909 E org.endlessos.Key:   File "/home/jenkins/workspace/kolibri-installer-android/_cache/.local/share/python-for-android/build/other_builds/python3-liblzma/x86_64__ndk_target_21/python3/Lib/importlib/__init__.py", line 127, in import_module
06-20 11:57:22.759  3699  3909 E org.endlessos.Key:   File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
06-20 11:57:22.759  3699  3909 E org.endlessos.Key:   File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
06-20 11:57:22.759  3699  3909 E org.endlessos.Key:   File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
06-20 11:57:22.759  3699  3909 E org.endlessos.Key:   File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
06-20 11:57:22.759  3699  3909 E org.endlessos.Key:   File "<frozen importlib._bootstrap_external>", line 850, in exec_module
06-20 11:57:22.759  3699  3909 E org.endlessos.Key:   File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
06-20 11:57:22.759  3699  3909 E org.endlessos.Key:   File "/home/jenkins/workspace/kolibri-installer-android/_cache/.local/share/python-for-android/build/python-installs/kolibri/x86_64/kolibri_explore_plugin/api_urls.py", line 5, in <module>
06-20 11:57:22.759  3699  3909 E org.endlessos.Key:   File "/home/jenkins/workspace/kolibri-installer-android/_cache/.local/share/python-for-android/build/python-installs/kolibri/x86_64/kolibri_explore_plugin/collectionviews.py", line 24, in <module>
06-20 11:57:22.759  3699  3909 E org.endlessos.Key:   File "/home/jenkins/workspace/kolibri-installer-android/_cache/.local/share/python-for-android/build/python-installs/kolibri/x86_64/kolibri_explore_plugin/tasks.py", line 21, in <module>
06-20 11:57:22.759  3699  3909 E org.endlessos.Key: ModuleNotFoundError: No module named 'kolibri_explore_plugin.vendor'

[manuq]

A hotfix has been merged: #646

Deploying.

[manuq]

Deployed:

• kolibri-explore-plugin v6.17.0
• Android internal testers: Ladybird 6.17-348
• Windows alpha test flight: v6.17.0