When a sync is in progress, the syncsession can be prematurely closed if zeroconf indicates the instance has left the network

[rtibbles]

Observed behavior

The Android App runs its syncs in an independent task runner managed by the Android Work Manager library - this means it is possible for the sync to start running while the app is open, but then for the app to be closed.

If this is the case then zeroconf will shutdown during the sync, causing the server to think the instance is no longer syncing and try to cleanup the active sync session. This will cause errors for the syncing client, as it will receive a 404 for its active sync session.

Errors and logs

From the app side:

INFO 2023-11-02 13:17:57,015 kolibri.core.content.kolibri_plugin Processing content requests for synced dataset: dadee434baef8f7ee819e715c5b9cb9f
INFO 2023-11-02 13:17:57,017 kolibri.core.content.utils.content_request Processing new content assignment requests
INFO 2023-11-02 13:17:57,028 kolibri.core.content.utils.content_request Processing new content removal requests
INFO 2023-11-02 13:17:57,036 kolibri.core.content.kolibri_plugin Completed content requests for synced dataset: dadee434baef8f7ee819e715c5b9cb9f
ERROR 2023-11-02 13:17:57,380 morango.sync.session HTTPError Reason: (no response)
ERROR 2023-11-02 13:17:57,382 morango.sync.controller 404 Client Error: Not Found for url: http://192.168.100.251:8080/api/morango/v1/transfersessions/94b8e7be25d54207949f21ef8f25914e/
ERROR 2023-11-02 13:17:57,394 kolibri.core.device.soud [user=2389bbe0d3cf341764625349a5c594e8] [server=6a40d55608b125e4a9e45fb854476922] Critical error occurred during syncing: Stage `cleanup` failed
ERROR 2023-11-02 13:17:57,466 kolibri.core.device.soud Stage `cleanup` failed
Traceback (most recent call last):
  File "/home/runner/work/kolibri-installer-android/kolibri-installer-android/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/dist/morango/sync/controller.py", line 254, in _invoke_middleware
  File "/home/runner/work/kolibri-installer-android/kolibri-installer-android/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/dist/morango/registry.py", line 228, in __call__
  File "/home/runner/work/kolibri-installer-android/kolibri-installer-android/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/dist/morango/sync/operations.py", line 942, in __call__
  File "/home/runner/work/kolibri-installer-android/kolibri-installer-android/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/dist/morango/sync/operations.py", line 1726, in handle
  File "/home/runner/work/kolibri-installer-android/kolibri-installer-android/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/dist/morango/sync/operations.py", line 1355, in close_transfer_session
  File "/home/runner/work/kolibri-installer-android/kolibri-installer-android/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/dist/morango/sync/syncsession.py", line 475, in _close_transfer_session
  File "/home/runner/work/kolibri-installer-android/kolibri-installer-android/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/dist/requests/sessions.py", line 611, in delete
  File "/home/runner/work/kolibri-installer-android/kolibri-installer-android/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/dist/morango/sync/session.py", line 77, in request
  File "/home/runner/work/kolibri-installer-android/kolibri-installer-android/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/dist/morango/sync/session.py", line 66, in request
  File "/home/runner/work/kolibri-installer-android/kolibri-installer-android/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/dist/requests/models.py", line 960, in raise_for_status
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://192.168.100.251:8080/api/morango/v1/transfersessions/94b8e7be25d54207949f21ef8f25914e/

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/runner/work/kolibri-installer-android/kolibri-installer-android/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/core/device/soud.py", line 363, in execute_sync
  File "/home/runner/work/kolibri-installer-android/kolibri-installer-android/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/dist/django/core/management/__init__.py", line 131, in call_command
  File "/home/runner/work/kolibri-installer-android/kolibri-installer-android/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/dist/django/core/management/base.py", line 330, in execute
  File "/home/runner/work/kolibri-installer-android/kolibri-installer-android/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/core/tasks/management/commands/base.py", line 28, in handle
  File "/home/runner/work/kolibri-installer-android/kolibri-installer-android/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/core/auth/management/commands/resumesync.py", line 57, in handle_async
  File "/home/runner/work/kolibri-installer-android/kolibri-installer-android/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/core/auth/management/utils.py", line 437, in _sync
  File "/home/runner/work/kolibri-installer-android/kolibri-installer-android/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/core/auth/management/utils.py", line 569, in _pull
  File "/home/runner/work/kolibri-installer-android/kolibri-installer-android/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/dist/morango/sync/syncsession.py", line 677, in finalize
  File "/home/runner/work/kolibri-installer-android/kolibri-installer-android/python-for-android/build/python-installs/kolibri/arm64-v8a/kolibri/dist/morango/sync/syncsession.py", line 630, in proceed_to_and_wait_for
  File "<string>", line 2, in raise_from
morango.errors.MorangoError: Stage `cleanup` failed

From the server side:

INFO 2023-11-02 15:17:56,882 kolibri.core.discovery.utils.network.broadcast Kolibri instance 'a09f2c50b5dc9bb3e9d0e7c4a4d61359' has left the zeroconf network.
INFO 2023-11-02 15:17:57,280 morango.management.commands.cleanupsyncs TransferSession 1 of 1: deleting 0 Buffers and 0 RMC Buffers...
INFO 2023-11-02 15:17:57,287 morango.management.commands.cleanupsyncs Closing SyncSession 1 of 1

Full logs available here: learningequality/kolibri-installer-android#171 (comment)

Expected behavior

The server should not clean up while syncs are active! It was suggested by @bjester that we could perhaps reduce the aggressiveness of this behaviour, and instead rely on #11101 to keep the sync sessions in check.

User-facing consequences

Breakage of syncing!

Steps to reproduce

Setup an Android device as an LOD.
Exit the app while syncing is in progress.

[pcenov]

@rtibbles adding here the logs of today's regression testing on Blaine's PR #11489 as syncing of the Android LODs kept breaking and seems to be caused by the issue described above:

https://drive.google.com/drive/folders/1TU0snLOp1cWg801To7ypNp1_HkshMg3F?usp=sharing