[feature/se-concurrency] Sync Engine concurrency support

[felix-schwarz]

Description

This PR adds Sync Engine concurrency support to the app, so that more than one Sync Action (like move, copy, delete, download, upload, …) can be executed at once. To enable this, the Sync Engine:

• builds a tree of dependencies between the scheduled actions by putting them in Sync Lanes
• performs actions without dependencies on other actions immediately
• queues actions that depend on the completion of preceding actions and performs them when it is their turn

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Test plan:

https://github.com/owncloud/QA/blob/master/Mobile/iOS-app/Concurrency.md

Bugs & improvements:

• (1) Bottom progress bar gets crazy [feature/se-concurrency] Sync Engine concurrency support #369 (comment) [FIXED]
• (2) Countdown in progress message [feature/se-concurrency] Sync Engine concurrency support #369 (comment)

[felix-schwarz]

(1) should be addressed by d744a44 with the following changes:

• progress objects of the same type are now automatically grouped and summarized like this:

• more recent progress objects (and groups) are now prioritized over older ones (it was the other way round before), so that if you're uploading 100 files, it initially shows the summary, but will switch to the progress info of whatever you decided to do after that for as long as that other action is ongoing.

• looks like the code submitting the progress of downloads was removed at some point. Looking at adding the missing lines back in now.

[felix-schwarz]

Re (1) again: re-added the missing lines, so download progress is now once again shown in the progress summary bar

[jesmrec]

(2)

With the last update related the behaviour of the progress bar, i notice this:

Using the default budget:

OCSyncActionCategoryTransfer : @(6),	// Limit total number of concurrent transfers to 6
OCSyncActionCategoryUpload   : @(3),	// Limit number of concurrent upload transfers to 3
OCSyncActionCategoryDownload : @(3)	// Limit number of concurrent download transfers to 3

1. Tap on seven files to download
2. Three of them start to download, remaining four enqueued -> Downloading 7 of 7 files
3. Two of them finish -> Downloading 5 of 7 files
4. From the file list add three extra files to download

Current: Downloading 8 of 10 files. But in the list, there are 8 files instead of 10. When everything is finished, new transfers trigger a new countdown.

Suggested (more ideas welcome):

if the progress message is Downloading X of Y files:

X -> ongoing transfers (maximum 6 for the default budget, or separate for uploads/downloads)
Y -> pending/non-finished transfers (number of cells in status view)

so, it counts only the pending stuff because the finished one is out of scope in terms of Status.

what do you think? CC @michaelstingl

[felix-schwarz]

Thanks for the feedback @jesmrec!

if the progress message is Downloading X of Y files:

X -> ongoing transfers (maximum 6 for the default budget, or separate for uploads/downloads)
Y -> pending/non-finished transfers (number of cells in status view)

so, it counts only the pending stuff because the finished one is out of scope in terms of Status.

I think that would be confusing to the average user and not provide much additional information. Most of the time it'd be Downloading 3 of Y files, maybe with brief fluctuations to Downloading 2 of Y files or Downloading 1 of Y files in-between as previous downloads finish and it takes a moment for new ones to be started.

There's also a mathematical problem with dropping the finished actions from the count. Imagine 5 transfers, with one at 90%, the rest at 0%. That'd result in an overall progress of 18% ((0.9 + 0 + 0 + 0 + 0)/5 = 0.18).

If all downloads then progress by 10%, the one with 90% hits 100% and is removed. The previously evenly advancing overall progress then suddenly drops from 18% to 10% ((0.1 + 0.1 + 0.1 + 0.1)/4 = 0.1), so the progress bar would jump around, which a progress bar should ideally never do.

That's why actions keep getting added to the count until the count is back to 1, at which point the progress of the last download can be shown directly.

I originally started out with messages following the pattern Uploading X files…, which is what f.ex. the macOS Finder does. It only shows the number of remaining files.

I then changed it to the current Uploading X of Y files… part out of curiosity how it'd feel, part because it helped me debug that feature. At this point, I'm still not fully convinced that change was/is a real improvement.

So with you bringing it up, I'd be totally fine to change it back to Uploading [number of remaining uploads] files… ("Finder style").

[jesmrec]

Ok, i got your idea and make sense to do it in that way. Let's give a try.

[jesmrec]

Approved on my side, correct application of concurrency taking in account the budgets.