Possible quest processing performance enhancement

[smichel17]

When testing for #2794 (comment), I did a lot of manually downloading an area. One thing I noticed was that even when 100% of the tiles were cached already (and thus, completed immediately), the "searching" indicator was still spinning for a while (just over 2 seconds on an emulator on my desktop, split evenly between download and processing; presumably it would be quite a bit longer on an older phone). I assume this is downloading and/or processing the OSM data into quests.

If I search for quests two times in a row without panning the map at all, there are almost certainly no new quests. So, I wonder if it would be useful to skip the download and/or processing of some data in some situations like this.

[matkoniecz]

See 25cca13

Cache in such case is deliberately ignored, as requested in #2554

[mnalis]

I agree with @matkoniecz here - not just JOSM, but on the ground in Vespucci, or OsmAnd, or MapComplete etc. Forcing area to be downloaded manually should continue to do just that IMHO: forcefully re-download whole area, exactly because the user wants the latest data, even if it is maybe just seconds "stale".

The automatic downloader however (the one that works without users manually clicking Scan for quests here) should keep the cache as much as possible, absolutely.

[westnordost]

Which it does

[smichel17]

@mnalis I would probably agree with you, except that manual download is useful for more than just re-downloading an already-downloaded area to get the latest data; it is also the only way to download an area in advance— particularly for offline mapping and when you are on a train or other vehicle, where there is not time to wait for the automatic download. In these cases, I often want to overlap my bboxes so there's not gaps in the data, but I do not want to forcibly re-download data that I just downloaded seconds earlier (especially on a train, when the re-download wastes precious time and data).

[mnalis]

@smichel17 I agree that is useful additional functionality, and I myself am using it for that too, but how much do you overlap? I usually position map so the railroad "forward" faces "up" on the mobile, and then just press my finger at the top of the way (and screen) and drag it down to bottom, and then do rescan. I have not measured it precisely, but it looks like maybe 5% overlap for this (quickest I've found) method of downloading which guarantees no data missed? IMHO any speed lost due to 5% bigger download is gained by it being extremely quick screen gesture, and is fastest to solve quests as it requires no mental calculations (what is left on the screen is left of you, etc).
If I was that concerned about that 5% data/time loss, I'd probably move it a little more after that, even if it means no overlap, so only 98% of the quests are captured instead of 100% (I've never managed to solve all of the quests on some route in the first try anyway).

But if that 5% is that important to you (or for some reason in your case it is closer to 50% or whatever), how would you propose to solve those two different use cases (one of which requires ignoring cache, and other requires using cache)?

Different download buttons for cached and non-cached download? Persistent checkbox "do not redownload if cached"? Keeping timestamp of downloaded data and only do force-redownload if more than 5 seconds passed since the last force-redownload? Remember last manual download bbox and substract if from new download bbox if difference is less than 10%?

Those all look like overkill to me (and/or do too much magic in the background which leads to unexplainable behaviors which are bad UX), but nothing better comes to my mind.

[smichel17]

Remember last manual download bbox and subtract it from new download bbox if difference is less than 10%?

Something like this. It is hard to put into words, so I'll use a picture instead:

• The index card represents the new manual download bbox which the user has just requested.
• The overlapping, penciled-in rectangles represent bboxes which have been downloaded automatically or manually in the last 10 minutes.

The strategy is:

• If less than 75% of the index card is penciled in, attempt to skip the download for penciled-in areas.
  • This would work better if the split is just one bbox overlapping. I've shown a more complex case, where there were some smaller automatic downloads and then the user zoomed out and requested a bigger one; in that case, we'd just proceed with the whole download instead. Persisting the already-recently-downloaded data and re-generating the quests for it could still be skipped, however.
• If more than 75% of the index card is penciled in, re-download the entire box. Also, in the main menu, instead of ":mag: Search for quests", show the item as ":arrows_counterclockwise: Force re-scan for quests" (or something like that). The idea is to give the user some indication that they've already downloaded this area and they're unlikely to find anything new by doing it again.
  • If 100% of the index card is penciled in, re-download all elements but only persist elements which have changed in the last 10 minutes.

After spending the time to write this out, I think the proposal is probably still too complex. I think there might be a simpler version that still maintains some of the benefit (maybe just the last/4th bullet point?), but I don't want to spend any more time on it right now. It's not a top priority for me, just an idea I had for improvement.

[smichel17]

In #3235 (comment), I wrote:

It's funny, when we were using the nextzen tiles which were years out of date, we got relatively few big reports of this type. Now that the map tiles are updated quickly, people seem much more likely to assume that means they are updated instantly.

When @Helium314 referenced it in #3260 (comment), it got me thinking again: it seems like the vast majority (if not 100%) of these reports come from people who are looking for their own edits in a different editor to show up quickly. This makes sense, because someone who uses only StreetComplete won't even know that some OSM data was updated.

…which got me thinking about this thread: @mnalis, you said, the user wants the latest data, even if it is maybe just seconds "stale". Do you think that's true of all users, or just those who also edit OSM with other editors? If the latter, maybe "unconditionally fetch the latest data" is a feature better suited for #3003 ("StreetComplete for experienced mappers").

[mnalis]

I would think vast majority of users would want that to reload data, yes. Just like when you press F5 in browser, you do it in most cases because you want to browser to redownload data. For SC, it might be because you are editing with other editor too, it might be because you're using Team mode and somebody split the way, or you might want it just because the quests were downloaded some time ago and you suspect they're stale (or you just want to make sure they aren't stale, for example because you have no mobile data on your device), etc.

If the area already has data, I do not see much users clicking manually on button to (re)download the area unless they really want to redownload it (and if they don't have data in that area already, it does not make any difference for the download strategy, as there is no cache to use anyway). Just as users don't usually randomly press F5 in web browser unless they feel the need to try to reload webpage (for whatever reason).