Display download progress for file downloads #2327

legoktm · 2024-12-09T23:50:37Z

Status

Work in progress

TODO:

Visual alignment issue; need padding on the right side
update apparmor stuff?
tests?

Description

Display a vanilla progress bar for file downloads that simply shows how much of the file has been downloaded so far.

A new FileDownloadProgressBar widget replaces the existing animated spinner, and we inject a signal through to the SDK to pass the current progress back to the widget.

Fixes #1104.

Test Plan

Spin up dev server; then inside the container run NUM_SOURCES=10 --random-file-size 100000 to generate 10 sources with 100MB attachments
Download them, including starting multiple parallel downloads and switching to other sources
Progress bar continues to update as expected, etc.

Checklist

These changes should not need testing in Qubes
- everything should be testable with the dev client, since no proxy changes are needed
I have updated the AppArmor profile
No database schema changes are needed

legoktm · 2024-12-10T00:34:48Z

So right now I've implemented an exponential moving average based on https://stackoverflow.com/questions/2779600/how-to-estimate-download-time-remaining-accurately; however I missed the comment which says, "EMA will only work if the time-sampling rate is about the same. If, for example the download is updated every 1 MB rather than every 1 second and the speed fluctuates the output will most likely be nonsense". Unfortunately that's exactly what we're doing, we update on chunks rather than on a regular time interval. Will require some more 🤔

legoktm · 2024-12-10T19:13:57Z

I got a smoothed version of the download speed to work (by adjusting the SMOOTHING_FACTOR to be relative to the time period), but there's a more practical problem: we only update the speed after we receive a chunk, so if it stalls, we don't update the speed and it'll get stuck at e.g. 4MB/s.

So I want to rework this a bit, I'll move the calculation stuff into the widget and we can use a QTimer to update the widget's speed on an actual time interval.

rocodes

Thanks so much for working on this @legoktm . I haven't run the code yet, just went through a fast visual review. If you're still looking for input on tests I can come back to that tomorrow :)

rocodes · 2024-12-18T20:50:54Z

client/securedrop_client/gui/base/progress.py

+    # One of:
+    # {"size": int}
+    # {"decrypting": True}
+    signal = pyqtSignal(dict)


I have a slight preference for a signal like pyqtSignal(int) where the size is always what's expected to be passed along. Along those lines, I wonder if it may be simpler to have "decrypting" ui behaviour fully separate from the download progress bar (eg: download progress is finished -> hide the progress bar and show a widget relevant to decryption).

Yeah, same. That's what I originally had but switched it to a dict to accommodate the decrypting signal.

I wonder if it may be simpler to have "decrypting" ui behaviour fully separate from the download progress bar (eg: download progress is finished -> hide the progress bar and show a widget relevant to decryption).

Do you think that other widget would be something other than a progress bar? Once we switch to Sequoia I expect we'll be able to calculate decryption progress just like we can with downloads now. At that point I imagine we could do something like the first 90% of the progress bar is download and the last 10% is decrypting/unzippping. Or just two progress bars, but either way I'd imagine it's the same underlying QProgressBar widget.

Certainly we can split it into two signals though; one for the size and then another for the "decrypting" state.

rocodes · 2024-12-18T20:53:30Z

client/securedrop_client/gui/base/progress.py

+        self.setObjectName("FileDownloadProgressBar")
+        self.setMaximum(file_size)
+        # n.b. we only update the bar's value and let the text get updated by
+        # the timer in update_speed


rocodes · 2024-12-18T20:55:39Z

client/securedrop_client/gui/base/progress.py

+        self.update_display()
+
+    def proxy(self) -> ProgressProxy:
+        """Get a proxy that updates this widget."""


This stuck out to me a bit, because it means that the widget, which is otherwise "naive"/encapsulated and needs no outside knowledge of the sdk or any components, now has an internal dependency on the ProgressProxy. What would you think about the widget either taking a ProgressProxy object in its constructor (clearer dependency graph +/ easier to test) or being completely agnostic to how it's receiving updates as long as it exposes a @pyqtSlot that takes an int parameter and updates itself accordingly?

# in the parent widget progressbar = FileDownloadProgressBar(self.file.size) proxy = ProgressProxy(progressbar.handle) self.controller.on_submission_download(File, self.file.uuid, proxy)

I wanted to encapsulate the construction of ProgressProxy in a single place so that if we e.g. wanted to add another signal, it only needs to be changed in one file. This isn't strictly true because we do have ProgressProxy(None) but it's close.

Also, my original design was that the SDK would have a Progress class that would be independent of PyQt but that didn't really work and I ended up needing the signal. What do you think if I moved the ProgressProxy class into the same file as the widget, since it's effectively tied together with that?

client/securedrop_client/logic.py

client/securedrop_client/sdk/progress.py

rocodes · 2024-12-18T21:10:38Z

client/tests/gui/test_widgets.py

@@ -3334,7 +3334,9 @@ def test_FileWidget_on_left_click_download(mocker, session, source):

    fw._on_left_click()
    get_file.assert_called_once_with(file_.uuid)
-    controller.on_submission_download.assert_called_once_with(db.File, file_.uuid)
+    # Because the ProgressProxy is created dynamically and not retained, we can't assert
+    # the specific value of it, so use ANY.


(This is true - optionally you could inspect the call_args and ensure that the thing that was passed in the 3rd arg is of type ProgressProxy)

rocodes · 2024-12-18T21:16:15Z

client/securedrop_client/sdk/__init__.py

@@ -328,9 +334,12 @@ def _send_json_request(
        if self.development_mode:
            env["SD_PROXY_ORIGIN"] = self.server

+        if not progress:
+            progress = ProgressProxy(None)


This is a dummy progressproxy that doesn't send update signals anywhere, in order to satisfy the signature of _streaming_download ?

Since there are other places where you have ProgressProxy | None, and this could be a bit confusing, I wonder about at minimum adding a comment to explain, but potentially changing the signature so that the sdk is always guaranteed to be passed a progressproxy (and therefore punting the decision strictly to the ui), or doing the opposite and accepting Proxy | None all the way through the sdk. No strong feelings just thinking aloud.

Yeah. It's definitely inconsistent, my thinking was that _streaming_download is already such a complicated and nested function that let's not add anymore complexity to it...but I was really just saving:

if progress: progress.set_value(bytes_written)

which isn't really that bad, vs having the extra load of inner being None.

rocodes · 2024-12-18T21:17:59Z

client/securedrop_client/logic.py

@@ -842,6 +850,7 @@ def _submit_download_job(
            job.success_signal.connect(self.on_file_download_success)
            job.failure_signal.connect(self.on_file_download_failure)

+        job.progress = progress


I was trying to figure out if there was any way of logically gating this (eg if it's a db.File then it always should have a non-null progress or there's an error worth logging), but that may not be true.

Right, I think we just treat it as ProgressProxy | None all the way through until we actually call methods on it.

Display a vanilla progress bar for file downloads that simply shows how much of the file has been downloaded so far. A new FileDownloadProgressBar widget replaces the existing animated spinner, and we inject a signal through to the SDK to pass the current progress back to the widget. The widget both displays the overall total progress as a percentage and also calculates the download speed by using an exponential moving average to smooth it out. A timer runs every 100ms to recalculate the speed. A new utils.humanize_speed() is used to translate the raw bytes/second into a human-readable version with a focus on keeping the length of the string roughly consistent so there's less visual shifting. Once the download has finished, an indeterminate progress bar is shown while decrypting since we don't (currently) have a way to report specific progress on that. TODO: * tests? Fixes #1104.

legoktm · 2024-12-20T02:08:03Z

Thanks for all the feedback @rocodes!

If you're still looking for input on tests I can come back to that tomorrow :)

Yes please, whenever you have time.

legoktm requested a review from a team as a code owner December 9, 2024 23:50

legoktm marked this pull request as draft December 9, 2024 23:50

legoktm mentioned this pull request Dec 9, 2024

Display download progress #1104

Open

legoktm force-pushed the download-progress branch from 41d1174 to ec75f0b Compare December 10, 2024 00:27

legoktm force-pushed the download-progress branch 2 times, most recently from cb0c67f to 1571296 Compare December 10, 2024 00:56

legoktm force-pushed the download-progress branch 4 times, most recently from 6ae75e9 to 0eb51f6 Compare December 12, 2024 18:23

rocodes reviewed Dec 18, 2024

View reviewed changes

legoktm force-pushed the download-progress branch from 0eb51f6 to 78020fc Compare December 20, 2024 02:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Display download progress for file downloads #2327

Display download progress for file downloads #2327

legoktm commented Dec 9, 2024 •

edited

Loading

legoktm commented Dec 10, 2024

legoktm commented Dec 10, 2024

rocodes left a comment

rocodes Dec 18, 2024

legoktm Dec 20, 2024

rocodes Dec 18, 2024

rocodes Dec 18, 2024

legoktm Dec 20, 2024

rocodes Dec 18, 2024

rocodes Dec 18, 2024

legoktm Dec 20, 2024

rocodes Dec 18, 2024

legoktm Dec 20, 2024

legoktm commented Dec 20, 2024

Display download progress for file downloads #2327

Are you sure you want to change the base?

Display download progress for file downloads #2327

Conversation

legoktm commented Dec 9, 2024 • edited Loading

Status

Description

Test Plan

Checklist

legoktm commented Dec 10, 2024

legoktm commented Dec 10, 2024

rocodes left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

legoktm commented Dec 20, 2024

legoktm commented Dec 9, 2024 •

edited

Loading