pull/fetch: progress bar is all over the console output

[natikgadzhi]

DVC version:

DVC version: 0.91.1
Python version: 3.7.7
Platform: Darwin-19.4.0-x86_64-i386-64bit
Binary: False
Package: brew
Supported remotes: azure, gdrive, gs, http, https, s3, ssh, oss
Cache: reflink - supported, hardlink - supported, symlink - supported
Repo: dvc, git

When I run dvc pull, dvc tries to show the progress bar for downloading data files, but it looks like there's a bug in rendering it.

Here's the video: https://share.getcloudapp.com/jkuKG6rp

Please let me know if I can provide more details on this or help in any way. I know this reproduces for one of my teammates, too, on Ubuntu with DVC 0.91.0.

[skshetry]

Hi, @xnutsive. Thanks for opening this issue.

The files that you are downloading are small, and hence looks flickering, and is a problem on the
fetch part.

#3500 set a threshold for showing maximum number of bars and hides them (it was more crazier before), but we still need to unify multiple bars into a single bar. This was done on upstream tqdm/tqdm#924.

Related/part-of #2905.

@casperdcl, I assume this is going to be a bit nicer with tqdm/tqdm#930. I notice that the bars are getting placed randomly on each render making it look flickery?

[casperdcl]

The ordering isn't random - the screen is actually "full" of bars "in order." To maintain the order, the whole screen below a bar is cleared when said bar completes, and the cleared bars are moved upwards. However, some (most) are randomly skipping display for speed purposes.

tqdm/tqdm#930 will actually fix this by changing to random ordering. This means finished bars get replaced without clearing others.

(disclaimer: this is a simplified explanation; in practice it's a little more complicated because concurrency + UI + cross-platform support + speed optimisation = 4 x hell more fun)

[natikgadzhi]

Experience-wise (please take this with a grain of salt, I'm not by any means a serious ML practitioner), a single pbar would be nicer — I don't really care how much of a specific image / text is downloaded, I just care about the whole data downloading / uploading process. I've read another issue where you discuss this and I imagine doing a single pbar is a ton of work though ;) Let me know if I can be useful in the meantime!

…

On Tue, Mar 31, 2020 at 11:52 PM, Casper da Costa-Luis < ***@***.*** > wrote: The ordering isn't random - the screen is actually "full" of bars "in order." To maintain the order, the whole screen below a bar is cleared when said bar completes, and the cleared bars are moved upwards. However, some (most) are randomly skipping display for speed purposes. tqdm/ tqdm#930 ( tqdm/tqdm#930 ) will actually fix this by changing to random ordering. This means finished bars get replaced without clearing others. (disclaimer: this is a simplified explanation; in practice it's a little more complicated because concurrency + UI + cross-platform support + speed optimisation = 4 x hell hard) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub ( #3565 (comment) ) , or unsubscribe ( https://github.com/notifications/unsubscribe-auth/AAAKU4IJX6UYETDADQ3QVLDRKLQDXANCNFSM4LYGK3HA ).

[casperdcl]

#3583 should mostly fix this issue.

@xnutsive I agree with your last comment - #3452 should be fixed by adding missing "main bars." The nested ones are relatively minor by comparison.