progress: inconsistent nested format when no main bar

[gurobokum]

Overview

When storage is local (copy_file) on dvc pull first pbar is displayed as normal one (self.pos = 0) and all others are nested

Description

When we do remote.download for all scheme except local there are main bar
So only first pbar is normal, and all others are nested

$ dvc push grand_files
 51% grand_files/1/file-1|█████████████████████████████████████████████                                           |164M/320M [00:14<00:07,    21.1MB/s]
 38%|███▊      |grand_files/1/another-file-1                                                                       122M/320M [00:14<00:18,    11.3MB/s]
 66%|██████▋   |grand_files/1/second-file-1                                                                        212M/320M [00:14<00:08,    13.0MB/s]
 35%|███▌      |grand_files/1/third-file

but expects that all pbars are as 1-pbar

Relates other commands as well:

$ dvc import ../repo grand_files
Importing 'grand_files (../repo)' -> 'grand_files'                      
 56% grand_files.9vFFJ8yBMA9QLmwq5PL7xj.tmp/1/second-file-1|██████████████████████████████                        |178M/320M [00:03<00:06,    23.8MB/s]
 54%|█████▍    |grand_files.9vFFJ8yBMA9QLmwq5PL7xj.tmp/1/another-file-1                                            174M/320M [00:03<00:06,    23.1MB/s]
 57%|█████▋    |grand_files.9vFFJ8yBMA9QLmwq5PL7xj.tmp/1/file-1                                                    181M/320M [00:03<00:06,    23.0MB/s]
 54%|█████▍    |grand_files.9vFFJ8yBMA9QLmwq5PL7xj.tmp/1/third-file-1     

$ dvc pull -j3 grand_files
 78% ../../../../../../../tmp/tmp3zld67wsdvc-erepo/grand_files/1/another-file-1|██████████████████████████▍       |249M/320M [00:04<00:03,    21.7MB/s]
 80%|████████  |../../../../../../../tmp/tmp3zld67wsdvc-erepo/grand_files/1/file-1                                 256M/320M [00:05<00:04,    15.2MB/s]
 93%|█████████▎|../../../../../../../tmp/tmp3zld67wsdvc-erepo/grand_files/1/second-file-1                          297M/320M [00:04<00:01,    21.5MB/s]

Relates:

• pull: going completely crazy on large dataset #2905

Please provide information about your setup

$ dvc --version
0.87.0+f05e6e

[efiop]

CC @casperdcl

[casperdcl]

@JIoJIaJIu not sure what the problem is - are you experiencing any unexpected behaviour?

[casperdcl]

Ah I see it's that the format of the first bar should be the same as the others (nested)...

[gurobokum]

Ah I see it's that the format of the first bar should be the same as the others (nested)...

I think we sometimes need to have nested bars and 1 main pbar, for example when compute cache we would like to see 1 main pbar - Computing file/dir.. and then large files are displayed as nested

I see the problem here is how we determine main bar
Also I've noticed that it's not guaranteed that main bar gets 0 position

[casperdcl]

I see the problem here is how we determine main bar

It's best to add a main bar in cases where there currently is no main bar (that would fix this issue)

Also I've noticed that it's not guaranteed that main bar gets 0 position

Really? When?

[gurobokum]

Really? When?

with the script you may reproduce the behavior

$ python pbar.py 10 --slow
  0% File#724.80|                         |0/100 [00:00<?,       ?it/s]
  3%|▎         |File#120.64            3/100 [00:02<01:16,    1.27it/s]
  0%|          |File#755.66                0/100 [00:00<?,       ?it/s]
  0%|          |File#756.29                0/100 [00:00<?,       ?it/s]
  0%|          |File#292.52                0/100 [00:00<?,       ?it/s]
  3%|▎         |File#117.65            3/100 [00:02<01:14,    1.31it/s]
  2%|▏         |File#164.46            2/100 [00:01<01:20,    1.21it/s]
  0%|          |File#340.28                0/100 [00:00<?,       ?it/s]
  0%|          |Downloading directory       0/10 [00:00<?,    ?Files/s]
  0%|          |File#377.57                0/100 [00:00<?,       ?it/s]
  0%|          |File#463.49                0/100 [00:00<?,       ?it/s]

we expect that Downloading directory appears on the top

[casperdcl]

That's a different issue... Opened #3521. Will fix separately.

[casperdcl]

@JIoJIaJIu can you post some code for reproducing the import case? Also does it happen with get?

$ dvc import ../repo grand_files
Importing 'grand_files (../repo)' -> 'grand_files'                      
 56% grand_files.9vFFJ8yBMA9QLmwq5PL7xj.tmp/1/second-file-1|██████████████████████████████                        |178M/320M [00:03<00:06,    23.8MB/s]
 54%|█████▍    |grand_files.9vFFJ8yBMA9QLmwq5PL7xj.tmp/1/another-file-1                                            174M/320M [00:03<00:06,    23.1MB/s]
 57%|█████▋    |grand_files.9vFFJ8yBMA9QLmwq5PL7xj.tmp/1/file-1                                                    181M/320M [00:03<00:06,    23.0MB/s]
 54%|█████▍    |grand_files.9vFFJ8yBMA9QLmwq5PL7xj.tmp/1/third-file-1   

[gurobokum]

Hey @casperdcl, sorry for the delay with answer!

can you post some code for reproducing the import case

I was just using dvc with local storage

Steps for reproducing:

1. Create source_repo

   $ mkdir source_repo
   $ cd source_repo
   $ git init
   $ dvc init
   $ git commit -m 'init'
   

2. Setup local remote

   $ mkdir ../storage
   $ dvc remote add -d storage ../storage
   

3. Generate files with the script

   $ ./grand.sh grand_files
   $ dvc add grand_files
   $ git add grand_files.dvc
   $ git commit -m 'add grand_files'
   $ dvc push
   

   will show

   $ dvc push
   36% grand_files/1/second-file-1|█████         |116M/320M [00:11<00:43,    4.97MB/s]
   14%|█▍        |grand_files/1/another-file-1   45.0M/320M [00:11<00:58,    4.90MB/s]  
   86%|████████▌ |grand_files/1/third-file-1      275M/320M [00:11<00:10,    4.64MB/s]
   29%|██▉       |grand_files/1/file-1           93.0M/320M [00:12<00:26,    9.07MB/s]
   

4. Create target_repo

   $ cd ../
   $ mkdir target_repo
   $ cd target_repo
   $ git init
   $ dvc init
   $ git commit -m 'init'
   

5. Import the output

   $ dvc import ../source_repo/ grand_files 
   

Also does it happen with get?

$ dvc get ../source_repo/ grand_files
 21% grand_files.i6zUbRovXheDBpNh3kVkS9.tmp/1/another-file-1|▏|66.0M/320M [00:0
  9%|▉         |grand_files.i6zUbRovXheDBp28.0M/320M [00:02<00:47,    6.49MB/s]
  9%|▉         |grand_files.i6zUbRovXheDBp29.0M/320M [00:01<00:22,    13.6MB/s]
  6%|▌         |grand_files.i6zUbRovXheDBp19.0M/320M [00:00<00:36,    8.67MB/s]

Scripts for creating bulk of files