progress: inconsistent nested format when no main bar

[gurobokum]

Overview

When storage is local (copy_file) on dvc pull first pbar is displayed as normal one (self.pos = 0) and all others are nested

Description

When we do remote.download for all scheme except local there are main bar
So only first pbar is normal, and all others are nested

$ dvc push grand_files
 51% grand_files/1/file-1|█████████████████████████████████████████████                                           |164M/320M [00:14<00:07,    21.1MB/s]
 38%|███▊      |grand_files/1/another-file-1                                                                       122M/320M [00:14<00:18,    11.3MB/s]
 66%|██████▋   |grand_files/1/second-file-1                                                                        212M/320M [00:14<00:08,    13.0MB/s]
 35%|███▌      |grand_files/1/third-file

but expects that all pbars are as 1-pbar

Relates other commands as well:

$ dvc import ../repo grand_files
Importing 'grand_files (../repo)' -> 'grand_files'                      
 56% grand_files.9vFFJ8yBMA9QLmwq5PL7xj.tmp/1/second-file-1|██████████████████████████████                        |178M/320M [00:03<00:06,    23.8MB/s]
 54%|█████▍    |grand_files.9vFFJ8yBMA9QLmwq5PL7xj.tmp/1/another-file-1                                            174M/320M [00:03<00:06,    23.1MB/s]
 57%|█████▋    |grand_files.9vFFJ8yBMA9QLmwq5PL7xj.tmp/1/file-1                                                    181M/320M [00:03<00:06,    23.0MB/s]
 54%|█████▍    |grand_files.9vFFJ8yBMA9QLmwq5PL7xj.tmp/1/third-file-1     

$ dvc pull -j3 grand_files
 78% ../../../../../../../tmp/tmp3zld67wsdvc-erepo/grand_files/1/another-file-1|██████████████████████████▍       |249M/320M [00:04<00:03,    21.7MB/s]
 80%|████████  |../../../../../../../tmp/tmp3zld67wsdvc-erepo/grand_files/1/file-1                                 256M/320M [00:05<00:04,    15.2MB/s]
 93%|█████████▎|../../../../../../../tmp/tmp3zld67wsdvc-erepo/grand_files/1/second-file-1                          297M/320M [00:04<00:01,    21.5MB/s]

Relates:

• pull: going completely crazy on large dataset #2905

Please provide information about your setup

$ dvc --version
0.87.0+f05e6e

[efiop]

CC @casperdcl

[casperdcl]

@JIoJIaJIu not sure what the problem is - are you experiencing any unexpected behaviour?

[casperdcl]

Ah I see it's that the format of the first bar should be the same as the others (nested)...

[gurobokum]

Ah I see it's that the format of the first bar should be the same as the others (nested)...

I think we sometimes need to have nested bars and 1 main pbar, for example when compute cache we would like to see 1 main pbar - Computing file/dir.. and then large files are displayed as nested

I see the problem here is how we determine main bar
Also I've noticed that it's not guaranteed that main bar gets 0 position

[casperdcl]

I see the problem here is how we determine main bar

It's best to add a main bar in cases where there currently is no main bar (that would fix this issue)

Also I've noticed that it's not guaranteed that main bar gets 0 position

Really? When?

[gurobokum]

Really? When?

with the script you may reproduce the behavior

$ python pbar.py 10 --slow
  0% File#724.80|                         |0/100 [00:00<?,       ?it/s]
  3%|▎         |File#120.64            3/100 [00:02<01:16,    1.27it/s]
  0%|          |File#755.66                0/100 [00:00<?,       ?it/s]
  0%|          |File#756.29                0/100 [00:00<?,       ?it/s]
  0%|          |File#292.52                0/100 [00:00<?,       ?it/s]
  3%|▎         |File#117.65            3/100 [00:02<01:14,    1.31it/s]
  2%|▏         |File#164.46            2/100 [00:01<01:20,    1.21it/s]
  0%|          |File#340.28                0/100 [00:00<?,       ?it/s]
  0%|          |Downloading directory       0/10 [00:00<?,    ?Files/s]
  0%|          |File#377.57                0/100 [00:00<?,       ?it/s]
  0%|          |File#463.49                0/100 [00:00<?,       ?it/s]

we expect that Downloading directory appears on the top

[casperdcl]

That's a different issue... Opened #3521. Will fix separately.