`pull`: `-R` does not check immediate target

[mattseddon]

Bug Report

Description

Firstly, from the docs I realise that pull -R <target> is probably working exactly as advertised.

In the VS Code extension, we show a tracked tree which can be used to selectively pull files from the remote.

We currently use the output of dvc list . -R --show-json --dvc-only to generate this tree (we will shortly be using the output from the new data:status command). We mark everything provided by the list output as tracked.

When calling pull against these tracked paths we check to see if the path exists in the list output. If it does then we call dvc pull <target>. If it does not we call dvc pull -R <target>.

When calling dvc pull -R we get mixed results. Here is an example of -R stating that everything is up to date when things clearly haven't changed:

Screen.Recording.2022-05-17.at.3.35.58.pm.mov

dvc.yaml for the above project is here. training_metrics is tracked but there is no way currently for us to easily/consistently tell this from the combined output of list, status & diff.

Reproduce

1. Open demo project for the first time.
2. Run dvc pull -R training_metrics from the root.
3. “everything is up to date” will be returned by the command
4. No data will have been updated.

Expected

dvc pull -R target checks the target as well as all searching inside the target.

We could take the alternative approach of including the appropriate information in the new data:status command. I.e training_metrics/ would be provided as part of the output to let us know that it is tracked.

Environment information

Output of dvc doctor:

$ dvc doctor
DVC version: 2.10.2 (pip)
---------------------------------
Platform: Python 3.9.9 on macOS-12.3.1-x86_64-i386-64bit
Supports:
        webhdfs (fsspec = 2022.3.0),
        http (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
        https (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
        s3 (s3fs = 2022.3.0, boto3 = 1.21.21)
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk1s5s1
Caches: local
Remotes: s3
Workspace directory: apfs on /dev/disk1s5s1
Repo: dvc (subdir), git

Additional Information (if any):

Please let me know if you need anything else from me. Thanks

[skshetry]

Related: #5326

EDIT: not related, it's a bit different.

[dberenbaum]

Reproduce

1. Open demo project for the first time.

2. Run `dvc pull -R training_metrics` from the root.

3. “everything is up to date” will be returned by the command

4. No data will have been updated.

I'm either unable to reproduce or misunderstanding:

$ git clone git@github.com:iterative/vscode-dvc.git
Cloning into 'vscode-dvc'...
remote: Enumerating objects: 28550, done.
remote: Counting objects: 100% (196/196), done.
remote: Compressing objects: 100% (132/132), done.
remote: Total 28550 (delta 87), reused 106 (delta 56), pack-reused 28354
Receiving objects: 100% (28550/28550), 10.73 MiB | 32.90 MiB/s, done.
Resolving deltas: 100% (20665/20665), done.
dave@davids-air:/tmp 11:23:05
$ cd vscode-dvc/demo
$ dvc pull -R training_metrics
A       training_metrics/
1 file added and 2 files fetched
$ dvc doctor
DVC version: 2.10.3.dev54+g570e3a535
---------------------------------
Platform: Python 3.9.2 on macOS-12.3.1-arm64-arm-64bit
Supports:
        azure (adlfs = 2022.2.0, knack = 0.9.0, azure-identity = 1.8.0),
        gdrive (pydrive2 = 1.10.1),
        gs (gcsfs = 2022.2.0),
        hdfs (fsspec = 2022.2.0, pyarrow = 7.0.0),
        webhdfs (fsspec = 2022.2.0),
        http (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
        https (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
        s3 (s3fs = 2022.2.0, boto3 = 1.20.24),
        ssh (sshfs = 2021.11.2),
        oss (ossfs = 2021.8.0),
        webdav (webdav4 = 0.9.5),
        webdavs (webdav4 = 0.9.5)
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk3s1s1
Caches: local
Remotes: https
Workspace directory: apfs on /dev/disk3s1s1
Repo: dvc (subdir), git

[mattseddon]

@dberenbaum in order to test this I upgraded from 2.10.2 to the latest version of dvc from main. It appears that the files in our subrepo have been renamed:

I cannot get the repo out of this renamed state. I have tried:

1. dvc pull
2. dvc commit
3. dvc commit -f
4. dvc exp run

Still stuck here:

Reverting to 2.10.2 brings things back to normal. I can also recreate on a fresh clone.

> dvc doctor            
DVC version: 2.10.3.dev66+g71a4b482 
---------------------------------
Platform: Python 3.9.9 on macOS-12.3.1-x86_64-i386-64bit
Supports:
        webhdfs (fsspec = 2022.3.0),
        http (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
        https (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
        s3 (s3fs = 2022.3.0, boto3 = 1.21.21)
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk1s5s1
Caches: local
Remotes: https
Workspace directory: apfs on /dev/disk1s5s1
Repo: dvc (subdir), git

[shcheklein]

Reading the description one more time it might be a bigger issue for DVC extension. -R semantics is clearly can lead that if there is a dvc.yaml inside that directory that describes a dataset outside we'll also pull it. And, yes this semantics mean that we can't do dvc pull -R target where target is a DVC-tracked directory.

1. is there a workaround on the extension side?
2. DVC product question - how consistent -R is across commands?
3. how useful this semantics vs looking for outputs inside the specified directory? (can we find some tickets where we can find the motivation? when we first introduced this?).

My guess would be that it is as old as dvc pull accepting as a target only .dvc files. (I would double check this). If were adding -R before expanding targets to .dvc + actual paths - it makes sense to me, and we can reconsider -R (or introduce --recusrsive?)

[mattseddon]

@efiop looks like the change that introduce the renamed issue was this one: a80a85e.

Was there anything in there that would make a subrepo exhibit this diff no matter what (~/vscode-dvc is the git repo, ~/vscode-dvc/demo is the dvc project):

~/vscode-dvc/demo ❯  dvc diff
Renamed:                                                              
    demo/data/MNIST/raw/ -> data/MNIST/raw/
    demo/data/MNIST/raw/t10k-images-idx3-ubyte -> data/MNIST/raw/t10k-images-idx3-ubyte
    demo/data/MNIST/raw/t10k-images-idx3-ubyte.gz -> data/MNIST/raw/t10k-images-idx3-ubyte.gz
    demo/data/MNIST/raw/t10k-labels-idx1-ubyte -> data/MNIST/raw/t10k-labels-idx1-ubyte
    demo/data/MNIST/raw/t10k-labels-idx1-ubyte.gz -> data/MNIST/raw/t10k-labels-idx1-ubyte.gz
    demo/data/MNIST/raw/train-images-idx3-ubyte -> data/MNIST/raw/train-images-idx3-ubyte
    demo/data/MNIST/raw/train-images-idx3-ubyte.gz -> data/MNIST/raw/train-images-idx3-ubyte.gz
    demo/data/MNIST/raw/train-labels-idx1-ubyte -> data/MNIST/raw/train-labels-idx1-ubyte
    demo/data/MNIST/raw/train-labels-idx1-ubyte.gz -> data/MNIST/raw/train-labels-idx1-ubyte.gz
    demo/misclassified.jpg -> misclassified.jpg
    demo/model.pt -> model.pt
    demo/predictions.json -> predictions.json
    demo/training_metrics.json -> training_metrics.json
    demo/training_metrics/ -> training_metrics/
    demo/training_metrics/scalars/acc.tsv -> training_metrics/scalars/acc.tsv
    demo/training_metrics/scalars/loss.tsv -> training_metrics/scalars/loss.tsv

I can see that scmrepo got bumped from 0.0.19 to 0.0.22 in that commit.

LMK if you want a separate issue for this.

Thanks.

[dberenbaum]

@mattseddon Thanks, I can confirm the renaming issue.

I'm still not following the original issue since it seems that pull -R is working as expected for me. Can you clarify?

[mattseddon]

I can confirm that #7756 (comment) is correct (did not work for 2.10.2 but since fixed).

In testing I think have found a new issue where if a .dvc files is present within a directory then -R will pull that file but not any other directory deps/outs available within that directory.

Using example-get-started:

~/example-get-started master ❯ tree data/ 
data/
├── data.xml
├── data.xml.dvc
├── features
│   ├── test.pkl
│   └── train.pkl
└── prepared
    ├── test.tsv
    └── train.tsv

2 directories, 6 files
~/example-get-started master ❯ rm -rf data/features  
~/example-get-started master ❯ rm -rf data/prepared
~/example-get-started master ❯ rm data/data.xml
~/example-get-started master ❯ tree data/ 
data/
└── data.xml.dvc

0 directories, 1 file
~/example-get-started master ❯ dvc pull -R data
A       data/data.xml                                                                                                                                
1 file added
~/example-get-started master ❯ tree data/  
data/
├── data.xml
└── data.xml.dvc

0 directories, 2 files

~/example-get-started master ❯ dvc doctor
DVC version: 2.10.3.dev71+gf23d31af 
---------------------------------
Platform: Python 3.9.9 on macOS-12.4-x86_64-i386-64bit
Supports:
        webhdfs (fsspec = 2022.1.0),
        http (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
        https (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
        s3 (s3fs = 2022.1.0, boto3 = 1.20.24)
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk1s5s1
Caches: local
Remotes: https
Workspace directory: apfs on /dev/disk1s5s1
Repo: dvc, git

Does this make sense? Is this the intended behaviour of -R?

[shcheklein]

@mattseddon yes, that's expected, it's not a bug. This is the current behavior of the -R . I've tried to explain it here. Now we need to find a workaround with the DVC team.

[shcheklein]

@dberenbaum

I'm still not following the original issue since it seems that pull -R is working as expected for me. Can you clarify?

It's not a bug, but behavior might be confusing to be honest. But even that doesn't matter - we need to find a way to all tracked files inside a directory. -R gives something completely different (good question how useful it is, but it's a separate topic). Question - do we have some workaround for the behavior we are looking for?

[dberenbaum]

For VS Code, can you use dvc list --dvc-only to get the appropriate targets and pass those to dvc pull?