Support pulling named subsets of data, or excluding files from pull

[r-zip]

I've been working on a large project with multiple datasets. One of these datasets is large (>100 GB). If I simply run dvc pull, then it will pull the huge dataset, which takes up most available disk space on my machine.

The only way around this appears to be providing the file name to every data file to download. This is inconvenient, however, because there are many files I do want, and only one that I don't want.

I see two solutions to this:

1. Allow named file groups. The user could specify groups of files in some sort of config, and pull them individually by name. I.e., dvc pull mnist. The user would also be able to exclude them: dvc pull all --exclude mnist.
2. Allow exclusion of certain files from the command line. I.e., dvc pull --exclude data/mnist.dvc.

[shcheklein]

To some extent related to this #2095 since one of the possible solutions to this can be specifying some setting per output in the DVC-file. Is it pulled/pushed by default, for example.

[Suor]

@shcheklein I don't think this is related, this is not about different remotes.

@r-zip There are two tricks you can employ:

1. Recursive dvc-file collection:

   dvc pull --recursive some_dir

   This will collect all dvc-files in some_dir and pull all their outs.

2. Use scripting, say bash files:

   In pull_123.sh:

   #!/bin/bash
   
   dvc pull stage1.dvc stage2.dvc stage3.dvc

   Then run:

   chmod +x pull_123.sh  # make it executable
   ./pull_123.sh         # pull stages 1,2,3

   You may also use makefiles or anything else you like.

[shcheklein]

@shcheklein I don't think this is related, this is not about different remotes.

@Suor yep, I think we are on the same page. It's very different. The only relation to the ticket is that potentially we can an option per output that specifies the default behavior for push/pull. Or it even can be a Null remote for some outputs? What do you think? Can it be a solution for this one?

[Suor]

@shcheklein I was thinking about remote: null option or not mentioning some out in any of the remotes. This might not what is expected for this issue.

[dberenbaum]

I think this can be closed now that there is --glob support in dvc pull. Feel free to reopen if you think it's still a valid feature request.

[pbailey-hf]

Hello. I'm looking for a solution to a similar problem. I don't see the --glob option in dvc pull for version 3.51.2. Has it been superseded?

[dberenbaum]

@pbailey-hf You can pull any subpath without --glob, like dvc pull data/subpath.