dvc: provide granularity for commands that could target specific tracked files

[jorgeorpinel]

This can be seen as revisiting feature request #1026

UPDATE: Please scroll down to #2458 (comment) for most recent, summarized requirement.
Here is the original context also (still relevant):

---

There's different scenarios in which being able to manipulate files granularly independently of how they were committed/pushed to DVC could be useful. The problem with using dvc add -R now is that it can generate lots of .dvc files, but what if a directory could be added without -R (producing a single DVC-file) and yet other commands (lock, update, get, etc) could be applied to individual files inside the added directory tree?

Example (from treeverse/dataset-registry@7476a85)

Project 1:

$ tree
.
└── tutorial
    └── nlp
        ├── Posts.xml.zip
        └── pipeline.zip
$ dvc add tutorial
...
$ dvc push
...

Project 2:

$ dvc import {project-1-url} tutorial/nlp/pipeline.zip
...
$ tree
.
├── tutorial
│   └── nlp
│       └── pipeline.zip
└── tutorial.dvc

Not sure about where the .dvc would have to be placed in this example though.

And also this is how Git works, I believe. Files are tracked individually (in fact it doesn't even recognize empty dirs).

[pared]

@jorgeorpinel so, you propose storing particular files metadata inside stage file, instead of storing metadata for whole directory?

[jorgeorpinel]

I didn't think on the implementation details but that sounds reasonable... Unless there's thousand of files in there (which is possible): this could make it hard for Git to handle the DVC-file produced.

I would pull a "how does Git do it?" card here ♠️

[Suor]

A dir as a single entity is a desirable thing as I see it.

[pared]

Well I think that is a bit dangerous ground:

big directory == a lot of metadata entries

which can cause problem when loading it, and also, big metadata file could be problematic to handle by git.

[shcheklein]

@jorgeorpinel I agree with @Suor on this. In most cases the way ML projects are organized the current default behavior is desirable choice. And we have -R in case someone needs to override it.

It might make sense though to implement -R in a different way that allows for more granular control - may be repurpose the ticket for some research on this?

[efiop]

What kind of granularity (other than the already existing -R) are you guys talking about?

[shcheklein]

@efiop It reminds me the same discussion we had with some other users and with the team about partial checkouts, for example. Basically, there is a feeling that people are using -R in some cases when dvc add and a single DVC files it creates are not granular enough. But on the other hand -R is cumbersome with all these DVC-files it is creating - it's another painful issue.

[jorgeorpinel]

OK I repurposed this issue for exploring about providing more granular control of files in added dirs. (No longer related to -R I think)

@efiop

What kind of granularity... are you guys talking about?

Partial syncs like Ivan mentioned. Please see the description of this issue for an example.

[dashohoxha]

I would pull a "how does Git do it?" card here 

My impression is that the analogy between GIT and DVC is not very good, because they try to do different things (code management vs. data management). So, we should not strive to imitate git on everything.

[jorgeorpinel]

I did not say we should imitate Git on everything. What is your suggestion?

[dashohoxha]

@dashohoxha what is your suggestion?

I don't have any suggestion. I cannot even understand the problem. But if people are complaining about it, maybe there is some problem.

[efiop]

@shcheklein @jorgeorpinel Thanks for the explanation guys. Yeah, I remember those users that were asking about checking out or pulling only a particular number of files out of a directory. One of the big concerns at that time was that dvc was pretty slow with directories, which was significantly improved since then (still pretty long way to go though), so maybe it is acceptable now. But I can still see cases where someone would only want to pull one file out of the directory, and it seems to me that we could support that for most of the operations, we just need to do a few improvements to our logic.

For example, first step would be to support using paths instead of dvc files as arguments. E.g. dvc pull dir would be the same as dvc pull dir.dvc. After that, we will need to adjust the commands to understand subpaths, so that dvc pull dir/file would only download cache for file and checkout it.

So to summarize, in terms of the current architecture, granular operations are possible with dvc add dir(without -R), but we need to adjust particular dvc commands to understand granular arguments.

Related #2180