ls: optimize --dvc-only

[skshetry]

For dvc ls --dvc-only, we walk through all files and then filter those that are not dvc tracked files and list them. This could be optimized, and changed to just walk through dvc-tracked files from the trie we have.

[skshetry]

I decided to wait for fsspec changes. As discussed with @efiop, I like the idea of using listdir() to achieve this, which would be implemented with fsspec changes.

cc @isidentical

[skshetry]

It'd be great if we could clarify what --dvc-only means. Right now, it's not actually clear.

Let's say I have a data directory in modified state, tracked with dvc, having the following entries and their status:

1. data/foo (tracked and unchanged)
2. data/bar (tracked but now deleted)
3. data/foobar (recently added, not tracked, i.e. was not when dvc add was done previously)

When I do dvc list -R . --dvc-only, what should happen? Should I see all of them?
Similarly, what should happen if I do dvc list -R . data --dvc-only?

The following is the current behaviour:

$ dvc list -R . --dvc-only
data/bar
data/foo
data/foobar
$ dvc list . data -R --dvc-only
data/foo
data/foobar

cc @dberenbaum

[dberenbaum]

Is this a --dvc-only question? I think even without that argument, you will get data/bar in the first command and not in the second. I agree it's a consistency issue. Is it expected that tracked but deleted files ever show up in dvc list?

[mattseddon]

@skshetry, @dberenbaum,

We are planning to use this functionality for the VS Code extension. We will need the list of tracked files for at least two reasons:

1. Informing the Explorer view not to grey out these files / directories because they are not 'git ignored' (see Story: Match Git extension decoration for paths tracked by DVC vscode-dvc#207).
2. Knowing which files / directories to watch for changes so that we can run dvc status at the appropriate times and provide the status information to our SCM view (see Represent dvc status data in VS Code API vscode-dvc#169).

Ideally we would like to see all files / directories that are being tracked by DVC appear whenever running dvc list -R . --dvc-only, we do not currently care about untracked. We would like it to be as quick as possible (can we run a "local only" option).

Do you have a rough timeframe for when the initial optimisation work will happen?

Thanks,
Matt

[dberenbaum]

Good to know, @mattseddon! We didn't have this prioritized yet, but we can try to do so soon now that we know you need it. If you have any particular timeline in mind or relative priority with other vscode requests, let us know.

[mattseddon]

Thank you @dberenbaum. Within the next 8 weeks is probably soon enough for us. Will let you know if anything changes.

[mattseddon]

@dberenbaum got a further requirement for this one in that we(vscode-dvc)'d like to have the option to query against the remote / cache (as well) for files/dirs that are not on disk, see this comment for a better explanation and treeverse/vscode-dvc#176 for the why.

Can we track this request here or would you like a separate ticket?

[dberenbaum]

@mattseddon I'm not sure I follow what you're trying to do. You want to show files in the vscode file explorer view even though those files don't exist in the local workspace?

Do you actually want to list the contents of the remote, or is that a proxy for listing dvc-tracked files that are not in the workspace? If it's the latter, you should not need to access the remote. It might be better to keep this in a separate issue, but it does seem like the fix suggested by @skshetry might be a prerequisite to list every "expected" dvc-tracked file, even if that file is missing in the local workspace. Thoughts @skshetry?

[mattseddon]

@dberenbaum sorry for the confusion,

Do you actually want to list the contents of the remote, or is that a proxy for listing dvc-tracked files that are not in the workspace? If it's the latter, you should not need to access the remote.

To begin with the latter but we may need the former in the future.

After working with the command for a while I realised that @skshetry had already detailed most of what I was seeing in his earlier post. This comment details my exact findings.

It would be good to have the option to choose between showing all tracked in a specific folder and only those that are on disk. I.e dvc list . --dvc-only (shows all entries by default) vs dvc list . --dvc-only --local-only.

Let me know what you think.

Note: It seems that moving the -R before the . kind of gives me what I need but I will still have to take the immediate contents of the folder and dedup.

[skshetry]

@dberenbaum, --dvc-only is quite an ambiguous option, I am not quite sure what it should reflect, the committed version or the to-be committed one or mix of both.

To begin with the latter but we may need the former in the future.

Regarding the use in the vscode-dvc, maybe it should use dvc list to get the current state and then use dvc status to fill in the gaps?

We might need to add more granular support for that in status though. dvc list serves a different purpose, so trying to get the state of the repo might not be a good fit there.

[dberenbaum]

@skshetry I'm still confused about whether any of the issues you noted are about --dvc-only or more generally about dvc list. Do we have any background on the rationale behind --dvc-only? Is there some known scenario or internal use for it?

Regarding the use in the vscode-dvc, maybe it should use dvc list to get the current state and then use dvc status to fill in the gaps?

We might need to add more granular support for that in status though. dvc list serves a different purpose, so trying to get the state of the repo might not be a good fit there.

Using status might make sense since it seems that dvc list is intended to be like ls with some additional dvc-specific fields for the files listed. However, having to use two commands to get a full description of what dvc is tracking in the repo seems cumbersome, so we probably need one command that can do it.

[skshetry]

I'm still confused about whether any of the issues you noted are about --dvc-only or more generally about dvc list

@dberenbaum, yeah, I didn't realize that difference.

Do we have any background on the rationale behind --dvc-only?

This was never a problem before 2.0, as we always used to clone the repo and list from them, so there were no modified states. Initially, it was implemented as --outs-only, and I could find the following information in #2509 (comment).

On the other hand it can be good to show just a list of all DVC outputs. It can be done with dvc ls --recursive --outputs-only for example.

---

so we probably need one command that can do it.

We could extend the dvc status to list all files with their status respectively with status --granular or status --long-form or similar flags.