support adding/transfering data straight to cache/remote

[efiop]

We often see people trying to use --external to add some big dataset that they have on an external drive, where they also have an external cache dir. People often do that because they can't/don't want to copy their data to dvc repo to dvc add it normally, e.g. because their HDD/SSD won't be able to physically fit two copies of that dataset.

Same thing with s3/gs/etc, where people want to just move data straight to their remote, without having to download/add/push it, because, again, it might not even fit on their local machine.

That's why it would be great to introduce feature(s) to be able to move(or copy) data straight to cache/remote from it's original location. Potentially this is not only useful for dvc add but also for dvc import[-url], where you want to use some data (e.g. through streaming with our API) in your project, that won't fit on your machine.

Related to #3920

[jorgeorpinel]

Would this also solve #3305?

[jorgeorpinel]

And yes, it seems like this feature could be valuable. It sounds like a "lightweight" data registry (see https://dvc.org/doc/use-cases/data-registries) for people to use DVC as an intermediary between data already hosted on cloud storage and other projects. So they can access the data with a unified interface (DVC CLI or API).

[efiop]

Would this also solve #3305?

No, it won't. These are unrelated. Unless I'm missing something.

[isidentical]

Hey @efiop, is this copy/tranfer operation will be on a new command or an already existing one?

Also, may I request 2 workflow command-group samples (some sort of small, self-contained replication of the whole process so that I can test one my local environment)? One is from the current workflow of people, where they try to do on 2 different local file system points, and one is from what they are going to execute when this issue will be resolved.

[efiop]

@isidentical I'm not a 100% sure if this might fit into some existing command, would love to hear your thoughts on it.

Sure, here are some examples. Please make sure you are familiar with our workflow already (see our get-started guide)

straight-to-remote scenario:

Imagine having some file/directory on s3(or other cloud) and you want to add it to your dvc repo and push to your remote (say it is an s3 remote too, say dvc remote add -d mys3 s3://bucket/dvc-remote), you would need to

aws s3 cp s3://bucket/path/to/data data
dvc add data
dvc push

but now what if data is so big that it doesn't fit as a whole on your machine? You won't be able to perform the same set of commands. DVC is able to pull partial data(e.g. if data is a diretory: dvc pull data/subdir) or provide you streaming access to the data on remote (e.g. if data is a file, you could use api.open/read) but we need to somehow shove that giant file into the remote first. So we need some way to put that data into remote without downloading it fully locally. Again have to stress that this new operation, when it is done, should leave the dvc repo in a state that is indistinguishable from that regular worklow described above with the exception that you won't have that data in the local cache or in your workspace, only present in the remote (you'll have data.dvc though, so if you ever move to another machine that can fit the whole file, dvc pull should be able to bring it and checkout it as normal). Regarding particular command name and semantics - it is up to you to research and suggest the CLI design for it 😉

straight-to-cache scenario:

Now imagine you have giant HDD and a small SSD that you keep your dvc repo(aka workspace) on. Since SSD is too small, you set up a dvc cache on HDD (see https://dvc.org/doc/use-cases/shared-development-server) (say /storage/dvc-cache) and use symlinks to link from cache into your local workspace (see https://dvc.org/doc/user-guide/large-dataset-optimization). Now say you have a giant dataset in /storage/some/path/data that you want to add to your repo as if you'd be able to:

cp /storage/some/path/data data
dvc add data

your first instinct might be to dvc add /storage/some/path/data but that file is outside of dvc repo, but dvc will complain. At this point people often misuse --external and dvc add /storage/some/path/data --extranal, but that will make /storage/some/path/data a part of your workspace (dvc will change it there) instead of having data in your dvc repo. So it would be nice to have a way to transfer that file to your workspace. The solution for this feature might be related to straight-to-remote solution, or it might be a separate command/flag, it is unclear right now. If you wish, we can focus on one straight-to-remote or straight-to-cache exclusively for now, to narrow the scope, but it is just useful to keep in mind that these might have a common solution/interface.

[isidentical]

As a command-line model, I am thinking something like this;

dvc transfer to-remote [-r] <from> <file>
dvc transfer to-cache <from>

# straigh-to-remote's example
dvc transfer to-remote s3://bucket/path/to/data data
# straight-to-cache's example
dvc transfer to-cache /storage/some/path/data