user-guide: document data sharing with unsupported storage types

[dashohoxha]

Since a remote cache is basically a mirror of the local cache, then any tool (not just DVC) can be used to synchronize them.

This can be another workaround for storage types that are not yet supported by DVC (besides mounting the remote storage on a local directory). Maybe this is not as efficient as DVC, and some things are not going to work (for example dvc status --cloud), but it still should allow people to share data.

Should we create a page (on UC or UG) that documents this workaround?

Related issues:

• dvc: use box.com as remote storage solution dvc#1940
• remote: support NFS dvc#1183
• Add support for ftp server dvc#1696

[shcheklein]

It feels like it's more or less obvious (after you read how DVC stores data) that cache (or the whole project )can be copied - to a remote machine, to another machine (the same way as you can use rsync to sync Git project). But I would not consider this as a workaround for unsupported remotes - it's too limited and breaks data granularity and basic DVC workflow - push/pull along with gc/status, dvc get\import, etc

We can mention this, but don't think it deserves a full page. Any suggestions how to frame this properly (considering that DVC support becomes quite limited) and where can we put this? May be just mention this in those tickets?

[dashohoxha]

I would not consider this as a workaround for unsupported remotes - it's too limited and breaks data granularity and basic DVC workflow - push/pull along with gc/status, dvc get\import, etc

Ok, let's not call it a "workaround for unsupported remotes", let's call it "sharing data with the help of a remote storage".
DVC can be used even without a remote storage, so I would say that commands like push and pull are not essential for its operation. This is the case for example with the "shared development machine".

I had in mind something like this as well: How can we share DVC data with NextCloud? Maybe I am wrong, but it seems to me that the first answer that comes to mind is this: NextCloud is not a supported DVC remote type, so this can't be done just yet, until support for NextCloud is added to DVC.

As a matter of fact this is wrong. If we store the cache directory on a NextCloud folder, it will be synchronized automatically with the server, and from the server with all the other clients. It is similar to the case of "shared development machine", but the synchronization is a bit slower, since it happens through the internet.

It doesn't seem to me that this solution is obvious.

This can be generalized for all the other storage types or cloud services. Even if they don't have some automatic synchronization features, it should be possible to use a cron job to synchronize them automatically to the server.

We can mention this, but don't think it deserves a full page. Any suggestions how to frame this properly (considering that DVC support becomes quite limited) and where can we put this?

I think it deserves a Use Case page. We can explain it for the NextCloud as an example, and then say that it can be generalized to all the other cloud services.

[shcheklein]

First of all, let me point that I'm still figuring out a better way to structure different ways of managing data with DVC, and ML in general. DVC gives so many different ways of organizing things that we need to come up with some simple rules and some common scenarios that will organize and help people navigate depending on the task they are trying to solve. So, all comments are not "precise", not "strong" opinions. It's just I don't see a good "full" picture yet.

This is the case for example with the "shared development machine"

This is not exactly true. It depends on your setup, of course. But in most cases you would use both - shared cache to utilize resource better and a remote to move data from one machine to another (to get an experiment or model to your local machine), to get model to production, etc.

If we store the cache directory on a NextCloud folder

And mount it? Why don't we mount a remote and use /mnt/remote-storage as a remote? - to have all the benefits - project's cache and remotes.

It doesn't seem to me that this solution is obvious.

I think I didn't understand the workaround itself you were suggesting. Not sure I understand it now.

I think it deserves a Use Case page.

I think there is sill some misunderstanding about this section. This is not a technical section, neither this section for describing workarounds, or how to solve a specific DVC-related staff (update DVC tracked file, etc). Titles in this section ideally should match workflow problems people/users are trying to solve even without knowing about DVC existence. So, in this case it should be sufficient to have a note in dvc remote or UG page + links to it from use cases, from dvc remote, etc.

On a side note - should we rename this section - to When DVC can help? or DVC applications to make it extra clear for everyone? I'm surprised a bit that "Use Case" name is triggering so technical low level understanding.

[dashohoxha]

And mount it? Why don't we mount a remote and use /mnt/remote-storage as a remote? - to have all the benefits - project's cache and remotes.

I think I get your point. You are right about getting all the benefits (being able to use all the features of DVC, including push/pull). In this case, no mounting is involved (although for practical purposes it looks like mounting). However if set up correctly this might actually be better than mounting.

This is one of the things that can be better demonstrated how it works, than explained with words. So, let me write an interactive Katacoda example about it (not right now, a bit later) and then we can continue discussing on it.

I think there is sill some misunderstanding about [Use Case] section. This is not a technical section, neither this section for describing workarounds, or how to solve a specific DVC-related staff (update DVC tracked file, etc). Titles in this section ideally should match workflow problems people/users are trying to solve even without knowing about DVC existence. So, in this case it should be sufficient to have a note in dvc remote or UG page + links to it from use cases, from dvc remote, etc.

My understanding of the Use Case section (described on this post: https://discuss.dvc.org/t/restructuring-docs/) is this:

I think that use cases are very helpful for the user because they reduce the amount of the possibilities and options that he has to consider in order to setup his workspace properly. So, we should consider adding a few more use cases which hopefully should cover all the possible ways that DVC can be setup and used. It is OK if there is some small overlap among the use cases (for example a particular situation can be covered by more than one use case).

If we have a comprehensive set of use cases, then for each query by the users about how to best setup DVC, we should be able to refer him to a use case that matches his situation. If we have to explain him the details of what he should do, then we should consider revising our use cases or adding a new one.

I don't claim that this is the best understanding or best definition, I am open to changing my mind about it.

[dashohoxha]

let me write an interactive Katacoda example about it

This is the best I could come up with Katacoda:

• https://katacoda.com/dvc/courses/examples/mounted-storage
• https://katacoda.com/dvc/courses/examples/synced-storage

I am not sure whether they make things perfectly clear, but I could not simulate a networking environment and had to explain everything using ssh://localhost as a server.

On a side note - should we rename this section (Use Cases) - to When DVC can help? or DVC applications to make it extra clear for everyone? I'm surprised a bit that "Use Case" name is triggering so technical low level understanding.

I think that we should dissolve the "Use Cases" and introduce the section "User Guide / HowTo".

[coroa]

This is the best I could come up with Katacoda:

* https://katacoda.com/dvc/courses/examples/mounted-storage

* https://katacoda.com/dvc/courses/examples/synced-storage

Thanks for these examples, they were already quite helpful to my understanding. I'd suggest to link to them from the user guide.

The technical limitation that everything sits on the same virtual server did not bother me at all.

A note on the synced-storage setup: For deduplication you are creating and mounting XFS images. Wouldn't it be a lot easier to use dvc's support for hardlinks to avoid the duplicates?