Support multiple external cache locations

[maxme1]

I have a huge dataset (bigger than my hard drive) but there are several HDDs in my machine. It would be cool if DVC could manage several external caches, and, for example, switch to a new one, when the previous is full (but keep track of the previous one, of course).

If I get it correctly, the core of DVC is a hash table stored on the disk, with very neat features, such as synchronization between different machines.
It seems that combining several such tables is pretty straightforward for all operations except insertion - this is the only place where you need to pick a specific table.
This could be implemented with a wrapper around LocalCache, which takes several instances and a selection strategy and implements the same interface.

Of course I may be oversimplifying things - I am not very familiar with your codebase. Just sharing some thoughts 😉

[efiop]

For the record: might be related to multiple remotes somehow #3371

[efiop]

Discord context https://discordapp.com/channels/485586884165107732/485596304961962003/738737246345101313

[hhoeflin]

I think this would also be interesting for groups working together, where you could have one/several (read-only) caches for large datasets and one (writable) active cache that the repo uses.

Is this something you are still considering?

[karajan1001]

I think this would also be interesting for groups working together, where you could have one/several (read-only) caches for large datasets and one (writable) active cache that the repo uses.

I think you can set the large datasets as remotes in this case.

For the original problem. Some of the file systems (for example zfs) can combine several HDDs to a pool and mount them to one single directory.

[rick-van-veen]

I think you can set the large datasets as remotes in this case.

@karajan1001 Could you explain this? Within the context of:

I think this would also be interesting for groups working together, where you could have one/several (read-only) caches for large datasets and one (writable) active cache that the repo uses.

[karajan1001]

I think you can set the large datasets as remotes in this case.

@karajan1001 Could you explain this? Within the context of:

I think this would also be interesting for groups working together, where you could have one/several (read-only) caches for large datasets and one (writable) active cache that the repo uses.

Looks like the problem comes from read-only, maybe the group can make this control on the cloud account side. A typical DVC repo actually consists one local (writable) active cache in the repo and one remote large cache to share.

[johnyaku]

Here is another use case for the concept of multiple caches.

We use several dvc registries to track raw data as it comes in. This allows us to ensure that data is properly tracked (via the dvc/git repo) and archived (via the dvc remote) immediately. The peace of mind that comes from this is priceless, especially with data streaming in from disparate sources.

Later, this raw data gets imported into datasets. This registry/dataset split is very useful, as it allows data to be recycled across projects (datasets) without having to duplicate it (and then having to track duplicate copies). As well as recycling, we can also trace data back to the original registries, which is good for reproducibility.

Another convenient part of our setup is a shared external cache. So long as the data is in the shared cache, we can check out a teammate's dataset in seconds, even if that dataset is many TB in size.

The problem comes with managing the shared cache. We have 65 TB to play with, but this is nowhere near enough to hold all of our data, and it shouldn't need to. Data that is not in the cache can always be downloaded from the remote, and so cache really only needs to hold "live" data. But this implies regular cache cleanups, and garbage collecting a shared cache is tricky.

Yes, we can protect other projects from accidental garbage collection with --projects but we have a LOT of projects and keeping this list up to date is challenging, and more than a little nerve-wracking. It might be convenient to be able to maintain a central, global list of projects that could be referenced by --projects, but that would be a separate feature request :)

Here what I'm suggesting is the utility of mutiple caches. In this scenario, the raw data (in the registries) would be cached in the shared external cache, so that imported data can be checked out almost instantly. But the processed data generated in a particular dataset would be cached in a local cache specific to that dataset. This way, the analysts can garbage collect the local caches associated with their own datasets whenever they like, without having to worry about affecting other projects. The shared external cache, on the other hand, would only contain raw data and so the data manager (me!) would be able to clean this shared cache without worrying about affecting everybody else.